4.Hessian matrix

Recall the Calculus

1) One varialbe(변수 1개)

In calculus, the second derivative test(이계도 함수 판정법) is a criterion for determining whether a given critical point(임계점) of a real function of one variable is a local maximum or a local minimum using the value of the second derivative at the point.

The test states: if the function f is twice differentiable at a critical point x (i.e. f'(x) = 0), then:

• If  then  has a local maximum(극대) at .
• If  then  has a local minimum(극소) at .
• If , the test is inconclusive.

2) Two variable(변수 2개)

Suppose that $(x_{0}, y_{0})$ is a critical point of $f(x,y)$ and that $f(x,y)$ has continuous second-order parital derivatives in some circular centered at $(x_{0}, y_{0})$

Define D(xy) to be the determinant

$D(x,y)=f_{xx}(x_0,y_0)f_{yy}(x_0,y_0)-(f_{xy}(x_0,y_0))^2$,

Then the second partial derivative test asserts the following:

1. If  $D(x_{0},y_{0})>0$  and $f_{xx}(x_{0},y_{0})>0$  then $(x_{0},y_{0})$ is a local minimum(극소)of f.
2. If  $D(x_{0},y_{0})>0$ and $f_{xx}(x_{0},y_{0})<0$  then $(x_{0},y_{0})$is a local maximum(극대) of f.
3. If $D(x_{0},y_{0})<0$  then $(x_{0},y_{0})$ is a saddle point(안장점) of f.
4. If $D(x_{0},y_{0})=0$ then the second derivative test is inconclusive, and the point (ab) could be any of a minimum, maximum or saddle point.

Example) Locate the maxima, minima, and saddle points of the functions.

$f(x,y)=\frac{1}{3}x^3+xy^2-4xy+1$.

3) In 3 more variable,

How can we determine that those critical points are local minium or local maximum or saddle point?

We can think the matrix,

, called  as the Hessian matrix of $f(x,y)$.

Define $D(x,y)$ to be the determinant .

Suppose that $(x_{0},y_{0})$ is a critical point of $f(x,y)$ and that $f(x,y)$ has continuous second-order parital derivatives in some circular centered at $(x_{0}, y_{0})$

Then, we can rewrite the Second Derivative Test(이계도 함수 판정법) as following:

1. $f$ has a local minimum at $(x_{0},y_{0})$ if $H$ is positive definite.

2. $f$ has a local maximum at $(x_{0},y_{0})$ if $H$ is negative definite.

3. $f$ has a saddle point at $(x_{0},y_{0})$ if $H$ is indefinite.

4. The test is inconclusive otherwise.

Definition4

Let  $A=[a_{ij}] \in M_{n\times n}(\mathbb{R})$ be symmetric matrix and let $\rm{x}= (\it x_{1}, x_{2}, \cdots x_{n})\in \mathbb{R}^n$. Then :

• Positive definite(양의 정부호) if   $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}>\rm0$ for all nonzero $\rm{x}$.
x
  x  

• Positive semidefinite(양의 준정부호) if  $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}\geq \rm0$ for all $\rm{x}$.
• Negative definite(음의 정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}< \rm0$ for all nonzero $\rm{x}$.
• Negative semidefinite(음의 준정부호) if  $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}\leq \rm0$  for all $\rm{x}$.
• Indefinite(부정부호) if  $\rm{x}^{\it{T}}\it{A}\rm{x}$ takes both positive and negative values.

Theorem3

Let $A$ is a symmetric matrix, then:

•  $\rm{x}^{\it{T}}\it{A}\rm{x}$ is positive definite if and only if all eigenvalues of $A$ are positive.
•  $\rm{x}^{\it{T}}\it{A}\rm{x}$ is negative definite if and only if all eigenvalues of $A$ are negative.
•  $\rm{x}^{\it{T}}\it{A}\rm{x}$ is indefinite if and only if $A$ has at least one positive eigenvalue and at least one negative eigenvalue.

Example) For  $q(x,y)=2x^2-4xy+5y^2$ , determine the natrue of critical point $(0,0)$.

Sol)

The matrix $A$ of quadratic form is  $\begin{bmatrix} 2 &-2 \\ -2 & 5 \end{bmatrix}$  and its eigenvalues are $\lambda_1=6$ and $\lambda_2=1$

다음은 이변수함수에 관한 local extrema를 보여주는 interact입니다.(단, 임계점이 $2n\pi$의 형태면 구해지지 않음)

$\large \bigstar$ In general(일반적으로),

Given the real-valued function

if all second partial derivatives of $f$ exist and are continuous over the domain of the function, then the Hessian matrix of $f$ is

where x = (x1x2, ..., xn) and $D_{i}$ is the differentiation operator with respect to the th argument. Thus

The following test can be applied at any critical point (ab, ...) for the Hessian matrix

1. If the Hessian is positive definite (equivalently, has all eigenvalues positive) at (ab, ...), then f attains a local minimum(극소) at (ab, ...).
2. If the Hessian is negative definite (equivalently, has all eigenvalues negative) at (ab, ...), then f attains a local maximum(극대) at (ab, ...).
3. If the Hessian has both positive and negative eigenvalues then (ab, ...) is a saddle point(안장점) for f   (and in fact this is true even if (ab, ...) is degenerate)

Remark

The mixed derivatives(혼합 도함수) of f are the entries off the main diagonal in the Hessian. Assuming that they are continuous, the order of differentiation does not matter (Clairaut's theorem). For example,

:If the second derivatives of f are all continuous in a neighborhood D, then the Hessian of f is a symmetric matrix(대칭행렬) throughout D

Example) For the function  $f(x,y,z)=x^3+xyz+y^2-3x$ , find the critical points and determine whether the critical points are local maximum , local minimum or saddle point.

Sol)

The Hessian matrix of $f$ is  $\begin{bmatrix} 6x &z &y \\ z & 2 & x\\ y &x &0 \end{bmatrix}$ .

At  $(1,0,0)$, the eigenvalues of Hessian matrix are positive and negative values.

Hence, $(1,0,0)$ is saddle point.

At  $(-1,0,0)$,  the eigenvalues of Hessian matrix are positive and negative values.

Hence, $(-1,0,0)$  is saddle point.

Example) Locate the maxima, minima, and saddle points of the functions.

$f(x,y,z)=x^3 - 3y^3 + xz + z^2$

다음은 삼변수함수에 관한 local extrema를 보여주는 interact입니다.(단, 임계점이 $2n\pi$의 형태면 구해지지 않음)

Theorem4

The following are equivalent for a real symmetrix matrix $A$:

•  $A$ is positive definite
•  All the eigenvalues of $A$ are positive
•  All the leading pricipal submatrix $A_{k}$'s have positve determinants
•  All the pivots (without row interchanges) are positive
•  There exists a nonsigular matrix $W$ such that $A=W^{T}W$
•  There is an symmetric positive definite matrix $B$ such that $A=B^2$

There is sample of proof in case that $A$ is positive definite (Principal axes Theorem(2013.12.02), Cholesky Theorm(2013.12.02))

The following are equivalent for a real symmetrix matrix $A$:

•  $A$ is negaitive definite
•  All the eigenvalues of $A$ are negaitive
•  The determiant $\small (-1)^{k} \det\it (A_{k})>\rm 0$ for $\small 1\leq k\leq n$, i.e,  $\small \det(\it A_{1})<\rm 0, det(\it A_{2})>\rm 0, \cdots , (-1)^{\it n}det(\it A_{n})=(-1)^{n}\rm det(\it A)>\rm 0$
•  The quadratic forms associated to all $A_{k}$ are negative definite
•  All the pivots (without row interchanges) are negaitive
• We can prove other things using above the proof of positive definite.

The following are equivalent for a real symmetrix matrix $A$:

•  $A$ is positive semidefinite
•  All the eigenvalues of $A$ are nonnegative
•  All the leading pricipal submatrix $A_{k}$'s have nonnegative determinants
•  All the pivots (without row interchanges) are nonnegative
•  There exists a matrix $W$, possible singular, such that $A=W^{T}W$
• The followibg are equivalent for a real symmetrix matrix $A$:

•  $A$ is negative semidefinite
•  All the eigenvalues of $A$ are nonpositive
•  All the leading pricipal submatrix $A_{k}$'s have nonpositive determinants
•  All the pivots (without row interchanges) are nonpositive
•  There exists a matrix $W$, possible singular, such that $A=W^{T}W$

Reference)

Recall the calclus-one variable- Second derivative test, wikipedia, http://goo.gl/W11Os,2013.11.26

- two variable& in 3 more variable- Second partial deribvative test, wikipedia, http://goo.gl/R0LQMS,2013.11.26

In General, Information of Hessian matrix- Hessian matrix, wikipedia, http://goo.gl/s24ns, 2013.11.26

Definition 4, Theorem3 , Theorem4 and examples- Linear Algebra, Kwak&Hong, p294~p297.

Back TO the HOME