4.Hessian matrix -- Sage

Recall the Calculus

1) One varialbe(변수 1개)

In calculus, the second derivative test(이계도 함수 판정법) is a criterion for determining whether a given critical point(임계점) of a real function of one variable is a local maximum or a local minimum using the value of the second derivative at the point.

The test states: if the function f is twice differentiable at a critical point x (i.e. f'(x) = 0), then:

If $\ f^{\prime\prime}(x) < 0$ then $\ f$ has a local maximum(극대) at $\ x$ .
If $\ f^{\prime\prime}(x) > 0$ then $\ f$ has a local minimum(극소) at $\ x$ .
If $\ f^{\prime\prime}(x) = 0$ , the test is inconclusive.

2) Two variable(변수 2개)

Suppose that $(x_{0}, y_{0})$ is a critical point of $f(x,y)$ and that $f(x,y)$ has continuous second-order parital derivatives in some circular centered at $(x_{0}, y_{0})$ .

Define D(x, y) to be the determinant

$D(x,y)=f_{xx}(x_0,y_0)f_{yy}(x_0,y_0)-(f_{xy}(x_0,y_0))^2$ ,

Then the second partial derivative test asserts the following:

If $D(x_{0},y_{0})>0$ and $f_{xx}(x_{0},y_{0})>0$ then $(x_{0},y_{0})$ is a local minimum(극소)of f.
If $D(x_{0},y_{0})>0$ and $f_{xx}(x_{0},y_{0})<0$ then $(x_{0},y_{0})$ is a local maximum(극대) of f.
If $D(x_{0},y_{0})<0$ then $(x_{0},y_{0})$ is a saddle point(안장점) of f.
If $D(x_{0},y_{0})=0$ then the second derivative test is inconclusive, and the point (a, b) could be any of a minimum, maximum or saddle point.

Example) Locate the maxima, minima, and saddle points of the functions.

$f(x,y)=\frac{1}{3}x^3+xy^2-4xy+1$ .

3) In 3 more variable,

How can we determine that those critical points are local minium or local maximum or saddle point?

We can think the matrix,

$H(x,y) = \begin{pmatrix}f_{xx}(x,y) &f_{xy}(x,y)\\f_{yx}(x,y) &f_{yy}(x,y)\end{pmatrix}$ , called as the Hessian matrix of $f(x,y)$ .

Define $D(x,y)$ to be the determinant $D(x,y)=\det(H(x,y)) = f_{xx}(x,y)f_{yy}(x,y) - \left( f_{xy}(x,y) \right)^2$ .

Suppose that $(x_{0},y_{0})$ is a critical point of $f(x,y)$ and that $f(x,y)$ has continuous second-order parital derivatives in some circular centered at $(x_{0}, y_{0})$ .

Then, we can rewrite the Second Derivative Test(이계도 함수 판정법) as following:

1. $f$ has a local minimum at $(x_{0},y_{0})$ if $H$ is positive definite.

2. $f$ has a local maximum at $(x_{0},y_{0})$ if $H$ is negative definite.

3. $f$ has a saddle point at $(x_{0},y_{0})$ if $H$ is indefinite.

4. The test is inconclusive otherwise.

Definition4

Let $A=[a_{ij}] \in M_{n\times n}(\mathbb{R})$ be symmetric matrix and let $\rm{x}= (\it x_{1}, x_{2}, \cdots x_{n})\in \mathbb{R}^n$ . Then :

Positive definite(양의 정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}>\rm0$ for all nonzero $\rm{x}$ .
x

  x  
Positive semidefinite(양의 준정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}\geq \rm0$ for all $\rm{x}$ .
Negative definite(음의 정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}< \rm0$ for all nonzero $\rm{x}$ .
Negative semidefinite(음의 준정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}=\it{\sum a_{ij}x_{i}x_{j}}\leq \rm0$ for all $\rm{x}$ .
Indefinite(부정부호) if $\rm{x}^{\it{T}}\it{A}\rm{x}$ takes both positive and negative values.

Theorem3

Let $A$ is a symmetric matrix, then:

$\rm{x}^{\it{T}}\it{A}\rm{x}$ is positive definite if and only if all eigenvalues of $A$ are positive.
$\rm{x}^{\it{T}}\it{A}\rm{x}$ is negative definite if and only if all eigenvalues of $A$ are negative.
$\rm{x}^{\it{T}}\it{A}\rm{x}$ is indefinite if and only if $A$ has at least one positive eigenvalue and at least one negative eigenvalue.

Example) For $q(x,y)=2x^2-4xy+5y^2$ , determine the natrue of critical point $(0,0)$ .

Sol)

The matrix $A$ of quadratic form is $\begin{bmatrix} 2 &-2 \\ -2 & 5 \end{bmatrix}$ and its eigenvalues are $\lambda_1=6$ and $\lambda_2=1$

다음은 이변수함수에 관한 local extrema를 보여주는 interact입니다.(단, 임계점이 $2n\pi$ 의 형태면 구해지지 않음)

$\bigstar$ In general(일반적으로),

Given the real-valued function $f(x_1, x_2, \dots, x_n),\,\!$

if all second partial derivatives of $f$ exist and are continuous over the domain of the function, then the Hessian matrix of $f$ is

$H(f)_{ij}(\mathbf x) = D_i D_j f(\mathbf x)\,\!$

where x = (x₁, x₂, ..., x_n) and $D_{i}$ is the differentiation operator with respect to the i th argument. Thus

$H(f) = \begin{bmatrix} \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_1\,\partial x_n} \\[2.2ex] \dfrac{\partial^2 f}{\partial x_2\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_2^2} & \cdots & \dfrac{\partial^2 f}{\partial x_2\,\partial x_n} \\[2.2ex] \vdots & \vdots & \ddots & \vdots \\[2.2ex] \dfrac{\partial^2 f}{\partial x_n\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_n\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2} \end{bmatrix}.$

The following test can be applied at any critical point (a, b, ...) for the Hessian matrix

If the Hessian is positive definite (equivalently, has all eigenvalues positive) at (a, b, ...), then f attains a local minimum(극소) at (a, b, ...).
If the Hessian is negative definite (equivalently, has all eigenvalues negative) at (a, b, ...), then f attains a local maximum(극대) at (a, b, ...).
If the Hessian has both positive and negative eigenvalues then (a, b, ...) is a saddle point(안장점) for f (and in fact this is true even if (a, b, ...) is degenerate)

Remark

The mixed derivatives(혼합 도함수) of f are the entries off the main diagonal in the Hessian. Assuming that they are continuous, the order of differentiation does not matter (Clairaut's theorem). For example,

$\frac {\partial}{\partial x} \left( \frac { \partial f }{ \partial y} \right) = \frac {\partial}{\partial y} \left( \frac { \partial f }{ \partial x} \right).$

:If the second derivatives of f are all continuous in a neighborhood D, then the Hessian of f is a symmetric matrix(대칭행렬) throughout D

Example) For the function $f(x,y,z)=x^3+xyz+y^2-3x$ , find the critical points and determine whether the critical points are local maximum , local minimum or saddle point.

Sol)

The Hessian matrix of $f$ is $\begin{bmatrix} 6x &z &y \\ z & 2 & x\\ y &x &0 \end{bmatrix}$ .

At $(1,0,0)$ , the eigenvalues of Hessian matrix are positive and negative values.

Hence, $(1,0,0)$ is saddle point.

At $(-1,0,0)$ , the eigenvalues of Hessian matrix are positive and negative values.

Hence, $(-1,0,0)$ is saddle point.

Example) Locate the maxima, minima, and saddle points of the functions.

$f(x,y,z)=x^3 - 3y^3 + xz + z^2$

다음은 삼변수함수에 관한 local extrema를 보여주는 interact입니다.(단, 임계점이 $2n\pi$ 의 형태면 구해지지 않음)

Theorem4

The following are equivalent for a real symmetrix matrix $A$ :

$A$ is positive definite
All the eigenvalues of $A$ are positive
All the leading pricipal submatrix $A_{k}$ 's have positve determinants
All the pivots (without row interchanges) are positive
There exists a nonsigular matrix $W$ such that $A=W^{T}W$
There is an symmetric positive definite matrix $B$ such that $A=B^2$

There is sample of proof in case that $A$ is positive definite (Principal axes Theorem(2013.12.02), Cholesky Theorm(2013.12.02))

The following are equivalent for a real symmetrix matrix $A$ :

$A$ is negaitive definite
All the eigenvalues of $A$ are negaitive
The determiant $(-1)^{k} \det\it (A_{k})>\rm 0$ for $1\leq k\leq n$ , i.e, $\det(\it A_{1})<\rm 0, det(\it A_{2})>\rm 0, \cdots , (-1)^{\it n}det(\it A_{n})=(-1)^{n}\rm det(\it A)>\rm 0$
The quadratic forms associated to all $A_{k}$ are negative definite
All the pivots (without row interchanges) are negaitive

We can prove other things using above the proof of positive definite.

The following are equivalent for a real symmetrix matrix $A$ :

$A$ is positive semidefinite
All the eigenvalues of $A$ are nonnegative
All the leading pricipal submatrix $A_{k}$ 's have nonnegative determinants
All the pivots (without row interchanges) are nonnegative
There exists a matrix $W$ , possible singular, such that $A=W^{T}W$

The followibg are equivalent for a real symmetrix matrix $A$ :

$A$ is negative semidefinite
All the eigenvalues of $A$ are nonpositive
All the leading pricipal submatrix $A_{k}$ 's have nonpositive determinants
All the pivots (without row interchanges) are nonpositive
There exists a matrix , possible singular, such that

Reference)

Recall the calclus-one variable- Second derivative test, wikipedia, http://goo.gl/W11Os,2013.11.26

- two variable& in 3 more variable- Second partial deribvative test, wikipedia, http://goo.gl/R0LQMS,2013.11.26

In General, Information of Hessian matrix- Hessian matrix, wikipedia, http://goo.gl/s24ns, 2013.11.26

Definition 4, Theorem3 , Theorem4 and examples- Linear Algebra, Kwak&Hong, p294~p297.

Back TO the HOME