5.Special case of Hessian matrix

Remark

$\small q(\rm\bold x)=\rm\bold x^{\it T}\it A\rm \bold x=\it \begin{bmatrix} x &y \end{bmatrix}\begin{bmatrix} a &b \\ b&c \end{bmatrix}\begin{bmatrix} x\\y \end{bmatrix}=ax^2+2bxy+cy^2$

for  $\small \rm\bold{x}=\it \begin{bmatrix} x & y \end{bmatrix}^{T} \in \mathbb{R}^{2}$ is itself a function of two variables, and its frist partial derivatives are

$\small q_{x} = 2ax+2by$

$\small q_{y} = 2bx+2cy$

So, the Hessian of $\small q$ is

$\small H=2\begin{bmatrix} a &b \\ b &c \end{bmatrix}=2A$.     Thus  $\small H$ is nonsigular if and only if $\small ac-b^2\neq 0$.

Since $\small q(\rm\bold 0)=0$, it follows that the quadratic form $\small q$ takes the local minimum(극소) at $\small \rm\bold 0$ if and only if

$\small q(\rm\bold x)=\rm\bold x^{\it T}\it A\rm\bold x >0$   for all  $\small \rm\bold x\neq \rm\bold 0$,

and  $\small q$ takes the local maximum(극대) at  $\small \rm\bold 0$  if and only if

$\small q(\rm\bold x)=\rm\bold x^{\it T}\it A\rm\bold x <0$  for all $\small \rm\bold x\neq \rm\bold 0$.

If $\small \rm\bold x^{\it T}\it A\rm \bold x$ takes bothe positive and negative values, then $\small \rm\bold 0$ is a saddle point.

In general(일반적으로), for $\small \rm\bold x \in \mathbb{R}^{\it n}$

$\small A$, the matrix representation(행렬표현) of the quadratic form $\small q$ can be rewritten as

$\small \frac{1}{2}H=A$   ($\small H$ is the Hessian matrix of $\small q$)

Therefore, we can determine the extrema of $\small q$ using the matrix $\small A$.

Theorem 5

Let $\small A$ be a symmetric $\small n\times n$ matrix whose eigenvalues(고유값) are $\small \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n$ in descending order.

If $\small \rm\bold x$ is constrained so that  $\small \left \| \rm\bold x \right \|=1$ relative to the Euclidean inner product on $\small \mathbb{R}^{n}$ , then

1) $\small \lambda_1 \geq \rm\bold x^{\it T} \it A\rm \bold x \geq \lambda_n$,

2)  $\small \rm\bold x^{\it T} \it A\rm \bold x = \lambda$  if $\small \rm\bold x$ is eigenvectors(고유벡터) of $\small A$ belonging to an eigenvalue $\small \lambda$.

(참고 : Exercise Rqyleigh quatient http://math1.skku.ac.kr/home/pub/1661/ )

Proof)

1) Since $\small A$ is symmetric(대칭), there is an orthonormal basis $\small \alpha=\left \{ \rm\bold v_{1},\rm\bold v_{2},\cdots , \rm\bold v_{\it n} \right \}$ for $\small \mathbb{R}^{n}$ cosisting of eigenvectors(고유벡터) of $\small A$ belonging to the eigenvlues $\small \lambda_1 , \lambda_{2}, \cdots , \lambda_n$, respectively.

If  $\small \left \langle , \right \rangle$ denotes the Euclidean inner product , then any $\small \rm\bold x$ in $\small \mathbb{R}^{n}$ may be expressed as

$\small \rm\bold x =\left \langle \rm\bold x,\rm\bold v_1 \right \rangle\rm\bold v_1+\left \langle \rm\bold x,\rm\bold v_2 \right \rangle\rm\bold v_2 +\cdots+\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle\rm\bold v_\it n$

Thus,

$\small A\rm\bold x =\left \langle \rm\bold x,\rm\bold v_1 \right \rangle\it A\rm\bold v_1+\left \langle \rm\bold x,\rm\bold v_2 \right \rangle\it A\rm\bold v_2 +\cdots+\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle\it A\rm\bold v_\it n$

$\small =\lambda_1\left \langle \rm\bold x,\rm\bold v_1 \right \rangle\rm\bold v_1+\lambda_2\left \langle \rm\bold x,\rm\bold v_2 \right \rangle\rm\bold v_2 +\cdots+\lambda_{\it n}\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle\rm\bold v_\it n$.

If  $\small \left \| \rm\bold x \right \|^2= \left \langle \rm\bold x,\rm\bold v_1 \right \rangle^2+\left \langle \rm\bold x,\rm\bold v_2 \right \rangle^2 +\cdots+\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle^2=1$, then we obtain

$\small \rm\bold x^{\it T}\it A\rm\bold x=\left \langle \rm\bold x, \it A\rm\bold x \right \rangle=\lambda_1\left \langle \rm\bold x,\rm\bold v_1 \right \rangle^2+\lambda_2\left \langle \rm\bold x,\rm\bold v_2 \right \rangle^2 +\cdots+\lambda_{\it n}\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle^2$

$\small \leq \lambda_1\left \langle \rm\bold x,\rm\bold v_1 \right \rangle^2+\lambda_1\left \langle \rm\bold x,\rm\bold v_2 \right \rangle^2 +\cdots+\lambda_{\it 1}\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle^2$

$\small = \lambda_1(\left \langle \rm\bold x,\rm\bold v_1 \right \rangle^2+\left \langle \rm\bold x,\rm\bold v_2 \right \rangle^2 +\cdots+\left \langle \rm\bold x,\rm\bold v_\it n \right \rangle^2)$

$\small =\lambda_1$,

since $\small \lambda_1$ is the largest eigenvalue. Simliarly, One can show $\small \lambda_n \leq \rm\bold x^{\it T}\it A\rm\bold x$.

2)  If $\small \rm\bold x$ is an eigenvector of $\small A$ belonging to $\small \lambda$ and  $\small \left \| \rm\bold x \right \|=1$, then

$\small \rm\bold x^{\it T}\it A \rm\bold x =\left \langle \rm\bold x,\it A\rm\bold x \right \rangle=\left \langle \rm\bold x,\lambda\rm\bold x \right \rangle=\lambda\left \langle \rm\bold x,\rm\bold x \right \rangle=\lambda\left \| \rm\bold x \right \|^2=\lambda$.

Example) Find the maximum and minimum values of the quadratic form

$x^2+y^2+4xy$

subject to the constraint  $x^2+y^2=1$, and determine values of $x$ and $y$ at which the maximum and minimum occur.

Sol)

The matrix representation of the quadratic form is  $A=\begin{bmatrix} 1 &2 \\ 2 &1 \end{bmatrix}$ .

The largest eigenvalue of $A$ is  $\lambda_{max}=3$ at  $(x,y)=(1,1)$ .

The smallest eigenvalue of $A$ is  $\lambda_{min}=-1$ at   $(x,y)=(1,-1)$ .

The unit vector of $(x,y)=(1,1)$  is   $(x,y)=(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})$ , which maximum value of the form occur.

The unit vector of $(x,y)=(1,-1)$  is   $(x,y)=(\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$, which minimum value of the form occur.

Example)

Find the maximum and minimum values of the quadratic form

$\small 2x^2+2y^2+3xy$

subject to the constraint  $\small x^2+y^2=1$ , and determine values of $\small x$ and $\small y$ at which the minimum and maximum occur.

다음은 함수가 단위원 안에서 최대값과 최소값이 어느 점에서 일어나는 지를 보여주는 interact입니다.

Example)

Find the maximum and monimum of the following quadratic forms subject to the constraint  $\small x^2+y^2+z^2=1$

and determine the values of $\small x,y$ and $\small z$ at which the maximum and minimum occur:

1) $x^2+y^2+z^2-2xy+2xz$.

2) $\small x^2+y^2+2z^2-2xy+4xz+4yz$.

Sol)

1) The matrix representation of quadratic form is  $A=\begin{bmatrix} 1 & -1 &1 \\ -1 &1 &0 \\ 1 &0 & 1 \end{bmatrix}$.

The largest values of $A$ is  $\lambda_{max}=\sqrt{2}+1$ at   $(x,y,z)=(\frac{1}{\sqrt{2}},-\frac{1}{2},\frac{1}{2})$.

The samllest value of $A$ is  $\lambda_{min}=-\sqrt{2}+1$ at  $(x,y,z)=(\frac{1}{\sqrt{2}},\frac{1}{2},-\frac{1}{2})$ .

2) 다음은 함수가 단위구에서 최대값과 최소값이 어느 점에서 일어나는 지를 보여주는 interact입니다.

Reference:

Remark3, Theorem5, Example- Linear Algebra, Kwak&Hong, Birkhauser,p298~p300

Back TO the HOME