LA Chapter 7 by SGLee
Chapter 8
Diagonalization
8.1 Matrix Representation of Linear Transformation
8.2 Similarity and Diagonalization
8.3 Diagonalization with orthogonal matrix,
8.4 Quadratic forms
*8.5 Applications of Quadratic forms
8.6 SVD and Pseudo-Inverse https://youtu.be/AxL4Q83IdAA
8.7 Complex eigenvalues and eigenvectors
8.8 Hermitian, Unitary, Normal Matrices
*8.9 Linear system of differential equations
8.10 Exercises
(English Textbook) http://matrix.skku.ac.kr/2015-Album/Big-Book-LinearAlgebra-Eng-2015.pdf
(e-book : Korean) http://matrix.skku.ac.kr/2015-Album/BigBook-LinearAlgebra-2015.pdf
http://matrix.skku.ac.kr/LA/
http://matrix.skku.ac.kr/LA-Sage/
선형대수학 http://matrix.skku.ac.kr/LinearAlgebra.htm
Credu http://matrix.skku.ac.kr/Credu-CLA/index.htm
OCW http://matrix.skku.ac.kr/OCW-MT/index.htm
행렬 계산기 http://matrix.skku.ac.kr/2014-Album/MC-2.html
그래프 이론 http://matrix.skku.ac.kr/2014-Album/Graph-Project.html
행렬론 http://matrix.skku.ac.kr/MT2010/MT2010.htm
JFC http://matrix.skku.ac.kr/JCF/index.htm
In Chapter 6, we have studied how to represent a linear transformation from 
into 
as a matrix using its corresponding standard matrix.
We were able to compute the standard matrix of the linear transformation based on the fact that
every vector in 
or 
can be expressed as a linear combination of the standard basis vectors.
In this chapter, we study how to represent a linear transformation from 
to 
with respect to arbitrary ordered bases for 
and 
.
In addition, we study relationship between different matrix representations of a linear transformation from 
to itself using transition matrices.
We also study matrix diagonalization.
Further, we study spectral properties of symmetric matrices and show that every symmetric matrix is orthogonally diagonalizable.
* A quadratic form is a quadratic equation which we come across in mathematics, physics, economics, statistics, and image processing, etc.
Symmetric matrices play a significant role in the study of quadratic forms.
We will learn how orthogonal diagonalization of symmetric matrices is used in the study of quadratic forms.
We introduce one of the most important concept in matrix theory called the singular value decomposition (SVD)
which has many applications in science and engineering.
We will generalize matrix diagonalization of 
matrices and study least squares solutions and a pseudoinverse.
Furthur we deal with complex matrices having complex eigenvalues and eigenvectors.
We also introduce Hermitian matrices and unitary matrices that are complex counterparts corresponding to symmetric matrices and orthogonal matrices respectively.
Lastly, we study diagonalization of complex matrices.
8.1 Matrix Representation
Lecture Movie : https://youtu.be/V_TBd7O_a70 http://youtu.be/jfMcPoso6g4
Lab : http://matrix.skku.ac.kr/knou-knowls/cla-week-11-sec-8-1.html
In Chapter 6, we have studied how to represent a linear transformation from 
into 
as a matrix using the standard bases for 
and 
.
In this section, we find a matrix representation of a linear transformation from 
into 
with respect to arbitrary ordered bases for 
and 
.
Matrix Representation Relative to the Standard Bases
Matrix Representation Using Arbitrary Ordered Bases for 
and 
|
Theorem |
8.1.1 |
|
Let
be ordered bases for
where the matrix representation |
|
be a linear transformation, and let
and 
respectively. Let 
. Then
,
of 
with respect to the ordered bases
and 
is
.
Note that the matrix 
is called the matrix associated with the linear transformation 
with respect to the bases 
and 
.
Recall that any vector 
can be uniquely represented as a linear combination of vectors in 
, say
.
Then the coordinate vector for 
relative to the basis 
is
.
By the linearity of 
, we have
. Since 
is a vector in 
, the coordinate vector of 
relative to 
satisfies
. ■
Thus the matrix 
is the matrix whose 
-th column is the coordinate vector 
of 
with respect to the basis 
.
|
[Remarks] |
|
||
|
|
|
|
|
|
|
(1) By Theorem 8.1.1 we can compute
(2) The matrix For example, if we change the order of the vectors in the ordered base (3) The matrices
(4) If is called the matrix representation of |
|
|
|
|
|
|
|
|
|
|||
by a matrix-vector multiplication, that is,
.
varies according the ordered bases 
.
, then the columns of 
change as well.
and 
are distinct, but they have the following relationship:
.
and 
, then 
is denoted by 
and
relative to the ordered basis 
.
Define a linear transformation 
via 
and let
, 
be ordered bases for 
and 
respectively. Compute 
.
Since 
, we first compute 
.:
, 
, 
We now find the coordinate vectors of the above vectors relative to the ordered basis 
. Since
,
we need to solve the corresponding linear systems with the same coefficient matrix 
.
The augmented matrix for all of the three linear systems is 
. By converting this into its RREF, we get
.
Therefore, 
, 
, 
and hence
.
(Note that 
.) ■
Let 
be defined via 
and
be ordered bases for 
and 
respectively. Find 
.
Since 
, 
, we have
.
We can get its RREF as follows:
.
Therefore, we get 
as
. □
① Write 
.
be a linear transformation defined as 
and
, 
for 
and 
respectively.
.
using 
in (1).
, find the standard matrix 
and 
,
and 
are the standard bases of 
and 
respectively.
, 
and 
where 
and
. Hence 
.
, we have
.)
, 
. 
be a linear transformation from a vector space 
with an ordered basis 
into a vector space 
with an ordered basis 
,
be a linear transformation from a vector sapce 
with an ordered basis 
into a vector space 
with an ordered basis 
.
and 
respectively.
. Then its matrix representation is
,
and 
.
, the matrix 
is called the 
to ordered basis 
.
.
be linear transformations defined as
and 
, 
, 
for 
.
with respect to the ordered bases 
and 
.
defined by
and 
and the matrix associated with composition.
and 
.
Lecture Movie :
Lab : 
from 
to itself
and 
.
be a linear transformation and 
and 
be ordered bases for 
.
, then we have
,
is the transition matrix from 
to 
.
.
be a linear transformation defined by 
.
is the standard basis 
for 
and 
is a basis for 
, find 
using the transition matrix 
.
be the standard matrix relative to the standard basis 
for linear transformation 
.
. If 
, then
.
and by Theorem 8.2.1 we get 
as follows:
. 
and 
such that
,
is 
. We denote it as 
.
, 
, 
, it can be shown that 
. Hence 
is similar to 
. 
of the same order, the following hold:
of the same order, if 
are similar to each other, then we have the following:
.
.
Since 
, there exists an invertible matrix 
such that 
.
(
)
(
)
Since similar matrices have the same determinants, it follows that they have the same characteristic equations and hence the same eigenvalues.
is similar to a diagonal matrix, that is, there exists an invertible matrix 
such that 
is a diagonal matrix. Then 
is called 
is called 
.
If 
, then 
and hence we have
(
multiplications of 
)
and matrix 
, we have
.
is diagonalizable. 
satisfies 
, it is
is not diagonalizable. 
is diagonalizable, that is, there exist an invertible matrix 
and a diagonal matrix 
with
, 
, 
,
. Since 
, we have
,
. Hence 
.
, then 
and 
(
). Hence 
. Similarly, we can show that 
.
and 
give a contradiction to 
. Therefore, 
is not diagonalizable. 
be a square matrix of order 
. Then 
is diagonalizable
has 
linearly independent eigenvectors. 
is diagonalizable, then 
is similar to diagonal matrix 
of 
, and the 
th column of a diagonalizing matrix
is an eigenvector of 
corresponding to eigenvalue 
.
If 
is diagonalizable, then there exists an invertible matrix
where 
.
are eigenvalues of 
and 
. Note that 
are eigenvectors of 
respectively.
is invertible, it follows that its columns 
are linearly independent.
Suppose 
has eigenvalues 
and their corresponding eigenvectors 
that are linearly independent. We define 
as
.
.
are linearly independent, the matrix 
is invertible, giving 
.
is diagonalizable. 
linearly independent eigenvectors 
of 
.
whose columns are 
in this order.
diagonalize 
and 
is a diagonal matrix
of 
.
has eigenvalues 
and
respectively.
is diagonalizable.
, then we have
is diagonalizable and find the diagonalizing matrix 
of 
.
has eigenvalues 
. We now compute linearly independent eigenvectors of 
.
, we solve 
(that is, 
) for 
.
, we get 
;
, we solve 
(that is, 
) for 
.
and hence
.
is not zero, 
is invertible.
are linearly independent.
is diagonalizable. 
are eigenvectors of 
corresponding to distinct eigenvalues 
, then 
is linearly independent.
.
of order 
distinct eigenvalues, then 
is diagonalizable.
Let 
be eigenvectors of 
corresponding to distinct eigenvalues
.
is diagonalizable. 
in Example 6 has two distinct eigenvalues. Thus, by Theorem 8.2.6, 
is diagonalizable. 
can have a repeated eigenvalue. Therefore, the converse of Theorem 8.2.6 is not necessarily true.
be distinct eigenvalues of 
.
can be written as
is equal to 
.
is called the
and 
.
be a square matrix of order 
. Then 
is diagonalizable 
is equal to 
.
By Theorem 8.2.4, an equivalent condition for a square matrix 
of order 
to diagonalizable is to have 
linearly independent eigenvectors.
is equal to the number of linearly independent eigenvectors of 
and
, the result follows.
be a square matrix and 
be an eigenvalue of 
. Then the 
is greater than or equal to the geometric multiplicity of 
.
Let 
be the geometric multiplicity of an eigenvalue 
of 
, and let 
be the 
matrix
linearly independent eigenvectors of 
corresponding to 
.
by adding 
linearly independent columns to 
.
be the resulting invertible matrix and let 
be the inverse of 
.
. Note that 
and 
have same characteristic polynomials.
column of
has have 
in its diagonal,
has a factor of at least 
.
is greater than or equal to the geometric multiplicity of 
. 
be a square matrix of order 
.
is diagonalizable 
of 
has the same algebraic and geometric multiplicity.
If 
is diagonalizable, then there exists an invertible matrix 
and a diagonal matrix 
, or equivalently 
.
times column 
of 
is equal to scalar multiple of the column 
of 
.
columns of 
are eigenvectors of 
,
has the same algebraic and geometric multiplicity.
, the characteristic equation is 
.
are 
and 
has algebraic multiplicity 2. 
, 
and 
respectively.
does not have three linearly independent eigenvectors.
is not diagonalizable. 
with algebraic multiplicity 2 has
.
has eigenvalues 3 and 
with algebraic multiplicity 1 and 2 respectively,
is diagonalizable 
has 3 linearly independent eigenvectors
diagonalize 
and 
, where 
.
.
Lecture Movie : 
Lab : 
, if 
is invertible and 
, then 
is called an
is an orthogonal matrix of order 
, then the following hold:
are unit vectors and they are perpendicular to each other.
are unit vectors and they are perpendicular to each other.
is invertible.
for any 
vector 
Similar to the proof of Theorem 6.2.3.
, we have 
. Since
, 
is an orthogonal matrix. 
and 
be square matrices of the same order.
such that 
, then 
is said to be 
.
, if there exists an orthogonal matrix diagonalizing 
,
is called 
is called 
.
Next we try to answer the following question. What matrices are orthogonally diagonalizable? 
be an eigenvalue of 
and 
, an eigenvector corresponding to 
.
. 
to have 
.
which implies
.
. Note that 
. Hence 
which means 
is real.
has characteristic equation
.
which are all real numbers. 
is symmetric, then eigenvectors of 
corresponding to distinct eigenvalues are perpendicular to each other.
, 
be eigenvectors corresponding to distinct eigenvalue 
and 
(
) respectively.
.
, the above equation implies 
. 
be a square matrix. Then 
is orthogonally diagonalizable 
is symmetric.
(
) Suppose 
is orthogonally diagonalizable.
and a diagonal matrix 
such that 
.
, we have
.
.
is symmetric. 
) : Read : 
is a symmetric matrix of order 
, then 
has 
eigenvectors forming an orthonormal set.
Since 
is symmetric, by Theorem 8.3.4, 
is orthogonally diagonalizable,
and a diagonal matrix 
such that 
.
are the eigenvalues of 
and the columns of 
are 
eigenvectors of 
.
form an orthonormal set, the 
eigenvectors of 
are orthonormal. 
of order 
, 
is orthogonally diagonalizable.
has 
eigenvectors that are orthonormal.
is symmetric.
How to find an orthogonal matrix 
diagonalizing a given symmetric matrix 
?
, find an orthogonal matrix 
diagonalizing 
.
Since the characteristic equation of 
is 
,
are 
,
.
that are orthogonal.
are
.
, we get an 
diagonalizing 
:
has eigenvalues 
(algebraic multiplicity 2) and 
.
has two linearly independent eigenvectors.
, 
,
, 
.
corresponding to the eigenvalue 
and
.
diagonalizing 
is given by 
. 
Lecture Movie : 
Lab : 
, 
for a quadratic curve can 
. (1)
. (2)
satisfying (1)
(Ellipse) (3)
or 
(Hyperbola) (4)
or 
(Parabola, 
) (5)
can be written as 
,
has the standard form 
and
can be put into 
, its graph is a parabola. 
is the 
-axis. The graph of 
consists of the two horizontal lines 
, 
.
consists of the two lines 
and 
.
consists of one point 
. The graph of 
has no points. 
The graph of a quadratic equations with both 
and 
terms or
and 
terms 
.
, we get
. (6)
, we get
-coordinate plane. This equation gives a hyperbola of the standard position in the 
-coordinate plane.
-axis and 1 unit along the 
-axis. 
(7)
are quadratic forms,
has a linear term 
and
A quadratic form can be written in the form of 
. For example,
or
.
above is not unique.
We will use 
,
.
be a symmetric matrix of order 
and 
for 
real values 
.
is called a 
.
For a quadratic form in 
and 
, the 
-term is called a cross-product term.
For a quadratic form
,
is symmetric, we can find 
corresponding to the eigenvalues 
, 
of 
.
is orthogonal and 
orthogonally diagonalizes 
, that is, 
.
and 
by switching the roles of 
and 
, 
.
as the rotation matrix 
in 
.
for some 
. Then
is a quadratic form 
-coordinate system.
has 
as its eigenvalues. Then, by rotating the coordinate axes, the quadratic form 
can be written as follows in the 
-coordinate system
. (8)
is 1 and 
diagonalize 
, then 
or 
.
. (9)
can be written as
.
is
, the eigenvalues of 
are 
.
.
.
are
,
diagonalizing 
is
-coordinate axes are obtained 
-axis 45° clockwise (
-coordinate system. 
. (10)
, we can rewrite the equation (10) as follows:
. (11)
is
, 
are 
, 
and their corresponding orthonormal eigenvectors are 
, 
respectively.
.
, we get 
and 
and hence from (11), we obtain
. (12)
-term in (12). By completing the squares in (12) we get
.
)
-coordinate system
-coordinate system is obtained by 
-coordinate system 1 unit along the 
-axis.
■
Let 
. (13)
(14)
-coordinate system. This enables us to identify the graph of the equation (13) in 
.
, 
are positive,
and 
are negative,
In (14) if both of 
and 
are nonzero but have different signs, then the graphs looks like a saddle in (a) and
and 
is zero, then the graph is 
.
(15)
.
diagonalizing the symmetric matrix 
.
, and hence using 
, we can transform (15) into the following:
. (16)
-coordinate system.
-coordinate system is obtained by rotating the 
-coordinate by angle 
counterclockwise. Hence, in 
, 
is given by
. Now we sketch the cross-section of equation (15) at 
.
into (16), we get 
and hence the graph looks like the following:
Let's use Sage to graph equation (15)
diagonalizing 
Lecture Movie : 
Lab : 
Lecture Movie : 
Lab : 
matrices (not necessarily square or symmetric)
be an 
real matrix. Then there exist orthogonal matrices 
of order 
and 
of order 
,
matrix 
such that 
, (1)
are positive and listed in the monotonically decreasing order,
is a zero-matrix. That is,
,
.
.
are called 
.
are called the 
are called the 
.
The following theorem shows that matrices 
and 
are orthogonal matrices diagonalizing 
and 
respectively.
be the singular value decomposition (SVD) of an 
matrix 
where 
are positive diagonal entries of 
. Then
.
.
Since 
, we get
.
and 
,
and
, 
.
The eigenvalues of 
are 
and
are 
.
is 
, and a unit eigenvector of 
corresponding to 
. We can 
:
.
.
is
. 
and eigenvectors of 
be an 
matrix. Then 
is a nonsingular matrix 
is nonzero.
Since 
, matrix 
is nonsingular if and only if 
is nonsingular.
is nonsingular, then all the eigenvalues of 
are nonzero.
are the square roots of the positive eigenvalues of 
.
are nonzero. 
are the singular values of an 
matrix 
. Then the matrix 
can be expressed as follows:
. (R)
.
We can express an 
. (2)
, 
, 
are 
nonsingular matrices and 
, 
are orthogonal matrices.
can be expressed as
. (3)
is not a square matrix or 
is singular, 
.
of 
in (2) into the form 
(where 
is nonsingular).
matrix 
, the 
matrix 
is called a 
,
, 
are orthogonal matrices in the SVD of 
and 
is
(where 
is nonsingular).
We read 
as 
‘dagger.’ If 
, then we define 
.
so that 
.
. Remember 
is a diagonal matrix where
and 
are the singular values of the matrix 
.
is usually denoted by 
where 
and 
are the matrices obtained by taking the first 
columns of 
and 
respectively.
-rank approximation (or truncated SVD) to 
.
:
.
.
If 
has 
, then 
is said to be of the 
has full column rank, then 
is nonsingular. If 
is nonsingular, then the pseudo-inverse of 
is equal to 
.
matrix 
has full column rank, then the pseudo-inverse of 
is
.
Let 
be the SVD of 
. Then 
where 
is nonsingular. Then
.
has full column rank, 
is nonsingular and matrix 
is an 
orthogonal matrix. Hence 
and
. 
using theorem 8.6.5.
.
has full column rank,
is nonsingular and
. 
is a pseudo-inverse of 
, then the following hold:
.
(Exercise) 
is the solution to the normal equation 
. 
has full column rank, then the matrix 
is nonsingular and hence
.
has full column rank, the least squares solution to 
is the pseudo-inverse 
times the vector 
.
be an 
matrix and 
be a vector in 
. Then 
is the least squares solution to 
.
Let 
be the SVD of 
with 
(
is nonsingular).
and hence 
, Since
,
satisfies 
. 
.
be an equation for the line that fits to the points 
.
, the given condition can be written as the linear system 
for which 
and 
.
has full column rank, we get 
which is
. Hence 
. Therefore, the least squares line is given by 
. 
Lecture Movie : 
Lab : 
complex components is denoted by
.
similar to those for 
,
is 
and its dimension is equal to 
.
If
, 
, 
, 
,
in 
can be expressed as 
where 
’s are complex numbers,
, 
} is a basis for 
. This basis is called the standard basis for 
For a complex number 
, 
is called the conjugate of 
and 
is called the modulus of 
.
as 
, then 
and 
.
, we define its conjugate as 
.
and 
be vectors in 
. Then
in particular, 
is called the 
be vectors in 
.
, we can define the Euclidean norm 
of 
and
between 
as the following:
.
.
If 
, then we say that 
and 
are orthogonal to each other.
, compute the Euclidean inner product and their Euclidean distance. 
be a real square matrix of order 
.
are zeros of its characteristic polynomial, eigenvalue can be a complex number.
is a complex eigenvalue of an 
real matrix 
and 
is its corresponding eigenvector,
of 
is also an eigenvalue of 
and 
is an eigenvector corresponding to 
.
Since an eigenvector is a nonzero vector, 
and 
. Since 
and 
is real (i.e.,
),
.
= [ bar(A)]^T ( = A^* ), then all the eigenvalues of 
are real numbers.
Let 
be an eigenvalue of 
, that is, there exists a nonzero vector 
such that 
.
on the left-hand side, we get 
.
. Since 
is a nonzero real number, we just need to show that 
is a real number. Note that
.
is a real number. 
are 
. In addition show that if 
, then 
can be decomposed into
,
is the angle between the 
-axis and the line passing through the origin and the point 
.
is 
, the eigenvalues of 
are 
.
, then 
, 
. Therefore,
. 
Lecture Movie : 
Lab : 
to denote the set of all 
real matrices.
to denote the set of all 
complex matrices. 
can be generalized
, 
is defined by
.
of the complex conjugate of 
is called the 
, that is, 
.
: 
, 
is real, then 
.
, 
, 
, their conjugate transposes are
, 
, 
. 
and a complex number 
, the following hold:
.
.
.
.
Proof of the above theorem is easy and left as exercises.
satisfying 
, 
is called a 
, 
and hence 
is not Hermitian. However, since 
, 
is Hermitian. 
is Hermitian. Then the following hold:
, the product 
is a 
is a 
corresponding to distinct eigenvalues are
. Since 
, 
is Hermitian. 
is 
and hence the eigenvalues of 
are 
1, 
, 
are real numbers
, 
, and 
, where 
, 
, 
, 
, and 
respectively, are orthogonal to each other. 
satisfies 
, then 
is called a 
and 
below are skew-Hermitian:
and
Every matrix 
can be expressed as 
, where 
is Hermitian and 
is skew-Hermitian. 
is Hermitian and 
is skew-Hermitian, every complex square matrix 
can be rewritten as
.
satisfies 
, then 
is called a 
is unitary, then 
. In addition, if the 
th column vector of 
is denoted by 
, then
.
is a unitary matrix if and only if the columns of 
form an orthonormal set in 
.
is unitary:
Since 
, the product
. Hence
is a unitary matrix. We can also show that 
.
. 
has the Euclidean inner product and 
is a unitary matrix. Then the following hold:
, 
, which implies 
.
is an eigenvalue of 
, then 
.
corresponding to distinct eigenvalues are orthogonal to each other. 
The property 
of a unitary matrix 
shows that a unitary matrix is an isometry, preserving the norm.
, if there exists a unitary matrix 
such that 
, 
and 
are 
is unitarily similar to a diagonal matrix, then 
is called 
and 
. Then it can be checked that 
is a unitary matrix and 
.
is unitarily diagonalizable.
If 
is unitarily diagonalizable, then there exists a unitary matrix 
such that 
and
. Letting 
, we get,
. 
This implies that 
th 
of the unitary matrix 
is a unit eigenvector of 
corresponding to the eigenvalue 
.
diagonalizing matrix 
.
are 
and their corresponding eigenvectors are 
, 
.
, 
, it follows that
,
is a unitary matrix. 
Transforming a complex square matrix into an upper triangular matrix
is unitarily similar to an upper triangular matrix 
.
and an upper triangular matrix 
such that
, 
(
),
’s are eigenvalues of 
.
be the eigenvalues of 
. We prove this by mathematical induction.
, then the statement holds because 
.
.
be an eigenvector corresponding to eigenvalue 
.
including 
, say 
.
is orthonormal, the matrix 
is a unitary matrix.
, the first column of 
is 
. Hence 
is of the following form:
.,
. Since 
, the eigenvalues of 
are 
.
such that
.
,
.
is a unitary matrix, the result follows.
Not every square matrix is unitarily diagonalizable. (see Chapter 10)
satisfies
,
is called 
and 
are normal:
, 
satisfies 
and hence 
. This implies that any Hermitian matrix is normal.
satisfies 
, it is a normal matrix. 
, the following are equivalent:
is unitarily diagonalizable.
is a normal matrix.
has 
orthonormal eigenvectors.
and 
.
is a normal matrix and the columns of 
are orthonormal eigenvectors of 
.
, 
and hence 
is a normal matrix.
, we get
, 
.
and 
are eigenvectors of 
. In addition, since 
and 
, 
and 
are orthonormal eigenvectors of 
. 
is unitarily diagonalizable.
is Hermitian and its eigenvalues are 
.
is 
.
corresponding to 
.
, we get 
. 
is diagonalizable, using the Schur’s Theorem,
(close to a diagonal matrix) similar to 
.
is called the Jordan canonical form of 
.
Lecture Movie : 
Lab : 
