Chaper 8
Diagonalization
8.1 Matrix Representation of Linear Transformation
8.2 Similarity and Diagonalization
8.3 Diagonalization with orthogonal matrix,
8.4 Quadratic forms
*8.5 Applications of Quadratic forms
8.6 SVD and PseudoInverse
8.7 Complex eigenvalues and eigenvectors
8.8 Hermitian, Unitary, Normal Matrices
*8.9 Linear system of differential equations
Exercises
In Chapter 6, we have studied how to represent a linear transformation from into as a matrix using its corresponding standard matrix.
We were able to compute the standard matrix of the linear transformation based on the fact that
every vector in or can be expressed as a linear combination of the standard basis vectors.
In this chapter, we study how to represent a linear transformation from to with respect to arbitrary ordered bases for and .
In addition, we study relationship between different matrix representations of a linear transformation from to itself using transition matrices.
We also study matrix diagonalization.
Further, we study spectral properties of symmetric matrices and show that every symmetric matrix is orthogonally diagonalizable.
* A quadratic form is a quadratic equation which we come across in mathematics, physics, economics, statistics, and image processing, etc.
Symmetric matrices play a significant role in the study of quadratic forms.
We will learn how orthogonal diagonalization of symmetric matrices is used in the study of quadratic forms.
We introduce one of the most important concept in matrix theory called the singular value decomposition (SVD) which has many applications in science and engineering.
We will generalize matrix diagonalization of matrices and study least squares solutions and a pseudoinverse.
Furthur we deal with complex matrices having complex eigenvalues and eigenvectors.
We also introduce Hermitian matrices and unitary matrices that are complex counterparts corresponding to symmetric matrices and orthogonal matrices respectively.
Lastly, we study diagonalization of complex matrices.
8.1 Matrix Representation
Lecture Movie : https://youtu.be/V_TBd7O_a70 http://youtu.be/jfMcPoso6g4
Lab : http://matrix.skku.ac.kr/knouknowls/claweek11sec81.html
In Chapter 6, we have studied how to represent a linear transformation from into as a matrix using the standard bases for and .
In this section, we find a matrix representation of a linear transformation from into with respect to arbitrary ordered bases for and .
Matrix Representation Relative to the Standard Bases
Matrix Representation Using Arbitrary Ordered Bases for and
Theorem 
8.1.1 
Let be a linear transformation, and let
be ordered bases for and respectively. Let . Then
,
where the matrix representation of with respect to the ordered bases and is . 
Note that the matrix is called the matrix associated with the linear transformation with respect to the bases and .
Recall that any vector can be uniquely represented as a linear combination of vectors in , say
.
Then the coordinate vector for relative to the basis is
.
By the linearity of , we have
. Since is a vector in , the coordinate vector of relative to satisfies
. ¡á
Thus the matrix is the matrix whose th column is the coordinate vector of with respect to the basis .
[Remarks] 







(1) By Theorem 8.1.1 we can compute by a matrixvector multiplication, that is,
.
(2) The matrix varies according the ordered bases . For example, if we change the order of the vectors in the ordered base , then the columns of change as well. (3) The matrices and are distinct, but they have the following relationship: .
(4) If and , then is denoted by and is called the matrix representation of relative to the ordered basis . 







Define a linear transformation via and let
,
be ordered bases for and respectively. Compute .
Since , we first compute .:
, ,
We now find the coordinate vectors of the above vectors relative to the ordered basis . Since
,
we need to solve the corresponding linear systems with the same coefficient matrix .
The augmented matrix for all of the three linear systems is . By converting this into its RREF, we get
.
Therefore, , , and hence
.
(Note that .) ¡á
Let be defined via and
be ordered bases for and respectively. Find .
Since , , we have
.
We can get its RREF as follows:
.
Therefore, we get as
. ¡à
¨ç Write .
x, y = var('x, y')
h(x, y) = [x+y, x3*y, 2*x+y]
T = linear_transformation(QQ^2, QQ^3, h)
x1=vector([1, 1])
x2=vector([2, 1])
y1=vector([1, 0, 1])
y2=vector([1, 2, 1])
y3=vector([0, 1, 1])
B=column_matrix([y1, y2, y3, T(x1), T(x2)]) # Matrix whose columns are the vectors defined above
print B
[ 1 1 0 2 3]
[ 0 2 1 2 1]
[1 1 1 1 3]
¨è RREF
C=B.echelon_form()
print C
[ 1 0 0 1/2 5/2]
[ 0 1 0 3/2 1/2]
[ 0 0 1 1 0]
¨é Finding
A=C.submatrix(0, 3, 3, 2) # C.submatrix(a, b, c, d)
# submatrix with c consecutive rows of C starting from row a+1 and d
# consecutive columns of C starting from column b+1
print A
[ 1/2 5/2]
[3/2 1/2]
[ 1 0]
We shall include calculation using the inbuilt function. Following are the codes.
var('x,y')
h(x,y)=[x+y,x3*y,2*x+y]
V=QQ^2;W=QQ^3
T=linear_transformation(V,W,h)
y1=vector(QQ,[1,1]);y2=vector(QQ,[2,1])
x1=vector(QQ, [1,0,1]);x2=vector(QQ, [1,2,1]);x3=vector(QQ,[0,1,1]);
alpha=[y1,y2]; beta=[x1,x2,x3]
V1=V.subspace_with_basis(alpha); W1=W.subspace_with_basis(beta)
T1=(T.restrict_domain(V1)).restrict_codomain(W1)
T1.matrix(side='right')
[ 1/2 5/2]
[3/2 1/2]
[ 1 0] ¡á
Let be a linear transformation defined as and
consider the ordered bases , for and respectively.
Answer the following questions:
(1) Find .
(2) Compute using in (1).
(3) Using the definition of , find the standard matrix and ,
where and are the standard bases of and respectively.
(1) Since , and
where and
. Hence .
(2) Since , we have
(Note that
.)
(3) , . ¡á







Composition of Linear Transformations
Let be a linear transformation from a vector space with an ordered basis into a vector space with an ordered basis , and be a linear transformation from a vector sapce with an ordered basis into a vector space with an ordered basis . Suppose these linear transformations have their corresponding matrix representations and respectively. We can consider the composition . Then its matrix representation is
,
That is, the product of the two matrix representations of and . 









[Remark] 
Transition Matrix 


as 



As we have discussed earlier, , the matrix is called the transition matrix from ordered basis to ordered basis . We can consider the transition matrix as linear transformation .







Let be linear transformations defined as
and
respectively. Consider the ordered bases , , for .
Find the matrix representation of the composition with respect to the ordered bases and .
x, y = var('x, y')
ht(x, y) = [2*x+y, xy];hs(x, y) = [x+5*y, 2*x+3*y]
T = linear_transformation(QQ^2, QQ^2, ht)
S = linear_transformation(QQ^2, QQ^2, hs)
x1=vector([1, 0]);x2=vector([0, 1]);x3=vector([1, 1])
B=column_matrix([x2, x1, T(x1), T(x2)])
C=B.echelon_form()
MT=C.submatrix(0, 2, 2, 2)
print "Matrix of T="
print MT
D=column_matrix([x1, x3, S(x2), S(x1)])
E=D.echelon_form()
MS=E.submatrix(0, 2, 2, 2)
print "Matrix of S="
print MS
print "MS*MT="
print MS*MT
Matrix of T=
[ 1 1]
[ 2 1]
Matrix of S=
[ 2 3]
[ 3 2]
MS*MT=
[4 5]
[ 7 1]
F=column_matrix([x1, x3, S(T(x1)), S(T(x2))])
G=F.echelon_form()
MST=G.submatrix(0, 2, 2, 2)
print "Matrix of S*T="
print MST
Matrix of S*T=
[4 5]
[ 7 1] ¡á
Consider the linear transformation defined by
and
respectively. Find the composition and the matrix associated with composition.
Hence show that the matrix of the composition is a product of the matrices associated with and .
x, y, z,w = var('x, y, z,w')
output1 = [x+z+w,x+y+2*zw,2*x+y+3*z2*w]
output2 = [x+z,x+3*y+2*z,2*xy+3*z,yz]
T_symbolic(x, y, z,w)=output1
T=linear_transformation(QQ^4, QQ^3, T_symbolic)
S_symbolic(x, y, z)=output2
S=linear_transformation(QQ^3, QQ^4, S_symbolic)
A= T.matrix(side='right')
B= S.matrix(side='right')
print A
print B
U=S*T;
C=U.matrix(side='right')
print C
B*A==C
[ 3 1 4 1]
[ 8 5 13 6]
[ 7 2 9 3]
[1 0 1 1]
True ¡á
8.2 Similarity and Diagonalization
Lecture Movie : https://youtu.be/NHEyutjknyo http://youtu.be/xirjNZ40kRk
Lab : http://matrix.skku.ac.kr/knouknowls/claweek11sec82.html
In this section, we present various matrix representations of a linear transformation from to itself in terms of transition matrix.
We also study when the transition matrix becomes a diagonal matrix.
[Remark] 
Relationship between matrix representations and 






. 







Theorem 
8.2.1 
Let be a linear transformation and and be ordered bases for .
If , then we have
,
where is the transition matrix from to . 
.
http://www.math.tamu.edu/~yvorobet/MATH304503/Lect212web.pdf
Let be a linear transformation defined by .
If is the standard basis for and is a basis for , find using the transition matrix .
Let be the standard matrix relative to the standard basis for linear transformation .
Then we can find . If , then
.
Therefore, and by Theorem 8.2.1 we get as follows:
.
x, y = var('x, y')
h(x, y) = [2*xy, x+3*y]
T = linear_transformation(QQ^2, QQ^2, h)
x1=vector([1, 0]);x2=vector([0, 1])
y1=vector([0, 1]);y2=vector([1, 1])
B=column_matrix([x1, x2, y1, y2])
C=B.echelon_form()
P=C.submatrix(0, 2, 2, 2)
print "Transition Matrix="
print P
A = T.matrix(side='right')
print "A="
print A
print "P.inverse()*A*P"
print P.inverse()*A*P
D=column_matrix([y1, y2, T(y1), T(y2)])
E=D.echelon_form()
print "Matrix of A wrt beta="
print E.submatrix(0, 2, 2, 2)
Transition Matrix=
[ 0 1]
[ 1 1]
A=
[ 2 1]
[ 1 3]
P.inverse()*A*P
[ 2 1]
[ 1 3]
Matrix of A wrt beta=
[ 2 1]
[ 1 3] ¡á
Similarity
Definition 
[Similarity] 






For square matrices and of the same order, if there exists an invertible matrix such that , then we say that is similar to . We denote it as . 






For , , , it can be shown that . Hence is similar to . ¡á
Theorem 
8.2.2 
For square matrices of the same order, the following hold: (1) (2) (3) Therefore, the similarity relation is an equivalence relation. 
Theorem 
8.2.3 
For square matrices of the same order, if are similar to each other, then we have the following: (1) . (2) . 
Since , there exists an invertible matrix such that .
(1) By the multiplicative property of determinant,
()
(2) ( )
¡á
Since similar matrices have the same determinants, it follows that they have the same characteristic equations and hence the same eigenvalues.
In ordered to solve problems of determinant and/or eigenvalues of a square matrix, we can use its similar matrices which make the problems simpler.
Diagonalizable Matrices
Definition 
[Diagonalizable Matrices] 






Suppose a square matrix is similar to a diagonal matrix, that is, there exists an invertible matrix such that is a diagonal matrix. Then is called diagonalizable and the invertible matrix is called a diagonalizing matrix for . 






If , then and hence we have
( multiplications of )
This implies that if a matrix is diagonalizable, then its powers can be very easily computed.
For invertible matrix and matrix , we have
.
Hence is diagonalizable. ¡à
¡Ü http://matrix.skku.ac.kr/RPG_English/8TFdiagonalizable.html
A=matrix(QQ, [[1, 1], [2, 4]])
print A.is_diagonalizable() # Checking if diagonalizable
True ¡á
Since every diagonal matrix satisfies , it is
diagonalizable. ¡á
Note that not every matrix is diagonalizable.
Show that is not diagonalizable.
Suppose to the contrary that is diagonalizable, that is, there exist an invertible matrix and a diagonal matrix with
, , ,
such that . Since , we have
,
which gives . Hence .
If , then and (). Hence . Similarly, we can show that .
The conditions and give a contradiction to . Therefore, is not diagonalizable. ¡á
Equivalent Conditions for Diagonalizability
Theorem 
8.2.4 [Equivalent Condition] 
Let be a square matrix of order . Then is diagonalizable if and only if has linearly independent eigenvectors. Furthermore, if is diagonalizable, then is similar to diagonal matrix whose main diagonal entries are equal to the eigenvalues of , and the th column of a diagonalizing matrix is an eigenvector of corresponding to eigenvalue . 
If is diagonalizable, then there exists an invertible matrix
such that where . Since , we get .
Hence are eigenvalues of and . Note that are eigenvectors of corresponding to eigenvalues respectively.
Since is invertible, it follows that its columns are linearly independent.
Suppose has eigenvalues and their corresponding eigenvectors that are linearly independent. We define as
.
Then
.
Since the columns of are linearly independent, the matrix is invertible, giving . Therefore is diagonalizable.
¡á
[Remark] 
Procedure for diagonalizing a matrix 






¡Ü Step 1: Find linearly independent eigenvectors of . ¡Ü Step 2: Construct a matrix whose columns are in this order. ¡Ü Step 3: The matrix diagonalize and is a diagonal matrix whose main diagonal entries are eigenvalues of . 






It can be shown that the matrix has eigenvalues and their corresponding eigenvectors are respectively.
Since these eigenvectors are linearly independent, by Theorem 8.2.4, is diagonalizable.
If , then we have
. ¡á
Show that is diagonalizable and find the diagonalizing matrix of .
A=matrix([[0, 0, 2], [1, 2, 1], [1, 0, 3]])
print A.eigenvalues() # Eigenvalue Computation
[1, 2, 2]
has eigenvalues . We now compute linearly independent eigenvectors of .
For , we solve (that is, ) for .
E=identity_matrix(3)
print (EA).echelon_form()
[ 1 0 2]
[ 0 1 1]
[ 0 0 0]
Since , we get ; For , we solve (that is, ) for .
print (2*EA).echelon_form()
[1 0 1]
[0 0 0]
[0 0 0]
This gives and hence
.
x1=vector([2, 1, 1])
x2=vector([1, 0, 1])
x3=vector([0, 1, 0])
P=column_matrix([x1, x2, x3])
print P
print P.det()
[2 1 0]
[ 1 0 1]
[ 1 1 0]
1
Since the above computation shows that the determinant of is not zero, is invertible.
Hence its columns are linearly independent. Therefore, by Theorem 8.2.4, is diagonalizable.
print P^1*A*P # Computing diagonal matrix whose main diagonal entries are eigenvalues of A.
[1 0 0]
[0 2 0]
[0 0 2] ¡á
Theorem 
8.2.5 
If are eigenvectors of corresponding to distinct eigenvalues , then the set is linearly independent. 
(Exercise) Hint. This can be proved by the mathematical induction on .
Theorem 
8.2.6 
If a square matrix of order has distinct eigenvalues, then is diagonalizable. 
Let be eigenvectors of corresponding to distinct eigenvalues.
Then, by Theorem 8.2.5, the eigenvectors are linearly independent. Therefore, Theorem 8.2.4 implies that is diagonalizable. ¡á
The matrix in Example 6 has two distinct eigenvalues. Thus, by Theorem 8.2.6, is diagonalizable. ¡á
Note that a diagonal matrix can have a repeated eigenvalue. Therefore, the converse of Theorem 8.2.6 is not necessarily true.
Algebraic Multiplicity and Geometric Multiplicity of an Eigenvalue
Definition 
[Algebraic and Geometric Multiplicity] 






Let be distinct eigenvalues of . Then the characteristic polynomial of can be written as In the above expression the sum of the exponents is equal to . The positive integer is called the algebraic multiplicity of and the number of linearly independent eigenvectors corresponding to the eigenvalue is called the geometric multiplicity of . 






Theorem 
8.2.7 [Equivalent Condition for Diagonalizability] 
Let be a square matrix of order . Then is diagonalizable if and only if the sum of the geometric multiplicities of eigenvalues of is equal to . 
By Theorem 8.2.4, an equivalent condition for a square matrix of order to diagonalizable is to have linearly independent eigenvectors.
Since the sum of the geometric multiplicities of eigenvalues of is equal to the number of linearly independent eigenvectors of and it is equal to , the result follows.
¡á
Theorem 
8.2.8 
Let be a square matrix and be an eigenvalue of . Then the algebraic multiplicity of is greater than or equal to the geometric multiplicity of . 
Let be the geometric multiplicity of an eigenvalue of , and let be the matrix whose columns are the linearly independent eigenvectors of corresponding to .
We can construct an invertible matrix by adding linearly independent columns to . Let be the resulting invertible matrix and let be the inverse of .
Then . Note that and have same characteristic polynomials. Since the first column of has have in its diagonal,
the characteristic polynomial of has a factor of at least .
Hence, the algebraic multiplicity of is greater than or equal to the geometric multiplicity of . ¡á
Theorem 
8.2.9 [Equivalent Condition for Diagonalizability] 
Let be a square matrix of order . Then is diagonalizable if and only if each eigenvalue of has the same algebraic and geometric multiplicity. 
If is diagonalizable, then there exists an invertible matrix and a diagonal matrix such that , or equivalently .
This implies that times column of is equal to scalar multiple of the column of .
Hence, all the columns of are eigenvectors of , which implies that each eigenvalue of has the same algebraic and geometric multiplicity.
The converse follows from Theorem 8.2.7. ¡á
For , the characteristic equation is .
Hence the eigenvalues of are and has algebraic multiplicity 2.
The following two vectors are linearly independent eigenvectors of
,
corresponding to and respectively.
However, matrix does not have three linearly independent eigenvectors.
Hence Theorem 8.2.4 implies that is not diagonalizable. ¡á
Note that the eigenvalue with algebraic multiplicity 2 has only one linearly independent eigenvector of .
It can be shown that has eigenvalues 3 and with algebraic multiplicity 1 and 2 respectively,
We can further show that geometric multiplicity of 3 and 2 are 1 and 2 respectively.
Hence is diagonalizable since has 3 linearly independent eigenvectors.
It can be verified that diagonalize and , where . Let us further compute .
¡á
8.3 Diagonalization with orthogonal matrix,
*Function of matrix
Lecture Movie : https://youtu.be/B8ESZZQIlzA http://youtu.be/jimlkBGAZfQ
Lab : http://matrix.skku.ac.kr/knouknowls/claweek11sec83.html
Symmetric matrices appear in many applications.
In this section, we study useful properties of symmetric matrices and show that every symmetric matrix is orthogonally diagonalizable.
*Furthermore, we study matrix functions using matrix diagonalization.
Orthogonal Matrix
Definition 
[Orthogonal Matrix] 






For a real square matrix , if is invertible and , then is called an orthogonal matrix. 






Theorem 
8.3.1 
If is an orthogonal matrix of order , then the following hold:
(1) The rows of are unit vectors and they are perpendicular to each other. (2) The columns of are unit vectors and they are perpendicular to each other. (3) is invertible. (4) for any vector (Norm Preserving Property). 
Similar to the proof of Theorem 6.2.3.
For , we have . Since
, is an orthogonal matrix. ¡á
Definition 
[Orthogonal Similarity] 






Let and be square matrices of the same order. If there exists an orthogonal matrix such that , then is said to be orthogonally similar to . 






Definition 
[Orthogonally Diagonalizable] 






For a square matrix , if there exists an orthogonal matrix diagonalizing , then is called orthogonally diagonalizable and is called a matrix orthogonally diagonalizing . 






Next we try to answer the following question. What matrices are orthogonally diagonalizable?
Theorem 
8.3.2 
Every eigenvalue of a real symmetric matrix is a real number. 
Let be an eigenvalue of and , an eigenvector corresponding to .
Then . Now premultiplying the first equation both sides by to have .
Taking the complex conjugate both sides, we have which implies .
Now, we get . Note that . Hence which means is real. ¡á
Note: Compare this proof with the one in Theorem 8.7.1
The symmetric matrix has characteristic equation .
Hence its eigenvalues are which are all real numbers. ¡á
Theorem 
8.3.3 
If a square matrix is symmetric, then eigenvectors of corresponding to distinct eigenvalues are perpendicular to each other. 
Let , be eigenvectors corresponding to distinct eigenvalue and () respectively.
We have .
Since , the above equation implies . ¡á
Theorem 
8.3.4 
Let be a square matrix. Then is orthogonally diagonalizable if and only if the matrix is symmetric. 
() Suppose is orthogonally diagonalizable. Then there exists an orthogonal matrix and a diagonal matrix such that .
Since , we have
.
Hence
.
Therefore, is symmetric.
() : Read : http://www.maths.manchester.ac.uk/~peter/MATH10212/notes10.pdf ¡á
Theorem 
8.3.5 
If is a symmetric matrix of order , then has eigenvectors forming an orthonormal set. 
Since is symmetric, by Theorem 8.3.4, is orthogonally diagonalizable,
that is, there exists an orthogonal matrix and a diagonal matrix such that .
Hence the main diagonal entries of are the eigenvalues of and the columns of are eigenvectors of .
Since the columns of the orthogonal matrix form an orthonormal set, the eigenvectors of are orthonormal. ¡á
Theorem 
8.3.6 
For a square matrix of order , the following are equivalent:
(1) is orthogonally diagonalizable. (2) has eigenvectors that are orthonormal. (3) is symmetric. 
How to find an orthogonal matrix diagonalizing a given symmetric matrix ?
For symmetric matrix , find an orthogonal matrix diagonalizing .
Since the characteristic equation of is , the eigenvalues of are ,.
Note that all the eigenvalues are distinct. Hence there exist eigenvectors of that are orthogonal.
The corresponding eigenvectors of are
.
By normalizing , we get an (real) orthogonal matrix diagonalizing :
¡á
It can be shown that the matrix has eigenvalues (algebraic multiplicity 2) and .
Hence we need to check if has two linearly independent eigenvectors. After eigenvector computation, we get
that are linearly independent eigenvectors corresponding to eigenvalue 3.
Using the GramSchmidt Orthonormalization, we get
, ,
, .
We can find an eigenvector corresponding to the eigenvalue and normalization gives
.
Therefore, the orthogonal matrix diagonalizing is given by
. ¡á
[Remark] 
* Function of Matrices 






¡Ü There are several techniques for extending a real function to a square matrix function such that interesting properties are maintained. You can find more the details on the following website: ¡Ü https://en.wikipedia.org/wiki/Matrix_function ¡Ü http://youtu.be/BABwoKAN4 ¡Ü http://www.siam.org/books/ot104/OT104HighamChapter1.pdf 






8.4 Quadratic forms
Lecture Movie : https://youtu.be/SxCLGhylkIk http://youtu.be/vWzHWEhAdk
Lab : http://matrix.skku.ac.kr/knouknowls/claweek12sec84.html
A quadratic form is a polynomial each of whose terms is quadratic.
Quadratic forms appear in many scientific areas including mathematics, physics, economics, statistics, and image processing.
Symmetric matrices play an important role in analyzing quadratic forms.
In this section, we study how diagonalization of symmetric matrices can be applied to analyse quadratic forms.
Definition 







An implicit equation in variables , for a quadratic curve can be expressed as . (1) . This can be rewritten in a matrixvector form as follows: . (2) 






[Remark] 
Graph for a quadratic curve (conic section) 






The following are the types of conic sections: ¨ç Nondegenerate conic sections: Circle, Ellipse, Parabola, Hyperbola. See Figure 1 and Example 1. ¨è Imaginary conic section: There are no points satisfying (1) ¨é Degenerate conic section: The graph of the equation (1) is one point, one line, a pair of lines, or having no points. See Example 2. 







Figure 1
[Remark] 
Conic Sections in the Standard Position 






¡Ü (Ellipse) (3)
¡Ü or (Hyperbola) (4)
¡Ü or (Parabola, ) (5)








(Nondegenerate conic section)
Since the equation can be written as ,
the graph of this equation is an ellipse.
The equation has the standard form and
hence its graph is a hyperbola. Since the equation can be put into , its graph is a parabola.
var('x,y')
g1=implicit_plot(9*x^2+4*y^2144==0,(x,8,8),(y,8,8),figsize=4,axes=True)
g2=implicit_plot(9*x^24*y^2+144==0,(x,20,20),(y,20,20),figsize=4,axes=True)
g3=implicit_plot(y^2+3*x==0,(x,5,5),(y,5,5),figsize=4,axes=True)
show(graphics_array([g1,g2,g3]))
¡á
(Degenerate conic section)
The graph of the equation is the axis. The graph of consists of the two horizontal lines , .
The graph of consists of the two lines and .
The graph of consists of one point . The graph of has no points. ¡á
The graph of a quadratic equations with both and terms or both and terms is a translation of a conic section in the standard position.
Let us plot the graph of . By completing squares in , we get
. (6)
Hence by using the substitutions , we get
in the coordinate plane. This equation gives a hyperbola of the standard position in the coordinate plane.
Hence the graph of the equation (6) is obtained by translating the hyperbola in the standard position 3 units along the axis and 1 unit along the axis.
var('x,y')
c1=implicit_plot(x^2/3y^2/31==0,(x,8,8),(y,8,8),axes=True,figsize=5,color='red')
c2=implicit_plot((x3)^2/3(y1)^2/31==0,(x,8,8),(y,8,8),axes=True,figsize=5,color='blue')
c1+c2
¡á
Quadratic Form
Definition 
[Quadratic Form] 






(7)
is called the quadratic form of the quadratic equation (1). 






The quadratic equations are quadratic forms, but the quadratic equation has a linear term and constant term 1 and hence it is not a quadratic form. ¡á
A quadratic form can be written in the form of . For example,
or .
This means that the matrix above is not unique.
We will use a symmetric matrix to write a quadratic forms:
,
.
We use symmetric matrices to represent quadratic forms because symmetric matrices are orthogonally diagonalizable.
Definition 







Let be a symmetric matrix of order and for real values . Then is called a quadratic form in . 






For a quadratic form in and , the term is called a crossproduct term.
Using orthogonal diagonalization, we can eliminate the crossproduct term.
For a quadratic form
,
the matrix is symmetric, we can find orthonormal eigenvectors corresponding to the eigenvalues , of .
The matrix is orthogonal and orthogonally diagonalizes , that is, .
Since we can switch the roles of and by switching the roles of and , without loss of generality, we can assume .
Therefore, we can consider as the rotation matrix in . Let for some . Then
and hence is a quadratic form without any crossproduct term in the coordinate system. Therefore, we get the following theorem.
Theorem 
8.4.1 [Diagonalization of a Quadratic Form] 
Suppose a symmetric matrix has as its eigenvalues. Then, by rotating the coordinate axes, the quadratic form can be written as follows in the coordinate system
. (8)
If the determinant of is 1 and diagonalize , then the rotation can be obtained by or . 
Using diagonalization of a quadratic form, determine which conic section the following quadratic equation describes.
. (9)
The quadratic equation can be written as
.
Since the characteristic equation of the symmetric matrix is , the eigenvalues of are .
By Theorem 8.4.1, . Hence, in the new coordinate system, the quadratic equation becomes
.
Since eigenvectors corresponding to are
,
respectively, the orthogonal matrix diagonalizing is
and .
Therefore coordinate axes are obtained by rotating the axis 45° clockwise () and
the equation (9) is an ellipse in the standard position relative to the coordinate system. ¡á
[Remark] 
Simulation for quadratic forms 






¡Ü http://www.geogebratube.org/student/m121534








Sketch the graph of the following quadratic equation
. (10)
Letting , we can rewrite the equation (10) as follows:
. (11)
Using rotation we first eliminate the crossproduct terms. Since the characteristic equation of is
,
the eigenvalues of are , and their corresponding orthonormal eigenvectors are , respectively.
Hence we can take .
Using axis rotation , we get and and hence from (11), we obtain
. (12)
We now use horizontal translation to remove term in (12). By completing the squares in (12) we get
.
That is, (Note: ). Therefore, the equation (12) represents an ellipse in the coordinate system
where the coordinate system is obtained by horizontally translating the coordinate system 1 unit along the axis.
¡á
Surface in 3dimensional space
Let
. (13)
Then, after diagonalization, we get
(14)
in the rotated coordinate system. This enables us to identify the graph of the equation (13) in .
In equation (14), if both , are positive, then the graph of equation (14) is a paraboloid opening upward (see figure (a) below). If both and are negative,
then the graph is a paraboloid opening downward (see figure (b) below).
Since the horizontal crosssection of each paraboloid is an ellipse, the above graphs are called elliptic paraboloids.
Elliptic Paraboloid
In (14) if both of and are nonzero but have different signs, then the graphs looks like a saddle in (a) and
is called a hyperbolic paraboloid. If one of and is zero, then the graph is parabolic cylinder in (b).
Show that the graph of the following equation is an elliptic paraboloid and sketch its crosssection at .
(15)
The quadratic form in (15) can be written as .
We first find an orthogonal matrix diagonalizing the symmetric matrix .
It can be shown that , and hence using , we can transform (15) into the following:
. (16)
The equation (16) represents an elliptic paraboloid in the coordinate system.
Note that the coordinate system is obtained by rotating the coordinate by angle counterclockwise. Hence, in , is given by
and . Now we sketch the crosssection of equation (15) at .
By substituting into (16), we get and hence the graph looks like the following:
¡à
Let's use Sage to graph equation (15)
¨ç Computing eigenvalues of
A=matrix(2, 2, [34, 12, 12, 41])
print A.eigenvalues()
[50, 25]
¨è Computing eigenvectors of
print A.eigenvectors_right()
[(50, [(1, 4/3)], 1),
(25, [(1, 3/4)], 1)]
¨é Computing diagonalizing
G=matrix([[1, 3/4], [1, 4/3]]) # Constructing a matrix whose columns are eigenvectors
P=matrix([1/G.row(j).norm()*G.row(j) for j in range(0,2)])
# Normalizing the row vectors (The orthogonality follows from the
# fact that the eigenvalues are distinct)
P=P.transpose() # Constructing a matrix whose columns are
# orthonormal eigenvectors
print P
[ 4/5 3/5]
[ 3/5 –4/5]
¨ê Sketching two ellipses simultaneously
var('u, v')
s=vector([u, v])
B=P.transpose()*A*P
p1=implicit_plot(s*A*s==50, (u, 2, 2), (v, 2, 2), axes='true')
p2=implicit_plot(s*B*s==50, (u, 2, 2), (v, 2, 2), color='red', axes='true')
show(p1+p2) # Ploting two graphs simultaneously
¡á
8.5 *Applications of Quadratic forms
Lecture Movie : http://youtu.be/cOW9qT64e0g
Lab : http://matrix.skku.ac.kr/knouknowls/claweek12sec85.html
By the theorem of principal axis (theorem 8.4.1), the graph of a 3D curve can be shown in the form of circle, ellipse or parabola in 2D.
The specific shape is uniquely determined by signs of eigenvalues of the corresponding quadratic form.
In this section, we define the sign of the quadratic form to identify the type of graph of given quadratic forms, and learn how to obtain the extrema of multivariable functions using them.
Given a system of springs and masses, there will be one quadratic form that represents the kinetic energy of the system, and another which represents the potential energy of the system in position variables.
Can one supply more details or omit this section. More details can be found in the following websites:
¡Ü Application of Quadratic Forms and Sage:
http://matrix.skku.ac.kr/2014Album/Quadraticform/
¡Ü http://ocw.mit.edu/ans7870/18/18.013a/textbook/HTML/chapter32/section09.html
8.6 SVD and Pseudoinverse
Lecture Movie : https://youtu.be/0FYcU4DWhWQ https://youtu.be/ejCge6Zjf1M
Lab : http://matrix.skku.ac.kr/knouknowls/claweek12sec86.html
We have learned that symmetric matrices are diagonalizable.
We now extend the concept of diagonalization to matrices (not necessarily square or symmetric) resulting in a matrix decomposition and
study pseudoinverse and least square solutions using a matrix decomposition.
Theorem 
8.6.1 [Singular Value Decomposition] 
Let be an real matrix. Then there exist orthogonal matrices of order and of order , and an matrix such that
, (1)
where the main diagonal entries of are positive and listed in the monotonically decreasing order, and is a zeromatrix. That is,
, where . 
Definition 







Equation (1) is called the singular value decomposition (SVD) of . The main diagonal entries of the matrix are called the singular values of . In addition, the columns of are called the left singular vectors of and the columns of are called the right singular vectors of . 






The following theorem shows that matrices and are orthogonal matrices diagonalizing and respectively.
Theorem 
8.6.2 
Let the decomposition be the singular value decomposition (SVD) of an matrix where are positive diagonal entries of . Then
(1) . (2) . 
Since , we get
.
Hence, by considering, and ,
and
,
respectively. ¡á
Find the SVD of .
The eigenvalues of are and
hence the singular values of are .
A unit eigenvector of corresponding to is , and a unit eigenvector of corresponding to
is . We can also find unit eigenvectors of :
.
Hence we get
.
Therefore, the SVD of is
. ¡à
¨ç Computing the singular values of and eigenvectors of
A=matrix([[sqrt(3), 2], [0, sqrt(3)]])
B=A.transpose()*A
eig=B.eigenvalues()
sv=[sqrt(i) for i in eig] # Computing singular values
print B.eigenvectors_right() # Computing eigenvectors of
[(9, [(1, sqrt(3))], 1), (1, [(1, 1/3*sqrt(3))], 1)]
¨è Computing
G=matrix([[1, sqrt(3)], [1, 1/3*sqrt(3)]])
Vh=matrix([1/G.row(j).norm()*G.row(j) for j in range(0,2)])
Vh=Vh.simplify() # Transpose of V
print Vh
[ 1/2 1/2*sqrt(3)]
[1/2*sqrt(3) 1/2]
¨é Computing eigenvectors of
C=A*A.transpose()
print C.eigenvectors_right() # Computing eigenvectors of
[(9, [(1, 1/3*sqrt(3))], 1), (1, [(1, sqrt(3))], 1)]
¨ê Computing
F=matrix([[1, 1/3*sqrt(3)], [1, sqrt(3)]])
U=matrix([1/F.row(j).norm()*F.row(j) for j in range(0,2)])
U=U.simplify().transpose() # U
print U
[ 1/2*sqrt(3) 1/2]
[ 1/2 –1/2*sqrt(3)]
¨ë Computing diagonal matrix
S=diagonal_matrix(sv); S
[3 0]
[0 1]
¨ì Verifying
U*S*Vh
[sqrt(3) 2]
[ 0 sqrt(3)]
http://matrix.skku.ac.kr/2014Album/MC.html (SVD) ¡á
Equivalent statement of invertible matrix on SVD
Theorem 
8.6.3 
Let be an matrix. Then is a nonsingular matrix if and only if every singular value of is nonzero. 
Since , matrix is nonsingular if and only if is nonsingular.
Hence, if is nonsingular, then all the eigenvalues of are nonzero.
By Theorem 8.6.2, the singular values of are the square roots of the positive eigenvalues of .
Hence the singular values of are nonzero. ¡á
Theorem 
8.6.4 
Suppose are the singular values of an matrix . Then the matrix can be expressed as follows: . (R) The equation (R) is called a rankone decomposition of . 
Note that the pseudoinverse of a matrix is important in the study of the least squares solutions for optimization problems.
We can express an nonsingular matrix using the SVD
. (2)
Note that all of , , are nonsingular matrices and , are orthogonal matrices. Hence the inverse of can be expressed as
. (3)
If is not a square matrix or is singular, then (3) does not give an inverse of .
However, we can construct a pseudoinverse of by putting in (2) into the form (where is nonsingular).
Definition 
[PseudoInverse] 






For an matrix , the matrix is called a pseudoinverse of , where , are orthogonal matrices in the SVD of and is
(where is nonsingular). 






We read as ¡®dagger.¡¯ If , then we define .
Truncated SVD
http://langvillea.people.cofc.edu/DISSECTIONLAB/Emmie'sLSISVDModule/p5module.html
What is a truncated SVD?
We learned that singular value decomposition for any matrix so that .
Let's take a closer look at the matrix . Remember is a diagonal matrix where
and are the singular values of the matrix .
A full rank decomposition of is usually denoted by where and are the matrices obtained by taking the first columns of and respectively.
We can find a rank approximation (or truncated SVD) to by taking only the first k largest singular values and the first k columns of U and V.
Find a pseudoinverse of .
We first compute the (truncated) SVD^{1)} of :
http://matrix.skku.ac.kr/2014Album/MC.html
.
Then
. ¡á
¡Ü http://matrix.skku.ac.kr/RPG_English/8MApseudoinverse.html
A=matrix(RDF,[[1,1],[0,1],[1,0]])
import numpy as np
Pinv=matrix(np.linalg.pinv(A))
Pinv
[ 0.333333333333 0.333333333333 0.666666666667]
[ 0.333333333333 0.666666666667 0.333333333333]
If has , then is said to be of the full column rank.
If has full column rank, then is nonsingular. If is nonsingular, then the pseudoinverse of is equal to .
Theorem 
8.6.5 
If an matrix has full column rank, then the pseudoinverse of is
. 
Let ¡¡be the SVD of . Then where is nonsingular. Then
.
Since has full column rank, is nonsingular and matrix is an orthogonal matrix. Hence and
. ¡á
Find the pseudoinverse of using theorem 8.6.5.
.
Since has full column rank,
is nonsingular and
. ¡á
Theorem 
8.6.6 
If is a pseudoinverse of , then the following hold:
(1) (2) (3) (4) (5) (6) . 
(Exercise) https://en.wikipedia.org/wiki/Proofs_involving_the_Moore%E2%80%93Penrose_pseudoinverse
[Remark] 







A pseudoinverse provides a tool for solving a least square problem. It is known that the least squares solution to the linear system is the solution to the normal equation . If has full column rank, then the matrix is nonsingular and hence . This means that if has full column rank, the least squares solution to is the pseudoinverse times the vector .








Theorem 
8.6.7 
Let be an matrix and be a vector in . Then is the least squares solution to . 
Let be the SVD of with ( is nonsingular).
Then and hence , Since
,
it follows that satisfies . ¡á
Find the least squares line passing through the four points .
Let be an equation for the line that fits to the points .
Then, by letting , the given condition can be written as the linear system for which and .
Since has full column rank, we get which is
. Hence . Therefore, the least squares line is given by . ¡á
var('x, y')
p1=plot(x + 3/2, x, 1,3, color='blue');
p2 = text("$x+ 3/2 $", (2,2), fontsize=20, color='blue')
show(p1+p2, ymax=4, ymin=1)
in http://matrix.skku.ac.kr/CalBook/part1/CSSec13.htm
8.7 Complex eigenvalues and eigenvectors
Lecture Movie : https://youtu.be/ejGNmo9hhfI http://youtu.be/8_uNVj_OIAk
Lab : http://matrix.skku.ac.kr/knouknowls/claweek13sec87.html
We have so far focused on real eigenvalues and real eigenvectors.
However, real square matrices can have complex eigenvalues and eigenvectors. In this section, we introduce complex vector spaces, complex matrices, complex eigenvalues and complex eigenvectors.
Complex vector spaces
Definition 
[Complex Vector Space] 






The set of vectors with complex components is denoted by .
If we define the vector addition and the scalar multiple of a vector in similar to those for , then is a vector space over and its dimension is equal to . 






If
, , , ,
then a vector in can be expressed as where ¡¯s are complex numbers,
and the set {, } is a basis for . This basis is called the standard basis for .
For a complex number , is called the conjugate of and is called the modulus of .
Furthermore, if we denote a complex number as , then and .
For a complex vector , we define its conjugate as .
¡Ü [Example] http://matrix.skku.ac.kr/RPG_English/9VTconjugate.html
Inner product
Definition 
[Inner Product] 






Let and be vectors in . Then
satisfies the following properties:
(1) (2) (3) (4) in particular,
The inner product is called the Euclidean inner product for the vector space as a special case. 






Definition 







Let be vectors in . Then, using the Euclidean inner product , we can define the Euclidean norm of and the Euclidean distance between as the following:
(1) . (2) . 






If , then we say that and are orthogonal to each other.
For vectors , compute the Euclidean inner product and their Euclidean distance.
¡à
u=vector([2*I, 0, 1+3*I]) # I is the imaginary unit.
v=vector([2I, 0, 1+3*I])
print v.hermitian_inner_product(u) # Taking the conjugate for v
# < u, v > = v.hermitian_inner_product(u)
print (uv).norm()
4*I + 8
sqrt(13) ¡á
Complex Eigenvalues and Eigenvectors of Real
Matrices
Let be a real square matrix of order .
Since eigenvalues of are zeros of its characteristic polynomial, eigenvalue can be a complex number.
If we find eigenvector corresponding to a complex eigenvalue, it will a complex vector.
Theorem 
8.7.1 
If is a complex eigenvalue of an real matrix and is its corresponding eigenvector, then the complex conjugate of is also an eigenvalue of and is an eigenvector corresponding to . 
Since an eigenvector is a nonzero vector, and . Since and is real (i.e.,),
it follows that .
Eigenvalues of Real Symmetric Matrices
Theorem 
8.7.2 
If is a real symmetric matrix, then all the eigenvalues of are real numbers. 
Let be an eigenvalue of , that is, there exists a nonzero vector such that .
By multiplying both sides by on the lefthand side, we get .
Hence . Since is a nonzero real number, we just need to show that is a real number. Note that
.
Therefore, is a real number. ¡á
Show that the eigenvalues of are . In addition show that if , then can be decomposed into
,
where is the angle between the axis and the line passing through the origin and the point .
Since the characteristic equation of is , the eigenvalues of are .
If , then , . Therefore,
. ¡á
8.8 Hermitian, Unitary, Normal Matrices
Lecture Movie : http://youtu.be/8_uNVj_OIAk , http://youtu.be/GLGwj6tzd60
Lab : http://matrix.skku.ac.kr/knouknowls/claweek13sec88.html
We used to denote the set of all real matrices.
In this section, we introduce to denote the set of all complex matrices.
Symmetric matrices and orthogonal matrices in can be generalized to be Hermitian matrices and unitary matrices in ,
We shall further study the diagonalization of Hermitian and Unitary matrices.
Conjugate Transpose
Definition 
[Conjugate Transpose] 






For a matrix, is defined by
.
The transpose of the complex conjugate of is called the conjugate transpose and
is denoted by , that is, . 






[Remark] 







¡Ü The Euclidean inner product in : , ¡Ü If a matrix is real, then . 







For matrices , , , their conjugate transposes are
Theorem 
8.8.1 [Properties of Conjugate Transpose] 
For complex matrices and a complex number , the following hold:
(1) . (2) . (3) . (4) . 
Proof of the above theorem is easy and left as exercises.
Hermitian Matrix
Definition 
[Hermitian Matrix] 






If a complex square matrix satisfying , is called a Hermitian matrix. 






In , and hence is not Hermitian. However, since , is Hermitian. ¡á
Theorem 
8.8.2 [Properties of Hermitian Matrix] 
Suppose is Hermitian. Then the following hold:
(1) For any vector , the product is a real number. (2) Every eigenvalue of is a real number. (3) Eigenvectors of corresponding to distinct eigenvalues are orthogonal to each other. 
Theorem 8.7.2 & http://people.math.gatech.edu/~meyer/MA6701/module11.pdf
Let . Since , is Hermitian.
The characteristic equation of is and hence the eigenvalues of are 1, ,
which confirms that all the eigenvalues of a Hermitian matrix are real numbers
. Furthermore, it can be shown that the eigenvectors are , , and , where
, ,
corresponding to , , and respectively, are orthogonal to each other. ¡á
SkewHermitian Matrices
Definition 
[SkewHermitian Matrix] 






If a complex square matrix satisfies , then is called a skewHermitian matrix. 






It can be verified that both matrices and below are skewHermitian:
and
¡á
Every matrix can be expressed as , where is Hermitian and is skewHermitian.
In particular, since is Hermitian and is skewHermitian, every complex square matrix can be rewritten as
.
Unitary Matrices
Definition 
[Unitary Matrix] 






If matrix satisfies , then is called a unitary matrix. If is unitary, then . In addition, if the th column vector of is denoted by , then
.
Therefore, is a unitary matrix if and only if the columns of form an orthonormal set in . 






Show that the following matrix is unitary:
Since , the product
. Hence
is a unitary matrix. We can also show that
.
For example, . ¡á
Theorem 
8.8.3 [Properties of a Unitary Matrix] 
Suppose has the Euclidean inner product and is a unitary matrix. Then the following hold:
(1) For , , which implies . (2) If is an eigenvalue of , then . (3) Eigenvectors of corresponding to distinct eigenvalues are orthogonal to each other. 
The property of a unitary matrix shows that a unitary matrix is an isometry, preserving the norm.
Unitary Similarity and Unitarily Diagonalizable
Matrices
Definition 
[Unitary Similarity and Unitary Diagonalization] 






For matrices , if there exists a unitary matrix such that , then we say that and are unitarily similar to each other. Furthermore, if is unitarily similar to a diagonal matrix, then is called unitarily diagonalizable. 






Let and . Then it can be checked that is a unitary matrix and .
Therefore, is unitarily diagonalizable. ¡á
If is unitarily diagonalizable, then there exists a unitary matrix such that and
hence . Letting , we get,
.
This implies that each th column of the unitary matrix is a unit eigenvector of corresponding to the eigenvalue .
Find a unitary matrix diagonalizing matrix .
The eigenvalues of are and their corresponding eigenvectors are
, .
Letting ,
and , it follows that
,
where is a unitary matrix. ¡á
Schur¡¯s Theorem
Transforming a complex square matrix into an upper triangular matrix
Theorem 
8.8.4 [Schur¡¯s Theorem] 
A square matrix is unitarily similar to an upper triangular matrix whose main diagonal entries are the eigenvalues of . That is, there exists a unitary matrix and an upper triangular matrix such that
, (),
where ¡¯s are eigenvalues of . 
Let be the eigenvalues of . We prove this by mathematical induction.
First, if , then the statement holds because . We now assume that the statement is true for any square matrix of order less than or equal to .
¨ç Let be an eigenvector corresponding to eigenvalue .
¨èBy the GramSchmidt Orthonormalization, there exists an orthonormal basis for including , say .
¨é Since is orthonormal, the matrix is a unitary matrix.
In addition, since , the first column of is . Hence is of the following form:
.,
where . Since , the eigenvalues of are .
¨ê By the induction hypothesis, there exists a unitary matrix such that
.
¨ë Letting ,
we get
.
Since is a unitary matrix, the result follows. ¡á
[Lecture on this proof] http://youtu.be/lL0VdTStJDM
Not every square matrix is unitarily diagonalizable. (see Chapter 10)
Normal matrix
Definition 
[Normal Matrix] 






If matrix satisfies , then is called a normal matrix. 






It can be shown that the following matrices and are normal:
, ¡á
A Hermitian matrix satisfies and hence . This implies that any Hermitian matrix is normal.
In addition, since a unitary matrix satisfies , it is a normal matrix. ¡á
Equivalent Conditions for a Matrix to be Normal
Theorem 
8.8.5 
For matrix , the following are equivalent:
(1) is unitarily diagonalizable. (2) is a normal matrix. (3) has orthonormal eigenvectors. 
Let and .
Show that is a normal matrix and the columns of are orthonormal eigenvectors of .
Since , and hence is a normal matrix.
Letting , we get
,
.
Thus and are eigenvectors of . In addition, since and , and are orthonormal eigenvectors of . ¡á
Show that is unitarily diagonalizable.
Note that matrix is Hermitian and its eigenvalues are .
An eigenvector corresponding to is . By normalizing it, we get .
Similarly, we can get a unit eigenvector corresponding to .
Taking , we get . ¡á
[Remark] 







Although not every matrix is diagonalizable, using the Schur¡¯s Theorem, we can obtain an upper triangular matrix (close to a diagonal matrix) similar to . The upper triangular matrix is called the Jordan canonical form of . The Jordan canonical form will be discussed in Chapter 10. 







8.9 *Linear system of differential equations
Lecture Movie : http://www.youtube.com/watch?v=c0y5DcNQ8gs
Lab : http://matrix.skku.ac.kr/knouknowls/claweek11sec81.html
Many problems in science and engineering can be written as a mathematical problem of solving linear system of differential equations.
In this section, we learn how to solve linear system of differential equations by using a matrix diagonalization.
¡Ü Details can be found in the following websites:
http://www.math.psu.edu/tseng/class/Math251/NotesLinearSystems.pdf
http://matrix.skku.ac.kr/EMSage/EMathChapter5.html
http://matrix.skku.ac.kr/CLAMC/chap8/Page83.htm
http://matrix.skku.ac.kr/CLAMC/chap8/Page84.htm
http://matrix.skku.ac.kr/CLAMC/chap8/Page85.htm
http://matrix.skku.ac.kr/CLAMC/chap8/Page86.htm
http://matrix.skku.ac.kr/CLAMC/chap8/Page87.htm
1) http://langvillea.people.cofc.edu/DISSECTIONLAB/Emmie'sLSISVDModule/p5module.html