Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 5.4 Symmetric matrices and quadratic forms

In this section, we discuss diagonalization for symmetric matrices, which were defined in Unit 1. Symmetric matrices play a fundamental role in optimization because Hessian matrices are symmetric when the mixed second partial derivatives agree. Another recurring example is \(A^T A\text{,}\) which appeared in the Unit 4 normal equations and will return below in least squares.
🔗

Subsection Orthogonality of eigenvectors

A key property of symmetric matrices is that for any two vectors \(\mathbf{x}\) and \(\mathbf{y}\text{,}\) \((A \mathbf{x}) \cdot \mathbf{y} = \mathbf{x} \cdot (A\mathbf{y})\text{.}\) Indeed, we calculate that
\begin{align*} (A\mathbf{x}) \cdot \mathbf{y} \amp = (A\mathbf{x})^T\mathbf{y}\\ \amp = \mathbf{x}^TA^T\mathbf{y}\\ \amp = \mathbf{x}^TA\mathbf{y} \quad \text{(since } A^T = A\text{)}\\ \amp = \mathbf{x} \cdot (A\mathbf{y}) \end{align*}
This property leads to a fundamental result about the eigenvectors of symmetric matrices.
🔗

Why is this true?.

Let \(A\mathbf{x} = \lambda\mathbf{x}\) and \(A\mathbf{y} = \mu\mathbf{y}\text{,}\) where \(\lambda \neq \mu\text{.}\) Using the property above:
\begin{align*} \lambda(\mathbf{x} \cdot \mathbf{y}) \amp = (\lambda\mathbf{x}) \cdot \mathbf{y}\\ \amp = (A\mathbf{x}) \cdot \mathbf{y}\\ \amp = \mathbf{x} \cdot (A\mathbf{y})\\ \amp = \mathbf{x} \cdot (\mu\mathbf{y})\\ \amp = \mu(\mathbf{x} \cdot \mathbf{y}) \end{align*}
This gives us \((\lambda - \mu)(\mathbf{x} \cdot \mathbf{y}) = 0\text{.}\) Since \(\lambda \neq \mu\text{,}\) we must have \(\mathbf{x} \cdot \mathbf{y} = 0\text{,}\) which means the eigenvectors are orthogonal.
🔗
🔗
As we will see, it turns out that for any symmetric matrix, there exists a pairwise orthogonal set of eigenvectors which form a basis of \(\R^n\text{.}\) In particular, every symmetric matrix is diagonalizable.
🔗
🔗

Subsection Orthogonal matrices and orthogonal diagonalization

Recall the definition of an orthogonal matrix from Definition 2.5.34.
🔗

Why is this true?.

Let us start by showing the first property is equivalent to the third. Suppose \(Q\) is invertible, and \(Q^{-1} = Q^T\text{.}\) Let \(\mathbf{v}_1, \dots, \mathbf{v}_n\) be the columns of \(Q\text{.}\) Recall from Theorem 1.4.2 that the \((i,j)\)-entry of \(Q^T Q\) is equal to the dot product \(\mathbf{v}_i \cdot \mathbf{v}_j\text{.}\) Since \(Q^{-1} = Q^T\text{,}\) \(Q^T Q = I\text{,}\) and so for \(i \neq j\text{,}\) \(\mathbf{v}_i \cdot \mathbf{v}_j = 0\text{,}\) and for each \(i\text{,}\) \(\mathbf{v}_i \cdot \mathbf{v}_i = 1\text{.}\) Thus the column vectors \(\mathbf{v}_1,\dots,\mathbf{v}_n\) are orthonormal. Conversely, if the column vectors \(\mathbf{v}_1,\dots,\mathbf{v}_n\) are orthonormal, then we immediately see that \(Q^T Q = I\text{.}\)
🔗
The proof that the first property is equivalent to the second is very similar, using the rows of \(Q\) rather than the columns and that if \(Q^T Q = I\text{,}\) then \(Q Q^T = I\text{,}\) and vice versa.
🔗
🔗

Why is this true?.

Let \(\mathbf{v}\) and \(\mathbf{w}\) be column vectors. Recalling the fifth property of Theorem 1.4.3, we calculate that
\begin{align*} (Q\mathbf{v}) \cdot (Q\mathbf{w}) \amp = (Q\mathbf{v})^T(Q\mathbf{w})\\ \amp = \mathbf{v}^T Q^T Q \mathbf{w}\\ \amp = \mathbf{v}^T I_n \mathbf{w}\\ \amp = \mathbf{v}^T \mathbf{w}\\ \amp = \mathbf{v} \cdot \mathbf{w}\text{.} \end{align*}
Thus \(Q\) preserves dot products.
🔗
Next, using that \(Q\) preserves dot products, and also Note 1.1.14, we find
\begin{equation*} \| Q\mathbf{v} \|^2 = (Q \mathbf{v}) \cdot (Q \mathbf{v}) = \mathbf{v} \cdot \mathbf{v} = \| \mathbf{v} \|^2\text{.} \end{equation*}
Taking square roots on both sides of this equation gives \(\|Q\mathbf{v}\| = \|\mathbf{v}\|\text{.}\)
🔗
🔗
Figure 5.4.4 provides a visualization of an orthogonal change of coordinates.
🔗
A cube and its rotated image under an orthogonal matrix.
A gray reference cube is shown behind a colored image cube obtained by applying an orthogonal matrix \(Q\text{.}\) The colored cube has red, blue, and green faces and is labeled with the images \(f(\mathbf{e}_1)\text{,}\) \(f(\mathbf{e}_2)\text{,}\) and \(f(\mathbf{e}_3)\text{.}\) The image cube is rotated relative to the gray cube, but its edge lengths and angles are preserved.
🔗
Figure 5.4.4. The effect of applying the linear map \(\mathbf{f}(\mathbf{x}) = Q\mathbf{x}\) to a cube, where \(Q\) is orthogonal. Notice that the map is “rigid”, i.e., applying the map to the cube results in a cube with the same dimensions. Adapted from: Stanford’s MATH 51 textbook.
🔗

Definition 5.4.5. Orthogonal diagonalizability.

A matrix \(A\) is orthogonally diagonalizable when an orthogonal matrix \(Q\) can be found such that \(Q^{-1}A Q\) is a diagonal matrix. Since \(Q^{-1} = Q^T\text{,}\) this is equivalent to \(Q^T A Q\) being a diagonal matrix.
🔗
🔗

Activity 5.4.1.

Consider the symmetric matrix
\begin{equation*} A = \begin{bmatrix} 2 \amp -1 \\ -1 \amp 2 \end{bmatrix} \end{equation*}
Check that the vectors
\begin{equation*} \mathbf{u} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \quad\text{and}\quad \mathbf{v} = \begin{bmatrix} 1 \\ -1 \end{bmatrix} \end{equation*}
are eigenvectors of \(A\) which are orthogonal to each other. What are the corresponding eigenvalues?
🔗
Solution.
The vectors \(\mathbf{u}\) and \(\mathbf{v}\) are eigenvectors of \(A\) with eigenvalues \(\lambda_1 = 1\) and \(\lambda_2 = 3\) respectively, since
\begin{equation*} A\mathbf{u} = \begin{bmatrix} 2 \amp -1 \\ -1 \amp 2 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} = 1\mathbf{u} \end{equation*}
and
\begin{equation*} A\mathbf{v} = \begin{bmatrix} 2 \amp -1 \\ -1 \amp 2 \end{bmatrix}\begin{bmatrix} 1 \\ -1 \end{bmatrix} = \begin{bmatrix} 3 \\ -3 \end{bmatrix} = 3\mathbf{v} \end{equation*}
We also calculate that
\begin{equation*} \mathbf{u} \cdot \mathbf{v} = 1 \cdot 1 + 1 \cdot (-1) = 0\text{.} \end{equation*}
Thus the two vectors \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal.
🔗
🔗
🔗
Before-and-after grid diagram for a symmetric matrix with orthogonal eigenvector directions.
The left panel shows a square coordinate grid with a red line in the direction \(\mathbf{u}=(1,1)\) and a blue line in the direction \(\mathbf{v}=(1,-1)\text{.}\) The right panel shows the image of the grid under \(A\text{:}\) the red direction is unchanged, while the blue direction is stretched to three times its original length. The red and blue eigenvector directions remain perpendicular in both panels.
🔗
Figure 5.4.6. Visualization of the action of matrix \(A\) from Activity 5.4.1 on a grid. The red line along eigenvector \(\mathbf{u}\) is unchanged (since \(\mathbf{u}\) is an eigenvector with eigenvalue 1), while the blue line along eigenvector \(\mathbf{v}\) is stretched by a factor of 3 (since \(\mathbf{v}\) is an eigenvector with eigenvalue 3). Adapted from: Stanford’s MATH 51 textbook.
🔗
Figure 5.4.6 should be compared to Figure 5.3.12. Notice that since the matrix \(A\) in Activity 5.4.1 is symmetric, the red and blue line in Figure 5.4.6 meet at right angles, since the vectors \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal, whereas the red and blue lines in Figure 5.3.12 do not meet at right angles, since the eigenvectors for that example are not orthogonal to one another.
🔗

Activity 5.4.2.

Consider the matrix
\begin{equation*} A = \begin{bmatrix} 1 \amp 1 \\ 1 \amp -1 \end{bmatrix}\text{.} \end{equation*}
Find an orthogonal matrix \(Q\) such that \(Q^{-1} A Q\) is diagonal.
🔗
Solution.
We proceed as in Activity 5.3.6, i.e., finding a basis of eigenvectors of \(A\text{,}\) and defining \(Q\) by taking those columns as eigenvectors. By Theorem 5.4.2, as long as the eigenvectors are orthonormal, the matrix \(Q\) will be orthogonal. We start by finding the eigenvalues of \(Q\text{,}\) by computing the characteristic polynomial. We calculate that
\begin{align*} c_A(\lambda) \amp = \det\begin{bmatrix} 1-\lambda \amp 1 \\ 1 \amp -1-\lambda \end{bmatrix}\\ \amp = (1-\lambda)(-1-\lambda) - 1\\ \amp = \lambda^2 - 2\\ \amp = (\lambda - \sqrt{2})(\lambda + \sqrt{2})\text{.} \end{align*}
The solutions to \(c_A(\lambda) = 0\) are thus given by \(\lambda = \pm \sqrt{2}\text{,}\) and so \(\lambda_1 = \sqrt{2}\) and \(\lambda_2 = - \sqrt{2}\) are the two eigenvalues of \(A\text{.}\)
🔗
To calculate an eigenvector with eigenvalue \(\sqrt{2}\text{,}\) we solve the system
\begin{equation*} \begin{bmatrix} 1-\sqrt{2} \amp 1 \\ 1 \amp -1-\sqrt{2} \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}\text{.} \end{equation*}
Expanding the left hand side and reading off the second entry of the resulting vector gives that
\begin{equation*} v_1 + (-1 - \sqrt{2}) v_2 = 0\text{.} \end{equation*}
Since the entries of the first column are distinct, we immediately see that the first column will contain a pivot when we row-reduce the matrix on the left hand side of this equation, and thus (since the matrix is not invertible), \(\mathbf{v}_2\) will be a free variable when specifying the set of solutions. Setting \(v_2 = 1\text{,}\) we obtain the basic eigenvector
\begin{equation*} \begin{bmatrix} 1 + \sqrt{2} \\ 1 \end{bmatrix}\text{.} \end{equation*}
We are looking for an orthonormal basis of eigenvectors, and so in particular, the eigenvectors we choose as a basis must all have length \(1\text{.}\) We thus consider a multiple of the basic eigenvector which has length \(1\text{.}\) Since the basic eigenvector has length
\begin{equation*} \sqrt{ (1 + \sqrt{2})^2 + (1)^2 } = \sqrt{ 4 + 2\sqrt{2} } \end{equation*}
The vector
\begin{equation*} \mathbf{v} = \frac{1}{\sqrt{4 + 2 \sqrt{2}}} \begin{bmatrix} 1 + \sqrt{2} \\ 1 \end{bmatrix} = \begin{bmatrix} \frac{1 + \sqrt{2}}{\sqrt{4 + 2 \sqrt{2}}} \\ \frac{1}{\sqrt{4 + 2 \sqrt{2}}} \end{bmatrix} \end{equation*}
has length \(1\text{.}\) This will be the first vector in our basis.
🔗
With a similar technique, one may find the basic eigenvector with eigenvalue \(- \sqrt{2}\text{.}\) It is the vector
\begin{equation*} \begin{bmatrix} 1 - \sqrt{2} \\ 1 \end{bmatrix}\text{.} \end{equation*}
Normalizing gives the vector
\begin{equation*} \mathbf{w} = \frac{1}{\sqrt{4 - 2 \sqrt{2}}} \begin{bmatrix} 1 - \sqrt{2} \\ 1 \end{bmatrix} = \begin{bmatrix} \frac{1 - \sqrt{2}}{\sqrt{4 - 2 \sqrt{2}}} \\ \frac{1}{\sqrt{4 - 2 \sqrt{2}}} \end{bmatrix}\text{,} \end{equation*}
which has length \(1\text{.}\) This is the second vector in our basis.
🔗
Using Theorem 5.4.1, since \(A\) is symmetric, \(\mathbf{v}\) and \(\mathbf{w}\) are orthogonal, and thus, by Theorem 4.1.8, are linearly independent, thus forming a basis. But this means that the matrix
\begin{equation*} Q = \begin{bmatrix} \frac{1 + \sqrt{2}}{\sqrt{4+2\sqrt{2}}} \amp \frac{1 - \sqrt{2}}{\sqrt{4-2\sqrt{2}}} \\ \frac{1}{\sqrt{4+2\sqrt{2}}} \amp \frac{1}{\sqrt{4-2\sqrt{2}}} \end{bmatrix} \end{equation*}
is orthogonal, and
\begin{equation*} Q^{-1}AQ = \begin{bmatrix} \sqrt{2} \amp 0 \\ 0 \amp - \sqrt{2} \end{bmatrix} \end{equation*}
is diagonal.
🔗
🔗
🔗

Activity 5.4.3.

Suppose that \(A\) is orthogonally diagonalizable, i.e., we can find an orthogonal matrix \(Q\) so that \(Q^TAQ = D\text{,}\) where \(D\) is diagonal. Then \(A = QDQ^T\text{.}\)
🔗
Solution.
We calculate that
\begin{equation*} QDQ^T = Q(Q^TAQ)Q^T = (QQ^T) A (QQ^T) = I A I = A\text{.} \end{equation*}
🔗
🔗
🔗

Remark 5.4.7.

The factorization \(A = QDQ^T\) writes \(A\) as a product of three matrices: an orthogonal matrix, a diagonal matrix, and the transpose of the orthogonal matrix. This product is sometimes called the spectral decomposition of a symmetric matrix \(A\text{.}\)
🔗
🔗
🔗

Subsection Principal axes theorem

Recall Theorem 5.3.10. That theorem tells us that if \(A\) is orthogonally diagonalizable, where \(Q^{-1} A Q\) is diagonal, then the columns of \(Q\) must be linearly independent eigenvectors of \(A\text{.}\) In addition, in order for \(Q\) to be orthogonal, Theorem 5.4.2 tells us that the eigenvectors which form the columns of \(Q\) must be orthonormal.
🔗
Remarkably, all symmetric matrices are orthogonally diagonalizable.
🔗

Why is this true?.

We do not prove the “hard” part of the theorem (that the third property implies the second). But the proof that the second property implies the third is more simple (see Activity 5.4.4).
🔗
🔗

Remark 5.4.9.

Theorem 5.4.8 is also called The Spectral Theorem for symmetric matrices (the set of eigenvalues of a matrix is often called the spectrum of the matrix).
🔗
🔗
We do not prove the "hard" part of this theorem, that is, that (3) implies (2). Let’s however check that (2) implies (3).
🔗

Activity 5.4.4.

Show that if \(A\) is orthogonally diagonalizable, then \(A\) is symmetric.
🔗
Solution.
Suppose \(A\) is orthogonally diagonalizable. Then there exists an orthogonal matrix \(Q\) and a diagonal matrix \(D\) such that \(Q^TAQ = D\text{.}\) Activity 5.4.3 tells us that \(A = QDQ^T\text{.}\) But now we check that
\begin{align*} A^T \amp = (QDQ^T)^T\\ \amp = (Q^T)^T D^T Q^T\\ \amp = Q D Q^T = A\text{.} \end{align*}
The equation \(A^T = A\) means precisely that \(A\) is symmetric.
🔗
🔗
🔗

Activity 5.4.5.

Let
\begin{equation*} A = \begin{bmatrix} 1 \amp 0 \amp -1 \\ 0 \amp 1 \amp 2 \\ -1 \amp 2 \amp 5 \end{bmatrix} \end{equation*}
Find an orthogonal matrix \(Q\) such that \(Q^{-1} A Q\) is diagonal.
🔗
Solution.
First, compute the eigenvalues. We calculate that
\begin{align*} \det(A - \lambda I) \amp = \det\begin{bmatrix} 1-\lambda \amp 0 \amp -1 \\ 0 \amp 1-\lambda \amp 2 \\ -1 \amp 2 \amp 5-\lambda \end{bmatrix}\\ \amp = (1- \lambda) [(1 -\lambda)(5 - \lambda) - 4] + (-1)[0 - (1-\lambda)(-1)]\\ \amp = (1- \lambda) [(1 -\lambda)(5 - \lambda) - 4] - (1-\lambda)\\ \amp = (1- \lambda)[(1 -\lambda)(5 - \lambda) - 4 - 1]\\ \amp = (1- \lambda)[-6 \lambda + \lambda^2]\\ \amp = -\lambda(\lambda-1)(\lambda-6) \end{align*}
This calculation verifies that the eigenvalues of \(A\) are \(\lambda_1 = 0\text{,}\) \(\lambda_2 = 1\text{,}\) and \(\lambda_3 = 6\text{.}\) Now we proceed to find an eigenvector for each of these eigenvalues.
🔗
For the eigenvalue \(\lambda_1 = 0\text{,}\) we solve the equation \(A\mathbf{v}_1 = \mathbf{0}\text{,}\) which we expand as
\begin{equation*} \begin{bmatrix} 1 \amp 0 \amp -1 \\ 0 \amp 1 \amp 2 \\ -1 \amp 2 \amp 5 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \end{equation*}
By observation, we see that the first and second columns will contain pivots in the reduced row echelon form of the matrix in the left hand side, so setting \(v_3 = 1\) gives the basic eigenvector
\begin{equation*} \begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix}\text{.} \end{equation*}
Normalizing gives the normalized eigenvector
\begin{equation*} \mathbf{v}_1 = \frac{1}{\sqrt{6}} \begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix} = \begin{bmatrix} 1/\sqrt{6} \\ -2/\sqrt{6} \\ 1/\sqrt{6} \end{bmatrix}\text{,} \end{equation*}
which will be the first vector in our basis.
🔗
For the eigenvalue \(\lambda_2 = 1\text{,}\) we solve the equation \((A - I)\mathbf{v}_2 = \mathbf{0}\text{,}\) which we expand as
\begin{equation*} \begin{bmatrix} 0 \amp 0 \amp -1 \\ 0 \amp 0 \amp 2 \\ -1 \amp 2 \amp 4 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \end{equation*}
We now see that first and third columns will contain pivots when we row reduce this matrix, so setting \(v_2 = 1\) and solving gives the basic eigenvector
\begin{equation*} \begin{bmatrix} 2 \\ 1 \\ 0 \end{bmatrix}\text{.} \end{equation*}
Normalizing gives the eigenvector
\begin{equation*} \mathbf{v}_2 = \frac{1}{\sqrt{5}} \begin{bmatrix} 2 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2/\sqrt{5} \\ 1/\sqrt{5} \\ 0 \end{bmatrix}\text{.} \end{equation*}
This will be the second vector in our basis.
🔗
For \(\lambda_3 = 6\text{,}\) we solve \((A - 6I)\mathbf{v}_3 = \mathbf{0}\text{,}\) i.e.,
\begin{equation*} \begin{bmatrix} -5 \amp 0 \amp -1 \\ 0 \amp -5 \amp 2 \\ -1 \amp 2 \amp -1 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \end{equation*}
Solving this system gives the basic eigenvector
\begin{equation*} \begin{bmatrix} -1/5 \\ 2/5 \\ 1 \end{bmatrix}\text{.} \end{equation*}
Multiplying by \(5\) (to prevent working with fractions) gies the integer eigenvector
\begin{equation*} \begin{bmatrix} -1 \\ 2 \\ 5 \end{bmatrix} \end{equation*}
Normalizing gives the eigenvector
\begin{equation*} \mathbf{v}_3 = \frac{1}{\sqrt{30}} \begin{bmatrix} -1 \\ 2 \\ 5 \end{bmatrix} = \begin{bmatrix} -1/\sqrt{30} \\ 2/\sqrt{30} \\ 5/\sqrt{30} \end{bmatrix}\text{.} \end{equation*}
This will be the third and final vector in our basis.
🔗
We now set
\begin{equation*} Q = \begin{bmatrix} 1/\sqrt{6} \amp 2/\sqrt{5} \amp -1/\sqrt{30} \\ -2/\sqrt{6} \amp 1/\sqrt{5} \amp 2/\sqrt{30} \\ 1/\sqrt{6} \amp 0 \amp 5/\sqrt{30} \end{bmatrix}\text{.} \end{equation*}
Then
\begin{equation*} Q^{-1}AQ = \begin{bmatrix} 0 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \\ 0 \amp 0 \amp 6 \end{bmatrix}\text{.} \end{equation*}
We have thus found an orthogonal diagonalization of the matrix \(A\text{.}\)
🔗
🔗
🔗
In Activity 5.4.5, the eigenvalues of \(A\) were distinct. To come up with a general strategy to obtain an orthogonal diagonalization, we need to make one further observation.
🔗

Definition 5.4.10. Eigenspaces of a matrix.

Suppose \(A\) is a square matrix. Then for any scalar \(\lambda\text{,}\) the set
\begin{equation*} E_\lambda(A) = \{ \mathbf{x} \in \R^n: A \mathbf{x} = \lambda \mathbf{x} \} \end{equation*}
is called the eigenspace of \(A\) corresponding to \(\lambda\text{.}\)
🔗
🔗

Why is this true?.

Note that \(\mathbf{x} \in E_\lambda(A)\) precisely when \(\mathbf{x}\) solves the equation \((A - \lambda I) \mathbf{x} = \mathbf{0}\text{.}\) Thus \(E_\lambda(A)\) is precisely the null space of the matrix \(A - \lambda I\text{,}\) and the theorem follows from Theorem 2.3.11.
🔗
🔗

Note 5.4.12. Procedure for orthogonally diagonalizing a symmetric matrix \(A\).

  1. Find the eigenvalues of \(A\text{.}\)
    🔗
  2. For each eigenvalue \(\lambda\) of \(A\text{,}\) find a basis for \(E_\lambda(A)\text{,}\) and convert it to an orthonormal basis using the Gram-Schmidt orthogonalization process (see Theorem 4.2.8) if necessary.
    🔗
  3. Collecting the orthonormal bases for each of the eigenspaces gives an orthonormal basis of eigenvector for \(A\text{.}\) Taking these eigenvectors as column vectors of a matrix \(Q\) gives an orthonormal matrix such that \(Q^{-1}AQ\) is diagonal.
    🔗
🔗
🔗

Activity 5.4.6.

If
\begin{equation*} A = \begin{bmatrix} 8 \amp -2 \amp 2 \\ -2 \amp 5 \amp 4 \\ 2 \amp 4 \amp 5 \end{bmatrix}, \end{equation*}
compute an orthonormal diagonalization of \(A\text{.}\) You may use the fact that the characteristic polynomial of \(A\) is \(\lambda (\lambda - 9)^2\text{,}\) and the null space of \(A\) is spanned by the vector
\begin{equation*} \begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix}\text{.} \end{equation*}
🔗
Solution.
The eigenvalues are \(\lambda_1 = 0\) and \(\lambda_2 = 9\text{.}\) Since the null space of \(A\) is spanned by the vector
\begin{equation*} \begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix}\text{,} \end{equation*}
the eigenvectors of \(A\) with eigenvalue \(0\) are all multiples of this vector. Thus an orthonormal basis for \(E_0(A)\) is given by
\begin{equation*} \left\{ \begin{bmatrix} 1/3 \\ 2/3 \\ -2/3 \end{bmatrix} \right\}\text{.} \end{equation*}
On the other hand, to find an orthonormal basis for \(E_9(A)\) we form the matrix
\begin{equation*} 9I - A = \begin{bmatrix} 9 \amp 0 \amp 0 \\ 0 \amp 9 \amp 0 \\ 0 \amp 0 \amp 9 \end{bmatrix} - \begin{bmatrix} 8 \amp -2 \amp 2 \\ -2 \amp 5 \amp 4 \\ 2 \amp 4 \amp 5 \end{bmatrix} = \begin{bmatrix} 1 \amp 2 \amp -2 \\ 2 \amp 4 \amp -4 \\ -2 \amp -4 \amp 4 \end{bmatrix}\text{.} \end{equation*}
Using the row operations \(R_2 \rightarrow R_2 - 2R_1\) and \(R_3 \rightarrow R_3 + 2R_1\text{,}\) we obtain the reduced row echelon form for the matrix \(9I - A\text{:}\) the matrix
\begin{equation*} \begin{bmatrix} 1 \amp 2 \amp -2 \\ 0 \amp 0 \amp 0 \\ 0 \amp 0 \amp 0 \end{bmatrix}\text{.} \end{equation*}
We thus obtain the two basic solutions
\begin{equation*} \mathbf{v}_1 = \begin{bmatrix} -2 \\ 1 \\ 0 \end{bmatrix} \quad\text{and}\quad \mathbf{v}_2 = \begin{bmatrix} 2 \\ 0 \\ 1 \end{bmatrix}\text{.} \end{equation*}
We now apply Gram-Schmidt to this pair of vectors, which form a basis of \(E_9(A)\text{,}\) in order to obtain an orthogonal basis. We define
\begin{equation*} \mathbf{w}_1=\mathbf{v}_1 \qquad\text{and}\qquad \mathbf{w}_2=\mathbf{v}_2-\frac{\mathbf{v}_2\cdot\mathbf{w}_1}{\mathbf{w}_1\cdot\mathbf{w}_1}\mathbf{w}_1. \end{equation*}
We calculate that
\begin{equation*} \frac{\mathbf{v}_2 \cdot \mathbf{w}_1}{\mathbf{w}_1 \cdot \mathbf{w}_1} = \frac{-4}{5} \end{equation*}
and thus that
\begin{equation*} \mathbf{w}_2 = \begin{bmatrix} 2 \\ 0 \\ 1 \end{bmatrix} + \frac45 \begin{bmatrix} -2 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2/5 \\ 4/5 \\ 1 \end{bmatrix}\text{.} \end{equation*}
Normalizing gives the orthonormal basis
\begin{equation*} \left\{ \frac{\mathbf{w}_1}{\| \mathbf{w}_1 \|}, \frac{\mathbf{w}_2}{\| \mathbf{w}_2 \|} \right\} = \left\{ \begin{bmatrix} -2/\sqrt(5) \\ 1/\sqrt{5} \\ 0 \end{bmatrix}, \begin{bmatrix} 2/\sqrt{45} \\ 4/\sqrt{45} \\ 5/\sqrt{45} \end{bmatrix} \right\} \end{equation*}
for \(E_9(A)\text{.}\)
🔗
Combining the two orthonormal bases we have calculated gives an orthonormal basis
\begin{equation*} \left\{ \begin{bmatrix} 1/3 \\ 2/3 \\ -2/3 \end{bmatrix}, \begin{bmatrix} -2/\sqrt{5} \\ 1/\sqrt{5} \\ 0 \end{bmatrix}, \begin{bmatrix} 2/\sqrt{45} \\ 4/\sqrt{45} \\ 5/\sqrt{45} \end{bmatrix} \right\} \end{equation*}
of \(\R^n\) consisting of eigenvectors, and so if we define
\begin{equation*} Q = \begin{bmatrix} 1/3 \amp -2/\sqrt{5} \amp 2/\sqrt{45} \\ 2/3 \amp 1/\sqrt{5} \amp 4/\sqrt{45} \\ -2/3 \amp 0 \amp 5/\sqrt{45} \end{bmatrix}\text{,} \end{equation*}
then
\begin{equation*} Q^{-1}AQ = \begin{bmatrix} 0 \amp 0 \amp 0 \\ 0 \amp 9 \amp 0 \\ 0 \amp 0 \amp 9 \end{bmatrix} \end{equation*}
gives an orthogonal diagonalization of \(A\text{.}\)
🔗
🔗
🔗

Definition 5.4.13. Positive definite, negative definite, and indefinite matrices.

A symmetric matrix is called positive definite if all of its eigenvalues are strictly positive. It is called negative definite if all of its eigenvalues are strictly negative. It is called indefinite if it has at least one strictly positive eigenvalue and at least one strictly negative eigenvalue.
🔗
🔗
The fact that every symmetric matrix is diagonalizable can often be used to determine structural properties of such matrices. For instance, consider the theorem below.
🔗

Why is this true?.

By Theorem 5.4.8, we can write \(C = QDQ^T\text{,}\) where \(Q\) is an orthogonal matrix, and \(D = \diag(\lambda_1,\dots,\lambda_n)\) is a diagonal matrix. Then if \(\mathbf{y} = Q^T \mathbf{x}\text{,}\)
\begin{equation*} \mathbf{x}^T C \mathbf{x} = \mathbf{x}^T (Q D Q^T) \mathbf{x} = \mathbf{y}^T D \mathbf{y} = \lambda_1 y_1^2 + \lambda_2 y_2^2 + \cdots + \lambda_n y_n^2. \end{equation*}
This quantity is positive for all \(\mathbf{x} \neq 0\text{,}\) or equivalently, since \(Q^T\) is invertible, for all \(\mathbf{y} \neq 0\text{,}\) if and only if \(\lambda_1,\dots,\lambda_n \gt 0\text{.}\)
🔗
🔗
🔗

Subsection Quadratic forms and definiteness

Definition 5.4.15.

The quadratic form associated with an \(n \times n\) matrix \(A\) is the scalar-valued function \(Q_A\) on \(\R^n\) defined by \(Q_A(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}\text{.}\)
🔗
🔗

Activity 5.4.7. A quadratic form as curvature.

Let
\begin{equation*} H= \begin{bmatrix} 4\amp0\\ 0\amp1 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf{h}^T H\mathbf{h}\) for
    \begin{equation*} \mathbf{h}= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf{h}= \begin{bmatrix} 0\\ 1 \end{bmatrix}, \qquad \mathbf{h}= \begin{bmatrix} 1\\ 1 \end{bmatrix}. \end{equation*}
    🔗
    🔗
  2. Write \(\mathbf{h}^T H\mathbf{h}\) for a general vector
    \begin{equation*} \mathbf{h}= \begin{bmatrix} h_1\\ h_2 \end{bmatrix}. \end{equation*}
    🔗
    🔗
  3. In which coordinate direction is the quadratic form larger?
    🔗
    🔗
Tags. [U5-LO4, U5-LO5 | P+C | Core]
🔗
Solution.
We compute
\begin{equation*} \begin{bmatrix} 1\\ 0 \end{bmatrix}^T H \begin{bmatrix} 1\\ 0 \end{bmatrix} =4, \end{equation*}
\begin{equation*} \begin{bmatrix} 0\\ 1 \end{bmatrix}^T H \begin{bmatrix} 0\\ 1 \end{bmatrix} =1, \end{equation*}
and
\begin{equation*} \begin{bmatrix} 1\\ 1 \end{bmatrix}^T H \begin{bmatrix} 1\\ 1 \end{bmatrix} =5. \end{equation*}
For a general vector,
\begin{equation*} \mathbf{h}^T H\mathbf{h}=4h_1^2+h_2^2. \end{equation*}
The quadratic form is larger in the first coordinate direction than in the second coordinate direction. For Hessians, this kind of comparison measures curvature in different directions.
🔗
🔗
🔗
If we can understand quadratic forms, we can understand the quadratic approximations of functions at critical points.
🔗

Activity 5.4.8.

For a symmetric \(3 \times 3\) matrix \(A = [A_{ij}]\text{,}\) write out the quadratic form \(q_A(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}\) explicitly in terms of the entries of \(A\) and the components of \(\mathbf{x} = (x_1, x_2, x_3)\text{.}\)
🔗
Solution.
\begin{align*} q_A(\mathbf{x}) \amp = \begin{bmatrix} x_1 \amp x_2 \amp x_3 \end{bmatrix} \begin{bmatrix} A_{11} \amp A_{12} \amp A_{13} \\ A_{21} \amp A_{22} \amp A_{23} \\ A_{31} \amp A_{32} \amp A_{33} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}\\ \amp = \begin{bmatrix} x_1 \amp x_2 \amp x_3 \end{bmatrix} \begin{bmatrix} A_{11}x_1 + A_{12}x_2 + A_{13}x_3 \\ A_{21}x_1 + A_{22}x_2 + A_{23}x_3 \\ A_{31}x_1 + A_{32}x_2 + A_{33}x_3 \end{bmatrix}\\ \amp = x_1(A_{11}x_1 + A_{12}x_2 + A_{13}x_3)\\ \amp + x_2(A_{21}x_1 + A_{22}x_2 + A_{23}x_3)\\ \amp + x_3(A_{31}x_1 + A_{32}x_2 + A_{33}x_3)\\ \amp = A_{11}x_1^2 + A_{22}x_2^2 + A_{33}x_3^2\\ \amp + A_{12}x_1x_2 + A_{13}x_1x_3 + A_{21}x_2x_1\\ \amp + A_{23}x_2x_3 + A_{31}x_3x_1 + A_{32}x_3x_2 \end{align*}
Since \(A\) is symmetric, we have \(A_{ij} = A_{ji}\) for all \(i, j\text{.}\) Using this property:
\begin{align*} \amp q_A(\mathbf{x}) = A_{11}x_1^2 + A_{22}x_2^2 + A_{33}x_3^2 + (A_{12} + A_{21})x_1x_2\\ \amp + (A_{13} + A_{31})x_1x_3 + (A_{23} + A_{32})x_2x_3\\ \amp = A_{11}x_1^2 + A_{22}x_2^2 + A_{33}x_3^2\\ \amp + 2A_{12}x_1x_2 + 2A_{13}x_1x_3 + 2A_{23}x_2x_3 \end{align*}
🔗
🔗
🔗

Remark 5.4.16.

More generally, for a symmetric \(n \times n\) matrix \(A = [A_{ij}]\text{,}\) the quadratic form \(q_A(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}\) can be written as:
\begin{equation*} q_A(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} = \sum_{i=1}^n \sum_{j=1}^n A_{ij} x_i x_j = \sum_{i=1}^n A_{ii}x_i^2 + 2\sum_{1 \leq i \lt j \leq n} A_{ij}x_i x_j \end{equation*}
In particular, note that if \(A\) is a diagonal matrix, then
\begin{equation*} q_A(\mathbf{x}) = A_{11} x_1^2 + \cdots + A_{nn} x_n^2 \end{equation*}
which is simple to understand. We will use spectral theory to show that a rotation of a coordinate system reduces a quadratic form to a quadratic form defined by a diagonal matrix. The signs of the diagonal entries will then tell us the behavior of the critical point.
🔗
🔗
Recall the spectral theorem, i.e., Theorem 5.4.8, which says that there is an orthogonal matrix \(Q\) so that \(Q^T A Q\) is a diagonal matrix \(D\text{.}\)
🔗

Why is this true?.

Since \(\mathbf{x}=Q\mathbf{y}\text{,}\) we have
\begin{align*} Q_A(\mathbf{x}) \amp = \mathbf{x}^TA\mathbf{x}\\ \amp = (Q\mathbf{y})^TA(Q\mathbf{y})\\ \amp = \mathbf{y}^TQ^TAQ\mathbf{y}\\ \amp = \mathbf{y}^TD\mathbf{y}\\ \amp = \lambda_1y_1^2+\cdots+\lambda_ny_n^2.\text{.} \end{align*}
🔗
🔗

Definition 5.4.18. The principal axes.

If \(A\) is a symmetric, \(n \times n\) matrix, then an orthonormal basis of eigenvectors of \(A\) is called a set of principal axes for the quadratic form \(Q_A\text{.}\)
🔗
🔗

Activity 5.4.9.

Consider the quadratic form \(q(x,y) = 13x^2 - 6xy + 5y^2\text{.}\)
🔗
(a)
Find the symmetric matrix \(A\) such that \(q = q_A\text{,}\) i.e., such that
\begin{equation*} q(x,y) = \begin{bmatrix} x \amp y \end{bmatrix} A \begin{bmatrix} x \\ y \end{bmatrix}\text{.} \end{equation*}
🔗
Solution.
If
\begin{equation*} A = \begin{bmatrix} a \amp b \\ b \amp c \end{bmatrix}\text{,} \end{equation*}
then
\begin{equation*} q_A(x,y)=ax^2+2bxy+cy^2. \end{equation*}
So \(a = 13\text{,}\) \(b = -3\text{,}\) and \(c = 5\text{,}\) i.e.,
\begin{equation*} A = \begin{bmatrix} 13 \amp -3 \\ -3 \amp 5 \end{bmatrix}\text{.} \end{equation*}
🔗
🔗
🔗
(b)
Find the eigenvalues and a basis of eigenvectors for \(A\text{.}\)
🔗
Solution.
The characteristic polynomial of \(A\) (recall Definition 5.3.2) is given by
\begin{align*} c_A(\lambda) \amp = \det(\lambda I_2 - A)\\ \amp = \det \begin{bmatrix} \lambda - 13 \amp 3 \\ 3 \amp \lambda - 5 \end{bmatrix}\\ \amp = (\lambda - 13) (\lambda - 5) - 9\\ \amp = \lambda^2 - 18 \lambda + 56\\ \amp = (\lambda - 14)(\lambda - 4)\text{.} \end{align*}
So \(\lambda = 4\) and \(\lambda = 14\) are the two eigenvalues of \(A\text{.}\)
🔗
For the eigenvalue \(\lambda_1 = 14\text{,}\) we solve \((A - \lambda_1 I)\mathbf{v}_1 = \mathbf{0}\text{:}\)
\begin{equation*} \begin{bmatrix} -1 \amp -3 \\ -3 \amp -9 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation*}
That is, \(v_1 = -3v_2\text{.}\) Taking \(v_2 = 1\text{,}\) we get \(v_1 = -3\text{.}\) So an eigenvector for \(\lambda_1\) is
\begin{equation*} \mathbf{v}_1 = \begin{bmatrix} -3 \\ 1 \end{bmatrix}\text{.} \end{equation*}
For the eigenvalue \(\lambda_2 = 4\text{,}\) we solve \((A - \lambda_2 I)\mathbf{v}_2 = \mathbf{0}\text{:}\)
\begin{equation*} \begin{bmatrix} 9 \amp -3 \\ -3 \amp 1 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation*}
That is, \(v_2 = 3v_1\text{.}\) Taking \(v_1 = 1\text{,}\) we get \(v_2 = 3\text{.}\) So an eigenvector for \(\lambda_2\) is
\begin{equation*} \mathbf{v}_2 = \begin{bmatrix} 1 \\ 3 \end{bmatrix}\text{.} \end{equation*}
🔗
🔗
🔗
(c)
(c) Apply the previous theorem to express \(q(x_1, x_2)\) in terms of new coordinates \(y_1\) and \(y_2\) that eliminate the cross term.
🔗
Solution.
We first normalize the eigenvectors to get:
\begin{align*} \mathbf{v}'_1 \amp = \frac{\mathbf{v}_1}{||\mathbf{v}_1||} = \frac{\begin{bmatrix} -3 \\ 1 \end{bmatrix}}{\sqrt{(-3)^2 + 1^2}} = \frac{1}{\sqrt{10}}\begin{bmatrix} -3 \\ 1 \end{bmatrix}\\ \mathbf{v}'_2 \amp = \frac{\mathbf{v}_2}{||\mathbf{v}_2||} = \frac{\begin{bmatrix} 1 \\ 3 \end{bmatrix}}{\sqrt{1^2 + 3^2}} = \frac{1}{\sqrt{10}}\begin{bmatrix} 1 \\ 3 \end{bmatrix} \end{align*}
The orthogonal matrix \(Q\) with these unit eigenvectors as columns diagonalizes \(A\text{.}\) When we express a vector \(\mathbf{x} = Q\mathbf{y} = y_1\mathbf{v}'_1 + y_2\mathbf{v}'_2\text{,}\) the quadratic form becomes:
\begin{align*} q(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} \amp = \mathbf{y}^T Q^T A Q \mathbf{y}\\ \amp = \mathbf{y}^T \begin{bmatrix} \lambda_1 \amp 0 \\ 0 \amp \lambda_2 \end{bmatrix} \mathbf{y}\\ \amp = \lambda_1 y_1^2 + \lambda_2 y_2^2\\ \amp = 14 y_1^2 + 4 y_2^2\text{.} \end{align*}
🔗
🔗
🔗
🔗

Activity 5.4.10.

Consider the quadratic form
\begin{equation*} q(x,y) = x^2 + 4xy - 2y^2\text{.} \end{equation*}
🔗
(a)
Find the symmetric matrix \(A\) such that \(q = q_A\text{.}\)
🔗
Solution.
The matrix is
\begin{equation*} A = \begin{bmatrix} 1 \amp 2 \\ 2 \amp -2 \end{bmatrix}\text{.} \end{equation*}
🔗
🔗
🔗
(b)
Find the eigenvalues and corresponding eigenvectors of \(A\text{.}\)
🔗
Solution.
The characteristic polynomial is
\begin{align*} \det(A-\lambda I) \amp = \det \begin{bmatrix} 1-\lambda \amp 2\\ 2 \amp -2-\lambda \end{bmatrix}\\ \amp = (1-\lambda)(-2-\lambda)-4\\ \amp = \lambda^2+\lambda-6\\ \amp = (\lambda-2)(\lambda+3).\text{.} \end{align*}
Thus the eigenvalues are \(2\) and \(-3\text{.}\) For eigenvalue \(\lambda_1 = 2\text{,}\) we solve \((A - \lambda_1 I)\mathbf{v}_1 = \mathbf{0}\text{:}\)
\begin{equation*} \begin{bmatrix} -1 \amp 2 \\ 2 \amp -4 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation*}
That is, \(v_1 = 2v_2\text{.}\) Taking \(v_2 = 1\text{,}\) we get \(v_1 = 2\text{.}\) So
\begin{equation*} \mathbf{v}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix} \end{equation*}
is an eigenvector for \(\lambda_1\text{.}\) For the eigenvalue \(\lambda_2 = -3\text{,}\) we solve \((A - \lambda_2 I)\mathbf{v}_2 = \mathbf{0}\text{:}\)
\begin{equation*} \begin{bmatrix} 4 \amp 2 \\ 2 \amp 1 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation*}
That is, \(v_2 = -2v_1\text{.}\) Taking \(v_1 = 1\text{,}\) we get \(v_2 = -2\text{.}\) So an eigenvector for \(\lambda_2\) is \(\mathbf{v}_2 = \begin{bmatrix} 1 \\ -2 \end{bmatrix}\text{.}\)
🔗
🔗
🔗
(c)
Apply the previous theorem to express \(q(x_1, x_2)\) in terms of new coordinates \(y_1\) and \(y_2\) that eliminate the cross term, and classify the level curves \(q(x_1, x_2) = c\) for different values of \(c\text{.}\)
🔗
Solution.
We first normalize the eigenvectors to get:
\begin{align*} \mathbf{v}'_1 \amp = \frac{\mathbf{v}_1}{||\mathbf{v}_1||} = \frac{\begin{bmatrix} 2 \\ 1 \end{bmatrix}}{\sqrt{2^2 + 1^2}} = \frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}\\ \mathbf{v}'_2 \amp = \frac{\mathbf{v}_2}{||\mathbf{v}_2||} = \frac{\begin{bmatrix} 1 \\ -2 \end{bmatrix}}{\sqrt{1^2 + (-2)^2}} = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix} \end{align*}
The orthogonal matrix \(Q\) with these unit eigenvectors as columns diagonalizes \(A\text{.}\) When we express a vector \(\mathbf{x} = Q\mathbf{y} = y_1\mathbf{v}'_1 + y_2\mathbf{v}'_2\text{,}\) the quadratic form becomes:
\begin{align*} q(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} \amp = \mathbf{y}^T Q^T A Q \mathbf{y}\\ \amp = \mathbf{y}^T \begin{bmatrix} \lambda_1 \amp 0 \\ 0 \amp \lambda_2 \end{bmatrix} \mathbf{y}\\ \amp = \lambda_1 y_1^2 + \lambda_2 y_2^2\\ \amp = 2 y_1^2 - 3 y_2^2 \end{align*}
🔗
🔗
🔗
🔗

Why is this true?.

🔗
🔗
PrevTopNext