Section C.5 Unit 5 applied, geometric, and computational interpretation
These solution sketches correspond to the additional applied and computational interpretation exercises at the end of Unit 5.
Subsection Eigenvectors as special directions
Learning outcomes. U5-LO1.
We compute
\begin{equation*}
A\mathbf{v}_1=
\begin{bmatrix}
3\amp1\\
0\amp2
\end{bmatrix}
\begin{bmatrix}
1\\
0
\end{bmatrix}
=
\begin{bmatrix}
3\\
0
\end{bmatrix}
=
3\mathbf{v}_1.
\end{equation*}
So \(\mathbf{v}_1\) is an eigenvector with eigenvalue \(3\text{.}\)
Next,
\begin{equation*}
A\mathbf{v}_2=
\begin{bmatrix}
3\amp1\\
0\amp2
\end{bmatrix}
\begin{bmatrix}
-1\\
1
\end{bmatrix}
=
\begin{bmatrix}
-2\\
2
\end{bmatrix}
=
2
\begin{bmatrix}
-1\\
1
\end{bmatrix}
=
2\mathbf{v}_2.
\end{equation*}
So \(\mathbf{v}_2\) is an eigenvector with eigenvalue \(2\text{.}\) The matrix map stretches the \(\mathbf{v}_1\)-direction by a factor of \(3\) and the \(\mathbf{v}_2\)-direction by a factor of \(2\text{.}\)
Subsection Diagonalization and matrix powers
Learning outcomes. U5-LO1, U5-LO2.
The columns of \(P\) are
\begin{equation*}
\mathbf v_1=
\begin{bmatrix}
1\\0
\end{bmatrix},
\qquad
\mathbf v_2=
\begin{bmatrix}
1\\1
\end{bmatrix}.
\end{equation*}
Direct calculation gives
\begin{equation*}
A\mathbf v_1=2\mathbf v_1,\qquad A\mathbf v_2=3\mathbf v_2.
\end{equation*}
Thus the columns of \(P\) are eigenvectors with eigenvalues \(2\) and \(3\text{,}\) respectively.
We have
\begin{equation*}
P^{-1}=
\begin{bmatrix}
1\amp-1\\
0\amp1
\end{bmatrix}.
\end{equation*}
Therefore
\begin{align*}
PDP^{-1}\amp=
\begin{bmatrix}1\amp1\\0\amp1\end{bmatrix}
\begin{bmatrix}2\amp0\\0\amp3\end{bmatrix}
\begin{bmatrix}1\amp-1\\0\amp1\end{bmatrix}\\
\amp=\begin{bmatrix}2\amp1\\0\amp3\end{bmatrix}=A.
\end{align*}
Since \(D^k=\operatorname{diag}(2^k,3^k)\text{,}\)
\begin{align*}
A^k\amp=PD^kP^{-1}\\
\amp=
\begin{bmatrix}1\amp1\\0\amp1\end{bmatrix}
\begin{bmatrix}2^k\amp0\\0\amp3^k\end{bmatrix}
\begin{bmatrix}1\amp-1\\0\amp1\end{bmatrix}\\
\amp=
\begin{bmatrix}
2^k\amp3^k-2^k\\
0\amp3^k
\end{bmatrix}.
\end{align*}
In particular,
\begin{equation*}
A^4=
\begin{bmatrix}
16\amp65\\
0\amp81
\end{bmatrix}.
\end{equation*}
Diagonalization reduces repeated matrix actions to taking powers of the diagonal entries of \(D\) and then changing coordinates with \(P\) and \(P^{-1}\text{.}\)
Subsection Quadratic form and definiteness
Learning outcomes. U5-LO4, U5-LO5.
For the three listed vectors,
\begin{equation*}
\begin{bmatrix}
1\\
0
\end{bmatrix}^T
H
\begin{bmatrix}
1\\
0
\end{bmatrix}
=4,
\end{equation*}
\begin{equation*}
\begin{bmatrix}
0\\
1
\end{bmatrix}^T
H
\begin{bmatrix}
0\\
1
\end{bmatrix}
=1,
\end{equation*}
and
\begin{equation*}
\begin{bmatrix}
1\\
1
\end{bmatrix}^T
H
\begin{bmatrix}
1\\
1
\end{bmatrix}
=5.
\end{equation*}
For a general vector
\begin{equation*}
\mathbf{h}=
\begin{bmatrix}
h_1\\
h_2
\end{bmatrix},
\end{equation*}
\begin{equation*}
\mathbf{h}^TH\mathbf{h}=4h_1^2+h_2^2.
\end{equation*}
This is positive for every nonzero \(\mathbf{h}\text{,}\) so \(H\) is positive definite. If \(H\) is the Hessian at a critical point, the second derivative test gives a local minimum.
Subsection Hessian eigenvalue classification
Learning outcomes. U5-LO5.
If the Hessian eigenvalues are \(3,5\text{,}\) the Hessian is positive definite, so the critical point is a local minimum.
If the Hessian eigenvalues are \(-2,-7\text{,}\) the Hessian is negative definite, so the critical point is a local maximum.
If the Hessian eigenvalues are \(4,-1\text{,}\) the Hessian is indefinite, so the critical point is a saddle point.
If the Hessian eigenvalues are \(0,2\text{,}\) the test is inconclusive. The zero eigenvalue means the quadratic approximation has a flat direction, so the second-order term alone does not decide the local behavior.
Subsection Least squares from gradients
Learning outcomes. U5-LO6, U4-LO4.
We compute
\begin{equation*}
A^TA=
\begin{bmatrix}
3\amp3\\
3\amp5
\end{bmatrix},
\qquad
A^T\mathbf{b}=
\begin{bmatrix}
5\\
6
\end{bmatrix}.
\end{equation*}
The normal equations are
\begin{equation*}
A^TA\hat{\mathbf{x}}=A^T\mathbf{b},
\end{equation*}
or
\begin{equation*}
\begin{bmatrix}
3\amp3\\
3\amp5
\end{bmatrix}
\hat{\mathbf{x}}
=
\begin{bmatrix}
5\\
6
\end{bmatrix}.
\end{equation*}
Solving gives
\begin{equation*}
\hat{\mathbf{x}}=
\begin{bmatrix}
7/6\\
1/2
\end{bmatrix}.
\end{equation*}
Then
\begin{equation*}
A\hat{\mathbf{x}}=
\begin{bmatrix}
7/6\\
5/3\\
13/6
\end{bmatrix},
\end{equation*}
so the residual is
\begin{equation*}
\mathbf{r}
=
\mathbf{b}-A\hat{\mathbf{x}}
=
\begin{bmatrix}
-1/6\\
1/3\\
-1/6
\end{bmatrix}.
\end{equation*}
Finally,
\begin{equation*}
A^T\mathbf{r}
=
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
This is the residual-orthogonality condition from Unit 4 and the gradient-zero condition from Unit 5.
Subsection Residual orthogonality in code
Learning outcomes. U5-LO6, U4-LO4.
The vector
xhat is a least-squares coefficient vector. The vector r is the residual
\begin{equation*}
\mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}.
\end{equation*}
The expression
A.T @ r checks whether the residual is orthogonal to the columns of \(A\text{.}\) It can be close to zero even when r is not close to zero. Unit 4 used this residual-orthogonality condition to derive the normal equations. Unit 5 obtains the same equations by setting the gradient of \(\|A\mathbf{x}-\mathbf{b}\|^2\) equal to zero.
