Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 5.7 Coding recap

Subsection Linked notebook

Run Lab U5: Hessian eigenvalues, least squares, and fixed nonlinear features
 1 
sebroc.github.io/MATH345-Course-Materials/labs/#lab-u5
. The opening block reviews gradient descent and a final-layer update from Unit 3. The Unit 5 material interprets Hessian eigenvalues, checks least squares through residual and gradient conditions, distinguishes global minimality from coefficient uniqueness, and fits fixed nonlinear features.
🔗
For a quick reference on entrywise functions, design matrices, eigenvalue and least-squares commands, transposes, and numerical checks such as “close to zero”, see the programming appendix sections B.4, B.8, B.10, and B.9.
🔗
🔗

Subsection Review activities

Activity 5.7.1. Reading Hessian eigenvalues (U5-LO4, U5-LO5).

Suppose a critical point has Hessian matrix
🔗
H = np.array([[4.0, 0.0],
              [0.0, -1.0]])

evals, evecs = np.linalg.eig(H)
evals
Output:
🔗
array([ 4., -1.])
  1. What matrix is being studied?
    🔗
    🔗
  2. What do the signs of the eigenvalues say about the quadratic form \(\mathbf{h}^T H\mathbf{h}\text{?}\)
    🔗
    🔗
  3. If \(H\) is the Hessian at a critical point, what does the second derivative test conclude?
    🔗
    🔗
  4. Why is the phrase “at a critical point” important?
    🔗
    🔗
Solution.
The matrix being studied is the Hessian matrix. Its eigenvalues have opposite signs, so the quadratic form \(\mathbf{h}^T H\mathbf{h}\) takes both positive and negative values. If \(H\) is the Hessian at a critical point, the second derivative test classifies the critical point as a saddle point. The phrase “at a critical point” is important because the Hessian test classifies local behavior after the first-order term has disappeared.
🔗
🔗
🔗

Activity 5.7.2. Reading a least-squares residual check (U5-LO6, U4-LO4).

Consider the code:
🔗
xhat = np.linalg.lstsq(A, b, rcond=None)[0]
r = b - A @ xhat
A.T @ r
  1. What is xhat?
    🔗
    🔗
  2. What is r?
    🔗
    🔗
  3. What condition is checked by A.T @ r?
    🔗
    🔗
  4. Does A.T @ r being “close to zero” mean that r is “close to zero”?
    🔗
    🔗
  5. Which earlier unit used this same condition?
    🔗
    🔗
Solution.
The vector xhat is a least-squares coefficient vector. The vector r is the residual
\begin{equation*} \mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}. \end{equation*}
The expression A.T @ r checks whether the residual is orthogonal to the columns of \(A\text{.}\) It can be close to zero even when r is not close to zero. Unit 4 used this same residual-orthogonality condition to derive the normal equations. Unit 5 obtains the same equations by setting the gradient of \(\|A\mathbf{x}-\mathbf{b}\|^2\) equal to zero.
🔗
🔗
🔗

Activity 5.7.3. Reading spectral tools (U5-LO1, U5-LO3, U5-LO4).

Consider the code:
🔗
A = np.array([[2.0, 1.0],
              [1.0, 2.0]])

evals, Q = np.linalg.eigh(A)
D = Q.T @ A @ Q
x = np.array([1.0, -1.0])
q = x @ (A @ x)
One possible output is
🔗
\begin{align*} \texttt{evals}\amp=\begin{bmatrix}1\\3\end{bmatrix},\\ Q\amp=\frac{1}{\sqrt2}\begin{bmatrix}-1\amp1\\1\amp1\end{bmatrix},\\ D=Q^TAQ\amp=\begin{bmatrix}1\amp0\\0\amp3\end{bmatrix},\\ q=\mathbf x^TA\mathbf x\amp=2. \end{align*}
  1. What do the entries of evals and the columns of Q represent?
    🔗
    🔗
  2. Why does Q.T @ A @ Q give a diagonal matrix?
    🔗
    🔗
  3. What does x @ (A @ x) compute?
    🔗
    🔗
  4. Use the eigenvalues to classify the quadratic form \(q_A(\mathbf x)=\mathbf x^TA\mathbf x\text{.}\)
    🔗
    🔗
Solution.
The entries of evals are the eigenvalues \(1\) and \(3\text{.}\) The corresponding columns of Q are orthonormal eigenvectors. NumPy may return either sign for either eigenvector, so the signs of the displayed columns may change without changing the mathematics.
🔗
Because \(A\) is symmetric, its orthonormal eigenvectors form the columns of an orthogonal matrix \(Q\text{,}\) and \(Q^TAQ\) is the diagonal matrix of eigenvalues. The expression x @ (A @ x) computes the quadratic form \(q_A(\mathbf x)=\mathbf x^TA\mathbf x\text{,}\) which equals \(2\) for the displayed vector. Both eigenvalues are positive, so \(q_A\) is positive definite.
🔗
🔗
🔗
Reopen The same fixed-feature fit in code for the fixed-feature design-matrix calculation.
🔗
🔗
🔗
PrevTopNext