Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 5.8 Exercises

Learning outcomes. The labels below identify the learning outcomes for each exercise group. Individual problems may also involve earlier outcomes.
🔗

Subsection Symmetric matrices and orthogonal diagonalization

Learning outcomes. U5-LO3.
🔗
  • Nicholson 8.2.5(b,d,f)
     1 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/08%3A_Orthogonality/8.02%3A_Orthogonal_Diagonalization/8.2E%3A_Orthogonal_Diagonalization_Exercises
    🔗
    🔗
🔗

Subsection Hessians and critical-point classification

Learning outcomes. U3-LO9, U5-LO5.
🔗
🔗

Subsection Additional exercises

These exercises connect eigenvectors, quadratic forms, Hessian eigenvalues, least squares, residual checks, fixed nonlinear features, and code interpretation. Solutions are collected in Appendix C.5.
🔗

Additional exercise 5.8.1. Eigenvectors as special directions.

Let
\begin{equation*} A= \begin{bmatrix} 3\amp1\\ 0\amp2 \end{bmatrix}, \qquad \mathbf{v}_1= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf{v}_2= \begin{bmatrix} -1\\ 1 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(A\mathbf{v}_1\text{.}\)
    🔗
    🔗
  2. Compute \(A\mathbf{v}_2\text{.}\)
    🔗
    🔗
  3. Identify the eigenvalue for each eigenvector.
    🔗
    🔗
  4. Explain what the matrix map does to these two special directions.
    🔗
    🔗
Learning outcomes. U5-LO1.
🔗
🔗

Additional exercise 5.8.2. Diagonalization and matrix powers.

Let
\begin{equation*} A= \begin{bmatrix} 2\amp1\\ 0\amp3 \end{bmatrix}, \qquad P= \begin{bmatrix} 1\amp1\\ 0\amp1 \end{bmatrix}, \qquad D= \begin{bmatrix} 2\amp0\\ 0\amp3 \end{bmatrix}. \end{equation*}
🔗
  1. Verify that the columns of \(P\) are eigenvectors of \(A\text{,}\) and identify their eigenvalues.
    🔗
    🔗
  2. Compute \(P^{-1}\) and verify that \(A=PDP^{-1}\text{.}\)
    🔗
    🔗
  3. Use \(A^k=PD^kP^{-1}\) to derive a formula for \(A^k\text{.}\)
    🔗
    🔗
  4. Compute \(A^4\text{.}\)
    🔗
    🔗
  5. Explain why diagonalization makes repeated matrix actions easier to compute.
    🔗
    🔗
Learning outcomes. U5-LO1, U5-LO2.
🔗
🔗

Additional exercise 5.8.3. Quadratic form and definiteness.

Let
\begin{equation*} H= \begin{bmatrix} 4\amp0\\ 0\amp1 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf{h}^T H\mathbf{h}\) for
    \begin{equation*} \mathbf{h}= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf{h}= \begin{bmatrix} 0\\ 1 \end{bmatrix}, \qquad \mathbf{h}= \begin{bmatrix} 1\\ 1 \end{bmatrix}. \end{equation*}
    🔗
    🔗
  2. Write \(\mathbf{h}^T H\mathbf{h}\) for a general vector
    \begin{equation*} \mathbf{h}= \begin{bmatrix} h_1\\ h_2 \end{bmatrix}. \end{equation*}
    🔗
    🔗
  3. Is \(H\) positive definite, negative definite, or indefinite?
    🔗
    🔗
  4. If \(H\) is the Hessian at a critical point, what does the second derivative test conclude?
    🔗
    🔗
Learning outcomes. U5-LO4, U5-LO5.
🔗
🔗

Additional exercise 5.8.4. Hessian eigenvalue classification.

Suppose \(\mathbf{a}\) is a critical point of a scalar-valued function \(f\text{.}\) For each possible list of Hessian eigenvalues, classify the critical point as a local minimum, local maximum, saddle point, or inconclusive.
🔗
  1. \(\displaystyle 3,5\)
    🔗
    🔗
  2. \(\displaystyle -2,-7\)
    🔗
    🔗
  3. \(\displaystyle 4,-1\)
    🔗
    🔗
  4. \(\displaystyle 0,2\)
    🔗
    🔗
  5. Explain why the zero eigenvalue case is different.
    🔗
    🔗
Learning outcomes. U5-LO5.
🔗
🔗

Additional exercise 5.8.5. Least squares from gradients.

Let
\begin{equation*} A= \begin{bmatrix} 1\amp0\\ 1\amp1\\ 1\amp2 \end{bmatrix}, \qquad \mathbf{b}= \begin{bmatrix} 1\\ 2\\ 2 \end{bmatrix}. \end{equation*}
Define
\begin{equation*} h(\mathbf{x})=\|A\mathbf{x}-\mathbf{b}\|^2. \end{equation*}
🔗
  1. Compute \(A^T A\) and \(A^T\mathbf{b}\text{.}\)
    🔗
    🔗
  2. Use \(\nabla h(\mathbf{x})=2A^T(A\mathbf{x}-\mathbf{b})\) to write the normal equations.
    🔗
    🔗
  3. Solve the normal equations.
    🔗
    🔗
  4. Compute the residual \(\mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}\text{.}\)
    🔗
    🔗
  5. Check that \(A^T\mathbf{r}=\mathbf{0}\text{.}\)
    🔗
    🔗
Learning outcomes. U5-LO6, U4-LO4.
🔗
🔗

Additional exercise 5.8.6. Residual orthogonality in code.

Consider the code:
🔗
xhat = np.linalg.lstsq(A, b, rcond=None)[0]
r = b - A @ xhat
A.T @ r
  1. What is xhat?
    🔗
    🔗
  2. What is r?
    🔗
    🔗
  3. What condition is checked by A.T @ r?
    🔗
    🔗
  4. Does A.T @ r being close to zero mean that r is close to zero?
    🔗
    🔗
  5. Which earlier unit used this same condition?
    🔗
    🔗
Learning outcomes. U5-LO6, U4-LO4.
🔗
🔗

Additional exercise 5.8.7. Fixed-feature design matrix.

Let \(\sigma(t)=\tanh(t)\text{,}\) and define
\begin{equation*} N_{\mathbf{c}}(t)=c_0+c_1\sigma(t)+c_2\sigma(t-1)+c_3\sigma(t+1). \end{equation*}
Use the input values \(t=-1,0,1\text{.}\)
🔗
  1. Write the design matrix \(A\) for the model \(A\mathbf{c}\approx \mathbf{y}\text{.}\)
    🔗
    🔗
  2. Which vector is trained?
    🔗
    🔗
  3. Why is this a least-squares problem?
    🔗
    🔗
  4. Is the model linear as a function of \(t\text{?}\)
    🔗
    🔗
  5. Is the model linear as a function of \(\mathbf{c}\text{?}\)
    🔗
    🔗
Learning outcomes. U5-LO6, U5-LO7.
🔗
🔗
🔗
🔗
PrevTopNext