Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 4.6 Exercises

Exercise tags. The tags below identify the primary outcomes and competency types for each exercise group. Individual problems may also involve earlier outcomes.
🔗

Subsection Orthogonality and orthogonal bases

Primary outcomes. U4-LO1, U4-LO2. Competencies. P+C.
🔗
  • Nicholson 5.3.3(b)
     1 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/05%3A_Vector_Space_R/5.03%3A_Orthogonality/5.3E%3A_Orthogonality_Exercises
    🔗
    🔗
  • Nicholson 5.3.6(b)
     2 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/05%3A_Vector_Space_R/5.03%3A_Orthogonality/5.3E%3A_Orthogonality_Exercises
    🔗
    🔗
  • Nicholson 5.3.9
     3 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/05%3A_Vector_Space_R/5.03%3A_Orthogonality/5.3E%3A_Orthogonality_Exercises
    🔗
    🔗
🔗

Subsection Orthogonal projection and Gram-Schmidt

Primary outcomes. U4-LO2, U4-LO3. Competencies. P+C.
🔗
  • Nicholson 8.1.1(d)
     4 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/08%3A_Orthogonality/8.01%3A_Orthogonal_Complements_and_Projections/8.1E%3A_Orthogonal_Complements_and_Projections_Exercises
    🔗
    🔗
  • Nicholson 8.1.2(b,d)
     5 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/08%3A_Orthogonality/8.01%3A_Orthogonal_Complements_and_Projections/8.1E%3A_Orthogonal_Complements_and_Projections_Exercises
    🔗
    🔗
  • Nicholson 8.1.4(b)
     6 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/08%3A_Orthogonality/8.01%3A_Orthogonal_Complements_and_Projections/8.1E%3A_Orthogonal_Complements_and_Projections_Exercises
    🔗
    🔗
🔗

Subsection QR factorization and least squares

Primary outcomes. U4-LO6. Competencies. P+R.
🔗
  • Nicholson 8.4.1(b,d)
     7 
    math.libretexts.org/Bookshelves/Linear_Algebra/Linear_Algebra_with_Applications_(Nicholson)/08%3A_Orthogonality/8.04%3A_QR-Factorization/8.4E%3A_QR-Factorization_Exercises
    🔗
    🔗
🔗

Subsection Additional applied and computational problems

These exercises connect the main Unit 4 ideas: orthogonality, orthogonal bases, projections, residuals, least squares, normal equations, regression, QR factorization, and code interpretation. Solutions are collected in Appendix C.4.
🔗

Checkpoint 4.6.1. Orthogonal-basis coefficients.

Let
\begin{equation*} \mathbf{f}_1 = \begin{bmatrix}1\\1\\0\end{bmatrix},\quad \mathbf{f}_2 = \begin{bmatrix}1\\-1\\0\end{bmatrix},\quad \mathbf{f}_3 = \begin{bmatrix}0\\0\\2\end{bmatrix}, \end{equation*}
and let
\begin{equation*} \mathbf{x} = \begin{bmatrix}3\\1\\4\end{bmatrix}. \end{equation*}
🔗
  1. Verify that \(\mathbf{f}_1,\mathbf{f}_2,\mathbf{f}_3\) are pairwise orthogonal.
    🔗
    🔗
  2. Use dot products to write \(\mathbf{x}\) as a linear combination of \(\mathbf{f}_1,\mathbf{f}_2,\mathbf{f}_3\text{.}\)
    🔗
    🔗
  3. Explain why each dot product isolates one coefficient.
    🔗
    🔗
Tags. [U4-LO1, U4-LO2 | P+C+R | Core]
🔗
🔗

Checkpoint 4.6.2. Projection onto a line.

Let
\begin{equation*} \mathbf{u} = \begin{bmatrix}1\\2\end{bmatrix},\quad \mathbf{x} = \begin{bmatrix}3\\1\end{bmatrix}, \end{equation*}
and let \(L = \operatorname{span}\{\mathbf{u}\}\text{.}\)
🔗
  1. Compute \(\operatorname{proj}_L(\mathbf{x})\text{.}\)
    🔗
    🔗
  2. Compute \(\mathbf{r} = \mathbf{x} - \operatorname{proj}_L(\mathbf{x})\text{.}\)
    🔗
    🔗
  3. Check that \(\mathbf{r} \cdot \mathbf{u} = 0\text{.}\)
    🔗
    🔗
  4. Explain what the residual measures.
    🔗
    🔗
Tags. [U4-LO3 | P+C | Core]
🔗
🔗

Checkpoint 4.6.3. Column-space orthogonal complement.

Let
\begin{equation*} A = \begin{bmatrix} 1\amp 0\\ 0\amp 1\\ 1\amp 1 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(A^T\mathbf{r}\) for \(\mathbf{r} = \begin{bmatrix}-1\\-1\\1\end{bmatrix}\text{.}\)
    🔗
    🔗
  2. Explain why \(\mathbf{r}\) is orthogonal to \(\operatorname{col}(A)\text{.}\)
    🔗
    🔗
  3. Describe \(\operatorname{col}(A)^\perp\text{.}\)
    🔗
    🔗
  4. Explain the identity \(\operatorname{col}(A)^\perp = \operatorname{null}(A^T)\) in this example.
    🔗
    🔗
Tags. [U4-LO3, U4-LO4, U2-LO3 | C+R | Core]
🔗
🔗

Checkpoint 4.6.4. Projection onto a plane.

Let \(U\) be the plane \(x + y + z = 0\text{,}\) and let
\begin{equation*} \mathbf{p} = \begin{bmatrix}1\\2\\4\end{bmatrix}. \end{equation*}
🔗
  1. Give a normal vector \(\mathbf{n}\) for \(U\text{.}\)
    🔗
    🔗
  2. Compute \(\operatorname{proj}_U(\mathbf{p})\) by subtracting the component of \(\mathbf{p}\) in the normal direction.
    🔗
    🔗
  3. Check that the projected point lies in \(U\text{.}\)
    🔗
    🔗
Tags. [U4-LO3 | P+C+R | Core]
🔗
🔗

Checkpoint 4.6.5. Unreachable target, closest output.

Let
\begin{equation*} A = \begin{bmatrix} 1\amp 0\\ 0\amp 1\\ 1\amp 1 \end{bmatrix}, \qquad \mathbf{b} = \begin{bmatrix}2\\1\\5\end{bmatrix}. \end{equation*}
🔗
  1. Explain why \(A\mathbf{x} = \mathbf{b}\) has no exact solution.
    🔗
    🔗
  2. Form \(A^T A\) and \(A^T\mathbf{b}\text{.}\)
    🔗
    🔗
  3. Solve the normal equations.
    🔗
    🔗
  4. Compute \(A\hat{\mathbf{x}}\) and \(\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}}\text{.}\)
    🔗
    🔗
  5. Check \(A^T\mathbf{r} = \mathbf{0}\text{.}\)
    🔗
    🔗
Tags. [U4-LO4, U4-LO5, U2-LO3 | P+C+R | Core]
🔗
🔗

Checkpoint 4.6.6. Regression design matrix and residual.

Fit a line \(y = c_0 + c_1 t\) to the data
\begin{equation*} (0,1),\quad (1,2),\quad (2,2). \end{equation*}
🔗
  1. Build the design matrix \(X\text{.}\)
    🔗
    🔗
  2. Form \(X^T X\) and \(X^T\mathbf{y}\text{.}\)
    🔗
    🔗
  3. Solve the normal equations.
    🔗
    🔗
  4. Compute the fitted values and residual.
    🔗
    🔗
  5. Check \(X^T\mathbf{r} = \mathbf{0}\text{.}\)
    🔗
    🔗
Tags. [U4-LO5 | P+C+M | Core]
🔗
🔗

Checkpoint 4.6.7. Code interpretation: residual orthogonality.

Consider the code:
🔗
xhat = np.linalg.lstsq(A, b, rcond=None)[0]
r = b - A @ xhat
A.T @ r
Suppose the output of the last line is approximately array([2.2e-16, -1.1e-16]).
🔗
  1. What does xhat represent?
    🔗
    🔗
  2. What does \(\mathbf{r}\) represent?
    🔗
    🔗
  3. What geometric condition does A.T @ r check?
    🔗
    🔗
  4. Does this output mean \(\mathbf{r}\) is the zero vector?
    🔗
    🔗
Tags. [U4-LO4, U4-LO7 | C+T | Core]
🔗
🔗

Checkpoint 4.6.9. Rank and nonunique coefficients.

Let
\begin{equation*} A = \begin{bmatrix} 1\amp 0\amp 1\\ 0\amp 1\amp 1\\ 1\amp 1\amp 2 \end{bmatrix}, \end{equation*}
and let \(\mathbf{z} = \begin{bmatrix}1\\1\\-1\end{bmatrix}\text{.}\)
🔗
  1. Compute \(A\mathbf{z}\text{.}\)
    🔗
    🔗
  2. Suppose \(\hat{\mathbf{x}}\) is a least-squares solution. Compare \(A\hat{\mathbf{x}}\) and \(A(\hat{\mathbf{x}} + 5\mathbf{z})\text{.}\)
    🔗
    🔗
  3. Which can be nonunique: the fitted vector or the coefficient vector?
    🔗
    🔗
  4. How does this connect to dependent columns?
    🔗
    🔗
Tags. [U4-LO4, U2-LO6 | C+R | Core]
🔗
🔗

Checkpoint 4.6.10. QR code reading.

Consider the code:
🔗
Q, R = np.linalg.qr(A)
x_qr = np.linalg.solve(R, Q.T @ b)
x_lstsq = np.linalg.lstsq(A, b, rcond=None)[0]

Q.T @ Q, x_qr, x_lstsq
  1. What should Q.T @ Q be close to?
    🔗
    🔗
  2. Why does the code solve a system with R?
    🔗
    🔗
  3. Why should x_qr and x_lstsq agree when \(A\) has independent columns?
    🔗
    🔗
  4. Does this prove \(A\mathbf{x} = \mathbf{b}\) has an exact solution?
    🔗
    🔗
Tags. [U4-LO6, U4-LO7 | C+T | Core]
🔗
🔗

Checkpoint 4.6.11. Attention versus projection.

Classify each phrase as attention-like, projection-like, or both.
🔗
  1. Uses dot products.
    🔗
    🔗
  2. Forms a weighted average of value vectors.
    🔗
    🔗
  3. Finds the closest vector in a subspace.
    🔗
    🔗
  4. Produces a residual orthogonal to a subspace.
    🔗
    🔗
  5. Normalizes scores into weights.
    🔗
    🔗
  6. Can be checked by \(A^T\mathbf{r} = \mathbf{0}\text{.}\)
    🔗
    🔗
Tags. [U4-LO1, U4-LO3, U4-LO4 | C+M | Core]
🔗
🔗
🔗
🔗
PrevTopNext