Section C.4 Unit 4 applied, geometric, and computational interpretation
These solution sketches correspond to the additional applied and computational interpretation exercises at the end of Unit 4.
Subsection Orthogonal-basis coefficients
The vectors are pairwise orthogonal. The coefficients are
\begin{align*}
\frac{\mathbf{x} \cdot \mathbf{f}_1}{\mathbf{f}_1 \cdot \mathbf{f}_1} \amp= 2,\\
\frac{\mathbf{x} \cdot \mathbf{f}_2}{\mathbf{f}_2 \cdot \mathbf{f}_2} \amp= 1,\\
\frac{\mathbf{x} \cdot \mathbf{f}_3}{\mathbf{f}_3 \cdot \mathbf{f}_3} \amp= 2.
\end{align*}
So \(\mathbf{x} = 2\mathbf{f}_1 + \mathbf{f}_2 + 2\mathbf{f}_3\text{.}\) Orthogonality makes the cross terms vanish.
Subsection Projection onto a line
\(\mathbf{x} \cdot \mathbf{u} = 5\) and \(\mathbf{u} \cdot \mathbf{u} = 5\text{,}\) so \(\operatorname{proj}_L(\mathbf{x}) = \mathbf{u} = \begin{bmatrix}1\\2\end{bmatrix}\text{.}\) The residual is \(\begin{bmatrix}2\\-1\end{bmatrix}\text{.}\) Its dot product with \(\mathbf{u}\) is \(2 - 2 = 0\text{.}\) The residual is the part of \(\mathbf{x}\) perpendicular to the line.
Subsection Column-space orthogonal complement
\(A^T\mathbf{r} = \begin{bmatrix}0\\0\end{bmatrix}\text{.}\) Therefore \(\mathbf{r}\) is orthogonal to each column of \(A\text{.}\) The orthogonal complement is \(\operatorname{span}\{\begin{bmatrix}-1\\-1\\1\end{bmatrix}\}\text{.}\) This is exactly the null space of \(A^T\text{.}\)
Subsection Projection onto a plane
A normal vector is \(\mathbf{n} = \begin{bmatrix}1\\1\\1\end{bmatrix}\text{.}\) Since \(\mathbf{p} \cdot \mathbf{n} = 7\) and \(\mathbf{n} \cdot \mathbf{n} = 3\text{,}\)
\begin{equation*}
\operatorname{proj}_U(\mathbf{p}) = \mathbf{p} - (7/3)\mathbf{n}
=
\begin{bmatrix}-4/3\\-1/3\\5/3\end{bmatrix}.
\end{equation*}
The entries add to 0, so the projected point lies in \(U\text{.}\)
Subsection Unreachable target, closest output
The output of \(A\) has the form \((x_1, x_2, x_1+x_2)\text{.}\) The first two target entries force \(x_1 = 2\) and \(x_2 = 1\text{,}\) which would make the third entry \(3\text{,}\) not \(5\text{.}\)
\(A^T A = \begin{bmatrix}2\amp 1\\1\amp 2\end{bmatrix}\text{,}\) \(A^T\mathbf{b} = \begin{bmatrix}7\\6\end{bmatrix}\text{.}\)
Solving gives \(\hat{\mathbf{x}} = \begin{bmatrix}8/3\\5/3\end{bmatrix}\text{.}\) Then
\begin{equation*}
A\hat{\mathbf{x}} =
\begin{bmatrix}8/3\\5/3\\13/3\end{bmatrix},
\end{equation*}
\begin{equation*}
\mathbf{r} =
\begin{bmatrix}-2/3\\-2/3\\2/3\end{bmatrix},
\end{equation*}
and \(A^T\mathbf{r} = \begin{bmatrix}0\\0\end{bmatrix}\text{.}\)
Subsection Regression design matrix and residual
\(X = \begin{bmatrix}1\amp 0\\1\amp 1\\1\amp 2\end{bmatrix}\text{,}\) \(\mathbf{y} = \begin{bmatrix}1\\2\\2\end{bmatrix}\text{.}\)
\(X^T X = \begin{bmatrix}3\amp 3\\3\amp 5\end{bmatrix}\text{,}\) \(X^T\mathbf{y} = \begin{bmatrix}5\\6\end{bmatrix}\text{.}\)
Solving gives \(\boldsymbol{\beta} = \begin{bmatrix}7/6\\1/2\end{bmatrix}\text{.}\) The fitted values are \(\begin{bmatrix}7/6\\5/3\\13/6\end{bmatrix}\text{.}\) The residual is \(\begin{bmatrix}-1/6\\1/3\\-1/6\end{bmatrix}\text{,}\) and \(X^T\mathbf{r} = \begin{bmatrix}0\\0\end{bmatrix}\text{.}\)
Subsection Code interpretation: residual orthogonality
xhat is a least-squares coefficient vector. The residual is \(\mathbf{b} - A\hat{\mathbf{x}}\text{.}\) The expression A.T @ r checks whether \(\mathbf{r}\) is orthogonal to every column of \(A\text{.}\) The near-zero output does not mean \(\mathbf{r} = \mathbf{0}\text{;}\) it means the residual is orthogonal to the column space, up to roundoff.
Subsection Debug: wrong residual check
If \(A\) is \(m\) by \(n\text{,}\) then
A @ r is usually not defined because \(\mathbf{r}\) has length \(m\text{,}\) not \(n\text{.}\) The residual must be checked against the columns of \(A\text{.}\) Those dot products are encoded by A.T @ r. The correct mathematical check is \(A^T\mathbf{r}\approx\mathbf{0}\text{;}\) in code, use A.T @ r.
Subsection Rank and nonunique coefficients
\(A\mathbf{z} = \mathbf{0}\text{,}\) so \(\mathbf{z}\) is a null-space direction. Therefore
\begin{equation*}
A(\hat{\mathbf{x}} + 5\mathbf{z}) = A\hat{\mathbf{x}} + 5A\mathbf{z} = A\hat{\mathbf{x}}.
\end{equation*}
The coefficient vector can change while the fitted vector stays fixed. This happens because the third column is the sum of the first two columns.
Subsection QR code reading
Q.T @ Q should be close to the identity matrix because \(Q\) has orthonormal columns. Since \(A = QR\text{,}\) least squares can be reduced to the triangular system \(R\hat{\mathbf{x}} = Q^T\mathbf{b}\text{.}\) The two coefficient vectors should agree because they compute the same least-squares solution. This does not prove \(A\mathbf{x} = \mathbf{b}\) has an exact solution.
Subsection Attention versus projection
Both use dot products. Attention-like: weighted average of value vectors; normalizes scores into weights. Projection-like: closest vector in a subspace; residual orthogonal to a subspace; \(A^T\mathbf{r} = \mathbf{0}\) check.
