Note 4.7.1. Unit highlights.
Big question. When \(A\mathbf{x}=\mathbf{b}\) has no solution, what is the best approximate solution?
Notation. \(U^\perp\text{,}\) \(\operatorname{proj}_U(\mathbf{b})\text{,}\) \(\|A\mathbf{x}-\mathbf{b}\|^2\text{,}\) \(A^TA\hat{\mathbf{x}}=A^T\mathbf{b}\text{.}\)
Learning outcomes. By the end of this unit, students should be able to use U4-LO1 through U4-LO7 to compute projections, formulate least squares as projection, identify residual orthogonality, derive and solve normal equations, solve small regression problems, and explain why QR helps least squares.
Concepts. Orthogonality, projection, residual, least squares, regression, QR factorization.
Results. Best approximation is projection; residual is orthogonal to the column space; normal equations characterize least-squares solutions.
Toolbox skills. U4-LO1 through U4-LO7 use dot products, orthogonality tests, normalization, Gram-Schmidt, projections, residuals, normal equations, design matrices, line fitting, \(A^T\mathbf{r}\approx \mathbf{0}\) checks,
Q.T @ Q checks, and QR factorization.
Main application. Regression as best approximation in a column space.
Connection. Attention and projection both use dot products, but attention forms weighted averages while projection finds closest points in subspaces.
Connections. Unit 2 rank controls uniqueness of coefficients. Unit 5 derives the same normal equations by gradients. Unit 7 reuses projection with polynomial inner products.
Study anchors. U4-LO1: Orthogonality and orthogonal bases. U4-LO2 and U4-LO3: Orthogonal projection. U4-LO4 and U4-LO5: Least squares as projection. U4-LO6: \(QR\) factorization and least squares. U4-LO7: Code snapshot: fitting and checking the residual, Applications and computation recap, and Linked notebook.
Exam-readiness checklist. I can use U4-LO3 to compute projection residuals; use U4-LO4 to identify least-squares residuals and explain why \(A^T\mathbf{r}\approx\mathbf{0}\) is the residual-orthogonality check; use U4-LO5 to build design matrices and solve normal equations; and use U4-LO6 and U4-LO7 to read QR checks,
Q.T @ Q, and least-squares code outputs.
Common mistakes. Thinking least squares solves \(A\mathbf{x}=\mathbf{b}\) exactly; forgetting the residual is orthogonal to the column space, not usually zero; using QR language without orthonormal columns.
