Unit 5 Eigenvalues and second-order optimization
Eigenvectors identify directions that a square matrix maps back to the same line. Diagonalization collects enough of these directions into a coordinate system in which the matrix acts by separate scalings. Symmetric matrices admit orthonormal eigenvector coordinates, which also simplify quadratic forms. We then apply these tools to Hessians and revisit least squares through projection, gradients, and curvature.
Big questions. Which directions make a matrix simple? How do those directions describe quadratic forms and the local shape of a function? How do projection, zero gradient, and curvature describe the same least-squares problem?
Learning outcomes. By the end of this unit, students should be able to:
-
U5-LO1. Compute eigenvalues, eigenvectors, and eigenspaces, and interpret eigenvectors as directions whose lines are preserved by a square matrix.
-
U5-LO2. Determine whether a matrix is diagonalizable, construct a diagonalization\begin{equation*} A=PDP^{-1} \end{equation*}when possible, and use it to simplify matrix actions and powers.
-
U5-LO3. Orthogonally diagonalize symmetric matrices and interpret the spectral theorem as an orthonormal change of coordinates.
-
U5-LO4. Represent and evaluate quadratic forms, explain their scaling behavior, analyze them in principal-axis coordinates, and classify symmetric matrices as positive definite, negative definite, or indefinite.
-
U5-LO5. Construct and use Taylor polynomials and quadratic approximations, and classify critical points using Hessian eigenvalues, contour plots, and the two-variable discriminant shortcut.
-
U5-LO6. Derive and interpret least squares through residual orthogonality, zero gradient, Hessian curvature, global minimality, and uniqueness.
-
U5-LO7. Construct a design matrix from fixed nonlinear features, write the squared-error loss\begin{equation*} L(\mathbf c)=\|A\mathbf c-\mathbf y\|^2, \end{equation*}and explain why fitting only \(\mathbf c\) is a least-squares problem.
Toolbox skills. Compute eigenvalues, eigenvectors, eigenspaces, diagonalizations, matrix powers, orthogonal diagonalizations, quadratic forms, definiteness, Hessian eigenvalue classifications, normal equations, residual checks, and fixed-feature design matrices.
