Unit 5 Optimization and training
In Unit 3, the gradient appeared in the linear approximation \(f(\mathbf{a}+\mathbf{h})\approx f(\mathbf{a})+\nabla f(\mathbf{a})\cdot \mathbf{h}\text{.}\) Now we use that linear approximation to choose directions that decrease a function.
Big question. How do we choose parameters to minimize a loss?
Learning outcomes. By the end of this unit, students should be able to:
Core outcomes.
-
U5-LO1. Find critical points and classify local behavior using gradients, contour plots, the two-variable second derivative test, and Hessian eigenvalues.
-
U5-LO2. Compute directional derivatives and use the gradient to identify steepest-increase and descent directions.
-
U5-LO3. Take gradient-descent steps and diagnose the effect of learning rate from iterates, code, or loss tables.
-
U5-LO4. Compute eigenvalues, eigenvectors, eigenspaces, and symmetric diagonalizations, and explain why they matter for Hessians and quadratic forms.
-
U5-LO5. Use quadratic approximations and Hessian definiteness to interpret local loss landscapes.
-
U5-LO6. Derive the normal equations for least squares using gradients and connect the projection, calculus, and Hessian viewpoints.
-
U5-LO7. Model fixed-hidden-layer training as a least-squares problem using a design matrix, and interpret the fitted parameters.
Toolbox skills. Compute gradients, directional derivatives, Hessians, eigenvalues, eigenvectors, eigenspaces, symmetric diagonalizations, quadratic forms, descent steps, second-derivative classifications, and least-squares normal equations from gradients.
Application and computation outcomes. Interpret Lab U5
[U5-LO2, U5-LO3, U5-LO4, U5-LO5, U5-LO6, U5-LO7 | C+M+T | Core] by reading computations involving gradient descent updates, learning-rate comparisons, Hessian eigenvalues, least squares as a zero-gradient problem, residual orthogonality, fixed-hidden-layer design matrices, and rank-one outer-product updates.
