Section 5.10 Unit 5 highlights
Big question. How do we choose parameters to minimize a loss?
Notation. \(D_{\mathbf{v}}f(\mathbf{a})\text{,}\) \(\nabla f(\mathbf{x})\text{,}\) \(\mathbf{x}_{k+1}=\mathbf{x}_k-\alpha\nabla f(\mathbf{x}_k)\text{,}\) \(L(\theta)\text{,}\) \(\theta_{k+1}=\theta_k-\alpha\nabla L(\theta_k)\text{,}\) \(H_f(\mathbf{a})\text{,}\) \(\mathbf{h}^T H_f(\mathbf{a})\mathbf{h}\text{,}\) \(Q^THQ=D\text{,}\) \(\nabla\|A\mathbf{x}-\mathbf{b}\|^2=2A^T(A\mathbf{x}-\mathbf{b})\text{,}\) \(A^T\mathbf{r}=\mathbf{0}\text{,}\) \(A\mathbf{c}\approx\mathbf{y}\text{.}\)
Learning outcomes. By the end of this unit, students should be able to use U5-LO1 through U5-LO7 to identify critical points, compute directional derivatives, take and interpret gradient descent steps, compute eigenvalues and eigenvectors, use Hessian eigenvalues to classify local behavior, derive least-squares normal equations from gradients, and explain why fixed-hidden-layer training can reduce to least squares.
Concepts. Optimization problem, feasible region, local extremum, global extremum, critical point, directional derivative, gradient, steepest descent, gradient descent, learning rate, logits, loss, eigenvalue, eigenvector, eigenspace, diagonalization, symmetric matrix, quadratic form, positive definite matrix, negative definite matrix, indefinite matrix, Hessian, saddle point, least squares, residual, design matrix, fixed feature, trained coefficient.
Results. Directional derivatives are gradient dot products. The negative gradient is the direction of steepest local decrease. Hessian eigenvalue signs classify many critical points. Symmetric matrices have orthogonal eigenvectors and can be diagonalized by an orthogonal change of coordinates. Quadratic forms describe the second-order part of a local approximation. Least-squares normal equations can be derived by setting
\begin{equation*}
\nabla\|A\mathbf{x}-\mathbf{b}\|^2=\mathbf{0}.
\end{equation*}
If fixed nonlinear features are collected into a design matrix \(A\text{,}\) then training only the final coefficient vector \(\mathbf{c}\) is a least-squares problem.
Toolbox skills. U5-LO1 through U5-LO7 use gradients, directional derivatives, dot products, descent directions, learning-rate checks, eigenvalues, eigenvectors, eigenspaces, symmetric diagonalization, quadratic forms, Hessian matrices, second-derivative tests, normal equations, residual orthogonality, design matrices, least-squares solves, and short code interpretation.
Main applications. Loss minimization, learning-rate diagnosis, simplified next-token loss, least squares as gradient zero, residual-orthogonality checks, fixed-hidden-layer training, and reading short optimization code.
Connections. Unit 1 supplies vectors, dot products, norms, cosine-style alignment, and matrix-vector products. Unit 2 supplies rank, null-space, and non-uniqueness language for coefficient vectors. Unit 3 supplies gradients, Hessians, Jacobians, local linearization, and the chain rule. Unit 4 supplies projection, residual orthogonality, design matrices, and normal equations. Unit 6 adds constraints and special directions through Lagrange multipliers and SVD. Unit 7 returns to quadratic forms, Hessians, projection, and least squares in polynomial spaces.
Study anchors. U5-LO1: Optimization problems and critical points. U5-LO2 and U5-LO3: Directional derivatives and gradient descent; Applications and computation recap; Lab U5. U5-LO4: Eigenvalues, eigenvectors, and diagonalization; Symmetric matrices and quadratic forms. U5-LO5: Symmetric matrices and quadratic forms; Hessians and the second derivative test. U5-LO6: Least squares from gradients; residual-orthogonality code checks. U5-LO7: Fixed-hidden-layer training; Applications and computation recap; Lab U5.
Applications and computations readiness checklist. Given a short Unit 5 calculation, code snippet, loss table, or matrix output, I can identify a gradient descent update; explain the role of the learning rate; diagnose slow, useful, or unstable descent from loss values; read a simplified next-token loss as an optimization objective; read Hessian eigenvalues as curvature information at a critical point; classify positive definite, negative definite, and indefinite Hessians; read
np.linalg.eig or np.linalg.eigvalsh as an eigenvalue computation; read np.linalg.lstsq(A, b, rcond=None)[0] as a least-squares solve; interpret A.T @ r as a residual-orthogonality check; identify fixed feature columns in a design matrix; and explain why np.outer(g, h) forms a rank-one update.
Common mistakes. Treating every critical point as a maximum or minimum; ignoring boundaries or constraints when a problem has them; using the Hessian test away from a critical point; treating a zero Hessian eigenvalue as a classification instead of an inconclusive case; forgetting the minus sign in gradient descent; assuming a larger learning rate is always better; confusing eigenvalues with eigenvectors; thinking \(A^T\mathbf{r}=\mathbf{0}\) means \(\mathbf{r}=\mathbf{0}\text{;}\) forgetting that least-squares uniqueness depends on column independence; and missing that a fixed-hidden-layer model can be nonlinear in \(t\) but linear in the trained coefficients \(\mathbf{c}\text{.}\)
