MATH 345: Linear Algebra and Optimization

This course is about linear algebra, together with the parts of multivariable calculus most relevant to optimization problems: given a quantity depending on multiple parameters, finding parameters that maximize or minimize that quantity. These tools are fundamental to data science, where one often builds models of data by choosing parameters that make a model best approximate a given data set. The course has one recurring theme: linear algebra studies linear maps; differential calculus studies nonlinear functions by replacing them locally with linear maps; optimization uses both to solve data problems.

The book is organized into seven units. Unit 1 introduces vectors, matrices, and linear maps. Unit 2 studies the anatomy of a linear map: systems, geometry, redundancy, rank, and forgotten directions. Unit 3 moves beyond linear maps to general functions and local linearization. Unit 4 develops orthogonality, projection, and least squares. Unit 5 studies optimization, gradient descent, eigenvalues, Hessians, and training. Unit 6 introduces constrained optimization and the singular value decomposition. Unit 7 uses abstract vector spaces and polynomial approximation to synthesize projection, least squares, Taylor approximation, Gram-Schmidt, and Hessians.

Applications. The application threads are intentionally recurrent. Vectors, distance, and dot products introduce embeddings, nearest neighbors, and cosine similarity in Unit 1 and return in Units 4 and 5. Weighted linear combinations and matrix-vector products preview attention in Unit 1 and return in Units 4 and 5. Linear maps \(\mathbf{x}\mapsto A\mathbf{x}\) begin in Unit 1 and return through Units 2 and 3. Systems \(Ax=b\text{,}\) null spaces, rank, lines and planes, and local linearization connect solution sets, forgotten directions, tangent planes, and sensitivity across Units 2 and 3. Projection and least squares drive regression in Unit 4 and return in Units 5 and 7. Gradients, Hessians, Lagrange multipliers, SVD, and polynomial inner products organize optimization, constraints, compression, Taylor approximation, and regression in Units 5-7. The Transformer Wrap-Up Worksheet revisits token embeddings, attention scores, weighted averages, affine layers, nonlinear blocks, training loss, gradients, and low-rank updates. The course is not a course on large language models. The goal is to learn mathematical tools that are useful across many areas: data science, optimization, modeling, numerical computation, statistics, engineering, and machine learning.

How to use learning outcomes. Each unit begins with a small set of core learning outcomes, labeled U1-LO1, U1-LO2, and so on. The labels return in exercises, labs, interpretation checks, exam-style checks, and unit highlights. A core outcome names a skill students should be able to use in a new problem. Toolbox skills are smaller computations that support the core outcomes, and optional outcomes mark extension material that is useful but not required on the main path through the course. Exercise and lab tags use the following competency labels: P = procedural computation, C = conceptual interpretation, R = representation or translation between notation, geometry, words, and code, M = modeling or application, J = justification or proof, and T = technology or code interpretation. For example, [U4-LO4 | C+T | Core] means that the task develops conceptual interpretation of Unit 4’s least-squares residual outcome and interpretation of code, numerical output, or a computational check.

Outcome recurrence. Dot products and cosine similarity first appear in U1-LO1 and U1-LO2, then return in Units 4 and 5 through embeddings, nearest neighbors, and attention previews. Matrix-vector products and composition first appear in U1-LO4 and U1-LO6, then return in Units 2 and 3 through linear maps, matrix actions, and local linearization. Reachable outputs and null directions first appear in U2-LO3 and U2-LO6, then return in Units 4 and 6 through redundancy, forgotten directions, and SVD. Local linearization and Jacobian matrices first appear in U3-LO5 and U3-LO6, then return in Unit 5 through sensitivity, the chain rule, and training. Projection and least squares first appear in U4-LO4 and U4-LO5, then return in Units 5 and 7 through regression, normal equations, and polynomial approximation. Gradients, Hessians, and loss landscapes first appear in U5-LO1 and U5-LO5, then return in Unit 7 through optimization, quadratic approximation, and Hessian forms. SVD and low-rank approximation first appear in U6-LO4 and U6-LO7, then return in Unit 7 through compression, dominant directions, and low-rank structure. Polynomial inner products and best approximation first appear in U7-LO3 and U7-LO5 and close the final synthesis through Taylor approximation, continuous least squares, and sampled least squares.

Labs. The notebooks are linked supplements to the textbook. The core lab sequence includes Lab U1 on vectors, similarity, attention, and matrix actions; Lab U2 on auditing a linear map; Lab U3 on Jacobian matrices and local linearization; Lab U4 on regression as projection; Lab U5 on gradient descent and tiny training; Lab U6 on SVD and compression; and Lab U7 on polynomial approximation. The Transformer Wrap-Up Lab is an extension attached to the Transformer Wrap-Up Worksheet appendix. A detailed lab index appears at the end of the book.