Note 3.9.1. Unit highlights.
Big question. If a function is not linear, what linear map best describes it near one point?
Notation. \(F:\mathbb R^n\to\mathbb R^m\text{,}\) \(f:\mathbb R^n\to\mathbb R\text{,}\) \(F_i\text{,}\) \(\nabla f\text{,}\) \(H_f\text{,}\) \(J_F\text{,}\) \(F(\mathbf a+\mathbf h)\approx F(\mathbf a)+J_F(\mathbf a)\mathbf h\text{,}\) \(J_{F\circ G}(\mathbf a)=J_F(G(\mathbf a))J_G(\mathbf a)\text{,}\) \(\sigma(t)\text{,}\) \(D_{\mathbf a}\text{,}\) \(A_{\mathrm{att}}\text{,}\) \(H=A_{\mathrm{att}}V\text{.}\)
Concepts. Scalar-valued function, vector-valued map, component function, domain, range, graph, level curve, ball, limit, continuity, interior point, boundary point, closed set, bounded set, tangent vector, partial derivative, gradient, Hessian, Jacobian matrix, differentiability, tangent plane, local linearization, square-grid visualization, coordinatewise sigmoid, diagonal derivative matrix, local matrix for a sigmoid block, softmax weighting rule, attention-style weighted average, input-dependent weights, chain rule, composed nonlinear block.
Results. A partial derivative measures change in one coordinate direction. The gradient collects first partial derivatives of a scalar-valued function. The Hessian collects second partial derivatives. The Jacobian matrix collects the partial derivatives of all component functions of a vector-valued map. Differentiability gives the local model \(F(\mathbf a+\mathbf h)\approx F(\mathbf a)+J_F(\mathbf a)\mathbf h\text{.}\) For compositions, Jacobian matrices multiply in the order the local linear maps compose. In a sigmoid block, the local matrix has the form \(W_2D_{\mathbf a}W_1\text{,}\) where \(D_{\mathbf a}\) stores sigmoid derivatives at the base point. In an attention-style block, matrix multiplications can appear inside a nonlinear rule because the weights depend on the input.
Toolbox skills. Classify function types; find domains; read level curves; compute derivatives of vector-valued functions; compute partial derivatives, gradients, Hessians, and Jacobian matrices; write tangent planes; evaluate local linear approximations; check Jacobian matrix shapes; multiply Jacobian matrices in the chain rule; read diagonal derivative matrices for coordinatewise nonlinear rules; compare actual and predicted changes.
Main applications. Tangent planes, local sensitivity, local forgotten directions, finite-difference checks, symbolic Jacobian matrix computations, square-grid visualizations for comparing linear and nonlinear maps, square-grid visualizations for small smooth nonlinear blocks, square-grid visualizations for attention-style softmax weights, chain-rule shape checks, tiny sigmoid blocks, attention-style nonlinear blocks, and dependency checks for input-dependent weights.
Connections. Unit 1 supplies matrix-vector products, affine maps, composition, unit-square visualizations, and attention-style weighted averages. Unit 2 supplies lines, planes, normal vectors, and null-space language. Unit 3 turns these into local tools for nonlinear functions. Unit 5 uses gradients, Hessians, Jacobian matrices, and the chain rule for optimization and training.
Study anchors. U3-LO1: General functions and domains. U3-LO2: Balls, limits, and continuity; One-parameter maps and changes. U3-LO3: Partial derivatives, gradients, Hessians, and Jacobian matrices. U3-LO4 and U3-LO5: Local linearization and differentiability. U3-LO6: Chain rule as matrix multiplication. U3-LO7: Tiny sigmoid blocks, attention-style nonlinear blocks, dependency checks, and Applications and computation recap.
Applications and computations readiness checklist. Given a short Unit 3 calculation, code snippet, table, or diagram, I can classify scalar-valued and vector-valued functions; identify domains and boundary points; read tangent vectors; compute gradients, Hessians, and Jacobian matrices; use a tangent plane or local linear approximation; interpret \(J_F(\mathbf a)\mathbf h\) as a predicted output change; identify a locally forgotten direction from the null space of a Jacobian matrix; check Jacobian matrix shapes in the chain rule; read a diagonal derivative matrix for a sigmoid block; explain why softmax weights are input-dependent; read \(H=A_{\mathrm{att}}V\) as weighted averages when \(A_{\mathrm{att}}\) is fixed; and explain why an attention-style rule can be nonlinear even when it contains matrix multiplications.
Common mistakes. Confusing scalar-valued and vector-valued functions; treating a local approximation as exact far from the base point; using a Jacobian matrix with the wrong shape; multiplying chain-rule factors in the wrong order; confusing the input change \(\mathbf h\) with the original input \(\mathbf x\text{;}\) assuming a local matrix is the same at every base point; calling an attention-style rule linear just because it contains matrix multiplications; forgetting to specify which quantities are fixed and which depend on the input.
