Section C.3 Unit 3 conceptual, geometric, and computational checkpoints
These solution sketches correspond to the additional conceptual, geometric, and computational checkpoints at the end of Unit 3.
Subsection Directional derivative as a dot product
Use
\begin{equation*}
D_{\mathbf{v}}f(\mathbf{a})=\nabla f(\mathbf{a})\cdot \mathbf{v}.
\end{equation*}
Then
\begin{equation*}
D_{\mathbf{v}_1}f(\mathbf{a})=6,
\qquad
D_{\mathbf{v}_2}f(\mathbf{a})=8.
\end{equation*}
Also,
\begin{equation*}
D_{\mathbf{v}_3}f(\mathbf{a})
=
\begin{bmatrix}
6\\
8
\end{bmatrix}
\cdot
\frac15
\begin{bmatrix}
3\\
4
\end{bmatrix}
=
\frac{18+32}{5}=10,
\end{equation*}
and
\begin{equation*}
D_{\mathbf{v}_4}f(\mathbf{a})=-10.
\end{equation*}
The largest listed directional derivative occurs in direction \(\mathbf{v}_3\text{,}\) and the smallest occurs in direction \(\mathbf{v}_4\text{.}\) This is the Unit 1 dot product measuring alignment: the derivative is largest when the direction is aligned with the gradient and smallest when it points opposite the gradient.
Subsection Final-layer rank-one update
The outer product is
\begin{equation*}
\mathbf{g}\mathbf{h}^T
=
\begin{bmatrix}
1\\
-2\\
3
\end{bmatrix}
\begin{bmatrix}
4\amp0
\end{bmatrix}
=
\begin{bmatrix}
4\amp0\\
-8\amp0\\
12\amp0
\end{bmatrix}.
\end{equation*}
Its shape is \(3\times 2\text{.}\) Since every column is a scalar multiple of \(\mathbf{g}\text{,}\) the column space has dimension at most one, so the rank is at most one.
The update
\begin{equation*}
W_{\mathrm{new}}=W-\alpha \mathbf{g}\mathbf{h}^T
\end{equation*}
changes \(W\) by subtracting a scaled rank-one matrix. In NumPy,
np.outer(g, h) is appropriate when g and h are stored as one-dimensional arrays because it explicitly forms the matrix \(\mathbf{g}\mathbf{h}^T\text{.}\)
The update matrix has rank at most one. This does not imply that the updated matrix \(W_{\mathrm{new}}\) has rank at most one.
Subsection Critical points are candidates
We have
\begin{equation*}
\nabla f(x,y)=
\begin{bmatrix}
2x\\
-2y
\end{bmatrix}.
\end{equation*}
Thus the only critical point is \((0,0)\text{.}\)
Along the \(x\)-axis,
\begin{equation*}
f(t,0)=t^2,
\end{equation*}
which is positive for \(t\ne 0\text{.}\) Along the \(y\)-axis,
\begin{equation*}
f(0,t)=-t^2,
\end{equation*}
which is negative for \(t\ne 0\text{.}\) Nearby points therefore occur both above and below \(f(0,0)=0\text{.}\) The critical point is a saddle point, not a local minimum or local maximum.
Subsection Reading a gradient descent loop
The loop represents
\begin{equation*}
\mathbf{x}_{k+1}=\mathbf{x}_k-\alpha\nabla f(\mathbf{x}_k).
\end{equation*}
The learning rate is
alpha. The vector grad_f(x) must have the same shape as x. Replacing the minus sign by a plus sign would move in the direction of steepest increase instead of steepest decrease. The code does not prove that a minimum has been found. It only describes the update rule; convergence depends on the function, starting point, learning rate, and stopping rule.Subsection Learning-rate diagnosis
The learning rate \(\alpha=0.05\) is making slow but steady progress. The learning rate \(\alpha=0.20\) is making faster useful progress. The learning rate \(\alpha=1.05\) appears unstable because the loss increases.
A decreasing loss table does not prove that the global minimum has been found. It only reports what happened for the displayed iterates.
Subsection Function type and component functions
The function is vector-valued:
\begin{equation*}
F:\mathbb R^2\to\mathbb R^3.
\end{equation*}
Its component functions are
\begin{equation*}
F_1(x,y)=x^2+y,\qquad F_2(x,y)=e^{x-y},\qquad F_3(x,y)=x-y.
\end{equation*}
We have
\begin{equation*}
F(1,0)=\begin{bmatrix}1\\e\\1\end{bmatrix}.
\end{equation*}
Since the output has three components and the input has two variables, \(J_F(1,0)\) has shape \(3\times2\text{.}\)
Subsection Domain, boundary, closed, and bounded
The logarithm requires \(y-x^2>0\text{,}\) and the square root requires \(x\ge 0\text{.}\) Thus the domain is
\begin{equation*}
D=\{(x,y)\in\mathbb R^2:x\ge0,\ y>x^2\}.
\end{equation*}
One interior point is \((1,2)\text{.}\) Boundary points include points on \(y=x^2\) with \(x\ge0\text{,}\) such as \((1,1)\text{,}\) and points on \(x=0\) with \(y\ge0\text{,}\) such as \((0,1)\text{.}\) The set is not closed because it does not include the parabola \(y=x^2\text{.}\) It is not bounded because points in \(D\) can have arbitrarily large coordinates.
Subsection One-parameter map and tangent vector
We have
\begin{equation*}
\mathbf r(2)=
\begin{bmatrix}1\\0\\-1\end{bmatrix}
+2\begin{bmatrix}2\\-1\\3\end{bmatrix}
=
\begin{bmatrix}5\\-2\\5\end{bmatrix}.
\end{equation*}
The derivative is constant:
\begin{equation*}
\mathbf r'(t)=\begin{bmatrix}2\\-1\\3\end{bmatrix}.
\end{equation*}
This is the direction vector of the line, matching the Unit 2 parametric-line form point plus parameter times direction.
Subsection Gradient and Hessian
The first partial derivatives are
\begin{equation*}
f_x=2x+y,\qquad f_y=x+4y.
\end{equation*}
Hence
\begin{equation*}
\nabla f(x,y)=\begin{bmatrix}2x+y\\x+4y\end{bmatrix},
\qquad
\nabla f(1,-1)=\begin{bmatrix}1\\-3\end{bmatrix}.
\end{equation*}
The Hessian is
\begin{equation*}
H_f=\begin{bmatrix}2&1\\1&4\end{bmatrix},
\end{equation*}
which is symmetric. The entries of \(\nabla f(1,-1)\) measure the sensitivity of \(f\) to changes in the \(x\)- and \(y\)-directions at that point.
Subsection Jacobian matrix rows and columns
The Jacobian matrix is
\begin{equation*}
J_F(x,y)=
\begin{bmatrix}
2xy&x^2\\
e^y&xe^y
\end{bmatrix}.
\end{equation*}
At \(\mathbf a=(1,0)\text{,}\)
\begin{equation*}
J_F(\mathbf a)=
\begin{bmatrix}
0&1\\
1&1
\end{bmatrix}.
\end{equation*}
The input and output dimensions are both \(2\text{.}\) The second row corresponds to the second component \(F_2(x,y)=xe^y\text{.}\) The columns measure the predicted output changes from changing the first and second input coordinates.
Subsection Tangent plane as a Unit 2 plane
We have \(f(1,2)=3\text{,}\) \(f_x(1,2)=4\text{,}\) and \(f_y(1,2)=1\text{.}\) The tangent plane is
\begin{equation*}
z-3=4(x-1)+(y-2).
\end{equation*}
Equivalently,
\begin{equation*}
\begin{bmatrix}-4\\-1\\1\end{bmatrix}\cdot
\begin{bmatrix}
x-1\\
y-2\\
z-3
\end{bmatrix}
=0.
\end{equation*}
A normal vector is \(\mathbf n=(-4,-1,1)^T\text{.}\) The Unit 2 idea being reused is a plane described by a point and a normal vector.
Subsection Scalar local linear prediction
From the previous function, \(f(\mathbf a)=f(1,2)=3\) and
\begin{equation*}
\nabla f(\mathbf a)=\begin{bmatrix}4\\1\end{bmatrix}.
\end{equation*}
The local linear prediction is
\begin{equation*}
3+\begin{bmatrix}4&1\end{bmatrix}
\begin{bmatrix}0.1\\-0.2\end{bmatrix}
=3+0.2=3.2.
\end{equation*}
The actual value is
\begin{equation*}
f(1.1,1.8)=1.1^2+1.1(1.8)=3.19.
\end{equation*}
The error actual minus prediction is \(3.19-3.2=-0.01\text{.}\)
Subsection Vector-valued local linear prediction
We have
\begin{equation*}
F(\mathbf a)=F(1,0)=\begin{bmatrix}0\\1\end{bmatrix}
\end{equation*}
and, from the previous Jacobian matrix exercise,
\begin{equation*}
J_F(\mathbf a)\mathbf h=
\begin{bmatrix}
0&1\\
1&1
\end{bmatrix}
\begin{bmatrix}0.1\\-0.2\end{bmatrix}
=
\begin{bmatrix}-0.2\\-0.1\end{bmatrix}.
\end{equation*}
Thus the local prediction is
\begin{equation*}
F(\mathbf a)+J_F(\mathbf a)\mathbf h=\begin{bmatrix}-0.2\\0.9\end{bmatrix}.
\end{equation*}
The actual value is approximately
\begin{equation*}
F(1.1,-0.2)=\begin{bmatrix}-0.242\\0.9006\end{bmatrix}.
\end{equation*}
The error vector measures actual output minus predicted output.
Subsection Locally forgotten direction
One choice is
\begin{equation*}
\mathbf h=\begin{bmatrix}-2\\1\end{bmatrix},
\end{equation*}
since
\begin{equation*}
\begin{bmatrix}1&2\\2&4\end{bmatrix}
\begin{bmatrix}-2\\1\end{bmatrix}
=
\begin{bmatrix}0\\0\end{bmatrix}.
\end{equation*}
This direction is in the null space of the Jacobian matrix, so the local linear map forgets it to first order. This connects directly to Unit 2 null spaces. It does not prove \(F(\mathbf a+\mathbf h)=F(\mathbf a)\) exactly, because the Jacobian matrix only gives a local linear approximation.
Subsection Chain-rule shape check
Since \(G:\mathbb R^2\to\mathbb R^3\text{,}\) the Jacobian matrix \(J_G(\mathbf a)\) has shape \(3\times2\text{.}\) Since \(F:\mathbb R^3\to\mathbb R^2\text{,}\) the Jacobian matrix \(J_F(G(\mathbf a))\) has shape \(2\times3\text{.}\)
Therefore
\begin{equation*}
J_{F\circ G}(\mathbf a)=J_F(G(\mathbf a))J_G(\mathbf a)
\end{equation*}
has shape \(2\times2\text{.}\) The reverse order is not correct because the shapes do not match the order of composition, and in general Jacobian matrix products do not commute.
Subsection Reading a tiny sigmoid block
The lines
\begin{equation*}
\mathbf{s}=W_1\mathbf{x}+\mathbf{b}_1,\qquad \mathbf{y}=W_2\mathbf{u}+\mathbf{b}_2
\end{equation*}
are affine maps. The line
u = sigmoid(s) contains the coordinatewise nonlinear step.The variables
x, s, u, b1, b2, and y represent vectors. The matrix \(D\) stores the sigmoid derivative at each coordinate of \(\mathbf{s}\text{.}\) The line J_at_x = W2 @ D @ W1 computes the Jacobian matrix at \(\mathbf{x}\text{.}\) The full block is usually not one linear map because of the bias terms and the coordinatewise sigmoid step.
Subsection Corners are not enough
Both maps send the four corners to the same four outputs:
\begin{equation*}
\begin{bmatrix}0\\0\end{bmatrix}\mapsto\begin{bmatrix}0\\0\end{bmatrix},
\quad
\begin{bmatrix}1\\0\end{bmatrix}\mapsto\begin{bmatrix}1\\0\end{bmatrix},
\quad
\begin{bmatrix}1\\1\end{bmatrix}\mapsto\begin{bmatrix}2\\1\end{bmatrix},
\quad
\begin{bmatrix}0\\1\end{bmatrix}\mapsto\begin{bmatrix}1\\1\end{bmatrix}.
\end{equation*}
But
\begin{equation*}
T\left(\begin{bmatrix}0\\1/2\end{bmatrix}\right)
=
\begin{bmatrix}1/2\\1/2\end{bmatrix},
\qquad
F\left(\begin{bmatrix}0\\1/2\end{bmatrix}\right)
=
\begin{bmatrix}1/4\\1/2\end{bmatrix}.
\end{equation*}
So the corner test misses the difference. A grid samples more points, including edge and interior points, and can show bending.
Subsection Local square-grid error
The Jacobian matrix is
\begin{equation*}
J_F(x,y)
=
\begin{bmatrix}
1&2y\\
0&1
\end{bmatrix}.
\end{equation*}
At
\begin{equation*}
\mathbf{a}
=
\begin{bmatrix}
0\\
1/2
\end{bmatrix},
\end{equation*}
we get
\begin{equation*}
J_F(\mathbf{a})
=
\begin{bmatrix}
1&1\\
0&1
\end{bmatrix}.
\end{equation*}
For
\begin{equation*}
\mathbf{h}
=
\begin{bmatrix}
h_1\\
h_2
\end{bmatrix},
\end{equation*}
we have
\begin{equation*}
F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})
=
\begin{bmatrix}
h_1+h_2+h_2^2\\
h_2
\end{bmatrix}
\end{equation*}
and
\begin{equation*}
J_F(\mathbf{a})\mathbf{h}
=
\begin{bmatrix}
h_1+h_2\\
h_2
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})-J_F(\mathbf{a})\mathbf{h}
=
\begin{bmatrix}
h_2^2\\
0
\end{bmatrix}.
\end{equation*}
If \(|h_2|\le r\text{,}\) then \(h_2^2\le r^2\text{,}\) so the error goes to zero as the square shrinks.
Subsection Code interpretation: square-grid arrays
The array \(P\) has shape \(2\times 5\text{.}\) Its columns are input points. The last column is
\begin{equation*}
\begin{bmatrix}
0\\
1/2
\end{bmatrix}.
\end{equation*}
The corresponding output column is
\begin{equation*}
\begin{bmatrix}
1/4\\
1/2
\end{bmatrix}.
\end{equation*}
The expression
np.vstack([x + y**2, y]) builds the two output components of \(F\) as rows of a new \(2\times 5\) array. This is not matrix multiplication because the first output component contains \(y^2\text{,}\) a nonlinear operation.Subsection Sigmoid derivative and local matrix
At
\begin{equation*}
\mathbf{a}
=
\begin{bmatrix}
0\\
0
\end{bmatrix},
\end{equation*}
we get
\begin{equation*}
\mathbf{s}_{\mathbf a}=W_1\mathbf a+\mathbf b_1
=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
\mathbf{u}_{\mathbf a}
=
\sigma(\mathbf{s}_{\mathbf a})
=
\begin{bmatrix}
1/2\\
1/2\\
1/2
\end{bmatrix},
\qquad
D_{\mathbf a}
=
\operatorname{diag}(1/4,1/4,1/4).
\end{equation*}
Thus
\begin{equation*}
W_2D_{\mathbf a}W_1
=
\begin{bmatrix}
1&0&-1/2\\
0&1&1/2
\end{bmatrix}
\begin{bmatrix}
1/4&0&0\\
0&1/4&0\\
0&0&1/4
\end{bmatrix}
\begin{bmatrix}
1&0\\
0&1\\
1&1
\end{bmatrix}
=
\begin{bmatrix}
1/8&-1/8\\
1/8&3/8
\end{bmatrix}.
\end{equation*}
Its determinant is
\begin{equation*}
(1/8)(3/8)-(-1/8)(1/8)=1/16,
\end{equation*}
so this local matrix is invertible.
For \(\mathbf c=\begin{bmatrix}1\\0\end{bmatrix}\text{,}\) we get
\begin{equation*}
\mathbf{s}_{\mathbf c}
=
\begin{bmatrix}
1\\
0\\
1
\end{bmatrix},
\qquad
D_{\mathbf c}=\operatorname{diag}(d,1/4,d),
\qquad
d=\sigma(1)(1-\sigma(1)).
\end{equation*}
Since \(d\ne 1/4\text{,}\) the first and third entries of \(D_{\mathbf c}\) differ from the corresponding entries of \(D_{\mathbf a}\text{.}\) The block is smooth, but the sigmoid derivative changes with the input, so the Jacobian matrix can change with the base point.
Subsection Code interpretation: sigmoid block arrays
The array \(P\) has shape \(2\times 3\text{.}\) Its columns are input points. Since \(W_1\) has shape \(3\times 2\text{,}\) the array \(S=W_1P+\mathbf{b}_1\) has shape \(3\times 3\text{.}\) The array \(U=\sigma(S)\) has the same shape.
The nonlinear operation is
U = sigmoid(S)
The first column of \(P\) is
\begin{equation*}
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
For this input,
\begin{equation*}
\mathbf{s}=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix},
\qquad
\mathbf{u}=
\begin{bmatrix}
1/2\\
1/2\\
1/2
\end{bmatrix}.
\end{equation*}
Therefore the first output column is
\begin{equation*}
W_2\mathbf{u}+\mathbf{b}_2
=
\begin{bmatrix}
1/4\\
3/4
\end{bmatrix}.
\end{equation*}
The full computation is not one matrix multiplication \(AP\) because the sigmoid step is nonlinear.
Subsection Softmax weights are data-dependent
The score vector is
\begin{equation*}
\mathbf{s}
=
K\mathbf{q}
=
\begin{bmatrix}
3q_1\\
3q_2\\
-3q_1-3q_2
\end{bmatrix}.
\end{equation*}
At \(\mathbf{q}=\mathbf{0}\text{,}\)
\begin{equation*}
\mathbf{s}
=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
\end{equation*}
All three scores are equal, so the softmax weights are
\begin{equation*}
\boldsymbol{\alpha}(\mathbf{0})
=
\begin{bmatrix}
1/3\\
1/3\\
1/3
\end{bmatrix}.
\end{equation*}
Therefore
\begin{equation*}
F(\mathbf{0})
=
V^T
\begin{bmatrix}
1/3\\
1/3\\
1/3
\end{bmatrix}
=
\begin{bmatrix}
1/3\\
1/3
\end{bmatrix}.
\end{equation*}
The softmax entries are positive and add to \(1\text{,}\) so \(F(\mathbf{q})\) is always a weighted average of the rows of \(V\text{.}\) The nonlinear step is the softmax rule that turns scores into weights.
Subsection Code interpretation: softmax square-grid arrays
The array \(P\) has shape \(2\times 4\text{.}\) Its columns are query vectors. Since \(K\) has shape \(3\times 2\text{,}\) the product
\begin{equation*}
S=KP
\end{equation*}
has shape \(3\times 4\text{.}\) The array
Alpha also has shape \(3\times 4\text{,}\) with one softmax weight vector in each column.Since \(V^T\) has shape \(2\times 3\text{,}\) the product
\begin{equation*}
Q=V^T\mathrm{Alpha}
\end{equation*}
has shape \(2\times 4\text{.}\) Its columns are output vectors.
The nonlinear operation is
Alpha = softmax_columns(S)
The full computation is not one matrix multiplication \(AP\) because the softmax step uses exponentials and column sums. The weights depend nonlinearly on the scores, and the scores depend on \(P\text{.}\)
Subsection Reading square-grid behavior
Map A is the nonlinear shear. It bends grid lines smoothly because of the \(y^2\) term, and it can agree with a linear shear on four corners while differing at edge and interior points.
Map B is the sigmoid block. Its local matrix has the form \(W_2D_{\mathbf a}W_1\text{,}\) where \(D_{\mathbf a}\) stores sigmoid derivatives. Its grid can bend and compress because those derivatives depend on the input.
Map C is the attention-style softmax map. The output is
\begin{equation*}
V^T\boldsymbol{\alpha},
\end{equation*}
where the entries of \(\boldsymbol{\alpha}\) are positive, add to \(1\text{,}\) and depend on the input. Therefore the output is a weighted average of the value vectors and stays inside the triangle they form.
Thus behaviors 1 and 4 match Map A, behaviors 2 and 5 match Map B, and behaviors 3 and 6 match Map C. Map A best shows why four corners alone can miss nonlinear behavior. Map B best shows the chain rule for Jacobian matrices. Map C best shows attention-style weighted averages. A square-grid visualization can show bending, compression, and changing local behavior that four corners alone may miss.
