Section C.3 Unit 3 conceptual, geometric, and computational checkpoints
These solution sketches correspond to the additional conceptual, geometric, and computational checkpoints at the end of Unit 3.
Subsection Function type and component functions
The function is vector-valued:
\begin{equation*}
F:\mathbb R^2\to\mathbb R^3.
\end{equation*}
Its component functions are
\begin{equation*}
F_1(x,y)=x^2+y,\qquad F_2(x,y)=e^{x-y},\qquad F_3(x,y)=x-y.
\end{equation*}
We have
\begin{equation*}
F(1,0)=\begin{bmatrix}1\\e\\1\end{bmatrix}.
\end{equation*}
Since the output has three components and the input has two variables, \(J_F(1,0)\) has shape \(3\times2\text{.}\)
Subsection Domain, boundary, closed, and bounded
The logarithm requires \(y-x^2>0\text{,}\) and the square root requires \(x\ge 0\text{.}\) Thus the domain is
\begin{equation*}
D=\{(x,y)\in\mathbb R^2:x\ge0,\ y>x^2\}.
\end{equation*}
One interior point is \((1,2)\text{.}\) Boundary points include points on \(y=x^2\) with \(x\ge0\text{,}\) such as \((1,1)\text{,}\) and points on \(x=0\) with \(y\ge0\text{,}\) such as \((0,1)\text{.}\) The set is not closed because it does not include the parabola \(y=x^2\text{.}\) It is not bounded because points in \(D\) can have arbitrarily large coordinates.
Subsection One-parameter map and tangent vector
We have
\begin{equation*}
\mathbf r(2)=
\begin{bmatrix}1\\0\\-1\end{bmatrix}
+2\begin{bmatrix}2\\-1\\3\end{bmatrix}
=
\begin{bmatrix}5\\-2\\5\end{bmatrix}.
\end{equation*}
The derivative is constant:
\begin{equation*}
\mathbf r'(t)=\begin{bmatrix}2\\-1\\3\end{bmatrix}.
\end{equation*}
This is the direction vector of the line, matching the Unit 2 parametric-line form point plus parameter times direction.
Subsection Gradient and Hessian
The first partial derivatives are
\begin{equation*}
f_x=2x+y,\qquad f_y=x+4y.
\end{equation*}
Hence
\begin{equation*}
\nabla f(x,y)=\begin{bmatrix}2x+y\\x+4y\end{bmatrix},
\qquad
\nabla f(1,-1)=\begin{bmatrix}1\\-3\end{bmatrix}.
\end{equation*}
The Hessian is
\begin{equation*}
H_f=\begin{bmatrix}2&1\\1&4\end{bmatrix},
\end{equation*}
which is symmetric. The entries of \(\nabla f(1,-1)\) measure the sensitivity of \(f\) to changes in the \(x\)- and \(y\)-directions at that point.
Subsection Jacobian matrix rows and columns
The Jacobian matrix is
\begin{equation*}
J_F(x,y)=
\begin{bmatrix}
2xy&x^2\\
e^y&xe^y
\end{bmatrix}.
\end{equation*}
At \(\mathbf a=(1,0)\text{,}\)
\begin{equation*}
J_F(\mathbf a)=
\begin{bmatrix}
0&1\\
1&1
\end{bmatrix}.
\end{equation*}
The input and output dimensions are both \(2\text{.}\) The second row corresponds to the second component \(F_2(x,y)=xe^y\text{.}\) The columns measure the predicted output changes from changing the first and second input coordinates.
Subsection Tangent plane as a Unit 2 plane
We have \(f(1,2)=3\text{,}\) \(f_x(1,2)=4\text{,}\) and \(f_y(1,2)=1\text{.}\) The tangent plane is
\begin{equation*}
z-3=4(x-1)+(y-2).
\end{equation*}
Equivalently,
\begin{equation*}
\begin{bmatrix}-4\\-1\\1\end{bmatrix}\cdot
\begin{bmatrix}
x-1\\
y-2\\
z-3
\end{bmatrix}
=0.
\end{equation*}
A normal vector is \(\mathbf n=(-4,-1,1)^T\text{.}\) The Unit 2 idea being reused is a plane described by a point and a normal vector.
Subsection Scalar local linear prediction
From the previous function, \(f(\mathbf a)=f(1,2)=3\) and
\begin{equation*}
\nabla f(\mathbf a)=\begin{bmatrix}4\\1\end{bmatrix}.
\end{equation*}
The local linear prediction is
\begin{equation*}
3+\begin{bmatrix}4&1\end{bmatrix}
\begin{bmatrix}0.1\\-0.2\end{bmatrix}
=3+0.2=3.2.
\end{equation*}
The actual value is
\begin{equation*}
f(1.1,1.8)=1.1^2+1.1(1.8)=3.19.
\end{equation*}
The error actual minus prediction is \(3.19-3.2=-0.01\text{.}\)
Subsection Vector-valued local linear prediction
We have
\begin{equation*}
F(\mathbf a)=F(1,0)=\begin{bmatrix}0\\1\end{bmatrix}
\end{equation*}
and, from the previous Jacobian matrix exercise,
\begin{equation*}
J_F(\mathbf a)\mathbf h=
\begin{bmatrix}
0&1\\
1&1
\end{bmatrix}
\begin{bmatrix}0.1\\-0.2\end{bmatrix}
=
\begin{bmatrix}-0.2\\-0.1\end{bmatrix}.
\end{equation*}
Thus the local prediction is
\begin{equation*}
F(\mathbf a)+J_F(\mathbf a)\mathbf h=\begin{bmatrix}-0.2\\0.9\end{bmatrix}.
\end{equation*}
The actual value is approximately
\begin{equation*}
F(1.1,-0.2)=\begin{bmatrix}-0.242\\0.9006\end{bmatrix}.
\end{equation*}
The error vector measures actual output minus predicted output.
Subsection Locally forgotten direction
One choice is
\begin{equation*}
\mathbf h=\begin{bmatrix}-2\\1\end{bmatrix},
\end{equation*}
since
\begin{equation*}
\begin{bmatrix}1&2\\2&4\end{bmatrix}
\begin{bmatrix}-2\\1\end{bmatrix}
=
\begin{bmatrix}0\\0\end{bmatrix}.
\end{equation*}
This direction is in the null space of the Jacobian matrix, so the local linear map forgets it to first order. This connects directly to Unit 2 null spaces. It does not prove \(F(\mathbf a+\mathbf h)=F(\mathbf a)\) exactly, because the Jacobian matrix only gives a local linear approximation.
Subsection Chain-rule shape check
Since \(G:\mathbb R^2\to\mathbb R^3\text{,}\) the Jacobian matrix \(J_G(\mathbf a)\) has shape \(3\times2\text{.}\) Since \(F:\mathbb R^3\to\mathbb R^2\text{,}\) the Jacobian matrix \(J_F(G(\mathbf a))\) has shape \(2\times3\text{.}\)
Therefore
\begin{equation*}
J_{F\circ G}(\mathbf a)=J_F(G(\mathbf a))J_G(\mathbf a)
\end{equation*}
has shape \(2\times2\text{.}\) The reverse order is not correct because the shapes do not match the order of composition, and in general Jacobian matrix products do not commute.
Subsection Reading a tiny sigmoid block
The lines
\begin{equation*}
\mathbf{s}=W_1\mathbf{x}+\mathbf{b}_1,\qquad \mathbf{y}=W_2\mathbf{u}+\mathbf{b}_2
\end{equation*}
are affine maps. The line
u = sigmoid(s) contains the coordinatewise nonlinear step.The variables
x, s, u, b1, b2, and y represent vectors. The matrix \(D\) stores the sigmoid derivative at each coordinate of \(\mathbf{s}\text{.}\) The line J_at_x = W2 @ D @ W1 computes the Jacobian matrix at \(\mathbf{x}\text{.}\) The full block is usually not one linear map because of the bias terms and the coordinatewise sigmoid step.
Subsection Corners are not enough
Both maps send the four corners to the same four outputs:
\begin{equation*}
\begin{bmatrix}0\\0\end{bmatrix}\mapsto\begin{bmatrix}0\\0\end{bmatrix},
\quad
\begin{bmatrix}1\\0\end{bmatrix}\mapsto\begin{bmatrix}1\\0\end{bmatrix},
\quad
\begin{bmatrix}1\\1\end{bmatrix}\mapsto\begin{bmatrix}2\\1\end{bmatrix},
\quad
\begin{bmatrix}0\\1\end{bmatrix}\mapsto\begin{bmatrix}1\\1\end{bmatrix}.
\end{equation*}
But
\begin{equation*}
T\left(\begin{bmatrix}0\\1/2\end{bmatrix}\right)
=
\begin{bmatrix}1/2\\1/2\end{bmatrix},
\qquad
F\left(\begin{bmatrix}0\\1/2\end{bmatrix}\right)
=
\begin{bmatrix}1/4\\1/2\end{bmatrix}.
\end{equation*}
So the corner test misses the difference. A grid samples more points, including edge and interior points, and can show bending.
Subsection Local square-grid error
The Jacobian matrix is
\begin{equation*}
J_F(x,y)
=
\begin{bmatrix}
1&2y\\
0&1
\end{bmatrix}.
\end{equation*}
At
\begin{equation*}
\mathbf{a}
=
\begin{bmatrix}
0\\
1/2
\end{bmatrix},
\end{equation*}
we get
\begin{equation*}
J_F(\mathbf{a})
=
\begin{bmatrix}
1&1\\
0&1
\end{bmatrix}.
\end{equation*}
For
\begin{equation*}
\mathbf{h}
=
\begin{bmatrix}
h_1\\
h_2
\end{bmatrix},
\end{equation*}
we have
\begin{equation*}
F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})
=
\begin{bmatrix}
h_1+h_2+h_2^2\\
h_2
\end{bmatrix}
\end{equation*}
and
\begin{equation*}
J_F(\mathbf{a})\mathbf{h}
=
\begin{bmatrix}
h_1+h_2\\
h_2
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})-J_F(\mathbf{a})\mathbf{h}
=
\begin{bmatrix}
h_2^2\\
0
\end{bmatrix}.
\end{equation*}
If \(|h_2|\le r\text{,}\) then \(h_2^2\le r^2\text{,}\) so the error goes to zero as the square shrinks.
Subsection Code interpretation: square-grid arrays
The array \(P\) has shape \(2\times 5\text{.}\) Its columns are input points. The last column is
\begin{equation*}
\begin{bmatrix}
0\\
1/2
\end{bmatrix}.
\end{equation*}
The corresponding output column is
\begin{equation*}
\begin{bmatrix}
1/4\\
1/2
\end{bmatrix}.
\end{equation*}
The expression
np.vstack([x + y**2, y]) builds the two output components of \(F\) as rows of a new \(2\times 5\) array. This is not matrix multiplication because the first output component contains \(y^2\text{,}\) a nonlinear operation.Subsection Sigmoid derivative and local matrix
At
\begin{equation*}
\mathbf{a}
=
\begin{bmatrix}
0\\
0
\end{bmatrix},
\end{equation*}
we get
\begin{equation*}
\mathbf{s}_{\mathbf a}=W_1\mathbf a+\mathbf b_1
=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
\mathbf{u}_{\mathbf a}
=
\sigma(\mathbf{s}_{\mathbf a})
=
\begin{bmatrix}
1/2\\
1/2\\
1/2
\end{bmatrix},
\qquad
D_{\mathbf a}
=
\operatorname{diag}(1/4,1/4,1/4).
\end{equation*}
Thus
\begin{equation*}
W_2D_{\mathbf a}W_1
=
\begin{bmatrix}
1&0&-1/2\\
0&1&1/2
\end{bmatrix}
\begin{bmatrix}
1/4&0&0\\
0&1/4&0\\
0&0&1/4
\end{bmatrix}
\begin{bmatrix}
1&0\\
0&1\\
1&1
\end{bmatrix}
=
\begin{bmatrix}
1/8&-1/8\\
1/8&3/8
\end{bmatrix}.
\end{equation*}
Its determinant is
\begin{equation*}
(1/8)(3/8)-(-1/8)(1/8)=1/16,
\end{equation*}
so this local matrix is invertible.
For \(\mathbf c=\begin{bmatrix}1\\0\end{bmatrix}\text{,}\) we get
\begin{equation*}
\mathbf{s}_{\mathbf c}
=
\begin{bmatrix}
1\\
0\\
1
\end{bmatrix},
\qquad
D_{\mathbf c}=\operatorname{diag}(d,1/4,d),
\qquad
d=\sigma(1)(1-\sigma(1)).
\end{equation*}
Since \(d\ne 1/4\text{,}\) the first and third entries of \(D_{\mathbf c}\) differ from the corresponding entries of \(D_{\mathbf a}\text{.}\) The block is smooth, but the sigmoid derivative changes with the input, so the Jacobian matrix can change with the base point.
Subsection Code interpretation: sigmoid block arrays
The array \(P\) has shape \(2\times 3\text{.}\) Its columns are input points. Since \(W_1\) has shape \(3\times 2\text{,}\) the array \(S=W_1P+\mathbf{b}_1\) has shape \(3\times 3\text{.}\) The array \(U=\sigma(S)\) has the same shape.
The nonlinear operation is
U = sigmoid(S)
The first column of \(P\) is
\begin{equation*}
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
For this input,
\begin{equation*}
\mathbf{s}=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix},
\qquad
\mathbf{u}=
\begin{bmatrix}
1/2\\
1/2\\
1/2
\end{bmatrix}.
\end{equation*}
Therefore the first output column is
\begin{equation*}
W_2\mathbf{u}+\mathbf{b}_2
=
\begin{bmatrix}
1/4\\
3/4
\end{bmatrix}.
\end{equation*}
The full computation is not one matrix multiplication \(AP\) because the sigmoid step is nonlinear.
Subsection Softmax weights are data-dependent
The score vector is
\begin{equation*}
\mathbf{s}
=
K\mathbf{q}
=
\begin{bmatrix}
3q_1\\
3q_2\\
-3q_1-3q_2
\end{bmatrix}.
\end{equation*}
At \(\mathbf{q}=\mathbf{0}\text{,}\)
\begin{equation*}
\mathbf{s}
=
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
\end{equation*}
All three scores are equal, so the softmax weights are
\begin{equation*}
\boldsymbol{\alpha}(\mathbf{0})
=
\begin{bmatrix}
1/3\\
1/3\\
1/3
\end{bmatrix}.
\end{equation*}
Therefore
\begin{equation*}
F(\mathbf{0})
=
V^T
\begin{bmatrix}
1/3\\
1/3\\
1/3
\end{bmatrix}
=
\begin{bmatrix}
1/3\\
1/3
\end{bmatrix}.
\end{equation*}
Since a linear map must send \(\mathbf{0}\) to \(\mathbf{0}\text{,}\) the map \(F\) is not linear.
The softmax entries are positive and add to \(1\text{,}\) so \(F(\mathbf{q})\) is always a weighted average of the rows of \(V\text{.}\) The nonlinear step is the softmax rule that turns scores into weights.
Subsection Code interpretation: softmax square-grid arrays
The array \(P\) has shape \(2\times 4\text{.}\) Its columns are query vectors. Since \(K\) has shape \(3\times 2\text{,}\) the product
\begin{equation*}
S=KP
\end{equation*}
has shape \(3\times 4\text{.}\) The array
Alpha also has shape \(3\times 4\text{,}\) with one softmax weight vector in each column.Since \(V^T\) has shape \(2\times 3\text{,}\) the product
\begin{equation*}
Q=V^T\mathrm{Alpha}
\end{equation*}
has shape \(2\times 4\text{.}\) Its columns are output vectors.
The nonlinear operation is
Alpha = softmax_columns(S)
The full computation is not one matrix multiplication \(AP\) because the softmax step uses exponentials and column sums. The weights depend nonlinearly on the scores, and the scores depend on \(P\text{.}\)
Subsection Reading square-grid behavior
Map A is the nonlinear shear. It bends grid lines smoothly because of the \(y^2\) term, and it can agree with a linear shear on four corners while differing at edge and interior points.
Map B is the sigmoid block. Its local matrix has the form \(W_2D_{\mathbf a}W_1\text{,}\) where \(D_{\mathbf a}\) stores sigmoid derivatives. Its grid can bend and compress because those derivatives depend on the input.
Map C is the attention-style softmax map. The output is
\begin{equation*}
V^T\boldsymbol{\alpha},
\end{equation*}
where the entries of \(\boldsymbol{\alpha}\) are positive, add to \(1\text{,}\) and depend on the input. Therefore the output is a weighted average of the value vectors and stays inside the triangle they form.
Thus behaviors 1 and 4 match Map A, behaviors 2 and 5 match Map B, and behaviors 3 and 6 match Map C. Map A best shows why four corners alone can miss nonlinear behavior. Map B best shows the chain rule for Jacobian matrices. Map C best shows attention-style weighted averages. A square-grid visualization can show bending, compression, and changing local behavior that four corners alone may miss.
