Coding recap

Section 3.7 Coding recap

Subsection Linked notebook

Run Lab U3: Jacobian matrices and local linearization

sebroc.github.io/MATH345-Course-Materials/labs/#lab-u3

. The main sequence practices symbolic Jacobian matrices, evaluating a Jacobian matrix at a point, local linear prediction, square-grid visualizations, checking Jacobian matrix shapes in the chain rule, and short nonlinear model-block computations.

🔗

Directional derivatives and gradient descent are revisited computationally at the beginning of Lab U5.

🔗

For a quick reference on array shapes, matrix products, numerical checks, and SymPy symbolic commands, see the programming appendix sections B.2, B.4, B.9, and B.11.

🔗

Subsection Review activities

Activity 3.7.1. Reading a symbolic Jacobian matrix (U3-LO1, U3-LO3).

x, y = sp.symbols("x y")
F = sp.Matrix([x*y, x**2 + y])
J = F.jacobian([x, y])
J

Output:

🔗

\begin{equation*} J= \begin{bmatrix} y\amp x\\ 2x\amp 1 \end{bmatrix}. \end{equation*}

What are the component functions of F?
🔗

🔗
What is the shape of J?
🔗

🔗
What does the first column of J measure?
🔗

🔗

Solution.

The component functions are \(F_1(x,y)=xy\) and \(F_2(x,y)=x^2+y\text{.}\) The Jacobian matrix is \(\begin{bmatrix}y&x\\2x&1\end{bmatrix}\text{,}\) so it is \(2\times 2\text{.}\) Its first column measures how the two output components change with respect to the first input variable \(x\text{.}\)

🔗

Activity 3.7.2. Actual output versus local prediction (U3-LO5).

def F(v):
    x, y = v
    return np.array([x**2 + y, x - y**2])

a = np.array([1.0, 2.0])
h = np.array([0.1, -0.2])
J_at_a = np.array([[2.0, 1.0],
                   [1.0, -4.0]])

actual = F(a + h)
linear = F(a) + J_at_a @ h
actual, linear, actual - linear

Output:

🔗

\begin{align*} \texttt{actual}\amp=\begin{bmatrix}3.01\\-2.14\end{bmatrix},\\ \texttt{linear}\amp=\begin{bmatrix}3.00\\-2.10\end{bmatrix},\\ \texttt{actual}-\texttt{linear}\amp=\begin{bmatrix}0.01\\-0.04\end{bmatrix}. \end{align*}

Which output is the true nonlinear value?
🔗

🔗
Which output is the local linear prediction?
🔗

🔗
What does actual - linear measure?
🔗

🔗

Solution.

The true value is actual, which is [3.01, -2.14]. The local linear prediction is linear, which is [3.0, -2.1]. The difference actual - linear is [0.01, -0.04], the local approximation error for this input change.

🔗

Activity 3.7.3. Reading Jacobian matrix columns (U3-LO3, U3-LO5).

J_at_a = np.array([[2.0, -1.0, 0.0],
                   [0.0,  3.0, 4.0]])

e1_change = np.array([0.1, 0.0, 0.0])
J_at_a[:, 0], J_at_a @ e1_change

Output of J_at_a[:, 0]:

🔗

\begin{equation*} \begin{bmatrix}2.0\\0.0\end{bmatrix}. \end{equation*}

Output of J_at_a @ e1_change:

🔗

\begin{equation*} \begin{bmatrix}0.2\\0.0\end{bmatrix}. \end{equation*}

What are the input and output dimensions?
🔗

🔗
What does J_at_a[:, 0] select?
🔗

🔗
What output change is predicted for e1_change?
🔗

🔗

Solution.

The input dimension is \(3\) and the output dimension is \(2\text{.}\) The slice J_at_a[:, 0] selects the first column of the Jacobian matrix, [2.0, 0.0]. For the input change [0.1, 0.0, 0.0], the predicted output change is [0.2, 0.0].

🔗

Activity 3.7.4. Chain-rule shape check (U3-LO6).

Jg = np.ones((3, 2))
Jf_at_g = np.ones((4, 3))

Jf_at_g @ Jg

Output of Jf_at_g @ Jg:

🔗

\begin{equation*} \begin{bmatrix} 3.0\amp3.0\\ 3.0\amp3.0\\ 3.0\amp3.0\\ 3.0\amp3.0 \end{bmatrix}. \end{equation*}

Which map is applied first in the composition?
🔗

🔗
What are the input and output dimensions of the composition?
🔗

🔗
What is the shape of the product?
🔗

🔗

Solution.

The map whose Jacobian matrix is Jg is applied first. Since Jg is \(3\times 2\text{,}\) it takes two input directions to three intermediate directions; Jf_at_g then takes those three directions to four output directions. The product has shape \(4\times 2\text{,}\) and the displayed output is a \(4\times 2\) array whose entries are all \(3\text{.}\)

🔗

Activity 3.7.5. Reading a tiny sigmoid block (U3-LO6, U3-LO7).

def sigmoid(t):
    return 1 / (1 + np.exp(-t))

W1 = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0],
])

W2 = np.array([
    [1.0, 0.0, -0.5],
    [0.0, 1.0,  0.5],
])

b1 = np.zeros(3)
b2 = np.zeros(2)

x = np.array([0.0, 0.0])

s = W1 @ x + b1
u = sigmoid(s)
D = np.diag(u * (1 - u))
y = W2 @ u + b2
J_at_x = W2 @ D @ W1

s, u, y, J_at_x

Output:

🔗

\begin{align*} \mathbf s\amp=\begin{bmatrix}0.0\\0.0\\0.0\end{bmatrix},\\ \mathbf u\amp=\begin{bmatrix}0.5\\0.5\\0.5\end{bmatrix},\\ \mathbf y\amp=\begin{bmatrix}0.25\\0.75\end{bmatrix},\\ J_{\mathbf x}\amp=\begin{bmatrix}0.125\amp-0.125\\0.125\amp0.375\end{bmatrix}. \end{align*}

Which lines contain affine maps?
🔗

🔗
Which line contains the coordinatewise nonlinear step?
🔗

🔗
What do s, u, and y represent?
🔗

🔗
Which line forms the diagonal derivative matrix?
🔗

🔗
Which line computes the Jacobian matrix of the block at x?
🔗

🔗
What is the shape of J_at_x?
🔗

🔗

Solution.

The lines s = W1 @ x + b1 and y = W2 @ u + b2 are affine maps. The line u = sigmoid(s) is the coordinatewise nonlinear step.

🔗

The vector s is the pre-activation vector. The vector u is the hidden vector after applying sigmoid. The vector y is the output vector.

🔗

The line D = np.diag(u * (1 - u)) forms the diagonal matrix of sigmoid derivatives at this input, since \(\sigma'(t)=\sigma(t)(1-\sigma(t))\text{.}\)

🔗

The line J_at_x = W2 @ D @ W1 computes the Jacobian matrix of the block at x by the chain rule for Jacobian matrices. The shape of J_at_x is \(2\times 2\text{:}\) the input has two coordinates and the output has two coordinates.

🔗

Activity 3.7.6. Reading a gradient descent loop (U3-LO8, U3-LO9).

Assume grad_f(x) computes \(\nabla f(\mathbf{x})\text{.}\) Consider the code:

🔗

x = x0
for k in range(num_steps):
    x = x - alpha * grad_f(x)

What mathematical update rule is represented?
🔗

🔗
Which quantity is the learning rate?
🔗

🔗
What shape must grad_f(x) have?
🔗

🔗
What would change if the minus sign were a plus sign?
🔗

🔗
Does this code prove that a minimum has been found?
🔗

🔗

Solution.

The loop represents the update

\begin{equation*} \mathbf{x}_{k+1}=\mathbf{x}_k-\alpha\nabla f(\mathbf{x}_k). \end{equation*}

The learning rate is alpha. The vector grad_f(x) must have the same shape as x, since the update subtracts one vector from another. If the minus sign were replaced by a plus sign, the update would move in the direction of steepest increase instead of steepest decrease. The loop does not prove that a minimum has been found. It only describes the repeated update; convergence depends on the function, starting point, learning rate, and stopping rule.

🔗

Activity 3.7.7. Diagnosing learning rates (U3-LO9).

For \(f(x)=x^2\text{,}\) start at \(x_0=4\) and use the update

\begin{equation*} x_{k+1}=x_k-\alpha f'(x_k). \end{equation*}

The table shows the loss values \(f(x_k)\) for three learning rates.

🔗

\(k\) 🔗	\(\alpha = 0.05\) 🔗	\(\alpha = 0.20\) 🔗	\(\alpha = 1.05\) 🔗
0 🔗	16.000 🔗	16.000 🔗	16.000 🔗
1 🔗	12.960 🔗	5.760 🔗	19.360 🔗
2 🔗	10.498 🔗	2.074 🔗	23.426 🔗
3 🔗	8.503 🔗	0.746 🔗	28.345 🔗

Which learning rate is making slow but steady progress?
🔗

🔗
Which learning rate is making faster useful progress?
🔗

🔗
Which learning rate appears unstable?
🔗

🔗
Does a decreasing loss table prove that the global minimum has been found?
🔗

🔗

Solution.

The learning rate \(\alpha=0.05\) is making slow but steady progress. The learning rate \(\alpha=0.20\) is making faster useful progress. The learning rate \(\alpha=1.05\) appears unstable because the loss is increasing. A decreasing loss table does not prove that the global minimum has been found. It only shows what happened for the displayed iterates.

🔗

Activity 3.7.8. Reading a next-token loss (U3-LO8, U3-LO9).

A language model produces scores for possible next tokens, converts those scores into probabilities, and uses a loss to measure the probability assigned to the observed next token. A simplified training objective for a sequence \(t_1,\ldots,t_L\) can be written

\begin{equation*} \mathcal L(\boldsymbol{\theta}) = -\sum_{i=1}^{L-1} \log p_{\boldsymbol{\theta}} \bigl(t_{i+1}\mid t_1,\ldots,t_i\bigr). \end{equation*}

The parameter vector \(\boldsymbol{\theta}\) may collect many matrices and bias vectors.

🔗

A gradient descent step has the form

\begin{equation*} \boldsymbol{\theta}_{k+1} = \boldsymbol{\theta}_k -\alpha\nabla\mathcal L(\boldsymbol{\theta}_k). \end{equation*}

🔗

What are the parameters?
🔗

🔗
What quantity is being minimized?
🔗

🔗
Which symbol is the learning rate?
🔗

🔗
What does the gradient point toward?
🔗

🔗
Why is this an optimization problem, even though we are not studying full model training?
🔗

🔗
Why is this not automatically a least-squares problem?
🔗

🔗

Solution.

The parameters are collected in \(\boldsymbol{\theta}\text{.}\) The objective \(\mathcal L(\boldsymbol{\theta})\) is the loss. The learning rate is \(\alpha\text{.}\) The gradient \(\nabla\mathcal L(\boldsymbol{\theta}_k)\) gives the local direction of steepest increase, so the negative gradient is used for descent. This is an optimization problem because training means adjusting parameters to reduce a loss.

🔗

Nothing in the displayed objective has the form

\begin{equation*} \|A\mathbf c-\mathbf y\|^2 \end{equation*}

with a fixed matrix \(A\) and a single trained coefficient vector \(\mathbf c\text{.}\) It is a general differentiable loss optimized by gradient descent.

🔗

Activity 3.7.9. Gradient of a final linear layer (U3-LO6, U3-LO9, U2-LO3).

Suppose a fixed hidden representation is

\begin{equation*} \mathbf h\in\mathbb R^d, \end{equation*}

and a final linear layer computes

\begin{equation*} \boldsymbol{\ell}=W\mathbf h, \qquad W\in\mathbb R^{m\times d}. \end{equation*}

Let

\begin{equation*} \mathbf g=\nabla_{\boldsymbol{\ell}}\mathcal L \in\mathbb R^m \end{equation*}

be the loss gradient with respect to the output scores.

🔗

For

\begin{equation*} \ell_i=\sum_{j=1}^d W_{ij}h_j, \end{equation*}

use the chain rule to show that

\begin{equation*} \frac{\partial\mathcal L}{\partial W_{ij}} = g_i h_j. \end{equation*}

🔗

🔗
Conclude that

\begin{equation*} \nabla_W\mathcal L=\mathbf g\mathbf h^T. \end{equation*}

🔗

🔗
What is the shape of this matrix?
🔗

🔗
Why does it have rank at most one?
🔗

🔗
Interpret the update

\begin{equation*} W_{\mathrm{new}} = W-\alpha\mathbf g\mathbf h^T. \end{equation*}

🔗

🔗

Solution.

Since

\begin{equation*} \ell_k=\sum_{j=1}^d W_{kj}h_j, \end{equation*}

we have

\begin{equation*} \frac{\partial \ell_k}{\partial W_{ij}} = \begin{cases} h_j,&k=i,\\ 0,&k\neq i. \end{cases} \end{equation*}

🔗

The chain rule gives

\begin{equation*} \frac{\partial\mathcal L}{\partial W_{ij}} = \sum_{k=1}^m \frac{\partial\mathcal L}{\partial\ell_k} \frac{\partial\ell_k}{\partial W_{ij}} = g_i h_j. \end{equation*}

Therefore

\begin{equation*} \nabla_W\mathcal L=\mathbf g\mathbf h^T. \end{equation*}

🔗

If \(\mathbf g\in\mathbb R^m\) and \(\mathbf h\in\mathbb R^d\text{,}\) then this outer product has shape \(m\times d\text{.}\) Every column is a scalar multiple of \(\mathbf g\text{,}\) so its rank is at most one. The update changes \(W\) in the negative-gradient direction for this training example.

🔗

Activity 3.7.10. The same final-layer update in code (U3-LO6, U3-LO9, U2-LO3).

G = np.outer(g, h)
W_new = W - alpha * G

Which earlier activity gives the formula represented by G?
🔗

🔗
What is the shape of G?
🔗

🔗
Why is np.outer(g, h) used here?
🔗

🔗
What mathematical update is represented by the second line?
🔗

🔗
Why is this connected to rank?
🔗

🔗

Solution.

By Gradient of a final linear layer,

\begin{equation*} G=\mathbf g\mathbf h^T=\nabla_W\mathcal L. \end{equation*}

🔗

If \(\mathbf g\) has length \(m\) and \(\mathbf h\) has length \(d\text{,}\) then G has shape \(m\times d\text{.}\) The command np.outer(g, h) forms the outer product even when the two vectors are stored as one-dimensional NumPy arrays. The second line represents

\begin{equation*} W_{\mathrm{new}} = W-\alpha\nabla_W\mathcal L. \end{equation*}

🔗

The gradient matrix has rank at most one for this single training example. This does not imply that \(W\) or \(W_{\mathrm{new}}\) has rank one.

🔗

Prev Top Next