Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 3.8 Exercises

Learning outcomes. The labels below identify the learning outcomes for each exercise group. Individual problems may also involve earlier outcomes.
🔗

Subsection General functions and domains

Learning outcomes. U3-LO1.
🔗
🔗

Subsection One-parameter maps and changes

Learning outcomes. U3-LO1, U3-LO2.
🔗
🔗

Subsection Partial derivatives, gradients, Hessians, and Jacobian matrices

Learning outcomes. U3-LO2, U3-LO3.
🔗
🔗

Subsection Local linearization and tangent planes

Learning outcomes. U3-LO4, U3-LO5.
🔗
🔗

Subsection Chain rule as matrix multiplication

Learning outcomes. U3-LO6.
🔗
🔗

Subsection Directional derivatives

Learning outcomes. U3-LO8.
🔗
🔗

Subsection Additional exercises

These exercises cover Unit 3 ideas not fully represented in the OpenStax practice: scalar-valued and vector-valued functions, domains, local language, derivatives, gradients, Hessians, Jacobian matrices, tangent planes, local linear prediction, the chain rule, directional derivatives, critical points, gradient descent, and code interpretation. Solutions are collected in Appendix C.3.
🔗

Additional exercise 3.8.1. Directional derivative as a dot product.

Suppose
\begin{equation*} \nabla f(\mathbf{a})= \begin{bmatrix} 6\\ 8 \end{bmatrix}. \end{equation*}
Let
\begin{equation*} \mathbf{v}_1= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf{v}_2= \begin{bmatrix} 0\\ 1 \end{bmatrix}, \qquad \mathbf{v}_3= \frac{1}{5} \begin{bmatrix} 3\\ 4 \end{bmatrix}, \qquad \mathbf{v}_4= -\frac{1}{5} \begin{bmatrix} 3\\ 4 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(D_{\mathbf{v}_i}f(\mathbf{a})\) for \(i=1,2,3,4\text{.}\)
    🔗
    🔗
  2. Which listed direction gives the largest directional derivative?
    🔗
    🔗
  3. Which listed direction gives the smallest directional derivative?
    🔗
    🔗
  4. Explain how this uses the Unit 1 dot product.
    🔗
    🔗
Learning outcomes. U3-LO8.
🔗
🔗

Additional exercise 3.8.2. Critical points are candidates.

Let
\begin{equation*} f(x,y)=x^2-y^2. \end{equation*}
🔗
  1. Find all critical points of \(f\text{.}\)
    🔗
    🔗
  2. Evaluate \(f(t,0)\) and \(f(0,t)\text{.}\)
    🔗
    🔗
  3. Explain why the critical point is not a local minimum or a local maximum.
    🔗
    🔗
Learning outcomes. U3-LO9.
🔗
🔗

Additional exercise 3.8.3. Reading a gradient descent loop.

Assume grad_f(x) computes \(\nabla f(\mathbf{x})\text{.}\) Consider the code:
🔗
x = x0
for k in range(num_steps):
    x = x - alpha * grad_f(x)
  1. What mathematical update rule is represented?
    🔗
    🔗
  2. Which quantity is the learning rate?
    🔗
    🔗
  3. What shape must grad_f(x) have?
    🔗
    🔗
  4. What would change if the minus sign were a plus sign?
    🔗
    🔗
  5. Does this code prove that a minimum has been found?
    🔗
    🔗
Learning outcomes. U3-LO8, U3-LO9.
🔗
🔗

Additional exercise 3.8.4. Learning-rate diagnosis.

For \(f(x)=x^2\text{,}\) start at \(x_0=4\) and use
\begin{equation*} x_{k+1}=x_k-\alpha f'(x_k). \end{equation*}
The table shows the loss values \(f(x_k)\) for three learning rates.
\begin{equation*} \begin{array}{c|ccc} k \amp \alpha=0.05 \amp \alpha=0.20 \amp \alpha=1.05\\ \hline 0 \amp 16.000 \amp 16.000 \amp 16.000\\ 1 \amp 12.960 \amp 5.760 \amp 19.360\\ 2 \amp 10.498 \amp 2.074 \amp 23.426\\ 3 \amp 8.503 \amp 0.746 \amp 28.345 \end{array} \end{equation*}
🔗
  1. Which learning rate is making slow but steady progress?
    🔗
    🔗
  2. Which learning rate is making faster useful progress?
    🔗
    🔗
  3. Which learning rate appears unstable?
    🔗
    🔗
  4. Does a decreasing loss table prove that the global minimum has been found?
    🔗
    🔗
Learning outcomes. U3-LO9.
🔗
🔗

Additional exercise 3.8.5. Function type and component functions.

Let
\begin{equation*} F(x,y)= \begin{bmatrix} x^2+y\\ e^{x-y}\\ x-y \end{bmatrix}. \end{equation*}
🔗
  1. Is \(F\) scalar-valued or vector-valued?
    🔗
    🔗
  2. What are the domain and codomain?
    🔗
    🔗
  3. Write the component functions \(F_1,F_2,F_3\text{.}\)
    🔗
    🔗
  4. Compute \(F(1,0)\text{.}\)
    🔗
    🔗
  5. What should be the shape of \(J_F(1,0)\text{?}\)
    🔗
    🔗
Learning outcomes. U3-LO1, U3-LO3.
🔗
🔗

Additional exercise 3.8.6. Domain, boundary, closed, and bounded.

Let
\begin{equation*} D=\{(x,y)\in\mathbb R^2:x\ge 0,\ y>x^2\}. \end{equation*}
🔗
  1. Explain why this is the domain of \(f(x,y)=\ln(y-x^2)+\sqrt{x}\text{.}\)
    🔗
    🔗
  2. Give one interior point of \(D\text{.}\)
    🔗
    🔗
  3. Give one boundary point of \(D\text{.}\)
    🔗
    🔗
  4. Is \(D\) closed?
    🔗
    🔗
  5. Is \(D\) bounded?
    🔗
    🔗
Learning outcomes. U3-LO1, U3-LO2.
🔗
🔗

Additional exercise 3.8.7. One-parameter map and tangent vector.

Let
\begin{equation*} \mathbf r(t)= \begin{bmatrix} 1\\ 0\\ -1 \end{bmatrix} +t \begin{bmatrix} 2\\ -1\\ 3 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf r(2)\text{.}\)
    🔗
    🔗
  2. Compute \(\mathbf r'(t)\text{.}\)
    🔗
    🔗
  3. What does \(\mathbf r'(t)\) say about the curve?
    🔗
    🔗
  4. How does this connect to the Unit 2 parametric equation of a line?
    🔗
    🔗
Learning outcomes. U3-LO2, U2-LO1.
🔗
🔗

Additional exercise 3.8.8. Gradient and Hessian.

Let
\begin{equation*} f(x,y)=x^2+xy+2y^2. \end{equation*}
🔗
  1. Compute \(f_x\) and \(f_y\text{.}\)
    🔗
    🔗
  2. Compute \(\nabla f(1,-1)\text{.}\)
    🔗
    🔗
  3. Compute \(H_f(x,y)\text{.}\)
    🔗
    🔗
  4. Is the Hessian symmetric?
    🔗
    🔗
  5. What does each entry of \(\nabla f(1,-1)\) measure?
    🔗
    🔗
Learning outcomes. U3-LO3.
🔗
🔗

Additional exercise 3.8.9. Jacobian matrix rows and columns.

Let
\begin{equation*} F(x,y)= \begin{bmatrix} x^2y\\ xe^y \end{bmatrix}, \qquad \mathbf a= \begin{bmatrix} 1\\ 0 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(J_F(x,y)\text{.}\)
    🔗
    🔗
  2. Compute \(J_F(\mathbf a)\text{.}\)
    🔗
    🔗
  3. What are the input and output dimensions?
    🔗
    🔗
  4. Which row corresponds to the second component of \(F\text{?}\)
    🔗
    🔗
  5. What do the two columns of \(J_F(\mathbf a)\) measure?
    🔗
    🔗
Learning outcomes. U3-LO3, U3-LO5, U1-LO4.
🔗
🔗

Additional exercise 3.8.10. Tangent plane as a Unit 2 plane.

Let
\begin{equation*} f(x,y)=x^2+xy. \end{equation*}
🔗
  1. Compute \(f(1,2)\text{,}\) \(f_x(1,2)\text{,}\) and \(f_y(1,2)\text{.}\)
    🔗
    🔗
  2. Write the tangent plane to \(z=f(x,y)\) at \((1,2,f(1,2))\text{.}\)
    🔗
    🔗
  3. Rewrite the tangent plane in the form
    \begin{equation*} \mathbf n\cdot \begin{bmatrix} x-1\\ y-2\\ z-f(1,2) \end{bmatrix} =0. \end{equation*}
    🔗
    🔗
  4. Identify a normal vector \(\mathbf n\text{.}\)
    🔗
    🔗
  5. Which Unit 2 idea is being reused?
    🔗
    🔗
Learning outcomes. U3-LO4, U2-LO1.
🔗
🔗

Additional exercise 3.8.11. Scalar local linear prediction.

Use the same function \(f(x,y)=x^2+xy\) at
\begin{equation*} \mathbf a= \begin{bmatrix} 1\\ 2 \end{bmatrix}, \qquad \mathbf h= \begin{bmatrix} 0.1\\ -0.2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(f(\mathbf a)\text{.}\)
    🔗
    🔗
  2. Compute \(\nabla f(\mathbf a)\text{.}\)
    🔗
    🔗
  3. Compute the local linear prediction \(f(\mathbf a)+\nabla f(\mathbf a)^T\mathbf h\text{.}\)
    🔗
    🔗
  4. Compute the actual value \(f(\mathbf a+\mathbf h)\text{.}\)
    🔗
    🔗
  5. Compare the actual value with the prediction.
    🔗
    🔗
Learning outcomes. U3-LO4, U3-LO5.
🔗
🔗

Additional exercise 3.8.12. Vector-valued local linear prediction.

Use
\begin{equation*} F(x,y)= \begin{bmatrix} x^2y\\ xe^y \end{bmatrix}, \qquad \mathbf a= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf h= \begin{bmatrix} 0.1\\ -0.2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(F(\mathbf a)\text{.}\)
    🔗
    🔗
  2. Use \(J_F(\mathbf a)\) from Jacobian matrix rows and columns.
    🔗
    🔗
  3. Compute \(F(\mathbf a)+J_F(\mathbf a)\mathbf h\text{.}\)
    🔗
    🔗
  4. Compute \(F(\mathbf a+\mathbf h)\text{.}\)
    🔗
    🔗
  5. What does the error vector measure?
    🔗
    🔗
Learning outcomes. U3-LO5.
🔗
🔗

Additional exercise 3.8.13. Locally forgotten direction.

Suppose
\begin{equation*} J_F(\mathbf a)= \begin{bmatrix} 1&2\\ 2&4 \end{bmatrix}. \end{equation*}
🔗
  1. Find a nonzero vector \(\mathbf h\) such that \(J_F(\mathbf a)\mathbf h=\mathbf 0\text{.}\)
    🔗
    🔗
  2. What does this say about the local linear map?
    🔗
    🔗
  3. How does this connect to Unit 2 null spaces?
    🔗
    🔗
  4. Does this prove that \(F(\mathbf a+\mathbf h)=F(\mathbf a)\) exactly? Explain.
    🔗
    🔗
Learning outcomes. U3-LO5, U2-LO3, U2-LO6.
🔗
🔗

Additional exercise 3.8.14. Chain-rule shape check.

Suppose
\begin{equation*} G:\mathbb R^2\to\mathbb R^3, \qquad F:\mathbb R^3\to\mathbb R^2. \end{equation*}
🔗
  1. What is the shape of \(J_G(\mathbf a)\text{?}\)
    🔗
    🔗
  2. What is the shape of \(J_F(G(\mathbf a))\text{?}\)
    🔗
    🔗
  3. What is the shape of \(J_{F\circ G}(\mathbf a)\text{?}\)
    🔗
    🔗
  4. Which product gives \(J_{F\circ G}(\mathbf a)\text{?}\)
    🔗
    🔗
  5. Why is the reverse order not correct?
    🔗
    🔗
Learning outcomes. U3-LO6, U1-LO6.
🔗
🔗

Additional exercise 3.8.15. Reading a tiny sigmoid block.

Consider the code:
🔗
def sigmoid(t):
    return 1 / (1 + np.exp(-t))

s = W1 @ x + b1
u = sigmoid(s)
D = np.diag(u * (1 - u))
y = W2 @ u + b2
J_at_x = W2 @ D @ W1
  1. Which lines are affine maps?
    🔗
    🔗
  2. Which line contains the coordinatewise nonlinear step?
    🔗
    🔗
  3. Which variables represent vectors?
    🔗
    🔗
  4. What does \(D\) store?
    🔗
    🔗
  5. Which line computes the Jacobian matrix of the block at \(\mathbf{x}\text{?}\)
    🔗
    🔗
  6. Why is the whole block usually not one linear map?
    🔗
    🔗
Learning outcomes. U3-LO7, U1-LO7.
🔗
🔗

Additional exercise 3.8.16. Corners are not enough.

Let
\begin{equation*} T\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y\\ y \end{bmatrix}, \qquad F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}. \end{equation*}
🔗
  1. Compute the images of the four corners of the unit square under \(T\text{.}\)
    🔗
    🔗
  2. Compute the images of the four corners of the unit square under \(F\text{.}\)
    🔗
    🔗
  3. Compute the images of \(\begin{bmatrix}0\\1/2\end{bmatrix}\) under both maps.
    🔗
    🔗
  4. Explain why the four corners do not prove that two maps are the same.
    🔗
    🔗
  5. Explain why a grid gives more information than the four corners.
    🔗
    🔗
Learning outcomes. U3-LO1, U1-LO5.
🔗
🔗

Additional exercise 3.8.17. Local square-grid error.

Let
\begin{equation*} F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}, \qquad \mathbf{a} = \begin{bmatrix} 0\\ 1/2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(J_F(x,y)\text{.}\)
    🔗
    🔗
  2. Compute \(J_F(\mathbf{a})\text{.}\)
    🔗
    🔗
  3. Let \(\mathbf{h}=\begin{bmatrix}h_1\\h_2\end{bmatrix}\text{.}\) Compute \(F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})\text{.}\)
    🔗
    🔗
  4. Compute \(J_F(\mathbf{a})\mathbf{h}\text{.}\)
    🔗
    🔗
  5. Compute the error
    \begin{equation*} F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})-J_F(\mathbf{a})\mathbf{h}. \end{equation*}
    🔗
    🔗
  6. If \(|h_2|\le r\text{,}\) explain why the error goes to zero as \(r\to 0\text{.}\)
    🔗
    🔗
Learning outcomes. U3-LO3, U3-LO5.
🔗
🔗

Additional exercise 3.8.18. Code interpretation: square-grid arrays.

Consider the code:
🔗
import numpy as np

def F(P):
    x = P[0, :]
    y = P[1, :]
    return np.vstack([x + y**2, y])

P = np.array([
    [0.0, 1.0, 1.0, 0.0, 0.0],
    [0.0, 0.0, 1.0, 1.0, 0.5],
])

Q = F(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are input points stored as rows or columns?
    🔗
    🔗
  3. Which column stores \(\begin{bmatrix}0\\1/2\end{bmatrix}\text{?}\)
    🔗
    🔗
  4. What is the corresponding output column of \(Q\text{?}\)
    🔗
    🔗
  5. What does the expression np.vstack([x + y**2, y]) construct?
    🔗
    🔗
  6. Why is this computation not a matrix multiplication?
    🔗
    🔗
Learning outcomes. U3-LO1, U3-LO5, U1-LO8.
🔗
🔗

Additional exercise 3.8.19. Sigmoid derivative and local matrix.

Let
\begin{equation*} \sigma(t)=\frac{1}{1+e^{-t}}, \qquad \sigma'(t)=\sigma(t)(1-\sigma(t)), \end{equation*}
applied coordinatewise. Define
\begin{equation*} W_1= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}, \qquad \mathbf{b}_1= \begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix}, \end{equation*}
\begin{equation*} W_2= \begin{bmatrix} 1&0&-1/2\\ 0&1&1/2 \end{bmatrix}, \qquad \mathbf{b}_2= \begin{bmatrix} 0\\ 0 \end{bmatrix}. \end{equation*}
🔗
For
\begin{equation*} \mathbf{x} = \begin{bmatrix} x\\ y \end{bmatrix}, \end{equation*}
set
\begin{equation*} N(\mathbf{x})=W_2\sigma(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2. \end{equation*}
🔗
Let
\begin{equation*} \mathbf{a} = \begin{bmatrix} 0\\ 0 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf{s}_{\mathbf a}=W_1\mathbf{a}+\mathbf{b}_1\text{.}\)
    🔗
    🔗
  2. Compute \(\mathbf{u}_{\mathbf a}=\sigma(\mathbf{s}_{\mathbf a})\text{.}\)
    🔗
    🔗
  3. Compute \(D_{\mathbf a}=\operatorname{diag}(\sigma'((\mathbf{s}_{\mathbf a})_1),\sigma'((\mathbf{s}_{\mathbf a})_2),\sigma'((\mathbf{s}_{\mathbf a})_3))\text{.}\)
    🔗
    🔗
  4. Compute the local matrix \(W_2D_{\mathbf a}W_1\text{.}\)
    🔗
    🔗
  5. Is this local matrix invertible?
    🔗
    🔗
  6. Let \(\mathbf{c}=\begin{bmatrix}1\\0\end{bmatrix}\text{.}\) Which entries of \(D_{\mathbf c}\) differ from entries of \(D_{\mathbf a}\text{?}\)
    🔗
    🔗
  7. Why can the local matrix change from one base point to another?
    🔗
    🔗
Learning outcomes. U3-LO3, U3-LO5, U3-LO7.
🔗
🔗

Additional exercise 3.8.20. Code interpretation: sigmoid block arrays.

Consider the code:
🔗
import numpy as np

W1 = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0],
])

b1 = np.zeros((3, 1))

W2 = np.array([
    [1.0, 0.0, -0.5],
    [0.0, 1.0, 0.5],
])

b2 = np.zeros((2, 1))

def sigmoid(Z):
    return 1 / (1 + np.exp(-Z))

def N(P):
    S = W1 @ P + b1
    U = sigmoid(S)
    return W2 @ U + b2

P = np.array([
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
])

Q = N(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are input points stored as rows or columns?
    🔗
    🔗
  3. What is the shape of \(S=W_1P+\mathbf{b}_1\text{?}\)
    🔗
    🔗
  4. What is the shape of \(U=\sigma(S)\text{?}\)
    🔗
    🔗
  5. Which line applies the nonlinear operation?
    🔗
    🔗
  6. Compute the first column of \(Q\text{.}\)
    🔗
    🔗
  7. Why is the full computation not one matrix multiplication \(AP\text{?}\)
    🔗
    🔗
Learning outcomes. U3-LO5, U3-LO7, U1-LO8.
🔗
🔗

Additional exercise 3.8.21. Softmax weights are data-dependent.

Let
\begin{equation*} K= \begin{bmatrix} 3&0\\ 0&3\\ -3&-3 \end{bmatrix}, \qquad V= \begin{bmatrix} 1&0\\ 0&1\\ 0&0 \end{bmatrix}. \end{equation*}
🔗
For
\begin{equation*} \mathbf{q} = \begin{bmatrix} q_1\\ q_2 \end{bmatrix}, \end{equation*}
define
\begin{equation*} \mathbf{s}=K\mathbf{q}, \qquad \boldsymbol{\alpha}(\mathbf{q})=\operatorname{softmax}(\mathbf{s}), \qquad F(\mathbf{q})=V^T\boldsymbol{\alpha}(\mathbf{q}). \end{equation*}
🔗
  1. Compute \(\mathbf{s}\) in terms of \(q_1\) and \(q_2\text{.}\)
    🔗
    🔗
  2. Compute \(\boldsymbol{\alpha}(\mathbf{0})\text{.}\)
    🔗
    🔗
  3. Compute \(F(\mathbf{0})\text{.}\)
    🔗
    🔗
  4. Explain why \(F\) is not linear.
    🔗
    🔗
  5. Explain why \(F(\mathbf{q})\) is always a weighted average of the rows of \(V\text{.}\)
    🔗
    🔗
  6. Which step is nonlinear?
    🔗
    🔗
Learning outcomes. U3-LO1, U3-LO5, U3-LO7, U1-LO8.
🔗
🔗

Additional exercise 3.8.22. Code interpretation: softmax square-grid arrays.

Consider the code:
🔗
import numpy as np

K = np.array([
    [3.0, 0.0],
    [0.0, 3.0],
    [-3.0, -3.0],
])

V = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [0.0, 0.0],
])

def softmax_columns(S):
    S_shifted = S - S.max(axis=0, keepdims=True)
    E = np.exp(S_shifted)
    return E / E.sum(axis=0, keepdims=True)

def F(P):
    S = K @ P
    Alpha = softmax_columns(S)
    return V.T @ Alpha

P = np.array([
    [0.0, 1.0, 0.0, -1.0],
    [0.0, 0.0, 1.0, -1.0],
])

Q = F(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are query vectors stored as rows or columns?
    🔗
    🔗
  3. What is the shape of \(S=KP\text{?}\)
    🔗
    🔗
  4. What is the shape of Alpha?
    🔗
    🔗
  5. What is the shape of \(Q\text{?}\)
    🔗
    🔗
  6. Which line applies the nonlinear operation?
    🔗
    🔗
  7. Why is the full computation not one matrix multiplication \(AP\text{?}\)
    🔗
    🔗
Learning outcomes. U3-LO5, U3-LO7, U1-LO8.
🔗
🔗

Additional exercise 3.8.23. Reading square-grid behavior.

A square-grid diagram shows an input grid and its image under a map. Match each behavior with the most likely map type.
🔗
Behaviors:
🔗
  1. Grid lines bend smoothly because of a \(y^2\) term.
    🔗
    🔗
  2. Grid bends and compresses smoothly because sigmoid derivatives change with input.
    🔗
    🔗
  3. The output points stay inside a triangle formed by three value vectors.
    🔗
    🔗
  4. The four corners match a linear shear, but an edge midpoint does not.
    🔗
    🔗
  5. The local matrix has form \(W_2D_{\mathbf a}W_1\text{.}\)
    🔗
    🔗
  6. The nonlinear step comes from softmax weights depending on input.
    🔗
    🔗
Map types:
🔗
  1. Nonlinear shear
    \begin{equation*} F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}. \end{equation*}
    🔗
    🔗
  2. Two-output sigmoid block
    \begin{equation*} N(\mathbf{x})=W_2\sigma(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2. \end{equation*}
    🔗
    🔗
  3. Attention-style softmax map
    \begin{equation*} F_{\mathrm{att}}(\mathbf{q}) = V^T\operatorname{softmax}(K\mathbf{q}). \end{equation*}
    🔗
    🔗
  1. Match each behavior to Map A, Map B, or Map C.
    🔗
    🔗
  2. Which map best shows why four corners alone can miss nonlinear behavior?
    🔗
    🔗
  3. Which map best shows the chain rule for Jacobian matrices?
    🔗
    🔗
  4. Which map best shows attention-style weighted averages?
    🔗
    🔗
  5. What can a square-grid visualization show that four corners alone may miss?
    🔗
    🔗
Learning outcomes. U3-LO1, U3-LO5, U3-LO7, U1-LO8.
🔗
🔗

Additional exercise 3.8.24. Final-layer rank-one update.

Let
\begin{equation*} \mathbf{g}= \begin{bmatrix} 1\\ -2\\ 3 \end{bmatrix}, \qquad \mathbf{h}= \begin{bmatrix} 4\\ 0 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf{g}\mathbf{h}^T\text{.}\)
    🔗
    🔗
  2. What is the shape of \(\mathbf{g}\mathbf{h}^T\text{?}\)
    🔗
    🔗
  3. Explain why \(\mathbf{g}\mathbf{h}^T\) has rank at most one.
    🔗
    🔗
  4. Interpret the update \(W_{\mathrm{new}}=W-\alpha \mathbf{g}\mathbf{h}^T\text{.}\)
    🔗
    🔗
  5. In NumPy, why is np.outer(g, h) appropriate when g and h are stored as one-dimensional arrays?
    🔗
    🔗
Learning outcomes. U3-LO6, U3-LO9, U2-LO3.
🔗
🔗
🔗
🔗
PrevTopNext