Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 3.8 Exercises

Subsection OpenStax practice

Exercise tags. The tags below identify the primary outcomes and competency types for each exercise group. Individual problems may also involve earlier outcomes.
🔗

Subsubsection General functions and domains

Primary outcomes. U3-LO1. Competencies. P+R+C.
🔗
🔗

Subsubsection One-parameter maps and changes

Primary outcomes. U3-LO1, U3-LO2. Competencies. P+R.
🔗
🔗

Subsubsection Partial derivatives, gradients, Hessians, and Jacobian matrices

Primary outcomes. U3-LO2, U3-LO3. Competencies. P.
🔗
🔗

Subsubsection Local linearization and tangent planes

Primary outcomes. U3-LO4, U3-LO5. Competencies. P+C.
🔗
🔗

Subsubsection Chain rule as matrix multiplication

Primary outcomes. U3-LO6. Competencies. P+R.
🔗
🔗

Subsubsection Additional conceptual, geometric, and computational checkpoints

These checkpoints cover Unit 3 ideas not fully represented in the OpenStax practice: scalar-valued and vector-valued functions, domains, local language, derivatives, gradients, Hessians, Jacobian matrices, tangent planes, local linear prediction, the chain rule, and code interpretation. Solutions are collected in Appendix C.3.
🔗
Checkpoint 3.8.1. Function type and component functions.
Let
\begin{equation*} F(x,y)= \begin{bmatrix} x^2+y\\ e^{x-y}\\ x-y \end{bmatrix}. \end{equation*}
🔗
  1. Is \(F\) scalar-valued or vector-valued?
    🔗
    🔗
  2. What are the domain and codomain?
    🔗
    🔗
  3. Write the component functions \(F_1,F_2,F_3\text{.}\)
    🔗
    🔗
  4. Compute \(F(1,0)\text{.}\)
    🔗
    🔗
  5. What should be the shape of \(J_F(1,0)\text{?}\)
    🔗
    🔗
Tags. [U3-LO1, U3-LO3 | C+R | Core]
🔗
🔗
Checkpoint 3.8.3. One-parameter map and tangent vector.
Let
\begin{equation*} \mathbf r(t)= \begin{bmatrix} 1\\ 0\\ -1 \end{bmatrix} +t \begin{bmatrix} 2\\ -1\\ 3 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf r(2)\text{.}\)
    🔗
    🔗
  2. Compute \(\mathbf r'(t)\text{.}\)
    🔗
    🔗
  3. What does \(\mathbf r'(t)\) say about the curve?
    🔗
    🔗
  4. How does this connect to the Unit 2 parametric equation of a line?
    🔗
    🔗
Tags. [U3-LO2, U2-LO1 | P+R | Core]
🔗
🔗
Checkpoint 3.8.5. Jacobian matrix rows and columns.
Let
\begin{equation*} F(x,y)= \begin{bmatrix} x^2y\\ xe^y \end{bmatrix}, \qquad \mathbf a= \begin{bmatrix} 1\\ 0 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(J_F(x,y)\text{.}\)
    🔗
    🔗
  2. Compute \(J_F(\mathbf a)\text{.}\)
    🔗
    🔗
  3. What are the input and output dimensions?
    🔗
    🔗
  4. Which row corresponds to the second component of \(F\text{?}\)
    🔗
    🔗
  5. What do the two columns of \(J_F(\mathbf a)\) measure?
    🔗
    🔗
Tags. [U3-LO3, U3-LO5, U1-LO4 | P+C+R | Core]
🔗
🔗
Checkpoint 3.8.6. Tangent plane as a Unit 2 plane.
Let
\begin{equation*} f(x,y)=x^2+xy. \end{equation*}
🔗
  1. Compute \(f(1,2)\text{,}\) \(f_x(1,2)\text{,}\) and \(f_y(1,2)\text{.}\)
    🔗
    🔗
  2. Write the tangent plane to \(z=f(x,y)\) at \((1,2,f(1,2))\text{.}\)
    🔗
    🔗
  3. Rewrite the tangent plane in the form
    \begin{equation*} \mathbf n\cdot \begin{bmatrix} x-1\\ y-2\\ z-f(1,2) \end{bmatrix} =0. \end{equation*}
    🔗
    🔗
  4. Identify a normal vector \(\mathbf n\text{.}\)
    🔗
    🔗
  5. Which Unit 2 idea is being reused?
    🔗
    🔗
Tags. [U3-LO4, U2-LO1 | P+R | Core]
🔗
🔗
Checkpoint 3.8.7. Scalar local linear prediction.
Use the same function \(f(x,y)=x^2+xy\) at
\begin{equation*} \mathbf a= \begin{bmatrix} 1\\ 2 \end{bmatrix}, \qquad \mathbf h= \begin{bmatrix} 0.1\\ -0.2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(f(\mathbf a)\text{.}\)
    🔗
    🔗
  2. Compute \(\nabla f(\mathbf a)\text{.}\)
    🔗
    🔗
  3. Compute the local linear prediction \(f(\mathbf a)+\nabla f(\mathbf a)^T\mathbf h\text{.}\)
    🔗
    🔗
  4. Compute the actual value \(f(\mathbf a+\mathbf h)\text{.}\)
    🔗
    🔗
  5. Compare the actual value with the prediction.
    🔗
    🔗
Tags. [U3-LO4, U3-LO5 | P+C+R | Core]
🔗
🔗
Checkpoint 3.8.8. Vector-valued local linear prediction.
Use
\begin{equation*} F(x,y)= \begin{bmatrix} x^2y\\ xe^y \end{bmatrix}, \qquad \mathbf a= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad \mathbf h= \begin{bmatrix} 0.1\\ -0.2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(F(\mathbf a)\text{.}\)
    🔗
    🔗
  2. Use \(J_F(\mathbf a)\) from Jacobian matrix rows and columns.
    🔗
    🔗
  3. Compute \(F(\mathbf a)+J_F(\mathbf a)\mathbf h\text{.}\)
    🔗
    🔗
  4. Compute \(F(\mathbf a+\mathbf h)\text{.}\)
    🔗
    🔗
  5. What does the error vector measure?
    🔗
    🔗
Tags. [U3-LO5 | P+C+R | Core]
🔗
🔗
Checkpoint 3.8.9. Locally forgotten direction.
Suppose
\begin{equation*} J_F(\mathbf a)= \begin{bmatrix} 1&2\\ 2&4 \end{bmatrix}. \end{equation*}
🔗
  1. Find a nonzero vector \(\mathbf h\) such that \(J_F(\mathbf a)\mathbf h=\mathbf 0\text{.}\)
    🔗
    🔗
  2. What does this say about the local linear map?
    🔗
    🔗
  3. How does this connect to Unit 2 null spaces?
    🔗
    🔗
  4. Does this prove that \(F(\mathbf a+\mathbf h)=F(\mathbf a)\) exactly? Explain.
    🔗
    🔗
Tags. [U3-LO5, U2-LO3, U2-LO6 | C+R | Core]
🔗
🔗
Checkpoint 3.8.10. Chain-rule shape check.
Suppose
\begin{equation*} G:\mathbb R^2\to\mathbb R^3, \qquad F:\mathbb R^3\to\mathbb R^2. \end{equation*}
🔗
  1. What is the shape of \(J_G(\mathbf a)\text{?}\)
    🔗
    🔗
  2. What is the shape of \(J_F(G(\mathbf a))\text{?}\)
    🔗
    🔗
  3. What is the shape of \(J_{F\circ G}(\mathbf a)\text{?}\)
    🔗
    🔗
  4. Which product gives \(J_{F\circ G}(\mathbf a)\text{?}\)
    🔗
    🔗
  5. Why is the reverse order not correct?
    🔗
    🔗
Tags. [U3-LO6, U1-LO6 | C+R | Core]
🔗
🔗
Checkpoint 3.8.11. Reading a tiny sigmoid block.
Consider the code:
🔗
def sigmoid(t):
    return 1 / (1 + np.exp(-t))

s = W1 @ x + b1
u = sigmoid(s)
D = np.diag(u * (1 - u))
y = W2 @ u + b2
J_at_x = W2 @ D @ W1
  1. Which lines are affine maps?
    🔗
    🔗
  2. Which line contains the coordinatewise nonlinear step?
    🔗
    🔗
  3. Which variables represent vectors?
    🔗
    🔗
  4. What does \(D\) store?
    🔗
    🔗
  5. Which line computes the Jacobian matrix of the block at \(\mathbf{x}\text{?}\)
    🔗
    🔗
  6. Why is the whole block usually not one linear map?
    🔗
    🔗
Tags. [U3-LO7, U1-LO7 | C+R+T | Core]
🔗
🔗
Checkpoint 3.8.12. Corners are not enough.
Let
\begin{equation*} T\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y\\ y \end{bmatrix}, \qquad F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}. \end{equation*}
🔗
  1. Compute the images of the four corners of the unit square under \(T\text{.}\)
    🔗
    🔗
  2. Compute the images of the four corners of the unit square under \(F\text{.}\)
    🔗
    🔗
  3. Compute the images of \(\begin{bmatrix}0\\1/2\end{bmatrix}\) under both maps.
    🔗
    🔗
  4. Explain why the four corners do not prove that two maps are the same.
    🔗
    🔗
  5. Explain why a grid gives more information than the four corners.
    🔗
    🔗
Tags. [U3-LO1, U1-LO5 | C+R | Core]
🔗
🔗
Checkpoint 3.8.13. Local square-grid error.
Let
\begin{equation*} F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}, \qquad \mathbf{a} = \begin{bmatrix} 0\\ 1/2 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(J_F(x,y)\text{.}\)
    🔗
    🔗
  2. Compute \(J_F(\mathbf{a})\text{.}\)
    🔗
    🔗
  3. Let \(\mathbf{h}=\begin{bmatrix}h_1\\h_2\end{bmatrix}\text{.}\) Compute \(F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})\text{.}\)
    🔗
    🔗
  4. Compute \(J_F(\mathbf{a})\mathbf{h}\text{.}\)
    🔗
    🔗
  5. Compute the error
    \begin{equation*} F(\mathbf{a}+\mathbf{h})-F(\mathbf{a})-J_F(\mathbf{a})\mathbf{h}. \end{equation*}
    🔗
    🔗
  6. If \(|h_2|\le r\text{,}\) explain why the error goes to zero as \(r\to 0\text{.}\)
    🔗
    🔗
Tags. [U3-LO3, U3-LO5 | P+C+R | Core]
🔗
🔗
Checkpoint 3.8.14. Code interpretation: square-grid arrays.
Consider the code:
🔗
import numpy as np

def F(P):
    x = P[0, :]
    y = P[1, :]
    return np.vstack([x + y**2, y])

P = np.array([
    [0.0, 1.0, 1.0, 0.0, 0.0],
    [0.0, 0.0, 1.0, 1.0, 0.5],
])

Q = F(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are input points stored as rows or columns?
    🔗
    🔗
  3. Which column stores \(\begin{bmatrix}0\\1/2\end{bmatrix}\text{?}\)
    🔗
    🔗
  4. What is the corresponding output column of \(Q\text{?}\)
    🔗
    🔗
  5. What does the expression np.vstack([x + y**2, y]) construct?
    🔗
    🔗
  6. Why is this computation not a matrix multiplication?
    🔗
    🔗
Tags. [U3-LO1, U3-LO5, U1-LO8 | C+R+T | Core]
🔗
🔗
Checkpoint 3.8.15. Sigmoid derivative and local matrix.
Let
\begin{equation*} \sigma(t)=\frac{1}{1+e^{-t}}, \qquad \sigma'(t)=\sigma(t)(1-\sigma(t)), \end{equation*}
applied coordinatewise. Define
\begin{equation*} W_1= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}, \qquad \mathbf{b}_1= \begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix}, \end{equation*}
\begin{equation*} W_2= \begin{bmatrix} 1&0&-1/2\\ 0&1&1/2 \end{bmatrix}, \qquad \mathbf{b}_2= \begin{bmatrix} 0\\ 0 \end{bmatrix}. \end{equation*}
🔗
For
\begin{equation*} \mathbf{x} = \begin{bmatrix} x\\ y \end{bmatrix}, \end{equation*}
set
\begin{equation*} N(\mathbf{x})=W_2\sigma(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2. \end{equation*}
🔗
Let
\begin{equation*} \mathbf{a} = \begin{bmatrix} 0\\ 0 \end{bmatrix}. \end{equation*}
🔗
  1. Compute \(\mathbf{s}_{\mathbf a}=W_1\mathbf{a}+\mathbf{b}_1\text{.}\)
    🔗
    🔗
  2. Compute \(\mathbf{u}_{\mathbf a}=\sigma(\mathbf{s}_{\mathbf a})\text{.}\)
    🔗
    🔗
  3. Compute \(D_{\mathbf a}=\operatorname{diag}(\sigma'((\mathbf{s}_{\mathbf a})_1),\sigma'((\mathbf{s}_{\mathbf a})_2),\sigma'((\mathbf{s}_{\mathbf a})_3))\text{.}\)
    🔗
    🔗
  4. Compute the local matrix \(W_2D_{\mathbf a}W_1\text{.}\)
    🔗
    🔗
  5. Is this local matrix invertible?
    🔗
    🔗
  6. Let \(\mathbf{c}=\begin{bmatrix}1\\0\end{bmatrix}\text{.}\) Which entries of \(D_{\mathbf c}\) differ from entries of \(D_{\mathbf a}\text{?}\)
    🔗
    🔗
  7. Why can the local matrix change from one base point to another?
    🔗
    🔗
Tags. [U3-LO3, U3-LO5, U3-LO7 | C+R | Core]
🔗
🔗
Checkpoint 3.8.16. Code interpretation: sigmoid block arrays.
Consider the code:
🔗
import numpy as np

W1 = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0],
])

b1 = np.zeros((3, 1))

W2 = np.array([
    [1.0, 0.0, -0.5],
    [0.0, 1.0, 0.5],
])

b2 = np.zeros((2, 1))

def sigmoid(Z):
    return 1 / (1 + np.exp(-Z))

def N(P):
    S = W1 @ P + b1
    U = sigmoid(S)
    return W2 @ U + b2

P = np.array([
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
])

Q = N(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are input points stored as rows or columns?
    🔗
    🔗
  3. What is the shape of \(S=W_1P+\mathbf{b}_1\text{?}\)
    🔗
    🔗
  4. What is the shape of \(U=\sigma(S)\text{?}\)
    🔗
    🔗
  5. Which line applies the nonlinear operation?
    🔗
    🔗
  6. Compute the first column of \(Q\text{.}\)
    🔗
    🔗
  7. Why is the full computation not one matrix multiplication \(AP\text{?}\)
    🔗
    🔗
Tags. [U3-LO5, U3-LO7, U1-LO8 | C+R+T | Core]
🔗
🔗
Checkpoint 3.8.17. Softmax weights are data-dependent.
Let
\begin{equation*} K= \begin{bmatrix} 3&0\\ 0&3\\ -3&-3 \end{bmatrix}, \qquad V= \begin{bmatrix} 1&0\\ 0&1\\ 0&0 \end{bmatrix}. \end{equation*}
🔗
For
\begin{equation*} \mathbf{q} = \begin{bmatrix} q_1\\ q_2 \end{bmatrix}, \end{equation*}
define
\begin{equation*} \mathbf{s}=K\mathbf{q}, \qquad \boldsymbol{\alpha}(\mathbf{q})=\operatorname{softmax}(\mathbf{s}), \qquad F(\mathbf{q})=V^T\boldsymbol{\alpha}(\mathbf{q}). \end{equation*}
🔗
  1. Compute \(\mathbf{s}\) in terms of \(q_1\) and \(q_2\text{.}\)
    🔗
    🔗
  2. Compute \(\boldsymbol{\alpha}(\mathbf{0})\text{.}\)
    🔗
    🔗
  3. Compute \(F(\mathbf{0})\text{.}\)
    🔗
    🔗
  4. Explain why \(F\) is not linear.
    🔗
    🔗
  5. Explain why \(F(\mathbf{q})\) is always a weighted average of the rows of \(V\text{.}\)
    🔗
    🔗
  6. Which step is nonlinear?
    🔗
    🔗
Tags. [U3-LO1, U3-LO5, U3-LO7, U1-LO8 | C+R | Core]
🔗
🔗
Checkpoint 3.8.18. Code interpretation: softmax square-grid arrays.
Consider the code:
🔗
import numpy as np

K = np.array([
    [3.0, 0.0],
    [0.0, 3.0],
    [-3.0, -3.0],
])

V = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [0.0, 0.0],
])

def softmax_columns(S):
    S_shifted = S - S.max(axis=0, keepdims=True)
    E = np.exp(S_shifted)
    return E / E.sum(axis=0, keepdims=True)

def F(P):
    S = K @ P
    Alpha = softmax_columns(S)
    return V.T @ Alpha

P = np.array([
    [0.0, 1.0, 0.0, -1.0],
    [0.0, 0.0, 1.0, -1.0],
])

Q = F(P)
Q
  1. What is the shape of \(P\text{?}\)
    🔗
    🔗
  2. Are query vectors stored as rows or columns?
    🔗
    🔗
  3. What is the shape of \(S=KP\text{?}\)
    🔗
    🔗
  4. What is the shape of Alpha?
    🔗
    🔗
  5. What is the shape of \(Q\text{?}\)
    🔗
    🔗
  6. Which line applies the nonlinear operation?
    🔗
    🔗
  7. Why is the full computation not one matrix multiplication \(AP\text{?}\)
    🔗
    🔗
Tags. [U3-LO5, U3-LO7, U1-LO8 | C+R+T | Core]
🔗
🔗
Checkpoint 3.8.19. Reading square-grid behavior.
A square-grid diagram shows an input grid and its image under a map. Match each behavior with the most likely map type.
🔗
Behaviors:
🔗
  1. Grid lines bend smoothly because of a \(y^2\) term.
    🔗
    🔗
  2. Grid bends and compresses smoothly because sigmoid derivatives change with input.
    🔗
    🔗
  3. The output points stay inside a triangle formed by three value vectors.
    🔗
    🔗
  4. The four corners match a linear shear, but an edge midpoint does not.
    🔗
    🔗
  5. The local matrix has form \(W_2D_{\mathbf a}W_1\text{.}\)
    🔗
    🔗
  6. The nonlinear step comes from softmax weights depending on input.
    🔗
    🔗
Map types:
🔗
  1. Nonlinear shear
    \begin{equation*} F\left(\begin{bmatrix}x\\y\end{bmatrix}\right) = \begin{bmatrix} x+y^2\\ y \end{bmatrix}. \end{equation*}
    🔗
    🔗
  2. Two-output sigmoid block
    \begin{equation*} N(\mathbf{x})=W_2\sigma(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2. \end{equation*}
    🔗
    🔗
  3. Attention-style softmax map
    \begin{equation*} F_{\mathrm{att}}(\mathbf{q}) = V^T\operatorname{softmax}(K\mathbf{q}). \end{equation*}
    🔗
    🔗
  1. Match each behavior to Map A, Map B, or Map C.
    🔗
    🔗
  2. Which map best shows why four corners alone can miss nonlinear behavior?
    🔗
    🔗
  3. Which map best shows the chain rule for Jacobian matrices?
    🔗
    🔗
  4. Which map best shows attention-style weighted averages?
    🔗
    🔗
  5. What can a square-grid visualization show that four corners alone may miss?
    🔗
    🔗
Tags. [U3-LO1, U3-LO5, U3-LO7, U1-LO8 | C+R | Core]
🔗
🔗
🔗
🔗
🔗
PrevTopNext