Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 3.7 Applications and computation recap

Subsection Review checkpoints

Activity 3.7.1. Reading a symbolic Jacobian matrix. 

x, y = sp.symbols("x y")
F = sp.Matrix([x*y, x**2 + y])
J = F.jacobian([x, y])
J
Output:
🔗
Matrix([
[y,   x],
[2*x, 1]
])
  1. What are the component functions of F?
    🔗
    🔗
  2. What is the shape of J?
    🔗
    🔗
  3. What does the first column of J measure?
    🔗
    🔗
Solution.
The component functions are \(F_1(x,y)=xy\) and \(F_2(x,y)=x^2+y\text{.}\) The Jacobian matrix is \(\begin{bmatrix}y&x\\2x&1\end{bmatrix}\text{,}\) so it is \(2\times 2\text{.}\) Its first column measures how the two output components change with respect to the first input variable \(x\text{.}\)
🔗
🔗
🔗

Activity 3.7.2. Actual output versus local prediction. 

def F(v):
    x, y = v
    return np.array([x**2 + y, x - y**2])

a = np.array([1.0, 2.0])
h = np.array([0.1, -0.2])
J_at_a = np.array([[2.0, 1.0],
                   [1.0, -4.0]])

actual = F(a + h)
linear = F(a) + J_at_a @ h
actual, linear, actual - linear
Output:
🔗
(array([ 3.01, -2.14]),
 array([ 3. , -2.1]),
 array([ 0.01, -0.04]))
  1. Which output is the true nonlinear value?
    🔗
    🔗
  2. Which output is the local linear prediction?
    🔗
    🔗
  3. What does actual - linear measure?
    🔗
    🔗
Solution.
The true value is actual, which is [3.01, -2.14]. The local linear prediction is linear, which is [3.0, -2.1]. The difference actual - linear is [0.01, -0.04], the local approximation error for this input change.
🔗
🔗
🔗

Activity 3.7.3. Reading Jacobian matrix columns. 

J_at_a = np.array([[2.0, -1.0, 0.0],
                   [0.0,  3.0, 4.0]])

e1_change = np.array([0.1, 0.0, 0.0])
J_at_a[:, 0], J_at_a @ e1_change
Output:
🔗
(array([2., 0.]), array([0.2, 0. ]))
  1. What are the input and output dimensions?
    🔗
    🔗
  2. What does J_at_a[:, 0] select?
    🔗
    🔗
  3. What output change is predicted for e1_change?
    🔗
    🔗
Solution.
The input dimension is \(3\) and the output dimension is \(2\text{.}\) The slice J_at_a[:, 0] selects the first column of the Jacobian matrix, [2.0, 0.0]. For the input change [0.1, 0.0, 0.0], the predicted output change is [0.2, 0.0].
🔗
🔗
🔗

Activity 3.7.4. Chain-rule shape check. 

Jg = np.ones((3, 2))
Jf_at_g = np.ones((4, 3))

Jf_at_g @ Jg
Output:
🔗
array([[3., 3.],
       [3., 3.],
       [3., 3.],
       [3., 3.]])
  1. Which map is applied first in the composition?
    🔗
    🔗
  2. What are the input and output dimensions of the composition?
    🔗
    🔗
  3. What is the shape of the product?
    🔗
    🔗
Solution.
The map whose Jacobian matrix is Jg is applied first. Since Jg is \(3\times 2\text{,}\) it takes two input directions to three intermediate directions; Jf_at_g then takes those three directions to four output directions. The product has shape \(4\times 2\text{,}\) and the displayed output is a \(4\times 2\) array whose entries are all \(3\text{.}\)
🔗
🔗
🔗

Activity 3.7.5. Reading a tiny sigmoid block. 

def sigmoid(t):
    return 1 / (1 + np.exp(-t))

W1 = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0],
])

W2 = np.array([
    [1.0, 0.0, -0.5],
    [0.0, 1.0,  0.5],
])

b1 = np.zeros(3)
b2 = np.zeros(2)

x = np.array([0.0, 0.0])

s = W1 @ x + b1
u = sigmoid(s)
D = np.diag(u * (1 - u))
y = W2 @ u + b2
J_at_x = W2 @ D @ W1

s, u, y, J_at_x
Output:
🔗
(array([0., 0., 0.]),
 array([0.5, 0.5, 0.5]),
 array([0.25, 0.75]),
 array([[ 0.125, -0.125],
        [ 0.125,  0.375]]))
  1. Which lines contain affine maps?
    🔗
    🔗
  2. Which line contains the coordinatewise nonlinear step?
    🔗
    🔗
  3. What do s, u, and y represent?
    🔗
    🔗
  4. Which line forms the diagonal derivative matrix?
    🔗
    🔗
  5. Which line computes the Jacobian matrix of the block at x?
    🔗
    🔗
  6. What is the shape of J_at_x?
    🔗
    🔗
Solution.
The lines s = W1 @ x + b1 and y = W2 @ u + b2 are affine maps. The line u = sigmoid(s) is the coordinatewise nonlinear step.
🔗
The vector s is the pre-activation vector. The vector u is the hidden vector after applying sigmoid. The vector y is the output vector.
🔗
The line D = np.diag(u * (1 - u)) forms the diagonal matrix of sigmoid derivatives at this input, since \(\sigma'(t)=\sigma(t)(1-\sigma(t))\text{.}\)
🔗
The line J_at_x = W2 @ D @ W1 computes the Jacobian matrix of the block at x by the chain rule for Jacobian matrices. The shape of J_at_x is \(2\times 2\text{:}\) the input has two coordinates and the output has two coordinates.
🔗
🔗
🔗
Interpretation check. Given a Jacobian matrix \(J_F(\mathbf{a})\text{,}\) identify the input dimension, output dimension, and what each column measures. Tags. [U3-LO3, U3-LO5 | C+R+T | Core]
🔗
Exam-style check. Given a \(3\times 2\) Jacobian matrix \(J_F(\mathbf{a})\text{,}\) identify the input dimension, output dimension, and compute the local change \(J_F(\mathbf{a})\mathbf{h}\) for a given small vector \(\mathbf{h}\text{.}\) Tags. [U3-LO3, U3-LO5 | P+C+R | Core]
🔗
Exam skill. Interpret a Jacobian matrix as a local linear map and use it to predict a small output change. Tags. [U3-LO5 | C+R | Core]
🔗
🔗

Subsection Linked notebook

Run Lab U3: Jacobian matrices and local linearization
 1 
sebroc.github.io/MATH345-Course-Materials/labs/#lab-u3
. The core path practices symbolic Jacobian matrices, evaluating a Jacobian matrix at a point, local linear prediction, square-grid visualizations, checking Jacobian matrix shapes in the chain rule, and short nonlinear model-block computations.
🔗
For a quick reference on array shapes, matrix products, numerical checks, and SymPy symbolic commands, see the programming appendix sections B.2, B.4, B.9, and B.11.
🔗
🔗
🔗
PrevTopNext