Applications and computation recap

Section 5.8 Applications and computation recap

Application question. How do we read the short computations used in Unit 5?

This section checks whether you can connect a short calculation or code fragment to the mathematics in this unit. The goal is not to memorize Python. The goal is to identify the objects, check the shapes, and interpret the mathematical operation.

🔗

A short Unit 5 computation should be read in three steps.

🔗

Identify the mathematical objects.
🔗

🔗
Identify the update, matrix product, eigenvalue computation, or least-squares problem.
🔗

🔗
Interpret what the output says about optimization.
🔗

🔗

Table 5.8.1. Reading Unit 5 computations

🔗

Code fragment 🔗	First question to ask 🔗	Typical interpretation 🔗
`grad_f(x)`	What function is being minimized? 🔗	Gradient at the current point. 🔗
`x - alpha * grad_f(x)`	Is the sign negative? 🔗	One gradient descent step. 🔗
`losses.append(f(x))`	What is being recorded? 🔗	Loss history during an iteration. 🔗
`np.linalg.eig(H)`	Is `H` a Hessian or another square matrix? 🔗	Eigenvalues and eigenvectors. 🔗
`Q.T @ H @ Q`	Are the columns of `Q` orthonormal eigenvectors? 🔗	Hessian in principal-axis coordinates. 🔗
`np.linalg.lstsq(A, b, rcond=None)[0]`	What are the columns of `A`? 🔗	Least-squares coefficient vector. 🔗
`r = b - A @ xhat`	What is being compared? 🔗	Residual vector. 🔗
`A.T @ r`	Which Unit 4 condition is being checked? 🔗	Residual orthogonality. 🔗
`np.column_stack([...])`	What features are being collected? 🔗	Design matrix construction. 🔗
`np.outer(g, h)`	What are the shapes of `g` and `h`? 🔗	Rank-one outer product. 🔗

Subsection Review activities

Activity 5.8.1. Reading a gradient descent loop.

Assume grad_f(x) computes \(\nabla f(\mathbf{x})\text{.}\) Consider the code:

🔗

x = x0
for k in range(num_steps):
    x = x - alpha * grad_f(x)

What mathematical update rule is represented?
🔗

🔗
Which quantity is the learning rate?
🔗

🔗
What shape must grad_f(x) have?
🔗

🔗
What would change if the minus sign were a plus sign?
🔗

🔗
Does this code prove that a minimum has been found?
🔗

🔗

Tags. [U5-LO2, U5-LO3 | C+T | Core]

🔗

Solution.

The loop represents the update

\begin{equation*} \mathbf{x}_{k+1}=\mathbf{x}_k-\alpha\nabla f(\mathbf{x}_k). \end{equation*}

The learning rate is alpha. The vector grad_f(x) must have the same shape as x, since the update subtracts one vector from another. If the minus sign were replaced by a plus sign, the update would move in the direction of steepest increase instead of steepest decrease. The loop does not prove that a minimum has been found. It only describes the repeated update; convergence depends on the function, starting point, learning rate, and stopping rule.

🔗

Activity 5.8.2. Diagnosing learning rates.

For \(f(x)=x^2\text{,}\) start at \(x_0=4\) and use the update

\begin{equation*} x_{k+1}=x_k-\alpha f'(x_k). \end{equation*}

The table shows the loss values \(f(x_k)\) for three learning rates.

🔗

\(k\) 🔗	\(\alpha = 0.05\) 🔗	\(\alpha = 0.20\) 🔗	\(\alpha = 1.05\) 🔗
0 🔗	16.000 🔗	16.000 🔗	16.000 🔗
1 🔗	12.960 🔗	5.760 🔗	19.360 🔗
2 🔗	10.498 🔗	2.074 🔗	23.426 🔗
3 🔗	8.503 🔗	0.746 🔗	28.345 🔗

Which learning rate is making slow but steady progress?
🔗

🔗
Which learning rate is making faster useful progress?
🔗

🔗
Which learning rate appears unstable?
🔗

🔗
Does a decreasing loss table prove that the global minimum has been found?
🔗

🔗

Tags. [U5-LO3 | C+T | Core]

🔗

Solution.

The learning rate \(\alpha=0.05\) is making slow but steady progress. The learning rate \(\alpha=0.20\) is making faster useful progress. The learning rate \(\alpha=1.05\) appears unstable because the loss is increasing. A decreasing loss table does not prove that the global minimum has been found. It only shows what happened for the displayed iterates.

🔗

Activity 5.8.3. Reading Hessian eigenvalues.

Suppose a critical point has Hessian matrix

🔗

H = np.array([[4.0, 0.0],
              [0.0, -1.0]])

evals, evecs = np.linalg.eig(H)
evals

Output:

🔗

array([ 4., -1.])

What matrix is being studied?
🔗

🔗
What do the signs of the eigenvalues say about the quadratic form \(\mathbf{h}^T H\mathbf{h}\text{?}\)
🔗

🔗
If \(H\) is the Hessian at a critical point, what does the second derivative test conclude?
🔗

🔗
Why is the phrase "at a critical point" important?
🔗

🔗

Tags. [U5-LO4, U5-LO5 | C+T | Core]

🔗

Solution.

The matrix being studied is the Hessian matrix. Its eigenvalues have opposite signs, so the quadratic form \(\mathbf{h}^T H\mathbf{h}\) takes both positive and negative values. If \(H\) is the Hessian at a critical point, the second derivative test classifies the critical point as a saddle point. The phrase "at a critical point" is important because the Hessian test classifies local behavior after the first-order term has disappeared.

🔗

Activity 5.8.4. Reading a least-squares residual check.

Consider the code:

🔗

xhat = np.linalg.lstsq(A, b, rcond=None)[0]
r = b - A @ xhat
A.T @ r

What is xhat?
🔗

🔗
What is r?
🔗

🔗
What condition is checked by A.T @ r?
🔗

🔗
Does A.T @ r being close to zero mean that r is close to zero?
🔗

🔗
Which earlier unit used this same condition?
🔗

🔗

Tags. [U5-LO6, U4-LO4 | C+T | Core]

🔗

Solution.

The vector xhat is a least-squares coefficient vector. The vector r is the residual

\begin{equation*} \mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}. \end{equation*}

The expression A.T @ r checks whether the residual is orthogonal to the columns of \(A\text{.}\) It can be close to zero even when r is not close to zero. Unit 4 used this same residual-orthogonality condition to derive the normal equations. Unit 5 obtains the same equations by setting the gradient of \(\|A\mathbf{x}-\mathbf{b}\|^2\) equal to zero.

🔗

Activity 5.8.5. Reading fixed-hidden-layer training.

Let \(\sigma(t)=\tanh(t)\text{.}\) Consider the code:

🔗

A = np.column_stack([
    np.ones_like(t),
    np.tanh(t),
    np.tanh(t - 1),
    np.tanh(t + 1)
])

c = np.linalg.lstsq(A, y, rcond=None)[0]
yhat = A @ c

What are the columns of \(A\text{?}\)
🔗

🔗
Which vector is being trained?
🔗

🔗
Why is this a least-squares problem?
🔗

🔗
Is the model linear as a function of \(t\text{?}\)
🔗

🔗
Is the model linear as a function of \(\mathbf{c}\text{?}\)
🔗

🔗

Tags. [U5-LO6, U5-LO7 | C+M+T | Core]

🔗

Solution.

The columns of \(A\) are the fixed feature vectors

\begin{equation*} 1,\qquad \tanh(t),\qquad \tanh(t-1),\qquad \tanh(t+1), \end{equation*}

evaluated at the data inputs. The vector being trained is

\begin{equation*} \mathbf{c}= \begin{bmatrix} c_0\\ c_1\\ c_2\\ c_3 \end{bmatrix}. \end{equation*}

This is a least-squares problem because the prediction vector is \(A\mathbf{c}\text{,}\) and training chooses \(\mathbf{c}\) so that \(A\mathbf{c}\) is close to \(\mathbf{y}\text{.}\) The model is not linear as a function of \(t\text{,}\) because it contains \(\tanh(t)\text{,}\) \(\tanh(t-1)\text{,}\) and \(\tanh(t+1)\text{.}\) It is linear as a function of \(\mathbf{c}\text{,}\) which is why least squares applies.

🔗

Activity 5.8.6. Reading a rank-one update.

Suppose \(\mathbf{g}\in\mathbb{R}^m\) and \(\mathbf{h}\in\mathbb{R}^d\text{.}\) Consider the code:

🔗

G = np.outer(g, h)
W_new = W - alpha * G

What is the shape of G?
🔗

🔗
Why is np.outer(g, h) used here?
🔗

🔗
What mathematical update is represented by the second line?
🔗

🔗
Why is this connected to rank?
🔗

🔗

Tags. [U5-LO7, U2-LO3 | C+T | Core]

🔗

Solution.

The matrix G has shape \(m\times d\text{.}\) The command np.outer(g, h) forms the matrix \(\mathbf{g}\mathbf{h}^T\text{.}\) This is the correct outer product when g and h are stored as one-dimensional NumPy arrays. The second line represents the update

\begin{equation*} W_{\mathrm{new}}=W-\alpha \mathbf{g}\mathbf{h}^T. \end{equation*}

The matrix \(\mathbf{g}\mathbf{h}^T\) has rank at most one, so the update changes \(W\) in one rank-one direction.

🔗

Subsection Linked notebook

Run Lab U5: Gradient descent and tiny training

sebroc.github.io/MATH345-Course-Materials/labs/#lab-u5

. Focus on four reading habits: identify the gradient descent update, diagnose a learning-rate table, recognize a least-squares call, and identify the fixed feature columns in the design matrix.

🔗

For a quick reference on Python loops, helper functions, entrywise functions, design matrices, eigenvalue and least-squares commands, transposes, and numerical checks such as "close to zero," see the programming appendix sections B.12, B.4, B.8, B.10, and B.9.

🔗

Exam skill. Given a short Unit 5 code snippet, loss table, or matrix output, identify the mathematical operation and interpret the result as a descent step, Hessian/eigenvalue check, least-squares fit, residual-orthogonality check, fixed-feature training problem, or rank-one update. Tags. [U5-LO2, U5-LO3, U5-LO4, U5-LO5, U5-LO6, U5-LO7 | C+M+T | Core]

🔗

Prev Top Next