Section C.5 Unit 5 applied, geometric, and computational interpretation
These solution sketches correspond to the additional applied and computational interpretation exercises at the end of Unit 5.
Subsection Critical points are candidates
We have
\begin{equation*}
\nabla f(x,y)=
\begin{bmatrix}
2x\\
-2y
\end{bmatrix}.
\end{equation*}
Thus the only critical point is \((0,0)\text{.}\)
Along the \(x\)-axis,
\begin{equation*}
f(t,0)=t^2,
\end{equation*}
which is positive for \(t\ne 0\text{.}\) Along the \(y\)-axis,
\begin{equation*}
f(0,t)=-t^2,
\end{equation*}
which is negative for \(t\ne 0\text{.}\) Nearby points therefore occur both above and below \(f(0,0)=0\text{.}\) The critical point is a saddle point, not a local minimum or local maximum.
Subsection Directional derivative as a dot product
Use
\begin{equation*}
D_{\mathbf{v}}f(\mathbf{a})=\nabla f(\mathbf{a})\cdot \mathbf{v}.
\end{equation*}
Then
\begin{equation*}
D_{\mathbf{v}_1}f(\mathbf{a})=6,
\qquad
D_{\mathbf{v}_2}f(\mathbf{a})=8.
\end{equation*}
Also,
\begin{equation*}
D_{\mathbf{v}_3}f(\mathbf{a})
=
\begin{bmatrix}
6\\
8
\end{bmatrix}
\cdot
\frac15
\begin{bmatrix}
3\\
4
\end{bmatrix}
=
\frac{18+32}{5}=10,
\end{equation*}
and
\begin{equation*}
D_{\mathbf{v}_4}f(\mathbf{a})=-10.
\end{equation*}
The largest listed directional derivative occurs in direction \(\mathbf{v}_3\text{,}\) and the smallest occurs in direction \(\mathbf{v}_4\text{.}\) This is the Unit 1 dot product measuring alignment: the derivative is largest when the direction is aligned with the gradient and smallest when it points opposite the gradient.
Subsection Reading a gradient descent loop
The loop represents
\begin{equation*}
\mathbf{x}_{k+1}=\mathbf{x}_k-\alpha\nabla f(\mathbf{x}_k).
\end{equation*}
The learning rate is
alpha. The vector grad_f(x) must have the same shape as x. Replacing the minus sign by a plus sign would move in the direction of steepest increase instead of steepest decrease. The code does not prove that a minimum has been found. It only describes the update rule; convergence depends on the function, starting point, learning rate, and stopping rule.Subsection Learning-rate diagnosis
The learning rate \(\alpha=0.05\) is making slow but steady progress. The learning rate \(\alpha=0.20\) is making faster useful progress. The learning rate \(\alpha=1.05\) appears unstable because the loss increases.
A decreasing loss table does not prove that the global minimum has been found. It only reports what happened for the displayed iterates.
Subsection Eigenvectors as special directions
We compute
\begin{equation*}
A\mathbf{v}_1=
\begin{bmatrix}
3\amp1\\
0\amp2
\end{bmatrix}
\begin{bmatrix}
1\\
0
\end{bmatrix}
=
\begin{bmatrix}
3\\
0
\end{bmatrix}
=
3\mathbf{v}_1.
\end{equation*}
So \(\mathbf{v}_1\) is an eigenvector with eigenvalue \(3\text{.}\)
Next,
\begin{equation*}
A\mathbf{v}_2=
\begin{bmatrix}
3\amp1\\
0\amp2
\end{bmatrix}
\begin{bmatrix}
-1\\
1
\end{bmatrix}
=
\begin{bmatrix}
-2\\
2
\end{bmatrix}
=
2
\begin{bmatrix}
-1\\
1
\end{bmatrix}
=
2\mathbf{v}_2.
\end{equation*}
So \(\mathbf{v}_2\) is an eigenvector with eigenvalue \(2\text{.}\) The matrix map stretches the \(\mathbf{v}_1\)-direction by a factor of \(3\) and the \(\mathbf{v}_2\)-direction by a factor of \(2\text{.}\)
Subsection Quadratic form and definiteness
For the three listed vectors,
\begin{equation*}
\begin{bmatrix}
1\\
0
\end{bmatrix}^T
H
\begin{bmatrix}
1\\
0
\end{bmatrix}
=4,
\end{equation*}
\begin{equation*}
\begin{bmatrix}
0\\
1
\end{bmatrix}^T
H
\begin{bmatrix}
0\\
1
\end{bmatrix}
=1,
\end{equation*}
and
\begin{equation*}
\begin{bmatrix}
1\\
1
\end{bmatrix}^T
H
\begin{bmatrix}
1\\
1
\end{bmatrix}
=5.
\end{equation*}
For a general vector
\begin{equation*}
\mathbf{h}=
\begin{bmatrix}
h_1\\
h_2
\end{bmatrix},
\end{equation*}
\begin{equation*}
\mathbf{h}^TH\mathbf{h}=4h_1^2+h_2^2.
\end{equation*}
This is positive for every nonzero \(\mathbf{h}\text{,}\) so \(H\) is positive definite. If \(H\) is the Hessian at a critical point, the second derivative test gives a local minimum.
Subsection Hessian eigenvalue classification
If the Hessian eigenvalues are \(3,5\text{,}\) the Hessian is positive definite, so the critical point is a local minimum.
If the Hessian eigenvalues are \(-2,-7\text{,}\) the Hessian is negative definite, so the critical point is a local maximum.
If the Hessian eigenvalues are \(4,-1\text{,}\) the Hessian is indefinite, so the critical point is a saddle point.
If the Hessian eigenvalues are \(0,2\text{,}\) the test is inconclusive. The zero eigenvalue means the quadratic approximation has a flat direction, so the second-order term alone does not decide the local behavior.
Subsection Least squares from gradients
We compute
\begin{equation*}
A^TA=
\begin{bmatrix}
3\amp3\\
3\amp5
\end{bmatrix},
\qquad
A^T\mathbf{b}=
\begin{bmatrix}
5\\
6
\end{bmatrix}.
\end{equation*}
The normal equations are
\begin{equation*}
A^TA\hat{\mathbf{x}}=A^T\mathbf{b},
\end{equation*}
or
\begin{equation*}
\begin{bmatrix}
3\amp3\\
3\amp5
\end{bmatrix}
\hat{\mathbf{x}}
=
\begin{bmatrix}
5\\
6
\end{bmatrix}.
\end{equation*}
Solving gives
\begin{equation*}
\hat{\mathbf{x}}=
\begin{bmatrix}
7/6\\
1/2
\end{bmatrix}.
\end{equation*}
Then
\begin{equation*}
A\hat{\mathbf{x}}=
\begin{bmatrix}
7/6\\
5/3\\
13/6
\end{bmatrix},
\end{equation*}
so the residual is
\begin{equation*}
\mathbf{r}
=
\mathbf{b}-A\hat{\mathbf{x}}
=
\begin{bmatrix}
-1/6\\
1/3\\
-1/6
\end{bmatrix}.
\end{equation*}
Finally,
\begin{equation*}
A^T\mathbf{r}
=
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
This is the residual-orthogonality condition from Unit 4 and the gradient-zero condition from Unit 5.
Subsection Residual orthogonality in code
The vector
xhat is a least-squares coefficient vector. The vector r is the residual
\begin{equation*}
\mathbf{r}=\mathbf{b}-A\hat{\mathbf{x}}.
\end{equation*}
The expression
A.T @ r checks whether the residual is orthogonal to the columns of \(A\text{.}\) It can be close to zero even when r is not close to zero. Unit 4 used this residual-orthogonality condition to derive the normal equations. Unit 5 obtains the same equations by setting the gradient of \(\|A\mathbf{x}-\mathbf{b}\|^2\) equal to zero.Subsection Rank-one update
The outer product is
\begin{equation*}
\mathbf{g}\mathbf{h}^T
=
\begin{bmatrix}
1\\
-2\\
3
\end{bmatrix}
\begin{bmatrix}
4\amp0
\end{bmatrix}
=
\begin{bmatrix}
4\amp0\\
-8\amp0\\
12\amp0
\end{bmatrix}.
\end{equation*}
Its shape is \(3\times 2\text{.}\) Since every column is a scalar multiple of \(\mathbf{g}\text{,}\) the column space has dimension at most one, so the rank is at most one.
The update
\begin{equation*}
W_{\mathrm{new}}=W-\alpha \mathbf{g}\mathbf{h}^T
\end{equation*}
changes \(W\) by subtracting a scaled rank-one matrix. In NumPy,
np.outer(g, h) is appropriate when g and h are stored as one-dimensional arrays because it explicitly forms the matrix \(\mathbf{g}\mathbf{h}^T\text{.}\)
