Section C.6 Unit 6 applied, geometric, and computational interpretation
These solution sketches correspond to the additional applied and computational interpretation exercises at the end of Exercises.
Subsection Choosing candidates and equations
Solution to Choosing candidates and equations.
For the rectangle, first solve
\begin{equation*}
\nabla f(x,y)=\mathbf{0}
\end{equation*}
for critical points in the interior. Restrict \(f\) to each of the four edges and find the one-variable critical points on those edges. Include all endpoints and corners. Evaluate \(f\) at every candidate and compare the values.
With one equality constraint, solve
\begin{equation*}
\nabla f(\mathbf{x})
=
\lambda\nabla g(\mathbf{x})
\end{equation*}
together with
\begin{equation*}
g(\mathbf{x})=k.
\end{equation*}
Evaluate \(f\) at every resulting candidate and compare the values when absolute extrema are requested.
With two equality constraints, solve
\begin{equation*}
\nabla f(\mathbf{x})
=
\lambda_1\nabla g_1(\mathbf{x})
+
\lambda_2\nabla g_2(\mathbf{x})
\end{equation*}
together with
\begin{equation*}
g_1(\mathbf{x})=k_1,
\qquad
g_2(\mathbf{x})=k_2.
\end{equation*}
Again, evaluate \(f\) at all candidates and compare the values.
For the region \(g(\mathbf{x})\leq k\text{,}\) solve
\begin{equation*}
\nabla f(\mathbf{x})=\mathbf{0}
\end{equation*}
at feasible interior points satisfying \(g(\mathbf{x})<k\text{.}\) On the boundary, solve
\begin{equation*}
\nabla f(\mathbf{x})
=
\lambda\nabla g(\mathbf{x}),
\qquad
g(\mathbf{x})=k.
\end{equation*}
Compare all interior and boundary candidates. Since the region is closed and bounded, absolute extrema exist when \(f\) is continuous by the extreme value theorem.
Subsection One quadratic form, three readings
Solution to One quadratic form, three readings.
The characteristic polynomial of \(B\) is
\begin{equation*}
\det(B-\lambda I)
=
(5-\lambda)^2-16
=
(\lambda-9)(\lambda-1).
\end{equation*}
Thus the eigenvalues are
\begin{equation*}
\lambda_1=9,
\qquad
\lambda_2=1.
\end{equation*}
Corresponding unit eigenvectors are
\begin{equation*}
\mathbf{v}_1
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
1
\end{bmatrix},
\qquad
\mathbf{v}_2
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
-1
\end{bmatrix}.
\end{equation*}
\begin{equation*}
\max_{\|\mathbf{x}\|=1}q_B(\mathbf{x})=9
\end{equation*}
in direction \(\mathbf{v}_1\text{,}\) while
\begin{equation*}
\min_{\|\mathbf{x}\|=1}q_B(\mathbf{x})=1
\end{equation*}
in direction \(\mathbf{v}_2\text{.}\)
We calculate
\begin{equation*}
A^TA
=
\begin{bmatrix}
1 \amp 2\\
2 \amp 1
\end{bmatrix}
\begin{bmatrix}
1 \amp 2\\
2 \amp 1
\end{bmatrix}
=
\begin{bmatrix}
5 \amp 4\\
4 \amp 5
\end{bmatrix}
=
B.
\end{equation*}
Therefore the singular values of \(A\) are
\begin{equation*}
\sigma_1=\sqrt9=3,
\qquad
\sigma_2=\sqrt1=1,
\end{equation*}
and the right singular directions are \(\mathbf{v}_1\) and \(\mathbf{v}_2\text{.}\) In particular,
\begin{equation*}
\max_{\|\mathbf{x}\|=1}\|A\mathbf{x}\|=3.
\end{equation*}
Since
\begin{equation*}
C=\frac12B,
\end{equation*}
the covariance matrix has the same eigenvectors, with eigenvalues
\begin{equation*}
\frac92
\qquad\text{and}\qquad
\frac12.
\end{equation*}
Thus \(\mathbf{v}_1\) is a first principal direction and \(\mathbf{v}_2\) is a second principal direction. Their captured variations are \(9/2\) and \(1/2\text{.}\)
The first principal direction captures the fraction
\begin{equation*}
\frac{9/2}{9/2+1/2}
=
\frac9{10}
\end{equation*}
of the total centered variation.
The same directions appear because the three symmetric matrices being analyzed are
\begin{equation*}
B,\qquad A^TA=B,\qquad C=\frac12B.
\end{equation*}
Multiplying a matrix by a positive scalar changes its eigenvalues but not its eigenvectors.
Subsection Candidate table on a rectangle
The gradient is
\begin{equation*}
\nabla f(x,y)=
\begin{bmatrix}
2x-2\\
2y+4
\end{bmatrix}.
\end{equation*}
The interior critical point is
\begin{equation*}
(1,-2),
\end{equation*}
and
\begin{equation*}
f(1,-2)=-5.
\end{equation*}
On \(x=0\text{,}\)
\begin{equation*}
f(0,y)=y^2+4y,
\end{equation*}
so the boundary critical point is \(y=-2\text{,}\) giving \((0,-2)\) and value \(-4\text{.}\)
On \(x=3\text{,}\)
\begin{equation*}
f(3,y)=y^2+4y+3,
\end{equation*}
so the boundary critical point is \(y=-2\text{,}\) giving \((3,-2)\) and value \(-1\text{.}\)
On \(y=-3\text{,}\)
\begin{equation*}
f(x,-3)=x^2-2x-3,
\end{equation*}
so the boundary critical point is \(x=1\text{,}\) giving \((1,-3)\) and value \(-4\text{.}\)
On \(y=1\text{,}\)
\begin{equation*}
f(x,1)=x^2-2x+5,
\end{equation*}
so the boundary critical point is \(x=1\text{,}\) giving \((1,1)\) and value \(4\text{.}\)
The corner values are
\begin{equation*}
f(0,-3)=-3,\qquad
f(3,-3)=0,\qquad
f(0,1)=5,\qquad
f(3,1)=8.
\end{equation*}
Thus the absolute minimum value is
\begin{equation*}
-5
\end{equation*}
at
\begin{equation*}
(1,-2),
\end{equation*}
and the absolute maximum value is
\begin{equation*}
8
\end{equation*}
at
\begin{equation*}
(3,1).
\end{equation*}
Subsection Parallel gradients on a circle
Use
\begin{equation*}
g(x,y)=x^2+y^2.
\end{equation*}
Then
\begin{equation*}
\nabla f=
\begin{bmatrix}
3\\
4
\end{bmatrix},
\qquad
\nabla g=
\begin{bmatrix}
2x\\
2y
\end{bmatrix}.
\end{equation*}
The Lagrange equations are
\begin{equation*}
3=2\lambda x,
\qquad
4=2\lambda y,
\qquad
x^2+y^2=25.
\end{equation*}
The first two equations say that
\begin{equation*}
\begin{bmatrix}
x\\
y
\end{bmatrix}
\end{equation*}
is parallel to
\begin{equation*}
\begin{bmatrix}
3\\
4
\end{bmatrix}.
\end{equation*}
Since
\begin{equation*}
\left\|
\begin{bmatrix}
3\\
4
\end{bmatrix}
\right\|=5,
\end{equation*}
the constrained critical points are
\begin{equation*}
(3,4)
\quad\text{and}\quad
(-3,-4).
\end{equation*}
The function values are
\begin{equation*}
f(3,4)=25,
\qquad
f(-3,-4)=-25.
\end{equation*}
So the absolute maximum is \(25\) at \((3,4)\text{,}\) and the absolute minimum is \(-25\) at \((-3,-4)\text{.}\)
Geometrically, \(\nabla f\) points in the direction of fastest increase of \(f\text{,}\) while \(\nabla g\) is perpendicular to the circle. At the maximum and minimum, these directions are parallel or antiparallel.
Subsection Quadratic form on the unit circle
For
\begin{equation*}
\mathbf{x}=
\begin{bmatrix}
x\\
y
\end{bmatrix},
\end{equation*}
we have
\begin{equation*}
q(\mathbf{x})
=
\mathbf{x}^T
\begin{bmatrix}
5\amp0\\
0\amp2
\end{bmatrix}
\mathbf{x}
=
5x^2+2y^2.
\end{equation*}
On the unit circle,
\begin{equation*}
x^2+y^2=1.
\end{equation*}
The maximum value is \(5\text{,}\) attained at
\begin{equation*}
\mathbf{x}=
\begin{bmatrix}
1\\
0
\end{bmatrix}
\quad\text{and}\quad
\mathbf{x}=
\begin{bmatrix}
-1\\
0
\end{bmatrix}.
\end{equation*}
The minimum value is \(2\text{,}\) attained at
\begin{equation*}
\mathbf{x}=
\begin{bmatrix}
0\\
1
\end{bmatrix}
\quad\text{and}\quad
\mathbf{x}=
\begin{bmatrix}
0\\
-1
\end{bmatrix}.
\end{equation*}
These are the eigenvalues and eigenvectors of \(B\text{:}\) the eigenvalues are \(5\) and \(2\text{,}\) with eigenvector directions \(\mathbf{e}_1\) and \(\mathbf{e}_2\text{.}\)
Subsection Maximum stretch of a matrix
Since \(A\) is symmetric,
\begin{equation*}
A^TA=A^2.
\end{equation*}
Compute
\begin{equation*}
A^TA
=
\begin{bmatrix}
1\amp2\\
2\amp1
\end{bmatrix}
\begin{bmatrix}
1\amp2\\
2\amp1
\end{bmatrix}
=
\begin{bmatrix}
5\amp4\\
4\amp5
\end{bmatrix}.
\end{equation*}
The eigenvectors
\begin{equation*}
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
1
\end{bmatrix}
\quad\text{and}\quad
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
-1
\end{bmatrix}
\end{equation*}
have eigenvalues \(9\) and \(1\text{,}\) respectively. Therefore the singular values are
\begin{equation*}
3
\quad\text{and}\quad
1.
\end{equation*}
The largest singular value is \(3\text{,}\) so
\begin{equation*}
\max_{\|\mathbf{x}\|=1}\|A\mathbf{x}\|=3.
\end{equation*}
The most-stretched input direction is
\begin{equation*}
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
1
\end{bmatrix}.
\end{equation*}
Subsection Reading an SVD
The nonzero singular values are
\begin{equation*}
5
\quad\text{and}\quad
2.
\end{equation*}
The matrix has rank
\begin{equation*}
2.
\end{equation*}
Using
\begin{equation*}
A\mathbf{v}_i=\sigma_i\mathbf{u}_i
\end{equation*}
for positive singular values, and \(A\mathbf{v}_i=\mathbf{0}\) for a null direction,
\begin{equation*}
A\mathbf{v}_1=
5
\begin{bmatrix}
1\\
0
\end{bmatrix}
=
\begin{bmatrix}
5\\
0
\end{bmatrix},
\end{equation*}
\begin{equation*}
A\mathbf{v}_2=
2
\begin{bmatrix}
0\\
1
\end{bmatrix}
=
\begin{bmatrix}
0\\
2
\end{bmatrix},
\end{equation*}
and
\begin{equation*}
A\mathbf{v}_3=\mathbf{0}.
\end{equation*}
The forgotten input direction is
\begin{equation*}
\mathbf{v}_3=
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
-1\\
0
\end{bmatrix}.
\end{equation*}
The transmitted input directions are
\begin{equation*}
\mathbf{v}_1=
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
1\\
0
\end{bmatrix}
\end{equation*}
and
\begin{equation*}
\mathbf{v}_2=
\begin{bmatrix}
0\\
0\\
1
\end{bmatrix}.
\end{equation*}
Subsection Fundamental subspaces from an SVD
Since the rank is \(2\text{,}\) the first two right singular vectors form an orthonormal basis for the row space:
\begin{equation*}
\operatorname{row}(A)
=
\operatorname{span}
\left\{
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
1\\
0
\end{bmatrix},
\begin{bmatrix}
0\\
0\\
1
\end{bmatrix}
\right\}.
\end{equation*}
The remaining right singular vector gives the null space:
\begin{equation*}
\operatorname{null}(A)
=
\operatorname{span}
\left\{
\frac{1}{\sqrt2}
\begin{bmatrix}
1\\
-1\\
0
\end{bmatrix}
\right\}.
\end{equation*}
The first two left singular vectors form an orthonormal basis for the column space:
\begin{equation*}
\operatorname{col}(A)
=
\operatorname{span}
\left\{
\begin{bmatrix}
1\\
0
\end{bmatrix},
\begin{bmatrix}
0\\
1
\end{bmatrix}
\right\}
=
\mathbb R^2.
\end{equation*}
Since \(A\) has two rows and rank \(2\text{,}\)
\begin{equation*}
\operatorname{null}(A^T)=\{\mathbf{0}\},
\end{equation*}
whose basis is the empty set.
The dimensions are
\begin{equation*}
\dim(\operatorname{row}(A))=2,
\qquad
\dim(\operatorname{null}(A))=1,
\end{equation*}
and
\begin{equation*}
2+1=3,
\end{equation*}
which agrees with rank-nullity for a \(2\times 3\) matrix. Also,
\begin{equation*}
\dim(\operatorname{col}(A))=2,
\qquad
\dim(\operatorname{null}(A^T))=0,
\end{equation*}
and
\begin{equation*}
2+0=2.
\end{equation*}
Subsection Two constraints on a sphere
The feasible set is the intersection of the unit sphere and the plane
\begin{equation*}
x+y+z=0.
\end{equation*}
It is a circle, so it is closed and bounded. Since \(f\) is continuous, absolute extrema exist by the extreme value theorem.
Let
\begin{equation*}
g_1(x,y,z)=x^2+y^2+z^2,
\qquad
g_2(x,y,z)=x+y+z.
\end{equation*}
Then
\begin{equation*}
\nabla g_1
=
2
\begin{bmatrix}
x\\
y\\
z
\end{bmatrix},
\qquad
\nabla g_2
=
\begin{bmatrix}
1\\
1\\
1
\end{bmatrix}.
\end{equation*}
At a feasible point,
\begin{equation*}
\nabla g_1\cdot\nabla g_2
=
2(x+y+z)
=
0.
\end{equation*}
Both gradients are nonzero there. They are therefore linearly independent.
Define
\begin{equation*}
L(x,y,z,\lambda,\mu)
=
x-y
-
\lambda(x^2+y^2+z^2-1)
-
\mu(x+y+z).
\end{equation*}
The equations are
\begin{equation*}
1=2\lambda x+\mu,
\end{equation*}
\begin{equation*}
-1=2\lambda y+\mu,
\end{equation*}
\begin{equation*}
0=2\lambda z+\mu,
\end{equation*}
together with the two constraints.
Adding the first three equations and using \(x+y+z=0\) gives
\begin{equation*}
\mu=0.
\end{equation*}
Also, \(\lambda\ne0\text{,}\) since otherwise the first equation would be impossible. Thus
\begin{equation*}
x=\frac1{2\lambda},
\qquad
y=-\frac1{2\lambda},
\qquad
z=0.
\end{equation*}
The sphere constraint gives
\begin{equation*}
\frac1{2\lambda^2}=1,
\end{equation*}
so
\begin{equation*}
\lambda=\pm\frac1{\sqrt2}.
\end{equation*}
The candidates are
\begin{equation*}
\frac1{\sqrt2}
\begin{bmatrix}
1\\
-1\\
0
\end{bmatrix}
\qquad\text{and}\qquad
\frac1{\sqrt2}
\begin{bmatrix}
-1\\
1\\
0
\end{bmatrix}.
\end{equation*}
Their \(f\)-values are
\begin{equation*}
\sqrt2\qquad\text{and}\qquad-\sqrt2.
\end{equation*}
Therefore the absolute maximum is \(\sqrt2\text{,}\) and the absolute minimum is \(-\sqrt2\text{.}\)
Subsection Principal directions of a centered data set
The mean is
\begin{equation*}
\bar{\mathbf{x}}
=
\begin{bmatrix}
1\\
1
\end{bmatrix}.
\end{equation*}
Therefore
\begin{equation*}
Z=
\begin{bmatrix}
3\amp1\\
1\amp3\\
-3\amp-1\\
-1\amp-3
\end{bmatrix}.
\end{equation*}
The covariance matrix is
\begin{equation*}
C
=
\frac14Z^TZ
=
\begin{bmatrix}
5\amp3\\
3\amp5
\end{bmatrix}.
\end{equation*}
Its eigenvalues are
\begin{equation*}
\lambda_1=8,
\qquad
\lambda_2=2.
\end{equation*}
Corresponding unit eigenvectors are
\begin{equation*}
\mathbf{v}_1
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
1
\end{bmatrix},
\qquad
\mathbf{v}_2
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
-1
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
V=
\frac1{\sqrt2}
\begin{bmatrix}
1\amp1\\
1\amp-1
\end{bmatrix}.
\end{equation*}
The principal-coordinate matrix is
\begin{equation*}
T=ZV
=
\begin{bmatrix}
2\sqrt2\amp\sqrt2\\
2\sqrt2\amp-\sqrt2\\
-2\sqrt2\amp-\sqrt2\\
-2\sqrt2\amp\sqrt2
\end{bmatrix}.
\end{equation*}
Also,
\begin{equation*}
V^TCV
=
\begin{bmatrix}
8\amp0\\
0\amp2
\end{bmatrix}.
\end{equation*}
The first principal direction captures
\begin{equation*}
\frac{\lambda_1}{\lambda_1+\lambda_2}
=
\frac8{8+2}
=
\frac45
\end{equation*}
of the total centered variation.
Subsection PCA reconstruction and regression
The score is
\begin{equation*}
t_1
=
\mathbf{v}_1^T\mathbf{z}_1
=
\frac4{\sqrt2}
=
2\sqrt2.
\end{equation*}
Therefore
\begin{equation*}
\widehat{\mathbf{z}}_1
=
t_1\mathbf{v}_1
=
\begin{bmatrix}
2\\
2
\end{bmatrix},
\end{equation*}
and
\begin{equation*}
\mathbf{r}_1
=
\begin{bmatrix}
3\\
1
\end{bmatrix}
-
\begin{bmatrix}
2\\
2
\end{bmatrix}
=
\begin{bmatrix}
1\\
-1
\end{bmatrix}.
\end{equation*}
We check that
\begin{equation*}
\mathbf{r}_1\cdot\mathbf{v}_1
=
\frac1{\sqrt2}(1-1)
=
0.
\end{equation*}
Since
\begin{equation*}
\bar{\mathbf{x}}
=
\begin{bmatrix}
1\\
1
\end{bmatrix},
\end{equation*}
the reconstructed original data point is
\begin{equation*}
\widehat{\mathbf{x}}_1
=
\begin{bmatrix}
3\\
3
\end{bmatrix}.
\end{equation*}
The PCA line passes through \((1,1)\) in direction \((1,1)\text{,}\) so its equation is
\begin{equation*}
y=x.
\end{equation*}
For regression,
\begin{equation*}
m=\frac35,
\qquad
b=1-\frac35=\frac25.
\end{equation*}
Thus the regression line is
\begin{equation*}
\widehat y
=
\frac25+\frac35x.
\end{equation*}
PCA treats the two feature coordinates symmetrically and minimizes perpendicular reconstruction residuals. Regression treats \(x\) as the predictor and \(y\) as the response, and minimizes vertical prediction residuals. The two objectives need not produce the same line.
Subsection Redundant features revisited by singular values
Compute
\begin{equation*}
X
\begin{bmatrix}
1\\
1\\
-1
\end{bmatrix}
=
\begin{bmatrix}
1+2-3\\
2+4-6\\
0+1-1\\
1-1-0
\end{bmatrix}
=
\begin{bmatrix}
0\\
0\\
0\\
0
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
\begin{bmatrix}
1\\
1\\
-1
\end{bmatrix}
\in \operatorname{null}(X).
\end{equation*}
The third column is the sum of the first two columns, so the columns are dependent. The first two columns are not multiples of each other, so
\begin{equation*}
\operatorname{rank}(X)=2.
\end{equation*}
Because \(X\) has three columns but rank \(2\text{,}\) one singular value is zero.
If \(X\mathbf{c}=\mathbf{y}\text{,}\) then
\begin{equation*}
X\left(\mathbf{c}+t
\begin{bmatrix}
1\\
1\\
-1
\end{bmatrix}\right)
=
X\mathbf{c}
+
tX
\begin{bmatrix}
1\\
1\\
-1
\end{bmatrix}
=
\mathbf{y}.
\end{equation*}
So different coefficient vectors can give the same output. This is the same nonuniqueness from Unit 2, now detected by a zero singular value.
Subsection PCA from an SVD
The covariance eigenvalues are
\begin{equation*}
\lambda_1
=
\frac{(4\sqrt2)^2}{4}
=
8,
\qquad
\lambda_2
=
\frac{(2\sqrt2)^2}{4}
=
2.
\end{equation*}
Thus
\begin{equation*}
\mathbf{v}_1
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
1
\end{bmatrix}
\end{equation*}
is a first principal direction, and
\begin{equation*}
\mathbf{v}_2
=
\frac1{\sqrt2}
\begin{bmatrix}
1\\
-1
\end{bmatrix}
\end{equation*}
is a second principal direction.
The principal-coordinate matrix is
\begin{equation*}
T=U_2\Sigma_2
=
\begin{bmatrix}
2\sqrt2\amp\sqrt2\\
2\sqrt2\amp-\sqrt2\\
-2\sqrt2\amp-\sqrt2\\
-2\sqrt2\amp\sqrt2
\end{bmatrix}.
\end{equation*}
Keeping only the first principal direction gives
\begin{equation*}
\widehat Z_1
=
\sigma_1\mathbf{u}_1\mathbf{v}_1^T
=
\begin{bmatrix}
2\amp2\\
2\amp2\\
-2\amp-2\\
-2\amp-2
\end{bmatrix}.
\end{equation*}
The captured fraction is
\begin{equation*}
\frac{\sigma_1^2}
{\sigma_1^2+\sigma_2^2}
=
\frac{32}{32+8}
=
\frac45.
\end{equation*}
The two columns of \(T\) are the score vectors for the first and second principal directions.
