Section C.7 Unit 7 applied, geometric, and computational interpretation
These are solution sketches for the additional applied and computational problems in Unit 7.
Subsection Matrix-space coordinates
We have
\begin{equation*}
A=
\begin{bmatrix}
2\amp-1\\
0\amp3
\end{bmatrix}
=
2E_{11}-E_{12}+0E_{21}+3E_{22}.
\end{equation*}
Thus the coordinate vector in the ordered basis \((E_{11},E_{12},E_{21},E_{22})\) is
\begin{equation*}
\begin{bmatrix}
2\\
-1\\
0\\
3
\end{bmatrix}.
\end{equation*}
The space \(M_{22}\) has dimension \(4\text{.}\) Its zero vector is the zero matrix
\begin{equation*}
\begin{bmatrix}
0\amp0\\
0\amp0
\end{bmatrix}.
\end{equation*}
Subsection Polynomial coordinates and subspaces
In the ordered basis \((1,x,x^2,x^3)\text{,}\) the coordinate vector of \(p(x)=2-3x+x^3\) is
\begin{equation*}
\begin{bmatrix}
2\\
-3\\
0\\
1
\end{bmatrix}.
\end{equation*}
For \(U=\{p\in\mathcal P_{\le3}:p(0)=0\}\text{,}\) the constant coefficient must be zero, so
\begin{equation*}
U=\operatorname{span}\{x,x^2,x^3\}.
\end{equation*}
For \(W=\{p\in\mathcal P_{\le3}:p(0)=p'(0)=0\}\text{,}\) the constant and linear coefficients must be zero, so
\begin{equation*}
W=\operatorname{span}\{x^2,x^3\}.
\end{equation*}
Both are subspaces because they are solution sets of homogeneous linear conditions on the coefficients.
Subsection Derivative and evaluation as linear maps
The derivative map is linear because
\begin{equation*}
D(p+q)=(p+q)'=p'+q'
\end{equation*}
and
\begin{equation*}
D(cp)=(cp)'=cp'.
\end{equation*}
If \(p(x)=a_0+a_1x+a_2x^2+a_3x^3\text{,}\) then
\begin{equation*}
D(p)=a_1+2a_2x+3a_3x^2.
\end{equation*}
Thus the matrix of \(D\text{,}\) using the ordered bases \((1,x,x^2,x^3)\) and \((1,x,x^2)\text{,}\) is
\begin{equation*}
\begin{bmatrix}
0\amp1\amp0\amp0\\
0\amp0\amp2\amp0\\
0\amp0\amp0\amp3
\end{bmatrix}.
\end{equation*}
The evaluation map \(E_0(p)=p(0)\) is linear because evaluation distributes over addition and scalar multiplication. Its kernel is
\begin{equation*}
\{p\in\mathcal P_{\le3}:p(0)=0\}
=
\operatorname{span}\{x,x^2,x^3\}.
\end{equation*}
Subsection Matrix inner product
The trace inner product is the sum of entrywise products:
\begin{equation*}
\langle A,B\rangle
=
1\cdot3+2\cdot0+0\cdot4+(-1)\cdot1
=
2.
\end{equation*}
Also
\begin{equation*}
\|A\|^2
=
1^2+2^2+0^2+(-1)^2
=
6,
\end{equation*}
so \(\|A\|=\sqrt6\text{.}\) Next
\begin{equation*}
A-B=
\begin{bmatrix}
-2\amp2\\
-4\amp-2
\end{bmatrix},
\end{equation*}
so \(d(A,B)^2=4+4+16+4=28\text{.}\) Therefore \(d(A,B)=\sqrt{28}=2\sqrt7\text{.}\) Since \(\langle A,B\rangle=2\ne0\text{,}\) the matrices are not orthogonal.
Subsection Same polynomial space, different inner products
In the coefficient inner product,
\begin{equation*}
p(x)=1+x
\quad\leftrightarrow\quad
\begin{bmatrix}
1\\
1
\end{bmatrix},
\qquad
q(x)=1-x
\quad\leftrightarrow\quad
\begin{bmatrix}
1\\
-1
\end{bmatrix}.
\end{equation*}
Thus
\begin{equation*}
\langle p,q\rangle_{\mathrm{coef}}
=
1\cdot1+1\cdot(-1)=0.
\end{equation*}
For the \(L^2[-1,1]\) inner product,
\begin{equation*}
\langle p,q\rangle_{L^2}
=
\int_{-1}^{1}(1+x)(1-x)\,dx
=
\int_{-1}^{1}(1-x^2)\,dx.
\end{equation*}
This is \(2-\frac23=\frac43\text{.}\) Thus \(p\) and \(q\) are orthogonal for the coefficient inner product but not for the \(L^2[-1,1]\) inner product. Orthogonality depends on the chosen inner product.
Subsection Projection equations for the continuous least-squares line
With basis \((1,x)\text{,}\)
\begin{equation*}
G=
\begin{bmatrix}
\langle1,1\rangle\amp\langle x,1\rangle\\
\langle1,x\rangle\amp\langle x,x\rangle
\end{bmatrix}
=
\begin{bmatrix}
2\amp0\\
0\amp\frac23
\end{bmatrix}.
\end{equation*}
For \(p(x)=1+2x+x^2+x^3\text{,}\) we get
\begin{equation*}
b_0=\int_{-1}^{1}p(x)\,dx=2+\frac23=\frac83,
\end{equation*}
and
\begin{equation*}
b_1=\int_{-1}^{1}xp(x)\,dx
=
\int_{-1}^{1}(x+2x^2+x^3+x^4)\,dx
=
\frac43+\frac25
=
\frac{26}{15}.
\end{equation*}
Thus
\begin{equation*}
\begin{bmatrix}
2\amp0\\
0\amp\frac23
\end{bmatrix}
\begin{bmatrix}
c_0\\
c_1
\end{bmatrix}
=
\begin{bmatrix}
\frac83\\
\frac{26}{15}
\end{bmatrix}.
\end{equation*}
So \(c_0=\frac43\) and \(c_1=\frac{13}{5}\text{.}\) The continuous least-squares line is \(q_{L^2}(x)=\frac43+\frac{13}{5}x\text{.}\)
Subsection Taylor line versus fixed-interval least squares
The Taylor line of \(f(x)=x^2\) at \(0\) is \(T(x)=0\text{.}\) For the \(L^2[-1,1]\)-projection onto \(\operatorname{span}\{1,x\}\text{,}\) symmetry gives no \(x\)-term. The best constant is
\begin{equation*}
\frac{\int_{-1}^{1}x^2\,dx}{\int_{-1}^{1}1\,dx}
=
\frac{2/3}{2}
=
\frac13.
\end{equation*}
Thus the fixed-interval least-squares approximation is \(q(x)=\frac13\text{.}\) The answers differ because Taylor approximation is local, while fixed-interval least squares minimizes error over the whole interval.
Subsection Taylor projection from a local inner product
If \(p(x)=a_0+a_1x+a_2x^2+a_3x^3\) and \(q(x)=c_0+c_1x\text{,}\) then
\begin{equation*}
p-q=(a_0-c_0)+(a_1-c_1)x+a_2x^2+a_3x^3.
\end{equation*}
The squared norm from the local inner product is
\begin{equation*}
(a_0-c_0)^2+(a_1-c_1)^2+a_2^2+a_3^2.
\end{equation*}
The last two terms do not depend on \(c_0,c_1\text{.}\) The minimum occurs when \(c_0=a_0\) and \(c_1=a_1\text{.}\) Thus the projection is \(a_0+a_1x\text{.}\) Since \(a_0=p(0)\) and \(a_1=p'(0)\text{,}\) the answer is \(p(0)+p'(0)x\text{.}\)
Subsection Shrinking least squares
For \(x^2\text{,}\) symmetry gives slope \(0\text{,}\) and the constant is
\begin{equation*}
\frac{1}{2t}\int_{-t}^{t}x^2\,dx
=
\frac{1}{2t}\cdot\frac{2t^3}{3}
=
\frac{t^2}{3}.
\end{equation*}
Thus \(P_t(x^2)=\frac{t^2}{3}\text{.}\) For \(x^3\text{,}\) symmetry gives constant \(0\text{.}\) The slope is
\begin{equation*}
\frac{\int_{-t}^{t}x^4\,dx}{\int_{-t}^{t}x^2\,dx}
=
\frac{2t^5/5}{2t^3/3}
=
\frac{3t^2}{5}.
\end{equation*}
Thus \(P_t(x^3)=\frac{3t^2}{5}x\text{.}\) Both projections tend to \(0\) as \(t\to0\text{,}\) which is the Taylor linear approximation at the origin for both \(x^2\) and \(x^3\text{.}\)
Subsection Gram-Schmidt in \(\mathcal P_{\le3}\)
We have \(\langle1,x\rangle=\int_{-1}^{1}x\,dx=0\text{,}\) so \(1\) and \(x\) are orthogonal. For \(x^2\text{,}\)
\begin{equation*}
\operatorname{proj}_{\operatorname{span}\{1\}}(x^2)
=
\frac{\langle x^2,1\rangle}{\langle1,1\rangle}1
=
\frac{2/3}{2}
=
\frac13.
\end{equation*}
Also \(\langle x^2,x\rangle=\int_{-1}^{1}x^3\,dx=0\text{.}\) So the next orthogonal polynomial is \(x^2-\frac13\text{.}\) For \(x^3\text{,}\) the projections onto \(1\) and \(x^2-\frac13\) vanish by symmetry. The projection onto \(x\) is
\begin{equation*}
\frac{\langle x^3,x\rangle}{\langle x,x\rangle}x
=
\frac{2/5}{2/3}x
=
\frac35x.
\end{equation*}
So the next orthogonal polynomial is \(x^3-\frac35x\text{.}\) An orthogonal basis makes projection easier because the Gram matrix is diagonal, so each coefficient can be computed separately.
Subsection Code interpretation: polynomial regression
The columns of
A = np.column_stack([np.ones_like(xs), xs, xs**2])
represent sampled values of \(1,\ x,\ x^2\text{.}\) Thus the approximation subspace is \(\mathcal P_{\le2}=\operatorname{span}\{1,x,x^2\}\text{.}\) The vector
c stores coefficients of the fitted quadratic \(c_0+c_1x+c_2x^2\text{.}\) The product A @ c stores fitted sampled values. The vector r = y - A @ c stores sampled residuals. The computation A.T @ r checks that the residual is orthogonal to each sampled feature column:
\begin{equation*}
\sum_i r_i=0,\qquad
\sum_i x_ir_i=0,\qquad
\sum_i x_i^2r_i=0.
\end{equation*}
This does not mean \(\mathbf r\approx\mathbf0\text{.}\) It means \(\mathbf r\) is orthogonal to the approximation subspace.
Subsection Two-variable tangent plane and sampled design matrix
We compute \(f(0,0)=1\text{,}\) \(f_x(0,0)=2\text{,}\) and \(f_y(0,0)=-1\text{.}\) Thus the tangent plane is \(T(x,y)=1+2x-y\text{.}\) The residual is \(r_T(x,y)=f(x,y)-T(x,y)=x^2+xy+2y^2+x^3\text{.}\) It satisfies
\begin{equation*}
r_T(0,0)=0,
\qquad
\nabla r_T(0,0)=
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
For sampled points \((x_i,y_i)\text{,}\) the design matrix for fitting \(q(x,y)=\alpha+\beta x+\gamma y\) is
\begin{equation*}
A=
\begin{bmatrix}
1\amp x_1\amp y_1\\
1\amp x_2\amp y_2\\
\vdots\amp\vdots\amp\vdots\\
1\amp x_N\amp y_N
\end{bmatrix}.
\end{equation*}
The sampled residual equations are
\begin{equation*}
\sum_i r_i=0,
\qquad
\sum_i x_ir_i=0,
\qquad
\sum_i y_ir_i=0.
\end{equation*}
Subsection Hessian as a quadratic form
The second partial derivatives are
\begin{equation*}
f_{xx}(x,y)=2+6x,\qquad
f_{xy}(x,y)=1,
\end{equation*}
\begin{equation*}
f_{yx}(x,y)=1,\qquad
f_{yy}(x,y)=4.
\end{equation*}
Therefore
\begin{equation*}
H_f(\mathbf0)=
\begin{bmatrix}
2\amp1\\
1\amp4
\end{bmatrix}.
\end{equation*}
With
\begin{equation*}
\mathbf h=
\begin{bmatrix}
x\\
y
\end{bmatrix},
\end{equation*}
we get
\begin{align*}
\frac12\mathbf h^T H_f(\mathbf0)\mathbf h \amp=
\frac12
\begin{bmatrix}
x\amp y
\end{bmatrix}
\begin{bmatrix}
2\amp1\\
1\amp4
\end{bmatrix}
\begin{bmatrix}
x\\
y
\end{bmatrix}\\
\amp= x^2+xy+2y^2.
\end{align*}
This is the quadratic part of the second-order Taylor polynomial at the origin. Completing the square,
\begin{equation*}
x^2+xy+2y^2
=
\left(x+\frac12y\right)^2+\frac74y^2.
\end{equation*}
This is positive for every nonzero \(\mathbf h\text{.}\) However, the origin is not a local minimum of \(f\text{,}\) because
\begin{equation*}
\nabla f(\mathbf0)=
\begin{bmatrix}
2\\
-1
\end{bmatrix}
\ne
\mathbf0.
\end{equation*}
The second derivative test applies at critical points.
