Skip to main content
Contents Dark Mode Prev Up Next \(\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\dimens}{\operatorname{dim}}
\DeclareMathOperator{\row}{\operatorname{row}}
\DeclareMathOperator{\col}{\operatorname{col}}
\newcommand{\im}{\operatorname{im}}
\newcommand{\nulls}{\operatorname{null}}
\newcommand{\minor}{\operatorname{minor}}
\newcommand{\spans}{\operatorname{span}}
\newcommand{\nullity}{\operatorname{nullity}}
\newcommand{\kers}{\operatorname{ker}}
\newcommand{\proj}{\operatorname{proj}}
\newcommand{\diag}{\operatorname{diag}}
\newcommand{\Tr}{\operatorname{Tr}}
\newcommand{\rank}{\operatorname{rank}}
\newcommand{\lt}{<}
\newcommand{\gt}{>}
\newcommand{\amp}{&}
\definecolor{fillinmathshade}{gray}{0.9}
\newcommand{\fillinmath}[1]{\mathchoice{\colorbox{fillinmathshade}{$\displaystyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\textstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptscriptstyle\phantom{\,#1\,}$}}}
\)
Section 2.1 Understanding a linear map: a preview
In
Unit 1 , a matrix defined a linear map
\begin{equation*}
\mathbf{x}\mapsto A\mathbf{x}.
\end{equation*}
This unit answers the following questions about the behavior of this map. The second column previews the concepts introduced later in the unit for this purpose.
When do different inputs give the same output?
\(A(\mathbf{x}+\mathbf{z})=A\mathbf{x}\)
What input changes does \(A\) forget?
\(\operatorname{null}(A)\)
When can a desired output be hit?
\(A\mathbf{x}=\mathbf{b}\)
What outputs can \(A\) produce?
\(\operatorname{im}(A)\)
Can the map be reversed?
\(A^{-1}\text{,}\) when it exists
We give several preliminary examples.
Activity 2.1.1 . A map that forgets height.
Let
\begin{equation*}
P=
\begin{bmatrix}
1&0&0\\
0&1&0
\end{bmatrix}.
\end{equation*}
First compute \(P\mathbf{x}_1\text{,}\) \(P\mathbf{x}_2\text{,}\) and \(P\mathbf{x}_3\text{,}\) where
\begin{equation*}
\mathbf{x}_1=\begin{bmatrix}2\\1\\0\end{bmatrix},\qquad
\mathbf{x}_2=\begin{bmatrix}2\\1\\5\end{bmatrix},\qquad
\mathbf{x}_3=\begin{bmatrix}2\\1\\-3\end{bmatrix}.
\end{equation*}
Which input coordinate changed?
Which part of the input did the output remember?
Which part of the input did the output forget?
Suppose the desired output is
\((2,1)\text{.}\) What simple equations must the input coordinates satisfy?
Following up on the previous question, how many choices of the third coordinate work?
Solution .
We compute
\begin{equation*}
P\begin{bmatrix}x\\y\\z\end{bmatrix}
=
\begin{bmatrix}
1x+0y+0z\\
0x+1y+0z
\end{bmatrix}
=
\begin{bmatrix}x\\y\end{bmatrix}.
\end{equation*}
So the outputs are
\begin{equation*}
P\mathbf{x}_1=\begin{bmatrix}2\\1\end{bmatrix},\qquad
P\mathbf{x}_2=\begin{bmatrix}2\\1\end{bmatrix},\qquad
P\mathbf{x}_3=\begin{bmatrix}2\\1\end{bmatrix}.
\end{equation*}
The third coordinate changed, but the output did not. The map remembers the first two coordinates and forgets height.
To get the desired output \((2,1)\text{,}\) an input \(\mathbf{x}=(x,y,z)\) must satisfy
\begin{equation*}
x=2,\qquad y=1.
\end{equation*}
There is no equation restricting \(z\text{,}\) so many (in fact, infinitely many!) inputs work:
\begin{equation*}
\mathbf{x}=\begin{bmatrix}2\\1\\z\end{bmatrix}.
\end{equation*}
Activity 2.1.2 . The same height loss in code.
The following code repeats the same experiment.
import numpy as np
P = np.array([[1, 0, 0],
[0, 1, 0]])
x1 = np.array([4, -2, 0])
x2 = np.array([4, -2, 6])
x3 = np.array([4, -2, -5])
P @ x1, P @ x2, P @ x3
Predict the three code outputs.
Solution .
The three outputs are
[4, -2],
[4, -2], and
[4, -2]. The third coordinate changes across the inputs, but that change does not appear in the output.
Activity 2.1.3 . Can this map produce the requested output?
Let
\begin{equation*}
A=
\begin{bmatrix}
1&0\\
0&1\\
1&1
\end{bmatrix}.
\end{equation*}
First compute \(A\mathbf{x}_1\text{,}\) \(A\mathbf{x}_2\text{,}\) and \(A\mathbf{x}_3\text{,}\) where
\begin{equation*}
\mathbf{x}_1=\begin{bmatrix}1\\2\end{bmatrix},\qquad
\mathbf{x}_2=\begin{bmatrix}1\\3\end{bmatrix},\qquad
\mathbf{x}_3=\begin{bmatrix}0\\0\end{bmatrix}.
\end{equation*}
Can this map produce the output
\((1,2,3)\text{?}\) If yes, give an input.
What rule do the outputs above all obey?
For the desired output
\((1,2,4)\text{,}\) what equations would the unknown input
\(\mathbf{x}=(x,y)\) have to satisfy?
Following up on the previous question, can all of those equations be true at the same time?
Solution .
We compute
\begin{equation*}
A\begin{bmatrix}x\\y\end{bmatrix}
=
\begin{bmatrix}
1x+0y\\
0x+1y\\
1x+1y
\end{bmatrix}
=
\begin{bmatrix}
x\\
y\\
x+y
\end{bmatrix}.
\end{equation*}
So the outputs are
\begin{equation*}
A\mathbf{x}_1=\begin{bmatrix}1\\2\\3\end{bmatrix},\qquad
A\mathbf{x}_2=\begin{bmatrix}1\\3\\4\end{bmatrix},\qquad
A\mathbf{x}_3=\begin{bmatrix}0\\0\\0\end{bmatrix}.
\end{equation*}
The output \((1,2,3)\) is produced by the input \(\mathbf{x}_1\text{.}\) The outputs above all obey the rule: third coordinate equals first coordinate plus second coordinate.
For the desired output \((1,2,4)\text{,}\) the unknown input \(\mathbf{x}=(x,y)\) would have to satisfy
\begin{equation*}
x=1,\qquad y=2,\qquad x+y=4.
\end{equation*}
The first two equations force \(x+y=3\text{,}\) not \(4\text{.}\) So these equations cannot all be true at the same time, and the output \((1,2,4)\) cannot be produced.
Activity 2.1.4 . Two inputs, one output.
Activity 2.1.5 . Same output in code.
The following code compares two more inputs for the same map.
F = np.array([[1, 1, 0],
[0, 1, 1]])
u = np.array([0, 1, 2])
v = np.array([1, 0, 3])
d = v - u
F @ u, F @ v, d, F @ d
Predict the four code outputs.
Solution .
The four outputs are
[1, 3],
[1, 3],
[1, -1, 1], and
[0, 0]. Thus
F @ u and
F @ v are the same, and
F @ d is
[0, 0].
The equation F @ d = [0, 0] represents
\begin{equation*}
d_1+d_2=0,\qquad d_2+d_3=0.
\end{equation*}
The output does not always determine the input uniquely.
Activity 2.1.6 . Output scores and a hidden direction.
A final model layer can convert a hidden vector into output scores by
\begin{equation*}
\boldsymbol{\ell}=W\mathbf{h}+\mathbf{b}.
\end{equation*}
Let
\begin{equation*}
W=
\begin{bmatrix}
1&1&0\\
0&1&1
\end{bmatrix},
\qquad
\mathbf{b}=\begin{bmatrix}0.5\\-0.5\end{bmatrix},
\qquad
\mathbf{h}=\begin{bmatrix}2\\0\\1\end{bmatrix},
\qquad
\mathbf{d}=\begin{bmatrix}1\\-1\\1\end{bmatrix}.
\end{equation*}
Compute
\(W\mathbf{h}+\mathbf{b}\text{.}\)
Compute
\(W\mathbf{d}\text{.}\)
Use part 2 to compute
\(W(\mathbf{h}+\mathbf{d})+\mathbf{b}\text{.}\)
Are
\(\mathbf{h}\) and
\(\mathbf{h}+\mathbf{d}\) the same hidden vector?
Are their output-score vectors the same?
Did the bias vector
\(\mathbf{b}\) affect which hidden-vector change was invisible?
Solution .
First,
\begin{equation*}
W\mathbf{h}=\begin{bmatrix}2\\1\end{bmatrix},\qquad
W\mathbf{h}+\mathbf{b}=\begin{bmatrix}2.5\\0.5\end{bmatrix}.
\end{equation*}
Next,
\begin{equation*}
W\mathbf{d}
=
\begin{bmatrix}
1-1\\
-1+1
\end{bmatrix}
=
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation*}
Therefore,
\begin{equation*}
\begin{aligned}
W(\mathbf{h}+\mathbf{d})+\mathbf{b}
&= W\mathbf{h}+W\mathbf{d}+\mathbf{b}\\
&= W\mathbf{h}+\mathbf{b}\\
&= \begin{bmatrix}2.5\\0.5\end{bmatrix}.
\end{aligned}
\end{equation*}
The hidden vectors
\(\mathbf{h}\) and
\(\mathbf{h}+\mathbf{d}\) are different, but they produce the same output scores. The change
\(\mathbf{d}\) is invisible to
\(W\) because
\(W\mathbf{d}=\mathbf{0}\text{.}\)
The bias vector shifts the output scores, but it does not affect which hidden-vector changes are invisible.
Here is the same issue in a data-matrix setting.
Activity 2.1.7 . Two coefficient choices, same predictions.
A data matrix stores three measured features for four observations:
\begin{equation*}
X=
\begin{bmatrix}
1&2&3\\
2&4&6\\
0&1&1\\
1&-1&0
\end{bmatrix}.
\end{equation*}
Observe that the third feature is the sum of the first two features in every row.
Consider the two coefficient vectors
\begin{equation*}
\mathbf{c}=\begin{bmatrix}2\\1\\0\end{bmatrix},\qquad
\mathbf{c}'=\begin{bmatrix}1\\0\\1\end{bmatrix}.
\end{equation*}
In this example, the linear map takes a coefficient vector \(\mathbf{c}\) as input and returns the prediction vector \(X\mathbf{c}\text{.}\)
Compute
\(X\mathbf{c}\text{.}\)
Compute
\(X\mathbf{c}'\text{.}\)
Are the coefficient vectors the same?
Are the predictions the same?
Let
\(\mathbf{d}=\mathbf{c}-\mathbf{c}'\text{.}\) What equations does
\(X\mathbf{d}=\mathbf{0}\) represent?
Solution .
We have
\begin{equation*}
X\mathbf{c}=\begin{bmatrix}4\\8\\1\\1\end{bmatrix},\qquad
X\mathbf{c}'=\begin{bmatrix}4\\8\\1\\1\end{bmatrix}.
\end{equation*}
The coefficient vectors are different, but the predictions are the same.
The difference is
\begin{equation*}
\mathbf{d}=\mathbf{c}-\mathbf{c}'=\begin{bmatrix}1\\1\\-1\end{bmatrix}.
\end{equation*}
The equation \(X\mathbf{d}=\mathbf{0}\text{,}\) with \(\mathbf{d}=(d_1,d_2,d_3)\text{,}\) represents
\begin{equation*}
\begin{aligned}
d_1+2d_2+3d_3 &= 0,\\
2d_1+4d_2+6d_3 &= 0,\\
d_2+d_3 &= 0,\\
d_1-d_2 &= 0.
\end{aligned}
\end{equation*}
The vector \((1,1,-1)\) satisfies these equations. This happens because the third feature repeats information already contained in the first two features: feature 3 equals feature 1 plus feature 2 in every row.
Each preview led to the same kind of question: can certain linear equations in the unknown input coordinates be solved? The next section gives this question a formal name and a systematic method.
