Skip to main content
Contents Dark Mode Prev Up Next \(\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\dimens}{\operatorname{dim}}
\DeclareMathOperator{\row}{\operatorname{row}}
\DeclareMathOperator{\col}{\operatorname{col}}
\newcommand{\im}{\operatorname{im}}
\newcommand{\nulls}{\operatorname{null}}
\newcommand{\minor}{\operatorname{minor}}
\newcommand{\spans}{\operatorname{span}}
\newcommand{\nullity}{\operatorname{nullity}}
\newcommand{\kers}{\operatorname{ker}}
\newcommand{\proj}{\operatorname{proj}}
\newcommand{\diag}{\operatorname{diag}}
\newcommand{\Tr}{\operatorname{Tr}}
\newcommand{\rank}{\operatorname{rank}}
\newcommand{\lt}{<}
\newcommand{\gt}{>}
\newcommand{\amp}{&}
\definecolor{fillinmathshade}{gray}{0.9}
\newcommand{\fillinmath}[1]{\mathchoice{\colorbox{fillinmathshade}{$\displaystyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\textstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptscriptstyle\phantom{\,#1\,}$}}}
\)
Section 1.5 Applications and computation recap
Subsection About @
The same symbol
@ can mean different linear-algebra operations. Its meaning depends on the shapes of the objects being multiplied.
A short computation involving
@ should be read in three steps.
Table 1.5.1. Reading short computations
u @ vdot product; alignment or score
A @ xWhat are the rows and columns of
A?
X @ qAre rows of
X being scored by
q?
several dot products at once
K @ qWhich earlier activity computed these dot products by hand?
alpha @ VWhich vectors are being averaged?
weighted average of rows of
V
A @ XAre columns of
X input vectors?
one matrix applied to many inputs
Q @ K.TWhat are the query-key scores?
all query-key dot products
W @ x + bWhat is
\(f(\mathbf{0})\text{?}\)
affine map; linear only when
\(\mathbf{b}=\mathbf{0}\)
Subsection Review activities
Activity 1.5.1 . Reading a ranking.
Continuing the cosine-similarity ranking from
Activity 1.1.7 , the code is
scores = np.array([1.0, 0.5, 0.0])
doc_names = np.array(["D1", "D2", "D3"])
ranking = doc_names[np.argsort(scores)[::-1]]
ranking
What is the value of
ranking?
Is this ranking based on distance or direction?
Solution .
The value is
["D1", "D2", "D3"]. The documents are being ranked by cosine similarity with the query. This is a direction-based ranking because cosine similarity compares direction after normalization.
Activity 1.5.2 . Reading an attention computation.
s = K @ q
alpha = s / s.sum()
out = alpha @ V
Why do the entries of
alpha add to
\(1\text{?}\)
Solution .
The vector
s contains dot-product scores. The vector
alpha contains weights because the scores were divided by their sum. The vector
out is a weighted average of the value vectors stored in the rows of
V.
Activity 1.5.3 . Reading a geometric computation.
X = np.array([
[0, 1, 1, 0, 0],
[0, 0, 1, 1, 0],
], dtype=float)
A = np.array([
[1, 1],
[0, 1],
], dtype=float)
Y = A @ X
Y
What are the columns of
X?
What are the columns of
Y?
Which transformation from the gallery is this?
Solution .
The columns of
X are vertices of the unit square. The columns of
Y are their images under
\(A\text{.}\) This is a horizontal shear because
\(A\) sends
\((x,y)\) to
\((x+y,y)\text{.}\)
Activity 1.5.4 . Reading composition order.
S = np.array([
[2., 0.],
[0., 1.],
])
R = np.array([
[0., -1.],
[1., 0.],
])
R @ S, S @ R
Which product applies
S first and then
R?
Are the two products equal?
What does the difference mean geometrically?
Solution .
The product R @ S applies S first and then R, because the rightmost matrix acts first. The two products are not equal:
\begin{equation*}
RS=\begin{bmatrix}0\amp -1\\2\amp0\end{bmatrix},
\qquad
SR=\begin{bmatrix}0\amp -2\\1\amp0\end{bmatrix}.
\end{equation*}
Rotating after a horizontal stretch is not the same as stretching after a rotation.
Activity 1.5.5 . Reading an affine rule.
W = np.array([
[1., 1.],
[0., 2.],
])
b = np.array([3., -1.])
def f(x):
return W @ x + b
Is
f linear? Justify your answer using the zero vector.
Solution .
No. A linear map must send
\(\mathbf{0}\) to
\(\mathbf{0}\text{.}\) But
\(f(\mathbf{0})=W\mathbf{0}+\mathbf{b}=\mathbf{b}\text{,}\) and
\(\mathbf{b}\) is not the zero vector. The rule is affine, but not linear.
Subsection Linked notebook
Run
Lab U1: Vectors, similarity, attention, and matrix actions . The core path practices vector applications, document similarity, token attention, matrix-vector products, geometric matrix actions, applying one matrix to many points, composition, affine maps, and short code outputs. The final part contains review questions and a short forward look to square-grid visualizations.
For a quick reference on arrays, matrix products, norms, rankings, and slicing, see the programming appendix sections
B.2 ,
B.4 ,
B.5 ,
B.6 , and
B.3 .
