Section C.1 Unit 1 applied, geometric, and computational interpretation
These solution sketches correspond to the applied, geometric, and computational interpretation exercises at the end of Exercises.
Subsection Word-count vectors and cosine similarity
We have \(\|\mathbf{q}\|=\sqrt2\text{.}\) For \(\mathbf{D}_1=(2,0,2)\text{,}\)
\begin{equation*}
\mathbf{q}\cdot \mathbf{D}_1=4,\qquad \|\mathbf{D}_1\|=\sqrt8,
\end{equation*}
so
\begin{equation*}
\operatorname{cosim}(\mathbf{q},\mathbf{D}_1)=\frac{4}{\sqrt2\sqrt8}=1.
\end{equation*}
For \(\mathbf{D}_2=(1,1,0)\text{,}\)
\begin{equation*}
\mathbf{q}\cdot \mathbf{D}_2=1,\qquad \|\mathbf{D}_2\|=\sqrt2,
\end{equation*}
so
\begin{equation*}
\operatorname{cosim}(\mathbf{q},\mathbf{D}_2)=\frac{1}{\sqrt2\sqrt2}=\frac12.
\end{equation*}
For \(\mathbf{D}_3=(0,2,0)\text{,}\) \(\mathbf{q}\cdot\mathbf{D}_3=0\text{,}\) so \(\operatorname{cosim}(\mathbf{q},\mathbf{D}_3)=0\text{.}\) The cosine ranking is \(\mathbf{D}_1,\mathbf{D}_2,\mathbf{D}_3\text{.}\)
The vector \(\mathbf{D}_1\) has cosine similarity \(1\) because \(\mathbf{D}_1=2\mathbf{q}\text{.}\) The document \(\mathbf{D}_3\) is orthogonal to \(\mathbf{q}\text{.}\) In terms of the dictionary words, \(\mathbf{D}_3\) contains only “matrix”, while the query uses “linear” and “data”.
The Euclidean distances are
\begin{align*}
\operatorname{dist}(\mathbf{q},\mathbf{D}_1) \amp= \|\mathbf{q}-\mathbf{D}_1\| = \sqrt2,\\
\operatorname{dist}(\mathbf{q},\mathbf{D}_2) \amp= \|\mathbf{q}-\mathbf{D}_2\| = \sqrt2,\\
\operatorname{dist}(\mathbf{q},\mathbf{D}_3) \amp= \|\mathbf{q}-\mathbf{D}_3\| = \sqrt6.
\end{align*}
The distance ranking has \(\mathbf{D}_1\) and \(\mathbf{D}_2\) tied, followed by \(\mathbf{D}_3\text{.}\) It does not exactly match the cosine ranking \(\mathbf{D}_1,\mathbf{D}_2,\mathbf{D}_3\text{.}\)
Subsection Weighted average of value vectors
The weighted average is
\begin{equation*}
\frac14(10,0)+\frac14(0,10)+\frac12(10,10)=(2.5,0)+(0,2.5)+(5,5)=(7.5,7.5).
\end{equation*}
It is a weighted average because all coefficients are nonnegative and \(\frac14+\frac14+\frac12=1\text{.}\) The largest weight is \(\frac12\text{,}\) so \(\mathbf{v}_3\) receives the largest weight.
Subsection Attention bridge: hand scores versus \(K\mathbf{q}\)
The dot products are
\begin{equation*}
\mathbf{q}\cdot \mathbf{k}_{\mathrm{small}}=(1,1)\cdot (1,0)=1,
\end{equation*}
\begin{equation*}
\mathbf{q}\cdot \mathbf{k}_{\mathrm{red}}=(1,1)\cdot (0,1)=1,
\end{equation*}
\begin{equation*}
\mathbf{q}\cdot \mathbf{k}_{\mathrm{bird}}=(1,1)\cdot (1,1)=2.
\end{equation*}
Since the key vectors are rows of \(K\text{,}\)
\begin{equation*}
K\mathbf{q}=
\begin{pmatrix}1\amp 0\\ 0\amp 1\\ 1\amp 1\end{pmatrix}
\begin{pmatrix}1\\ 1\end{pmatrix}
=
\begin{pmatrix}1\\ 1\\ 2\end{pmatrix}.
\end{equation*}
The entries of \(K\mathbf{q}\) are the row dot products with \(\mathbf{q}\text{.}\) The sum of the scores is \(4\text{,}\) so \(\boldsymbol{\alpha}=(\frac14,\frac14,\frac12)\text{.}\)
Subsection Rows measure; columns contribute
The row-dot-product view gives
\begin{equation*}
A\mathbf{x}=
\begin{pmatrix}
(-1,4,-5)\cdot (2,-3,4)\\
(3,1,-2)\cdot (2,-3,4)
\end{pmatrix}
=
\begin{pmatrix}-34\\ -5\end{pmatrix}.
\end{equation*}
The column-combination view gives
\begin{equation*}
A\mathbf{x}
=
2\begin{pmatrix}-1\\ 3\end{pmatrix}
-3\begin{pmatrix}4\\ 1\end{pmatrix}
+
4\begin{pmatrix}-5\\ -2\end{pmatrix}
=
\begin{pmatrix}-34\\ -5\end{pmatrix}.
\end{equation*}
The first row measures the input by the dot product \((-1,4,-5)\cdot \mathbf{x}\text{.}\) The coordinates \(2,-3,4\) are the coefficients of the column combination.
Subsection The unit-square visualization
We have
\begin{equation*}
A\mathbf{e}_1=\begin{pmatrix}2\\ 1\end{pmatrix},\qquad A\mathbf{e}_2=\begin{pmatrix}-1\\ 1\end{pmatrix}.
\end{equation*}
Also,
\begin{equation*}
A\begin{pmatrix}1\\1\end{pmatrix}=A\mathbf{e}_1+A\mathbf{e}_2=\begin{pmatrix}1\\2\end{pmatrix}.
\end{equation*}
The four corners transform as
\begin{equation*}
A(0,0)=(0,0),\qquad A(1,0)=(2,1),\qquad A(1,1)=(1,2),\qquad A(0,1)=(-1,1).
\end{equation*}
The image of the unit square is the parallelogram with vertices \((0,0)\text{,}\) \((2,1)\text{,}\) \((1,2)\text{,}\) and \((-1,1)\text{.}\) The two transformed edges leaving the origin are \(A\mathbf{e}_1=(2,1)\) and \(A\mathbf{e}_2=(-1,1)\text{,}\) the columns of \(A\text{.}\)
Subsection Transformation matching
The matrix \(A_1=\begin{pmatrix}1\amp 1\\ 0\amp 1\end{pmatrix}\) is a horizontal shear because \(A_1(x,y)=(x+y,y)\text{.}\) The matrix \(A_2=\begin{pmatrix}0\amp -1\\ 1\amp 0\end{pmatrix}\) is a \(90^\circ\) counterclockwise rotation because \(\mathbf{e}_1\mapsto \mathbf{e}_2\) and \(\mathbf{e}_2\mapsto -\mathbf{e}_1\text{.}\)
The matrix \(A_3=\begin{pmatrix}1\amp 0\\ 0\amp 0\end{pmatrix}\) is projection onto the \(x\)-axis because \((x,y)\mapsto (x,0)\text{.}\) The matrix \(A_4=\begin{pmatrix}-1\amp 0\\ 0\amp 1\end{pmatrix}\) is reflection across the \(y\)-axis because \((x,y)\mapsto (-x,y)\text{.}\) The matrix \(A_5=\begin{pmatrix}2\amp 0\\ 0\amp 1\end{pmatrix}\) is a horizontal stretch because \((x,y)\mapsto (2x,y)\text{.}\)
Thus the matches are \(A_1\to 4\text{,}\) \(A_2\to 5\text{,}\) \(A_3\to 2\text{,}\) \(A_4\to 1\text{,}\) and \(A_5\to 3\text{.}\)
Subsection Composition order: \(RS\) versus \(SR\)
We compute
\begin{equation*}
RS=
\begin{pmatrix}0\amp -1\\ 1\amp 0\end{pmatrix}
\begin{pmatrix}2\amp 0\\ 0\amp 1\end{pmatrix}
=
\begin{pmatrix}0\amp -1\\ 2\amp 0\end{pmatrix},
\end{equation*}
while
\begin{equation*}
SR=
\begin{pmatrix}2\amp 0\\ 0\amp 1\end{pmatrix}
\begin{pmatrix}0\amp -1\\ 1\amp 0\end{pmatrix}
=
\begin{pmatrix}0\amp -2\\ 1\amp 0\end{pmatrix}.
\end{equation*}
The products are not equal. The product \(RS\) means apply \(S\) first, then \(R\text{.}\) The two orders differ because horizontal stretching before rotation is not the same as rotation before horizontal stretching.
Subsection Affine but not linear
We have \(T(0,0)=(1,0)\text{.}\) If \(T(\mathbf{x})=A\mathbf{x}\text{,}\) then \(T(\mathbf{0})=A\mathbf{0}=\mathbf{0}\text{.}\) But this transformation sends \(\mathbf{0}\) to \((1,0)\text{,}\) so it cannot be written as \(A\mathbf{x}\text{.}\)
It can be written as
\begin{equation*}
T(\mathbf{x})=I\mathbf{x}+\begin{pmatrix}1\\0\end{pmatrix}.
\end{equation*}
So \(T\) is affine but not linear.
Subsection Code interpretation: sorting similarity scores
The command
np.argsort(scores) returns the indices that sort the scores from smallest to largest:
np.argsort(scores)
gives
array([2, 1, 0])
array([0, 1, 2])
Therefore
ranking is
array(['D1', 'D2', 'D3'])
The code ranks documents by the given similarity scores. If the scores are cosine similarities, then the ranking is based on direction after normalization, not Euclidean distance or alphabetical order.
Subsection Transformer shape check
Since \(X\in \mathbb{R}^{6\times 5}\) and \(W_Q,W_K,W_V\in \mathbb{R}^{5\times 2}\text{,}\) we have
\begin{equation*}
Q=XW_Q\in \mathbb{R}^{6\times 2},\qquad K=XW_K\in \mathbb{R}^{6\times 2},\qquad V=XW_V\in \mathbb{R}^{6\times 2}.
\end{equation*}
Since \(K^T\in \mathbb{R}^{2\times 6}\text{,}\) we get \(QK^T\in \mathbb{R}^{6\times 6}\text{.}\) The \((i,j)\)-entry of \(QK^T\) is the dot product of query row \(i\) with key row \(j\text{.}\) It is a relevance score.
In the notation of the exercise, \(S=QK^T\) has shape \(6\times 6\text{,}\) so the later attention-weight matrix \(A_{\mathrm{att}}\) also has shape \(6\times 6\text{.}\) Then \(H=A_{\mathrm{att}}V\) has shape \(6\times 2\text{.}\) The product \(A_{\mathrm{att}}V\) forms the weighted averages of the value vectors. In this convention, token vectors are stored as rows of \(X\text{.}\)
