MATH 345: Linear Algebra and Optimization

Activity 1.1.1. Vectors as displacements in space.

Activity 1.1.2.

Solution.

Let \(\mathbf{v} = (x,y)\) be an arbitrary \(2\)-vector. Consider a triangle with vertices \(P(0,0)\text{,}\) \(Q(x,0)\text{,}\) and \(R(x,y)\) (see Figure 1.1.4).

🔗



The figure shows \(x\)- and \(y\)-axes with a vector \(\mathbf{v}\) drawn from \(P(0,0)\) to \(R(x,y)\text{.}\) A horizontal segment from \(P\) to \(Q(x,0)\) and a vertical segment from \(Q\) to \(R\) form a right triangle whose hypotenuse is the vector.

🔗

This triangle is right-angled (the two legs are horizontal and vertical), and the hypotenuse is the line-segment upon which the vector \(\mathbf{v}\) is built. The length of the leg from \(P(0,0)\) to \(Q(x,0)\) is \(|x|\text{,}\) and the length of the leg from \(Q(x,0)\) to \(R(x,y)\) is \(|y|\text{.}\) By Pythagoras’ theorem, we conclude that the length of the hypotenuse is

\begin{equation*} \sqrt{ |x|^2 + |y|^2 } = \sqrt{x^2 + y^2} = \| \mathbf{v} \|\text{.} \end{equation*}

🔗

🔗

Activity 1.1.3. A weighted average.

Activity 1.1.4.

Activity 1.1.5.

Activity 1.1.6. Word-count vectors.

Activity 1.1.7. The same ranking in code.

import numpy as np

D1 = np.array([2, 0, 2], dtype=float)
D2 = np.array([1, 1, 0], dtype=float)
D3 = np.array([0, 2, 0], dtype=float)
q = np.array([1, 0, 1], dtype=float)

score1 = (D1 @ q) / (np.linalg.norm(D1) * np.linalg.norm(q))
score2 = (D2 @ q) / (np.linalg.norm(D2) * np.linalg.norm(q))
score3 = (D3 @ q) / (np.linalg.norm(D3) * np.linalg.norm(q))

score1, score2, score3

(1.0, 0.5, 0.0)

1. Which score corresponds to \(\mathbf{D}_2\text{?}\)

   🔗
   🔗
2. Where does the code compute \(\mathbf{D}_1\cdot\mathbf{q}\text{?}\)

   🔗
   🔗
3. Which document is most similar to the query by cosine similarity?

   🔗
   🔗
4. Why does the output agree with the word-count vector activity?

   🔗
   🔗

Activity 1.1.8. A tiny attention-style token-weighting calculation.

1. Compute the three scores

   \begin{equation*} \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{small}},\qquad \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{red}},\qquad \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{bird}}. \end{equation*}

   🔗
   🔗
2. Normalize the positive scores by dividing each score by the sum of all three scores. Call the resulting weights

   \begin{equation*} \boldsymbol{\alpha}= \begin{bmatrix} \alpha_{\text{small}}\\ \alpha_{\text{red}}\\ \alpha_{\text{bird}} \end{bmatrix}. \end{equation*}

   🔗
   🔗
3. Use the value vectors

   \begin{equation*} \mathbf{v}_{\text{small}}=\begin{bmatrix}4\\0\end{bmatrix},\qquad \mathbf{v}_{\text{red}}=\begin{bmatrix}0\\4\end{bmatrix},\qquad \mathbf{v}_{\text{bird}}=\begin{bmatrix}4\\4\end{bmatrix} \end{equation*}

   to compute

   \begin{equation*} \mathbf{h}_{\text{bird}} = \alpha_{\text{small}}\mathbf{v}_{\text{small}} + \alpha_{\text{red}}\mathbf{v}_{\text{red}} + \alpha_{\text{bird}}\mathbf{v}_{\text{bird}}. \end{equation*}

   🔗
   🔗
4. Which token receives the largest weight? Why?

   🔗
   🔗