MATH 345: Linear Algebra and Optimization

Activity 1.1.3. Vectors as displacements in space (U1-LO2).

Activity 1.1.5. Vector length and geometry (U1-LO1).

Solution.

Let \(\mathbf{v} = (x,y)\) be an arbitrary \(2\)-vector. Consider a triangle with vertices \(P(0,0)\text{,}\) \(Q(x,0)\text{,}\) and \(R(x,y)\) (see Figure 1.1.6).

🔗



The figure shows \(x\)- and \(y\)-axes with a vector \(\mathbf{v}\) drawn from \(P(0,0)\) to \(R(x,y)\text{.}\) A horizontal segment from \(P\) to \(Q(x,0)\) and a vertical segment from \(Q\) to \(R\) form a right triangle whose hypotenuse is the vector.

🔗

This triangle is right-angled (the two legs are horizontal and vertical), and the hypotenuse is the line-segment upon which the vector \(\mathbf{v}\) is built. The length of the leg from \(P(0,0)\) to \(Q(x,0)\) is \(|x|\text{,}\) and the length of the leg from \(Q(x,0)\) to \(R(x,y)\) is \(|y|\text{.}\) By Pythagoras’ theorem, we conclude that the length of the hypotenuse is

\begin{equation*} \sqrt{ |x|^2 + |y|^2 } = \sqrt{x^2 + y^2} = \| \mathbf{v} \|\text{.} \end{equation*}

🔗

🔗

Activity 1.1.12. A weighted average (U1-LO8).

Activity 1.1.18. Computing cosine similarity (U1-LO1).

Activity 1.1.20. Finding the angle between vectors (U1-LO1).

Activity 1.1.24. Word-count vectors (U1-LO1, U1-LO2).

Activity 1.1.25. The same rankings in code (U1-LO2, U1-LO8).

import numpy as np

D1 = np.array([3., 0., 3.])
D2 = np.array([1., 1., 0.])
D3 = np.array([0., 2., 0.])
q = np.array([1., 0., 1.])

score1 = (D1 @ q) / (np.linalg.norm(D1) * np.linalg.norm(q))
score2 = (D2 @ q) / (np.linalg.norm(D2) * np.linalg.norm(q))
score3 = (D3 @ q) / (np.linalg.norm(D3) * np.linalg.norm(q))

distance1 = np.linalg.norm(D1 - q)
distance2 = np.linalg.norm(D2 - q)
distance3 = np.linalg.norm(D3 - q)

(score1, score2, score3), (distance1, distance2, distance3)

((1.0, 0.5, 0.0),
 (2.8284271247461903, 1.4142135623730951, 2.449489742783178))

1. Which score corresponds to \(\mathbf{D}_2\text{?}\)

   🔗
   🔗
2. Where does the code compute \(\mathbf{D}_1\cdot\mathbf{q}\text{?}\)

   🔗
   🔗
3. Which document is most similar to the query by cosine similarity?

   🔗
   🔗
4. Which line computes \(\|\mathbf{D}_2-\mathbf{q}\|\text{?}\)

   🔗
   🔗
5. Rank the documents from closest to farthest from the query by Euclidean distance. How does this ranking compare with the cosine-similarity ranking?

   🔗
   🔗

Activity 1.1.29. Gathering context for the bird position (U1-LO1, U1-LO8).

1. Compute the three scores

   \begin{equation*} \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{small}},\qquad \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{red}},\qquad \mathbf{q}_{\text{bird}}\cdot \mathbf{k}_{\text{bird}}. \end{equation*}

   🔗
   🔗
2. Normalize the positive scores by dividing each score by the sum of all three scores. Call the resulting weights

   \begin{equation*} \boldsymbol{\alpha}= \begin{bmatrix} \alpha_{\text{small}}\\ \alpha_{\text{red}}\\ \alpha_{\text{bird}} \end{bmatrix}. \end{equation*}

   🔗
   🔗
3. The coordinates of the value vectors are abstract toy features. Use

   \begin{equation*} \mathbf{v}_{\text{small}}=\begin{bmatrix}4\\0\end{bmatrix},\qquad \mathbf{v}_{\text{red}}=\begin{bmatrix}0\\4\end{bmatrix},\qquad \mathbf{v}_{\text{bird}}=\begin{bmatrix}4\\4\end{bmatrix} \end{equation*}

   to compute

   \begin{equation*} \mathbf{h}_{\text{bird}} = \alpha_{\text{small}}\mathbf{v}_{\text{small}} + \alpha_{\text{red}}\mathbf{v}_{\text{red}} + \alpha_{\text{bird}}\mathbf{v}_{\text{bird}}. \end{equation*}

   🔗
   🔗
4. Which token receives the largest weight? What total weight is assigned to “small” and “red” together? What does this say about the information gathered at the bird position?

   🔗
   🔗