Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 1.5 Coding recap

Subsection Linked notebook

Run Lab U1: Vectors, similarity, attention, and matrix actions
 1 
sebroc.github.io/MATH345-Course-Materials/labs/#lab-u1
. The main sequence practices vector applications, document similarity, token attention, matrix-vector products, geometric matrix actions, applying one matrix to many points, composition, affine maps, and short code outputs. The final part contains review questions and a short forward look to square-grid visualizations.
🔗
For a quick reference on arrays, matrix products, norms, rankings, and slicing, see the programming appendix sections B.2, B.4, B.5, B.6, and B.3.
🔗
🔗

Subsection Review activities

Activity 1.5.1. Reading a ranking (U1-LO2, U1-LO8).

Continuing the cosine-similarity ranking from Activity 1.1.25, the code is
🔗
scores = np.array([1.0, 0.5, 0.0])
doc_names = np.array(["D1", "D2", "D3"])
ranking = doc_names[np.argsort(scores)[::-1]]
ranking
  1. What is the value of ranking?
    🔗
    🔗
  2. What is being ranked?
    🔗
    🔗
  3. Is this ranking based on distance or direction?
    🔗
    🔗
Solution.
The value is ["D1", "D2", "D3"]. The documents are being ranked by cosine similarity with the query. This is a direction-based ranking because cosine similarity compares direction after normalization.
🔗
🔗
🔗

Activity 1.5.2. Reading an attention computation (U1-LO4, U1-LO8).

In Activity 1.3.13, the code was
🔗
s = K @ q
alpha = s / s.sum()
out = alpha @ V
  1. What does s contain?
    🔗
    🔗
  2. Why do the entries of alpha add to \(1\text{?}\)
    🔗
    🔗
  3. What does out represent?
    🔗
    🔗
Solution.
The vector s contains dot-product scores. The vector alpha contains weights because the scores were divided by their sum. The vector out is a weighted average of the value vectors stored in the rows of V.
🔗
🔗
🔗

Activity 1.5.3. Reading a geometric computation (U1-LO5, U1-LO8).

In Activity 1.4.9, the code was
🔗
X = np.array([
    [0, 1, 1, 0, 0],
    [0, 0, 1, 1, 0],
], dtype=float)

A = np.array([
    [1, 1],
    [0, 1],
], dtype=float)

Y = A @ X
Y
  1. What are the columns of X?
    🔗
    🔗
  2. What are the columns of Y?
    🔗
    🔗
  3. Which transformation from the gallery is this?
    🔗
    🔗
Solution.
The columns of X are vertices of the unit square. The columns of Y are their images under \(A\text{.}\) This is a horizontal shear because \(A\) sends \((x,y)\) to \((x+y,y)\text{.}\)
🔗
🔗
🔗

Activity 1.5.4. Reading composition order (U1-LO6). 

S = np.array([
    [2., 0.],
    [0., 1.],
])

R = np.array([
    [0., -1.],
    [1.,  0.],
])

R @ S, S @ R
  1. Which product applies S first and then R?
    🔗
    🔗
  2. Are the two products equal?
    🔗
    🔗
  3. What does the difference mean geometrically?
    🔗
    🔗
Solution.
The product R @ S applies S first and then R, because the rightmost matrix acts first. The two products are not equal:
\begin{equation*} RS=\begin{bmatrix}0\amp -1\\2\amp0\end{bmatrix}, \qquad SR=\begin{bmatrix}0\amp -2\\1\amp0\end{bmatrix}. \end{equation*}
Rotating after a horizontal stretch is not the same as stretching after a rotation.
🔗
🔗
🔗

Activity 1.5.5. Reading an affine rule (U1-LO7). 

W = np.array([
    [1., 1.],
    [0., 2.],
])

b = np.array([3., -1.])

def f(x):
    return W @ x + b
Is f linear? Justify your answer using the zero vector.
🔗
Solution.
No. A linear map must send \(\mathbf{0}\) to \(\mathbf{0}\text{.}\) But \(f(\mathbf{0})=W\mathbf{0}+\mathbf{b}=\mathbf{b}\text{,}\) and \(\mathbf{b}\) is not the zero vector. The rule is affine, but not linear.
🔗
🔗
🔗
🔗
🔗
PrevTopNext