This worksheet asks you to read a simplified transformer block using tools from the course: vectors, dot products, matrix multiplication, affine maps, null spaces, nonlinear functions, Jacobians, gradients, optimization, rank, and low-rank structure.
The goal is not to teach the full engineering of large language models. The goal is to recognize the mathematics and explain how information moves through a simplified model.
How to use this worksheet. Work through the activities in order. Each activity asks you to identify which course tool is being used, rather than to learn a new transformer topic from scratch.
This worksheet does not cover tokenization, real training datasets, hardware, backpropagation through attention, model evaluation, alignment, fine-tuning pipelines, or prompt engineering. Those topics belong to other courses. Here we focus on reading a simplified computation.
. The notebook is for interpreting shapes, dot products, masks, weighted averages, logits, and one small loss; it is not a real model-training assignment.
For attention-block code involving row-wise softmax, masks, np.triu, .astype(bool), .copy(), np.outer, and plotting, see the programming quick-reference sections B.13 and B.14.
Read a token matrix \(X\) and check the shapes of \(Q=XW_Q\text{,}\)\(K=XW_K\text{,}\)\(V=XW_V\text{,}\)\(S=QK^T\text{,}\) and \(H=A_{\mathrm{att}}V\text{.}\)
Notation card: one simplified attention head. Suppose a sequence has \(L\) tokens, each represented by a vector in \(\mathbb R^d\text{.}\) Store the token vectors as rows of
In transformer attention, the weighting rule is usually row-wise softmax, possibly after scaling and masking. Unit 3 treats row-wise softmax as a nonlinear function.
A vertical flowchart starts with \(X\) and proceeds downward through boxes for linear maps to \(Q\text{,}\)\(K\text{,}\) and \(V\text{,}\) dot-product scores, softmax or masking, weighted values, residual update, and a feed-forward layer. Red arrows connect the boxes in sequence. The final box indicates that the block outputs an updated \(X\text{.}\)
FigureA.0.3.A transformer block is a nonlinear information processor built from linear maps, dot products, weighted averages, and nonlinear operations.
\(Q,K,V\) are \(4\times 2\text{.}\) The score matrix \(S=QK^T\) is \(4\times 4\text{.}\) The attention-weight matrix \(A_{\mathrm{att}}\) is \(4\times 4\text{,}\) so \(H=A_{\mathrm{att}}V\) is \(4\times 2\text{.}\) The entry \(S_{ij}\) compares the query for token \(i\) with the key for token \(j\text{.}\) Row \(i\) of \(A_{\mathrm{att}}\) gives the weights used to combine rows of \(V\text{.}\) The product \(A_{\mathrm{att}}V\) forms weighted averages. The weighting rule that turns scores into attention weights is nonlinear. Units 1, 2, 3, 5, and 6 supply the main tools.
1 is linear in \(X\text{.}\) 2 is affine in \(x\text{.}\) 3 is nonlinear. 4 is nonlinear. 5 is bilinear in the pair \((A_{\mathrm{att}},V)\text{,}\) and linear in either one if the other is held fixed. 6 is linear as a map of the pair \((x,\Delta x)\text{,}\) and represents vector addition. 7 is generally nonlinear. Thus a transformer-style block can contain many matrix multiplications without being a linear map.
1 can appear in both settings. 2 is attention-like. 3 is projection. 4 is projection. 5 is attention-like. 6 is projection. The main distinction is that attention uses scores to mix value vectors, while orthogonal projection solves a closest-vector problem with an orthogonal residual.
The vector \(z\) is in \(\operatorname{null}(W)\text{.}\) Hidden vectors that differ by \(z\) give the same scores. The bias shifts the scores but does not change the condition \(Wz=0\text{.}\)
The parameters are collected in \(\theta\text{.}\) The loss \(L(\theta)\) is being minimized. The learning rate is \(\alpha\text{.}\) The gradient \(\nabla L(\theta_k)\) gives the local direction of steepest increase of the loss, so the negative gradient is used for descent. This is an optimization problem because training means adjusting parameters to reduce a loss.
The product \(BC\) has shape \(m\times n\text{,}\) matching \(W\text{.}\) Since \(BC\) factors through an \(r\)-dimensional middle space, its rank is at most \(r\text{.}\) The two factors store \(mr+rn=r(m+n)\) entries, compared with \(mn\) entries for \(W\text{.}\) This does not imply that \(W_{\mathrm{new}}\) has low rank. The update \(BC\) has low rank; the whole matrix \(W+BC\) may still have large rank.
Main idea. A transformer-style block is not one linear map. It is a nonlinear function built from linear maps, dot products, weighted averages, affine maps, nonlinear weighting rules, residual additions, normalization, and optimization.
Course synthesis. Unit 1 explains vectors, dot products, matrix products, weighted averages, and affine maps. Unit 2 explains forgotten directions. Unit 3 explains nonlinear functions and local linearization. Unit 4 separates attention from projection. Unit 5 explains loss and gradient descent. Unit 6 explains rank and low-rank updates.
