Skip to main content\(\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\dimens}{\operatorname{dim}}
\DeclareMathOperator{\row}{\operatorname{row}}
\DeclareMathOperator{\col}{\operatorname{col}}
\newcommand{\im}{\operatorname{im}}
\newcommand{\nulls}{\operatorname{null}}
\newcommand{\minor}{\operatorname{minor}}
\newcommand{\spans}{\operatorname{span}}
\newcommand{\nullity}{\operatorname{nullity}}
\newcommand{\kers}{\operatorname{ker}}
\newcommand{\proj}{\operatorname{proj}}
\newcommand{\diag}{\operatorname{diag}}
\newcommand{\Tr}{\operatorname{Tr}}
\newcommand{\rank}{\operatorname{rank}}
\newcommand{\lt}{<}
\newcommand{\gt}{>}
\newcommand{\amp}{&}
\definecolor{fillinmathshade}{gray}{0.9}
\newcommand{\fillinmath}[1]{\mathchoice{\colorbox{fillinmathshade}{$\displaystyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\textstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptscriptstyle\phantom{\,#1\,}$}}}
\)
Section B.13 Boolean Masks, Copying, and Attention Helpers
This optional section helps you read the array-manipulation code used in the optional attention material.
S = np.array([[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
[7.0, 8.0, 9.0]])
mask = np.triu(np.ones_like(S), k=1).astype(bool)
S_masked = S.copy()
S_masked[mask] = -1e9
def softmax_rows(S):
S_shifted = S - np.max(S, axis=1, keepdims=True)
e = np.exp(S_shifted)
return e / e.sum(axis=1, keepdims=True)
Matching ones.
Read as. Ones with the same shape as
S.
Used for. Mask construction.
Upper triangle.
Read as. Entries above the diagonal.
Boolean conversion.
Read as. Convert to
True/
False.
Shape/return. A Boolean array.
Independent copy.
Read as. Make an independent copy.
Used for. Safe modification.
Mask assignment.
Read as. Change selected entries.
Shape/return. A modified array.
Used for. Masked attention scores.
Row maximum.
np.max(S, axis=1, keepdims=True)
Shape/return. A column-shaped array.
Used for. Stable softmax.
Exponential.
Read as. Entry-by-entry exponential.
Row normalization.
e / e.sum(axis=1, keepdims=True)
Read as. Row normalization.
Shape/return. An array with row sums 1.
Used for. Attention weights.
Outer product.
Used for. Rank-one update.
