Linear algebra: Words turned into arrows

Open up a language model and what you find are numbers arranged in matrices: hundreds of billions of them. Answering a question consists almost entirely of multiplying vectors by matrices. That is why GPUs, chips designed to multiply matrices while drawing video games, became the engine of AI.

Linear algebra is the language in which models represent the world. Hermann Grassmann formulated the idea of a space of arbitrary dimension in 1844, and Arthur Cayley defined the matrix product in 1858. Neither could have imagined they were writing the language of AI.

Vectors and the dot product

A vector $𝐮 = (u_{1}, \dots, u_{d}) \in ℝ^{d}$ is a list of $d$ numbers that can be seen as a point or as an arrow from the origin. The key operation is the dot product:

𝐮 \cdot 𝐯 = \sum_{i = 1}^{d} u_{i} v_{i} = ‖ 𝐮 ‖ ‖ 𝐯 ‖ \cos θ .

It measures how much two vectors “point the same way”. Dividing by their lengths gives the cosine similarity, $\cos θ$ , which is 1 for parallel vectors, 0 for perpendicular ones and −1 for opposite ones. That it never leaves this range is guaranteed by a classic:

Theorem (Cauchy–Schwarz inequality)

For all $𝐮, 𝐯 \in ℝ^{d}$ , $| 𝐮 \cdot 𝐯 | \leq ‖ 𝐮 ‖ ‖ 𝐯 ‖$ , with equality if and only if they are proportional.

Proof

If $𝐯 = 𝟎$ it is trivial. Otherwise, for every $λ \in ℝ$ , $0 \leq ‖ 𝐮 - λ 𝐯 ‖^{2} = ‖ 𝐮 ‖^{2} - 2 λ 𝐮 \cdot 𝐯 + λ^{2} ‖ 𝐯 ‖^{2}$ . A quadratic in $λ$ that is never negative has discriminant $\leq 0$ : $4 (𝐮 \cdot 𝐯)^{2} - 4 ‖ 𝐮 ‖^{2} ‖ 𝐯 ‖^{2} \leq 0$ .

This simple operation, multiply and add, is what the attention mechanism uses to decide which words of a sentence should look at which.

Embeddings: meaning as geometry

How do you turn a word into a vector? The answer comes from linguistics. In 1954 Zellig Harris formulated the distributional hypothesis: words that appear in similar contexts have similar meanings. John R. Firth summed it up in 1957: “You shall know a word by the company it keeps.”

An embedding assigns each word a vector so that words with similar contexts end up close together. In 2003 Yoshua Bengio and his colleagues trained such vectors for the first time, together with a neural network that predicted the next word. In 2013 Tomas Mikolov and his team at Google published word2vec, which trained embeddings on billions of words in a few hours, and discovered something unexpected: relationships between concepts turn into directions.

𝐯_{king} - 𝐯_{man} + 𝐯_{woman} \approx 𝐯_{queen} .

A toy embedding space in two dimensions. Real ones have hundreds to thousands, and are learned from data instead of placed by hand. Pick three words: the dashed arrow is

B \to A

moved to start at

C

, and the result is compared with every word to find the closest. If the difference “king − man” encodes something like “royalty”, adding it to “woman” should land on “queen”.

Matrices: functions that transform space

An $m \times d$ matrix $W$ defines a linear transformation $𝐱 \mapsto W 𝐱$ from $ℝ^{d}$ to $ℝ^{m}$ . It can be understood at a glance by looking at its columns: column $j$ is where the unit vector $𝐞_{j}$ goes. Since the transformation respects sums and scaling, knowing where the basis goes is enough to know everything:

W 𝐱 = x_{1} (column 1) + x_{2} (column 2) + \dots + x_{d} (column d) .

The matrix

(\begin{matrix} a & b \\ c & d \end{matrix})

transforms the grid. The coloured arrows are the columns of the matrix, that is, where

𝐞_{1}

and

𝐞_{2}

go. The determinant

a d - b c

is the factor by which areas are multiplied. If it is negative the plane is flipped, and if it is zero it collapses onto a line and the lost information can no longer be recovered.

Multiplying matrices is composing transformations: $W_{2} (W_{1} 𝐱) = (W_{2} W_{1}) 𝐱$ . This innocent-looking property has a decisive consequence for neural networks:

Observation (linear layers collapse)

A composition of linear transformations is linear: $W_{L} \dots W_{2} W_{1} = W$ . Stacking a hundred linear layers gives no more expressive power than a single one.

That is why each layer of a neural network applies, after the matrix, a non-linear function $σ$ :

𝐡 = σ (W 𝐱 + 𝐛) .

Stretch, bend, stretch, bend… With that recipe any continuous function can be approximated, as we will see with the universal approximation theorem.

The surprising roominess of high dimensions

The embeddings of a modern LLM have between roughly a thousand and twenty thousand dimensions. Is that a lot or a little to represent every concept a model handles? Three-dimensional intuition is misleading: in high dimension there is a vast amount of room.

Theorem (Johnson–Lindenstrauss lemma, 1984)

For every $0 < ε < 1$ and every set of $n$ points in $ℝ^{D}$ there is a linear map $f : ℝ^{D} \to ℝ^{k}$ with $k = O (ε^{- 2} \log n)$ such that, for every pair of points,

(1 - ε) ‖ 𝐮 - 𝐯 ‖^{2} \leq ‖ f (𝐮) - f (𝐯) ‖^{2} \leq (1 + ε) ‖ 𝐮 - 𝐯 ‖^{2} .

Moreover, a random projection works with high probability.

The dimension needed grows only with the logarithm of the number of points. The flip side of the same geometry: $ℝ^{k}$ has room for exponentially many directions that are almost perpendicular to each other. The superposition hypothesis, studied by interpretability teams, holds that models exploit this to encode many more concepts than they have dimensions, each in a direction almost orthogonal to the rest.

Low rank: the essence of a matrix

Every real matrix, however large, breaks down into three simple transformations: a rotation, a stretch along the axes and another rotation.

Theorem (singular value decomposition)

Every $W \in ℝ^{m \times d}$ can be written as $W = U Σ V^{⊤}$ , with $U, V$ orthogonal and $Σ$ diagonal, whose entries $σ_{1} \geq σ_{2} \geq \dots \geq 0$ are the singular values. By the Eckart–Young theorem (1936), keeping the $r$ largest gives the best rank- $r$ approximation of $W$ .

Many real matrices have almost all their “information” concentrated in a few singular values. This idea underpins current techniques such as LoRA (2021), which adapts a huge LLM to a new task by training only a low-rank correction, $W + B A$ , with tiny $B$ and $A$ , instead of retraining the billions of parameters in $W$ .

We now know where knowledge is stored: in the numbers of the matrices. What is missing is the procedure for choosing those numbers, and for that we need to know how the error changes when each of them moves. That is calculus.

References

A. Cayley (1858). “A Memoir on the Theory of Matrices”. Philosophical Transactions of the Royal Society, 148.
Z. Harris (1954). “Distributional Structure”. Word, 10(2–3).
W. B. Johnson and J. Lindenstrauss (1984). “Extensions of Lipschitz mappings into a Hilbert space”. Contemporary Mathematics, 26.
Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin (2003). “A Neural Probabilistic Language Model”. JMLR, 3.
T. Mikolov, K. Chen, G. Corrado and J. Dean (2013). “Efficient Estimation of Word Representations in Vector Space”. arXiv:1301.3781.
E. J. Hu et al. (2021). “LoRA: Low-Rank Adaptation of Large Language Models”. arXiv:2106.09685.
N. Elhage et al. (2022). “Toy Models of Superposition”. Transformer Circuits Thread.

§Vectors and the dot product

§Embeddings: meaning as geometry

§Matrices: functions that transform space

§The surprising roominess of high dimensions

§Low rank: the essence of a matrix