Anthropic's foundational interpretability paper: treat a transformer as readable mathematics — a shared residual stream, attention heads that split into independent QK and OV circuits, and head composition that builds the first genuine in-context-learning algorithm.
Elhage, Nanda, Olsson, Henighan, Joseph, Mann, Askell, Olah et al. — Anthropic — 22 Dec 2021 Transformer Circuits transformer-circuits.pub
A trained transformer is a black box of matrices. We can measure what it does but not why. This paper asks a narrower, tractable question: can we describe, in exact mathematical terms, the small-scale algorithms a transformer actually runs — and read some of them straight off the weights?
The central object is the residual stream: the running vector that flows from the token embedding through every layer to the unembedding. Every component — the embedding, each attention head, each MLP, the unembedding — reads from and writes to subspaces of this one shared stream. Components do not talk to each other directly; the residual stream is the bus they all communicate over.
Because each component adds its output back into the stream, the layers are connected by addition, not by being wired in series. A crucial consequence: attention heads are independent and additive — each head can be understood on its own as an operation that reads its input by linear projection from the stream and writes its result by linear projection back into it. You can analyse one head without simulating the others.
Re-described this way, a transformer stops being a monolithic stack and becomes a set of components that communicate by reading and writing to one linear channel. Additivity is what makes the rest of the paper possible: you can pull out individual heads, individual paths, and study them in isolation.
The paper's sharpest tool is decomposing each attention head into two largely independent circuits. Conventionally we think of four matrices — query, key, value, output (W_Q, W_K, W_V, W_O). The framework regroups them into two products that each have a clear job:
W_Qᵀ W_K — governs where the head attends. It is a bilinear form on the residual stream that produces the attention pattern: which source position each destination position reads from.W_O W_V — governs what gets written. Once a destination has decided to attend to a source token, the OV circuit determines how that source token affects the destination's output, i.e. what vector is added back into the residual stream.Keys, queries and values are not the fundamental objects — they are intermediate results in computing these two low-rank matrices. Splitting "where to look" (QK) from "what to copy" (OV) lets you reason about an attention pattern and its content effect separately.
Every later interpretability result leans on this split. To understand a head you ask two separable questions: what makes it attend (QK) and what it does once it attends (OV). The four learned matrices are just the factored, low-rank way of writing these two products.
Attention is the only strongly non-linear thing in an attention-only transformer (the softmax). The paper's key manoeuvre: treat the attention patterns as fixed. If you freeze them — hold the softmax outputs constant — everything else the model does is linear in the residual stream.
That linearity is enormous leverage. An attention-only model can then be written as a sum of interpretable end-to-end functions, each mapping input tokens to changes in output logits. Each function corresponds to a path through the network — a chain of components the information flows through.
Because the paths are linear, analysing them is just multiplying chains of matrices. Expand the product of all the per-layer transforms and you get one term per path. Two components that are not adjacent but communicate through the residual stream have an effective "virtual weight" — the matrix product describing their direct interaction. As the paper puts it, you can learn a lot simply by breaking apart sums and multiplying together chains of matrices.
Freeze the patterns → one path = one matrix product = one interpretable term in the sum.
Transformers contain an enormous amount of linear structure hiding behind a single non-linearity. Quarantine the softmax and the rest decomposes into a sum of matrix-product paths you can read individually — the mathematical backbone of mechanistic interpretability.
Strip out every attention layer and you have embedding straight to unembedding. This model can only express bigram statistics — the probability of the next token given the current one. And you do not even need to run it: the bigram log-likelihoods are read directly from the matrix product W_E·W_U.
Add a single attention layer and the model becomes an ensemble of bigram and "skip-trigram" models. A skip-trigram has the form [A]…[B][C]: a head attends back to an earlier token A and uses it to adjust the prediction for what follows B. Both tables can again be extracted straight from the weights without running the model.
This is surprisingly expressive — it supports primitive copying / in-context behaviour: having seen A earlier, the head can raise the odds of tokens that tend to co-occur with it. But the mechanism is crude.
logits ≈ W_E · W_U
Pure bigram statistics: P(next | current). Read directly from the weights; no attention at all.
[A] … [B] → [C]
Bigram + skip-trigram ensemble. Source token A, attended to from B, shifts the prediction C. Primitive copying.
The crudeness shows up as the skip-trigram bug. A one-layer head's QK (which A to attend to) and OV (what to predict from it) are locked together across all positions — the head cannot make "what to copy" depend on the token immediately before the prediction. So a head that correctly learns "…keep…in mind" → "mind" is forced to also boost the symmetric, wrong continuation (e.g. "…keep…at mind"), because it cannot represent the two cases separately. The capability and the failure are the same limitation seen from two sides.
One layer gets you copying-flavoured behaviour, but it is a fixed lookup, not real in-context reasoning. The QK/OV coupling that makes skip-trigrams cheap is exactly what makes them buggy — and exactly what a second layer is needed to overcome.
With two attention layers, a head in the second layer can use the output of a head in the first. The paper names three ways heads compose, by which input of the later head the earlier head feeds:
Layer-1 output feeds the query of a layer-2 head — changing what it is searching for.
Layer-1 output feeds the key of a layer-2 head — changing which positions are matchable.
Layer-1 output feeds the value — changing what content gets moved. Different in kind from Q/K.
The payoff of composition is the induction head. Given a sequence containing [A][B]…[A], an induction head, sitting at the second [A], attends to the token that followed the earlier [A] — namely [B] — and predicts it. It is two steps composed: prefix-matching (find the earlier place this token appeared) plus copying (emit whatever came next there).
This is a qualitative jump over the one-layer model. A skip-trigram head can only copy via a fixed table; an induction head copies based on an actual repeated pattern in the current context, including token pairs it never saw in training. It is a small but genuine learning algorithm running at inference time — and it only exists with two or more layers, marking a real phase transition in capability.
An induction head is built by composition: a layer-1 head (often a "previous-token head") writes information that the layer-2 head's key uses to prefix-match (K-composition), and the OV circuit copies the matched continuation. Two simple heads compose into behaviour neither can do alone.
The paper is explicit that almost all of this analysis is of attention-only transformers — models with the MLP layers removed. That is a dramatic simplification: real transformers alternate attention and MLP blocks, and roughly two-thirds of the parameters live in the MLPs. The authors state plainly that the MLP portions have been much harder to get traction on, call it a major weakness of the work, and flag it as the open problem to tackle next.
This paper is where the shared vocabulary of interpretability comes from. "Residual stream", "QK and OV circuits", "Q/K/V-composition", "path expansion", "virtual weights", "induction head" — every later result in this series, and most external mechanistic-interpretability work, is phrased in these terms. If you ever read model-internals tooling, an activation-patching study, or a circuit diagram, this is the dictionary.
It also plants the seed for the next deck: induction heads are not a toy curiosity. The follow-up work shows they appear in large models and account for a substantial share of in-context learning — the very capability that makes few-shot prompting work.
You do not need to do interpretability research to benefit. Knowing that attention splits into "where it looks" and "what it copies", and that copying-by-pattern (induction) is a discrete, learnable circuit, gives you a sturdier mental model of why in-context learning and prompting behave the way they do.
W_QᵀW_K decides where a head attends; W_O W_V decides what it writes. Q, K, V are intermediate results, not the fundamental objects.W_E·W_U; one layer = bigram + skip-trigram ensemble (with the skip-trigram bug); two layers = composition.[A][B]…[A]→[B] — a qualitative jump that needs at least two layers.This framework predicted that one specific circuit — the induction head — should be a big deal. Presentation 02 — In-context Learning and Induction Heads follows it into real models, showing that induction heads emerge during training at a sharp phase change and explain a substantial chunk of how large transformers learn from their context.