A small two-layer circuit that copies "what came next last time" turns out to be the engine behind few-shot prompting — and it switches on in a single sharp phase change during training.
Olsson, Elhage, Nanda, Joseph, DasSarma, Henighan et al. — Anthropic — March 2022 Transformer Circuits transformer-circuits.pub
Presentation 01's Mathematical Framework showed that two-layer attention-only transformers contain a specific, identifiable circuit: the induction head. This paper takes that toy-model object and asks the much bigger question: is this little circuit actually responsible for in-context learning in real, large models?
An induction head implements a simple, almost algorithmic rule on a repeated sequence. Given context … [A][B] … [A] — where the current token is a second occurrence of [A] — it predicts [B]: "the last time I saw this token, what came next? Predict that again." It does this in two steps that two attention heads in adjacent layers compose to perform:
[A] and looks at the token after it.[B]'s probability of being emitted next.Crucially this needs two layers. A "previous-token head" in layer 1 writes each token's predecessor into the residual stream; the induction head in layer 2 uses that to match on prefixes. One-layer models cannot form induction heads at all — the composition is structurally impossible.
An induction head is not pattern-matching against a stored table of n-grams. It decouples the matching token [A] from the copied token [B], so it can complete patterns it has never seen during training — including on random, out-of-distribution sequences. That generality is the whole reason it can power in-context learning rather than mere memorization.
Trace the circuit on the sequence below. The model is at the second [A] and must predict the next token. The previous-token head has already tagged each position with its predecessor; the induction head uses those tags to find where [A] last appeared and copy what followed.
The same idea, written as the two composed heads in residual-stream terms:
tA head qualifies as an induction head if it does both things: prefix matching (attend to tokens that were followed by the current/recent token) and copying (increase the logit of the attended-to token). The authors characterised this behaviour across 34 transformers using more than 50,000 attention-head ablations — this is a measured, mechanistic claim, not a metaphor.
Induction heads do not fade in gradually. Early in training — in every transformer with more than one layer, of any size — there is a narrow window where induction heads abruptly crystallise. Three things happen at once in that window: a visible bump / kink appears in the training loss curve, induction heads form, and the model's in-context learning ability jumps sharply.
To quantify in-context learning, the authors define a simple heuristic. Take the loss on the 500th token in a context and subtract the loss on the 50th token, averaged over examples. If a model uses earlier context to predict later tokens better, this difference is large and negative. The indices 50 and 500 are "somewhat arbitrary" — just an early vs late probe. This score improves abruptly exactly at the phase change.
This is a clean, concrete example of a capability emerging via a discrete circuit forming, not a smooth slide down a loss curve. The synchronised timing — loss kink, induction-head formation, ICL jump — is the paper's first and most visible argument that the three are causally linked. It also makes the phase change a candidate early-warning signal for capability emergence, which is why it is of interest to safety.
No single experiment proves that induction heads cause in-context learning, so the paper stacks six independent arguments. The first five are rigorous in small attention-only models; the sixth is the circumstantial bridge that extends the story to large models.
| # | Argument | What it shows | Strength |
|---|---|---|---|
| 1 | Macroscopic co-occurrence | The phase change, induction-head formation and the ICL-score jump all happen in the same window. | rigorous (small) |
| 2 | Macroscopic co-perturbation | Architecture tweaks that shift when induction heads can form shift the loss kink and ICL jump by the same amount. | rigorous (small) |
| 3 | Direct ablation | Knocking out induction heads at test time significantly reduces in-context learning. | rigorous (small) |
| 4 | Examples of generality | The same heads defined for literal copying empirically do translation, pattern-matching and abstract tasks. | suggestive |
| 5 | Mechanistic plausibility | In small models the circuit is transparent and extends naturally to richer ICL by reuse. | suggestive |
| 6 | Continuity | Metrics tied to induction heads and to ICL vary smoothly from small to large models. | circumstantial (large) |
The honest framing matters: arguments 1–3 establish causation in small attention-only models where the circuit can be fully dissected; arguments 4–6 carry the claim toward large models without ever fully closing it. Where they cannot dissect a large model, they show that its behaviour lies on the same continuum as the small ones they can.
The authors are explicit that this is "only the beginnings of evidence" and that "a large number of subtle confounds or alternative hypotheses are possible." The strong claim — induction heads are the majority mechanism of in-context learning in large models — remains a hypothesis supported by converging evidence, not a closed proof.
If induction heads only copied exact repeats, they could not explain most in-context learning. The key finding is that the same heads, defined narrowly on literal copying, empirically implement a far more abstract operation: fuzzy, nearest-neighbour pattern completion, written [A*][B*] … [A] → [B] where A* ≈ A and B* ≈ B.
[A][B] … [A] → [B]
Exact repetition. The defining behaviour — and the easy case to verify mechanistically.
[A*][B*] … [A] → [B]
Match on tokens that are merely similar, not identical. Nearest-neighbour completion over the embedding space.
chat cat … chien → dog
The analogy generalises across languages and tasks — translation, pattern completion, "continue this structure."
Because the circuit decouples the matched token from the copied token, it never collapses into a fixed n-gram table. That decoupling is what lets it operate on novel and out-of-distribution sequences, and what makes "increase the probability of the analogous continuation" a plausible description of what a large fraction of attention heads are doing when you give a model examples in its prompt.
The narrow definition ("literal copy") and the broad behaviour ("do the analogous thing") are the same heads. That is precisely why the authors find it plausible that induction heads explain not just clever sequence-copying but the bulk of few-shot, in-context learning — the abstract version is the literal version run on soft, similarity-based matches.
This paper is the mechanistic answer to a question you rely on every day without thinking about it: why does putting examples in a prompt work at all?
[A][B] … [A] structure that induction heads consume. "The model learns from the examples" is, mechanistically, induction heads doing soft prefix-match-and-copy in a single forward pass — no weights change.The next time a prompt "just works" because you added two good examples, you now know the name of the thing doing the work, what shape of input it needs, and the single training event that gave the model the ability in the first place.
[A][B] … [A] → [B]. They require more than one layer to exist.Induction heads are clean because each head does roughly one thing. But real models pack far more features than they have neurons — individual neurons fire for many unrelated concepts. Presentation 03 — Toy Models of Superposition explains how networks cram more features than dimensions into the same activations, why that produces polysemantic neurons, and what it means for ever cleanly reading a model's internals.