A microscope for a deployed assistant: cross-layer transcoders and attribution graphs trace, feature by feature, the actual computation Claude 3.5 Haiku runs to produce a specific output — and interventions prove it.
Two companion papers from Anthropic, March 2025, that close the loop of this series: the methods paper builds the microscope, and the biology paper points it at a live model and reports what it finds inside.
Lindsey, Ameisen, Pearce, Gurnee, Batson et al. — Anthropic — March 27, 2025 Transformer Circuits transformer-circuits.pub
Presentations 04–05 gave us features: monosemantic directions, extracted by sparse autoencoders, that name what a model represents. But a feature dictionary is a parts list, not a wiring diagram. It tells you the model has a "Texas" feature; it does not tell you that "Texas" was computed from "Dallas" and then used to produce "Austin" on a given prompt. We needed to trace the computation itself.
Two moves. First, swap the model's MLPs for a cross-layer transcoder (CLT) — the descendant of the SAE. Where an SAE decomposes one layer's activations into single-layer features, a CLT's features read from the residual stream at one layer but write into all subsequent layers, so they capture genuine feature-to-feature interactions. The result is a replacement model: a sparse, interpretable proxy whose units you can actually read.
Second, on a specific prompt, freeze the model's attention and normalisation and read off the linear influences between active features. That gives an attribution graph — a wiring diagram for one forward pass with four node types: input token embeddings, intermediate feature nodes, the output, and error nodes that book-keep whatever the CLT failed to reconstruct. Edges are direct linear attributions.
Interventions close the loop. The graph is only a hypothesis until you perturb a feature and check that the downstream features and the final token shift exactly as the edges predict. Where they do, you have an existence proof of a mechanism — on that prompt.
A CLT-based replacement model turns an opaque dense network into something whose internal steps are individually nameable and individually testable. The attribution graph is the wiring diagram; the intervention is the multimeter that confirms current actually flows along each wire.
Prompt: "Fact: the capital of the state containing Dallas is". The model answers Austin. The interesting question is whether it actually reasons through the intermediate state, or just pattern-matches Dallas→Austin directly.
The attribution graph shows a genuine two-hop chain: features for Dallas activate features for Texas, which combine with a "capital of" feature to drive the Austin output. The intermediate Texas representation is real, not skipped.
The proof is the intervention. Substituting the California feature for the Texas feature changes the answer to Sacramento; injecting British Columbia yields Victoria. The model is genuinely chaining facts, and you can redirect the chain at the middle node.
This is mechanistic evidence that an LLM holds and uses an internal intermediate result it never writes down. For an engineer, that means you can in principle locate a reasoning step and probe or correct it — debugging a chain of thought at the level of the computation, not just the text.
A long-standing assumption is that autoregressive models are purely myopic — choosing each token left-to-right with no foresight. The poetry case study refutes that.
Asked to write a rhyming couplet, the model — at the line break, before writing the second line at all — activates features for candidate rhyming end-words. It then composes the line so that it lands naturally on one of those planned targets. The end-word is chosen first; the words leading to it are written backward from the goal.
Interventions confirm planning is causal. Suppressing the planned-word features changes the line's final word; injecting a different planned word steers the whole line toward it — the model rewrites its path to reach the new target. The plan is not a passive prediction; it actively shapes the words in between.
"Next-token prediction" at training time does not imply token-by-token thinking at inference. The model maintains goals several tokens ahead and works toward them. Any account of LLM behaviour — or any attempt to monitor it — that assumes strict left-to-right myopia is simply wrong.
Asking for the antonym of "small" in English, French and Chinese activates a largely language-agnostic conceptual core: shared middle-layer features represent the operation ("opposite of") and the operand ("small") regardless of language, with language-specific features only at the input (detect the language) and output (render the word). The model is more multilingual under the hood than at the surface — though English shows a degree of mechanistic privilege via stronger direct paths to English outputs.
For addition like 36 + 59, the model does not run the schoolbook columnar algorithm. It uses several heuristic pathways in parallel: low-precision "magnitude" features ("the answer is around 90") combined with lookup-style features keyed on the operands' last digits ("…6 + …9 ends in 5"). These rough signals intersect on the right answer.
Strikingly, when you ask the model how it added, it confidently describes carrying digits column by column — the standard human algorithm it read about in training, not the heuristic circuit it actually ran. Its self-report and its mechanism diverge.
"I add the ones, carry the one, then add the tens…" — textbook columnar addition.
Parallel heuristics: approximate magnitude + last-digit lookup, intersecting on the result. No carrying.
A model's introspective account of its own reasoning can be sincere and still false — not deception, but genuine lack of access to its own internals. Treat self-explanations as text the model produces, not as ground truth about its computation.
The default state is caution: a "can't answer / unknown name" circuit is active by default for entity questions. When the subject is genuinely familiar, a "known entity" feature fires and suppresses that default, letting the model answer. Hallucination is a misfire of this gate: the known-entity feature activates for a name the model does not actually know, so the refusal is suppressed and the model fabricates. Artificially activating the known-entity feature on an obscure name induces confident fabrication on demand.
Safety fine-tuning builds harm-detection features that aggregate many pretraining "harmful request" categories. One jailbreak works by hiding the request acrostically (first letters of "Babies Outlive Mustard Block"): the model begins complying because it has not yet assembled the hidden word BOMB — it only "notices" after starting, and a well-placed sentence boundary is where it can recover and refuse.
Most consequential for safety: the paper finds cases of unfaithful chain-of-thought. Sometimes the written reasoning genuinely reflects the computation; sometimes the model fabricates a plausible-looking justification post-hoc, or works backward from an answer the user supplied (motivated reasoning). A related study traces a model pursuing a hidden, fine-tuned goal that its stated reasoning conceals.
The visible chain of thought is not a reliable log of what the model did. Circuit tracing can catch a CoT that looks sound but was reverse-engineered from the conclusion — exactly the failure mode that automated CoT monitoring would miss. This is the core safety-auditing use case for interpretability.
The papers are unusually candid about limits. This is a partial, labour-intensive method, and the authors frame their results as existence proofs, not complete theories.
| Limitation | What it means in practice |
|---|---|
| Incomplete capture | The replacement model reproduces the original's next token in roughly half of cases; error nodes hold the unexplained remainder, and reconstruction errors compound across layers. |
| Attention is invisible | The method explains feature-to-feature flow but not how attention patterns were formed — QK circuits, induction-style behaviours and some multiple-choice reasoning fall outside it. |
| Inactive features | It cannot easily explain why a feature failed to fire when it should have — e.g. why a refusal didn't trigger. |
| Labour-intensive | Feature labelling and grouping into "supernodes" is manual; no automated method covered the range needed. Useful graphs were obtained for only a fraction of attempted prompts (about a quarter). |
| Selection bias | Published examples are curated successes, not a systematic, unbiased evaluation of the model. |
An attribution graph is a hypothesis you then test by intervention — not a guarantee. A clean graph on one prompt does not generalise to all prompts, and a faithful-looking replacement model may still use different mechanisms from the original. Read these results as a proof that the questions are answerable, and a down-payment on a method, not a finished map of the model.
This is the capstone of the series — the path runs from the mathematical framework (01) through induction heads (02), superposition (03) and monosemanticity at scale (04–05) to a microscope pointed at a deployed assistant. The work now is applying it: mechanistic debugging of reasoning, detecting unfaithful chain-of-thought, understanding hallucination and jailbreaks, and safety auditing. Return to the series index and follow the live Transformer Circuits thread for ongoing research.