Why a network can pack more features than it has neurons — and why that is the reason you cannot read a model one neuron at a time.
Elhage, Hume, Olsson, Schiefer, Henighan, Kravec, Hatfield-Dodds et al. — Anthropic — September 2022 Transformer Circuits transformer-circuits.pub
The paper builds on the linear representation hypothesis: the things a network “knows about” — its features — correspond to directions in activation space. A feature is present when activations point along its direction, and the strength of activation along that direction is the feature's value. Composition of features is then just vector addition.
If that picture held perfectly, interpretability would be easy: read each neuron, name its feature, done. The trouble is that real neurons rarely line up with single features.
The paper's central claim is that polysemanticity is not noise or sloppiness — it is the visible symptom of a deliberate, learnable strategy called superposition. The toy models exist to reproduce that strategy in a setting small enough to fully understand.
“What is a neuron?” is the wrong unit of analysis. The right unit is the feature direction — and features need not be axis-aligned with neurons. A single neuron can be a messy linear combination of many features at once.
Take n synthetic features, each a sparse, non-negative scalar, and force them through a bottleneck of only m < n dimensions, then ask the network to reconstruct them. The model is deliberately tiny:
# x : n sparse, non-negative input features (n > m)
h = W @ x # down-project into m-dim bottleneck
x' = ReLU(Wᵀ @ h + b) # tied weights, reconstruct n features
# weighted reconstruction loss — I_i is each feature's importance
L = Σ_i I_i · (x_i − x'_i)²Two design choices do all the work. The decoder reuses the encoder weights (tied weights, Wᵀ), so the only learned parameters are the m×n matrix W and a bias. And the reconstruction passes through a ReLU, whose non-negativity is exactly what lets the model clean up small interference between overlapping features.
Each feature i is represented by its column of W — a vector in the m-dimensional bottleneck. The Gram matrix WᵀW is the key diagnostic:
(WᵀW)ii = ‖Wi‖² tells you how much “dimension” feature i gets. Zero means it was dropped; one means it owns a clean direction.(WᵀW)ij = Wi·Wj are the interference between features i and j. Orthogonal features have zero overlap; superposed features have small non-zero overlap.So WᵀW directly visualises which features got a dedicated dimension and which are sharing space.
The whole phenomenon reduces to one trade-off the loss must resolve: represent more features (less reconstruction error) vs. keep them orthogonal (less interference). ReLU + sparsity is what tips that trade-off toward packing extra features in.
You cannot fit n mutually orthogonal directions into m < n dimensions — there is only room for m of them. Naively, the network must throw away n − m features. But it doesn't have to.
In high dimensions there are many vectors that are nearly orthogonal — almost perpendicular, with only small dot products. The network exploits this: it assigns the extra features near-orthogonal directions and tolerates a little interference. The result is superposition — representing more features than dimensions by accepting controlled cross-talk.
This only works because of feature sparsity: most features are zero on most inputs. If two superposed features almost never fire together, their interference almost never matters, and ReLU thresholds away the small residual noise on the rare collisions. Superposition is effectively a learned compressed code — the same principle as compressed sensing / sparse coding, where a sparse high-dimensional signal can be recovered from far fewer measurements.
Superposition makes the network an over-complete, lossy compression scheme. The price — a little interference noise — is cheap precisely when features are sparse, so a sparse world buys you far more representational capacity than you have neurons.
Whether a feature gets a clean dimension or is shoved into superposition is decided by two quantities the experiments sweep:
Ii): how much that feature reduces the loss — its weight in the reconstruction objective.1 − S, where S is the probability a feature is active): how often the feature is non-zero. Sparser features collide less often, so their interference costs less.Sweep sparsity and the model snaps between regimes rather than sliding smoothly — there are sharp phase transitions:
At low sparsity the model is monosemantic: it keeps the m most important features, one per dimension, and discards the rest. As sparsity rises past a threshold it abruptly switches to superposition, cramming in extra features. Importance breaks ties — the most important features get clean dedicated directions; less important ones are the ones forced to share.
Sparsity is the control knob for interpretability. The sparser a model's features — and real language features are extremely sparse — the more aggressively the network uses superposition, and the more polysemantic and unreadable its neurons become.
When the network does pack features in, it does not scatter them randomly. The feature directions snap into clean geometric arrangements that minimise interference — structures drawn from uniform polytopes. As sparsity increases, the model marches through a sequence of these shapes:
Two features sharing one dimension as opposite directions (180° apart). Two features in the space of one — 1/2 dimension per feature.
Three features at 120° in a 2D plane. Each direction is as far from the others as possible given the count.
Five features packed into a 2D plane at 72° spacing — an over-complete code in just two dimensions.
Four features as the vertices of a regular tetrahedron in 3D — maximally spread directions in a 3-dimensional bottleneck.
Because several features can share a dimension, the natural accounting is fractional dimensions per feature — an antipodal pair is 1/2, and other configurations give characteristic fractions such as 2/3 or 4/7. Higher-dimensional solutions are built as tegum products (direct sums of independent lower-dimensional polytopes), and the configurations are “sticky”: they sit in energy-minimising wells, so the model resists leaving a stable arrangement until sparsity changes enough to favour the next one.
Superposition has a crystalline structure, not a chaotic one. The same packing problems that produce regular polytopes in geometry show up inside trained networks — which is encouraging: a structured phenomenon is a tractable one to reverse-engineer.
Superposition is the mechanistic root cause of polysemanticity. Because neurons carry many features along non-axis-aligned directions, reading a network neuron-by-neuron is fundamentally lossy. The features are there — just not on the basis you are looking at.
The same near-orthogonal packing that makes superposition efficient also makes it fragile. Small, carefully chosen input perturbations can push activations along an unintended feature direction, exploiting the interference between superposed features. The paper draws a direct line from superposition geometry to the existence of adversarial examples.
The toy model exhibits its own version of double descent, and ties it to a regime distinction: networks can either memorise individual data points or learn generalising features, and these correspond to different ways of using the available dimensions. Superposition links the study of memorisation to the study of features — the same packing machinery underlies both.
Dimensions are spent storing specific data points. Good fit to the training set, weak generalisation.
Dimensions are spent on reusable features held in superposition. This is the regime interpretability cares about.
Superposition is not an isolated curiosity — it sits at the intersection of interpretability, robustness, and generalisation. Understanding it gives you one lever on three problems at once.
n > m features — an over-complete code, the same idea as compressed sensing.If superposition hides the real features on a basis we cannot read directly, can we train a second model to decompress them — to map a layer's activations onto a large, sparse dictionary of monosemantic features? Presentation 04 — Towards Monosemanticity does exactly this, using sparse autoencoders to pull genuinely interpretable features back out of a real transformer's superposition.