The chain rule, one operation at a time, from the loss back to the embedding: linear layers, GELU, softmax and the causal mask, LayerNorm, and the residual highway. What each step must save from the forward pass, and how a hand-written backward pass is checked.
This deck follows chapter 4, backprop through the model, where the gradient for the window “alice was” travels back through the Decoder’s model stop by stop. For the forward side of every operation, see the Decoder site: attention, feed-forward, LayerNorm and residuals. Checks: results.md §3.
For every one of the model’s 5,376 weights, the gradient is one number: how fast the loss would change if that weight alone were nudged up a little. Positive means “increasing this weight increases the loss”, so the optimiser moves the weight the other way.
$$g = \nabla_\theta L = \Big(\frac{\partial L}{\partial \theta_1}, \dots, \frac{\partial L}{\partial \theta_{P}}\Big)$$One number per parameter: a gradient for the embedding table, one for every attention and FFN matrix, one for every LayerNorm gain and shift.
It is computed fresh on every batch: a noisy estimate of the gradient over all the text (deck 05 measures the noise). In distributed training, this is the vector the workers sum or average (decks 10 and 12).
The forward pass went embedding → block 0 → block 1 → final LayerNorm → logits → loss. Backpropagation computes every weight’s gradient in one pass the other way, each stop one link of the chain rule.
Write \(\delta_T = \partial L/\partial T\). Each operation turns “how the loss changes with my output” into “how it changes with my inputs and my weights”, using only its own local derivative and what it saved during the forward pass. A weight’s gradient is finished at the stop where its operation is crossed. The embedding table gets its gradient twice: at the start (it is also the output head, tied) and at the very end.
Reverse-mode differentiation of a network was popularised for neural networks by Rumelhart, Hinton & Williams (1986); frameworks such as PyTorch record the forward operations and run exactly this walk automatically (autograd).
Every matrix in the model (the attention projections, the FFN, the head) is a linear layer \(Y = XW + b\). Its backward pass is two matrix products:
$$\begin{gathered}\frac{\partial L}{\partial W} = X^{\top}\delta_Y, \qquad \frac{\partial L}{\partial b} = \textstyle\sum_{\text{rows}} \delta_Y \\ \delta_X = \delta_Y W^{\top}\end{gathered}$$The input the layer saw, transposed, times the gradient of what it produced. So the input \(X\) must be kept from the forward pass.
Forward is one matrix product per layer; backward is two (one for \(\partial L/\partial W\), one for \(\delta_X\)). Hence the usual count of about \(6N\) FLOPs per token for training: 2 forward, 4 backward (deck 13).
For the tied head \(z = x_f E^{\top}\): \(\delta_{x_f} = \delta_z E\) and \(\partial L/\partial E \mathrel{+}= \delta_z^{\top} x_f\).
In attention, with scores \(S = QK^{\top}/\sqrt{d_k}\), weights \(P = \operatorname{softmax}(S)\) and output \(O = PV\) (Decoder, chapter 3):
$$\begin{gathered}\delta_V = P^{\top}\delta_O, \qquad \delta_P = \delta_O V^{\top} \\ \delta_S = P \odot \big(\delta_P - (\delta_P \odot P)\mathbf 1\big) \\ \delta_Q = \tfrac{\delta_S K}{\sqrt{d_k}}, \qquad \delta_K = \tfrac{\delta_S^{\top} Q}{\sqrt{d_k}}\end{gathered}$$Row by row, a score’s gradient is its weight times how much better than average its value vector would have been. A saturated softmax (one weight near 1, the rest near 0) passes almost nothing back: deck 07 shows heads stop learning exactly this way.
Masked (future) positions have weight 0, so they receive no gradient. Teacher forcing stays honest in both directions.
\(a = \operatorname{GELU}(u) \Rightarrow \delta_u = \delta_a \odot \operatorname{GELU}'(u)\). The gradient is multiplied by GELU’s slope at the value the neuron saw (Decoder, chapter 4): neurons far below zero pass almost nothing back. The pre-activation \(u\) must be kept.
LayerNorm’s output does not change if its input is shifted or scaled, so its backward pass removes those directions: the mean of the gradient and its component along the normalised input.
Bars are means over the row. RMSNorm (deck 06) drops the mean terms.
At each \(h = x + f(\operatorname{LN}(x))\) the gradient is simply copied into both paths: straight down the residual stream, and into the sublayer.
$$h = x + f(x) \;\Rightarrow\; \delta_x = \delta_h + f'(x)^{\top}\delta_h$$So the gradient reaching block 0 is the gradient at the top plus whatever each sublayer adds on the way: there is always a path from the loss to every layer along which the gradient is not multiplied by anything (Decoder, chapter 5). This is why deep residual networks train at all.
Every stop needs something from the forward pass: the input of each linear layer, the pre-activation of GELU, the attention weights, each LayerNorm’s input. Training must keep all of these, for every position of every sequence in the batch, until the backward pass reaches them.
2,320 values per sequence per block, counted in the reference trainer’s forward pass (results.md §10).
For a large model this activation memory is often larger than the weights. Trainers keep only some activations and recompute the rest during the backward pass: activation checkpointing (Chen et al. 2016), measured in deck 09.
The site writes every backward step out by hand (NumPy, float64) and checks it two ways, at initialisation and at perturbed weights:
| Backward pass checked against | Measured | Tolerance |
|---|---|---|
| PyTorch autograd, at initialisation | 4.4×10−17 (max err / (1 + max |grad|)) | 10−12 |
| PyTorch autograd, perturbed weights | 9.4×10−17 (same) | 10−12 |
| Central differences, h = 10−6, 552 entries, at initialisation | 7.2×10−10 (max abs) | 10−8 + 10−6|g| |
| Central differences, perturbed weights | 7.6×10−10 (max abs) | 10−8 + 10−6|g| |
PyTorch (CPU, pinned version) computes the same gradients automatically; agreement to about 10−16, relative, means the two do the same arithmetic in nearly the same order.
\(\dfrac{L(\theta + h e_i) - L(\theta - h e_i)}{2h}\) for 552 chosen entries with \(h = 10^{-6}\): independent of any derivative code, so it catches a wrong formula that autograd-style code could share.
Central differences of the cross-entropy at fixed logits \(z\) against the exact gradient \(p - y\) (deck 01). Move \(h\): too large and the truncation error (∝ \(h^2\)) dominates; too small and floating-point cancellation (∝ \(\epsilon/h\)) does.
The site’s check uses \(h = 10^{-6}\) on the whole model in float64 and reaches 7.6e-10 (results.md §3).
Animated: chapter 4, backprop through the model · from first principles: micrograd (Karpathy hub), Linear Algebra for AI · next: deck 04, optimisers