Live, animated companion: Training (Foundations track, after Decoder) — sixteen chapters in which the Decoder’s 5,376-parameter model trains in your browser. Each deck below links its chapters, and every measured number in the decks comes from the site’s results.md.
Presentations in This Series
- The Training Loop and the Objective →The loop (batch, forward, loss, backward, update); next-token cross-entropy and its gradient p−y; the default run's loss curve and plateau; training vs validation loss, perplexity; seeds and spread; how the trainer is checked.
- Data and Tokenisation →The corpus and its held-out split; windows, batches and teacher forcing; byte-pair encoding learnt live; BPE on the book; packing and document masking; deduplication, filtering and mixtures; tokens per parameter.
- Backpropagation Through a Transformer →What a gradient is; the backward order; linear, GELU, softmax/attention and LayerNorm backward passes; the residual highway; what must be saved (activation memory); checking against autograd and finite differences.
- Optimisers: SGD to AdamW →SGD in a narrow valley (live); momentum; Adam's moments and bias correction; AdamW's decoupled weight decay; the four optimisers on the real model; optimiser state; newer optimisers in context.
- Learning Rate, Schedules and Batch Size →Why the learning rate matters most; the LR sweep and its stable band; warmup; cosine and WSD (interactive); measured schedules; batch size, critical batch and gradient noise scale; the linear scaling rule.
- Initialisation, Normalisation and µP →Initial weight scale; pre-LN vs post-LN through depth and in training; RMSNorm; why the learning rate does not transfer across width; the µP coordinate check and learning-rate transfer.
- Stability: Spikes and Cures →Attention-logit growth and entropy collapse; QK-norm and z-loss; one bad batch; gradient clipping and skipping; what large runs report (PaLM restarts, Adam-driven divergence, lower β2).
- Mixed Precision →Floating-point formats and a live rounding calculator; BF16 vs FP16 range; master weights, FP32 accumulation, loss scaling; measured recipes; FP8 with per-tensor scaling; quantised gradients.
- Training Memory →Sixteen bytes per parameter; Korthikanti et al.'s activation formula; LLaMA 7B by the calculator (interactive); full and selective activation checkpointing; the Decoder's model counted against the formula.
- Data Parallelism, All-Reduce and ZeRO →Data parallelism; the ring all-reduce (stepped live); the α–β model and collective times; all-gather and why its bytes grow; payload vs wire bytes; ZeRO 1–3 and FSDP; step time, overlap and MFU.
- Model Parallelism →Pipeline parallelism and the bubble (live GPipe/1F1B simulation); interleaving; tensor parallelism's column/row splits; sequence, context and expert parallelism; 3-D/4-D layouts; the systems model against 12 published runs.
- Training over Slow Links →WAN bandwidth; top-k sparsification; error feedback; why compressed vectors are all-gathered; DeMo; DiLoCo; a four-worker toy of all four (labelled a toy, with the Compute in Transit results site for real runs); bytes against bandwidth; queueing and stragglers.
- Scaling Laws, Running a Job →C = 6ND and IsoFLOP profiles; Kaplan vs Chinchilla and a toy sweep; extrapolation and overtraining; monitoring; failures at scale; checkpoints and resuming; goodput and Young's interval (interactive); the bridge to SFT, RLHF and DPO.
A toy, said plainly. The measured runs use a 5,376-parameter model trained on one book on a CPU, three seeds per setting. They show the mechanisms; they do not say what happens at scale. Numbers about large runs are cited from the papers on each slide. GPU runs of the site’s Colab notebooks are pending, and no GPU number appears in the decks. For communication-efficient training measured on GPUs, see the Compute in Transit results site.
After pretraining: supervised fine-tuning, LoRA, RLHF, DPO and Constitutional AI are covered by the Training & Fine-Tuning series (live). Backpropagation from first principles: Linear Algebra for AI/ML and Karpathy: Neural Networks Zero to Hero.
Glossary
Every training term in the series, in one sentence, with the deck slide that explains it and the site chapter where it is animated.
| Term | In one sentence | Deck | Site |
|---|---|---|---|
| Training step | One turn of the loop: one batch, one forward and backward pass, one optimiser update. | deck 01 | ch 1 |
| Parameters (weights) | The model's adjustable numbers; training changes nothing else. The Decoder's model has 5,376. | deck 01 | ch 1 |
| Next-token prediction | Each position predicts the following token from the true text so far (teacher forcing), so a window of S+1 tokens gives S predictions. | deck 01 | ch 3 |
| Cross-entropy loss | Mean of minus the log probability given to the true next token; its gradient at the logits is p − y. | deck 01 | ch 3 |
| Loss curve and plateau | Loss against training step; this model sits at the letter-frequency loss for hundreds of steps before attention pays. | deck 01 | ch 1 |
| Validation loss | Loss on held-out text the model never trains on; the honest measure, and what runs are compared on. | deck 01 | ch 1 |
| Perplexity | eL: a model with loss L is as unsure as a uniform choice among eL tokens. | deck 01 | ch 3 |
| Seeds and spread | Seeded randomness makes runs reproducible; comparisons use several seeds, mean ± standard deviation. | deck 01 | ch 1 |
| Context length, windows and batches | How many tokens the model sees at once; a batch is many windows drawn at random from the training text. | deck 02 | ch 2 |
| Byte-pair encoding (BPE) | Learn a vocabulary by repeatedly merging the most frequent adjacent pair of symbols. | deck 02 | ch 2 |
| Packing, deduplication, mixtures | How pretraining corpora are prepared: documents packed into windows, near-duplicates removed, sources filtered and mixed. | deck 02 | ch 2 |
| Gradient | One number per weight: how fast the loss changes if that weight alone is nudged; recomputed every batch. | deck 03 | ch 4 |
| Backpropagation | The chain rule applied in reverse, operation by operation, from the loss to the first layer. | deck 03 | ch 4 |
| Activation memory | What the forward pass must keep for the backward pass; for large models often more than the weights. | deck 03 | ch 10 |
| SGD | Step by the learning rate times the gradient. | deck 04 | ch 5 |
| Momentum | Step along a running sum of past gradients; the buffer is per-weight optimiser state. | deck 04 | ch 5 |
| Adam (moments, bias correction) | Per-weight running means of the gradient (m) and its square (v), corrected for their zero start; step η·m/√v. | deck 04 | ch 5 |
| AdamW (decoupled weight decay) | Shrink every weight by (1 − ηλ) directly instead of adding the decay to the gradient. | deck 04 | ch 5 |
| Learning-rate sweep | Train at a ladder of learning rates about 3× apart, several seeds each, and keep the best; done per method. | deck 05 | ch 6 |
| Warmup, cosine and WSD | Learning-rate schedules: rise linearly at the start; decay smoothly (cosine) or hold then decay at the end (warmup–stable–decay). | deck 05 | ch 6 |
| Critical batch size, gradient noise scale | Below it, doubling the batch halves the steps; above it, it barely helps. Estimated by tr(Σ)/|G|2. | deck 05 | ch 6 |
| Pre-LN vs post-LN | Norm before each sublayer (stream grows, gradients even) or after each residual sum (stream at 1, needs warmup). | deck 06 | ch 7 |
| µP (maximal update parameterisation) | Width-scaled initialisation and learning rates so a narrow model's best learning rate transfers to a wide one. | deck 06 | ch 7 |
| QK-norm and z-loss | Normalise queries and keys so attention scores cannot grow with the weights; z-loss keeps the output softmax's log Z near 0. | deck 07 | ch 8 |
| Gradient clipping | Rescale any gradient whose global norm exceeds a threshold; or skip the batch. | deck 07 | ch 8 |
| BF16 and mixed precision | 16-bit compute with FP32's exponent range; FP32 master weights and accumulation; loss scaling for FP16. | deck 08 | ch 9 |
| Quantisation, full scale and clipping | Round values to a few integer levels between ± a full scale; values beyond it are clipped. | deck 08 | ch 9 |
| Sixteen bytes per parameter | Mixed-precision Adam: 2 weight + 2 gradient + 8 Adam moments + 4 FP32 master. | deck 09 | ch 10 |
| Activation checkpointing | Keep only some activations and recompute the rest in the backward pass: memory for compute. | deck 09 | ch 10 |
| Data parallelism | Every worker holds the whole model and its own share of the batch; gradients are averaged each step. | deck 10 | ch 11 |
| All-reduce, ring, reduce-scatter | Every worker ends with the sum; the ring does it as a reduce-scatter then an all-gather, sending 2(p−1)/p of the vector. | deck 10 | ch 11 |
| All-gather | Every worker ends with everyone's pieces; per-worker bytes grow with the number of workers. | deck 10 | ch 11 |
| Payload vs wire bytes | What a worker originates vs what it actually sends in the collective, forwarding included. | deck 10 | ch 11 |
| ZeRO and FSDP | Shard optimiser states, then gradients, then weights across data-parallel workers. | deck 10 | ch 11 |
| Step time and overlap | Step time = compute + communication; overlapping the gradient sync with the backward pass hides it. | deck 10 | ch 11 |
| MFU | Model FLOPs utilisation: the model's needed FLOPs per second over the hardware's peak. | deck 10 | ch 11 |
| Pipeline parallelism and the bubble | Stages of layers on different devices; idle fill-and-drain time is (p−1)/(vm) of the work. | deck 11 | ch 12 |
| Tensor parallelism | Split each layer's matrices across devices: column then row split, all-reduces every layer. | deck 11 | ch 12 |
| WAN / inter-data-centre bandwidth | Links between sites, far slower than NVLink or a cluster network, and shared. | deck 12 | ch 13 |
| Top-k sparsification | Send only the k largest values (and their positions). | deck 12 | ch 13 |
| Error feedback (residual) | Keep what a compressor dropped and add it to the next step's vector, so it is sent later. | deck 12 | ch 13 |
| Compressed all-gather | Compressed vectors differ in layout per worker, so they are all-gathered: wire = (W−1)×payload. | deck 12 | ch 13 |
| DeMo (decoupled momentum) | Local momentum per worker; send the top DCT coefficients of each chunk, keep the rest; sign step. | deck 12 | ch 13 |
| DiLoCo / local SGD | H local AdamW steps per worker, then an outer Nesterov step on the averaged pseudo-gradient. | deck 12 | ch 13 |
| Queueing and stragglers | Waiting for a busy shared link (M/D/1) and for the slowest worker each sync. | deck 12 | ch 13 |
| Scaling laws and IsoFLOP profiles | At fixed compute C ≈ 6ND, loss against model size has a minimum; how it moves with C is the law. | deck 13 | ch 14 |
| Extrapolation | Predicting a run too large to try from fits on small ones; only as good as the assumption that nothing changes. | deck 13 | ch 14 |
| Steps (and time) to a target loss | Compare methods by when they first reach a loss fixed in advance, not by final loss. | deck 13 | ch 15 |
| Checkpoint | Weights, optimiser state, data position and schedule step: enough to resume bit for bit. | deck 13 | ch 15 |
| Goodput and Young's interval | Share of time doing useful work under failures; best checkpoint interval near √(2δM). | deck 13 | ch 15 |
| SFT, RLHF and DPO | Fine-tuning after pretraining: response-only loss, reward-model RL, or direct preference optimisation. | deck 13 | ch 16 |