How big a model for a given budget (IsoFLOP profiles, Kaplan against Chinchilla, a toy sweep); what a running job is watched for; failures, checkpoints and goodput at the scale of thousands of GPUs; and the bridge to fine-tuning.
This deck follows chapter 14, scaling laws, chapter 15, running a real job and chapter 16, after pretraining. Measured numbers: results.md §11c, §14, §15 and §16.
Training costs about \(6N\) FLOPs per token for \(N\) parameters: 2 for the forward pass, 4 for the backward (deck 03), times \(D\) training tokens. A compute budget fixes the product, so every choice of model size fixes the number of tokens it can afford.
Learns everything it can and stops improving, however many tokens it sees.
Never sees enough tokens to use its size.
In between is the best size for the budget. Plot loss against model size at fixed compute and you get an IsoFLOP profile; how its minimum moves as the budget grows is a scaling law.
Power laws fitted to many runs: as compute grows, most of it should go into parameters.
Over 400 models, each run’s learning-rate schedule matched to its length: parameters and tokens should grow about equally, 0.5 / 0.49 / 0.46 by three methods (Table 2). Chinchilla, 70B parameters on 1.4T tokens (20 per parameter), beat much larger models trained on fewer tokens.
Their second method is the one on the next slide: IsoFLOP profiles, a parabola through each, and a straight line through the minima on log–log axes. Llama 3 did the same to choose its flagship’s size, fitting 0.53 for the optimal number of tokens (Grattafiori et al. 2024, §3.2.1). Chinchilla’s parametric form: \(L(N, D) = E + A/N^{\alpha} + B/D^{\beta}\).
The Decoder’s model at widths 6 to 64 (1,056 to 70,656 parameters), each trained for exactly the tokens its budget allows, AdamW at peak 0.01 with each run’s cosine stretched to its own length, three seeds (results.md §14). The rule, fixed before the sweep: a parabola in \(\log N\) through each budget’s lowest mean and its two neighbours; no fit if the lowest is at an edge. The smallest budget has no interior minimum (every wider model spends its few hundred steps on the plateau). At the other three the best size grows with the budget: 2,493, 3,550 and 6,590 parameters, a fitted a = 0.42.
The toy’s exponent lands near Chinchilla’s. That should not be over-read:
Real scaling studies span many decades of compute with fresh data. The toy shows the method and the shape of the trade-off. The site’s notebook 14 runs the same sweep on GPTs of 1–40M parameters: GPU run pending; no number here comes from it.
The point of a scaling law is to predict a run too expensive to try. Llama 3 fitted its law on runs of up to \(10^{22}\) FLOPs and extrapolated to its \(3.8\times10^{25}\) budget, predicting a 402B-parameter model on 16.55T tokens; it trained 405B, because IsoFLOP curves flatten near their minima as budgets grow, so a nearby size costs little (§3.2.1). Extrapolation is only as good as the assumption that nothing changes in between: a toy fit over one decade says nothing about a model a million times larger.
Compute-optimal minimises training cost for a given loss; it ignores the cost of using the model. A smaller model trained on far more tokens reaches a loss a bigger one would reach more cheaply to train, but costs less for every token it ever serves. Llama 3 trained its smaller models “for much longer than is compute-optimal” for that reason. Model size against serving cost: Inference Trade-offs Explained.
The default run, as a dashboard sees it (results.md §15):
| Step | Training loss | Gradient norm (before clipping) | Learning rate | Update / weight |
|---|---|---|---|---|
| 0 | 4.173 | 0.854 | 0.00020 | 1.58e-03 |
| 400 | 2.998 | 0.383 | 0.00930 | 5.16e-03 |
| 800 | 2.930 | 0.299 | 0.00710 | 5.15e-03 |
| 1,200 | 2.562 | 0.535 | 0.00425 | 2.82e-03 |
| 1,600 | 2.519 | 0.614 | 0.00190 | 1.10e-03 |
| 1,999 | 2.366 | 0.808 | 0.00100 | 6.43e-04 |
Training loss is noisy (each batch differs); validation loss, scored now and then on held-out text, is the honest version.
Before clipping: how hard each step pushes. Spikes start here (deck 07).
How far a step moves the weights relative to their size: rises through warmup, jumps as the model leaves the plateau, falls with the decay.
Steps to a target loss: when a method changes the cost of each step (compressed communication, say), compare steps or hours to a target fixed in advance, not final losses. The default run first reaches 2.8 at step 1,000, 2.6 at 1,150 and 2.4 at 1,400 (evaluated every 50 steps).
Over a 54-day period of Llama 3 405B’s pre-training the job was interrupted 466 times: 47 planned and 419 unexpected, about 78% of those hardware (faulty GPUs and their memory above all); only three needed significant manual intervention, and the job still spent more than 90% of its time training (Grattafiori et al. 2024, §3.3.4, Table 5).
between failures, for the job
between failures, per GPU
GPUs, any one of which stops the synchronous job
A synchronous job fails when any one GPU does: \(N\) GPUs each failing every \(M_1\) make a job that fails every \(M_1/N\). The same report names slow-but-working stragglers as a harder problem: every step waits for the slowest (deck 12). Throughput sets the calendar: 3.8×1025 FLOPs at about 400 TFLOP/s on each of 16,384 GPUs is 67 days of uninterrupted compute.
A checkpoint is everything needed to carry on exactly where training stopped: the weights, the optimiser’s state (AdamW’s two moments and its step count), the position in the data, and the schedule’s step. The reference trainer stopped at step 1,000 and resumed (three seeds, results.md §15):
| Resumed with | Validation loss 50 steps after | Final validation loss |
|---|---|---|
| never stopped | 2.643 | 2.277 ± 0.090 |
| weights + optimiser + data position | 2.643 | 2.277 ± 0.090 |
| weights only (fresh AdamW) | 2.652 | 2.327 ± 0.165 |
| weights + optimiser, data restarted | 2.636 | 2.286 ± 0.102 |
Resumed with everything, the run continues bit for bit on every seed. Drop the optimiser state and it continues from fresh moments, a gap inside the seed spread here; restart the data and it sees its first batches twice. At scale a checkpoint is large: Llama 3 saved between 1 MB and 4 GB per GPU, on a storage fabric that sustains 2 TB/s with peaks of 7 TB/s, which checkpoint writes saturate (§3.3.1).
Checkpoint often and the saves eat time; rarely, and each failure throws away more. With checkpoints every \(\tau\) of work, each taking \(\delta\), failures every \(M\) on average and a restart \(R\):
$$\begin{gathered}\text{goodput} \approx \frac{1}{1 + \delta/\tau}\Big(1 - \frac{\tau/2 + R}{M}\Big) \\ \tau^* = \sqrt{2\,\delta\,M}\end{gathered}$$Llama 3’s failure rate, \(\delta\) = 32.8 s (every GPU writing 4 GB at 2 TB/s, an upper bound from the paper’s numbers) and an illustrative 10-minute restart: the simulation’s best interval is 960 s, next to Young’s 854 s (Young 1974), and goodput peaks near 87.7% (results.md §11c).
Pretraining makes a model that continues text; assistants are that model trained further, on far less data chosen far more carefully. The objective barely changes; what it is applied to does. Each stage in outline, with the deck that covers it properly:
Llama 3 405B annealed the learning rate to zero over its final 40M tokens with a high-quality data mix, at its full 128K context, and averaged those checkpoints (§3.4.3): deck 05’s decay phase with the data changed too.
Deck 01’s cross-entropy on the response only; the prompt is masked out of the loss. On “who ran? ” + “the rabbit.” the trained model’s loss is 2.670 nats per response character, 2.721 unmasked (results.md §16). FT 01, FT 02 (LoRA).
RLHF (Ouyang et al. 2022) trains a reward model on human comparisons, then PPO; DPO (Rafailov et al. 2023) skips the reward model, raising \(\log p(y_w)\) relative to \(\log p(y_l)\). The trained model scores “queen.” -18.19 and “qxzjv.” -42.65 after “said the ”. FT 03, FT 04.
Where answers can be checked automatically, RL needs no reward model: DeepSeek-R1 (DeepSeek-AI 2025) trained reasoning this way with GRPO. Model-written feedback from principles: FT 05.
Every stage is chapter 1’s loop with a different dataset and loss: the same backward pass, optimisers, schedules, precision, memory and parallelism. The Training & Fine-Tuning series picks up from here.
Animated: chapter 14 · chapter 15 · chapter 16 · back to the series index