Real training computes in 16 and even 8 bits. Exponent against mantissa, BF16 against FP16, FP32 master weights and loss scaling, FP8 with per-tensor scaling, and gradients quantised to a few bits, each emulated on the model and measured.
This deck follows chapter 9, mixed precision, where all 5,376 gradients of the trained model are rounded in your browser. Measured numbers: results.md §9a–9b. The formats bit by bit: Numerics Explained (the site’s rounding code is vendored from it and matches it on 202,090 values).
Every number in the site’s trainer is a 64-bit double. Real training uses 16-bit and even 8-bit numbers, because they halve or quarter the memory and the traffic, and the matrix units run several times faster on them.
Weights, gradients and activations at 2 bytes instead of 4 (deck 09).
Two bytes per parameter per worker per step is what crosses the network when workers exchange BF16 gradients (deck 10, deck 12).
Precision and range: small numbers fall off the bottom of the format and become zero; large ones overflow.
A floating-point number has a sign, an exponent (its range) and a mantissa (its precision: how many values per factor of two). With \(e\) exponent and \(m\) mantissa bits and bias \(2^{e-1}-1\), the spacing near \(|x|\) is \(2^{\max(\lfloor\log_2|x|\rfloor,\,1-\text{bias}) - m}\), so the relative rounding error is at most \(2^{-m-1}\).
| Format | Exponent / mantissa bits | Smallest subnormal | Largest | Relative precision |
|---|---|---|---|---|
| FP32 | 8 / 23 | 2−149 | 3.4×1038 | 2−24 |
| BF16 | 8 / 7 | 2−133 | 3.4×1038 | 2−8 |
| FP16 | 5 / 10 | 2−24 | 65,504 | 2−11 |
| FP8 E5M2 | 5 / 2 | 2−16 | 57,344 | 2−3 |
| FP8 E4M3 | 4 / 3 | 2−9 | 448 | 2−4 |
BF16 (bfloat16) is 2 bytes like FP16 but keeps FP32’s 8-bit exponent: the same range as FP32 and about 2–3 significant decimal digits (Kalamkar et al. 2019). FP16 has more precision but a narrow range.
Type a value; it is rounded to nearest (ties to even) in each format, with subnormals and overflow. FP32 uses the browser’s own Math.fround; the others the spacing formula on the previous slide.
| Format | Rounded | Relative error |
|---|
All 5,376 gradients of the trained model, rounded to each format (results.json gradRounding, the site’s chapter 9 animation):
| Format (no scaling) | Gradients flushed to zero | Mean relative error |
|---|---|---|
| FP32 | 0 of 5,376 | 2.2e-08 |
| BF16 | 0 of 5,376 | 1.4e-03 |
| FP16 | 69 of 5,376 | 6.1e-03 |
| FP8 E5M2 | 405 of 5,376 | 4.8e-02 |
| FP8 E4M3 | 1,601 of 5,376 | 1.2e-01 |
BF16’s range keeps every gradient; FP16 loses the smallest; FP8 loses many. That trade is why most large runs today compute in BF16: it needs no loss scaling. Multiplying the loss by \(2^k\) before the backward pass shifts every gradient up by \(k\) binades, into FP16’s range:
gradRounding): too little scale flushes small gradients to zero, too much overflows large ones. Dynamic scaling looks for the gap.Micikevicius et al. (2018) made 16-bit training work with three ideas:
Compute in 16 bits, but the optimiser updates an FP32 copy of the weights. An update is often smaller than the spacing between 16-bit values near the weight, so adding it to a 16-bit weight would do nothing.
The matrix products accumulate their sums in FP32, so long dot products do not lose their small terms.
\(\tilde L = 2^k L \Rightarrow \tilde g = 2^k g\); the optimiser divides by \(2^k\) in FP32. Dynamic scaling halves the scale and skips the step on an overflow, and doubles it after a long run of good steps (PyTorch’s GradScaler).
The default run with every weight’s compute copy, every activation a matrix product reads or writes, and every gradient rounded to the format (float64 stands in for FP32 in the elementwise operations and the master weights); three seeds (results.md §9a):
| Recipe | Final validation loss | Seeds 1, 2, 3 | Gradient values flushed to zero |
|---|---|---|---|
| float64 (reference) | 2.277 ± 0.090 | 2.299, 2.354, 2.178 | 0.00% |
| FP32 | 2.316 ± 0.085 | 2.298, 2.408, 2.242 | 0.00% |
| BF16, FP32 master weights | 2.329 ± 0.059 | 2.269, 2.334, 2.386 | 0.00% |
| BF16, no master copy | 2.296 ± 0.027 | 2.325, 2.290, 2.272 | 0.00% |
| FP16, no loss scaling | 2.331 ± 0.027 | 2.340, 2.352, 2.301 | 4.56% |
| FP16, dynamic loss scaling | 2.288 ± 0.131 | 2.266, 2.429, 2.169 | 0.13% |
| FP8 (E4M3 / E5M2), no scaling | 2.957 ± 0.011 | 2.945, 2.959, 2.967 | 15.82% |
| FP8 (E4M3 / E5M2), per-tensor scaling | 2.574 ± 0.338 | 2.383, 2.965, 2.374 | 1.96% |
FP32, BF16 with master weights and FP16 with dynamic loss scaling all end within the seeds’ spread of the float64 reference. So, at this size, do the two recipes that should hurt: FP16 without scaling (although 4.6% of its gradient values were flushed to zero) and BF16 with no master copy (although 43% of its weight updates were rounded away). A 5,376-parameter model trained for 2,000 steps is forgiving; at scale, with longer runs and smaller late learning rates, the same losses accumulate, which is why master copies and loss scaling are standard.
The newest accelerators also multiply in 8-bit floats. Micikevicius et al. (2022) recommend E4M3 (largest 448) for weights and activations and E5M2 (more range, less precision) for gradients, with a scaling factor per tensor so each tensor’s largest value lands near the top of the format:
$$s = 2^{\lfloor \log_2(\max_{\text{fmt}} / \operatorname{amax}(x)) \rfloor}, \qquad \hat x = \operatorname{round}(s x)/s$$Never leaves the plateau: 2.957 ± 0.011, with 16% of gradient values flushed to zero.
Two of three seeds train (2.383, 2.965, 2.374): three mantissa bits are coarse, and at this scale whether the run escapes the plateau is a coin toss.
In the emulation only what the matrix products read is FP8; what they write stays BF16, as in those recipes. The site’s notebook 09 trains a 60-million-parameter model in each format on a GPU: GPU run pending, so no GPU number appears here.
Beyond floating point, values can be quantised to a few bits of integer: pick a full scale \(F\), divide it into \(n = 2^{b-1}-1\) steps each side of zero, round each value to the nearest step, and clip anything beyond the full scale.
$$\hat x = \Delta \cdot \operatorname{clip}\big(\operatorname{round}(x/\Delta), -n, n\big), \qquad \Delta = F/n$$| Bits | Levels | Final validation loss | Seeds 1, 2, 3 |
|---|---|---|---|
| 8 | 255 | 2.286 ± 0.069 | 2.321, 2.331, 2.207 |
| 4 | 15 | 2.413 ± 0.209 | 2.653, 2.269, 2.315 |
| 2 | 3 | 2.860 ± 0.182 | 2.650, 2.966, 2.964 |
Every weight gradient quantised per tensor (full scale = its largest value) before clipping and the optimiser step; three seeds (results.md §9b). Adam is forgiving here too, since it normalises each weight’s step (Alistarh et al. 2016 analyse quantised SGD). Quantised gradients are what compressed distributed training sends between workers (deck 12), and analogue or optical hardware imposes the same quantisation on whatever passes through it.
Animated: chapter 9, mixed precision · formats in depth: Numerics Explained · next: deck 09, memory