Between data centres the link can be a thousand times slower than NVLink, and sending the whole gradient every step stops being affordable. Send less (top-k, DeMo), or less often (DiLoCo): the mechanisms, the bytes, and a toy run of all four, with the real measurements one link away.
This deck follows chapter 13, training over slow links, where four workers compress their momentum DeMo’s way in your browser. Measured numbers: results.md §13 and §11d. Real measurements: the Compute in Transit results site.
The runs in this deck are a toy: the Decoder’s 5,376-parameter character model, 4 simulated workers in one process, each training on 4 of the default run’s 16 windows per step, 2,000 steps, one book, a CPU, three seeds per setting. They show the mechanisms running on a model small enough to follow. They do not say what works at scale.
The Compute in Transit results site measured the same ideas on a 15.8-million-parameter GPT trained by eight simulated workers for 3,000 steps on TinyStories, on GPUs, and then simulated data-centre-scale links with those measurements. Its numbers, not these, are the evidence; this deck explains the concepts its pages rely on. Where the two disagree (they do: slide 08), believe the results site.
Inside a server, NVLink moves hundreds of gigabytes a second per GPU; across a data centre’s network, tens (deck 10). A wide-area network (WAN) link between data centres is shared by a whole site and can offer each training job far less: a 100 GbE link is 12.5 GB/s for everything that crosses it.
450 GB/s each way per GPU
50 GB/s per GPU
12.5 GB/s for the whole site
At those rates, step time = compute + communication is dominated by the communication: LLaMA 7B’s dense BF16 gradient (13.5 GB) takes over a second to send at 12.5 GB/s, every step, before any other job shares the link. Training over slow links means sending less, sending less often, or both.
Send only the \(k\) largest values of a vector, and where they are: here 1 in 16 per tensor. Most of a gradient’s coordinates are small at any one step, so most of the step survives.
Each kept value needs its position: for the Decoder’s model 2 bytes of BF16 value plus 2 bytes of position, so 1 value in 16 costs 4/32 of the dense bytes, not 1/16: 1,344 bytes a step against 10,752 for the dense gradient.
Lin et al. (2018) apply it to an accumulated momentum: each worker adds its gradient into a local momentum and an accumulator, sends the top of the accumulator, and plain SGD steps with the average of what was sent (momentum 0.9 here).
What a compressor drops is not thrown away: each worker keeps it, the residual (\(v_i - s_i\) above), and adds it to the next step’s vector before compressing again. A coordinate too small to be sent this step keeps growing until it is large enough to be sent, so every update arrives eventually, only late.
Without it, the steady small coordinates of the gradient would never be applied at all; with it, compressed SGD provably converges at the uncompressed rate in the long run (Stich et al. 2018, sparsified SGD with memory; Karimireddy et al. 2019, error feedback fixes signSGD). 1-bit SGD (Seide et al. 2014) had used the same trick to send one bit per gradient value. DeMo’s momentum is its residual.
A ring all-reduce adds the workers’ chunks as they pass, which needs every worker’s vector to have the same layout (deck 10). Top-k vectors don’t: each worker picks its own positions. So the workers all-gather instead: every worker receives every other worker’s values and positions and adds them itself.
$$\begin{gathered}\text{dense: } 2\,\tfrac{W-1}{W}\,2P \qquad \text{compressed: } (W-1)\,c \\ \text{DiLoCo: } 2\,\tfrac{W-1}{W}\,2P\,/\,H\end{gathered}$$Bytes on the wire per worker per step for \(W\) workers, \(P\) parameters, 2-byte values and a compressed payload of \(c\) bytes. With four workers DeMo’s 1,008 bytes of payload become 3,024 on the wire, against the dense all-reduce’s 16,128. With eight workers the same payload costs 7 times itself, which is why the results site quotes both a payload ratio and a smaller wire ratio. Try it on slide 09.
Peng et al. (2024) let each worker keep its own momentum instead of syncing gradients (their Algorithm 1):
The DCT rearranges a chunk into frequency components without losing anything (it is orthonormal: what is sent and what is kept add up to the whole), and the momentum’s large, slowly changing components tend to collect in a few coefficients, the “fast components” that get sent. Values in BF16 and positions in one byte: 1,008 bytes a step per worker.
The other way to save bytes is to sync less often. In DiLoCo (Douillard et al. 2023, Algorithm 1) each worker trains alone with AdamW for \(H\) steps; then the workers average how far they moved (the pseudo-gradient) and an outer optimiser, Nesterov momentum (learning rate 0.7, momentum 0.9, the paper’s setting), moves the shared weights; then everyone starts the next \(H\) steps from them.
$$\begin{gathered}\Delta = \tfrac{1}{W}\textstyle\sum_i (\theta - \theta_i), \qquad b \leftarrow \mu b + \Delta \\ \theta \leftarrow \theta - \eta_{\text{out}}(\Delta + \mu b)\end{gathered}$$It sends the dense vector, but only once per \(H\) steps: 1,613 bytes a step on average at \(H = 10\), 323 at \(H = 50\). Local SGD, its ancestor, is analysed by Stich (2018).
| Method | Best lr | Final validation loss | Steps to 2.6 (seeds) | Payload | Wire | Wire vs dense |
|---|---|---|---|---|---|---|
| dense AdamW | 0.01 | 2.277 ± 0.090 | 1,150, 1,300, 1,000 | 10,752 B | 16,128 B | 1.0× less |
| top-k + EF | 1 | 2.225 ± 0.039 | 550, 550, 500 | 1,344 B | 4,032 B | 4.0× less |
| DeMo | 0.03 | 2.073 ± 0.047 | 350, 300, 300 | 1,008 B | 3,024 B | 5.3× less |
| DiLoCo H=10 | 0.03 | 2.578 ± 0.078 | not reached, 1,700, 1,950 | 1,075 B | 1,613 B | 10.0× less |
| DiLoCo H=50 | 0.1 | 2.706 ± 0.094 | not reached, not reached, not reached | 215 B | 323 B | 50.0× less |
Each method at the best of three learning rates (a sign step, a plain SGD step and Adam’s normalised step want very different rates), three seeds. DeMo’s best rate is the top of its grid, so it might do better higher still; top-k at 3.0 diverged in every seed. On this toy the compressors leave the plateau sooner than dense AdamW and DeMo ends lowest, while DiLoCo, whose workers each see only 4 windows a step between syncs, lags.
On the results site’s 15.8M-parameter GPT, dense AdamW ended lowest and DeMo’s top-k some way behind it. A 5,376-parameter character model on one book is a different optimisation problem, and the compressors’ big early steps (DeMo’s are a full learning rate on every weight) suit its plateau.
The formulas of slide 05 on the Decoder’s 5,376 parameters. Raise the number of workers and watch the compressors’ wire bytes grow while the all-reduce’s stay put.
| Method | Payload | Wire per step | vs dense | Communication to reach 2.6 |
|---|
A shared WAN carries other traffic; a sync that arrives while the link is busy waits. With transfers arriving at random and the link busy a fraction \(\rho\) of the time, a newcomer waits on average \(\rho\,s/(2(1-\rho))\) for transfers of length \(s\) (M/D/1, the Pollaczek–Khinchine formula). The wait grows without bound as the link fills, and hurts most the methods that send most.
Synchronous training waits for the slowest worker every step. Eight workers, each step 1 + 0.1×Exp(1) time units: syncing every step costs the slowest of eight; syncing every \(H\) steps, as DiLoCo does, waits for the slowest sum of \(H\) steps, which is closer to the mean.
| Load ρ | M/D/1 wait | Simulated |
|---|---|---|
| 0.3 | 0.214 | 0.214 |
| 0.6 | 0.750 | 0.765 |
| 0.9 | 4.500 | 4.518 |
| Sync every | Time per step | Mean worker’s step |
|---|---|---|
| 1 (every step) | 1.270 | 1.099 |
| 10 | 1.148 | 1.099 |
| 50 | 1.121 | 1.099 |
Seeded simulations against the formulas (results.md §11d). The results site’s transit page shows queueing on its recorded runs.
Animated: chapter 13, training over slow links · series: Compute in Transit · next: deck 13, scaling laws and running a job