Training Transit, Simulated

Topics We'll Cover

• The Question
• The Model
• Six Variants and Their Wire Bytes
• The Module as a Link Kernel (Interactive)
• Time to Target Loss
• Hot Spots, and a Shared WAN
• Break-even against WAN Bandwidth (Interactive)
• Break-even in Module GFLOP/s, ENOB and Bit Planes
• Energy
• Extrapolations (Outside the Experiment)
• What It Means for a Photonic Link Module
• Limits, Next Steps, and What to Take Away

The Question

Deck 04 counted steps and bytes. To reach a loss of 3.158 (the 2-D fixed band’s final loss, the one loss that the dense, top-k, exact-band and optics variants all reach), AdamW and top-k needed 400 steps, the exact fixed band 2,667, and the band through the emulated optics (ENOB 8) 1,800, from one seed. A step count is not a time: a band sends fewer bytes per step, and a module that computes it adds its own time.

The Model

W workers in D data centres, each worker an H100. A step is compute, then one sync of the update; the time to target is steps × step time. The diagram is the 2 DC × 4 topology; the model also runs 1 DC × 8 and 8 DC × 1.

Devices and link presets are imported from Disaggregated_Inference_Sim at 69ffd2b, not copied. The 400G and 800G presets and the 8 Gb/s WAN share (the 100 GbE preset at 1 GB/s) are illustrative. At zero background load the simulated step equals an independent closed form to 1e-12 (a test); slide 06 shows what only the discrete-event model can see. Source: results.md, header and section 2.

Six Variants and Their Wire Bytes

Four variants were measured in deck 04’s experiment. Two are hypothetical: every worker sends the same band positions and DeMo averages before taking the sign, so summing the bands gives the same update as gathering them, and C′ and E′ reuse C’s and E’s steps. They were not run.

Source: results.md, section 1. Model wire = bytes a worker sends in the model’s collective with 8 workers in one DC; for A, B, C and E it equals the experiment’s wire column exactly (a test).

The Module as a Link Kernel (Interactive)

The module is modelled as a link kernel: it declares operations per byte by type (FFT, add, multiply, MAC, floating-point MAC, modular, gather), latency, energy per bit and placement. A stream goes through four pipelined stages, and the slowest sets the rate:

• Optics: with binary (NRZ) input, a value quantised to 8 bits takes 8 optical passes, so the bit planes set throughput as well as accuracy. 4 × 14 Gb/s is 56 Gb/s, or 7 GB/s (derived: 56 ÷ 8): the series’ arithmetic from the paper, not a figure it states.
• Compute budget: the series’ reading of the paper’s “tens of GFLOPs”. Whether the optical transform is charged against it is not public, so it is a switch: [low] 10 GFLOP/s charged, [high] 99 GFLOP/s charged, [free] not charged.
• FPGA stage: modular arithmetic runs in the FPGA driver logic in the paper; here it also quantises, tracks the full scale and recombines the bit planes.

From the paper: 9.7 W static, the QSFP56 port, 4 × 14 GBaud NRZ. From the developer’s website: 2.76 pJ/bit (45 nm, excluding the laser). Illustrative: laser power, FPGA rate and work per value, latency, and the 10 and 99 GFLOP/s bracket. Source: results.md, sections 6 and 7b.

Time to Target Loss

Seconds to reach 3.158 = steps × step time, on six of the eight topologies (results.md has 400G and 800G too). 8 workers, H100s at MFU 0.40.

Source: results.md, section 2. [low] 10 GFLOP/s charged; [high] 99 GFLOP/s charged; [free] transform not charged. The 8 Gb/s WAN is illustrative.

• Top-k (B) is the fastest to 3.158 on every topology: 0.93 s on 2 DC × 4 over 100 GbE, against 1.63 s for AdamW.
• A pluggable with the transform charged at 10 GFLOP/s takes about a minute (59.01 to 73.14 s) wherever it is: the module, not the network, sets its time.
• The band summed rather than gathered is the band’s best case. C′ (no module) beats AdamW on the 25 GbE and 8 Gb/s WANs; E′ [free] beats AdamW only on the 8 Gb/s WAN (9.98 s against 13.28 s), and C′ is faster still there (7.02 s).

Hot Spots, and a Shared WAN

The share of the step each resource holds on 2 DC × 4, 100 GbE. Each round of the sync is charged to its slowest operation; the module’s time is split by its binding stage.

Source: results.md, section 6.

• AdamW’s step is mostly communication: compute is 12.1%, the inter-DC link 63.1%. That is the room compression has to work in.
• With the transform charged the module is the hot spot: module compute holds 95.0% of E’s step at [low] and 65.6% at [high]. With it free, the optics binds (57.4%): eight passes per value.
• A pluggable also caps its worker’s port: module port holds 13.6% of E’s [free] step, because a 25 GB/s QSFP56 module sits on a 50 GB/s port.
• C′ is compute-bound (60.8%): summing the band leaves little to communicate.

A shared WAN. With other jobs’ traffic on the inter-DC link (Poisson, 1 MB transfers, illustrative), the sync queues; the closed form cannot see this, the discrete-event model can. Mean step over 200 syncs, ms:

Source: results.md, section 9. At load 0.9 AdamW’s step grows from 4.075 to 26.291 ms; E′ [free], which sends only the band sum over the WAN, from 3.726 to 5.442 ms.

Break-even against WAN Bandwidth (Interactive)

Time to 3.158 on 2 DC × 4 as the inter-DC bandwidth changes (the 100 GbE preset’s latency throughout). Move the slider, or pick a recorded point; switch between E’s steps (the hypothesis) and the exact band’s.

How the curves are drawn: on 2 DC × 4 only the sync’s WAN term depends on the bandwidth, B / bw for an all-reduce or the switch sum, 4P / bw for an all-gather, and max(4P / bw, P / 25 GB/s) for a pluggable, whose port caps its flow. The rest of each step is taken from results.json’s 100 GbE step. A build-time check (_cit05_build/slider_model.py) confirms that this reproduces every time-to-target column on 2 DC × 4 and all 28 rows of section 7a.

Source: results.md, section 7a (bisection to a relative tolerance of 1e-4). Steps “E”: E’s single-seed steps, so the crossing is a hypothesis; “C (exact)”: the exact band’s steps for both in-transit and band variants; “measured”: no in-transit variant involved. Rows with a crossing in bold.

Break-even in Module GFLOP/s, ENOB and Bit Planes

The compute budget (transform charged), swept from 1 to 100,000 GFLOP/s:

Source: results.md, section 7b. Every row uses E’s single-seed steps: a hypothesis.

• On a 100 GbE WAN no budget is enough: even at 100,000 GFLOP/s the band in transit is slower than the band on the GPU, and E′ slower than AdamW. The optics binds before the budget runs out.
• On the 8 Gb/s WAN a pluggable beats the band on the GPU above 79.9 GFLOP/s, and E′ beats AdamW above 77.3 GFLOP/s; at no budget does E′ beat top-k or C′.
• The budget stops binding at 140.0 GFLOP/s (the 1-D 256-point band, 8 planes), above the series’ bracket of 10 to 99.

ENOB enters only through convergence: the steps of the E run at that ENOB (one seed each), as a step function between the measured 6, 8, 10 and 12.

Source: results.md, section 7c. ENOB 6 never reaches any ladder loss, and no ENOB reaches 1.785 or 1.646 (top-k’s and AdamW’s final losses). Where an in-transit variant wins at all, the break-even ENOB is 8, the lowest measured ENOB that trains. The mapping is not monotone (ENOB 10 needs fewer steps than 12): single seeds.

Bit planes trade throughput against convergence: each plane is one optical pass (2 DC × 4, 100 GbE, ENOB 8).

Source: results.md, section 8 (one seed per plane count). Four planes halve the optics stage but need 2,600 steps; twelve need 2,000. Eight is the fastest to 3.158 for both E and E′.

Energy

Per step, all 8 workers, 2 DC × 4, 100 GbE. Compute = each H100’s static power (100 W, illustrative) for the whole step + 1 pJ per FLOP; link = each preset’s pJ/bit over every byte; module = (9.7 W static + 2 W laser, illustrative) per module for the whole step + 2.76 pJ/bit over the bytes it processes. Energy to target = steps × energy per step.

Source: results.md, section 5. Device power coefficients are disagg-sim’s illustrative figures.

• Energy is mostly the devices’: the compute column is the largest in every row, because a slower step keeps every GPU powered for longer.
• The modules’ own share is small, 7.2% of E′ [free]’s energy per step and 10.0% at [low]; what costs energy is the time they add.
• To 3.158, E′ [free] uses 8.87 kJ against 1.95 kJ for AdamW and 1.38 kJ for top-k; C′, with no module, 5.90 kJ.

Extrapolations (Outside the Experiment)

Model size (2 DC × 4, 100 GbE):

DiLoCo-style sync every H steps, with the same steps to 3.158: an assumption, not a measurement (DiLoCo changes the optimiser too).

Source: results.md, section 10.

• Scaling the model does not change the order under these assumptions: top-k stays fastest and the module stays the hot spot of E at [high].
• Syncing less often shrinks every variant’s communication, so the band’s extra steps matter more: at H = 100 AdamW takes 0.21 s and E′ [free] 0.94 s.

What It Means for a Photonic Link Module

Against the five tests of deck 01, for the fixed band computed in transit, in time and energy rather than steps:

Limits, Next Steps, and What to Take Away

• Top-k is the fastest to 3.158 on every topology simulated. AdamW’s step is mostly communication (compute 12.1% on 2 DC × 4, 100 GbE), but the band needs many more steps.
• With the transform charged, the module is the bottleneck: at 10 GFLOP/s a pluggable step takes 33.348 ms against 1.443 ms for the band on the GPU.
• A pluggable all-gather never breaks even with AdamW or top-k over 0.1 to 1,000 GB/s of WAN.
• Summing in the network is where a band could gain: E′ beats AdamW below 1.57 GB/s with E’s steps (a hypothesis), 0.83 GB/s with the exact band’s.
• The same summation without a module does better: C′ beats AdamW below 6.58 GB/s, and on the exact band’s steps E′ never beats C′.
• Energy follows time: the devices dominate, and E′ [free] uses 8.87 kJ to 3.158 against 1.95 kJ for AdamW.