Compute in Transit

Presentations in This Series

1. Compute in Transit: the Idea and the Budget →

   Where computation can sit on a data path (endpoint, link, switch); the photonic link module (arXiv:2608.21536) and what its paper shows; the published operation set and what it lacks; the operations-per-byte budget, with an interactive calculator; whether the optical transform counts against it; precision, energy, and five tests every candidate must pass.

   livePlacement · Operation set · Budget · Five tests
2. Data Movement in LLM Training →

   The traffic each kind of parallelism creates and how big it is; ring all-reduce; in-network aggregation; low-communication training (DiLoCo, Streaming DiLoCo, MuLoCo, SparseLoCo); FFT and DCT gradient compression; DeMo mapped onto the module; the planned fixed-frequency-band experiment.

   liveAll-reduce · SwitchML · DiLoCo · DeMo
3. Data Movement in LLM Inference →

   The KV hand-off and when it binds, with an interactive sizer; quantised and transform-domain KV compression; KV tiering and prefix caching; query-aware KV selection; ring attention; MoE all-to-all; weight streaming; confidential AI; the candidate table and what a simulator needs.

   liveKV hand-off · KV tiering · Quest · Candidate table
4. The Fixed-Band Experiment →

   DeMo with its top-k replaced by a fixed frequency band, which a photonic link module (arXiv:2608.21536) could compute, on a 15.8M-parameter GPT with eight simulated workers: every fixed band ends more than a nat above top-k; payload and wire bytes, with an interactive ring calculator; captured energy on real momentum; why the emulated optics first diverged and how feeding back the values sent fixed it; ENOB and bit planes; what it means for a link module.

   liveDeMo · Captured energy · Error feedback · ENOB
5. Training Transit, Simulated →

   A SimPy model of eight workers in one or more data centres turns the experiment’s steps and wire bytes into time and energy to a target loss, with the module as a link kernel (port, optics, compute budget, FPGA; the slowest stage wins): top-k is fastest on every topology; with the transform charged the module is the bottleneck; a pluggable never breaks even; summing the band at the switch gains only on slow WANs, where a band summed by the GPUs gains more. Interactive break-even against WAN bandwidth; hot spots, energy, labelled extrapolations.

   liveLink kernel · Hot spots · Break-even · In-switch summing

The Proposed System, End to End

Variant E, from a worker’s gradient on the TX GPU to the weight update on every receiving GPU: the delta in 1-D chunks of 256, written onto light as bit planes, transformed by the optics, only the band’s outputs read and sent as values with no indices, then placed, averaged, inverse-transformed and applied as a sign; each sender feeds back what it sent. E′ computes and sums the band at the switch instead (the simulation’s hypothetical variant). Tags: P the module’s paper, X the experiment’s model, S the simulation, A an assumption. Every frame is computed by a port of the experiment’s code, checked against it; the chunk’s numbers are illustrative and the bytes are the experiment’s recorded ones. The full version, with bit planes and ENOB, is on deck 04, slide 01b.

Analysis and Code

1. </>
   analysis/ →

   cit_budget.py: the operations-per-byte budget; the work per byte of candidate kernels with the transform charged or free; training sync sizes and link times for Llama-3-8B and -70B; KV hand-off and weight-load sizes; a fixed frequency band against top-k on synthetic signals. results.md is the source of every number in the decks; pytest checks the conventions.

   codePython · NumPy · pytest
2. </>
   DeMo_Fixed_Band_Experiment →

   The experiment behind deck 04: DeMo implemented from the paper, top-k, fixed, calibrated and emulated-optics bands, simulated data parallelism, byte counting (payload and wire), energy diagnostics, the Colab notebook and every run log. results/tinystories-gpu/results.md is the source of every number in deck 04; docs/optics_investigation.md records the optics failure and its fix.

   codePyTorch · pytest · Jenkins
3. </>
   Training_Transit_Sim →

   The SimPy model behind deck 05: W workers in D data centres, hierarchical ring collectives, a roofline for compute, and the module as a link kernel with pipelined port, optics, compute-budget and FPGA stages; six variants (four measured, two hypothetical, including the band summed at the switch with a loopback); hot spots, energy, WAN queueing and break-even by bisection. Inputs pinned to the experiment’s results; examples/results.md is the source of every number in deck 05; 695 tests check it against closed forms.

   codeSimPy · pytest · Jenkins
4. </>
   Disaggregated_Inference_Sim →

   The SimPy disaggregated-serving simulator of the sister series: the model shapes and link presets used here, and the KV hand-off compression results (in transit and at the GPU) that decks 01 and 03 cite.

   codeSimPy · JavaScript port

Glossary: Concepts and Where They Are Explained

Every concept the decks rely on, with a short explanation and links to the slides that explain it. Concepts the sister series already explain are linked, not repeated: Fourier Optics for Inference (the lens as a Fourier transformer, the 4f system, the noise rule, conversion energy), LLM Inference Simulators (prefill and decode, the KV cache, disaggregation) and FHE Accelerator Simulators (NTTs, digit planes, ENOB).

The idea and the hardware

Compute in transit

Computation applied in the data path, on data that must move anyway, instead of fetching it to a processor and sending it back. It helps only where the transfer bounds the system.Explained in: CiT 01 · slide 02

Placement: endpoint, link, switch

Where the work sits decides what it can see: an endpoint sees its own data, a pluggable link module sees one flow at line rate, and a switch sees many flows, so only a switch can combine streams.Explained in: CiT 01 · slide 02; CiT 05 · slide 11

Photonic link module

A QSFP56 pluggable with an integrated optical Fourier transform device, reported to run FFT, NTT, MAC, and modular and floating-point arithmetic on link traffic at “tens of GFLOPs” (the authors' claim, evaluated on CKKS bootstrapping; arXiv:2608.21536).Explained in: CiT 01 · slide 03; its signal path, animated: FOptInf 01 · slide 15

Published operation set

The operations the paper lists for the module: radix-4 FFT and NTT, addition, subtraction, multiplication, MAC, floating-point MAC and modular arithmetic. Comparisons, rounding, top-k selection and data-dependent control are not listed, so a kernel must apply the same arithmetic to every value.Explained in: CiT 01 · slide 04

Recombined optical transforms

Small, low-precision on-die optical transforms (four optical channels with radix-4 decomposition in the prototype; Kundu et al.) combined into larger, higher-precision power-of-2 FFTs or NTTs.Explained in: CiT 01 · slide 05

Operations per byte

A link module's compute rate divided by its line rate: the arithmetic it can apply to each byte without slowing the link. 0.40 to 3.96 on a QSFP56 link, reading “tens of GFLOPs” as 10 to 99 GFLOP/s. At the prototype optics’ own rate (four modulators at 14 GBaud NRZ, about 7 GB/s) it is 1.43 to 14.14, on a link the prototype could not fill.Explained in: CiT 01 · slide 06; CiT 02 · slide 05

Transform charged or free

Whether the optical transform's FLOPs count against the budget. Charged, only BF16 FFTs up to length 8 fit; free, every fixed-band kernel fits and only the remaining arithmetic counts. Public sources do not say which.Explained in: CiT 01 · slide 07; CiT 05 · slide 04

Noise against exactness

Floating-point traffic needs only a noise level that matches its format's own rounding error; cryptography needs exact results, which the optics reaches by decomposition into smaller exact pieces.Explained in: CiT 01 · slide 08

Link energy per bit

Energy to move one bit over a link. The module’s developer estimates 2.76 pJ/bit for its 45 nm silicon photonics, excluding the laser; the simulator's whole-path link presets (3 to 15 pJ/bit) are illustrative and not directly comparable.Explained in: CiT 01 · slide 09; CiT 05 · slide 09

The five tests

Bottleneck (is the transfer what bounds the system), operations (built only from the published set), budget (fits the operations per byte), precision (tolerates noise or can be exact) and placement (one flow or many).Explained in: CiT 01 · slide 10; CiT 03 · slide 11

Transport-bound

A system whose time or throughput is set by a link, the only case in which compressing that link's traffic helps. In the simulator the KV hand-off bound serving only over slow links near saturation.Explained in: CiT 01 · slide 10; CiT 03 · slide 03; CiT 05 · slide 06

Training traffic

The traffic of each kind of parallelism

Data parallel moves gradients; FSDP/ZeRO moves weights and gradients; tensor and pipeline parallel move activations; ring attention moves KV blocks; expert parallel moves tokens.Explained in: CiT 02 · slide 01

FSDP and ZeRO

Each worker holds a shard of the weights, gradients and optimiser state; it all-gathers the weights before use and reduce-scatters the gradients after the backward pass.Explained in: CiT 02 · slide 01

Ring all-reduce

A reduce-scatter then an all-gather round a ring of N workers; each sends 2(N−1)/N times the buffer, close to twice the buffer whatever N is. The arithmetic is one add per value.Explained in: CiT 02 · slide 03; CiT 04 · slide 03; CiT 05 · slide 02

In-network aggregation

Summing workers' updates inside switches (SwitchML on programmable switches, reporting up to 5.5× training speed-up; SHARP in InfiniBand). The flows must meet, so it belongs in a switch, not in a module on one link.Explained in: CiT 02 · slide 05; CiT 03 · slide 08; CiT 05 · slide 11

Low-communication training (DiLoCo family)

Workers take many local optimiser steps, then exchange a pseudo-gradient and apply one outer step. DiLoCo, Streaming DiLoCo, MuLoCo and SparseLoCo cut how often and how much is sent, for training across data centres.Explained in: CiT 02 · slide 06; CiT 05 · slide 10

Error feedback

Keeping the part of an update that compression discarded and adding it back next time, so the error is delayed rather than lost.Explained in: CiT 02 · slide 06; CiT 04 · slide 08; CiT 04 · slide 09

Frequency-domain gradient compression

Transforming a gradient or momentum tensor (FFT or DCT) and sending only some coefficients, on the premise that its energy sits in a few of them (Wang et al., HPDC 2020).Explained in: CiT 02 · slide 07

DeMo (decoupled momentum)

Each worker keeps its own momentum, applies a DCT to it in chunks, sends the top-k components and subtracts what it sent; the paper reports up to 85× less data per GPU than AdamW-DDP at comparable loss (ICLR 2026).Explained in: CiT 02 · slide 08; CiT 04 · slide 01

Fixed band in place of top-k

Keeping a fixed set of transform coefficients instead of the largest ones: a gather with no comparisons, so it suits the published operations, but it keeps the energy only if the energy sits in that band.Explained in: CiT 01 · slide 04; CiT 02 · slide 09; CiT 04 · slide 04

The fixed-band experiment

DeMo with top-k replaced by a fixed frequency band, with the optics emulated, measured as validation loss against bytes communicated. On a 15.8M-parameter GPT every fixed band ended more than a nat above top-k (top-k 1.785, AdamW 1.646, the low and calibrated bands 3.087 to 3.158), and top-k reached the 1-D band’s final loss in 400 steps.Explained in: CiT 02 · slide 10 (the plan); CiT 04 · slide 04; CiT 04 · slide 05

The fixed-band experiment

Payload and wire bytes

Payload is what a worker originates each step; wire bytes are what it sends in its ring collective. An all-reduce sends about twice its buffer whatever the number of workers; an all-gather, which DeMo needs because each worker’s top-k positions differ, sends the payload once to each other worker. With eight workers, DeMo’s 8× and a fixed band’s 16× payload savings are 2.0× and 4.0× on the wire.Explained in: CiT 04 · slide 03; CiT 05 · slide 03

Captured energy

The share of a tensor’s energy that a selector keeps at a given number of coefficients: the measure of whether a fixed band can stand in for top-k. In DeMo’s 2-D chunks the fixed low band kept 0.013 of the momentum’s energy, about a fifth of a random set, against 0.643 for top-k.Explained in: CiT 04 · slide 06

Residual

What error feedback keeps: everything compressed away and not yet sent, added back to the next step’s update and decaying slowly. With a fixed band the band is removed every step, so the residual grows out of band until the band is a few parts in 100,000 of the energy DeMo compresses.Explained in: CiT 04 · slide 06; CiT 04 · slide 08

Bit planes

Feeding an integer through a linear optical transform one binary digit at a time, as on-off (NRZ) light, then recombining the outputs with weights 2^b. Each plane is one optical pass, so more planes mean more precision and more time.Explained in: CiT 04 · slide 07; CiT 04 · slide 10; CiT 05 · slide 08

ENOB

Effective number of bits: the precision a noisy analogue readout actually delivers, whatever its converter’s nominal width. In the experiment, ENOB 8 per plane was enough for the band once errors were fed back; ENOB 6 cost 0.3 in loss. The Fourier Optics glossary covers the noise rule behind it.Explained in: CiT 04 · slide 07; CiT 04 · slide 10; CiT 05 · slide 08

Feedback of the values sent (all-gather feedback)

Error feedback that subtracts what the optics actually produced, noise and all, rather than the exact band. It is possible because an all-gather returns every worker’s values, its own included. Feeding back the exact band let the optics’ errors repeat uncorrected and every such run diverged; feeding back the values sent fixed it.Explained in: CiT 04 · slide 09; CiT 04 · slide 11

The training-transit simulation

Roofline compute time

Compute time estimated from the arithmetic alone: 6 × parameters × tokens per worker per step, divided by the device’s peak rate times an achieved fraction (MFU; 0.40 here, illustrative). It sets the part of a step that compression cannot shorten.Explained in: CiT 05 · slide 02

Pipelined stages: the slowest wins

The module as a link kernel passes a stream through its port (25 GB/s), its optics (4 × 14 Gb/s NRZ, one pass per bit plane), its compute budget and an FPGA stage, all pipelined, so the slowest stage sets the rate. With the transform charged the budget binds below 140.0 GFLOP/s; with it free, the optics binds.Explained in: CiT 05 · slide 04; CiT 05 · slide 06

Hot spot

The resource that holds the largest share of a step, with each round of a sync charged to its slowest operation. AdamW’s step across two data centres is mostly the inter-DC link; a module with its transform charged holds 95.0% of the band’s step at 10 GFLOP/s.Explained in: CiT 05 · slide 06

Break-even

The value of one parameter (WAN bandwidth, module GFLOP/s or ENOB) at which two variants reach the target loss in the same time, found by bisection; “none in range” when one is faster throughout. A pluggable band never breaks even with AdamW or top-k over 0.1 to 1,000 GB/s.Explained in: CiT 05 · slide 07; CiT 05 · slide 08

In-switch summing with loopback

The band computed and summed by modules at the switch, so only the band sum crosses the WAN, with each worker’s own optical band sent back to it so that the fix for the optics’ errors (feedback of the values sent) still works. Hypothetical: not run. It beats AdamW only below 1.57 GB/s of WAN (0.83 GB/s on the exact band’s steps), and the same sum done by the GPUs does better.Explained in: CiT 05 · slide 03; CiT 05 · slide 07; CiT 05 · slide 11

E’s steps or the exact band’s

The repaired optics band reached the target in fewer steps than the exact band, but from one seed per run: a hypothesis. So every in-transit result is given twice, with E’s steps and with the exact band’s (three seeds), and each crossing says which it uses.Explained in: CiT 05 · slide 05; CiT 05 · slide 07

Inference traffic

KV bytes per token

A model's KV cache per prompt token: 131,072 bytes for Llama-3-8B and 327,680 for Llama-3-70B in BF16, so hand-offs grow linearly with the prompt.Explained in: CiT 03 · slide 02

KV hand-off

Moving a request's KV cache from the prefill pool that built it to the decode pool that uses it, in disaggregated serving.Explained in: CiT 03 · slide 03

Frequency keep-k KV compression

Transforming KV along the token axis and keeping a fixed band of components (FreqKV applies a DCT and keeps half by default): a transform, which an FFT can compute, and a gather, inside the published operations.Explained in: CiT 03 · slide 04

KV-cache quantisation

Storing KV at 2 to 4 bits (KIVI, KVQuant). It needs block maxima and rounding, which are outside the published operation set.Explained in: CiT 03 · slide 04

KV tiering and prefix caching

Keeping KV in CPU memory, SSD or a shared pool and fetching it back when a prefix is reused (LMCache, Mooncake): the series' best inference candidate.Explained in: CiT 03 · slide 05

Query-aware KV selection

Scoring page summaries against the query and fetching only the top pages (Quest). A plain dot-product score is MAC, 1 operation per BF16 byte; Quest's own bound and the ranking both need comparisons.Explained in: CiT 03 · slide 06

Photonic similarity engine (PRISM)

A proposed photonic engine, by other authors, that ranks KV blocks by inner product at low precision: evidence that KV selection is being explored optically.Explained in: CiT 03 · slide 06

Ring attention (context parallelism)

Splitting a long sequence across devices and passing KV blocks round a ring at every layer, overlapped with blockwise exact attention. Its transfers are usually hidden behind compute.Explained in: CiT 02 · slide 01; CiT 03 · slide 07

MoE all-to-all

Expert parallelism sends each token's activation to the device holding its expert and back, every MoE layer.Explained in: CiT 02 · slide 01; CiT 03 · slide 08

ML-KEM (FIPS 203)

NIST's lattice-based post-quantum key encapsulation. Its polynomial arithmetic uses NTTs, which suit the module, but handshakes are only kilobytes.Explained in: CiT 03 · slide 10

Link kernel

A simulator abstraction for work done on a link: it declares operations per byte, operation types, precision need, compression ratio and placement (pluggable or switch).Explained in: CiT 03 · slide 12; CiT 05 · slide 04