Compute in Transit 03 — Data Movement in LLM Inference

00

Topics We'll Cover

What Inference Moves
How Big: KV Cache and Weights (Interactive)
The KV Hand-off, and When It Binds
Compressing the Hand-off
KV Tiering and Prefix Caching
Fetch Only What Matters: Query-Aware KV Selection
Ring Attention and Context Parallelism
MoE All-to-All and Tensor-Parallel All-Reduce
Weight Streaming
Confidential AI: FHE and Encrypted Links
The Candidate Table
What a Simulator Needs to Answer This
What to Take Away

Where this deck sits

Deck 01 sets the budget and the five tests: a QSFP56 compute-in-transit module offers 0.40 to 3.96 operations per byte at line rate, and the published operations are FFT, NTT, MAC, and modular and floating-point arithmetic (Kundu et al., arXiv:2608.21536, a QSFP56 pluggable with an integrated optical Fourier transform device, evaluated only on FHE CKKS bootstrapping). Comparisons, rounding and top-k are not in the published set. Deck 02 applied the tests to training. This deck applies them to inference.

Concepts used here, and where they are explained. Each is defined, with links, in the series glossary.

KV hand-off KV tiering and prefix caching Operations per byte Published operation set Frequency keep-k Query-aware KV selection Ring attention MoE all-to-all ML-KEM Link-bound (transport-bound)

01

What Inference Moves

Inference moves less data than training in any single transfer, but it moves it often and on the critical path of a request. Six flows are worth naming.

Traffic	When it moves	Typical link
KV hand-off, prefill to decode	Once per request, after prefill (disaggregated serving: DistServe, Splitwise)	NVLink in a node; InfiniBand or Ethernet between nodes
KV tiering, offload and prefix fetch	When a cached prefix is reused, or KV is evicted from GPU memory	PCIe to host memory; network to a shared CPU/SSD pool
Context-parallel prefill (ring attention)	Every layer of a long-context prefill	NVLink or InfiniBand between the GPUs of one job
MoE all-to-all (and TP all-reduce)	Every MoE layer (every layer for TP), prefill and decode	NVLink in a node, InfiniBand across nodes
Weight loading and swapping	Cold start, model swap, adapter swap	PCIe from host or local SSD; network from object storage
Ciphertexts and encrypted traffic	Confidential inference: FHE ciphertexts, or encrypted links	Whatever link carries the data

Disaggregated serving: DistServe (arXiv:2401.09670, OSDI 2024) and Splitwise (arXiv:2311.18677, ISCA 2024). NVIDIA Dynamo's transfer library, NIXL, moves KV between prefill and decode workers and across GPU, CPU, SSD and object-storage tiers (Dynamo architecture; ai-dynamo/nixl).

02

How Big: KV Cache and Weights (Interactive)

A KV cache grows linearly with the prompt. With grouped-query attention, Llama-3-8B stores 131,072 bytes per token and Llama-3-70B 327,680 (BF16). The hand-off time is the link latency plus bytes over bandwidth:

Model	Prompt	KV hand-off (MB)	NVLink 4 (ms)	IB NDR (ms)	100 GbE (ms)	25 GbE (ms)
Llama-3-8B	2,048	268.4	0.60	5.38	21.49	85.92
Llama-3-8B	32,768	4,295.0	9.55	85.91	343.62	1,374.41
Llama-3-70B	2,048	671.1	1.50	13.43	53.71	214.77
Llama-3-70B	32,768	10,737.4	23.87	214.76	859.01	3,435.99

Source: analysis/results.md, section 4. BF16 weights: Llama-3-8B 16.06 GB, loaded over PCIe Gen5 x16 in 0.25 s; Llama-3-70B 141.10 GB in 2.20 s. For scale, one Llama-3-70B BF16 training update is 141.10 GB against 268.4 MB for a 2,048-token Llama-3-8B hand-off (section 6).

Model

Link

Prompt tokens2,048

Compression

Hand-off size

—

Transfer time

—

Hand-offs/s to fill the link

—

03

The KV Hand-off, and When It Binds

A single transfer time says little; what matters is whether hand-offs queue. Disaggregated_Inference_Sim runs a request trace through a prefill pool, a link and a decode pool (Llama-3-8B shape, one H100 per instance, 2,048-token prompts: a 268.4 MB GQA KV hand-off). Its coefficients are illustrative; read these as a simulator's results, not measurements.

Link	KV wait + transfer share of E2E	Link busy	TPOT p99	SLO met
NVLink 4	0.0%	0.5%	8.8 ms	100.0%
IB NDR	0.3%	4.0%	8.8 ms	100.0%
100 GbE	1.2%	16.1%	9.0 ms	100.0%
25 GbE	8.1%	64.4%	11.6 ms	100.0%

At the default load the hand-off is invisible on anything faster than 25 GbE. Push the 25 GbE case harder: the link saturates near 11.6 req/s (268.4 MB per prompt at 3.125 GB/s).

Load	Link busy, uncompressed	Hand-off p99	TPOT p99	SLO met: none	FP8	freq-keep-k
8 req/s	64.4%	471.9 ms	11.6 ms	100.0%	100.0%	100.0%
10 req/s	79.0%	750.0 ms	14.1 ms	100.0%	100.0%	100.0%
12 req/s	90.6%	2,959.0 ms	26.5 ms	98.3%	100.0%	100.0%
14 req/s	94.0%	9,657.8 ms	77.2 ms	41.0%	100.0%	100.0%

Source: Disaggregated_Inference_Sim, examples/results.md, sections 14 and 15. TPOT includes the hand-off, following DistServe's split (TTFT is the prefill phase; the rest counts towards TPOT); TTFT does not.

The hand-off binds only on a slow link near saturation. At 14 req/s over 25 GbE, uncompressed hand-offs met the SLO for 41.0% of requests; FP8 and freq-keep-k (both 2×) restored 100.0%, with hand-off p99 down from 9,657.8 ms to 166.6 ms.
Over NVLink, compression made no difference: TPOT p99 8.8 ms and SLO 100.0% with every preset.
What is handed off can matter more than the link: in section 14 a Hyena direct-decode cache (1,073.7 MB) saturates 25 GbE, while a distilled state (16.8 MB) is under 1% of end-to-end time on every link.

04

Compressing the Hand-off

Two families of KV compression, and how each fares against the published operation set.

Quantisation

KIVI: 2-bit KV, tuning-free, keys quantised per channel and values per token (arXiv:2402.02750).
KVQuant: 3-bit KV with under 0.1 perplexity loss (arXiv:2401.18079).
Needs: a block maximum (comparisons), a scale (multiply) and rounding. Rounding and comparison are not in the published set.

Transform domain

FreqKV: a DCT along the token axis, keeping the low-frequency components; half by default, with light fine-tuning (arXiv:2505.00570, ICLR 2026).
Needs: a transform and a fixed gather of the kept band. No comparisons, no rounding.
The decode side must accept frequency-domain KV, or invert the transform.

Kernel	Transform / byte	Other / byte	In set	Fits high budget, transform charged	Fits low budget, transform free
1-D FFT, length 8, keep a fixed band	3.75	0.00	yes	yes	yes
1-D FFT, length 64, keep a fixed band	7.50	0.00	yes	no	yes
1-D FFT, length 1024, keep a fixed band	12.50	0.00	yes	no	yes
Requantise BF16 to FP8 (per-block scale)	0.00	1.00	no	yes	no

Source: analysis/results.md, section 2. Budget on QSFP56: 0.40 (low) to 3.96 (high) operations per byte.

Verdict on the hand-off codec

Frequency keep-k is the codec that fits: built from FFT and a fixed gather, and it fits every budget if the optics does the transform without charge. FP8 would fit the arithmetic budget at the high end but needs rounding, which the module does not publish. In the simulator, FP8 at the GPU was as fast as FP8 in transit (both 166.6 ms hand-off p99 at 14 req/s): the GPU pass is short next to prefill, so in-transit compression saves GPU time, not latency. Whether a fixed band keeps enough of real KV is an accuracy question; FreqKV's result says yes for half, after fine-tuning.

05

KV Tiering and Prefix Caching

Production systems keep KV caches beyond GPU memory and fetch them back when a prefix is reused: a system prompt, a document, an earlier turn of a chat. Every fetch is a hand-off from storage instead of from a prefill pool.

GPU HBM ↔ CPU DRAM ↔ local SSD ↔ shared KV pool over the network

Mooncake (Kimi's serving platform) is KV-cache-centric: it separates prefill and decode clusters and builds a disaggregated KV cache from the cluster's otherwise under-used CPU, DRAM and SSD. Its scheduler moves reusable KV to the prefill instance and streams new KV to decode in chunks or layers (arXiv:2407.00079).
LMCache pulls KV out of GPU memory for vLLM and SGLang, supporting both offload for prefix reuse and prefill-decode transfer, with an API for movement and compression across GPU, CPU, storage and network (arXiv:2510.09665). The same group's CacheGen encodes KV into a compact bitstream for network fetch (arXiv:2310.07240).
Why reuse pays: in Disaggregated_Inference_Sim's prefix-caching model (Mistral-7B, one A100), a 2,048-token shared system prompt gave a 79.9% hit rate and cut TTFT p50 from 241.8 ms to 45.2 ms (section 21). That cache sits in GPU memory; a tiered cache adds the fetch.

Sizes are those of slide 02. Fetching a 32,768-token Llama-3-70B prefix (10,737.4 MB) over 100 GbE takes 859.01 ms, against 23.87 ms over NVLink 4 (results.md section 4). Over a network, the fetch is bandwidth-bound and sits in front of the first token.

Compression in transit, both ways

A tier is a better fit than the hand-off for an in-link codec: data is compressed once on the way out and decompressed on every fetch, at the network edge of a storage pool where no GPU is available to do it. The same frequency keep-k codec applies, and the same caveats: a fixed band, an accuracy check, and a free transform. The authors of LMCache and Mooncake report large throughput gains from tiering, which says the flow matters; neither says an in-link codec would help.

06

Fetch Only What Matters: Query-Aware KV Selection

Moving fewer bytes beats compressing them. For a long context, a decode step's attention is dominated by a few tokens, and which ones depends on the query.

query q → score each page summary (MAC) → choose the top pages (comparisons) → fetch only those pages

Quest keeps, for each KV page, the element-wise minimum and maximum of its keys, uses the query to estimate an upper bound on each page's attention score, and loads only the top-K pages. The authors report up to 7.03× self-attention speed-up and 2.23× lower inference latency with negligible accuracy loss on long-dependency tasks (arXiv:2406.10774, ICML 2024). Quest's bound takes a per-channel maximum, which is itself a comparison; a plain dot product against a representative key is pure MAC.
The arithmetic fits the budget's high end. Scoring a key against the query (head dimension 128) is a dot product: 2 FLOPs per BF16 value, 1.00 operation per byte, built from MAC. It fits the high budget (3.96) but not the low (0.40) (results.md, section 2).
Choosing the top pages needs comparisons, outside the published set. The module could score in the data path and hand the scores to a host that ranks them; the ranking is over one number per page, so it is cheap.

Being explored optically

PRISM (Park and Park, arXiv:2603.21576; other authors, not the module’s developer) proposes a thin-film lithium niobate photonic similarity engine for KV block selection, arguing that ranking needs only 4 to 6 bits. The authors report 100% needle-in-a-haystack accuracy from 4K to 64K tokens on Qwen2.5-7B with simulated hardware impairments, and 16× less KV traffic at 64K. These are the authors' claims, not yet reproduced. It shows the scoring step is a natural candidate for analogue optics; its value in transit depends on the KV living somewhere remote.

07

Ring Attention and Context Parallelism

A prompt too long for one GPU is split across devices. Each device keeps its query block and passes its KV block to the next device in a ring, computing blockwise attention against each block as it arrives, until every query block has seen every KV block.

Ring Attention (Liu, Zaharia and Abbeel, arXiv:2310.01889, ICLR 2024) overlaps the KV block transfers with blockwise attention, so sequence length scales with device count, and attention stays exact.
The traffic is KV, again: every layer of a long prefill moves each device's KV block round the whole ring. The per-token bytes are those of slide 02.
Compression fits the operation set: an FFT along the tokens of each block and a fixed band is the same kernel as slide 04.
But it breaks the point of the method. Ring attention's selling point is exact attention; a lossy codec on every block, every layer, accumulates error through the prefill. And the design already hides the transfer behind compute when the link is fast enough, so compression pays only when the ring's link is slower than the attention block it overlaps.

Verdict: weak. Technically in the set; the error budget and the overlap leave little to gain.

08

MoE All-to-All and Tensor-Parallel All-Reduce

Two collectives run inside a pool on every layer. They are frequent and on the critical path, but they carry activations, not cached state.

MoE expert-parallel all-to-all

Each token's activation is dispatched to the GPUs holding its chosen experts and the results combined back (GShard, arXiv:2006.16668).
In Disaggregated_Inference_Sim (Mixtral-8x7B on 2 H100s, section 24), EP2 prefill of 4 × 2,048 tokens takes 211.5 ms, of which 7.1% is the link; decode at batch 32, 3.4%.
Practice already compresses dispatch: DeepSeek-V3 quantises activations to FP8 before the MoE dispatch in training (arXiv:2412.19437, low-precision communication), and its DeepEP library offers FP8 dispatch for inference as well. That is rounding, outside the set.
Activations feed the expert's matmul directly; a frequency band along tokens has no reason to hold their energy.

Tensor-parallel all-reduce

Partial sums are added across GPUs after attention and after the MLP.
In the same simulator (Llama-3-70B, decode at batch 1), the all-reduce is 26.2% of an 18.3 ms step at TP4 and 61.5% of 18.2 ms at TP8 (section 24; first-order, illustrative).
Summing streams costs 0.50 operations per byte and is in the set, but needs both streams at one point: a switch, not a pluggable (results.md, section 2).
NVSwitch already reduces in the network (NVLink SHARP), so this place is taken.

Verdict: weak to uncertain. MoE dispatch is a modest share of step time in the simulator, and activations tolerate little noise without the rounding-based FP8 recipe. The all-reduce is a strong in-network candidate, but for a switch, where it already exists.

09

Weight Streaming

Weights move when a model is loaded, swapped for another, offloaded layer by layer to fit memory, or when a LoRA adapter is paged in for a request.

Model	BF16 weights (GB)	Load over PCIe Gen5 x16 (s)
Llama-3-8B	16.06	0.25
Llama-3-70B	141.10	2.20

Source: analysis/results.md, section 4.

It happens at cold start, not per token. A serving fleet loads a model once and serves it for hours; the load is amortised over every request that follows.
Adapters are small and frequent. S-LoRA serves thousands of adapters by paging them between host and GPU memory (arXiv:2311.03285). Small transfers are latency-bound, not bandwidth-bound.
Weights need high accuracy. Weight quantisation (FP8, INT4) is applied once, offline, with calibration; it is not something to redo lossily on every load. A transform codec on the wire would have to be lossless, and a fixed frequency band is not.
Offload during decode (streaming layers from host memory each step) is the one case where weights move per token, and there the cure is to keep the weights resident, not to compress them in flight.

Verdict: weak.

10

Confidential AI: FHE and Encrypted Links

Confidential inference is the one place the module's operations are already proven, because they were built for it.

FHE inference

Under fully homomorphic encryption the server computes on ciphertexts it cannot read. CKKS handles approximate real arithmetic; its cost is dominated by NTTs and modular arithmetic, and bootstrapping refreshes the noise budget.
The module’s own demonstration is CKKS bootstrapping on the QSFP56 module (Kundu et al., arXiv:2608.21536). Exact modular arithmetic comes from an analogue transform by digit decomposition, as the FHE Accelerator Simulators series explains.
FHE inference is compute-bound and orders of magnitude slower than plaintext; it is a niche, not a fleet workload.

Encrypted traffic

Bulk link encryption (MACsec, IPsec, TLS) typically uses AES, a block cipher of table look-ups and XORs over GF(2): not FFT, NTT or MAC.
Post-quantum key exchange, ML-KEM (FIPS 203, August 2024), multiplies polynomials of degree 256 modulo 3,329 using NTTs: in the set.
But a handshake is tiny: an ML-KEM-768 encapsulation key is 1,184 bytes and a ciphertext 1,088 (FIPS 203). It runs once per session, then AES carries the data.

Verdict: proven technically, small payoff in throughput. The module fits FHE by design, and ML-KEM's NTT fits too, but neither is a high-volume flow in an inference fleet. Bulk encryption, the high-volume part, uses an operation the module does not offer.

11

The Candidate Table

Every inference flow against the five tests of deck 01: bottleneck (link-bound), operations (fits published ops), budget (work in transit), precision (noise tolerance) and placement (a pluggable sees one flow, a switch many).

Candidate	Traffic	Work in transit	Fits published ops	Link-bound	Noise tolerance	Verdict
KV tier fetch / offload	KV to and from CPU/SSD pools	FFT + fixed band, both ways	yes	often, over a network	moderate (2- to 3-bit KV tolerated; FreqKV half)	best inference candidate
KV hand-off	KV, prefill to decode	FFT + fixed band; FP8	keep-k yes; FP8 no	only on slow links near saturation	moderate	plausible, narrow
Query-aware selection	KV page summaries	score pages (MAC, 1.00 / byte)	scoring yes; top-k no	when KV is remote	high (ranking only)	promising, needs host ranking
Ring attention	KV blocks round a ring	FFT + fixed band	yes	rarely; overlapped	low (exact attention)	weak
MoE all-to-all	token activations	FP8 dispatch	no (rounding)	7.1% of a prefill step (sim)	low to moderate	weak / uncertain
TP all-reduce	partial sums	add, 0.50 / byte	yes	yes at TP8 decode (61.5%, sim)	exact sum	switch, not pluggable; exists
Weight streaming	weights, adapters	lossless only	no useful kernel	cold start only	none	weak
FHE inference	ciphertexts	NTT, modular arithmetic	yes (demonstrated)	no, compute-bound	exact by decomposition	proven, niche
PQ key exchange	ML-KEM handshakes	NTT	yes	no, kilobytes	exact	proven, negligible volume

The strongest candidate is in training

Over a 100 GbE cross-data-centre link, one Llama-3-70B BF16 update takes 11.3 s uncompressed (results.md, section 3), against 53.71 ms for a 2,048-token Llama-3-70B hand-off on the same link (section 4). Deck 02 covers cross-data-centre gradient sync, the place compression in transit would act on the most bytes.

12

What a Simulator Needs to Answer This

Disaggregated_Inference_Sim already has an in-transit stage on the KV hand-off: no GPU time, 1 pJ per input bit, 1 µs added and 1.6 operations per line byte, all illustrative (results.md section 15). Answering the questions of slide 11 needs that stage generalised from “a codec on the hand-off” to a kernel on a link.

sketch: a kernel declared on a link (illustrative, not an existing API)

LinkKernel(
    name="freq-keep-half",
    ops_per_byte=0.0,          # arithmetic besides the transform
    op_types={"fft", "gather"},  # checked against {fft, ntt, modular, mac}
    transform_charged=False,  # the unknown that decides the verdict
    precision_bits=8,           # what the analogue path delivers
    ratio=2.0,
    placement="pluggable",       # or "switch": sees one stream, or many
)

Operations per byte and operation types, checked against the module's budget and published set. A kernel that needs a comparison or rounding is rejected or sent to the host.
Precision need, and an accuracy check for lossy codecs: run the codec on real KV or activations and measure perplexity or task accuracy. The simulator does not model accuracy today.
Placement: a pluggable sees one point-to-point stream; a switch sees many and can sum them.
Request traces with real prompt-length and prefix-reuse distributions, because the hand-off and the tier fetch bind only at high load on slow links.
All flows on one clock: hand-off, tier fetch, MoE all-to-all and all-reduce, so a codec on one link is judged by end-to-end SLO, not by its own transfer time.
An all-GPU baseline at equal power and cost: the same compression on the GPU, or simply a faster link. In section 15 the GPU matched the in-transit stage on latency.

Simulator: Disaggregated_Inference_Sim; recorded results: examples/results.md.

13

What to Take Away

Inference moves KV, mostly. 131,072 bytes per token for Llama-3-8B and 327,680 for Llama-3-70B: 268.4 MB for a 2,048-token 8B prompt, 10,737.4 MB for a 32,768-token 70B prompt.
The hand-off binds only on slow links near saturation. In the simulator, at 14 req/s over 25 GbE, 41.0% of requests met the SLO uncompressed and 100.0% with FP8 or freq-keep-k; over NVLink compression changed nothing. And the GPU compressed as fast as the link did.
Frequency keep-k is the codec that fits the published set (FFT and a fixed gather). Quantisation needs rounding. Keep-k fits the budget only if the optical transform is free.
KV tiering is the best inference candidate: a network-bound flow, compressed once and fetched many times, at a place with no GPU.
Query-aware scoring is a MAC at 1.00 operation per byte, which fits the high end of the budget; choosing the top pages needs comparisons. PRISM shows others are exploring the scoring optically.
Ring attention, MoE dispatch and weight streaming are weak: exact attention, noise-sensitive activations, cold-start-only weights. Confidential AI is proven but small in volume.
The biggest bytes are in training. Deck 02: cross-data-centre gradient sync.