Disaggregated Inference, Simulated

Topics We'll Cover

• The Interference Problem
• The Idea: Split the Phases
• Disaggregation in Production
• What the KV Transfer Costs
• The Simulator: Model and Assumptions
• Interactive: Run the Simulator
• Six Experiments to Try
• Results from the Python CLI
• Reading the Code
• Limitations and Extensions
• Asymmetric Models: Encoder-Only Prefill
• What to Take Away

The Interference Problem

On a colocated server, prefill and decode share each GPU. A vLLM-v0-style scheduler runs a waiting prefill before the next decode step, so every sequence mid-generation stalls for the whole prefill.

Chunked prefill (Sarathi-Serve) bounds the stall by slicing prompts into pieces. Disaggregation removes it by putting the two phases on different machines.

The Idea: Split the Phases

Two papers from 2024 made the case. Splitwise (Patel et al., Microsoft, ISCA 2024) characterised production traces and argued for separate prompt and token machine pools, even different GPU generations for each. DistServe (Zhong et al., OSDI 2024) framed the goal as goodput, the request rate served within both TTFT and TPOT SLOs, and searched placements by simulation.

Disaggregation in Production

These systems move quickly; treat the table as a map for further reading, not a feature matrix. The common thread is that the KV cache has become a first-class object that is stored, moved, routed on and reused, and that is what makes data movement the core modelling problem.

What the KV Transfer Costs

Bytes to move = prompt tokens × KV bytes per token. For Llama-3-70B in BF16 that is 320 KiB per token, so a 2,048-token prompt carries 671 MB.

• Transfer time is one cost; queueing for the link is another. At high utilisation the wait dominates, just as the M/D/1 formula predicts.
• Layer-wise streaming (send each layer's KV as soon as it is computed, as DistServe and Mooncake do) overlaps transfer with prefill compute. The companion simulator transfers after prefill completes, which is the pessimistic bound.
• Interconnect is the lever where optical and photonic technology has the most obvious near-term role.

The Simulator: Model and Assumptions

Interactive: Run the Simulator

This is a line-for-line JavaScript port of the SimPy model. On identical request streams it reproduces the Python simulator's per-request timestamps exactly; the repo's test suite checks this. Power cap, decode-pool cap and DVFS controls drive the power model from deck 07. Set a configuration and press Run, Compare (colocated on the same number of instances) or Sweep (goodput against load, both modes).

Six Experiments to Try

1. Find the interference. Defaults (70B, 4×H100 per instance, 1 prefill + 1 decode, 4 req/s), then Compare. Colocated runs two instances that each do both jobs; its dashed inter-token curve grows a long tail of ~130–200 ms stalls while disaggregated stays flat near 15 ms.
2. Push the load. Sweep from the defaults. Colocated loses goodput first, because the TPOT SLO fails; disaggregated holds until the single prefill instance saturates, and then TTFT fails instead. Add a second prefill instance (2P1D) and sweep again.
3. Starve the link. Set 2 prefill instances, 6 req/s and 25 GbE. The hot-spot moves to kv_wait → kv-link and most of each request's life is spent queueing for the network. Add channels or a faster link and watch it move back.
4. Make compute nearly free. Switch to the hypothetical optical MAC. Prefill (compute-bound) gets several times faster, TTFT drops, and decode barely changes, because it is bandwidth-bound. Fast compute shifts the bottleneck to memory and data movement.
5. Spend fewer watts. Turn DVFS on: energy per token drops and latency does not move. Then lower the decode-pool cap: down to about 250 W per device TPOT barely changes; near 200 W it collapses. Now cap everything at 400 W: TTFT suffers because prefill is compute-bound and runs near TDP. Add a second prefill instance and the latency comes back at lower power per device.
6. Change the workload. 8k-token prompts with 64-token outputs (summarisation) against 256-token prompts with 1k-token outputs (generation). The best prefill:decode ratio flips; no fixed ratio suits both, which is the provisioning risk of disaggregation.

Results from the Python CLI

The same experiments from the command line (disagg-sim --compare --sweep-rate 2 3 4 5 6 8): Llama-3-70B, 4×H100 per instance, 2,048-token mean prompts, 256-token mean outputs, SLOs TTFT ≤ 1 s and TPOT ≤ 25 ms, 800 requests.

The same runs report energy: colocated uses 3.09 J per output token at 2,993 W; disaggregated 2.77 J at 2,687 W, because decode steps never idle behind prefills. Deck 07 builds on this.

With two prefill instances (2P1D, three instances in total) the disaggregated system meets both SLOs for 100% of requests up to 10 req/s with TPOT p99 of 17 ms. At 4 req/s the decode instance runs at a mean batch of about 14 with an MBU of about 77%: memory-bound, as deck 03 predicted.

Reading the Code

The Python package Disaggregated_Inference_Sim is about 2,400 lines of Python (it was 1,150 before the 2026-10-04 and 2026-10-05 extensions), written to be read top to bottom.

pip install -e .[dev] && pytest
disagg-sim --compare                                  # colocated vs disaggregated
disagg-sim --prefill 2 --sweep-rate 4 6 8 10          # where does goodput collapse?
disagg-sim --prefill 2 --rate 6 --link eth-25g        # watch the hot-spot move to the link
disagg-sim --trace run.json                           # open in https://ui.perfetto.dev

Limitations and Extensions

Each of these is a self-contained exercise, roughly in order of difficulty, and each is a real feature of production serving simulators.

1. Layer-wise KV streaming: overlap the transfer with prefill by sending one layer's KV at a time.
2. Tensor-parallel communication: add the all-reduce term from deck 03 and a scale-up link per instance. Now built: --tp, --pp and --ep, measured in Inference Trade-offs Explained, chapter 8.
3. Chunked prefill in colocated mode: a per-step token budget mixing prefill chunks with decodes. Is it as good as disaggregation? Now built: --batch-policy chunked; the answer depends on the workload (Inference Trade-offs Explained, chapter 1).
4. Heterogeneous pools: H100 for prefill, A100 or an optical part for decode (Splitwise's proposal). Now built: --prefill-device and --decode-device, with an optical transform engine in the prefill pool, in Fourier Optics for Inference 03 (optical prefill pools, simulated).
5. Asymmetric models: a model whose prefill runs only part of the network (a causal encoder-decoder). Now built: --ced-encoder-layers, --ced-replay and --ced-replay-on, on the next slide.
6. Pre-emption and paged KV: reserve KV incrementally rather than up front; evict or swap when full. Now built: --kv-policy paged and --preemption recompute|swap, in both pools (chapter 2).
7. Prefix caching and a KV store (Mooncake): cache hits skip prefill and transfer. Prefix caching is now built (--prefix-caching, chapter 3); a KV store shared between pools is not.
8. Calibrated cost model: replace the roofline with a lookup table of measured or RTL-derived step times.
9. Per-request energy attribution: split each step's joules across the sequences in its batch, so energy can be reported per request class (the pool-level power model is already in place, deck 07).

Asymmetric Models: Encoder-Only Prefill

Everything so far assumed prefill and decode run the same model. A 2026 design breaks that. DeepSeek-V4.1-Flash’s Causal Encoder-Decoder (DeepSeek-AI, arXiv:2609.19969, section 2.2) makes the bottom half of its 40 layers a causal encoder and projects every decoder layer’s KV from the last encoder state, so a prompt token activates 8B parameters and a generated token 16B. The architecture is explained in Modern Architectures 06; this slide is about what it does to the pools.

Both placements are in the simulator (--model llama3-70b-ced, --ced-replay-on prefill|decode; Python and the JavaScript above agree exactly). On a dense Llama-3-70B shape split 40 + 40 (illustrative), one 8,192-token prefill takes 288.3 ms instead of 564.3 ms. Highest request rate with 90% of requests inside both SLOs, best split of six 4×H100 instances:

• The pool split moves. With half-price prefill, the best split shifts one instance towards decode at 2,048 : 512 and 8,192 : 128: provisioning ratios tuned for a decoder-only model are wrong for an asymmetric one.
• Gains above 2× are queueing, not FLOPs: the TTFT SLO is fixed (5× the decoder-only unloaded prefill), so halving service time cuts waiting by more than half.
• Smaller prefill instances fit. With the replay on decode a prefill instance holds 71.9 GB of weights instead of 141.1 GB; on two H100s that leaves KV room for 220,048 tokens instead of 8,835.

Every figure is in sections 16–18 of examples/results.md. The prompt:output ratios stand in for agent loops and are illustrative; the proxy has no MoE, sparse attention or prefix-cache hits. Animated, with the pools at work: Inference Trade-offs Explained, chapter 7.

What to Take Away

• Colocated prefill stalls decode, destroying p99 inter-token latency under load.
• Disaggregation separates the bottlenecks, so TTFT and TPOT can be provisioned independently. Its cost is moving the KV cache.
• KV transfer is milliseconds on NVLink and hundreds of milliseconds on commodity Ethernet; the link can become the hot-spot.
• A discrete-event simulator answers the sizing questions (ratio, link, batch, SLO) that analytical models cannot, because queueing and interference emerge only when you simulate them.
• The browser port and the Python package agree exactly, an example of the cross-validation habit deck 06 formalises.
• Whether to disaggregate depends on the workload: measured across five workloads, with every other lever, in Inference Trade-offs Explained (disaggregation, the KV hand-off).