What You Are Simulating — LLM Inference Workloads

Topics We'll Cover

• Two Phases: Prefill and Decode
• Counting FLOPs
• Counting Bytes
• The Roofline
• Interactive: Put a Batch on the Roofline
• KV-Cache Arithmetic
• Batching: Static, Continuous, Chunked, Paged
• Parallelism and Its Communication Bill
• The Metrics Users Feel: TTFT, TPOT, Goodput
• A Cost-Model Checklist
• What to Take Away

Two Phases: Prefill and Decode

Every LLM request runs in two phases with opposite hardware characters. Getting this distinction into the simulator is the single most important modelling step.

Counting FLOPs

For a dense decoder with P matmul parameters, L layers and model width d:

linear layers      2 · P                       # one multiply + one add per weight
attention at pos c 4 · L · d · c                 # QKᵀ and AV, each 2·d·c per layer

prefill(s)  = 2·P·s + 2·L·d·s(s+1)          # causal: sum of c over 1..s
decode(B,c) = 2·P·B + 4·L·d·(Σcᵢ + B)       # B sequences, cᵢ cached tokens each, + itself

The parameter counts come from the shapes alone (GQA attention, SwiGLU MLP, untied embeddings) and match the published model sizes; the simulator's unit tests check this. Note that attention FLOPs grow with context and become significant at long context even though the linear layers dominate at short context.

Counting Bytes

Time is set by whichever is slower: doing the arithmetic or moving the data. Bytes moved from HBM per forward pass:

weights      = (params − V·d) × bytes_per_weight   # layers + LM head, once per step
emb(n)       = n · d · bytes_per_weight            # embedding lookup: rows used only
kv_per_token = 2 · L · n_kv_heads · head_dim · bytes  # K and V

prefill(s)   = weights + emb(s) + s · kv_per_token          # write the new KV
decode(B,c)  = weights + emb(B) + (Σcᵢ + B) · kv_per_token  # read all KV, write one more token each

For Llama-3-70B in BF16 the weights occupy 141 GB. A decode step must stream 139 GB of them from memory however small the batch: everything except the 2.1 GB input-embedding table, of which it reads only the rows its tokens look up. On four H100s (4 × 3.35 TB/s at 80% achievable) that floor is about 13 ms per step before a single KV byte is read. Batching amortises this one read across many sequences, which is why serving systems batch hard.

Corrected on 2026-10-03. Until then the simulator charged every step all 141 GB, embedding table included, and left out each decoded token's attention to itself (slide 02). Tracing the real model found both (Toolkit deck 10). The floor moved from 13.16 to 12.97 ms.

The Roofline

The roofline model (Williams, Waterman & Patterson, 2009) plots attainable performance against arithmetic intensity I = FLOPs / bytes. Below the ridge point the kernel is memory-bound; above it, compute-bound.

t = max( FLOPs / (peak_flops · MFU_eff),  bytes / (mem_bw · MBU_eff) ) + overhead
ridge = (peak_flops · MFU_eff) / (mem_bw · MBU_eff)        # H100: 295 FLOP/B at peak, ~203 derated

Two lessons. In BF16 decode, intensity is roughly the batch size, so you need hundreds of concurrent sequences to approach the ridge, and the KV reads pull it further down. Prefill crosses the ridge after a few hundred tokens. These numbers come from the same cost model used in the simulator.

Interactive: Put a Batch on the Roofline

Choose a model and device, then move the decode batch, the context length and the prompt length. Both phases are plotted on a log–log roofline.

Try the hypothetical optical MAC: four times the matmul throughput, same memory. Prefill speeds up; decode hardly moves, because it is bandwidth-bound. Any novel compute fabric has to answer the memory question as well.

KV-Cache Arithmetic

The KV cache holds K and V for every layer and every token of every live sequence. With grouped-query attention (GQA) only the n_kv_heads are stored.

• Capacity limits the batch, and the batch sets decode efficiency. A serving simulator must track KV occupancy and admit requests only when their KV fits.
• Every byte of KV is read every decode step, so long contexts make decode more memory-bound still.
• The KV cache is what disaggregation must move. 671 MB per 2k-token prompt for 70B is 13 ms on a 50 GB/s InfiniBand NDR link, 1.5 ms on NVLink, and 215 ms on 25 GbE.

Batching: Static, Continuous, Chunked, Paged

Chunked prefill and disaggregation are two answers to the same interference problem. The first bounds the stall; the second removes it by putting the phases on different hardware.

Parallelism and Its Communication Bill

Large models span many devices. Each form of parallelism adds communication that the simulator must cost on the interconnect.

t_allreduce ≈ 2 · (n − 1) / n · bytes / link_bw  +  2 · (n − 1) · latency

The companion simulator treats a TP group as one larger device and ignores the all-reduce, a simplification its README states plainly. Adding it is a good first extension, and ASTRA-sim (deck 04) is the tool for when collectives are the question.

The Metrics Users Feel: TTFT, TPOT, Goodput

A Cost-Model Checklist

The roofline gets the first-order answer. Before trusting a simulator's absolute numbers, check what it does about each of these:

The professional fix is calibration: profile real kernels (or RTL), fit per-operator tables, and let the simulator interpolate. This is Vidur's approach, and the one deck 06 builds into a validation flow.

What to Take Away

• Prefill is compute-bound GEMM; decode is bandwidth-bound GEMV. Model them separately.
• FLOPs ≈ 2·P per token plus attention; bytes ≈ all the weights per step (bar the embedding table, a lookup) plus all the KV.
• The roofline gives step times good to first order, and shows why decode needs big batches.
• KV capacity limits batch size, and KV size is what disaggregation must move.
• Continuous batching is the scheduler loop every serving simulator implements.
• Report percentiles and goodput, not means and raw throughput.