Efficient Inference & Serving

The Five Papers

Training a large model is one problem; serving it cheaply, with low latency, at high throughput, is an entirely different one. These five publications define the modern inference and serving stack — each attacks a distinct bottleneck: memory bandwidth, FLOPs-per-token, model footprint, KV-cache fragmentation, and sequential decoding latency.

• FlashAttention — IO-Aware Exact Attention
• Switch Transformers — Sparse Mixture of Experts
• GPTQ — Accurate Post-Training Quantisation
• PagedAttention & vLLM — KV Cache as Virtual Memory
• Speculative Decoding — Draft, Verify, Accept

FlashAttention — Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao, Fu, Ermon, Rudra & Ré (Stanford), 2022 2022 · arXiv:2205.14135

The problem

Standard attention computes the full N×N score matrix, writes it to high-bandwidth memory (HBM), reads it back to apply softmax, writes again, reads again for the value matmul. On modern GPUs the arithmetic is cheap but the memory traffic dominates: attention is bandwidth-bound, not compute-bound. Materialising the quadratic matrix also makes memory scale as O(N²), capping context length.

The contribution

FlashAttention reorders the computation to be IO-aware. It tiles the query, key and value matrices into blocks small enough to fit in the GPU's fast on-chip SRAM, and uses an online softmax (running max and running sum, rescaled as new blocks arrive) so the full attention matrix is never written to HBM. The result is exact attention — not an approximation — that is several times faster and whose memory footprint drops from O(N²) to O(N).

Why it matters to a practising engineer

FlashAttention is the reason long-context models are economically viable. It is now the default attention kernel in PyTorch (scaled_dot_product_attention), vLLM, and every serious training stack. FlashAttention-2 (2023) improved work partitioning across warps for ~2× further speedup; FlashAttention-3 (2024) exploits Hopper-specific features (async TMA, FP8) to reach ~75% of an H100's theoretical FLOPs. You rarely call it directly — but if your attention is slow, the first question is whether the kernel is being dispatched at all.

Switch Transformers — Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Fedus, Zoph & Shazeer (Google), 2021 2021 · arXiv:2101.03961

The problem

In a dense transformer, every parameter is activated for every token, so raising capacity raises the per-token compute (FLOPs) in lockstep. That makes scaling to trillions of parameters prohibitively expensive both to train and to serve.

The contribution

Switch Transformers replace the dense feed-forward block with a sparse Mixture-of-Experts (MoE) layer: a learned router inspects each token and sends it to a single expert FFN (top-1 routing — the “switch”). With E experts the model holds roughly E× the parameters of a dense block, but each token still passes through only one expert, so per-token FLOPs stay flat. The paper simplifies prior MoE work (top-1 instead of top-2), adds a load-balancing auxiliary loss to keep experts evenly used, and uses selective-precision and capacity-factor tricks for stable training — reaching 1.6 trillion parameters.

Top-1 routing: the router picks exactly one expert per token. Total parameters scale with E; active parameters (and therefore FLOPs) per token do not.

Why it matters to a practising engineer

MoE is now mainstream: Mixtral 8x7B (Mistral, 2023) uses 8 experts with top-2 routing — ~47 B total parameters but only ~13 B active per token, giving the quality of a much larger model at the inference cost of a small one. The serving catch is memory: you must hold all experts in VRAM even though most are idle each step, and throughput depends on routing being well-balanced across the batch. When you see “A active of B total parameters”, you are looking at this paper's idea.

GPTQ — Accurate Post-Training Quantization for Generative Pre-trained Transformers

Frantar, Ashkboos, Hoefler & Alistarh, 2022 2022 · arXiv:2210.17323

The problem

A 175 B-parameter model in FP16 needs ~350 GB of memory just to hold its weights — far beyond a single GPU. Naive rounding to low precision (round-to-nearest) destroys accuracy on large models, and quantisation-aware training is far too expensive to redo on a foundation model.

The contribution

GPTQ is a one-shot, layer-wise post-training quantisation method. For each layer it minimises the reconstruction error of the layer's output using approximate second-order (Hessian) information — quantising weights one column at a time and updating the remaining full-precision weights to compensate for the error just introduced. It quantises down to 3–4 bits with negligible accuracy loss, and can process a 175 B model in a few GPU-hours using only a small calibration set.

Weight memory only; excludes activations and KV cache. INT4 turns a 70B model from a multi-GPU deployment into a single-A100 deployment.

Why it matters to a practising engineer

GPTQ (and the GPTQ file format) is one of the dominant ways quantised open-weights models are distributed and run — you have almost certainly downloaded a -GPTQ checkpoint. The complementary method to know is AWQ (Activation-aware Weight Quantization, Lin et al. 2023, arXiv:2306.00978): instead of Hessians it identifies the small fraction of salient weight channels — spotted via activation magnitudes — and protects them, which is often faster to apply and very competitive in accuracy. In practice you pick GPTQ or AWQ based on which has a clean kernel for your serving stack and your target hardware.

PagedAttention & vLLM — Efficient Memory Management for LLM Serving

Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang & Stoica (UC Berkeley), 2023 2023 · arXiv:2309.06180

The problem

During generation, every request accumulates a KV cache — the keys and values of all previous tokens. Classic serving systems allocate one large contiguous buffer per request, sized for the maximum possible length. This wastes enormous memory to internal and external fragmentation and reserved-but-unused slots, so GPU memory — not compute — becomes the limit on how many requests you can batch.

The contribution

PagedAttention applies the operating-system idea of virtual memory and paging to the KV cache. The cache is split into fixed-size blocks (pages) that need not be contiguous in physical GPU memory; a per-request block table maps logical positions to physical blocks. Blocks are allocated on demand (near-zero waste) and can be shared across requests — e.g. a common prompt prefix, or the parallel samples of one beam — via copy-on-write. Combined with continuous batching (admitting new requests as soon as slots free up), this multiplies serving throughput. It is the foundation of vLLM.

Why it matters to a practising engineer

vLLM is now one of the most widely deployed open-source inference servers. The paper reports up to 2–4× higher throughput than prior systems (FasterTransformer, Orca) at the same latency, by wasting under ~4% of KV memory instead of 60–80%. When you tune gpu_memory_utilization, block_size, or enable prefix caching, you are tuning the machinery from this paper.

Speculative Decoding — Fast Inference from Transformers

Leviathan, Kalman & Matias (Google), 2022/2023 2023 · arXiv:2211.17192

The problem

Autoregressive decoding is inherently sequential: each token needs the previous one. Generation latency is therefore dominated by the number of forward passes through the big model, and each pass is memory-bandwidth-bound (you reload all the weights to produce a single token), so the expensive GPU sits massively underutilised.

The contribution

Speculative decoding uses a small, cheap draft model to propose several future tokens, then runs the large target model once to score all of them in parallel. A clever acceptance/rejection test accepts the longest correct prefix and corrects the first wrong token from the target's own distribution. Crucially, the output distribution is provably identical to sampling from the target model alone — it is a pure latency optimisation, not an approximation — typically delivering a 2–3× speedup.

Why it matters to a practising engineer

Speculative decoding is now standard in production servers (vLLM, TensorRT-LLM, TGI). Speedup depends on the acceptance rate — how often the draft agrees with the target — so a well-matched draft model (or a self-drafting variant like Medusa heads, or n-gram / prompt-lookup drafting) matters. It pairs naturally with the previous papers: it reduces latency, while PagedAttention and quantisation reduce memory, and FlashAttention speeds each pass.

What to Take Away

• Inference is a systems problem. On modern GPUs the limiter is usually memory bandwidth and memory capacity, not raw FLOPs — the winning techniques respect the hardware.
• FlashAttention makes attention IO-aware: tiling + online softmax give exact attention with O(N) memory. It is the default kernel everywhere; FlashAttention-2/3 push it further on newer GPUs.
• Sparse MoE (Switch Transformers) decouples capacity from per-token compute. Mixtral and most frontier models now ship as MoE — great FLOPs efficiency, heavy on VRAM.
• GPTQ (and AWQ) shrink weights to ~4 bits post-training with minimal quality loss, turning multi-GPU models into single-GPU deployments.
• PagedAttention / vLLM manage the KV cache like OS virtual memory, eliminating fragmentation and enabling sharing — the basis of high-throughput open serving.
• Speculative decoding attacks sequential latency with a draft-then-verify scheme that preserves the target distribution exactly, for a 2–3× speedup.
• These five compose: FlashAttention (per-pass speed) + MoE (compute efficiency) + quantisation (footprint) + PagedAttention (throughput) + speculative decoding (latency) is the modern serving stack.