Making this kind of simulator fast without changing its answers: event abstraction, incremental state, lazy bookkeeping, exact macro-stepping, probe discipline, compiled kernels, parallel and multi-fidelity sweeps, surrogates, sampling, and parallel discrete-event simulation — each measured on the companion simulator.
Concepts used here, and where they are explained. Each links to its series glossary entry: a short explanation, then links to the slides that explain it in depth, in this series, the Local LLM Hosting and Key Publications decks, or the FHE Accelerator Simulators series.
A simulator's speed decides how many questions the architecture team can ask. At one minute per run, a 1,000-point design-space sweep is an overnight job; at one second it is a coffee break, and people start asking better questions. Speed is a feature, but only if the answers do not change.
| Family | Techniques | Changes answers? |
|---|---|---|
| Fewer events | Coarser event granularity; macro-stepping; analytic sub-models | Exact if nothing observable happens inside the merged events |
| Cheaper events | Incremental state, hoisted constants, lazy bookkeeping, fewer allocations | No (pure refactoring) |
| Cheaper probes | Lower sampling rates, streaming statistics, windowed tracing | Only the detail of the outputs |
| Faster execution | PyPy, Cython, C++ or Rust kernels | No, if verified differentially |
| Parallelism across runs | Process pools, job arrays, CI farms | No |
| Parallelism within a run | Conservative or optimistic PDES | No, but it is hard to get right |
| Smarter experiments | Analytic bounds, bisection, Bayesian optimisation, surrogates, common random numbers | Fewer runs for the same conclusion |
| Sampling | SimPoint-style representative regions, statistical sampling, checkpoints | Yes, with quantified error |
Guessing where a simulator spends its time is usually wrong. cProfile on the baseline (Llama-3-70B, 2,000 requests, 511 simulated seconds, before the power model was added) showed:
| Function | Own time | What it is |
|---|---|---|
decode_step_done | 0.32 s | Per-sequence token bookkeeping after every decode step |
list.append (1.09 M calls) | 0.17 s | Recording every inter-token latency |
decode_once | 0.13 s | Rebuilding the context list each step |
Cost model (decode, _time, properties) | ≈0.17 s | Recomputing model constants and allocating a result object every step |
SimPy kernel (step, _resume) | ≈0.11 s | The event loop itself |
sampler | 0.06 s | A passive probe |
The event kernel is a small fraction. Time goes into model code that runs per sequence per step. That points to two fixes: do less per step (incremental state), and stop touching every sequence on every step (lazy bookkeeping).
Tools: cProfile + snakeviz for call trees; py-spy for sampling without instrumentation overhead; perf and flame graphs for compiled cores.
The biggest acceleration happens before any optimisation: choosing what an event is. Nothing observable to the scheduler happens inside a decode step, so one event per batch step is exact for serving metrics.
| Event granularity, same 511 s of serving | Events | Relative |
|---|---|---|
| One per batch step (the simulator) | 37,099 | 1× |
| One per generated token | 505,895 | 14× |
| One per layer per step | 2,967,920 | 80× |
| One per clock cycle at 1.5 GHz | ≈7.7 × 1011 | ≈2 × 107× |
At batch-step granularity the plain SimPy model already runs at about 840× real time on a desktop i7-3770 (660–780× when first measured, on the same model before its 2026-10-03 correction; speed depends on how much the cost model computes per step and on the machine's load). The rule from deck 01 applies again: go finer only where the question needs it, for example per-layer events when studying layer-wise KV streaming.
ctx = [r.prompt_len + r.tokens_out
for r in self.running]
cost = self.cost.decode(ctx) # sums the list,
# recomputes params,
# allocates StepCost
for r in self.running: # touches every sequence
r.itls.append(now - r.last_token)
r.tokens_out += 1
r.last_token = now# once, outside the loop
fa, fb = 2*m.matmul_params, 4*m.n_layers*m.d_model
w = m.weight_bytes_streamed # layers + LM head
er = m.embedding_row_bytes # 1 row per token
kv = m.kv_bytes_per_token
# per step: a running sum and arithmetic
flops = fa*b + fb*(ctx + b) # + itself
nbytes = w + b*er + (ctx + b)*kv
dt = max(flops/fr, nbytes/br) + ovh
ctx += b # every sequence grew by 1
step_end.append(t) # one record per instancecached_property or locals; no recomputation per step.Between two state changes (an admission or a finish) a decode instance's future is deterministic: the batch is fixed and the context grows by one per step. So compute the whole run of steps in a tight loop and give SimPy one timeout for all of them. If new work arrives mid-way, interrupt: let the step in flight finish and admit the newcomer at that boundary, exactly as the baseline would.
k = min(steps_until_next_finish, horizon) # bounded look-ahead
ends = plan(k) # tight arithmetic loop
self._macro = True
try:
yield env.timeout(ends[-1] - env.now); done = k
horizon = min(4096, horizon * 2) # uninterrupted: look further
except simpy.Interrupt: # submit() interrupted us
j = bisect.bisect_right(ends, env.now) # the step in flight
yield env.timeout(ends[j] - env.now); done = j + 1
horizon = max(8, 2 * done) # interrupted: look less far
self.commit(ends[:done], ...)test_fast_forward_is_exact), because the step times are summed in the same order.From examples/benchmark_acceleration.py on an i7-3770 desktop (4 cores, 8 threads; Python 3.12, SimPy 4.1), with the power model (deck 07) enabled, as recorded in examples/results.md. Every speed-up is checked for identical results. Rerun on 2026-10-03 after the cost model was corrected: steps no longer read the whole embedding table, and decode attention includes each token's attention to itself (Toolkit deck 10). The fast-path code on slide 04 shows the corrected terms.
| Technique | Scenario | Before | After | Gain | Answers |
|---|---|---|---|---|---|
| Event abstraction (step, not token) | Defaults, 2,000 requests | 506k events | 37.1k events | 14× fewer events | Same model |
| Fast path (incremental + lazy + macro) | Defaults (1P1D, 4 req/s) | 0.74 s | 0.35 s | 2.1× | Bit-identical |
| Fast path | 1k-token outputs, 1 req/s | 2.20 s | 0.87 s | 2.5× | Bit-identical |
| Fast path | 2P2D, 6 req/s | 0.72 s | 0.36 s | 2.0× | Bit-identical |
| Probe rate (5 ms → off) | Defaults | 1.02 s | 0.55 s | 1.9× | Same metrics, less timeline |
| Parallel sweep, 8 processes | 8 rates | 1.46 s | 0.90 s | 1.6× | Identical |
| Analytic bracket + bisection | Max load at 90% SLO | 40 runs, 7.96 s | 7 runs, 1.23 s | 6.5× | 5.97 vs 5.75 req/s (grid step 0.25) |
Macro-stepping pays most when there are many steps per state change (long outputs, few arrivals per instance); with frequent arrivals it is cut short often. Adding the power model lowered the fast path's gain from 2.4–3.1× to 2.0–2.6× when it was added (2.0–2.5× in the current recording): the per-step DVFS and energy arithmetic is work macro-stepping cannot remove. Model fidelity costs simulator speed, and Amdahl's law (slide 14) says exactly how much. The parallel sweep is far below 8× because each run takes only about 0.2 s, so process start-up and data transfer dominate; with minute-long runs it approaches the core count. The largest gain came from doing fewer simulations, not faster ones.
Sampling the system every 5 ms of simulated time nearly doubled run time (1.02 s against 0.55 s with sampling off). Probes are code, and code costs time.
tracer.enabled before formatting anything).| Option | Effort | Typical fit | Caveat |
|---|---|---|---|
| PyPy | None (if dependencies allow) | Pure-Python, generator-heavy SimPy models | C-extension dependencies; measure, as gains vary |
| Cython / mypyc | Low to medium | Hot model functions with typed loops | Build step; the hot path must be type-annotated to gain much |
| Numba | Low | Numerical kernels (cost models, vectorised analytics) | Poor fit for generators and object-heavy event code |
| NumPy / JAX vectorisation | Medium | Analytical layers evaluated over thousands of configurations at once | Only for the closed-form parts |
| C++ core + pybind11 | High | The event loop and models, with Python configuration | Two languages to maintain; the usual industry end state |
| Rust core + PyO3/maturin | High | As C++, with memory safety and easy packaging | Team skills; ecosystem smaller than C++'s |
Whatever the route, keep the Python model as the reference and add a differential test in the style of test_javascript_port_matches_python: the compiled core must reproduce the reference exactly on a fixed set of workloads. That test is what makes a rewrite safe.
Sweeps, replications and searches are made of independent runs. Parallelising across runs is the simplest and most reliable speed-up there is.
def sweep(cfg, wl, rates, workers=1):
jobs = [(cfg, wl, r) for r in rates]
if workers <= 1:
return [_attainment(j) for j in jobs]
with ProcessPoolExecutor(max_workers=workers) as ex: # processes, not threads: the GIL
return list(ex.map(_attainment, jobs))A closed-form capacity bound costs microseconds. For the defaults it says prefill caps throughput at 7.49 req/s (decode 28.8, link 74.5). The simulator then only searches below that ceiling, because the bound is optimistic by construction.
SLO attainment falls with load, so bisection on the 90% threshold converged in 7 simulations to 5.97 req/s. A 0.25-step grid needed 40 to say 5.75. Add a tolerance for the noise in tail metrics, or bisect on the mean of a few replications.
For many parameters (prefill:decode ratio, TP degree, batch caps, link), fit a Gaussian-process surrogate to completed runs and choose the next run to maximise expected improvement in goodput per GPU. It typically needs far fewer runs than grids, and libraries such as Optuna and Ax make it routine.
Vidur learns operator runtimes so the simulator need not profile every shape. Going further, a regressor trained on simulator outputs can screen thousands of configurations before the simulator confirms the best few. Early stopping aborts a run once its SLO verdict is statistically certain.
Common random numbers (deck 06) cut the replications needed to compare two designs, often a bigger saving than any code optimisation.
Detailed simulators (cycle-level, or serving simulators with very fine models) cannot afford to simulate everything. Computer architecture solved this two decades ago.
For LLM serving the natural units to sample are traffic regimes (quiet, typical, burst), each simulated in detail and weighted by how often it occurs in production traces.
When one run is too big for one core, split the model into logical processes (LPs) that exchange timestamped messages.
An LP processes an event only when it is certain no earlier message can arrive. That needs lookahead: a minimum delay on every link between LPs. Physical link latency is natural lookahead; null messages prevent deadlock. SST and parallel gem5 work this way.
LPs run ahead speculatively and roll back when a straggler message arrives in their past, sending anti-messages to cancel what they sent. More parallelism, but state saving and rollback cost memory and time: the macro-stepping trade-off writ large.
Each event is tiny and the pools are tightly coupled through the routers, so synchronisation would cost more than it saves. PDES pays off for large models with many loosely coupled components and real lookahead: a datacentre of accelerators, a large NoC, a full SoC with many cores. Parallelism across runs comes first.
The same ideas apply when the simulator under pressure is RTL, which is where pre-tape-out verification spends its compute.
| Technique | What it does |
|---|---|
| Verilator | Compiles synthesisable SystemVerilog to optimised, optionally multi-threaded C++: a cycle-based (not event-driven) model, often much faster than interpretive event-driven simulators on large designs |
| Smaller RTL islands | Co-simulate only the block under test in RTL, with the rest of the system in TLM or the Python model (deck 01) |
| Emulation | Palladium, Veloce, ZeBu: around 1 MHz, enough to run real software and long workloads |
| FPGA prototyping | Tens of MHz; the fastest pre-silicon platform, at the cost of debug visibility and partitioning effort |
| Hybrid fast-forward | Boot and warm up in a virtual platform, checkpoint, then transfer state to emulation or RTL for the window of interest |
| Regression farm discipline | Prioritise tests by coverage gained per CPU hour; run the expensive tiers nightly |
Every technique speeds up only the fraction of time it touches. Set the profile shares (they are normalised) and the speed-up of each fix, and see the end-to-end gain. The defaults approximate the profile on slide 02 and the measured gains.
The sweep figure models a serial overhead per run (dispatch, pickling, result collection) that parallelism cannot hide: speed-up = single ÷ (1/workers + overhead). With the defaults it comes out well below the worker count, as measured on slide 06.
Deck 09 connects the simulator to the outside world: PyTorch, ONNX Runtime and HEIR.