Performance Analysis of Simulators and Systems

Topics We'll Cover

• Measure Before You Optimise
• Two Methods: USE for Resources, Profile First for Code
• USE in Practice: a Parallel Sweep
• Linux perf: Counters and Call Stacks
• Sampling Profilers and Flame Graphs
• Interactive: Flame Graphs of Two Simulators
• Reading a Profile: Self and Total Time
• Counting Instructions With Cachegrind
• Closing the Loop: Sort Once
• Benchmarks Are Distributions
• Interactive: Designing a Regression Gate
• Regression Detection in CI
• What to Take Away

Measure Before You Optimise

A simulator that takes a minute gets run a hundred times a day; one that takes an hour gets run once, and its users start guessing instead. So simulator speed matters, and the first rule of making it faster is the one LLM Inference Simulators 08 calls step zero: profile first. Intuition about where time goes is wrong often enough that it is not worth acting on.

This deck applies the standard tools to the simulators in this series, on one desktop machine (an Intel i7-3770 with 8 logical CPUs, with a desktop session running), and reports what they found, including two things that were not expected:

• The Rust port of Disaggregated_Inference_Sim spends about half its instructions sorting, in its metrics code, not in its simulation engine.
• On this machine, a single-run timing gate with a 5% margin raises a false alarm on unchanged code 25% of the time.

Every number comes from snippets/t11/run_t11.py and profile_first.py, recorded in RESULTS.md. They were all re-measured on 2026-10-03, after three changes. The profiled simulators' cost model was corrected (deck 10). The sort-once fix of slide 09 was applied. And an administrator enabled perf, which slide 04 now uses.

Two Methods: USE for Resources, Profile First for Code

• The USE method (Brendan Gregg) checks every resource (CPUs, memory, disks, network, locks) for Utilisation, Saturation (work queued because the resource is busy) and Errors. It finds system bottlenecks quickly, because it is a checklist rather than a guess.
• The profile-first loop finds code bottlenecks. Its last step is the one that is skipped most often: check that the answers did not change. For a simulator, a faster wrong answer is a regression.
• Profile the workload users actually run. A micro-benchmark of the event queue says nothing about a run dominated by metrics.

USE in Practice: a Parallel Sweep

Rust_DES_Kernel's disagg-rs --sweep runs one simulation per arrival rate in parallel with rayon. Here are eight rates of 4,000 requests each, under /usr/bin/time -v, on one thread and on eight:

Source: snippets/RESULTS.md in _simeng_build

• Utilisation: eight threads kept 76.1% of the 8 logical CPUs busy and ran 2.8× faster, not 8×. The i7-3770 has 4 cores with 2 hardware threads each, so 8 threads share 4 sets of execution units. The desktop was also running.
• Saturation: involuntary context switches rose from 10 to 201. Threads were pre-empted because there were more runnable threads than free CPUs.
• Memory: peak resident memory grew 7.2×, because each simulation holds its own state. On a 15 GB machine, memory is the resource to watch for wide sweeps.
• Errors: none here. In CI, out-of-memory kills and timeouts are the errors to look for.

Linux perf: Counters and Call Stacks

perf is the Linux profiler. perf stat counts hardware events: cycles, instructions, cache and branch misses, and from them instructions per cycle (IPC). perf record samples call stacks for a profile, and perf report and perf annotate show where the samples landed, down to the instruction. The perf wiki covers it in depth. Both were run here on the Rust and Python versions of the same simulator:

perf stat -r 5 -e cycles,instructions,cache-misses,branch-misses,... -- ./disagg-rs ...
perf record -e cycles:u -F 4999 --call-graph fp -- ./disagg-rs ...       # a frame-pointer build
perf record -e cycles:u -F 4999 --call-graph fp -- python3 -X perf -m disagg_sim ...
perf script | (fold the stacks) > flame graph                            # slide 06

Source: snippets/RESULTS.md in _simeng_build

• Neither is memory-bound. Rust retires 2.06 instructions per cycle, CPython 1.37. Rust's last-level cache misses are 50% of its last-level references, which looks alarming, but references are rare: it is about 1.2 misses per thousand instructions. Normalise counters per instruction before reading them.
• CPython mispredicts more branches (2.85% against 1.09%): the interpreter's dispatch on every bytecode is hard to predict.
• Getting usable stacks took two fixes. Release builds omit frame pointers, and DWARF unwinding lost the callers of most samples, so the Rust profile uses a build with -C force-frame-pointers=yes. Python's -X perf names Python functions for perf (Python 3.12 and later) but unwinds only through frame pointers, so it ran on Ubuntu's python3.12, which is built with them.

Sampling Profilers and Flame Graphs

A sampling profiler interrupts the program at a fixed rate (here 500 times a second) and records the call stack. Functions that take more time appear in more samples. The overhead is small and the program does not need changing. The price is statistical noise for anything that takes only a few samples.

A flame graph merges identical stack prefixes. The width of a frame is the share of samples in which it was on the stack (its total time). The top edge of each tower is where the time is actually spent (self time). The x-axis is not time, so a wide frame is a hot spot, not a long phase.

Interactive: Flame Graphs of Two Simulators

Real profiles. With py-spy: Disaggregated_Inference_Sim simulating 3,000 requests (792 samples), and Memory_System_Sim simulating 20,000 random requests on one HBM pseudo-channel (4,944 samples). With perf: the Rust port simulating 20,000 requests (1,723 samples), and the Python simulator under -X perf (8,091 samples; each stack keeps its Python frames plus the native function at the top). Green frames are the simulator's code, amber are SimPy's, blue are the standard library (Python's or Rust's), red are native C code and purple the rest. Frames under 0.4% of the samples are omitted.

Reading a Profile: Self and Total Time

Source: snippets/RESULTS.md in _simeng_build

• Total time locates the phase: 77% of the run is inside SimPy's event loop (simulate), with half of all samples in decode_once. The cost model's step_time and decode_sum are 10% of self time between them: the roofline arithmetic is not free.
• Self time locates the code: in the first profile, percentile was the second-largest self-time function at 12.8%. It is in the metrics, after the simulation has finished, and nobody optimising the event loop would look there. Slide 09 fixed it; in this profile, taken after the fix, _dist (which now sorts once) is 5.8%.
• perf sees the interpreter, py-spy sees the program. Under perf, 36.5% of the Python run's samples land in _PyEval_EvalFrameDefault, CPython's bytecode loop. With -X perf the Python functions appear as callers, so perf's total time agrees with py-spy's (simulate 80%), but self time belongs to the interpreter. Use py-spy to find the Python function and perf to see what the interpreter does inside it.

Source: snippets/RESULTS.md in _simeng_build

Memory_System_Sim is a different shape: 40% of its time is in the controller's main loop itself, and 19% in t_act, which computes the earliest legal activate time against every timing constraint. That is the model's intrinsic work, and a candidate for the Rust core its README proposes (deck 02).

Counting Instructions With Cachegrind

Cachegrind runs a program on a simulated CPU and counts every instruction, with a simple model of the caches. It is 20–100× slower than native, but deterministic: the same run gives the same count, so small changes can be compared without timing noise. Here are the same 500-request simulation in Rust and in Python:

Source: snippets/RESULTS.md in _simeng_build

• Start-up, measured separately: Python with the simulator's imports 286,251,713 instructions; a one-request Rust run 856,841. Net of start-up: Python 2,045,124 and Rust 109,743 instructions per request, 19x (gross: 23x).
• Neither run is cache-bound: the working set fits, and both miss the last-level cache on 0.1% or fewer of references. The difference is purely instructions, interpreter work against compiled code.

The Rust run's top functions by instructions executed:

Source: snippets/RESULTS.md in _simeng_build

The simulation engine (Simulation::run) is 12.5%. Sorting floats is 49.3%, and fmean another 17.5%: summarise, computing the latency percentiles. Three independent measurements agree. perf samples a 20,000-request run and puts 63.5% of the time under summarise, with the sort at 37.6% self (slide 04's profile). And Rust_DES_Kernel times summarise at 8.5 ms against 4.0 ms for simulate (deck 02).

Closing the Loop: Sort Once

Hypothesis: _dist called percentile for p50, p90 and p99, and each call sorted the whole list again. For inter-token latencies that list has 763,474 values. Change: sort once and take all three percentiles from the sorted list. Check: summarise must return exactly the same answers. The change is now in Disaggregated_Inference_Sim (2026-10-03), with a property test that the two versions agree, and profile_first.py keeps the original for this comparison:

def _dist(xs: list[float]) -> dict:
    # Sort once for all three percentiles (profiling showed the repeated sorts dominating
    # summarise). The mean stays fmean of the data, so every value is unchanged.
    s = sorted(xs)
    return {"mean": fmean(xs) if xs else math.nan, "p50": _percentile_sorted(s, 50),
            "p90": _percentile_sorted(s, 90), "p99": _percentile_sorted(s, 99),
            "max": s[-1] if s else math.nan}

Source: snippets/RESULTS.md in _simeng_build

• Speed-up of summarise: 2.25x; outputs identical: True. End to end (simulate + summarise): 1.24 s to 1.06 s (1.17x).
• Amdahl's law: summarise got 2.25× faster, the whole run only 1.17×, because the simulation itself was untouched (InfSim 08, slide 14).
• A correction to this slide. It used to predict that the same change would be worth more in the Rust port, where sorting is half the instructions. Applying it showed the prediction was wrong: the port already sorted each distribution once. Its sorting is a single, necessary O(n log n) sort. The nearest change, an unstable sort, gives identical output and saved only 7% of the sorting instructions (below). A real gain would need a different algorithm: selection finds three percentiles in O(n). Read the code before acting on a profile.

Source: snippets/RESULTS.md in _simeng_build

Benchmarks Are Distributions

The same command, run 40 times on an unchanged machine, takes 40 different times:

Source: snippets/RESULTS.md in _simeng_build

• The distribution is not a bell. In this run it is skewed, with most runs at 48–56 ms, a tail to 60 ms and one run at 67 ms. In the first run of this deck it had two clusters, around 53 and 61 ms. Frequency scaling, which core and hardware thread the process lands on, and the desktop all move it. A mean and a standard deviation describe this badly; the median and the MAD (median absolute deviation) describe it well.
• Report an interval: a bootstrap confidence interval for the median (resample the runs with replacement, take the median, repeat 5,000 times) needs no assumption about the distribution's shape.
• Warm-up depends on the runtime. memsim's first in-process run was no slower than the rest (0.99×): CPython 3.12 has no JIT to warm up. JIT-compiled runtimes, and anything with cold caches or lazy loading, need warm-up runs that are discarded (as criterion does by default; warm-up and replications).

Interactive: Designing a Regression Gate

A speed gate compares the median of k runs of the new build with the median of k runs of the baseline, and fails if the new one is slower by more than a margin. Using the 40 measured runs above, each setting was tried 20,000 times on unchanged code (how often it raises a false alarm) and on code made 5% and 10% slower (how often it catches the regression):

Regression Detection in CI

• Calibrate with an A/A test. Run the gate on unchanged code until you know its false-alarm rate. Here a single run against a single run with a 5% margin fails unchanged code 25% of the time.
• Choose k and the margin together. The margin must sit well below the regression you want to catch: with a 10% margin, a 10% slow-down is caught only about half the time, whatever k is. Ten runs per side with a 5% margin gave 5% false alarms and caught 95% of 10% slow-downs.
• Separate exact checks from timed ones. Behaviour (outputs, cycle counts, FLOPs) is gated exactly; time is gated statistically (deck 07). Instruction counts from cachegrind are a third option: deterministic and noise-free, though not the same thing as time.
• Control the machine when you can: a dedicated runner, a fixed CPU frequency, no other jobs. One heavy job at a time also keeps a small machine from running out of memory, as it did more than once during this series.
• Keep the raw samples as a build artefact, so a failure can be re-examined rather than re-run. criterion does this for Rust benchmarks and reports a confidence interval for each change (criterion user guide).

What to Take Away

• USE for resources, profile first for code, and always check the answers after a change.
• Read self and total time. Here the hot spot was in the metrics, not the engine: about half the Rust port's instructions are spent sorting.
• Read the code before acting on a profile: the same fix was a 2.25× win in Python and nearly nothing in Rust, which already sorted once.
• Use perf where it is allowed (counters, any language, frame pointers for usable stacks); py-spy and cachegrind work unprivileged, and cachegrind's instruction counts are deterministic.
• Timings are distributions: report medians and intervals, and keep the raw samples.
• Design gates from measured noise: an A/A test sets the false-alarm rate, and k and the margin set what is caught.

• snippets/t11The measurement scripts, the raw profiles and flame graphs, and the sort-once experiment.
• py-spy and cachegrindThe two profilers used here.
• The USE method and flame graphsBrendan Gregg's descriptions of both.
• LLM Inference Simulators 08Techniques for making simulators faster, measured.
• Interview_Computer_ArchitectureCaches, pipelines and performance counters.