Power & Energy in Inference Simulators

Topics We'll Cover

• Why Power Is a First-Class Simulator Output
• Where the Joules Go: Static, Dynamic, and Data Movement
• The Simulator's Power Model
• Prefill Runs Hot, Decode Runs Cool
• DVFS and Power Caps: The Third Roof
• Interactive: One Step Under a Power Cap
• System Results: Latency Against Energy
• Energy Proportionality: Static Power and Load
• From the Simulator to RTL Power and Thermal Analysis
• Power in Photonic and Novel Compute
• The Datacentre View
• Power Metrics Checklist
• What to Take Away

Why Power Is a First-Class Simulator Output

Speed is only half of performance. For AI hardware the other half is power and energy, and it is often the half that decides the product.

Where the Joules Go: Static, Dynamic, and Data Movement

P = P_static + P_dynamic
P_static  ≈ V · I_leak                 # leakage, clocks, always-on logic: paid whenever powered
P_dynamic ≈ α · C · V² · f               # switching: activity x capacitance x voltage^2 x frequency
E_op      ≈ α · C · V²                   # energy per operation: why lowering V (with f) pays quadratically

Horowitz's widely cited energy table (ISSCC 2014, 45 nm) makes the key point: moving data costs far more than computing on it.

Absolute values have fallen at newer nodes, but the ratios still hold. This is why the simulator charges separately for FLOPs and for HBM bytes, and why decode, which streams almost every weight on every step (all but the input-embedding table, which is a lookup), is an energy problem as well as a bandwidth problem.

The Simulator's Power Model

Four coefficients per device and one per link, attached to the same FLOP and byte counts the roofline already computes. Nothing new is counted; it is only priced.

E_total = Σinstances idle_W · T_run                    # static: charged busy or not
        + Σsteps (FLOPs · pJ/FLOP · s² + HBM bytes · pJ/byte)   # dynamic, with DVFS factor s
        + Σtransfers KV bits · pJ/bit                     # interconnect
P_step  = idle_W + E_step / t_step                       # tracked for the peak

Prefill Runs Hot, Decode Runs Cool

Per step, Llama-3-70B on 4×H100, from the simulator's cost model (as corrected on 2026-10-03; see slide 07):

• Compute-bound phases draw near-maximum power; every FLOP unit is switching.
• Memory-bound phases leave compute idle much of the time. The arithmetic units wait for HBM, so dynamic power is dominated by data movement.
• So decode has slack: the compute clock can drop until compute just keeps pace with memory, saving about 17% of decode's dynamic energy at no cost in time.
• Disaggregation makes this exploitable. On colocated machines the phases interleave, so the power setting must suit prefill. With separate pools, each pool gets its own power policy. Splitwise proposed exactly this: power-capped token machines.

DVFS and Power Caps: The Third Roof

Dynamic voltage and frequency scaling lowers the clock (and with it the voltage). In the model the compute clock runs at a fraction s of nominal; HBM is unaffected.

t(s) = max(tc / s, tm)                # compute slows; memory doesn't
E(s) = Ec · s² + Em                    # V tracks f, so energy per op ~ s^2
P(s) = idle + E(s) / t(s)             # increases with s: lower clocks always reduce power

dvfs on:     memory-bound step  →  s = max(s_min, tc/tm)        # same time, less energy
power cap:   choose the largest s with P(s) ≤ cap                   # closed form: sqrt, or a cubic (Cardano)
             still too hot at s_min → stretch the step to fit the budget

The simulator labels each step with whichever roof bound it and reports the fraction of busy time spent power-bound. Real DVFS has discrete operating points, voltage floors and transition latency; the continuous model is a first-order stand-in to refine when the hardware's operating points are known.

Interactive: One Step Under a Power Cap

This uses the simulator's own cost model (the browser engine from deck 05). Pick a step, then move the per-device cap and toggle DVFS. The chart sweeps the cap from TDP down to just above idle and plots time against energy: the energy–delay trade-off.

Things to notice: for prefill, tighter caps trade time for energy until static power (charged for the longer step) starts to dominate and total energy rises again. That minimum is the energy-optimal operating point: for a 2,048-token prefill on H100s it sits near a 160 W cap, at 186 J against 352 J uncapped, but at 2.2× the step time. For decode, the curve is flat in time over a wide range of caps: free savings.

System Results: Latency Against Energy

Full runs: Llama-3-70B, 4×H100 per instance, 4 req/s, 2,048/256-token mean prompts/outputs, SLOs TTFT ≤ 1 s and TPOT ≤ 25 ms, 800 requests. Every row is disagg-sim with the flags shown.

• Disaggregation saves about 10% energy per token here as well as fixing TPOT: no decode steps wasted waiting on prefills.
• Decode tolerates a cap down to about 250 W per device; at 200 W it falls off a cliff (the compute clock floor is reached and steps stretch past the TPOT SLO): only 12.6% of requests meet the SLOs. Finding that cliff is exactly what the simulator is for.
• Capping everything saves the most energy but costs TTFT; adding a second, capped prefill instance buys the latency back at 2.68 J/token. That is a provisioning trade only a simulator can price: more silicon at lower power.

Energy Proportionality: Static Power and Load

An idle accelerator still burns its static power. At low load that is spread over few tokens.

• Energy per token falls fivefold from light to heavy load: utilisation is an energy metric, not only a cost metric.
• Hence autoscaling and consolidation: fewer, busier instances plus power-gating idle ones. A simulator that models instance on/off transitions (with their latency) can evaluate such policies.
• Static-heavy hardware suffers most. The hypothetical optical part (cheap multiplies, 180 W of always-on lasers, tuning and converters) spends 65% of its energy on static power at 4 req/s. It wins on latency, but at 2.28 J/token it is no more efficient than the GPU under a 400 W cap with DVFS (2.26). Keeping such hardware fully busy, or gating its static consumers, is an architectural requirement.

From the Simulator to RTL Power and Thermal Analysis

Power modelling follows the same fidelity ladder as timing (deck 01), and the levels feed each other the same way.

• Simulator to RTL: the workload traces that drive performance verification also drive RTL power analysis, so activity factors reflect real inference, not toy vectors.
• RTL to simulator: per-operation energies extracted from RTL power runs replace the illustrative pJ/FLOP and pJ/byte. That is the power equivalent of cycle-count correlation, and its error belongs on the same correlation report.
• Thermal hot-spots: power is not uniform across a die. Power maps feed thermal models (HotSpot-style RC networks) that find thermal hot-spots, which in turn throttle clocks and close the loop back to timing. For photonics, temperature also shifts optical resonances, so thermal and functional behaviour are coupled.

Power in Photonic and Novel Compute

An optical multiply can cost very little energy. A photonic system pays for everything around it, and a simulator must account for all of it.

The Datacentre View

• Power oversubscription: provision more servers than the worst-case sum of TDPs allows, relying on statistical multiplexing and capping when needed. POLCA (Patel et al., 2023) studies this for LLM clusters, using the fact that inference rarely sustains peak power everywhere at once.
• Phase-aware power: Splitwise's power-capped token machines; per-pool caps in this simulator (--prefill-power-cap, --decode-power-cap).
• Energy-aware cluster control: DynamoLLM (2024) reconfigures instance counts, parallelism and GPU frequency with load to cut energy while meeting SLOs; the knobs are the ones the simulator exposes.
• Facility overheads: multiply IT energy by PUE (power usage effectiveness) for facility energy, and add embodied carbon if lifecycle impact is the question.
• Measure it: the ML.ENERGY leaderboard and its Zeus library measure real GPU energy per request and are a good source of baselines for validating a simulator's energy model on known hardware.

Power Metrics Checklist

Treat these like latency metrics: report them per configuration, with confidence intervals, under common random numbers, and track the energy-model correlation error alongside the timing one.

What to Take Away

• Power and energy are simulator outputs on a par with latency; for novel compute they are often the headline.
• Price the counts you already have: static power × time, pJ/FLOP, pJ/byte, pJ/bit. Calibrate the coefficients by regression on measurements or RTL power runs.
• Data movement dominates energy, as it dominates decode time.
• Prefill runs near TDP; decode has slack. DVFS and per-pool caps exploit it; the power cap is a third roof.
• Energy per token depends strongly on load through static power; static-heavy hardware must be kept busy.
• The fidelity ladder applies to power too: simulator traces drive RTL power analysis, and RTL energies calibrate the simulator.
• Every serving lever moves joules per token: step-by-step power under caps and DVFS, and J/token for each lever and workload, in Inference Trade-offs Explained, chapter 11 (and hardware and cost).