Simulating an FHE Accelerator

Topics We'll Cover

• What the Simulator Must Answer
• The Modelled Architecture
• The Trace: Contract Between Scheme and Hardware
• Where Traces Come From
• The SimPy Engine
• Scratchpad and Traffic
• Cost and Power Model
• Metrics and Hot-Spots
• Interactive: Run the Simulator
• Experiments to Try
• Validation
• A HEIR Front End: Real Programs
• Reading the Code
• What to Take Away

What the Simulator Must Answer

These are architecture-exploration questions, so the model sits at the transaction / discrete-event rung of the fidelity ladder (LLM Inference Simulators deck 01): one event per kernel and per memory chunk, not per cycle. A full bootstrap simulates in tens of milliseconds, so a design sweep of hundreds of points takes seconds.

The Modelled Architecture

The defaults are an ARK-class digital design at 1 GHz, sized like the published ASICs (ARK: 512 MB scratchpad, 1 TB/s HBM). The units are aggregates: four NTT pipelines become one resource with their combined throughput. All coefficients are illustrative.

The Trace: Contract Between Scheme and Hardware

The scheme model emits HE operations in program order. Each names its inputs and output, the evaluation key and plaintexts it needs, and its primitive kernels as [kind, amount, on-chip words]. Two real lines from fhe-sim --params small --dump-trace t.json (two hoisted baby-step rotations):

{"id": 2, "op": "hrot", "stage": "cts", "level": 11, "boot": 0, "inputs": ["b0.c1", "b0.c2"],
 "output": "b0.c3", "key": ["cts0.b1", 3145728], "pts": [],
 "kernels": [["auto", 245760, 491520], ["mac", 393216, 720896], ["intt", 8, 65536],
             ["bconv", 524288, 262144], ["ntt", 24, 196608]]}
{"id": 3, "op": "hrot", "stage": "cts", "level": 11, "boot": 0, "inputs": ["b0.c1", "b0.c2"],
 "output": "b0.c4", "key": ["cts0.b2", 3145728], ...}

• b0.c2 is the shared ModUp output (hoisting), so both rotations depend on it. Their keys differ, which is the whole data-movement story of deck 02.
• The kernel list is the ModDown half of a key switch (inner product, iNTT of the special limbs, base conversion, NTT), preceded by the automorphism.
• The hardware never sees "CKKS": only kernels, objects, sizes and dependencies. Any producer that writes this format can drive the engine.

Where Traces Come From

The SimPy Engine

Every HE operation is a SimPy process; every functional unit and HBM is a simpy.Resource. The core of sim.py, lightly trimmed:

def run_op(self, o, pl: OpPlan):
    env = self.env
    pf = env.process(self.prefetch(o, pl)) if (pl.key_load or pl.pt_load) else None  # keys don't wait for data
    for name, size in pl.writebacks:                      # spills decided at issue
        ev = env.process(self.writeback(name, size))
        self.wb_ev[name] = ev
    for x in o.inputs:                                    # dependencies
        p = self.producer.get(x)
        if p is not None:
            yield from self.wait_done(p)
    for x, size in pl.ct_load:                            # reload spilled ciphertexts
        w = self.wb_ev.get(x)
        if w is not None and not w.processed:
            yield w
        yield from self.xfer(size, "ct_read", o.stage)
    if pf is not None and not pf.processed:
        yield pf
    for k in o.kernels:
        for seg in self.cost.segments(k):                   # cost model: unit, time, joules
            req = self.units[seg.unit].request()
            yield req
            ...                                              # power and statistics bookkeeping
            yield env.timeout(seg.time)
            self.units[seg.unit].release(req)

• An issuer walks the trace in program order with a window of W in-flight operations (default 4): in-order issue, out-of-order completion within the window.
• HBM transfers are split into 4 MiB chunks, each a request on the shared HBM resource, so concurrent loads share bandwidth instead of serialising whole keys (deck 02 of the sister series: "hold the link or share it?").

Scratchpad and Traffic

• Functional, at issue. When an operation issues, the scratchpad decides in program order which of its inputs, key and plaintexts hit, and what must be evicted. Byte counts therefore do not depend on timing, which makes them deterministic and testable. This mirrors a compiler-managed scratchpad (F1's "explicitly managed memory hierarchy").
• LRU over named objects, with variable sizes. Objects in use by the issuing operation are pinned. A key is a hit only if the cached copy has at least the limbs this level needs.
• Liveness. A ciphertext is freed at its last use with no write-back. Evicting a live dirty ciphertext costs an HBM write now and a read later. A result with no consumer goes to HBM.
• Reserved working set. Kernel temporaries (ModUp digits, accumulators) live in a region sized for one top-level key switch: 90 MiB for the ark set. A scratchpad smaller than that is rejected rather than silently modelled.
• Traffic classes: evaluation keys, plaintexts, ciphertext reads and ciphertext writes, each counted separately.

Cost and Power Model

The manager guarantees the peak never exceeds the TDP, and a test checks it on every configuration, including Hypothesis-generated ones. It is greedy and first-come, with the clock chosen by bisection (no transcendentals, so the JavaScript port matches it bit for bit). The older alternative, one worst-case clock at which every unit and HBM flat out fit the TDP, is kept as power_mode="worst-case". It reserves power for a coincidence the workload never produces, so it can be 1.8× slower, or unable to run at all at a low TDP (deck 05). Methodology for all of this is in LLM Inference Simulators deck 07.

Metrics and Hot-Spots

Default run (ark set, ARK-class design): 13.94 ms, memory-bound (HBM 89% busy, NTT 13%, MAC 29%), CoeffToSlot 72% of the time with HBM as its hot-spot, EvalMod's hot-spot the MAC lanes. Keys are 54% of 12.44 GB of HBM traffic; 1,139 mJ per bootstrap at 82 W average and 176 W peak against a 250 W TDP.

Interactive: Run the Simulator

The JavaScript port of the SimPy model, with a minimal SimPy core that reproduces SimPy's event ordering. It matches the Python package bit for bit, including clocks, bytes, energy and per-operation end times; the repo's tests check this. Choose a configuration and press Run, SRAM sweep or Compare algorithms.

Experiments to Try

1. Find the bound. Run the defaults: memory-bound, with CoeffToSlot's hot-spot on HBM. The timeline shows HBM saturated while NTT and MAC idle in the orange stretches, then compute busy in EvalMod.
2. Buy SRAM. Press SRAM sweep. Key traffic does not move from 512 MiB upwards (each key is used once); only ciphertext spills disappear.
3. Change the algorithm instead. Tick Min-KS and seeded keys, or press Compare algorithms: key traffic falls about tenfold and the verdict moves to the MAC lanes.
4. Starve the NTTs. Choose small digital and compare algorithms again: on an NTT-bound design the same techniques make bootstrapping slower.
5. Hit the power wall. Raise NTT to 16,384 and MAC to 32,768, tick the three key techniques and lower the TDP to 150 W. The dynamic manager still runs at a mean clock near 100%; switch Power control to worst-case clock and the same design slows to about half speed. At 100 W the worst-case clock cannot run at all, and the manager becomes power-bound.
6. Back-to-back bootstraps. Set 2 bootstraps with a large SRAM: keys are reused across bootstraps only when the whole key set fits (about 16 GiB, beyond the slider).

Validation

89 tests, organised as a verification ladder (the method is in LLM Inference Simulators deck 06):

One parameter was fitted (0.18 modular operations per cycle per unit). The scheme model under-predicts the bootstrap mainly because OpenFHE rescales copies of each input before every multiplication, doubling EvalMod's transforms. Replaying the recorded stream removes that gap; most of the remaining −11% is element-wise work the tracer does not log. A full-slot N = 2¹⁶ OpenFHE bootstrap could not be measured or recorded: its keys and precomputed plaintexts exceeded the memory available on the 15 GB test machine.

A HEIR Front End: Real Programs

Three of HEIR's own example programs, compiled with the HEIR v2026.10.01 release binaries and simulated on the ARK-class design (fhe-sim --heir). HEIR picked every parameter, rotation, rescale and bootstrap; the front end expands each op into kernels. LoLa is an MNIST CNN with square activations (55 rotations); the MLP has a polynomial ReLU; the third row is LoLa compiled with a level budget of 2, so HEIR must place a bootstrap. All four runs are memory-bound.

• Inference is weight-bound too. The MLP moves more plaintext (its weights) than key material: compiler-chosen packing makes plaintext traffic a first-class cost.
• The compiler's parameters can outgrow the chip. To fit a bootstrap under a tiny level budget HEIR moved to N = 2¹⁷ with 47 primes. One key switch then needs 354 MiB of scratch, ciphertexts thrash the 512 MiB scratchpad, and the program takes 203 ms. That is a co-design conversation the simulator makes concrete.
• Cross-checked against OpenFHE. The same LoLa, through HEIR's OpenFHE code generation and the instrumented library, performs exactly the 55 rotations, 41 rotation keys and 2 relinearisations the front end reads from the IR. The library run costs 3× more on the ARK-class model (1.84 against 0.61 ms): its generated context carries 12 limbs where HEIR's level analysis needs 6, and its automatic rescaling adds 641 polynomial rescales to HEIR's 19 rescale ops.

Reading the Code

FHE_Accelerator_Sim follows the structure of the sister repo, Disaggregated_Inference_Sim: scheme model, cost model, engine and probes in separate files.

pip install -e .[dev] && pytest                        # 89 tests
fhe-sim                                                # the default run on slide 08
fhe-sim --counts                                       # deck 02's per-stage table
fhe-sim --sweep-sram 128 256 512 1024 2048             # traffic against SRAM
fhe-sim --min-ks --seeded-keys --otf-pt --trace b.json # open in ui.perfetto.dev
fhe-sim --openfhe-log calibration/openfhe_trace/sparse16.log.gz --hw cpu   # a real OpenFHE bootstrap
fhe-sim --heir calibration/heir/lola.ckks.mlir.gz         # LoLa, compiled by HEIR
python examples/results.py                             # every number in this series

What to Take Away

• A kernel-level trace is the contract between the scheme (or compiler) and the hardware model; keep them separate. HEIR's IR plugs straight in: real programs, with the compiler's own levels and bootstraps.
• Model keys as named objects in a capacity-limited scratchpad, or the SRAM question cannot be asked.
• Units and HBM as SimPy resources, dependencies as events and an issue window give overlap without cycle-level detail: a bootstrap in about 70 ms of Python or 12 ms of JavaScript.
• The default ARK-class design is memory-bound (13.94 ms, HBM 89% busy), with CoeffToSlot as the hot-spot.
• Validation is layered: hand formulas, invariants, flow-shop algebra, behaviour, properties, a functional model, recorded OpenFHE kernel streams, OpenFHE timings and a bit-exact twin.