C++ Performance Models with SystemC TLM-2.0

Topics We'll Cover

• Why a C++ Model in SystemC?
• Modern C++ for Models
• The SystemC Kernel: Processes, Events, Delta Cycles
• The Modelled Tile
• TLM-2.0: Payloads, Sockets and Coding Styles
• Approximately Timed: the Four-Phase Protocol
• Loosely Timed: Temporal Decoupling and the Quantum
• Ties Are Part of the Model
• Checked Against the SimPy Model
• Interactive: The Quantum Trade-Off
• GoogleTest and Sanitizers
• What Did Not Work, and the Limits
• What to Take Away

Why a C++ Model in SystemC?

A Python simulator is the fastest way to explore an architecture. Hardware teams standardise on C++ and SystemC (IEEE 1666-2023) when the model must live with the hardware: virtual platforms that boot firmware, models delivered to customers, and reuse of a vendor's IP models. The interfaces between them are standardised as TLM-2.0 (LLM Inference Simulators glossary; InfSim 02, slide 12).

This deck builds a TLM-2.0 model of the FHE accelerator tile from FHE Accelerator Simulators 03. It is driven by that simulator's own trace format and checked against it on the same traces. The code is SystemC_Accelerator_Model.

Modern C++ for Models

A performance model is mostly plain data and arithmetic. C++17 lets that part read like the Python it ports, with no SystemC in sight, so it can be unit-tested on its own:

struct HeOp {
    int id = 0;
    std::string op, stage;
    int level = 0;
    std::vector<std::string> inputs;
    std::string output;
    std::optional<std::pair<std::string, std::int64_t>> key;    // (key id, bytes)
    std::vector<std::pair<std::string, std::int64_t>> pts;      // (plaintext id, bytes)
    std::vector<Kernel> kernels;
};

Segment segment(const Hardware& hw, int log_n, const Kernel& k) {
    // hardware.py CostModel._seg and segments() (digital units only)
    Unit u = unit_of(k.kind);
    std::int64_t N = std::int64_t{1} << log_n;
    std::int64_t work_i = u == Unit::Ntt ? k.amount * (N / 2) * log_n : k.amount;
    double work = static_cast<double>(work_i);
    double s = hw.clock;
    double t_logic = work / (hw.rate(u) * s);
    double t_sram = static_cast<double>(k.words * 8) / (hw.sram_gbps * 1e9 * s);
    return {u, t_logic >= t_sram ? t_logic : t_sram, work};
}

• Value types and RAII (std::vector, std::unique_ptr, std::shared_ptr for a gate shared by two processes): no manual delete, so the sanitizers have little to find.
• Lambdas start processes: sc_spawn([this, pl]() { prefetch(pl); }).
• Same operation order as the reference. work / (rate * s), never "simplified", so every kernel's cost is bit-identical to the SimPy model's (deck 02, slide 06).

The SystemC Kernel: Processes, Events, Delta Cycles

• Modules (sc_module) hold processes: SC_THREADs that can wait(), or methods that cannot. Processes are also spawned at run time (sc_spawn), one per HE operation here.
• Time is sc_time, an integer count of a global resolution. This model sets 1 fs, so each delay is rounded to the nearest femtosecond: the only source of difference from the floating-point SimPy model.
• Order within an instant is unspecified. IEEE 1666 does not say which runnable process runs first. That is legal, and it matters (slide 08).

The Modelled Tile

The C++ ports the parts of FHE_Accelerator_Sim that decide what happens: the cost model and the LRU scratchpad planner. SystemC schedules when. The issuer plans each operation in program order and spawns a process that waits for its inputs, loads what missed, runs its kernels on the units and writes its output:

void Tile::issuer() {
    for (const auto& o : t_.ops) {
        while (inflight_ >= hw_.window) wait(slot_ev_);
        OpPlan pl = planner_.plan(o);
        ++inflight_;
        sc_spawn([this, &o, pl]() { run_op(o, pl); });
    }
}

TLM-2.0: Payloads, Sockets and Coding Styles

Transaction-level modelling replaces pin-level signals with function calls that carry a generic payload: a command, an address, a data pointer and length, a response status, plus optional extensions. The tile's initiator socket binds to the HBM model's target socket, and either side can be swapped for another vendor's model.

Here the payload is timing-only: the data pointer refers to a dummy buffer the target never reads, and an extension carries the requesting operation's id for arbitration:

struct Requester : tlm::tlm_extension<Requester> {
    int op = 0;
    tlm::tlm_extension_base* clone() const override { return new Requester(*this); }
    void copy_from(const tlm::tlm_extension_base& e) override { op = static_cast<const Requester&>(e).op; }
};

Approximately Timed: the Four-Phase Protocol

The base protocol has four phases: BEGIN_REQ and END_REQ on the forward path, BEGIN_RESP and END_RESP on the backward path. A target may shortcut a phase by returning an updated phase. This HBM accepts at once (END_REQ), waits for the channel, holds it for bytes / bandwidth, then answers with BEGIN_RESP. The initiator completes the transaction by returning TLM_COMPLETED, which implies END_RESP.

tlm::tlm_sync_enum Hbm::nb_transport_fw(tlm::tlm_generic_payload& trans, tlm::tlm_phase& phase, sc_time&) {
    if (phase == tlm::BEGIN_REQ) {
        // accept at once (END_REQ); a thread per request waits for the channel
        sc_spawn([this, &trans]() { serve(&trans); });
        phase = tlm::END_REQ;
        return tlm::TLM_UPDATED;
    }
    if (phase == tlm::END_RESP) return tlm::TLM_COMPLETED;
    SC_REPORT_ERROR("Hbm", "unexpected phase");
    return tlm::TLM_COMPLETED;
}

void Hbm::serve(tlm::tlm_generic_payload* trans) {
    Requester* who = nullptr;
    trans->get_extension(who);
    arb_->request(kHbmResource, who ? who->op : 0);
    wait(seconds(static_cast<double>(trans->get_data_length()) / bw_));
    arb_->release(kHbmResource);
    trans->set_response_status(tlm::TLM_OK_RESPONSE);
    ++transactions_;
    tlm::tlm_phase phase = tlm::BEGIN_RESP;
    sc_time delay = SC_ZERO_TIME;
    socket->nb_transport_bw(*trans, phase, delay);       // the initiator completes it (END_RESP implied)
}

Every process stays in step with simulated time, so requests meet the channel in the order they happen. AT reproduces the SimPy model op by op (slide 09). The price is a context switch per phase.

Loosely Timed: Temporal Decoupling and the Quantum

In LT, a process keeps a local time offset and runs ahead of the kernel's time, synchronising only when its offset passes the global quantum or when it must wait for another process. The target books the channel from the caller's local time and returns the finish time as the annotated delay:

void Hbm::b_transport(tlm::tlm_generic_payload& trans, sc_time& delay) {
    // LT: book the channel from the initiator's local time (now + delay), in call order
    double dt = static_cast<double>(trans.get_data_length()) / bw_;
    sc_time start = std::max(busy_until_, sc_time_stamp() + delay);
    busy_until_ = start + seconds(dt);
    delay = busy_until_ - sc_time_stamp();
    trans.set_response_status(tlm::TLM_OK_RESPONSE);
    ++transactions_;
}

} else {
    sc_time delay = qk.get_local_time();
    socket->b_transport(trans, delay);
    qk.set(delay);
    if (qk.need_sync()) qk.sync();

Fewer synchronisations make it faster. The cost: a process that runs ahead books the HBM channel or a unit before another process that would have asked earlier in simulated time. That is the accuracy LT trades away, and the quantum sets how much (slide 10).

Ties Are Part of the Model

Two operations finish waiting at the same instant and both want the NTT unit. Who goes first? A SimPy model answers by its event order. A SystemC kernel may answer either way: IEEE 1666 leaves the order of runnable processes unspecified. Left to the kernel, "first come" means whoever the implementation happens to run first. So the model makes the tie-break explicit:

void Arbiter::run() {
    for (;;) {
        wait(kick_);
        // let every process that can still act at this instant do so, then decide
        while (sc_pending_activity_at_current_time()) wait(SC_ZERO_TIME);
        for (auto& r : res_) {
            if (r.busy || r.pending.empty()) continue;
            auto first = r.pending.begin();
            r.busy = true;
            first->second->notify();
            r.pending.erase(first);
        }
    }
}

Source: examples/results.md in SystemC_Accelerator_Model

With the kernel's order, 6–24 operations per run end at different times from SimPy's, although the totals agree. With "oldest operation first", none do. This is the same lesson as the Rust event list's sequence numbers (deck 01, slide 03): a deterministic tie-break is part of the model, so write it down.

Checked Against the SimPy Model

tools/export_case.py writes three files from FHE_Accelerator_Sim: the trace (its own dump_trace JSON), the hardware, and the SimPy model's answers in worst-case clocking mode. The SystemC model reads the same trace and is compared with those answers:

Source: examples/results.md in SystemC_Accelerator_Model

• Seven of eight cases agree op by op, to 7 fs at worst (the time resolution is 1 fs), and on every byte and busy time.
• The 1 MiB-chunk case has four times as many HBM transactions and more exact ties. In 19 of them SimPy's order (its event-scheduling order) is not oldest-first, so those operations move by up to 26.7 µs. The horizon still agrees to 10⁻¹³.
• Tolerance, stated: horizon and every op end within 1 ps where ties resolve alike; exact bytes always. Anything else is a bug, and the GoogleTest suite fails.

Interactive: The Quantum Trade-Off

Four bootstraps (812 HE ops) on the ARK-class tile. Each point is a recorded run (median of five). Choose a quantum to see what LT buys and what it costs, against AT and SimPy.

GoogleTest and Sanitizers

Sixteen GoogleTest tests (deck 06, slide 10): the planner's bytes equal SimPy's exactly, unit busy times, AT agreement on four cases, LT invariants. One SystemC peculiarity shapes the suite: a process can elaborate and run a simulation only once, so each simulation runs in a forked child that sends its results back as JSON.

TEST_P(AtAgreement, MatchesSimPyToFemtoseconds) {
    const std::string c = GetParam();
    json r = simulate_in_child(c, Style::AT, 0), e = expected(c);
    double H = e["horizon_s"];
    EXPECT_NEAR(r["horizon"].get<double>(), H, 1e-12) << c;            // within 1 ps on a ms-scale run
    EXPECT_LT(max_abs_diff(r["op_end"], e["op_end"]), 1e-12) << c;
    EXPECT_EQ(r["bytes"], e["bytes"]);
    EXPECT_NEAR(r["busy"][0].get<double>(), e["busy"]["ntt"].get<double>(), 1e-12 * H);
    EXPECT_NEAR(r["hbm_busy"].get<double>(), e["busy"]["hbm"].get<double>(), 1e-12 * H);
}

• AddressSanitizer and UndefinedBehaviorSanitizer build (-DACCEL_SANITIZE=ON): GoogleTest, ASan + UBSan: 16 tests, 16 passed.
• CI builds SystemC 3.0.2 from source (cached), runs Release and sanitizer jobs, and has a job that exports fresh cases from the current FHE_Accelerator_Sim. If the Python model changes, the C++ model must change with it.
• sc_main replaces main (SystemC's library defines main), so the test binary calls RUN_ALL_TESTS() from sc_main.

What Did Not Work, and the Limits

• Emulating SimPy's event order. To close the last 19 differences, the arbiter was given SimPy's rule: order by priority and event-scheduling sequence, with process starts urgent. Stamping every wake-up that way made agreement worse, with hundreds of operations moved. That mode was removed rather than shipped half right. Reproducing another kernel's tie-breaks means reproducing its whole event queue.
• AT is slower than SimPy here: 330 ms against 204 ms for four bootstraps. A thread per HBM request and the arbiter's settling deltas cost more than they do on a workload of a few thousand coarse transactions. LT is about 4.5× faster than SimPy. C++ is not automatically fast; the modelling style decides.
• Scope. Digital designs, worst-case clocking. No dynamic power manager and no optical engine. Those exist only in the Python model.
• Payloads carry no data. This is a performance model. A virtual platform would move real bytes and use DMI (direct memory interface) to make that fast.

What to Take Away

• Keep the model's arithmetic out of SystemC. Plain C++17 that ports the reference in the same operation order is testable alone and bit-identical.
• AT for accuracy, LT for speed. The quantum is the dial: here up to 1 µs cost nothing, and 10 µs cost 0.8%.
• Ties are part of the model. IEEE 1666 leaves process order unspecified. An explicit arbiter made the result kernel-independent and matched SimPy op by op.
• Check against the reference on the same traces, with fresh exports in CI, and state the tolerance.
• GoogleTest with ASan and UBSan from the start; fork per simulation because SystemC elaborates once.

• SystemC_Accelerator_ModelThe model, the tests, the export tool and examples/results.md.
• Accellera SystemC and IEEE 1666The reference implementation and the standard, including TLM-2.0.
• FHE Accelerator Simulators 03The SimPy model this one reproduces.
• Deck 04: Modelling Memory SystemsWhat a better HBM target would model.
• Interview_CPPC++ interview questions.