Modelling Memory Systems: DRAM and HBM

Topics We'll Cover

• Why "Bandwidth × Efficiency" Fails
• How DRAM Is Organised
• Commands and Timing Parameters
• Row Hits, Misses and Conflicts
• Scheduling and Page Policy
• Address Mapping
• Refresh and the Four-Activate Window
• Interactive: Patterns, Policies and Load
• Load and Latency
• How We Know the Model Is Right
• Cross-Checked Against DRAMsim3
• Plugging It Into the FHE Simulator
• What to Take Away

Why "Bandwidth × Efficiency" Fails

Architecture simulators usually model memory as a pipe: bytes divided by peak bandwidth, sometimes times a derating factor of "about 70%". The FHE simulator's HBM works this way. The factor is not a property of the memory. It depends on the access pattern, the controller, the address mapping and the load. On a single HBM pseudo-channel with a single controller:

This deck builds a command-level model that derives the factor, checks it four ways, and plugs it into the FHE simulator. The code is Memory_System_Sim; every number comes from its examples/results.md.

How DRAM Is Organised

Parallelism comes from banks: while one bank opens a row, others can transfer data. A DDR4 bank group shares some internal circuitry, so back-to-back accesses within a group must be further apart than accesses across groups. HBM2-class stacks have 8 channels, each split into two pseudo-channels with their own banks and a 64-bit data bus.

Commands and Timing Parameters

The controller issues ACT (open a row), RD/WR (one burst), PRE (close) and REF (refresh), at most one per cycle on the command bus. The JEDEC standards (JESD79-4 for DDR4, JESD235 for HBM) name the minimum gaps between commands. The values below are illustrative: in the range of a DDR4-3200 speed bin and an HBM2E-class pseudo-channel, not quoted from the standards. The DDR4 values coincide with DRAMsim3's DDR4_8Gb_x8_3200 configuration, which slide 11 uses.

Source: examples/results.md in Memory_System_Sim

The controller computes, for each candidate command, the earliest cycle at which every constraint it is subject to holds. Per bank group, not just per rank: that distinction was a bug (slide 10).

def t_act(self, ra: int, bg: int, b: _Bank) -> int:
    t, rk = self.t, self.ranks[ra]
    e = max(b.act_ok, rk.ref_until, rk.last_act + t.RRD_S, rk.act_bg[bg] + t.RRD_L)
    if len(rk.acts) == 4:
        e = max(e, rk.acts[0] + t.FAW)
    return e

Row Hits, Misses and Conflicts

Unloaded latencies, simulated against those closed forms, plus the bandwidth limits of the next three slides:

Source: examples/results.md in Memory_System_Sim

Every latency matches exactly. A conflict costs almost three times a hit on this DDR4, so a controller's main job is to turn conflicts into hits by choosing the order of requests.

Scheduling and Page Policy

• FCFS: serve requests strictly in arrival order. Simple and fair, but when the oldest request needs a row opened, every other bank waits too.
• FR-FCFS (first-ready, first-come-first-served; Rixner et al., ISCA 2000): issue ready column commands (row hits) first, then the oldest ready command. It keeps a row open while queued requests still hit it, and overlaps opening rows in some banks with transfers in others.
• Open page leaves the row open after an access, betting on a hit. Closed page precharges at once (auto-precharge) unless a queued request hits the row, betting on a miss instead of a conflict.

Source: examples/results.md in Memory_System_Sim

FR-FCFS wins everywhere, by up to 12× on interleaved streams, whose rows FCFS keeps closing. Writes cost a stream 13% even under FR-FCFS, because the bus has to turn around and a read must wait tWTR after a write. Page policy matters less than the scheduler here: with a full queue, the controller already knows whether another hit is coming.

Address Mapping

The mapping decides which address bits select the channel, bank group, bank, row and column, and so which patterns spread across banks:

SCHEMES = {
    "RoRaBaCoBgCh": ("ro", "ra", "ba", "co", "bg", "ch"),    # default: a stream rotates over bank groups
    "RoRaBgBaCoCh": ("ro", "ra", "bg", "ba", "co", "ch"),    # a stream stays in one bank group (tCCD_L)
    "RoCoRaBgBaCh": ("ro", "co", "ra", "bg", "ba", "ch"),    # consecutive bursts rotate over banks
    "ChRaBgBaRoCo": ("ch", "ra", "bg", "ba", "ro", "co"),    # one bank per huge region (worst parallelism)
}

Source: examples/results.md in Memory_System_Sim

• Bank groups below the column bits. If a row's bursts stay in one bank group, back-to-back reads are tCCD_L apart instead of tCCD_S: streaming falls from 0.998 to 0.526 of peak. Real DDR4 controllers interleave bank groups at fine granularity for this reason.
• XOR bank hashing (Zhang, Zhu and Zhang, MICRO 2000) mixes low row bits into the bank index, so a stride of exactly one row stops hammering one bank: 0.054 becomes 0.456.

Refresh and the Four-Activate Window

Two more bounds follow the same pattern. A stride that hits the same bank's next row each time runs at one access per tRC: simulated 0.054 and 0.026, against (BL/2) / tRC = 0.054 and 0.026. Without refresh, streaming reaches 0.999 of peak. These closed forms are tests in the repository (tests/test_controller.py), not just slide material.

Interactive: Patterns, Policies and Load

Recorded results, not a live model. Pick a pattern and a policy to see the bandwidth it sustains. The chart shows mean and p99 latency against offered load for streaming and random traffic on one HBM pseudo-channel (FR-FCFS, open page). The dashed line marks the load you pick. Little's law (LLM Inference Simulators glossary) turns the latency into the number of requests in flight.

Load and Latency

Source: examples/results.md in Memory_System_Sim

• Streaming stays near its unloaded latency to about half of peak, then queueing takes over: 37 ns at 0.1 rises to 247 ns at 0.9, the familiar hockey stick (queueing checks).
• Random traffic saturates near 0.27 of peak (slide 05), so at 0.3 its queue grows without bound and the mean is already 1.25 µs.
• The tail is refresh. Streaming's p99 is 365–557 ns at every load. A refresh is in progress about 9% of the time (tRFC / tREFI), so the slowest few per cent of requests wait for one: 350 ns.

A memory model that ignores load would give the 37 ns number to a simulator running at 90% utilisation, and miss the tail completely.

How We Know the Model Is Right

What it found. Hypothesis produced a trace with a write to bank group 0, then a write to group 1, then a read from group 0. The controller remembered only the last write, so it applied tWTR_S from the group-1 write. It should have applied tWTR_L from the group-0 write. The checker flagged "WR-RD same bank group (WTR_L): 28 < 32". The fix keeps per-group history; the shrunk example is now a regression test:

def test_regression_wtr_l_after_a_later_write_to_another_group():
    """Found by Hypothesis: WR (group 0), WR (group 1), RD (group 0). The read must wait WTR_L after
    the group-0 write, not just WTR_S after the later group-1 write."""
    t = PRESETS["ddr4"]
    m = AddressMapping(t)
    g0, g1 = 0, 1 << m.shift["bg"]
    reqs = [Request(g0, write=True), Request(g1, write=True), Request(g0 + (1 << m.shift["co"]))]
    checked(t, reqs, mapping=m, refresh=False)
    wr0, rd = reqs[0], reqs[2]
    assert rd.issue >= wr0.done + t.WTR_L

Who checks the checker? Its own tests plant violations in legal logs, and run a controller subclass with tFAW deliberately removed. The checker must flag every one.

Cross-Checked Against DRAMsim3

DRAMsim3 (Li et al., IEEE CAL 2020) is a widely used cycle-accurate DRAM simulator. Both simulators ran the same traces, with the same timing (its DDR4_8Gb_x8_3200 config, one rank) and the same address mapping:

DRAMsim3 commit 2981759; 60,000 cycles per saturation run; throughput as a fraction of the data-bus peak.

Source: examples/results.md in Memory_System_Sim

• Close where the physics dominates: streaming with bank groups interleaved (+0.3%), random reads (−2 to −3%), the same-bank stride (exact) and unloaded latency (−1.2%).
• Apart where the controllers differ: interleaved streams differ by 16–54%. DRAMsim3 queues per bank (eight commands per bank, 32 transactions); memsim has one shared 32-entry queue. Which streams' rows stay open depends on the queue.

A cross-check that agrees everywhere usually means the two models share assumptions. This one shows which results depend on the controller, and therefore need its design specified before they are quoted.

Plugging It Into the FHE Simulator

FHE_Accelerator_Sim moves HBM traffic as 4 MiB chunks through one first-come resource. It now accepts an optional memory model, Accelerator(memory=...), that times each chunk given its size, its direction and the direction of the chunk before it. Memory_System_Sim's adapter simulates one pseudo-channel's share of the chunk at command level, memoised:

def chunk_time(self, nbytes: int, write: bool, prev_write: bool | None, peak_gbps: float) -> float:
    """Seconds to move ``nbytes`` through HBM whose peak is ``peak_gbps`` (FHE_Accelerator_Sim protocol)."""
    t = self.timing
    channels = peak_gbps / t.channel_peak_gbps()
    bursts = max(1, round(nbytes / t.access_bytes / channels))
    return self.channel_cycles(bursts, write, prev_write) * t.tck_ns * 1e-9

Source: examples/results.md in FHE_Accelerator_Sim

Source: examples/results.md in FHE_Accelerator_Sim

Reading it. FHE traffic is long sequential streams, so a good controller reaches about 0.90 of peak, mostly lost to refresh. A well-chosen flat 0.9 then agrees to 0.2%. A folklore 0.7 adds 38% to the bootstrap. An FCFS controller or a bank-group-unfriendly mapping adds 70–75%. Near the balance point the model changes the verdict: at 160 GB/s, peak bandwidth calls the optimised design MAC-bound and the HBM model calls it memory-bound. At 200 GB/s, flat 0.7 calls it memory-bound and the model agrees with peak that it is MAC-bound.

What to Take Away

• DRAM bandwidth is a function, not a number. One HBM pseudo-channel delivered 0.10–0.93 of peak across six patterns.
• Timing parameters are constraints between commands. Model them per bank, per bank group and per rank, and check them with an independent checker.
• FR-FCFS and a bank-group-aware mapping matter more than page policy for these patterns. XOR hashing rescues pathological strides.
• Validate four ways: closed forms, a protocol checker, property-based traces and another simulator. Each found something the others would not.
• Derive the derating factor, then reuse it. Plugged into a system simulator, the detailed model mostly confirms a well-chosen factor, rules out a folklore one, and decides the verdict near the balance point.

• Memory_System_SimThe simulator, checker, tests, DRAMsim3 cross-check and examples/results.md.
• FHE_Accelerator_SimResults section 20: the HBM model inside the FHE simulator.
• Ramulator (Kim, Yang and Mutlu, IEEE CAL 2016) and DRAMsim3The two most-used open DRAM simulators.
• FHE Accelerator Simulators 05: NTT-bound against memory-boundWhere the balance point comes from.
• Interview_SoC_ArchitectureSoC and memory-architecture interview questions.