The Memory Wall Compute throughput has grown at ~3× per generation; DRAM bandwidth grows at ~1.4×.
The gap compounds — modern ML accelerators are almost always memory-bandwidth-bound, not compute-bound.
Transformer attention: bandwidth scales O(n²) with sequence length
LLM auto-regressive decode: each token must reload all model weights
At advanced nodes (N5, N3) device mismatch increases dramatically. Bitcell failure rate rises exponentially as supply voltage falls. Assist techniques restore margin:
Write Assist
Negative bitline (NBL) pull-down
Wordline boost (VWL > VDD)
VDD collapse during write
Read Assist
Virtual VSS (raised source)
Replica bitline sensing
Self-timed sense amplifiers
SRAM Compiler & Characterisation
Memory compilers (ARM, Cadence, TSMC internal) generate timing/power models (Liberty) and layout (GDS).
Monte Carlo SPICE verifies yield targets across PVT corners.
ML SoCs integrate multi-port SRAM cuts (1RW, 2RW, 1R1W) tiled to form large banks.
Assist circuits lower effective Vmin
50–100mV
Vmin reduction from write-assist techniques
06
Cache Design
Set-associativity · Replacement · Write policy · Scratchpad vs cache
07
Cache Fundamentals for ML SoCs
Organisation
Direct-mapped: simple, fast, high conflict miss rate
N-way set-associative: N candidates per set; LRU/PLRU/random replacement
Fully associative: maximum flexibility, CAM cost limits size
ML: 4–16 way typical for shared L2/L3
Write Policy
Write-back + write-allocate: dominant; minimises DRAM traffic
Write-through: simple coherency; used for I/O coherent agents
DDR5: 8 banks × 4 bank groups per die; 32 banks per rank
Key Timing Parameters
Param
Meaning
DDR5 ~
tRCD
RAS→CAS delay (row activate)
14 ns
CL
CAS latency (read cmd→data)
14 ns
tRP
Row precharge time
14 ns
tRAS
Row active minimum time
33 ns
tFAW
4-activate window
15 ns
tREFI
Refresh interval per row
3.9 µs
ML Scheduler Implications
Memory controllers for ML SoCs use bank-aware scheduling and
row-buffer locality policies to maximise page-hit rate on
weight matrix streaming patterns. DRAM ECC (on-die ECC + SECDED at interface)
adds latency but is mandatory at datacenter.
Deep power-down, partial array self-refresh → edge efficiency
Package-on-package (PoP) or co-packaged with SoC die
Standard
Peak Rate
BW / ch
Voltage
Target
DDR5-6400
6400 MT/s
51 GB/s
1.1 V
Server
DDR5-8800
8800 MT/s
70 GB/s
1.1 V
Server
LPDDR5X
8533 MT/s
68 GB/s
1.05 V
Edge
LPDDR6
14.4 Gbps
~115 GB/s
0.8–1.05 V
Edge, AI
Why not DDR for large ML SoCs?
DDR bandwidth per pin is lower than HBM/GDDR because it operates at lower data rates and uses narrower interfaces (64/72 bits per channel vs HBM's 1024-bit stack interface). PCB routing limits channel count.
16×
More bandwidth pins HBM vs DDR5
12
GDDR7 — Graphics & Edge Inference
GDDR7 Architecture
Up to 32–36 Gbps per pin (JEDEC JESD239); PAM3 signalling replaces NRZ
32-bit bus per die; multiple dies per package (256-bit typical for GPUs)
Linux NUMA: CXL memory appears as NUMA node; numactl placement
Tiered memory: hot pages on DRAM, cold pages on CXL (TPP kernel feature)
PyTorch / CUDA Unified Memory: transparent CXL integration in progress
8 TB
CXL pool per fabric switch shared across multiple hosts
18
Roofline Model — Visualising Memory Bottlenecks
Roofline Fundamentals
Arithmetic Intensity (AI): FLOPs per byte of DRAM traffic
Memory-bound: Performance = AI × Peak_BW (left of ridge)
Compute-bound: Performance = Peak_FLOPs (right of ridge)
Ridge point: AI = Peak_FLOPs / Peak_BW
Typical AI Workloads
Operation
Arithmetic Intensity
Bound
GEMM (large batch)
100–1000 FLOP/B
Compute
GEMV (batch=1, LLM decode)
0.5–2 FLOP/B
Memory
Attention (prefill, long ctx)
20–80 FLOP/B
Mixed
Elementwise (ReLU, layernorm)
<1 FLOP/B
Memory
FlashAttention (fused)
↑ 10–50×
Compute
Roofline: performance vs arithmetic intensity
Key insight: LLM decode (batch=1) operates at <2 FLOP/B — always memory-bound. Batching raises AI and shifts the operating point right toward the compute roof.
19
Memory Technology Bandwidth Landscape
Bandwidth at three scopes — on-chip (per-SM / per-tile), per-package (all DRAM stacks aggregated), and off-package (per link). Colour-coded by memory family: register file / SRAM / HBM / GDDR / LPDDR / DDR / CXL.
On-chip — per SM / per tile
Register File (per SM)
~10+ TB/s per SM
L1 SRAM / SPM (per tile)
~5 TB/s per tile
Per package — all DRAM aggregated
HBM4 (8 stk)
~16 TB/s (Rubin target, 2026)
HBM3E (8 stk)
~8 TB/s (B200 / B300)
HBM3E (6 stk)
~4.8 TB/s (H200)
HBM3 (5 stk)
~3.35 TB/s (H100 SXM)
GDDR7 (512-b board)
~1.79 TB/s (RTX 5090)
LPDDR6 unified (1024-b @ 14.4 Gbps)
~1.8 TB/s (projected M-Ultra class)
LPDDR5X unified (1024-b @ 6.4 Gbps)
~800 GB/s (Apple M3 Ultra)
DDR5 (8-channel socket)
~700 GB/s (Gaudi 3 host)
LPDDR6 mobile (4ch × 128-b)
~230 GB/s
LPDDR5X mobile (4ch × 128-b)
~136 GB/s
Off-package — per link
CXL 3.x (×16, per dir)
~128 GB/s per link
Reading the chart: scopes are not interchangeable. Per-tile SRAM figures apply inside one SM/tile; per-package DRAM figures sum across all stacks/channels on the package; CXL is quoted per x16 link, per direction. Compare within a scope, not across.
Takeaway: On-chip SRAM delivers 10–100× more bandwidth per tile than any off-chip DRAM per-package aggregate. The ML SoC architect's core challenge is staging data across this hierarchy — prefetching weights, double-buffering activations, and fusing operations to maximise locality.
100×
SRAM bandwidth advantage over HBM
20
Memory Systems in Production ML SoCs
NVIDIA H100 / H200
L1: 256 KB / SM; L2: 50 MB shared
HBM3 (H100 SXM): 5 active stacks (6 sites; H100 NVL enables all 6 for 96 GB), 3.35 TB/s, 80 GB
Bank interleaving: distribute weight rows across banks
Co-design principle:
Compiler (XLA, TVM, Triton) must be memory-hierarchy-aware — tile sizes, prefetch distances, and fusion decisions should be tuned to the specific SRAM capacity and HBM bandwidth of the target SoC.
22
Summary
On-Chip SRAM
6T/8T bitcells, assist circuits, HD vs LP flavours. ~70% die area in large ML SoCs. Vmin engineering enables power gating.
Cache & Scratchpad
Hardware cache vs software SPM tradeoffs. ML NPUs favour SPM + DMA for predictable latency and zero tag overhead.
HBM Family
1024-bit TSV interface, 3.35–4.8+ TB/s aggregate. HBM4 targets 2 TB/s/stack with hybrid bonding to SoC.
CXL Memory
Disaggregated pooling for TB-scale LLMs. ~150–250 ns latency; PCIe 6 to ~256 GB/s per link.
The Unifying Principle:
ML SoC memory architecture is a bandwidth optimisation problem across five decades of capacity. The roofline model quantifies where each operator sits; the solution space spans bitcell design, packaging, compiler fusion, and algorithmic compression.