Put the weights in on-chip SRAM and memory bandwidth stops being the limit. The price is capacity: a whole wafer per few billion parameters, or hundreds of chips in a chain. Cerebras's wafer, Groq's statically scheduled LPU and Graphcore's bulk-synchronous IPU.
This deck follows chapter 4, wafer-scale, chapter 5, compiler-scheduled chips and the IPU part of chapter 6. Specifications, sources and access dates: the specifications page.
A decode step reads every weight once. On a GPU that read comes from HBM; put the weights in on-chip SRAM and the same read is orders of magnitude faster.
The price is capacity: 230 MB of SRAM per GroqChip and 44 GB per wafer, against 80 GB of HBM on one H100.
Cerebras skips the step every other chip takes, cutting the wafer into dies. The WSE-3 keeps 84 dies joined, with its fabric running across the scribe lines.
| Cerebras WSE-3 (CS-3) | Cerebras WSE-3 Turbo (CS-4) | |
|---|---|---|
| Silicon (mm²) | 46,225 cs3-datasheet | 46,225 cs4-datasheet |
| Cores | 900,000 cs3-datasheet | 900,000 cs4-datasheet |
| On-chip SRAM | 44 GB cs3-datasheet | 44 GB cs4-datasheet |
| SRAM bandwidth | 21 PB/s cs3-datasheet | 43.2 PB/s cs4-datasheet |
| FP16 (sparse; dense not given) | 125 PFLOP/s cs3-datasheet | 250 PFLOP/s cs4-datasheet |
| System I/O | 1.2 Tb/s cs3-datasheet | 2.4 Tb/s cs4-datasheet |
| System power | 27 kW cs3-datasheet | not disclosed |
The CS-4’s WSE-3 Turbo is announced, with first shipments “this quarter” on the access date. Cerebras gives compute only as a sparse FP16 figure; a compute term from it is a floor.
Defects land at random, a density \(D_0\) per unit area. At an illustrative \(D_0\) = 0.1 per cm² (TSMC publishes none for N5):
A perfect wafer is essentially impossible. Cerebras makes defects small and survivable instead: each core is about 38,000 µm², so a defect costs one core of 900,000, and “redundant compute cores, redundant routing, and a fail-in-place architecture” route round it.
Llama 3.1 70B in 16 bits with its KV cache is 141 GB: 4 wafers, cut at layer boundaries. Cerebras’s inference service says exactly that: “70B models fit on as few as four systems”.
For training Cerebras streams the weights in from an external memory unit, one layer at a time, each applied to a whole batch held on the wafer. That suits training’s huge batches, not one user’s decode: streaming the 70B weights once over a CS-3’s 1.2 Tb/s of I/O takes 927 ms.
Groq’s first LPU, the GroqChip, removes the last things a chip decides for itself. Its compiler fixes every instruction, memory read and chip-to-chip transfer to a cycle before the program runs (“statically scheduled, deterministic execution”, ISCA 2022): no cache misses, no arbitration, the same latency every time.
| GroqChip (first-generation LPU) | NVIDIA Groq 3 LPU (LPX rack) | |
|---|---|---|
| SRAM | 230 MB groqchip-brief | 500 MB lpx-page |
| SRAM bandwidth | 80 TB/s groqchip-brief | 150 TB/s lpx-page |
| INT8 dense | 750 TOP/s groqchip-brief | not disclosed |
| FP8 | not disclosed | 1.23 PFLOP/s groq-home |
| Chip-to-chip (both ways) | 400 GB/s groq-isca22 | 2.5 TB/s lpx-page |
| Hop latency | 722 ns groq-isca22 | not disclosed |
| Power | 215 W groqchip-brief | not disclosed |
The GroqChip is retired (kept because published results used it); NVIDIA’s Groq 3 LPU is announced (“When released”), its hop latency not disclosed.
The model is cut into \(S\) stages of \(g\) chips; a token runs down the chain. Groq’s 2023 Llama 2 70B demonstration used 576 GroqChips in nodes of 8 (the fewest that hold it in INT8: 301).
More chips per stage divide the reading time but add an all-reduce hop per layer; more stages add hand-on hops. The latency is paid in hops, not bandwidth.
| Check | Kind | Published | Model | Ratio | Source |
|---|---|---|---|---|---|
| Llama 3.1 70B, 16-bit: CS-3 systems needed | capacity | 4 | 4 | 1 | cerebras-inference |
| Llama 2 70B, INT8: GroqChips used vs the fewest that hold it | capacity | 576 | 301 | 1.91 | nextplatform-groq |
| 576 GroqChips, Llama 2 70B: tokens/s per user vs the bound | latency | 300 | 3,644 | 0.0823 | nextplatform-groq |
| 4x CS-3, Llama 3.1 70B: tokens/s per user vs the bound | latency | 450 | 150,700 | 0.00299 | cerebras-inference |
| 1x CS-3, Llama 3.1 8B: tokens/s per user vs the bound | latency | 1,800 | 1,387,000 | 0.0013 | cerebras-inference |
The chip counts match. The speeds do not: the SRAM designs’ published speeds for one user are 0.13% to 8.2% of the bandwidth bound. Once the weights are in SRAM, bandwidth is not the limit; the published speeds imply 17.3 µs to 38.2 µs per layer beyond the bound, which the sources do not break down. Even so, these are the fastest one-user speeds in the site’s sources (450 tokens/s for 70B on four CS-3s).
Graphcore’s first-generation IPU (GC2) has 1,216 tiles, each a core with 262 kB of its own SRAM, 319 MB in all at 45 TB/s, and no other memory on the device. It runs the bulk-synchronous parallel model:
$$t_{\text{superstep}} = \max_{\text{tiles}} t_{\text{compute}} + t_{\text{sync}} + t_{\text{exchange}}$$Every tile works on its own memory only.
All tiles stop at a barrier.
Data moves between tiles on a schedule the compiler fixed.
Like an LPU chain, a model must fit in the tiles: Llama 3.1 70B in 16 bits needs 444 GC2s. Graphcore’s site lists no IPU products on the access date; the part is marked retired.
Previous: deck 01, the design space. Next: deck 03, dataflow, meshes, edge and in-memory.