Every AI chip does the same sums. What separates the families is where the weights live, who decides when data moves, and how big the engines are. GPUs and systolic designs as the baseline, the decode memory wall, and every family run on the same decoder model.
This deck follows chapter 1, the design space, chapter 2, chapter 3 and chapter 10, choosing of AI Accelerators Explained, where every number here is animated. Specifications, sources and access dates: the specifications page.
Every AI chip does the same sums: multiply activations by matrices of weights, layer after layer. What separates the families is what happens around the arithmetic. Ask the same three questions of each:
Off-chip DRAM (HBM, GDDR, LPDDR), a tier of SRAM beside DRAM, on-chip SRAM only, or inside the memory array itself.
Every weight once per decode step, plus the KV cache. At batch 1 that is about two flops per weight read.
The hardware at run time (warps and caches), the compiler ahead of time (a static schedule), or a mix.
A fourth axis, how big the engines are, runs from a few large matrix units (a TPU) through a hundred or so cores (a GPU’s SMs) to 900,000 tiny cores on one wafer.
Each family’s position is a category, with a sourced basis on the site’s specifications page; a real chip can sit between two.
| Family | Weights live in | Who schedules | Grain | Site |
|---|---|---|---|---|
| GPUs NVIDIA H100, B200 | off-chip DRAM | hardware, at run time | tens to hundreds of cores | ch 2 |
| Systolic designs Google TPU, AWS Trainium | off-chip DRAM | compiler, ahead of time | a few large engines | ch 3 |
| Wafer-scale Cerebras WSE-3 | on-chip SRAM | mixed | hundreds of thousands of cores | ch 4 |
| Compiler-scheduled Groq LPU | on-chip SRAM | compiler, ahead of time | a few large engines | ch 5 |
| Reconfigurable dataflow SambaNova RDU, Graphcore IPU | tiered SRAM + DRAM | compiler, ahead of time | tens to hundreds of cores | ch 6 |
| Meshes of small cores Tenstorrent Blackhole | tiered SRAM + DRAM | mixed | tens to hundreds of cores | ch 7 |
| FPGAs and edge NPUs AMD Versal AI Engines, Apple Neural Engine | tiered SRAM + DRAM | compiler, ahead of time | tens to hundreds of cores | ch 8 |
| In-memory and analogue phase-change crossbars (research) | in the memory array | compiler, ahead of time | tens to hundreds of cores | ch 9 |
Every comparison runs the same models: Llama 3.1 8B and 70B, with the layer shapes of the Llama 3 paper. One layer of the 70B model holds 855,638,016 weights in four matrices (QKV, attention out, the gated MLP’s up-and-gate and down projections); the model has 70,548,193,280.
$$\frac{\text{flops}}{\text{byte}} \approx \frac{2\,N_w\,B}{N_w\,b/8} = \frac{16\,B}{b}$$for a batch of \(B\) sequences and \(b\)-bit weights: about two flops per byte for one user at 8 bits. A 2,048-token prefill reuses each weight 2,048 times.
A decode step at batch 1 sits hundreds of times below the ridge: it is a memory read with a little arithmetic attached. That single fact decides most of what follows.
| NVIDIA H100 SXM | NVIDIA B200 (HGX B200) | |
|---|---|---|
| FP8 dense | 1.98 PFLOP/s h100-page | 4.5 PFLOP/s hgx-page |
| HBM | 80 GB h100-page | 180 GB dgx-b200-page |
| HBM bandwidth | 3.35 TB/s h100-page | 8 TB/s dgx-b200-page |
| NVLink (both ways) | 900 GB/s h100-page | 1.8 TB/s hgx-page |
| Power | 700 W h100-page | 1.79 kW dgx-b200-page |
One H100 holds Llama 3.1 70B in FP8. A decode step for one user is bounded by reading it from HBM: 20.8 ms, 48.1 tokens per second, using 0.34% of the arithmetic. Batching fixes that: on eight H100s the same step becomes compute-bound at batch 1,024 (chapter 2 animates the climb).
B200 figures are per GPU from the HGX B200 totals (dense is half the sparse figure stated); its power is a share of the DGX B200 system’s maximum, not a chip TDP.
Google’s TPUs and AWS’s Trainium build their matrix engines as weight-stationary systolic arrays (256 × 256 MXUs on TPU v6e and TPU7x, a 128 × 128 grid on Trainium2), fed from HBM by a compiler.
| Google TPU7x (Ironwood) | AWS Trainium2 | |
|---|---|---|
| FP8 dense | 4.61 PFLOP/s tpu-7x | 1.3 PFLOP/s trn2-arch |
| HBM | 206 GB tpu-7x | 103 GB trn2-arch |
| HBM bandwidth | 7.38 TB/s tpu-7x | 2.9 TB/s trn2-arch |
| Array rows | 256 tpu-arch | 128 nki-trn2 |
| TensorCores / NeuronCores | 2 tpu-7x | 8 trn2-arch |
A large array needs a batch as wide as itself to fill it. More on the arrays themselves: the Google TPU series and Systolic Arrays Explained.
Where the weights live sets how many chips it takes to hold a model. The fewest chips of each family’s representative part that hold Llama 3.1 70B (weights plus one KV cache, no room for anything else):
One H100 or TPU7x; 4 Cerebras wafers in 16-bit (the number Cerebras publishes); 308 first-generation GroqChips; 16,570 research crossbar chips.
Llama 3.1 70B, one user, a 1,024-token context and a 2,048-token prompt, on the fewest chips of each family’s representative part (chapter 10 animates it):
| Family | Part, format | Chips | Per token | Tokens/s, one user | Prefill (2,048) | J/token | Decode utilisation |
|---|---|---|---|---|---|---|---|
| GPUs | NVIDIA H100 SXM, FP8 | 1 | 20.8 ms | 48.1 | 147 ms | 14.6 | 0.34% |
| Systolic designs | Google TPU7x, FP8 | 1 | 9.44 ms | 106 | 62.9 ms | not disclosed | 0.33% |
| Wafer-scale | Cerebras WSE-3, BF16 | 4 | 6.63 µs | 151,000 | 2.32 ms | 0.179 | 4.27% |
| Compiler-scheduled | GroqChip, INT8 | 308 | 1.09 ms | 915 | 387 ms | 0.235 | 0.06% |
| Reconfigurable dataflow | SambaNova SN40L RDU, BF16 | 3 | 25.8 ms | 38.8 | 152 ms | not disclosed | 0.29% |
| Meshes of small cores | Tenstorrent Blackhole, FP8 | 3 | 45.4 ms | 22 | 146 ms | 40.8 | 0.16% |
| FPGAs and edge NPUs | Apple M4, INT8 | 3 | 194 ms | 5.17 | 2.55 s | not disclosed | 0.64% |
| In-memory and analogue | IBM HERMES project chip, INT8 | 16,570 | 42.7 µs* | 23,400 | 315 µs | 0.0143 | 0.31% |
* crossbar time only (attention, digital partial sums and chip-to-chip transfers are not counted). Energy is TDP × the bound’s time, only where a power figure is given.
The model takes no published speed as an input; it is checked against them. Ratio = published / model:
| Check | Kind | Published | Model | Ratio | Source |
|---|---|---|---|---|---|
| Llama 3.1 70B, 16-bit: CS-3 systems needed | capacity | 4 | 4 | 1 | cerebras-inference |
| a 20B model, 16-bit: CS-3 systems needed (weights only) | capacity | 1 | 1 | 1 | cerebras-inference |
| Llama 2 70B, INT8: GroqChips used vs the fewest that hold it | capacity | 576 | 301 | 1.91 | nextplatform-groq |
| 8x H100, Llama 2 70B, FP8, MLPerf Offline tokens/s vs the compute ceiling | throughput | 21,806 | 115,200 | 0.189 | mlperf-h100 |
| 576 GroqChips, Llama 2 70B: tokens/s per user vs the bound | latency | 300 | 3,644 | 0.0823 | nextplatform-groq |
| 4x CS-3, Llama 3.1 70B: tokens/s per user vs the bound | latency | 450 | 150,700 | 0.00299 | cerebras-inference |
| 1x CS-3, Llama 3.1 8B: tokens/s per user vs the bound | latency | 1,800 | 1,387,000 | 0.0013 | cerebras-inference |
| 16 SN40L sockets, Llama 3.1 8B, BF16: tokens/s per user vs the bound | latency | 1,042 | 1,791 | 0.582 | sn40l |
| 16 SN40L sockets, Llama 3.1 70B, BF16, speculative decoding: tokens/s per user vs the one-token-per-step bound | speculative | 457 | 203.3 | 2.25 | sn40l |
Chip counts match. For designs that read their weights from DRAM the bound is close (MLPerf’s H100s reach 19% of the compute ceiling; the SN40L paper’s 8B figure is 58% of its bound). For SRAM designs it is not: once weights are in SRAM, per-layer latency the vendors do not break down limits one user. The “speculative” row beats a one-token-per-step bound because speculative decoding emits several tokens per weight read.
SRAM, HBM and DDR tiers on SambaNova’s RDUs; NVIDIA’s announced Groq 3 LPX racks pair SRAM-only LPUs with GPUs; Cerebras’s announced CS-4 raises SRAM bandwidth to 43.2 PB/s per wafer.
FP8 on H100, B200, TPU7x and Trainium2; block FP8 on Tenstorrent; INT4 on edge NPUs. Halving the weight width halves the bytes of a decode step.
XLA on TPUs, whole-layer fusion on RDUs, a static schedule on LPUs, per-core data-movement programs on meshes.
NVLink, TPU inter-chip links, NeuronLink, and Ethernet ports on Tenstorrent’s cards.
Next: deck 02, the SRAM machines, and deck 03, dataflow, meshes, edge devices and in-memory compute.