Four more ways to build an AI chip: fuse a layer into one on-chip pipeline (SambaNova), tile a mesh of small cores and join chips over Ethernet (Tenstorrent), live on a few watts and a phone's memory (NPUs and FPGAs), or compute inside the memory array itself (phase-change crossbars).
This deck follows chapters 6, reconfigurable dataflow, 7, meshes of small cores, 8, FPGAs and edge NPUs and 9, in-memory and analogue compute. Specifications, sources and access dates: the specifications page.
A GPU runs a layer as a sequence of kernels, each reading its input from HBM and writing its result back. SambaNova’s reconfigurable dataflow unit lays a whole layer out across its 1,040 compute units and wires one operation’s output straight into the next: HBM sees only the weights, the KV cache, and the layer’s input and output.
$$\begin{gathered} t_{\text{kernels}} = \sum_i \max\!\Big(\tfrac{B_i + A_i}{\beta}, \tfrac{F_i}{P}\Big) \\ \ge\; \max\!\Big(\tfrac{\sum_i B_i}{\beta}, \tfrac{\sum_i F_i}{P}\Big) = t_{\text{fused}} \end{gathered}$$One Llama 3.1 8B layer on one SN40L socket (BF16):
| Work (one layer, one socket) | Kernel by kernel | Time | Fused | Time | HBM traffic saved |
|---|---|---|---|---|---|
| decode, batch 1, context 8,192 | 471 MB | 262 µs | 470 MB | 261 µs | 0% |
| decode, batch 64, context 1,024 | 733 MB | 407 µs | 706 MB | 392 µs | 4% |
| prefill, 2,048 tokens | 1.36 GB | 1.75 ms | 487 MB | 1.45 ms | 64% |
In a prefill the intermediate activations (and an unfused attention’s score matrix) outweigh the weights, so fusion saves most of the traffic. In a batch-1 decode it saves almost none: one token’s activations are tiny. There the gain is launch overhead, which this model does not see.
| SambaNova SN40L RDU | SambaNova SN50 RDU | |
|---|---|---|
| BF16 dense | 638 TFLOP/s sn40l | 1.6 PFLOP/s sambanova-rdu |
| On-chip SRAM | 520 MB sn40l | 432 MB sambanova-rdu |
| HBM | 64 GB sn40l | 64 GB sambanova-rdu |
| HBM bandwidth | 1.8 TB/s sn40l | not disclosed |
| DDR | 1.5 TB sn40l | 512 GB sambanova-rdu |
| DDR bandwidth | 200 GB/s sn40l | – |
HBM holds the models being served; DDR holds many more, ready to swap in. On 16 sockets, HBM holds 63 copies of Llama 3.1 8B and DDR 1,494; loading one from DDR into HBM takes 5.02 ms. SambaNova’s “composition of experts” serving is built on this.
The SN40L figures are from its paper (Table II). The SN50 is listed with specifications but its page does not say it is shipping, and gives no HBM bandwidth, so the model does not run it.
| Check | Kind | Published | Model | Ratio | Source |
|---|---|---|---|---|---|
| 16 SN40L sockets, Llama 3.1 8B, BF16: tokens/s per user vs the bound | latency | 1,042 | 1,791 | 0.582 | sn40l |
| 16 SN40L sockets, Llama 3.1 70B, BF16, speculative decoding: tokens/s per user vs the one-token-per-step bound | speculative | 457 | 203.3 | 2.25 | sn40l |
On 16 sockets in BF16 at an 8K sequence length, Llama 3.1 8B ran at 1,042 tokens per second for one user: 58% of the bandwidth bound, the closest any published speed comes to its bound in these decks. The paper reports reaching “over 85% of HBM bandwidth” (a quoted claim, not a model input). Its 70B figure used speculative decoding, so it beats a one-token-per-step bound and is kept out of the validation band.
Tenstorrent builds its Blackhole chip from 120 Tensix cores on a network-on-chip, each a compute unit with its own SRAM and small RISC-V cores that move the data, plus 16 “big RISC-V” cores. Data movement is programmed core by core.
| Tenstorrent Blackhole (p150a card) | |
|---|---|
| Block FP8 | 664 TFLOP/s tt-hardware |
| SRAM | 180 MB tt-hardware |
| GDDR6 | 32 GB tt-hardware |
| GDDR6 bandwidth | 512 GB/s tt-hardware |
| Ethernet (4 x 800G; both ways) | 800 GB/s tt-hardware |
| Board power | 300 W tt-hardware |
The memory is a graphics card’s, not HBM; the scale-out is built into the card: four 800G Ethernet ports join cards directly, with no separate network card or switch chip.
One Llama 3.1 70B layer in FP8, tensor-parallel over Blackhole cards at batch 1: each card streams its share of every weight matrix from GDDR6 into its cores, and the cards all-reduce over Ethernet after the attention output and the MLP down projection.
| Cards | One layer | Ethernet part (bandwidth only) | Layer weights per core | Tokens/s, one user (layers only) |
|---|---|---|---|---|
| 1 (does not hold it) | 1.68 ms | 0 ns | 7.13 MB | 7.46 |
| 2 (does not hold it) | 838 µs | 81.9 ns | 3.57 MB | 14.9 |
| 3 | 559 µs | 109 ns | 2.38 MB | 22.4 |
| 4 | 419 µs | 123 ns | 1.78 MB | 29.8 |
| 8 | 210 µs | 143 ns | 891 kB | 59.7 |
A 70B layer is more than the cores’ SRAM, so it streams from GDDR6 on every token; more cards divide that almost exactly. At batch 1 the all-reduces move a few kilobytes, so their bandwidth time is tiny; their real cost is latency, which Tenstorrent does not state and the model leaves out.
Apple’s M4 Neural Engine is rated at 38 TOP/s (format not stated) and shares 120 GB/s of unified memory with the CPU and GPU. At batch 1 one user’s token reads every weight once, so the memory decides:
| Weights | Size | Per token | Tokens/s | Fits | J/token |
|---|---|---|---|---|---|
| 16-bit | 16.1 GB | 126 ms | 7.93 | yes | not disclosed |
| 8-bit | 8.03 GB | 63.1 ms | 15.9 | yes | not disclosed |
| 6-bit | 6.02 GB | 47.5 ms | 21.1 | yes | not disclosed |
| 5-bit | 5.02 GB | 39.6 ms | 25.2 | yes | not disclosed |
| 4-bit | 4.01 GB | 31.8 ms | 31.4 | yes | not disclosed |
| 3-bit | 3.01 GB | 24 ms | 41.7 | yes | not disclosed |
| 2-bit | 2.01 GB | 16.2 ms | 61.8 | yes | not disclosed |
The arithmetic is busy for 409 µs of each 63.1 ms step at 8 bits. Narrower weights are the lever: each halving of the width doubles the bound. Edge budgets are a few watts: Hailo’s 10H module is rated 40 TOP/s in INT4 at 2.5 W typical.
An FPGA, or an adaptive SoC pairing programmable logic with arrays of small AI engines, is hardware shaped to one model. AMD’s Alveo V70 (discontinued) is rated 404 TOP/s in INT8 with 47.6 TB/s of internal memory bandwidth but 76.8 GB/s to DDR, at 75 W.
Any width at all: a 5- or 3-bit weight costs exactly its bits in memory and logic. On the V70, Llama 3.1 8B at 4 bits is bounded at 20.1 tokens/s, 3.73 J per token at TDP.
A fixed pipeline per model: no instruction fetch, no kernel launches, no batch needed to fill it, the same short latency every time. It suits small models next to their data more than large language models, whose weights do not fit on chip.
Store each weight as a conductance \(G_{ij}\) in a crossbar, apply the inputs as voltages \(V_i\), and the currents down each column are the dot products. IBM’s HERMES project chip (research, not a product) has 64 phase-change crossbars of 256 × 256 in a 14 nm process, 4,194,304 weights per chip, one analogue multiply in 133 ns.
| Model | Tiles per layer | Chips per layer | Chips for the model | Crossbar time per layer | Per token | Crossbar J/token |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 3,328 | 52 | 1,789 | 532 ns | 17.2 µs | 0.00154 |
| Llama 3.1 70B | 13,056 | 204 | 16,570 | 532 ns | 42.7 µs | 0.0143 |
No weight ever moves: the crossbars spend well under a microsecond on a layer. But the model needs one crossbar cell group per weight.
16,570 research chips for Llama 3.1 70B, where one H100 holds its 8-bit weights.
The KV cache changes every token, so attention cannot live in a crossbar: it runs in digital logic beside it.
An exact sum of 256 products of 8-bit inputs and weights needs 24 bits; the chip’s path is 8-bit, with device noise and conductance drift below that.
The chip’s precise 4-phase read mode buys accuracy with 520 ns per multiply instead of 133 ns. Computing with light is the other analogue route: see the Fourier Optics for Inference series.
Back to deck 01 for every family side by side, or deck 02 for the SRAM machines.