An LLM Transformer Decoder in SystemVerilog

What this deck covers

Twenty-six slides. Half walk through every RTL module in the repo at the level of a code review; the other half put the design in context — the algorithm, the historical hardware, and the real shipping silicon that runs the same workload on a million-times bigger budget.

1. Why a transformer-decoder accelerator exists
2. The decoder block, end to end
3. Q8.8 fixed-point — the only number format in the design
4. Module: transformer_pkg.sv — the whole arithmetic library in one file
5. Module: processing_element.sv — the MAC tile
6. Module: systolic_array.sv — 4×4 array with fill / drain accounting
7. Module: softmax_unit.sv — PWL exp + reciprocal-by-LUT FSM
8. Module: layer_norm.sv — division-free norm via RSqrt + Newton–Raphson
9. Module: feed_forward.sv — two GEMMs and a ReLU
10. Module: multi_head_attention.sv — heads, masking, softmax, KV-cache
11. Module: transformer_decoder.sv — the top-level FSM
12. Streaming variant — 99.4% fewer registers
13. KV-cache: kv_cache_bram.sv and friends
14. Verification — three tiers, 84 tests
15. Synthesis on Artix-7 — what's tight, what's slack
16. A short history of transformer hardware
17. Real systems — NVIDIA Hopper Transformer Engine
18. Real systems — Blackwell, Trillium, Trainium, MI300
19. Real systems — Cerebras, Groq, Tenstorrent, Etched, SambaNova, Apple ANE
20. The KV-cache problem at scale — PagedAttention, NVFP4 KV
21. Q8.8 vs INT8 vs FP8 vs FP4 — when fixed-point wins
22. Transformer-on-FPGA — FTRANS, EdgeBERT, FQ-BERT, ViTA
23. Where this design fits
24. Interactive — decoder cycle counter
25. Interactive — Q8.8 fixed-point explorer
26. Limitations & what would change at the next scale

Why a transformer-decoder accelerator exists

"Attention Is All You Need" (Vaswani et al., NeurIPS 2017, arXiv:1706.03762, posted 12 June 2017) replaced the recurrence and convolution stack with self-attention plus position-wise MLPs. It exposed a compute pattern that has dominated silicon roadmaps ever since:

• ≥ 99% of the FLOPs are dense GEMM (Q/K/V/O projections, FFN expand & contract).
• ~1% is non-GEMM: softmax, LayerNorm/RMSNorm, residual adds, RoPE, embedding lookups.
• That 1% is on the critical path of every token — the GEMM cannot proceed without softmax having finished, and softmax cannot proceed without LayerNorm having finished.

By 2020, attention had become a measurable fraction of inference time, and the academic-hardware community responded with attention-as-a-primitive papers (A³ HPCA 2020, SpAtten HPCA 2021, ELSA ISCA 2021, FACT ISCA 2023). Industry caught up in 2022 when NVIDIA shipped the Hopper Transformer Engine — the first time a GPU had per-layer FP8 statistics built in. Blackwell Ultra (2025) doubled the throughput of the SFU's MUFU.EX2 instruction, the one softmax actually sits on — silicon admitting that softmax is finally a first-class concern.

What this repo is

This is the educational analogue of all of the above, scaled to fit in 16 GB of host memory and on a $200 Artix-7 FPGA. Every block — GEMM, softmax, LayerNorm, KV-cache, top-level FSM — is small enough to read in one screen of code. None of the moves above are skipped: the softmax is a real PWL approximation, the LayerNorm is a real division-free RSqrt path, the KV-cache really avoids re-projection of past tokens.

The decoder block, end to end

One transformer_decoder instance corresponds to one decoder layer. Stacking N of them gives an N-layer decoder; weight memories can be shared across layers (area-optimal, time-multiplexed) or replicated (throughput-optimal). The dataflow per token is:

  token_emb ──► [LayerNorm 1] ──► [Multi-Head Attention] ──►(+)──► [LayerNorm 2] ──► [Feed-Forward] ──►(+)──► out_emb
                                                              ▲ Residual 1                                ▲ Residual 2
                  └────────────────────────────────────────────┘             └─────────────────────────────┘

This is a pre-norm decoder: LayerNorm sits before each sub-layer, with a residual that bypasses the sub-layer. Pre-norm is the architecture used by GPT-2, GPT-3, GPT-4, LLaMA-1/2/3, Mistral, Qwen, and most modern decoder-only LLMs. The original "Attention Is All You Need" Transformer was post-norm; the switch happened around 2019 because pre-norm trains far more stably at large scale (Xiong et al., "On Layer Normalization in the Transformer Architecture", ICML 2020).

Per-token sequence of operations

1. LayerNorm 1 on the input embedding — zero mean, unit variance, learnable γ/β.
2. Multi-head attention: compute Q from current token; pull K, V for all prior positions from the KV-cache; compute scores Q·Kᵀ / √d_k; apply causal mask; softmax; weighted sum over V; concat heads; project with W_O.
3. Residual 1: token_emb + attention_out (saturating add in Q8.8).
4. LayerNorm 2 on the residual.
5. Feed-Forward: ReLU(x·W₁ + b₁) · W₂ + b₂. The "wide-then-narrow" 4× expansion that all transformer FFNs use.
6. Residual 2: residual1 + ffn_out.
7. Update KV-cache for this position; advance seq_pos.

Module hierarchy

transformer_decoder          ← Top-level FSM (12 states)
├── layer_norm × 2           ← Pre-attention & pre-FFN normalisation
├── multi_head_attention     ← Causal MHA with KV-cache + time-multiplexed softmax
│   └── softmax_unit         ← Shared, MAX_SEQ_LEN-wide
├── feed_forward             ← Two-layer FFN with ReLU
└── transformer_pkg          ← Parameters · types · FP utilities

Q8.8 fixed-point — the only number format in the design

Every value in the datapath — activations, weights, biases, partial sums, softmax outputs, normalised activations — is 16-bit two's-complement Q8.8. That means 8 integer bits, 8 fractional bits, signed. Range −128.000 to +127.996, resolution 1/256 ≈ 0.0039.

Why fixed-point at all?

• One DSP per multiply. Artix-7 DSP48E1 is 25×18 signed; two Q8.8 operands fit comfortably with both inputs sign-extended.
• No floating-point unit. An IEEE binary16 multiplier on a 7-series part costs roughly 4× the LUT and FF count of a 16×16 fixed multiply, plus pipelining for normalise/round.
• Transparent. You can read a Q8.8 value off a waveform and know what it represents — useful for hand-debugging an educational design.
• Workload fits. With pre-LayerNorm scaling and 1/√d_k on the attention scores, well-trained Q8.8 small-D activations sit comfortably inside [−8, +8] — five integer bits of headroom.

Where Q8.8 hurts

Q8.8 has the same 65 536-level dynamic range as INT16 — about 1/40 of FP16's range. For D_MODEL ≥ 1024 and unscaled activations (the GPT-2-large era and beyond), per-tensor Q8.8 starts to fail because individual attention activations hit the rails. That's the regime the industry left fixed-point behind for: SmoothQuant (per-channel), AWQ (per-group), GPTQ, FP8-with-per-tile-scale (the Hopper Transformer Engine), and now NVFP4 (per-block FP4 with E4M3 scale).

This design lives in the small-D regime where Q8.8 works, on purpose. The package-level types make it the single point of change: edit DATA_WIDTH, FRAC_BITS, and the FP utility functions, rebuild, re-run the 84-test suite.

Module: transformer_pkg.sv

233 lines. The whole arithmetic library in one place — every other module imports it. Edit this file to scale the design.

Parameters

The four utility functions every other module uses

// Multiply two Q8.8 values, return Q8.8 (truncated)
function automatic signed [15:0] fp_mul(input signed [15:0] a, input signed [15:0] b);
  logic signed [2*DATA_WIDTH-1:0] product;
  product = a * b;
  return to_data(product >>> FRAC_BITS);
endfunction

// Saturating add for Q8.8
function automatic signed [15:0] fp_sat_add(input signed [15:0] a, input signed [15:0] b);
  logic signed [DATA_WIDTH:0] sum;
  sum = {a[DATA_WIDTH-1], a} + {b[DATA_WIDTH-1], b};
  if (sum > $signed({1'b0, {(DATA_WIDTH-1){1'b1}}}))      return {1'b0, {(DATA_WIDTH-1){1'b1}}};
  else if (sum < $signed({1'b1, {(DATA_WIDTH-1){1'b0}}})) return {1'b1, {(DATA_WIDTH-1){1'b0}}};
  else return to_data(sum[DATA_WIDTH-1:0]);
endfunction

The interesting one: fp_inv_sqrt

A division-free reciprocal-square-root, ~12 bits of accuracy in Q8.8. The same shape of code (CLZ → normalise → 32-entry LUT → one Newton–Raphson step → denormalise) appears three times in the design: here for LayerNorm's 1/√σ², in softmax_unit for 1/Σexp, and as a pattern that students can transplant to any Q-format reciprocal they need. The key trick is CLZ-normalise once, denormalise at the end — it lets a 32-entry LUT cover the full 16-bit input range to identical accuracy across decades.

For odd leading-zero counts the code multiplies by a Q2.14 constant for √2 (= 23170) so the binary-only shift can resume after the LUT step. This is verbatim what x86's RSQRTSS and ARM's VRSQRTE.F do internally — minus the IEEE rounding modes.

Module: processing_element.sv

52 lines. The smallest possible MAC tile — three flops, one multiplier, one adder. Same shape as Kung & Leiserson 1978; same shape as a single TPU MXU lane in 2025. The pattern is forty-eight years old and still right.

always @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    a_out <= 0; w_out <= 0; accumulator <= 0;
  end else if (clear) begin
    a_out <= 0; w_out <= 0; accumulator <= 0;
  end else if (enable) begin
    a_out       <= a_in;                            // skew east
    w_out       <= w_in;                            // skew south
    accumulator <= accumulator + (to_acc(a_in) * to_acc(w_in));
  end
end

assign acc_out = accumulator;

What's interesting

• Output-stationary. The accumulator never moves; A and B operands flow past. Identical pattern to Google's TPU MXU. Compare with the mmul_pe.sv in the companion AI_MMUL_Unit repo — line-for-line equivalent.
• Asynchronous active-low reset. Every flop in the design uses posedge clk or negedge rst_n; this is the Xilinx 7-series convention so it will stay in BRAM-aware retiming and not block CDC checks.
• Synchronous clear. Lets the array zero its accumulators between matrix tiles without a global reset.
• Sign-extension via to_acc. The 16→32-bit sign-extension is wrapped in a package-level helper because iverilog 10.1 mishandled inline {{16{a[15]}}, a}. A workaround that became a clarification.

Synthesis result on Artix-7 (xc7a35t, 100 MHz target)

One DSP, one PE. The 4×4 array uses 16 DSPs — under 30% of the xc7a35t's 90 DSP48E1 slices. Headroom for an 8×8 array is real on this part.

Module: systolic_array.sv

92 lines. Generates ROWS × COLS PEs, wires their A-east and B-south chains, and tracks a completion counter so the host can tell when the last accumulator has settled.

Topology

A streams left→right (blue), B streams top→bottom (purple), accumulators stay put (green).

Latency accounting

• Fill: ROWS + COLS − 1 = 7 cycles before every PE has a valid operand pair.
• Steady-state: K cycles, R·C MACs each.
• Drain: same as fill once enable drops.
• done assertion: a long-standing one-cycle bug here was fixed in commit bdc0614. The cycle counter pre-increments, so the comparison threshold is ROWS + COLS − 2, not − 1. The corrected RTL asserts done on exactly the cycle where PE(R−1, C−1) registers its final partial sum. The TB tb_systolic_array.sv was updated to verify this.

Why output-stationary, not weight-stationary?

This design loads a fresh weight tile for every projection (Q, K, V, O, FFN₁, FFN₂), so weights aren't reused enough to amortise weight-stationary's load cost. Output-stationary is also the natural shape for the QKᵀ score computation, which dominates per-head cycles. NVDLA, Apple ANE, and most edge NPUs go weight-stationary only because they hold a whole layer's weights in SRAM for batched inference — a regime this small reference design doesn't aim at.

Module: softmax_unit.sv

332 lines — the largest module in the repo by line count. Implements the standard max-subtract-exp-normalise recipe, every step of which is approximated:

  S_IDLE ─► S_FIND_MAX ─► S_SUBTRACT_EXP ─► S_SUM ─► S_RECIP ─► S_NORMALIZE ─► S_DONE

Step 1 — find max (numerical stability)

One pass over the input vector, holding the running maximum. Standard textbook trick: subtracting max(x) before exp prevents overflow without changing the result, because softmax is shift-invariant.

Step 2 — PWL exp approximation

A 4-segment piecewise-linear approximation of exp(x) over the negative range — that's all softmax ever sees after subtracting max:

function automatic signed [15:0] approx_exp(input signed [15:0] x);
  if (x >= 0)               return 16'sh0100;                      // exp(0) = 1.0
  else if (x > -16'sh0100)  return 16'sh0100 + x;                  // (-1, 0): linear
  else if (x > -16'sh0200) begin                                 // (-2, -1)
    result = 16'sh0060 + (x + 16'sh0100) >>> 1;                  // ~0.375 + 0.5*(x+1)
    return (result > 0) ? result : 16'sh0004;
  end else if (x > -16'sh0400) begin                          // (-4, -2)
    result = 16'sh0020 + (x + 16'sh0200) >>> 2;                  // ~0.125 + 0.25*(x+2)
    return (result > 0) ? result : 16'sh0004;
  end else                  return 16'sh0004;                      // floor for very negative
endfunction

This is good enough for token-selection ordering (which is all that matters for greedy/argmax decoding) and produces probabilities that sum to ≈ 1.0 within a few LSBs in Q8.8. The TB confirms Σprobs ≈ 0xFE / 0x100 for an 8-element uniform input.

Step 3 — sum the exps, then reciprocal

Computing p_i = exp(x_i) / Σexp would need a divider. Instead the unit computes recip = 65536 / exp_sum in one shared cycle, then multiplies each exp_i by recip with the same fp_mul the rest of the design uses. Because fp_mul implicitly does (a·b) >> 8, the formula works out to (exp_i << 8) / exp_sum — exactly the original division.

The reciprocal itself uses the same CLZ-normalise → 32-entry Q2.14 LUT → one Newton–Raphson step → denormalise pattern as fp_inv_sqrt. Newton–Raphson here is r ← r·(2 − x·r), which converges quadratically; ~12 bits of accuracy after one step is enough for Q8.8 outputs.

Synthesis result

The 47.9 MHz figure is one of the two reasons the full design doesn't meet the 100 MHz target. Pipelining the reciprocal across two FSM states would more-than-double Fmax — the deferred work is documented in README.md.

Module: layer_norm.sv

151 lines. Computes y_i = γ_i · (x_i − μ) / √(σ² + ε) + β_i across a VEC_LEN-element vector. The interesting part is what's not here: any divider.

  S_IDLE ─► S_MEAN ─► S_VARIANCE ─► S_NORMALIZE ─► S_DONE

Three division-replacement tricks in one module

• Mean: μ = (Σxi) >>> log₂(VEC_LEN). The runtime $fatal at instantiation enforces VEC_LEN = power of two — added in commit fed82e7 after a code review caught that >>> would silently round for non-power-of-2 lengths.
• Variance: same shift trick on Σ(x_i − μ)².
• 1/√(σ² + ε): fp_inv_sqrt from the package — a CLZ-normalised, LUT-seeded, Newton–Raphson-refined reciprocal square root. Same shape as compute_reciprocal in softmax_unit.

The +1 epsilon (literal 16'sh0001 = 1/256 in Q8.8) is enough to keep the LUT index away from zero on the constant-input case (variance = 0). The behavioural test Constant→zero exercises this: feed a vector of identical values, expect zero output.

Why this matters

A hardware integer divider for 32-bit values is ~600 LUTs and 12+ pipeline stages on Artix-7. Replacing it with a 32-entry LUT + two 16×16 multiplies (Newton–Raphson) keeps the path entirely inside the DSP fabric. The cost is ~12 bits of accuracy — more than enough for Q8.8.

Synthesis result

29.2 MHz is the slowest module in the design and therefore the system-level limiter. The path is the combinational chain inside the Newton–Raphson step: 16-bit multiply → subtract from constant → 16-bit multiply → variable shift. All inside a single FSM state. Pipelining across two FSM states is the documented path to ≥ 100 MHz.

RMSNorm vs LayerNorm

Modern LLMs (LLaMA, Qwen, Mistral) use RMSNorm instead of LayerNorm — same RSqrt path, but no mean-subtraction. RMSNorm is roughly 30% cheaper and slightly faster to train. Migrating this module to RMSNorm is mostly deletion: drop S_MEAN, drop the centered[] array, keep fp_inv_sqrt.

Module: feed_forward.sv

125 lines. Two linear layers and a ReLU. The wide-then-narrow shape every transformer FFN uses — D_MODEL=64 expands to D_FF=256, then collapses back to D_MODEL=64.

  S_IDLE ─► S_LINEAR1 ─► S_RELU ─► S_LINEAR2 ─► S_DONE
              (D_FF iters)      (parallel)   (D_MODEL iters)

Why FFN is the biggest GEMM in the layer

For one decoder layer at sequence length S, the FLOP budget breaks down (per "Attention Is All You Need" §3.3 and the standard expansion):

FFN is roughly 44% of the layer for typical S, and at single-token inference (S = 1) it's the dominant compute. This is why GPT-3, GPT-4, and LLaMA all keep the 4× FFN ratio — and why FlashAttention, which optimises the attention block, has so little to say about FFN.

The ReLU choice (not SwiGLU)

Modern LLMs use SwiGLU (Shazeer 2020): three weight matrices, two for the gate, one for the projection, sigmoid(W₁x) ⊙ (W₂x) → W₃ projection. SwiGLU outperforms ReLU on training perplexity but costs 3 GEMMs and a sigmoid per token instead of 2 GEMMs. The reference design uses plain ReLU because:

• One sign-bit test per element — zero hardware cost.
• Drop-in replacement: any FP-domain non-linearity (GeLU, SiLU, SwiGLU) plugs into S_RELU as a function.
• Educational clarity: the FFN is two GEMMs separated by a non-linearity. ReLU keeps that obvious.

What's missing for a "real" FFN

• No bias support for SwiGLU's gate path.
• S_LINEAR1 and S_LINEAR2 use a sequential for j loop accumulator — they could use the systolic array. Doing so would cut FFN cycles from O(D_MODEL · D_FF) to ~O(max(D_MODEL, D_FF)) but at the cost of weight-tile bookkeeping. Straightforward as future work; left out to keep the FSM readable.

Module: multi_head_attention.sv

362 lines. The single biggest module in the repo. Implements the canonical scaled-dot-product attention from "Attention Is All You Need" §3.2 — minus the divide-by-√d_k, which is folded into pre-scaling of the weights.

  Attention(Q,K,V) = softmax(Q·Kᵀ / √d_k) · V    over each of N_HEADS in parallel

Ten-state FSM

The shared softmax unit

One softmax_unit instance is time-multiplexed across heads. The instance is sized for VEC_LEN = MAX_SEQ_LEN, and positions beyond seq_pos are padded with the smallest representable Q8.8 value (−128.0) so the PWL exp approximation forces them to ~0. This implements the causal mask as a number, not as a control bit — saves a separate masking pipeline.

The KV-cache interface

K and V for all prior positions are held in dual-port memory. On every attention step:

• Read all K[0 .. seq_pos] for QKᵀ (parallel with V reads).
• Read all V[0 .. seq_pos] for the weighted sum.
• Write the freshly-computed K[seq_pos] and V[seq_pos] for next token.

In the streaming variant the cache is kv_cache_bram — a real dual-port BRAM that does element-by-element vector writes and single-element reads. In the non-streaming variant it's a flat unpacked array, intended only for simulation.

What's not here

• No RoPE. Position information is implicit (the next token is appended at seq_pos). LLaMA-class models add Rotary Positional Embedding to Q and K before the dot product. RoPE is two element-wise multiply-adds per element — easy to insert as an extra FSM state.
• No FlashAttention-style fusion. The score matrix is materialised fully before softmax. For MAX_SEQ_LEN = 128 this is fine; at longer contexts a tile-and-fuse rewrite à la FA-2/FA-3 saves a factor of S in K/V re-reads.
• No GQA / MQA. All N_HEADS have full K, V vectors. Modern inference systems (LLaMA-2-70B onwards) use Grouped-Query Attention to share K, V across multiple Q heads — cuts KV-cache memory by 4–8×.

Module: transformer_decoder.sv

249 lines. The conductor — instantiates LayerNorm-1, MHA, LayerNorm-2, FFN, threads them with two residual additions, and runs a 12-state top-level FSM.

  S_IDLE ─► S_LN1_START ─► S_LN1_WAIT ─►
            S_ATTN_START ─► S_ATTN_WAIT ─► S_RESIDUAL1 ─►
            S_LN2_START ─► S_LN2_WAIT ─►
            S_FFN_START ─► S_FFN_WAIT ─► S_RESIDUAL2 ─► S_DONE

Things the conductor handles that the sub-modules don't

• Single-cycle start pulses. The default in the always-block is to drive every *_start low; the case statement raises one for exactly the cycle of its S_xx_START state. Sub-modules launch on the rising edge.
• Saturating residual adds. Both residuals use fp_sat_add rather than wrap-around. Residual + LayerNorm guarantees mid-band activations, so saturation is rare in practice; checking it is one of the behavioural-tier tests.
• Weight pass-through. Every weight matrix and bias vector is a top-level port. In the streaming variant they're loaded over an AXI-Lite-style bus before the FSM is launched; in the simulation variant they're set as Verilog constants. This is the cleanest split between control plane (FSM) and data plane (weights) the design has.

Per-token cycle budget

Approximate, for D_MODEL=64, N_HEADS=4, seq_pos = 64 (mid-context):

At the achievable 29.2 MHz that's ~1.2 ms per token, ~830 tok/s — for a one-layer 64-dim toy model on a $200 FPGA. Stack 12 layers at this clock and you get ~70 tok/s, a reasonable rate for a small embedded model. The streaming variant trades throughput for area and is what synthesises within the 16 GB Vivado memory budget.

Streaming variant — 99.4% fewer registers

The repo ships two top-levels:

Why two?

For an educational repo, having both makes the trade-off concrete: the streaming variant is the bridge variant with the boundary registers stripped and the weights moved to BRAM. Diff one against the other and you can see, line by line, what "minimum-area" means in practice. The streaming variant's regression test (tb_transformer_decoder_stream.sv) produces bit-identical outputs to the bridge variant for the same weight load.

From stream_results.log:

--- Test 1: Single token at position 0 ---
  out_emb[0] = 0097 (0.5898)   ← matches bridge variant exactly
  out_emb[1] = ff7f (-0.5039)
  ...
[PASS] KV cache active: token-1 output differs from token-0
============================================
  Results: 4 PASSED, 0 FAILED
============================================

The streaming pattern, in one sentence

Replace combinational weight arrays with single-port BRAM and an address generator; replace pipeline registers between sub-modules with a single bus + handshake; let the FSM serialise everything that used to overlap. Trade ~10× fewer registers for ~3–5× more cycles per token. On Artix-7 that's the difference between "synthesises" and "exceeds 16 GB host memory".

KV-cache: kv_cache_bram.sv

Without a KV-cache, autoregressive generation re-projects every prior token's K and V on every step — O(n²·d) per token. With a KV-cache, only the fresh Q is computed; past K, V are read from memory — O(n·d) per token. The streaming variant uses a real dual-port BRAM (bram_dp) sized for MAX_SEQ_LEN × D_MODEL.

Address mapping

One BRAM each for K and V. Linear address: mem[pos · D_MODEL + dim] — row-major, position-major.

Two-port pattern

Sizing

For MAX_SEQ_LEN = 128, D_MODEL = 64, Q8.8: 128 × 64 × 16 bits × 2 (K and V) = 32 768 bits = 4 KB. Fits in two RAMB18E1 BRAMs comfortably (the xc7a35t has 50 of them).

By contrast, a real LLM:

That's the entire reason long-context LLM inference is HBM-bandwidth-bound rather than compute-bound. PagedAttention (vLLM, SOSP 2023) borrows OS virtual memory to cut KV waste from 60–80% to under 4%. NVIDIA has been pushing FP8 and NVFP4 KV quantisation to claw the size back. Same idea here, fifteen orders of magnitude smaller.

Verification — three tiers, 84 tests

What the behavioural tier covers

Recent fixes — git log as a verification record

• bdc0614 — Fix systolic_array done signal: asserted one cycle too late.
• b4cb919 — Fix clz16 return type and add default cases to LUTs. CLZ for 16-bit needs to return a 5-bit result so the all-zero input maps to 16; default cases protect synthesis against X/Z propagation.
• 0d99b46 — tb_transformer_decoder: replace fake Test 2 with real KV-cache check.
• a7ce1a1 — tb_softmax: fix Tests 1 and 4 to actually detect failures. A test that always passes is worse than no test. Two were rewritten to compare against expected probabilities, not just "non-zero".
• 6bda9a7 — run_sim.sh: detect and propagate [FAIL] assertion failures.
• fed82e7 — layer_norm: add runtime assertion for power-of-2 VEC_LEN.

That's six bugs caught after the initial release. Each one became a TB; each TB is in the regression suite.

Structural lint — scripts/lint_check.py

Walks all eight RTL files, checks module/endmodule balance, package imports, reset patterns (posedge clk or negedge rst_n), and cross-file instantiation resolution. Cheap to run, catches typos that iverilog's parser elides.

Synthesis on Artix-7 — what's tight, what's slack

Vivado 2025.2, target xc7a35tcpg236-1, 100 MHz clock constraint. The full top-level exceeded the host's 16 GB memory budget, so sub-modules were synthesised individually. The numbers below are per-sub-module and aren't additive (the array shares logic with the PEs it contains).

Why 100 MHz wasn't met

Two modules contain combinational logic inside a single FSM state that's too long for a 10 ns period:

• layer_norm: fp_inv_sqrt is fully combinational. Its critical path is mul → subtract → mul → variable shift, all in 1 cycle. Pipelining across two FSM states (a register between the LUT-and-mul and the second-mul-and-shift) restores ≥ 100 MHz.
• softmax_unit: compute_reciprocal has the same shape — same fix.

This isn't a bug; it's a deliberate trade. The combinational implementation is one screen of code; the pipelined version is two — and an educational design is allowed to leave the pipeline as homework. The README documents the gap.

What it costs on a 7-series xc7a35t

Total budget: 20 800 LUTs, 41 600 FFs, 50 BRAM, 90 DSP48E1.

• The streaming top-level fits in ~12% of LUTs, ~9% of FFs, ~30% of DSPs.
• An 8×8 systolic array (4× more compute) would still leave ≥ 50% of DSPs free.
• Adding RoPE, GQA, and a real RMSNorm would not move the needle on Artix-7 utilisation.

A short history of transformer hardware

The throughline: every generation, the silicon learns one more thing about transformers. Pre-2020 it learned "you need a lot of MACs". 2022 it learned "you need 8-bit numbers and per-layer scales". 2024 it learned "you need an explicit memory primitive (TMA, TMEM)". 2025 it learned "softmax is on the critical path; the SFU has to be wider". Forty-eight years from Kung's PE to the Blackwell Ultra SFU — the primitive doesn't change, the surroundings do.

Real systems — NVIDIA Hopper Transformer Engine

Hopper (2022) was the first GPU architecture marketed by reference to a non-GEMM workload. The Transformer Engine isn't a separate IP block — it's a software runtime + hardware features that turn each forward/backward pass into a per-layer scaled FP8 matmul.

What the Transformer Engine actually does

1. Profile the activation/weight tensor's max-abs in FP16.
2. Pick a per-layer scaling factor so the tensor fits FP8's range. Two FP8 formats:
   • E4M3: 4-bit exponent, 3-bit mantissa, range ~ ±448 — used for forward weights and activations, where precision matters.
   • E5M2: 5-bit exponent, 2-bit mantissa, range ~ ±57 344 — used for gradients, where range matters.
3. Cast the tensor to FP8, multiply on the 4th-gen Tensor Core, accumulate in FP32.
4. Cast back to BF16/FP16 for downstream non-GEMM ops (LayerNorm, softmax, residual).

The two enabling primitives

• TMA — Tensor Memory Accelerator. Async multi-dimensional DMA from HBM into shared memory. Decouples operand-fetch from compute warps so the SM is never operand-bound.
• 4th-gen Tensor Cores. Native FP8 ops, 2× the throughput per area vs Ampere's BF16 path.

How the reference design compares (very loosely)

The ratio is roughly 10⁹. The architectural moves are the same — block GEMM, max-subtract softmax, normalise, residual. Implementing them in 1 500 lines of SystemVerilog instead of 50 million transistors is the entire point.

Real systems — Blackwell, Trillium, Trainium, MI300

Three different bets

• NVIDIA — every generation, one new datatype + one new memory primitive. FP8+TMA (H100); FP4+TMEM+wider SFU (Blackwell). General-purpose by construction.
• Google — keep the MXU shape stable, double the data movement. SparseCore handles the parts of the model that aren't matmul (embeddings, lookups).
• AWS — heterogeneous engines per core; explicit Vector/Scalar paths for softmax/norm rather than relying on a "free" SFU.

Notice what no shipping system has: a hardwired fixed-point datapath of the kind in this repo. Production silicon picks per-layer scaled FP8/FP4 instead because (a) range adapts, and (b) the area cost of FP arithmetic is no longer dominant at 4 nm. Q8.8 lives on in educational and edge-FPGA niches — this design is squarely in the first.

Real systems — the rest of the field

Three philosophies, one workload

• Datacentre generality (NVIDIA, TPU, AMD, Trainium) — a big array, hierarchical memory, and a compiler that re-tiles for any model.
• Determinism & in-memory (Groq, Cerebras) — banish DRAM, schedule every cycle ahead of time. Spectacular tail-latency for inference; brittle outside the supported workloads.
• Workload specialisation (Etched Sohu, Apple ANE) — hardwire one model class. The asymptote of "do less, faster".

The reference design in this repo lives at the educational end of the third axis: it does only a transformer decoder and nothing else, and the hardwiring is its strongest feature for teaching purposes.

The KV-cache problem at scale

The KV-cache is the single biggest difference between transformer training and transformer inference. Training is GEMM-bound; inference, especially at long context, is HBM-bandwidth-bound — and the bandwidth is consumed by KV reads.

The arithmetic

Per token, per layer, the decoder reads 2 · seq · D_MODEL bytes of KV and computes 2 · seq · D_MODEL MACs. The arithmetic intensity is O(1) FLOPs per byte — the canonical memory-bound case. No matmul accelerator helps. On an H100 at single-batch inference, MMUL utilisation drops to 1–3%; the rest of the chip is idle waiting on HBM.

The two industry fixes

What this design does instead

For MAX_SEQ_LEN = 128, D_MODEL = 64, Q8.8: the entire KV cache is 4 KB. It fits in two BRAMs. There is no HBM, no PagedAttention, no quantisation pressure — the problem doesn't exist at this scale. What survives is the interface: append-on-token, broadcast-on-attention. That interface is the same on this Artix-7 design and on a 1 000 000 000 000-parameter model running on 16 H100s.

Q8.8 vs INT8 vs FP8 vs FP4 — when fixed-point wins

What this matters for

• Small-D inference (D_MODEL ≤ 256): Q8.8 and INT8 are roughly equivalent in accuracy and are the cheapest in hardware. Q8.8's edge is transparency; INT8's edge is tooling (TFLite, ONNX Runtime, llama.cpp Q4_K).
• Mid-D inference (D_MODEL 512–1024): per-channel INT8 is the production sweet spot. Per-tensor formats start to fail because of activation outliers in attention.
• Large-D inference (D_MODEL ≥ 2048, GPT-3+): FP8 with per-tile scaling. The Hopper Transformer Engine exists for this regime.
• Frontier training (LLaMA-3, GPT-4 successors): NVFP4 / MXFP4 with stochastic rounding. Block-scaled formats handle outliers without per-tile FP32 loss-scaling.

The area / energy curve

Multiplier area is roughly quadratic in mantissa width. So:

• FP32 → BF16: ~4× smaller multiplier (Google's 2018 BF16 paper, "free" for inference, near-free for training).
• BF16 → FP8: another 4×, but only via Transformer Engine + per-tile scaling. Not free.
• FP8 → FP4: another 4×, but only via MXFP/NVFP4 + stochastic rounding. Active research as of 2024–2026.
• Below FP4: papers, no shipping silicon.

The curve eventually meets a wall: at FP4, multiplier energy is so low (~0.03 pJ at 4 nm) that operand movement dominates the budget. That's the regime block scaling exists for — share one E4M3 scale across 16 elements, broadcast it once, save 15× on scale-fetch energy.

Transformer-on-FPGA — the genealogy

The fixed-point lineage

This repo's core algorithmic moves all trace back to the FQ-BERT line: integer softmax (PWL exp), integer LayerNorm (RSqrt LUT + Newton–Raphson), integer attention scoring, integer FFN. The FQ-BERT paper showed that you can run BERT entirely in the integer domain with < 1% accuracy loss — this design implements the same idea at smaller scale and with simpler approximations.

Why FPGAs are still relevant in the 2026 landscape

• Latency. Single-batch software-scheduled FPGAs eliminate the warm-up tail and runtime variability of batched GPU servers.
• Cost-per-inference at low utilisation. A $2 k Artix-7 board running 24/7 at 100% beats a $30 k H100 running at 5% — the regime most enterprise inference actually lives in.
• On-prem and air-gapped deployment. Data-sovereignty pressure (legal, defence, medical) pushes inference back behind the firewall; FPGAs are easier to procure, audit, and certify than cutting-edge ASICs.
• Educational. 1 500 lines of SystemVerilog you can read end-to-end. That's what this repo is.

Where this design fits

Honest scope statement

This is not a competitive transformer accelerator. It will not run LLaMA-7B; the systolic array is two orders of magnitude too small, the KV-cache is six orders of magnitude too small, the clock is one order of magnitude too slow. It is an educational reference implementation with three stated goals:

1. Make every architectural primitive of a transformer decoder small enough to read in one screen of code.
2. Show, by construction, that softmax + LayerNorm + KV-cache + GEMM compose into one autoregressive step.
3. Provide a regression-tested baseline for changing one variable at a time — RMSNorm, GQA, FlashAttention-style fusion, INT8 with per-channel scales, an 8×8 array — without rewriting the others.

If you're reading the RTL because you're learning hardware design, that's the audience. If you're reading it because you want to deploy a transformer to an FPGA, look at FQ-BERT or ViTA instead and bring the patterns from this repo to your own scale.

Interactive — decoder cycle counter

Choose a configuration; see the per-token cycle budget for one decoder layer at the chosen sequence position. Numbers come from the FSM cycle counts in the actual modules.

Notice how FFN's share of the budget climbs as D_MODEL grows (because FFN is O(D²) while the rest is O(D · seq)) and how seq_pos has almost zero effect on per-token cost at small D — the KV-cache is doing exactly the work it's there to do.

Interactive — Q8.8 fixed-point explorer

Type a value; see how Q8.8 stores it, the rounding error, and the worst-case error across the format's range. Compare with FP8 E4M3 (Hopper forward), FP8 E5M2 (Hopper backward), and FP4 E2M1 (Blackwell).

Try 0.5898 in Q8.8 — that's the actual out_emb[0] from the integration test (0x0097). Try 1e-4 in FP4 (rounds to zero) versus Q8.8 (also zero — Q8.8 is no better at small numbers than FP4 is). Try 200 in Q8.8 (saturates) versus FP8 E4M3 (representable). The format choices are full of these quiet trade-offs.

Limitations & what would change at the next scale

What this design doesn't do

• No RoPE. Position is implicit in seq_pos; LLaMA-class models rotate Q and K. Easy to add: one new FSM state, two element-wise multiplies per head per element.
• No GQA / MQA. Every head has its own K, V. Modern inference designs (LLaMA-2-70B+, Mistral) share K, V across multiple Q heads, cutting KV memory by 4–8×.
• No FlashAttention-style fusion. The S×S score matrix is fully materialised before softmax. For MAX_SEQ_LEN = 128 this is fine; at long context, tile-and-fuse is mandatory.
• No SwiGLU. ReLU is one sign-bit test; SwiGLU is three GEMMs and a sigmoid. Any FP-domain non-linearity can be swapped in at S_RELU; SwiGLU also needs a third weight matrix.
• RMSNorm is a 30-line refactor of layer_norm.sv (drop S_MEAN, drop the centered array, keep fp_inv_sqrt).
• No multi-layer top-level. One decoder block; stacking is documented but not synthesised.
• No AXI bus. Weights are loaded by simulation testbenches, not by a host bus master. AXI-Lite for control + AXI4 for weight DMA is the obvious productisation path.

What the next scale-up would touch

• Pipelined Newton–Raphson — splits the LayerNorm and softmax critical paths across two FSM states, restoring ≥ 100 MHz on Artix-7. Documented as the highest-priority follow-up in the README.
• BRAM-mapped weights with double buffering — lets the next layer's weights load while the current layer computes. Hides weight-fetch latency.
• Per-channel INT8 — same systolic array, scale registers next to each weight column. Halves activation memory at < 1% accuracy loss.
• 8×8 PE array — fits comfortably in xc7a35t's DSP budget (16 → 64 of 90).

The point of all of this

None of those changes are mysterious. They're all in the literature, in shipping silicon, or in the companion AI_MMUL_Unit repo. The reference design is small enough to make each change a self-contained patch, with the regression suite catching anything that drifts. That's the role of educational RTL — not to replace H100, but to make H100's architectural choices legible.