SoCs for
ML Inference

Training Characteristics

• Forward + backward pass — 3× the compute of inference per sample
• FP16 / BF16 / FP32 — precision critical for gradient stability
• All activations retained — for backprop, VRAM scales with batch × seq
• Throughput-optimised — maximise FLOPS utilisation, latency tolerated
• Cluster-scale — data, tensor, and pipeline parallelism across racks
• Intermittent — runs once; result is a model checkpoint

Inference Characteristics

• Forward pass only — activations discarded after each layer
• INT8 / INT4 / FP8 — lower precision acceptable; massive efficiency gain
• KV-cache retained — decode phase is severely memory-bandwidth-bound
• Latency-constrained — user-facing; time-to-first-token < 500 ms target
• Continuous batching — maximise GPU utilisation across many requests
• Always-on — edge devices run 24/7; power budget is paramount

Cloud / Hyperscale

• H100, A100, MI300X, TPU v5p
• Models: GPT-4, Gemini, Claude
• Batch sizes: 128–4096
• Latency budget: >100 ms acceptable
• TDP: 300–1000 W per accelerator
• Memory: HBM3/3E, 80–192 GB

On-Premise / Workstation

• RTX 4090, L40S, Gaudi 2, RX 7900
• Models: Llama 3, Mistral, Qwen
• Batch sizes: 1–32
• Latency budget: 50–200 ms
• TDP: 150–450 W
• Memory: GDDR7 / HBM3, 24–96 GB

Edge / Endpoint

• Apple M4, Snapdragon X Elite, Tensor G4
• Models: Phi-3, Gemma 2B, MobileNet
• Batch sizes: 1
• Latency budget: <50 ms
• TDP: 2–15 W
• Memory: LPDDR5X/6, 8–32 GB unified

Quantisation Taxonomy

PTQ vs QAT

• Post-Training Quantisation (PTQ) — no retraining; calibration dataset only; GPTQ, AWQ, SmoothQuant
• Quantisation-Aware Training (QAT) — fake-quant nodes in training graph; STE gradient; best accuracy
• GGUF / llama.cpp — CPU-friendly mixed-precision: Q4_K_M, Q5_K_M, Q8_0

Precision Formats & Efficiency

INT4 Weight-Only Quantisation (W4A16)

• Weights stored as INT4, unpacked to FP16 before MAC — no INT4 accumulator needed
• 2× weight load bandwidth saving vs INT8 — critical for memory-bound GEMV
• GPTQ: second-order Hessian-guided rounding; per-group scale every 128 weights
• AWQ (Activation-Aware): protects salient weights by per-channel scaling
• Hopper H100 has no native INT4 Tensor Core (removed after Ampere); W4A16 dequantises INT4 weights to FP16 before MMA. Ampere A100 retains INT4 at 1,248 TOPS sparse.

Tensor Core Evolution (NVIDIA)

Accuracy vs Efficiency Trade-off

KV-Cache Sizing

KV-Cache Optimisations

• Grouped Query Attention (GQA) — share K,V heads across Q groups; 8× KV reduction in Llama 3
• Multi-Query Attention (MQA) — extreme: one K,V head shared; used in Falcon
• PagedAttention (vLLM) — non-contiguous KV blocks in virtual memory; eliminates fragmentation
• KV quantisation — INT8/INT4 KV: 2–4× reduction; KV-Quant, KIVI
• Streaming LLM — sliding window + attention sinks; O(1) memory for unbounded generation

Arithmetic Intensity of Inference Operations

H100 SXM5 Roofline

Escaping the Memory Wall

Key Design Decisions

• Scratchpad vs cache — NPUs favour software-managed scratchpad; deterministic latency, no cache miss stalls, compiler controls data residency
• Double-buffering — DMA loads next weight tile while MAC processes current; hides DRAM latency entirely when compute time ≥ DMA time
• Dataflow: weight-stationary for conv (reuse weights), output-stationary for GEMV (accumulate partial sums)
• Activation unit — dedicated hardware for GELU, SiLU, RMSNorm, LayerNorm; avoids SRAM round-trip
• DMA engine — supports stride, gather, scatter; critical for non-contiguous KV-cache layouts

Scheduling & Operator Fusion

• Layer fusion — MatMul → Dequant → Add bias → GELU → next MatMul as single dispatch; eliminates SRAM store/load between ops
• Tile scheduler — partitions tensors into SRAM-sized tiles; handles bank conflict avoidance
• DMA priority — weight prefetch vs activation write-back arbitration; stall-free pipeline requires careful budget

Activation Sparsity

• ReLU networks: 50–80% of activations are zero after rectification
• Zero-gating: skip MAC if activation = 0 → power saving proportional to sparsity
• GELU / SiLU (LLMs) — near-zero but not exactly zero; less exploitable
• Hailo-8 NPU: dedicated zero-skipping hardware for ReLU CNNs

Weight Sparsity (Pruning)

• Unstructured pruning — individual weights zeroed; requires sparse accumulator or VLIW schedule; poor hardware efficiency
• Structured pruning — channels, heads, or layers removed; standard dense matmul on smaller tensors
• 2:4 structured sparsity — exactly 2 non-zeros per 4 weights; 2× throughput on NVIDIA Tensor Cores from Ampere onwards; negligible accuracy loss at 50%

Mixture-of-Experts (MoE)

• Architecture-level sparsity — only K of N expert FFN blocks activated per token
• Mixtral 8×7B: 8 experts, top-2 active — 46.7B total params, ~12.9B active per token
• Expert routing: gating network adds overhead; load-balance critical for utilisation
• All-to-all expert dispatch: dominant communication overhead in distributed inference
• DeepSeek-V3: 256 routed experts + 1 shared expert; top-8 routed per token; pairs with Multi-Head Latent Attention (MLA) for ~10–30× KV-cache reduction

NVIDIA 2:4 Structured Sparsity — Hardware Detail

• Compressed sparse weight matrix: 50% values stored + 2-bit metadata per group-of-4
• Sparse Tensor Core decompresses and computes in single pass
• 2× TOPS / 2× bandwidth saving — equivalent to doubling effective HBM bandwidth
• H100 sparse INT8: 3,958 TOPS → effectively 7,916 TOPS with 2:4 weights

Unified Memory Architecture Advantages

• Zero-copy — GPU tokenises, NPU runs LLM decode, CPU runs runtime logic — all share the same physical DRAM pages
• Coherent — no explicit DMA transfers or pinned buffers between engines
• LLM-friendly — 70B INT4 model ≈ 35 GB: fits M3 Ultra (192 GB) easily; fits M4 Pro (48 GB) at smaller models
• Bandwidth pooled — 120–800 GB/s shared; no PCIe bottleneck (≤64 GB/s)

Heterogeneous Dispatch Strategy

Inference Power Breakdown (Mobile NPU)

DVFS Strategy for NPU

• Compute-bound ops (prefill, large conv): max frequency, max voltage — utilise Tensor Cores fully
• Memory-bound ops (decode GEMV): reduce NPU frequency; performance limited by DRAM BW not clock
• Clock gating — idle MAC rows gated cycle-by-cycle; partial utilisation common in sparse layers
• Power gating — entire NPU domain powered down between inference requests; critical for mobile

Thermal Management

• TDP envelope — mobile: 3–8 W sustained; tablet: 5–15 W; thin laptop: 10–25 W
• Thermal throttling — PMIC and thermal sensor feedback; frequency reduced when Tj > 85–95°C
• Workload spreading — PPEM (Performance-Per-Milliwatt) scheduling; distribute load across CPU, GPU, NPU to balance heat
• Sustained vs burst — Apple M4: 38 TOPS sustained, ~50 TOPS burst (10 s); throttling policy is OS-controlled
• Vapour chamber / heat pipe — Snapdragon X Elite laptop: required for 45W PC TDP

Energy Minimisation Techniques

• Zero-skipping — detect zero activations before MAC; save dynamic power proportional to sparsity
• Data compression — compress activations in SRAM to DRAM transfers; reduce bandwidth energy (α 0.5–1 pJ/bit saved)
• Near-threshold compute — sub-0.5 V operation for background inference; 10× energy reduction at 10× slower frequency
• Quantisation — INT4 vs FP16: 4× fewer bits moved → ~4× DRAM energy saving (dominant energy source)

Key Optimisation Passes

• Operator fusion — MatMul + GELU + residual add as single kernel; eliminates intermediate tensor writes
• Layout transformation — NCHW → NHWC → custom NPU-native tile layout; maximises SRAM reuse
• Constant folding — embed quantisation scales, bias, and fixed masks into weight tensors at compile time
• Tiling & scheduling — partition tensors to fit SRAM; schedule DMA ahead of MAC by Δt = DMA_latency
• Kernel selection — cuDNN / TensorRT chooses fastest kernel variant at runtime via auto-tuning

Inference Runtimes Compared

Constraints

Model Architectures

• MobileNetV3-Small — 1.5 MB, INT8, ~2 ms on Cortex-M55
• DS-CNN (keyword spotting) — 80 KB, 20 classes, 98% accuracy
• TinyBERT 4-layer — INT8, 14 MB — tight on 16 MB MCU
• MCUNet — co-designed for 256 KB SRAM; neural architecture search targeting MCU
• Binary / ternary nets — XNOR-Net: replace MAC with bitwise AND+popcount; 32–64× MAC reduction

Hardware & Frameworks

• Arm Cortex-M55 + Ethos-U55 — SIMD Helium, 128-bit SIMD; Ethos-U55: 64–512 MAC/cycle, <0.5 mW
• STM32N6 (ST) — Neural-ART NPU, 600 GOPS, Cortex-M55+Ethos-U55
• RP2350 — Cortex-M33 × 2 + RISC-V option; CMSIS-NN optimised kernels
• TFLite Micro — zero-dynamic-allocation inference; static memory arena
• Edge Impulse — cloud training → TFLM/EON deployment pipeline

Conv2D as GEMM — im2col Transform

• Input feature map patches unrolled into rows of an Toeplitz matrix (im2col)
• Kernel filters become columns of a weight matrix
• Conv2D becomes GEMM → fully exploits Tensor Core / SIMD hardware
• Cost: im2col buffer = K×K×C_in×H_out×W_out — can exceed 10× input size
• Implicit GEMM (cuDNN): computes im2col on-the-fly; no extra buffer needed

Winograd Convolution (F(2,3))

• Reduces multiplications: 3×3 conv from 9 to 4 MACs per output element
• Adds additions — beneficial when multiply is more expensive than add
• Used in cuDNN for 3×3 conv with FP16; INT8 Winograd less common (accumulation error)
• 2.25× throughput improvement for 3×3 conv in ideal conditions

Depthwise Separable Convolution

• Depthwise — one filter per channel; K×K×C_in MACs; very low compute density
• Pointwise (1×1) — mixes channels; C_in×C_out×H×W MACs; GEMM-friendly
• MobileNetV2: 8–9× fewer operations vs standard conv; backbone of most mobile CNNs
• Efficiency trap — depthwise layers are memory-bandwidth-bound, not compute-bound; high TOPS NPUs underutilised

Loop Tiling & Scheduling

• 7-deep loop nest (N, K, C, H, W, R, S) — tiling order determines cache reuse distance
• Output-stationary: accumulate partial output in registers; min register spill
• Row-stationary (Eyeriss): maximise reuse of both weight and activation rows simultaneously
• Halide, TVM, MLIR Linalg — compiler frameworks for automated tiling search

Static vs Continuous Batching

• Static batching — pad all sequences to max length; wait for full batch before starting; simple but wasteful
• Continuous (iteration-level) batching — insert new requests at token boundaries; no waiting; GPU utilisation 2–3× higher
• Chunked prefill — interleave prefill chunks with decode tokens; prevents decode starvation during long prompts

PagedAttention (vLLM)

• KV-cache fragmentation wastes 20–40% of GPU memory in naive implementations
• PagedAttention: KV stored in fixed-size pages (e.g. 16 tokens); pages non-contiguous in physical memory like OS virtual memory
• Copy-on-write for prefix sharing — same prompt prefix shared across many requests
• Enables 3× more concurrent requests at same GPU memory budget

Prefix Caching & Radix Attention

• Prefix caching — reuse KV-cache of repeated system prompts (chatbot context, RAG documents) across requests
• SGLang RadixAttention — LRU cache of KV blocks indexed by token sequence hash; hit rate >80% in multi-turn workloads
• Speculative decoding integration — draft and verify within same batch, sharing KV-cache pages

Why Distribute?

• Model doesn't fit one device — Llama 3.1 405B FP16: 810 GB; requires 10+ A100 80GB
• Latency requirements — distribute GEMM across devices → smaller per-device matrix → lower latency per layer
• Throughput scaling — additional devices handle more concurrent requests

Tensor Parallelism (Megatron-LM)

• Column-parallel: split weight matrix along output dimension; each GPU computes partial output columns
• Row-parallel: split along input dimension; each GPU holds partial input rows; all-reduce to combine
• MLP sharding: FFN first Linear column-parallel, second row-parallel — one all-reduce per transformer block
• Attention sharding: Q,K,V heads distributed across GPUs; each GPU handles subset of heads
• Communication: all-reduce per layer — NVLink 5 (1.8 TB/s) vs PCIe (64 GB/s) matters enormously

Pipeline Parallelism

• Transformer layers partitioned into stages; each GPU holds a stage
• GPipe micro-batching: split batch into micro-batches; pipeline flush eliminates bubble
• 1F1B interleaved (Megatron): interleave forward/backward across stages; ~1/N pipeline bubble
• Communication: point-to-point activation tensors between stages — lower volume than all-reduce but sequential
• Inference-only: no backward pass — pipeline bubble eliminated; pure latency / throughput balance

Expert Parallelism (MoE)

• MoE experts distributed across GPUs — each GPU owns subset of experts
• All-to-all dispatch: route tokens to correct GPU for expert execution — dominant communication cost
• DeepSpeed-MoE, Tutel — dynamic load-balancing to handle expert load imbalance
• Combined TP + EP for Mixtral: tensor-parallel within expert, expert-parallel across nodes

H100 SXM5 Specifications

H200 — Memory Upgrade

Hopper Inference Features

• Transformer Engine — FP8 Tensor Cores; dynamic per-layer precision selection; loss-free FP8 inference for most LLMs
• WGMMA (Warpgroup MMA) — asynchronous matrix multiply across warp groups; decouples data movement from compute
• TMA (Tensor Memory Accelerator) — hardware-managed multi-dimensional tile DMA; no software prefetch code needed
• MIG (Multi-Instance GPU) — partition H100 into up to 7 isolated inference instances; efficient for small-model serving
• Confidential Computing — TEE for sensitive inference workloads (medical, financial)

Inference Performance (Llama 3.1 70B, INT8)

AI 100 Ultra Specifications

Inference-First Architecture Choices

• No FP64 / FP32 training paths — die area focused entirely on INT8/INT4 MAC arrays and SRAM
• 128 MB on-chip SRAM — KV-cache tiers: hot tokens in SRAM, cold in LPDDR; reduces off-chip reads for short-context decode
• Multi-chip module — two dies per Ultra card; 2× die yield benefit from smaller die; NVLink-equivalent internal interconnect
• Compiler-driven scheduling — Qualcomm's AIC compiler statically schedules all DMA and compute; no dynamic dispatch overhead
• Network-attached inference — designed for rack-scale pod with PCIe fabric; flexible scaling from 1 to 64 cards

Competitive Positioning

Groq TS-P1 Specifications

Tensor Streaming Processor Principles

• No caches, no branch prediction, no out-of-order execution — compiler determines all data movement statically at compile time
• Deterministic execution — every instruction scheduled at compile time; zero runtime overhead
• Spatial dataflow — data streams through fixed functional units in programmed order; no dynamic dispatch
• 230 MB SRAM — holds model weights of 8B INT8 entirely on-chip; zero HBM stall for decode
• Compiler = hardware — Groq Compiler expresses the full execution plan; chip is a programmable dataflow engine

Trade-offs & Limitations

• Model size ceiling — 70B INT4: ~35 GB; requires 200+ chips in a GroqRack for on-chip storage
• No dynamic shapes — variable-length sequences require recompilation or padding
• Older process (14nm) — raw TOPS far below H100; wins purely on bandwidth-to-compute ratio
• Ideal for latency-critical serving — chatbots, auto-complete, real-time speech — batch=1 scenarios

Model Intellectual Property

• Weight encryption — AES-256 encrypted weights in Flash/DRAM; decrypt only within NPU boundary
• Secure boot — NPU firmware authenticated with hardware root of trust before any model load
• Secure Enclave partition — model weights never accessible to application CPU in plaintext
• Apple Neural Engine: model loading through CoreML only; weights never exposed to user process

Trusted Execution for Cloud Inference

• NVIDIA Confidential Computing (H100) — GPU Trusted Execution Environment; attestation of GPU firmware integrity
• AMD SEV-SNP + Instinct — encrypted VM memory; inference data encrypted end-to-end
• Remote attestation — cloud customer verifies model and runtime integrity before submitting sensitive inputs
• Use case: medical imaging, financial document inference, PII-containing prompts

Inference-Time Attacks

• Prompt injection — adversarial input manipulates model output; not hardware-level but system design concern
• Model extraction — repeated queries to reconstruct weights; mitigated by rate-limiting and output perturbation
• Side-channel timing — inference latency leaks token probabilities; fixed-time decode with dummy padding
• Data poisoning — fine-tuning attacks; hardware signing of model checkpoints provides provenance

Speculative Decoding — Deep Dive

• Algorithm: draft model generates K tokens autoregressively; target model verifies all K+1 in single forward pass (parallel GEMM)
• Acceptance probability: p_accepted = min(1, p_target / p_draft) per token via rejection sampling
• Speedup: K_accepted × (T_target / T_draft) — effective when draft model is 10–50× smaller
• Self-speculative (Medusa, Eagle): draft heads attached to target model; share hidden states
• EAGLE-3 (NeurIPS'25): feature-level speculative decoding with multi-layer fusion; 2–6.5× speedup over vanilla decode; now the standard spec-dec head in vLLM and SGLang. Medusa / EAGLE-2 remain common.
• Hardware implication: turns memory-bound GEMV decode into compute-bound GEMM verification — transforms the roofline operating point

State Space Models (Mamba / S4)

• Alternative to Transformer attention; recurrent state representation with linear-time decode
• Inference advantage: O(1) memory per token (fixed state vector vs O(N) KV-cache growth)
• Hardware implication: no KV-cache → no KV bandwidth bottleneck; fits tiny SRAM even at long context
• Mamba-2: structured state space; selective SSM — matches Transformer quality at 1B+ parameters
• Hybrid architectures (Jamba, Zamba): interleaved attention + SSM layers — best of both

Analog & Photonic Compute

• Analog in-memory compute (IBM PCM/ReRAM): GEMV at 85 TOPS/W; no weight movement → zero DRAM bandwidth energy
• Photonic MAC (Lightmatter Passage): optical matrix multiplication at speed-of-light; 0 dynamic power for MAC
• 3D SRAM-on-logic (TSMC SoIC): >100 TB/s on-die bandwidth — Groq-style architecture at advanced node

Core Principles

Emerging Directions

• FP4 / FP6 — Blackwell native FP4; model quality at sub-INT4 bitwidths approaching INT8 with careful calibration
• State Space Models — Mamba / Jamba eliminate KV-cache; O(1) decode memory enables extreme context lengths
• Speculative + medusa decoding — becoming standard in production serving stacks
• Analog / photonic compute — elimination of weight movement energy; 10–100× efficiency gains demonstrated in research
• Near-memory compute — HBM-PIM, 3D SRAM-on-logic: bring compute to data rather than data to compute
• On-device foundation models — 7B–13B running locally at <10 W is now engineered reality

Series Navigation