SMMU — the System MMU

• Every device that does DMA (PCIe NIC, NVMe, GPU, display, video codec, DMA engine) bypasses the CPU's MMU entirely.
• Without an IOMMU, a device can write to any physical address — a bug or exploit in the driver turns into arbitrary-kernel-memory-write.
• The SMMU gives each device its own virtual address space and permissions — same semantic benefits as the CPU MMU, but for I/O.
• It also enables:
  • VFIO / SR-IOV pass-through — safely give a VM direct PCIe device access
  • Shared virtual memory (SVM) — the CPU and GPU/NPU use the same VA
  • PCIe ATS / PRI — device caches its own translations, traps on miss

• Every DMA transaction arrives at the SMMU with:
  • An AXI transaction with address, burst, ID
  • An AXI user side-band carrying StreamID (up to 32 bits) — identifies the source device
  • Optionally SubStreamID (aka PASID, up to 20 bits) — identifies the address space within a device
• SMMU indexes Stream Table with StreamID:
  • Entry contains a pointer to a Context Descriptor (CD), or CD table if SubStreamID is used
  • CD has TTBR0/1, TCR, MAIR — exactly like CPU MMU
• For virtualization, Stream Table Entry also points to a VTTBR (Stage-2).

// Flow of a PCIe DMA through SMMU-v3

PCIe device   ->  AXI transaction
  + StreamID (PCIe Requester ID)
  + PASID    (if present)
      |
      v
SMMU reads Stream Table[StreamID]
      |
      v
STE -> Context Descriptor Table[PASID]
      |
      v
CD -> TTBR, TCR, MAIR
      |
      v
Hardware Table Walker (Stage 1)
   -> each walk step further through
      Stage 2 if VM-owned
      |
      v
Output PA -> mesh / HN-F

• Stage 1: Virtual Address (from the device, in its address space) → Intermediate Physical Address (IPA).
• Stage 2: IPA (what the device/guest sees as "physical") → actual Physical Address (PA).
• Native-OS setup: Stage 1 only. Device sees kernel-managed VA space.
• Guest pass-through setup: Stage 1 (guest VM owns) + Stage 2 (hypervisor owns). Guest has full control of its device's VA without compromising host isolation.
• Same 4/16/64 KB granules as CPU MMU. Stage 2 can use coarse granularity (typically 1 GB blocks) for large VM memory regions.
• Translation tables in DRAM; SMMU has IOTLB + walk cache to accelerate repeat hits.

• ATS (Address Translation Service) — PCIe feature where the device has its own TLB cache. Device sends an ATS Translation Request over PCIe; SMMU responds with the translation; device caches it.
• Subsequent DMAs from that device include a Translated bit in the TLP prefix — SMMU treats them as already-translated.
• Reduces SMMU load dramatically for high-throughput devices (100 GbE NICs, NVMe).
• PRI (Page Request Interface) — device can request a page that isn't mapped yet. SMMU queues a page-request event; host kernel faults the page in; SMMU notifies device; device retries.
• Together ATS + PRI enable full demand-paged device access — no pinning, no pre-allocation. Essential for SVM (GPU compute sharing CPU memory).

// SVM: GPU and CPU share the same VA

// 1. User mallocs a large buffer
float *data = malloc(1 GiB);

// 2. Open the accelerator, bind to current mm
int fd = open("/dev/gpu0", O_RDWR);
bind_pasid(fd, getpid());

// 3. Submit a kernel, passing `data`
submit_kernel(fd, data, N);

// Under the hood, the GPU uses ATS to translate
// & PRI to fault in pages on demand. No pinning.
// glibc malloc works: swap + demand-paging just
// as with CPU access.

• SMMU-v3 moved control from MMIO registers to DRAM-resident queues — same pattern as NVMe.
• Command Queue (CMDQ) — CPU writes commands; SMMU consumes. Commands include:
  • CMD_CFGI_* — invalidate stream-table / context entries
  • CMD_TLBI_* — invalidate IOTLB entries
  • CMD_ATC_INV — invalidate device-side ATS TLB
  • CMD_SYNC — barrier; needs to be observed
• Event Queue (EVTQ) — SMMU posts faults/events; CPU drains it.
• PRI Queue (PRIQ) — page-request events, optional.
• SMMU MSI-signals the CPU via GIC-ITS when a queue has content — same pattern as NVMe.

• TBU — Translation Buffer Unit — sits close to each master on the mesh. Holds a fast local IOTLB (64-256 entries), services hits inline.
• TCU — Translation Control Unit — central, handles:
  • Stream-table lookups
  • Page-table walks
  • Command/event queue processing
  • PRI responses
• TBUs cache TCU translations; TBU misses query the TCU.
• This split lets the SMMU scale bandwidth with the number of masters.

• IORT (I/O Remapping Table) — ACPI table describing:
  • PCIe root complexes & segments
  • SMMU instances & base addresses
  • Stream ID mapping ranges
  • Named Components (on-chip DMA masters not behind PCIe)
• Linux parses IORT at boot, creates struct iommu_ops, binds each PCIe / on-chip device to its SMMU + StreamID.
• Device tree equivalent: iommus = <&smmu StreamID> property on each device node.

// IORT (ACPI) conceptual structure

Node 0: SMMU-v3 @ 0x4f000000
  event IRQ: 42
  cmdq IRQ:  43
  PRIQ IRQ:  44

Node 1: PCIe Root Complex (seg 0)
  PCI bus range 0-255
  ID mapping: PCI RID [0..65535] ->
              StreamID [0..65535] -> Node 0 (SMMU)

Node 2: Named Component "nvme@..."
  ID mapping: -> StreamID 0x1234 -> Node 0 (SMMU)

An OS that doesn't understand IORT will still boot — devices just won't have IOMMU isolation (and therefore VFIO won't work). Distros require IORT for PCIe passthrough on Arm.

• Core drivers: drivers/iommu/arm/arm-smmu-v3/ (v3), drivers/iommu/arm/arm-smmu/ (v2).
• Implements the Linux iommu_ops — attach/detach, map/unmap, TLB flush.
• VFIO stack:
  • vfio-pci claims the PCIe device
  • vfio_iommu_type1 provides per-container DMA mapping API
  • Qemu uses VFIO fd to set up guest DMA mappings via SMMU Stage 1
  • KVM maps guest memory via Stage 2 (VMID-tagged)
• SVM stack: /dev/iommufd (kernel 6.4+) + SVA API for PASID-based sharing.

// PCIe passthrough to VM, at a glance

// 1. Bind device to vfio-pci
echo "vfio-pci" > /sys/bus/pci/devices/.../driver_override
echo <pci_id> > /sys/bus/pci/drivers/vfio-pci/bind

// 2. Launch qemu
qemu-system-aarch64 \
  -machine virt,gic-version=3,iommu=smmuv3 \
  -cpu max \
  -device vfio-pci,host=0000:01:00.0 ...

// Under the hood:
//   SMMU stream table entry for device => CD of VM
//   Stage 2 = VM's IPA -> host PA
//   Guest owns Stage 1 entirely

• Per-device IOTLB hit rate is the #1 factor for high-throughput I/O. A 100 GbE NIC sees millions of distinct DMA buffers.
• Large-page DMA buffers (2 MB huge pages) dramatically reduce translation load — fewer IOTLB entries needed.
• Broadcast TLBIs from Linux cost CMDQ bandwidth — batch where possible.
• Linux iommu.strict=0 (non-strict mode) defers invalidation until a flush point. Higher perf but tiny freshness window.
• SMMU PMUs (SMMU-600/700 expose them) track: stream-table misses, walk-cache misses, translation latency, PRI events. Essential for debugging DMA-bandwidth-limited workloads.

• Hyperscalers run VFIO passthrough for GPU, NIC, NVMe — tenant VM gets bare-metal-level device speed.
• Stage-2 owned by KVM; Stage-1 owned by the guest.
• For a VM to sub-allocate the PCIe device to its own processes, the SMMU needs PASID support + the device must support SR-IOV VFs or SIOV.
• vIOMMU — qemu can expose a virtual SMMU to the guest so the guest kernel "sees" an SMMU and can do its own isolation for nested device use.
• With CCA / Realms, SMMU needs to honour Granule Protection Checks on its DMAs — a Realm VM's DMA target must be a Realm-owned granule.

• dmesg | grep smmu — probe + fault messages.
• SMMU fault format (SMMU-v3): Event type (C_BAD_STE, F_TRANSLATION, F_PERMISSION, F_ADDR_SIZE), StreamID, IOVA, Reason.
• Example:
  arm-smmu-v3 arm-smmu-v3.0.auto: event 0x10 received:
  0x000000007B2004C0 0x0000020000000000
  0x00000000FFFFFFF0 0x00000000FFFFFFF8
  → translation fault @ IOVA 0xFFFFFFF0, StreamID 0x2000
• Common causes: device DMA before map, use-after-unmap, page permission mismatch, writing past allocation.
• ATS failures show up with F_TRANSL_FORBIDDEN.

• "What problem does an SMMU solve?" → DMA isolation (device can't write arbitrary memory), virtualization (pass-through devices to VMs safely), shared virtual memory (GPU/NPU share CPU VAs).
• "What's a StreamID?" → identifier the bus carries with every DMA to uniquely identify the source device. For PCIe, typically the Requester-ID. SMMU uses it to index the stream table.
• "SMMU-v2 vs SMMU-v3?" → v2 = MMIO-register + context-bank; limited to 128 streams. v3 = DRAM tables + command queue; 1M+ streams, full virt + PASID + PRI.

• "What is ATS?" → PCIe Address Translation Service; device has its own TLB, queries SMMU for translations, tags subsequent DMAs as pre-translated. Massive throughput win.
• "What's a TBU?" → Translation Buffer Unit — per-master local IOTLB inside the SMMU, close to the master on the mesh. Misses go to the central TCU.
• "Why do we need IORT?" → ACPI table mapping PCIe Requester IDs → StreamIDs → SMMU instances. Without it Linux can't wire up device ↔ SMMU relationships on a server.