Memory System — VMSA, Caches & Ordering

• Cortex-M uses an MPU (region-based, no translation). Cortex-A uses a full MMU with paged virtual memory — the VMSA (Virtual Memory System Architecture).
• Every memory access goes through Stage-1 (OS) translation, and possibly Stage-2 (hypervisor) translation — that's up to 24 page-table walks for one access in the worst case.
• Memory is strongly weakly ordered — loads and stores can be reordered freely unless explicit barriers or release/acquire operations are used.
• The memory system is also a multi-core story: caches must be coherent across cores + GPU + NPU + DMA engines, using AXI/ACE or CHI.

• TTBR0_EL1 — user page-table root. Changed on every context_switch (tagged by ASID).
• TTBR1_EL1 — kernel page-table root. Stays the same for all processes.
• Effective VA width controlled by TCR_EL1.T0SZ / T1SZ — T0SZ=16 gives 48-bit user VA; T0SZ=12 gives 52-bit (LVA).
• If VA[63:48] is neither all-0 nor all-1, the CPU faults with a translation fault — the "canonical address" check.
• TBI (Top Byte Ignore) — allows bits [63:56] to carry a tag (used by MTE and HWASAN) without faulting.

• On a TLB miss the Hardware Table Walker (HTW) walks the in-memory page tables without any software help.
• Each descriptor is 8 bytes with a type (block / table / invalid) and PA + attribute bits.
• Walk is recursive: L0 → L1 → L2 → L3. Each step issues a memory read to the PA of the next table.
• A walk can itself miss in the cache and go to DRAM — expensive. Modern cores prefetch walks into a dedicated walker cache (sometimes called an "intermediate TLB" or "paging-structure cache").
• On Stage-1+Stage-2, each Stage-1 walk-step is itself Stage-2-translated — worst case 24 reads per VA translation.

// Simplified 4 KB, 48-bit VA walk (4 levels)

VA[47:39]  = L0 index (PGD)
VA[38:30]  = L1 index (PUD)
VA[29:21]  = L2 index (PMD)
VA[20:12]  = L3 index (PTE)
VA[11:0]   = page offset

PTE descriptor layout (simplified):
  [0]     valid
  [1]     table (1) or block/page (0)
  [11:2]  lower attrs (AP, SH, AttrIndx, AF, nG)
  [47:12] output address (PA bits)
  [63:51] upper attrs (PBHA, GP, DBM, Contig, PXN, UXN)

// AF = Access Flag: HW can set on access, or
// software-set (older cores). Armv8.1 adds HW update.

• ASID (Address Space ID) — 8- or 16-bit tag in every TLB entry for Stage-1 user translations. Comes from TTBR0_EL1[63:48].
• Lets the CPU switch user page tables without flushing the TLB — entries for the old process stay, tagged with its ASID.
• Kernel entries are global (nG=0) — not tagged by ASID, so they're visible from every process without duplication.
• VMID (VM ID) — similar concept at Stage-2, tags hypervisor guest translations. 8- or 16-bit. Set in VTTBR_EL2[63:48].
• Linux rolls its ASID allocator when all ASIDs used; rollover triggers a broadcast TLBI ASIDE1IS if ASID reassigned.

// TLBI maintenance ops (outer sharable, all ELs)

tlbi  vmalle1is          // all entries at EL1 (local ASID)
tlbi  aside1is, x0       // all entries matching ASID in x0
tlbi  vae1is, x1         // single VA (encoded in x1)
tlbi  vmalls12e1is       // all Stage-1+2 for this VMID
dsb   ish                // wait for it
isb                      // restart pipeline fetch

• L1-I / L1-D — split Harvard, per-core. Usually 32-64 KB, 2-4-way. VIPT tag (virtually-indexed, physically-tagged) — avoids synonyms.
• L2 — per core in modern cores (A76 onwards). 256 KB-3 MB. Inclusive or mostly-inclusive of L1.
• L3 (DSU) — shared by the whole DynamIQ cluster. Up to 16 MB. Managed by the DSU.
• SLC (System Level Cache) — attached to the CMN mesh/NoC. Sits on the path from cluster to DRAM. Shared with GPU, NPU, display.
• Line size is 64 bytes across the whole A-profile family.

• Point of Unification (PoU) — the level where I-cache and D-cache see the same data. Usually L2 or L3. Required for self-modifying code: after writing to memory, clean D-cache to PoU + invalidate I-cache to PoU.
• Point of Coherency (PoC) — the level where all coherent masters see the same data. Usually the SLC or just before DRAM. Required for DMA & non-coherent masters.
• Point of Persistence (PoPS) — Armv8.2 addition for non-volatile memory (NVDIMM, battery-backed). Not widely used on mobile.

// JIT compiler flushes after writing code
// (write-then-execute pattern)

bl    generate_code        // builds into [x0 .. x0+len]
mov   x2, x0               // save base

1:  dc    cvau, x2         // clean D to PoU
    add   x2, x2, #64
    cmp   x2, x3
    b.lo  1b
dsb   ish                  // wait for cleans

ic    ivau, x0             // invalidate I to PoU (by VA)
dsb   ish
isb                        // restart fetch

br    x0                   // now safe to execute

This sequence is what every JIT (V8, JSC, JVM) does on AArch64. Armv8.1+ cores can speed it up via CTR_EL0.DIC/IDC — if set, parts of the sequence are redundant.

• Arm A-profile is weakly ordered. The only ordering guaranteed is:
  • Same-address load→load (coherence)
  • Same-address write→read (SC-PO to same location)
  • Dependency ordering (address/data dep preserves program order for the dependent access)
• Everything else — load→store, store→store, unrelated load→load — can be reordered by hardware.
• Programmer recovers ordering with:
  • Barriers — DMB / DSB / ISB
  • Release/acquire — LDAR / STLR
  • LSE atomics (Armv8.1) with acquire/release semantics

• Before LSE, atomic ops were a LDXR / STXR loop — "Load-Linked / Store-Conditional". Under contention it can livelock.
• LSE — Large System Extension — adds single-instruction atomics that scale to many-core.
• Each takes acquire (A) / release (L) / acquire+release (AL) / plain variants — so ordering is baked in.
• Under the hood, big SMP systems implement LSE as a near-atomic sent to the Home Node (HN-F) in CHI — the cache-line's "owner" executes the RMW without the requester needing to own the line.
• Kernel compiles to LSE when present (-moutline-atomics produces a runtime picker).

// LSE atomics — all AArch64-only, Armv8.1-A+

cas    w0, w1, [x2]        // compare-and-swap
casal  w0, w1, [x2]        // CAS acquire+release
swp    w0, w1, [x2]        // SWP: atomic exchange

ldadd  w0, w1, [x2]        // atomic fetch-add
ldaxr  / ldxr             // legacy exclusive load

// Idiomatic acquire-semantics spinlock using CAS
acquire:
    mov   w3, #1
1:  mov   w1, wzr
    casa  w1, w3, [x0]     // CAS acquire: expect 0, set 1
    cbnz  w1, 1b           // retry until free
    ret

release:
    stlr  wzr, [x0]        // release-store clears
    ret

// Classic "publish data, then flag" pattern

// Producer (core A):
//   data is filled in, then published
str   x0, [data_ptr]       // 1: write data
stlr  w1, [flag]           // 2: release-store flag=1

// Consumer (core B):
//   spin on flag, then read data
1:  ldar  w2, [flag]       // acquire-load
    cbz   w2, 1b           // loop until non-zero
ldr   x3, [data_ptr]       // safe: sees the pre-STLR write

• The STLR in step 2 has release semantics: no prior store by this core may be reordered past it.
• The LDAR in step 1 of the consumer has acquire semantics: no subsequent load may be reordered before it.
• Together, the STLR-LDAR pair creates a happens-before edge — the consumer is guaranteed to see the producer's earlier STR x0, [data_ptr].
• Without release/acquire this program is racy on Arm — the consumer can read stale data. On x86 it happens to work because x86 is TSO.
• C++ memory_order_release / memory_order_acquire maps exactly to STLR / LDAR on AArch64.

• PRFM — hint to the caches; no architectural effect.
• Encoded target × type × locality:
  • Target: PLD (load) / PLI (instruction) / PST (store)
  • Type: L1 / L2 / L3
  • Locality: KEEP / STRM (streaming — hint cache it briefly)
• Hardware prefetchers also run in parallel — strided, stream, and increasingly Data-Dependent prefetchers in Cortex-X4+.
• In modern workloads (DB hashing, graph traversal) explicit PRFM still wins by 10-20% even with a good HW prefetcher.

// Linked-list walk with manual prefetch
// Classic cache-miss-hiding pattern

walk:
    ldr   x1, [x0, #NEXT]     // load next
    cbz   x1, 2f
    prfm  pldl1keep, [x1]     // prefetch next->next
    ldr   x2, [x0, #DATA]     // do the work
    bl    process_node
    mov   x0, x1
    b     walk
2:  ret

DC ZVA is effectively a "prefetch-for-write" — it zeroes a cache line in L1 without a fill from DRAM. Used in memset / calloc fast-paths.

• AMBA 03 — AXI Deep Dive — the bus that carries every cache-line refill.
• AMBA 04 — ACE & ACE-Lite — coherent transactions, MOESI on AXI.
• AMBA 05 — CHI Coherency — what the SLC and snoop filter look like at the mesh level.
• Cortex-M 04 — MPU & Memory — contrast with the region-based M-profile model.

• "Why tagged TLB with ASID?" → avoids flushing user entries on context switch; cut switch cost from hundreds of cycles to near zero.
• "What's the difference between PoU and PoC?" → PoU is where I and D caches see the same data (for self-modifying code); PoC is where everyone, including DMA masters, sees the same data.
• "Why does Linux use 4 KB granule mostly?" → x86 compat, small-page granularity for fine-grained mmap. Apple chose 16 KB for better TLB coverage and fewer page-table levels.

• "Why LSE atomics?" → LDXR/STXR livelocks under heavy contention; LSE sends a single RMW to the cache-line's home node, scaling to 64+ cores.
• "Explain a spinlock on AArch64" → CAS acquire to take, STLR to release, WFE while parked; avoids contended LL/SC livelock.
• "When is a DSB needed?" → after TLBI or IC/DC maintenance, to ensure it's globally observable before the ISB; rarely needed for plain data ordering (use release/acquire).
• "What is TBI?" → Top Byte Ignore — CPU ignores bits [63:56] of a VA on access. Enables MTE tags and HWASAN.