ACE & ACE-Lite

• By 2010, multi-core Arm SoCs (Cortex-A9 quad) had reached the point where software-managed cache maintenance was no longer acceptable.
• The Linux kernel needed hardware cache coherency for SMP scheduling and page-table updates to work reasonably.
• big.LITTLE (Cortex-A7 + A15, 2011) made it unavoidable — the OS had to migrate threads between clusters with live cache state.
• Arm needed a bus protocol that could carry coherency primitives. Bolting them onto AXI gave ACE — AXI Coherency Extensions — in AMBA 4 (2010).

• Classical cache-coherency states, named after the five-state protocol popularised by AMD / SPARC:
  • Modified — dirty, exclusive, must write back
  • Owned — dirty, shared (this cache responsible for write-back)
  • Exclusive — clean, only copy
  • Shared — clean, may be copies elsewhere
  • Invalid — not valid
• ACE uses a variant with five equivalent states but renamed to fit AXI's semantics — UniqueClean / UniqueDirty / SharedClean / SharedDirty / Invalid.

The MOESI "O" (Owned) state — the one responsible for supplying clean copies to peers and eventually writing back — is SharedDirty in ACE-speak.

Key invariants

• At most one cache can be Unique (UC or UD) for a given line.
• At most one cache can be SharedDirty for a given line.
• Multiple caches can be SharedClean simultaneously.

What's "Dirty"?

A cache line is Dirty if the data in it differs from the underlying memory's value. The responsibility to eventually write-back is always with exactly one cache (UD or SD).

• On top of the five AXI channels (AR, R, AW, W, B), ACE adds three snoop channels:
  • AC — Snoop Address (interconnect → cache)
  • CR — Snoop Response (cache → interconnect)
  • CD — Snoop Data (cache → interconnect, optional)
• When a new request arrives at the interconnect, the Home logic broadcasts an AC snoop to all peer caches; each responds with CR (state + hit/miss), and if it has modified data, drives CD.
• Plus a DVM logical channel (Distributed Virtual Memory) — piggybacks on the snoop channels for TLB invalidation, I-cache maintenance, and barrier messages.

• When a ReadShared / ReadUnique / CleanUnique arrives at the interconnect, it broadcasts a snoop to every peer cache on their AC channel.
• Snoop types mirror the request semantics:
  • ReadOnce / ReadClean / ReadNotSharedDirty / ReadShared / ReadUnique — peer must supply data if present, and transition its state accordingly.
  • CleanInvalid — peer must write back dirty data, then invalidate.
  • MakeInvalid — peer drops the line without writing back (used when the requester intends to overwrite the whole line).
• Peer responds with a 5-bit CR[4:0] encoding hit/miss, passed-dirty, passed-data, error.

AxDOMAIN[1:0]

Controls the scope of snooping:

• 00 Non-shareable — no snoop needed
• 01 Inner Shareable — snoop within the Inner domain (typically a cluster)
• 10 Outer Shareable — snoop across the outer domain (multiple clusters, system cache)
• 11 System — snoop to memory system

AxSNOOP[3:0]

Encodes the specific transaction type (ReadOnce / ReadShared / CleanUnique / etc.). Combined with AxDOMAIN, the interconnect knows exactly who to snoop and how.

• Most peripherals aren't caches — they're producers/consumers of memory. They don't need to hold cache state, but they benefit enormously from the CPU's caches being coherent with their traffic.
• ACE-Lite is a restricted ACE profile for such agents. Signals from the master side:
  • AXI + AxDOMAIN + AxSNOOP (so the interconnect can snoop CPU caches).
  • No AC / CR / CD channels on the master (it has no caches to snoop).
• The master reads from / writes to memory, and the interconnect automatically snoops CPU caches to return fresh data or invalidate stale copies.
• This is the coherent DMA / GPU / NIC / NPU interface.

• Cache coherency is only half of SMP correctness — the other half is TLB coherency.
• When one CPU invalidates a page-table entry (e.g. page removed, permission changed), every other CPU's TLBs must drop the corresponding entries before the OS can reuse the page.
• DVM messages travel on the ACE snoop channels (AC, with a special SNOOP type) and carry:
  • TLB Invalidate (TLBI) — by VA, by ASID, by all
  • Branch Predictor Invalidate (BPI)
  • Instruction-cache Invalidate (ICI)
  • Sync — pipelined barrier
• Every receiver must acknowledge; a DVM Sync waits for all acks before proceeding.

CCI = Cache Coherent Interconnect. Arm's off-the-shelf IP used in virtually every big.LITTLE and LITTLE-only SoC between 2011 and 2016.

• The snoop filter is a directory: for each cache line currently cached anywhere in the system, track which masters hold it.
• Typically implemented as a set-associative tag structure, inclusive of the caches it tracks.
• On a coherent request:
  • Lookup the address in the filter.
  • If no master holds it → skip snoop entirely, go direct to memory.
  • If one or more masters hold it → snoop only those.
• Filter capacity must be ≥ the sum of tracked cache capacities (or suffer back-invalidates to the slowest cache when the filter overflows).

• A CPU accessing a stack variable marks it Inner Shareable — only its own cluster needs to snoop.
• A kernel data structure shared across clusters is marked Outer Shareable — all clusters + all ACE-Lite masters snoop.
• DMA-mapped network buffer: typically Outer Shareable if the CPU and the NIC genuinely share it; Non-shareable if the driver manages cache maintenance manually.
• This is why Armv8 OS kernels tune page table attributes carefully — the wrong shareability domain can be a 10% performance bug.

• ACE5 (2017) brought AXI5's improvements to the coherent profile:
  • Atomic transactions — same offload as AXI5, but can participate in coherency (other caches get snooped accordingly).
  • Cache stashing — an ACE master can hint that a cache line be installed into a target cache's state (UC/SC).
  • Cache Maintenance Operations (CMO) — system-wide cache clean/invalidate over the bus rather than CPU-executed loops.
  • MTE support — tag bits on every coherent transaction.
  • RME hooks — AxNSE for 4-way security states (Root/Realm/Secure/Non-Secure).

• ACE's basic model is snoop-everyone. At 2 clusters → 1 peer to snoop. At 8 clusters → 7 peers per miss.
• Snoop filters amortise but don't eliminate the fan-out.
• The AC / CR / CD channels are signal-level buses — every master port has them physically. For 32+ masters, the wire count becomes unbearable on a reticle-sized die.
• Beyond ~8 coherent masters, the protocol's point-to-point wiring assumption runs out.
• Solution: move to a packet-based protocol where any number of nodes can share the same physical transport (mesh NoC). That protocol is CHI (next deck).

ACE systems that shipped big

• Samsung Exynos 5 Octa (2013) — Cortex-A15 + A7 big.LITTLE over CCI-400.
• Qualcomm Snapdragon 820 (2016) — Kryo clusters over CCI-550.
• Mediatek Helio X10/X20 (2015–16) — deca-core designs via CCI-500.
• Apple A6 → A11 — bespoke interconnects with ACE interfaces (though Apple runs custom cores).
• Amlogic, Allwinner, RockChip — countless set-top / automotive SoCs.

ACE-Lite on the periphery

• Mali GPUs (T6xx, T7xx, G-series) — ACE-Lite for coherent buffer sharing with the CPU.
• Video decoders & ISPs — ACE-Lite so decoded frames are visible to CPU caches.
• Cortex-R coherent peers in R82 (2020) — real-time cores joining an A-profile coherency domain.
• Arm SMMU MMU-600/MMU-700 — participates in DVM on behalf of I/O devices.

The cheapest way to be "coherent"

• An ACE-Lite master has no snoop channels. Physically it's just an AXI4 master with a few extra signals (AxDOMAIN, AxSNOOP, AxBAR).
• If all you need is I/O coherency ("CPU cache sees my writes"), you can bolt ACE-Lite onto an existing AXI master by driving the new attributes to sensible constants — zero extra logic.
• Interconnect (CCI) sees those constants and snoops the CPU on behalf of the master.

// Thin ACE-Lite wrapper around an AXI4 master.
// No snoop channels exist on the master side.
module acel_wrap #(parameter ID_W = 4) (
  // existing AXI4 port (connect to your master IP)
  output logic [ID_W-1:0] ARID,
  output logic [31:0]     ARADDR,
  output logic [7:0]      ARLEN,
  output logic [2:0]      ARSIZE,
  output logic [1:0]      ARBURST,
  output logic            ARVALID,
  input  logic            ARREADY,
  // ... (the rest of AR/R/AW/W/B as usual)

  // ACE-Lite extra attributes — pure combinational tie-offs
  output logic [1:0]      ARDOMAIN,
  output logic [3:0]      ARSNOOP,
  output logic [1:0]      ARBAR,
  output logic [1:0]      AWDOMAIN,
  output logic [2:0]      AWSNOOP,
  output logic [1:0]      AWBAR
);
  // I/O-coherent DMA targeting shared memory:
  //   Outer Shareable, ReadOnce / WriteUnique,
  //   No barriers.
  assign ARDOMAIN = 2'b10;    // Outer Shareable
  assign ARSNOOP  = 4'b0000;  // ReadOnce
  assign ARBAR    = 2'b00;    // No barrier

  assign AWDOMAIN = 2'b10;    // Outer Shareable
  assign AWSNOOP  = 3'b001;   // WriteUnique
  assign AWBAR    = 2'b00;
endmodule

Six assigns. That's it. Total new gates for coherent DMA = 0 — the CCI / CMN does all the work.

• "What are the five ACE cache states?" → UC, UD, SC, SD, I. They map to E, M, S, O, I in MOESI.
• "What's the difference between ReadShared and ReadUnique?" → ReadShared accepts shared copies elsewhere; ReadUnique invalidates all peers so the requester can modify.
• "What are the three snoop channels?" → AC (snoop address, interconnect → cache), CR (snoop response), CD (snoop data).
• "Why ACE-Lite for GPUs?" → GPUs are producers/consumers of memory, not holders of cache state. ACE-Lite lets them enter the coherency domain one-way, without exposing unused snoop channels.

• "What is DVM?" → Distributed Virtual Memory — TLB/I-cache maintenance messages piggybacked on the ACE snoop channels so SMP software doesn't need cross-CPU IPIs for every TLBI.
• "Why does a snoop filter matter?" → It turns a broadcast snoop into a targeted one. At 8+ masters, broadcasting is 10× more expensive than filtering.
• "Why doesn't ACE scale to 32 coherent masters?" → Point-to-point signal-level wiring; the AC/CR/CD channels must physically reach every master port. Solved by moving to packet-based CHI.
• "What did ACE5 add?" → Atomics, cache stashing, CMO, MTE tags, RME hooks.