CMN Mesh Interconnect

• Up to ~16 cores, a single shared coherent bus (AMBA ACE) works.
• Beyond ~16 cores the bus becomes the bottleneck: every snoop broadcasts to every core.
• Arm's answer (2017): a 2-D mesh with CHI (Coherent Hub Interface) transport. Each tile routes packets independently; no shared bus.
• Snoops become directory-filtered — only cores known to hold a line are snooped.
• Mesh is what makes 128-192 cores per socket feasible — AMBA 4 ACE topped out around 16 coherent cores.

• The SLC (System Level Cache) is distributed across HN-F nodes. Each HN-F holds a slice.
• Address hash picks the HN-F: usually hash(PA[47:6]) → HN-F index. Sometimes the hash is interleaved by cache line to spread bandwidth; sometimes by page to improve locality.
• Typical SLC slice: 2-8 MB per HN-F. A 32-HN-F mesh with 8 MB slices = 256 MB SLC.
• Neoverse V2 Grace: 117 MB SLC across the CMN-700 mesh.
• Graviton 3: ~64 MB. Graviton 4: 192 MB.
• Slice selection is normally fixed in hardware; MPAM partitioning adjusts how much of each slice a tenant can use.

• DMT — Direct Memory Transfer. When HN-F sees that its SLC doesn't have the line and it's going to DRAM, it tells the SN-F (memory controller) to send the data directly to the requester, bypassing the HN-F.
• DCT — Direct Cache Transfer. When another RN-F holds the line, HN-F arranges for that RN-F to send data directly to the requester, not through HN-F.
• Both save a hop + buffer at the HN-F. Big effect on latency-sensitive DB/KV workloads (typical L3-miss latency drops ~20 cycles).
• CMN-700 adds DVMv9 snoop filtering — TLB invalidates (DVM messages) are filtered by scoped VMID / ASID, avoiding bus-wide broadcasts.

• Launched in 2018. Up to 8×8 mesh = 64 nodes, 128 coherent RN-Fs.
• CHI-A transport. Up to 1 TB/s aggregate bandwidth on a single mesh.
• Supports cache stashing, MPAM v0, RAS framework.
• Shipping in: AWS Graviton 2, Ampere Altra, eMAG, Huawei Kunpeng 920.
• Limitation: no native PCIe Gen 4/5, no CXL. Had to use external NICs.

CMN-650 (2020)

• Mid-step refresh. CHI-B transport.
• Added CCIX 1.1 support (coherent accelerator attach over PCIe).
• First hooks for CXL 1.1.

• Up to 12×12 mesh = 144 nodes, up to 256 RN-Fs.
• CHI-E transport. Higher radix per switch, wider data paths (up to 512-bit).
• CXL 2.0 support via CXS-to-CXL bridge; allows memory pooling.
• MPAM v1 mandatory — QoS on bandwidth + cache partitioning.
• DVM filtering — targeted TLB invalidates rather than broadcast.
• Introduced slice hashing by locality as an option — reduces mesh traversal on sequential streams.
• Shipping in: Microsoft Cobalt 100, Alibaba Yitian 710, Graviton 4, NVIDIA Grace.

• Launched at Arm Neoverse Tech Day 2024. Paired with N3 / V3 + CSS.
• Key shifts vs CMN-700:
  • CHI-E2 transport — wider, higher-frequency data paths
  • Unified on-die + chiplet topology — the mesh spans multiple chiplets over UCIe, presenting a single coherence domain
  • CXL 3.0 — type 1/2/3 devices, multi-host, switchable fabric
  • CCA / Realms — Granule Protection Checks happen at HN-F
  • Higher HN-F radix; better SLC access latency at large mesh sizes
• Target silicon: AWS Graviton 5 (reported), Microsoft Cobalt 200, multi-chiplet HPC/AI servers.

• Each memory channel has an SN-F node on the mesh.
• SN-F nodes map to Arm memory-controller IP:
  • DMC-520 — LPDDR4 / DDR4, 32-bit
  • DMC-620 — DDR4, 64-bit, CMN-600 era
  • DMC-750 — DDR5 + RDIMM/LRDIMM, CMN-700 era
  • DMC-1000 (2024) — DDR5-6400, CXL 2.0, ECC-chipkill
• Additional SN-F variants implement HBM (Grace + 480 GB LPDDR5X is an exception), CXL.mem (pooled memory), and Flash/NVDIMM.
• Memory scrubbing happens at SN-F — patrol + on-demand scrubbers catch soft-error rates typical of large-DRAM server deployments.

• An RN-I (I/O Request Node) sits on the mesh for every I/O domain. It bridges non-coherent AXI/CXL/PCIe traffic into coherent CHI.
• Each RN-I is preceded by an SMMU-v3 that:
  • Translates I/O virtual addresses → PA (stage 1)
  • Enforces VM isolation (stage 2)
  • Checks PCIe STE/CD entries
  • Supports PCIe ATS + PRI (Address Translation Service, Page Request Interface)
• PCIe Gen 5 / 6 controllers typically integrated on the mesh edge with dedicated RN-I nodes.
• CXL devices can present as RN-F (type-1 accelerator, with its own cache) or SN-F (type-3 memory expander).

• SLC Cache Stashing — NIC/NVMe can stash packet data into the LLC before the CPU reads it.
• QoS classes — 16 QoS levels per REQ; MPAM-mapped.
• Dynamic snoop filter expansion — HN-Fs can switch between inclusive / partial-inclusive snoop filtering based on workload pressure.
• Ordering domains — REQ ordering can be relaxed per address space for bandwidth (used for accelerator + GPU traffic).
• Flow-control credits per VC — prevents head-of-line blocking between REQ/SNP/DAT.
• Error containment — RAS framework routes poison to the node best placed to handle it (usually HN-F or memory controller).

• "Why is HN-F the key node?" → owns the snoop-filter directory + a slice of SLC. Every coherent read/write to its address range goes there. Scales to >128 cores because snoops are targeted, not broadcast.
• "Explain DMT vs DCT" → DMT = HN-F tells SN-F to forward DRAM data directly to requester (skip HN-F buffer). DCT = HN-F tells another RN-F to forward cached data directly. Both cut L3-miss latency by ~20 cycles.

• "How does CXL.mem integrate?" → external memory expanders appear as SN-F nodes. HN-Fs own the CXL address ranges and enforce coherence. Software sees a normal RAM region with slightly higher latency.
• "What's MPAM?" → partition ID (PartID) tagged on every transaction; HN-F enforces cache-way + bandwidth limits per PartID. Used for cloud multi-tenancy QoS.
• "Why CHI and not AXI?" → AXI-ACE topped out at ~16 coherent nodes because snoops were broadcast. CHI is a packet-switched protocol that scales with the mesh.