CXL in ML Accelerator SoCs

The Memory Wall

• Capacity: GPT-4-class models require >1 TB for KV-cache at scale; HBM3E tops out at 144 GB per stack
• Bandwidth: LLM decode is memory-bandwidth-bound — each token reads the entire KV-cache
• Cost: HBM at ~$10-15/GB vs DDR5 at ~$2-3/GB vs CXL-attached DDR5 at ~$3-4/GB
• Stranded memory: Fixed per-device DRAM leads to 30-40% memory waste in heterogeneous clusters

CXL's Value Proposition

• Coherent access to terabytes of shared DDR5 memory
• Cache-line granularity — no DMA descriptor overhead
• PCIe PHY reuse — leverages existing SerDes IP
• Memory pooling — dynamic allocation eliminates stranding
• Fabric scalability — CXL 3.x enables rack-scale shared memory

CXL.io

Equivalent to PCIe TLP. Used for device discovery, configuration, interrupts, DMA. Required by all device types.

CXL.cache

Device caches host memory with hardware coherency. 3 channels: D2H Request, D2H Data, H2D Response. Uses MESI protocol variant. Enables accelerators to cache host data without software intervention.

CXL.mem

Host processor accesses device-attached memory (DDR, HBM) with load/store semantics. 2 channels: M2S (master-to-subordinate), S2M (subordinate-to-master). 64-byte cache-line granularity.

ARB/MUX

Multiplexes CXL.io, CXL.cache, and CXL.mem flits onto shared PCIe lanes. Supports 68B (CXL 2.0) and 256B (CXL 3.0) flit modes.

Functions

• Device discovery: Standard PCIe enumeration — BIOS/OS sees CXL devices as PCIe endpoints with DVSEC capabilities
• MMIO & config space: BAR-mapped registers, CXL DVSEC (Designated Vendor-Specific Extended Capability) for CXL-specific negotiation
• Interrupts: MSI-X interrupt delivery for completion notifications, error reporting
• Mailbox: CXL mailbox interface for firmware commands — identify, get/set config, scan media, sanitize
• DMA: Bulk data movement for non-coherent transfers (firmware loading, telemetry extraction)

CXL.io Transaction Types

Boot Flow

1. PCIe PHY link training (LTSSM) — flex bus negotiation
2. CXL DVSEC discovery — determine device type (1/2/3)
3. CXL.cache/mem link initialization if applicable
4. HDM (Host-managed Device Memory) decoder setup
5. Memory range mapping visible to host CPU

Channel Architecture

Snoop Flow Example

Device reads host memory line (cold miss):

ML Relevance

Type 2 accelerators (GPUs, NPUs) use CXL.cache to coherently access host-resident weight tensors, page tables, and scheduling queues — avoiding explicit DMA copy overhead.

M2S / S2M Channels

Bias Modes

Latency Budget

HDM Decoders

Host-managed Device Memory (HDM) decoders map CXL memory into the host physical address space. BIOS/firmware configures base/limit/interleave across up to 16 targets.

Architecture

Bandwidth Characteristics

Products & Deployments

• Samsung CMM-D: CXL 2.0 DDR5 memory module, 128 GB, E1.S form factor, deployed in hyperscale memory tiering
• SK hynix CMS: CXL Memory Semiconductor — 96 GB CXL 2.0 DRAM, PCIe 5.0 x8, <200 ns latency
• Micron CZ120: CXL 2.0 memory expander, 256 GB, E3.S, targets 4 TB per socket via pooling
• Samsung CMM-H: CXL-attached HBM, combining high bandwidth with CXL flexibility

Why Type 2 for ML?

• Unified Virtual Memory: GPU/NPU and CPU share a single virtual address space — no explicit copies
• Coherent Host Access: Accelerator caches host-resident data (page tables, scheduling metadata) via CXL.cache
• Host-Visible Device Memory: CPU can directly read/write accelerator HBM/GDDR via CXL.mem
• Demand Paging: CXL.cache supports ATS (Address Translation Services) — enables GPU page faults resolved by host OS

Bias Mode for ML

Device Bias during compute-intensive phases (GEMM, attention) — device owns its memory, minimal snooping. Host Bias during data staging — CPU writes model weights directly into device memory.

Type 2 Architecture

Single-Level Switch (CXL 2.0)

Each host sees a Virtual CXL Switch (VCS) with dedicated downstream ports. Fabric Manager dynamically binds devices to hosts via MLD (Multi-Logical Device) or SLD (Single-Logical Device) model.

Multi-Level Switching (CXL 3.0+)

Fabric Manager (FM)

• Software entity that discovers, configures, and manages CXL fabric
• Binds/unbinds device LDs to host VCS via FM-API
• Monitors link health, QoS, and handles hot-plug events
• CXL 3.0: Port-Based Routing (PBR) eliminates ID-based routing overhead

Pooling Architecture

• Multi-Logical Device (MLD): A single physical Type 3 device partitioned into up to 16 Logical Devices (LDs), each assigned to a different host
• Dynamic Capacity Device (DCD): CXL 3.0 feature — host requests memory extents on demand. FM allocates from a shared pool in configurable chunk sizes (256 MB to 1 TB)
• Shared memory regions: CXL 3.0 allows multiple hosts to map the same physical memory — enables inter-host communication without network traversal

Pooling for ML Inference Clusters

Host A dynamically allocates 1.5 TB for KV-cache. Host B uses 800 GB. When Host C goes idle, its 700 GB returns to the pool. FM reallocates to Host A within microseconds — no data migration needed via DCD.

Back-Invalidation (BI)

Type 3 devices gain limited coherency: device can issue BISnp (Back-Invalidate Snoop) to host caches when another host writes to shared memory. Eliminates stale-data hazards in multi-host pooling without full CXL.cache complexity.

Global Fabric Attach (GFA)

Extends CXL beyond single-switch topologies. Enables multi-level, multi-switch fabrics where any host can reach any device across the fabric. Uses Port-Based Routing (PBR) — switches route by physical port rather than device ID, enabling O(1) lookup latency in switch forwarding tables.

256-Byte Flits

Doubles flit size from CXL 2.0's 68B. Reduces header-to-payload ratio, enables carrying 4x 64B cache lines per flit. Combined with PCIe 6.0, delivers 128 GB/s per direction on x16 link.

CXL 3.1 Enhancements

• TSP (TEE Security Protocol): Hardware-enforced encryption for CXL.mem traffic — essential for multi-tenant ML inference clusters
• Unified I/O (UIO): Extends CXL coherency to non-PCIe fabrics (e.g., AMBA CHI bridges)
• Extended PBR: Supports up to 4096 ports per fabric, enabling warehouse-scale CXL deployments
• Per-device QoS: Fine-grained bandwidth/latency SLAs enforced at the switch level

The KV-Cache Problem

• LLM inference stores key/value tensors for every token in every layer
• Llama-3.1-405B @ 128K ctx: ~161 GB KV-cache per sequence at FP16 (GQA=8, 126 layers, 16,384 hidden)
• DeepSeek-V3 @ 128K ctx (MLA): ~8 GB per sequence — ~20× smaller than MHA-equivalent
• Llama-3-405B: ~160 GB KV-cache at 128k tokens, FP16
• Batching 64 concurrent requests: 3-10 TB KV-cache — far exceeds HBM capacity

CXL Tiered Memory Approach

Latency Analysis

Research results (Meta, 2024): CXL-tiered KV-cache achieves <5% throughput degradation vs all-HBM at 4x the context length capacity.

The Bandwidth Challenge

LLM decode is fundamentally memory-bandwidth-bound: each generated token reads all model weights once. For Llama-3-70B at FP16 (140 GB weights), generating at 30 tokens/s requires 4.2 TB/s — only achievable with 4+ HBM3E stacks.

CXL for Capacity, Not Bandwidth

• CXL x16 PCIe 5.0: ~64 GB/s — insufficient alone for weight streaming
• But ideal for capacity-bound workloads: embeddings, KV-cache, activation checkpoints
• Multiple CXL links aggregate bandwidth: 4x CXL x16 = ~256 GB/s
• CXL 3.0 x16 (PCIe 6.0): ~128 GB/s — approaching DDR5 channel bandwidth

Pooled Model Sharding

Each server keeps its pipeline-parallel shard in HBM for fast GEMM execution. The shared CXL pool holds the complete model copy for failover, plus the distributed KV-cache that both servers read during cross-shard attention.

CXL vs RDMA for ML

SoC Integration Architecture

Key Design Considerations

• PHY IP: CXL reuses PCIe PHY — Synopsys, Cadence, Alphawave offer CXL-ready PHY IP at 32 GT/s (Gen5) and 64 GT/s (Gen6). Area: ~5-8 mm² per x16 PHY at 5nm.
• Controller IP: CXL controller manages flit packing, credit flow, and coherency state machines. ~1-2 mm² area. Vendors: Synopsys, Cadence, Rambus, Montage.
• CHI-CXL Bridge: Translates on-chip AMBA CHI coherency transactions to CXL.cache/mem messages. Critical for ARM-based ML SoCs (e.g., Grace, Neoverse-based designs).
• Latency budget:

Specifications

• CXL 3.2 (Dec 2024) / CXL 4.0 (Nov 2025) Specifications — CXL Consortium. 4.0 runs on PCIe 7.0 PHY (128 GT/s). computeexpresslink.org
• CXL 3.0 Specification — CXL Consortium (2022). 256B flits, GFA, multi-level switching.
• CXL 2.0 Specification — CXL Consortium (2022). Single-level switching, memory pooling, FM-API.
• PCIe 6.0 Specification — PCI-SIG (2022). 64 GT/s PAM4 PHY underpinning CXL 3.x.

Research Papers

• Pond: CXL-based memory pooling systems for cloud — ASPLOS 2023
• TPP: Transparent page placement for CXL heterogeneous memory — ASPLOS 2023
• CXL-ANNS: Software-hardware collaborative memory disaggregation — USENIX ATC 2023
• Demystifying CXL memory with genuine devices and software — MICRO 2023

Industry Resources

• CXL Consortium — computeexpresslink.org — specifications, white papers, member list
• Linux CXL Subsystem — kernel docs: Documentation/driver-api/cxl/ — cxl_pci, cxl_mem, cxl_port drivers
• QEMU CXL Emulation — Type 3 device emulation for kernel/driver development
• Astera Labs — Leo CXL smart memory controller, Aries CXL retimer
• Montage Technology — CXL controller IP and memory buffer chips

Key Takeaways

Modern SoC Design Series · Presentation 08 · CXL in ML Accelerator SoCs