MPAM & CoreSight

MPAM — Partitioning & Monitoring

• Carves the memory system (caches, bandwidth) into partitions.
• Each partition tagged with a PartID; limits + counters enforced at every MPAM-aware resource (L2, L3, SLC, memory controller).
• Why: cloud multi-tenant QoS, automotive freedom-from-interference, mixed-criticality workloads.

CoreSight — Debug & Trace

• The fabric that carries JTAG / SWD, program trace, data trace, and cross-trigger events across an Arm SoC.
• Covers DAP (Debug Access Port), ETM (trace), ITM (instrumentation), STM (system), CTI (cross-trigger), TMC/TPIU (sinks).
• Why: post-mortem debug, performance analysis, live system tracing, security audits.

• Multi-tenant cloud: "noisy neighbour" VMs can thrash the LLC or saturate memory bandwidth, starving a better-paying tenant's SLA.
• Mixed-criticality automotive: a safety-critical thread must not be affected by an infotainment workload running on other cores.
• HPC: bin-packed workloads need cache + bandwidth isolation for reproducibility.
• x86 has had the equivalent since 2016:
  • Intel CAT (Cache Allocation Technology)
  • Intel CDP (Code/Data Prioritisation)
  • Intel MBA (Memory Bandwidth Allocation)
• Arm's answer: MPAM — Memory-system Partitioning and Monitoring, an architectural feature (Armv8.4-A onwards, mandatory in Neoverse N2+).

• PartID carried in CHI/AXI user-sideband.
• Every resource implements its own partition policy; discoverable via MPAM ID registers.
• Each resource has:
  • Configuration arrays (indexed by PartID)
  • Monitoring counters (indexed by PartID or PMG)
  • Enable / disable bits
• OS / hypervisor programs these via memory-mapped registers + MSR-like system regs (MPAM0/1/2/3_EL1/2/3).

• Linux kernel 5.19+ supports MPAM through the existing resctrl filesystem (originally built for Intel RDT/CAT).
• Mount: mount -t resctrl resctrl /sys/fs/resctrl.
• Inside the mount: one directory per partition, plus info/ showing capabilities (available PartIDs, resource caps).
• Each partition has:
  • cpus — which CPUs use it
  • tasks — which PIDs are tagged
  • schemata — the partition policy
  • mon_data/ — live counters

# Limit tenant B to 8 of 16 L3 ways + 30% BW
mkdir /sys/fs/resctrl/tenantB

cat > /sys/fs/resctrl/tenantB/schemata <<EOF
L3:0=00ff
MB:0=30
EOF

# Bind tenant B's processes
echo 12345 > /sys/fs/resctrl/tenantB/tasks
echo 12346 > /sys/fs/resctrl/tenantB/tasks

# Read live counters
cat /sys/fs/resctrl/tenantB/mon_data/mon_L3_00/llc_occupancy
cat /sys/fs/resctrl/tenantB/mon_data/mon_L3_00/mbm_local_bytes

Same API as x86 RDT. Portable to any MPAM-enabled Neoverse box. Kubernetes CRI-O and some OCI runtimes have hooks to set this up per container.

• Hypervisor at EL2 owns MPAM PartID allocation across VMs.
• Guest sees a virtualised MPAM (MPAM1_EL1 / MPAMIDR_EL1). HCR_EL2 traps writes to these for policing.
• KVM assigns a range of PartIDs to each guest; guest can sub-allocate internally (e.g., for resctrl inside the guest).
• PartID Narrowing: hypervisor can narrow a guest's PartID space so guests can't set values outside their allotted range.
• Cloud use: tenants request "gold" / "silver" / "bronze" SLA tiers → hypervisor assigns partition configs at VM start.

• CoreSight (2004) is Arm's architectural debug + trace infrastructure. Evolved from earlier EmbeddedICE / ETMv1 / CCC.
• Unifies every form of observability on an Arm SoC:
  • Program trace — ETM per core
  • Data trace — ETM data-watch, DWT on M-class
  • Debug access — DAP + APs
  • Event cross-triggering — CTI / CTM
  • System-level instrumentation — STM
• Everything is discoverable at runtime via ROM tables in memory-mapped regions.
• Used by JTAG probes (J-Link, ULINK), Linux perf, post-mortem dump tools, CTF tracers, and system-on-chip verification.

• External debugger (J-Link, DSTREAM, ULINK) connects via JTAG (TCK/TMS/TDI/TDO + TRSTn) or SWD (2-wire + clock).
• Enters the SoC at the DAP. DAP translates debug commands into APB/AXI reads/writes on the SoC's internal debug fabric.
• Multiple APs hang off the DAP:
  • MEM-AP — read/write any memory address (R/W as the CPU would see it)
  • APB-AP — reach CoreSight components' config registers
  • JTAG-AP — legacy secondary JTAG chain
• Authentication: DAP can be locked in field (DBGEN / NIDEN / SPIDEN) — no debug without matching silicon-fused keys.
• Secure-side debug separately gated via SPIDEN (Secure-Privileged Invasive Debug Enable).

• Each core has an ETM (Embedded Trace Macrocell) — snapshots every retired instruction + branch decision.
• Output: compressed trace stream (PE-address every N branches + deltas).
• Sinks options:
  • ETB / ETF — on-die SRAM (32 KB typical)
  • ETR — writes to DRAM (MiB-GiB, configurable)
  • TPIU — streams off-chip (limited by pad speed)
  • HSSTP — high-speed trace SerDes (multi-Gbps)
• Versions: ETMv3 (classic), ETMv4 (most current Cortex-A), ETE (Armv9 — merges ETM with TRBE for non-invasive DRAM-backed trace).

// Enable ETM via Linux perf

$ perf record -e cs_etm/@tmc_etr0/u ./workload

# results in a .data file with raw ETM trace bytes

$ perf script # decodes to instruction-accurate
              # PC history + branch decisions

# Alternatively:
$ /opt/ARM/DS/bin/trace_decode \
    --snapshot /proc/kcore \
    --etm-out trace.txt

The decode step is expensive but accurate: every retired instruction reconstructed. Used for post-crash analysis + finding rare races.

• Armv9-A / v8.6-A optional: TRBE — Trace Buffer Extension. Each core has a dedicated small trace buffer in its L2 tag area.
• Trace gets written directly to DRAM via a page-size circular buffer — no separate CoreSight funnel needed.
• Very low-overhead — the CPU manages its own trace buffer. Linux perf allocates DRAM pages and programs TRBPTR_EL1 / TRBBASER_EL1.
• Mandatory in Neoverse N2+ (SBSA Level 6).
• Use case: always-on instruction trace for flight-recorder-style post-mortem. If a production server crashes, last few million instructions are in DRAM, preserved across reboot if NVDIMM-backed.

• ITM (Instrumentation Trace Macrocell) — software writes to 32 stimulus ports. Each byte/halfword/word becomes a tagged trace packet.
• Classic use: Cortex-M "printf over SWO" — ITM stimulus + TPIU + SWO pin = real-time debug console without a UART.
• STM (System Trace Macrocell) — similar but system-wide: hundreds of channels, usable by any peripheral, interconnect node, or OS kernel.
• MIPI STP (System Trace Protocol) output. Multiple cores + peripherals can share.
• Linux stm_ftrace driver writes kernel ftrace events to STM channels — synchronised with hardware trace for cross-layer debugging.

// Printf over ITM on Cortex-M (pre-ETM)

#define ITM_STIM(n) (*(volatile uint32_t*)(0xE0000000 + 4*(n)))

int itm_putchar(int ch) {
  while ((ITM_STIM(0) & 1) == 0) ;   // wait for FIFO
  ITM_STIM(0) = ch;
  return ch;
}

// Or Linux / Cortex-A kernel STM
#include <linux/stm.h>
stm_log(stm_info, "trace %d\n", x);

STM gives you ftrace-style kernel tracing synchronized with ETM program trace — timeline reconstruction across kernel events + every instruction executed.

• CTI (Cross Trigger Interface) — a per-component set of input/output trigger channels. Each core has one; so do the DAP, ETMs, CTIs.
• CTM (Cross Trigger Matrix) — connects all CTIs into a broadcast fabric.
• Allows: "when core 0 hits this breakpoint, halt all cores, start tracing on core 3, and pulse a debug GPIO."
• Essential for debugging SMP bugs where cores need to be frozen synchronously — otherwise racy state changes while debugger is catching up.
• Also used to trigger trace capture from a PMU-counter overflow: CPU counts L3 misses, at threshold CTI fires, trace starts.

• "What does MPAM do?" → partitions cache + memory bandwidth per PartID so multi-tenant / mixed-criticality workloads can't interfere. Arm's answer to Intel CAT/MBA.
• "What's a PartID?" → 10-16-bit identifier tagged to every memory-system transaction. MPAM-aware resources check it to enforce allocations + count usage.
• "Linux resctrl on Arm?" → same /sys/fs/resctrl/ interface as Intel RDT, backed by MPAM. Write schemata like L3:0=ff;MB:0=50.
• "CoreSight purpose?" → unified debug + trace fabric. DAP for debug access, ETM for program trace, ITM/STM for instrumentation, CTI for cross-triggering.

• "How does an external debugger find every CoreSight component?" → ROM tables — discoverable memory regions listing base addresses + class IDs of every CoreSight IP on the SoC.
• "ETM vs SPE?" → ETM captures every retired instruction (full trace, heavy). SPE statistically samples 1 in N instructions with latency + attribution (lighter, still actionable).
• "What is TRBE?" → Armv9 Trace Buffer Extension: each core has its own DRAM-backed trace buffer. Low-overhead, suitable for always-on production flight recorder.
• "Why CTI?" → without cross-triggering, debuggers can't halt an SMP system atomically. CTI makes "all cores halt on this break" a hardware-guaranteed event.