Power Management in ML Accelerator SoCs

• TDP scaling — datacenter accelerators have surged from ~100 W (K80, 2014) to 300 W (A100), 700 W (H100 SXM5), and beyond 1000 W (B200 rack modules) in a decade
• Power density — advanced nodes (3 nm, 2 nm) pack more transistors but thermal density now exceeds 1 W/mm²; local hotspots reach 2–3 W/mm² in MAC arrays
• Utilization wall — at 3 nm, only ~30–50% of transistors can switch simultaneously within TDP; rest must be dark or voltage-scaled
• Dark silicon — Esmaeilzadeh et al. predicted 50–80% dark silicon by 2025; ML SoCs embrace this via specialized accelerator blocks activated on demand

Domain Types

• Always-on (AON) — PMU controller, RTC, wakeup logic, retention SRAM; typically 5–20 mW; never powered off
• Switchable compute — NPU clusters, GPU shader arrays, CPU cores; independently gated; 10–200 W per domain
• Switchable I/O — PCIe, HBM PHYs, SerDes; gated when link is idle; careful sequencing required
• Retention domains — subset of switchable domains that maintain register/SRAM state via retention cells at reduced voltage (~0.4 V)

Isolation & Level Shifting

• Isolation cells — clamp outputs of powered-off domains to known values (0/1/latch) to prevent floating signals from causing crowbar current in active domains
• Level shifters — translate signals between domains at different voltages; HL (high-to-low) and LH (low-to-high) variants
• Retention registers — balloon-latch or shadow-latch flip-flops; save state on SAVE signal, restore on RESTORE; ~15% area overhead per FF
• Always-on buffers — repeaters in AON domain carry signals across gated regions without interruption

Header vs Footer Switches

• Header switch (PMOS) — placed between VDD and virtual-VDD (VVDD); most common in CMOS; better noise isolation
• Footer switch (NMOS) — placed between virtual-VSS (VVSS) and ground; higher drive strength per unit area; used in some Intel designs
• Rush current control — switches are daisy-chained with staggered enable (100–500 ns delays) to limit di/dt; typical inrush limit: 100–500 mA/ns
• Switch sizing — ~5–10% of total gated area; IR drop across switch: <10 mV at peak current; on-resistance: 1–5 mΩ

Wake-up Latency

• Power gate turn-on: 0.5–5 μs (switch chain stagger)
• Rail stabilization: 5–20 μs (decap charging)
• State restore (retention): 1–10 μs
• PLL/clock relock: 10–50 μs
• Total cold wake-up: ~20–80 μs per domain

Power Gating Sequence

UPF Key Concepts

Power Domain

Logical grouping of cells sharing a supply set

Supply Network

VDD/VSS rails, switches, and connections

Power State Table

Defines legal combinations of domain states (ON/OFF/RETENTION)

Isolation Strategy

Rules for clamping signals at domain boundaries

Retention Strategy

Which registers save state; save/restore triggers

Level Shifter Strategy

Where and what type of shifters are inserted

Power-Aware Verification

• Simulation — UPF-aware simulators (Synopsys VCS, Cadence Xcelium) corrupt signals from gated domains; catch missing isolation bugs
• Static checks — formal verification of isolation completeness, level shifter coverage, and state table consistency
• Power-aware synthesis — Synopsys DC inserts isolation, retention, and level shifter cells per UPF directives
• P&R integration — Innovus/ICC2 places power switches, routes virtual rails, and checks IR drop per domain

## UPF snippet — NPU power domain
create_power_domain PD_NPU \
  -elements {npu_top}
create_supply_net VDD_NPU
create_supply_net VSS
create_supply_set SS_NPU \
  -function {power VDD_NPU} \
  -function {ground VSS}
set_isolation iso_npu \
  -domain PD_NPU \
  -isolation_power_net VDD_AON \
  -clamp_value 0
set_retention ret_npu \
  -domain PD_NPU \
  -save_signal {save_npu posedge} \
  -restore_signal {restore_npu posedge}

The Power Equation

• Quadratic voltage — reducing V by 20% cuts dynamic power by ~36%, but frequency must also drop (V ∝ f in near-threshold)
• Cubic savings — since f scales roughly with V, effective power scales as ~V³; 0.8× voltage → ~0.5× power
• Leakage floor — P_leak = I_sub(V,T) + I_gate(V); at advanced nodes, leakage is 30–50% of total power at nominal voltage
• P_total = P_dynamic + P_leak + P_{short-circuit}

Operating Points & Guardbands

Adaptive Voltage Scaling (AVS)

• Per-chip calibration — critical-path ring oscillators (CPMs) measure actual silicon speed
• Guardband reduction — AVS eliminates 50–100 mV of design margin, saving 10–20% power
• Closed-loop AVS — continuous voltage adjustment based on speed monitors; used in Apple M-series, AMD Zen, Intel SpeedShift

Voltage Droop Detection

• On-die voltage droop — sudden current transients (e.g., cache miss → all-bank activation) cause supply to droop 50–150 mV in <1 ns
• Droop detectors — analog comparators or critical-path replica monitors; Intel uses >100 distributed sensors per core
• Response — hardware clock stretching or frequency throttle within 1–5 ns; prevents timing violations
• Proactive mitigation — microarchitectural throttling (e.g., limit concurrent vector operations) to bound di/dt

PLL/FLL Relock Time

• PLL relock — 5–50 μs for phase-locked loop to acquire new frequency; domain stalls during transition
• FLL (frequency-locked loop) — 1–5 μs relock; preferred for fast DVFS; used in Arm DynamIQ, Apple M-series
• Glitch-free mux — switch between PLL output and backup clock during transitions to maintain operation

Voltage Regulator Response

• Slew rate — external VRMs: 10–50 mV/μs; on-die regulators: 100–500 mV/μs
• Voltage first or frequency first? — scaling up: raise voltage, wait, then raise frequency; scaling down: lower frequency first, then voltage
• Transition energy — total energy for V/F transition: 10–100 μJ depending on domain capacitance and voltage delta

Firmware DVFS Governor

• Performance governor — always max OPP; used for latency-critical inference
• On-demand governor — samples utilization every 1–10 ms; scales up on >80% busy, down on <30%
• Predictive governor — ML workload phase detection; pre-scale voltage before compute burst arrives
• Cooperative firmware/HW — hardware handles fast droops (ns); firmware handles OPP transitions (ms); OS provides policy hints

Coordinated Scaling

• Shared power budget — total SoC power capped (e.g., 700 W for H100); domains compete for thermal headroom
• Cross-domain dependencies — NPU must not outrun memory controller bandwidth; coordinated OPP tables
• Power balancing — if NPU is idle during all-reduce, shift power budget to NVLink/network domain

Workload-Aware Policies

• Training: compute phase — NPU/tensor at max OPP; GPU shaders at nominal; CPU idle
• Training: gradient sync — NPU throttled; NVLink/network at max; memory at peak bandwidth OPP
• Inference: batched — NPU at moderate OPP for energy efficiency; latency budget allows lower clocks
• Inference: real-time — NPU at turbo OPP; tail-latency SLA requires max frequency

Multi-Rail PMICs

• Rail count — modern AI accelerator PMICs supply 8–20+ independent rails (core, I/O, SRAM, PLL, analog, HBM)
• Current capacity — core rails: 200–1000 A per phase; multiphase buck converters (6–16 phases) for high-current domains
• Vendors — Renesas (Intersil), MPS, Infineon, TI, onsemi; custom PMICs for hyperscaler designs
• Efficiency — modern multiphase bucks achieve 90–95% efficiency at full load; drops to 70–80% at light load

Sequencing & Soft-Start

• Power-up sequence — core VDD → I/O VDD → PLL supply → memory VDD; strict ordering prevents latch-up and ESD damage
• Soft-start — ramp rate limited to 1–10 mV/μs; limits inrush current; prevents supply overshoot
• Power-down — reverse sequence; graceful shutdown with drain timeout; emergency fast-discharge on fault

Control Interfaces

I2C / PMBus

Standard management bus; voltage margining, telemetry, OCP thresholds; 100–400 kHz

SPI

Higher bandwidth configuration; used for initial PMIC programming at boot

SVID (Intel)

Serial Voltage ID; dedicated 1-wire protocol for fast voltage requests (~1 μs latency); used in x86 platforms

AVSBus (AMD)

Adaptive Voltage Scaling bus; communicates target voltage from SoC to VRM in real-time

GPIO/PWM

Simple VID codes for coarse voltage selection; legacy but still used in cost-sensitive edge AI

Fault Protection

• OVP — over-voltage protection; crowbar or shunt; response <1 μs
• OCP — over-current protection; hiccup mode or latch-off; per-phase current sensing
• OTP — over-temperature protection; PMIC-internal thermal shutdown at 125–150 °C junction
• UVP — under-voltage lockout; prevents SoC from operating at dangerously low voltage

Low-Dropout Regulator (LDO)

• Topology — series pass transistor; V_out = V_in − V_dropout; dropout as low as 50–200 mV
• Efficiency — η = V_out/V_in; if V_in=1.0V, V_out=0.75V, η=75%; poor for large V_in–V_out gaps
• Noise — extremely low output noise (<10 μV_rms); ideal for PLL, ADC, and analog supplies
• Response — fast transient response (~10 ns); no switching artifacts
• Area — small die area; no inductors; integrates well on-die

Switching Regulator (Buck/Boost)

• Topology — inductor-based converter; PWM switching at 1–10 MHz; requires external L and C
• Efficiency — 85–95% across wide V_in/V_out range; dominates for high-current loads
• Noise — switching noise at carrier frequency and harmonics; requires filtering for sensitive loads
• Response — limited by LC filter bandwidth; 1–10 μs transient response
• Current — scales to 100s of amps with multiphase designs; each phase handles 30–60 A

Digital LDOs

• Architecture — array of binary-weighted PMOS transistors controlled by a digital loop; replaces analog error amplifier
• Quantization — 64–256 unit switches provide 6–8 bit voltage resolution (~3–5 mV steps)
• Loop bandwidth — up to 100 MHz with fast comparator; suitable for suppressing on-die droop
• Area — 0.01–0.05 mm² at 5 nm; dozens of instances per SoC for granular voltage islands
• Limitations — lower efficiency than analog LDO; quantization ripple; clock noise injection

FIVR (Intel) / PIVR

• Fully Integrated Voltage Regulator — on-package buck converter using thin-film inductors on package substrate
• Intel FIVR — introduced in Haswell (2013); 140+ MHz switching; sub-ns load response. Dropped after Broadwell in client; restored for server in Ice Lake-SP / Sapphire Rapids. Modern Intel client (Meteor/Arrow/Lunar Lake) uses package VR plus on-die DLVR (digital linear VR).
• Package-Integrated VR (PIVR) — inductors embedded in organic substrate or silicon interposer; AMD exploring for Zen 5+
• Efficiency — 85–90% at full load; enables >20 independent voltage domains per die

Granular Voltage Islands

Intel FIVR

140 MHz, thin-film inductors, 90% peak η

Apple M3

Per-core DVFS, FLL-based, <1 μs transitions

AMD Zen 4

Discrete VRM + on-die clock stretching for droop

Google TPU

Custom PMIC with domain-level voltage control

The Dark Silicon Problem

• Definition — transistors that cannot be powered on simultaneously due to thermal/power constraints; coined by Esmaeilzadeh et al. (ISCA 2011)
• Scaling trend — at 7 nm: ~50% dark; at 3 nm: ~60–70% dark; Moore's Law adds transistors faster than power budgets grow
• ML SoC advantage — heterogeneous accelerators (tensor, vector, scalar, DMA) are naturally used in phases; dark silicon becomes"specialized silicon"
• Thermal density — local hotspot power density of 2–3 W/mm² exceeds air cooling limits (~0.5 W/mm²); liquid cooling extends to ~1.5 W/mm²

Power Scheduling Strategies

• Spatial scheduling — activate non-adjacent compute tiles to spread thermal load; checkerboard patterns reduce hotspot temperature by 10–15 °C
• Temporal scheduling — rotate active tiles over time (e.g., 100 ms quanta); thermal time constant of silicon die is 1–10 ms
• Chiplet power budgeting — in multi-chiplet designs (MI300X: 12 chiplets), per-chiplet power caps ensure package-level TDP compliance
• NVIDIA H100 — GH100 has 144 physical SMs; SXM5 enables 132, PCIe enables 114 (yield + power binning); max SM concurrency further limited by TDP at boost clocks

Clock Gating Fundamentals

• ICG cells — Integrated Clock Gating cells: latch + AND gate; glitch-free clock enable on negative edge; standard cell library primitive
• RTL inference — synthesis tools (DC, Genus) automatically insert ICG cells for conditional register loads: if(en) q <= d; → ICG
• Clock gating efficiency — percentage of clock cycles where gating is active; target >80% for data-path registers
• Granularity — fine-grained (per-register group, 4–16 FFs) vs coarse-grained (per-module, 1K–100K FFs)

Fine-Grained vs Coarse-Grained

Clock Mesh Power

• Clock tree power — global clock spine + sector buffers + leaf buffers; H-tree or mesh topology; 100K–1M buffers per GHz domain
• Clock mesh — used in high-performance SoCs (Apple, Intel) for low-skew distribution; higher power than tree but better timing
• Mesh gating — sector-level clock gating in mesh architectures; when a tile is idle, its entire mesh sector is gated
• Dynamic clock mesh — programmable mesh connectivity; disable mesh segments for inactive regions

ML-Specific Clock Gating

• Sparse activation gating — in transformers, gating zero-weight MAC units saves 30–60% dynamic power
• Operand-aware gating — gate upper bits of multiplier when operands are narrow (INT8 vs INT16)
• Pipeline stage gating — gate empty pipeline stages in systolic arrays during bubble cycles
• Activation sparsity — ReLU outputs ~50% zeros; gating corresponding datapath saves proportional energy

Training Phase Behavior

• Forward pass — high MAC utilization (80–95%); peak power; memory bandwidth demand for weight loading
• Backward pass — similar compute intensity; slightly different memory access pattern; gradient accumulation
• Gradient synchronization — all-reduce across GPUs/nodes; compute nearly idle; network at peak; 20–40% of iteration time in large clusters
• Optimizer step — element-wise operations; low compute intensity; memory-bound; opportunity for voltage scaling

Inference Batching & Power

• Batch size vs efficiency — larger batches amortize power overhead; energy per token decreases 2–5× from batch=1 to batch=64
• Continuous batching — vLLM-style batching keeps utilization high, reducing idle power waste by 30–50%
• Power proportionality — ideal: power ∝ throughput; reality: 40–60% idle power; improving with aggressive gating

Dynamic Power Capping

• Hardware power meter — on-die current sensors (shunt resistors or replica circuits) + voltage monitors; sampled at 1–10 kHz
• Rolling average — power capping uses exponential moving average (EMA) over 10–100 ms window; prevents transient violations
• Throttle actions — reduce clock frequency → reduce voltage (if time permits) → power-gate idle domains → reduce batch throughput
• Rack-level capping — NVIDIA DCGM / AMD ROCm SMI coordinate power caps across GPUs in a node

RAPL-like Interfaces

Intel RAPL

Running Average Power Limit; MSR-based; per-package, per-core, DRAM domains; 1 ms update rate

NVIDIA NVML

Power limit enforcement via driver API; per-GPU; configurable power cap with <10 ms response

AMD HSMP

Host System Management Port; socket-level power capping; CCD-level granularity in EPYC

Custom SoC PMU

ML SoCs implement proprietary power management units with per-domain telemetry and limits

Multi-Vt Libraries

• SVT (Standard-Vt) — baseline speed and leakage; used for most logic
• HVT (High-Vt) — ~10× lower leakage than SVT; ~20% slower; used for non-critical paths
• LVT (Low-Vt) — ~10× higher leakage than SVT; ~15% faster; used for critical paths only
• ULVT (Ultra-Low-Vt) — fastest; extreme leakage; sparingly used on most-critical timing paths
• Typical mix — 60% HVT, 30% SVT, 8% LVT, 2% ULVT yields near-maximum performance at 2–3× lower leakage than all-SVT

Gate-Length Biasing

• Longer gate length — adding 1–2 nm to gate length on non-critical paths reduces sub-threshold leakage by 2–3×
• Automated by P&R — Innovus/ICC2 can swap cells to long-channel variants post-timing closure
• Stacking effect — series-connected transistors in NAND/NOR have exponentially lower leakage than parallel (inverter) structures

Power-Aware Synthesis & P&R

• Power-driven synthesis — tools optimize for min-power subject to timing constraints; multi-Vt cell selection, operand isolation, clock gating insertion
• Power-aware placement — co-place high-switching cells near decaps; minimize clock routing length; thermal-aware placement for hotspot mitigation
• MCMM optimization — multi-corner multi-mode: optimize for both active power (typical corner) and leakage (slow corner, high temp)
• ECO power optimization — post-route leakage recovery: swap SVT→HVT on paths with positive slack; typically recovers 10–20% leakage

Leakage at Advanced Nodes

• FinFET (5–7 nm) — excellent electrostatic control; leakage manageable; gate leakage reduced by high-k dielectric
• GAA/Nanosheet (3–2 nm) — even better electrostatic control; I_off reduced 20–30% vs FinFET at same performance
• CFET (future) — complementary FET stacks NMOS on PMOS; promises 30–50% area reduction; leakage TBD
• Backside power delivery (BSPDN) — Intel PowerVia on Intel 18A (2025), TSMC "Super Power Rail" on A16 (mass production late 2026). TSMC N2/N2P retain frontside power; ~5–15% power improvement from lower IR drop.

DTPM (Dynamic Thermal & Power Management)

• Thermal sensors — on-die diode sensors (1–2 °C accuracy); 10–50 distributed across die; sampled at 1–10 kHz
• T_j,max — maximum junction temperature: 100–105 °C for datacenter GPUs; 110 °C for some mobile SoCs
• Thermal throttling — progressive: first reduce boost clock → reduce to nominal → reduce to minimum → emergency power-off at T_critical (125 °C)
• Predictive throttling — model thermal trajectory; begin throttling before T_j,max to avoid abrupt performance cliff

Power-Thermal Feedback Loops

• Positive feedback risk — higher T → higher leakage → higher P → higher T; thermal runaway in extreme cases
• Control loop — PID controller: T_target = 95 °C; adjust power cap every 1–10 ms; proportional band: ±5 °C
• Multi-input control — power, temperature, current, and voltage all feed into unified throttling decision
• Time constants — die: 1–10 ms; package: 100 ms–1 s; heatsink: 10–60 s; room: minutes

Edge Device Constraints

• Skin temperature — phones/tablets limited to 40–45 °C skin; sustained SoC power: 3–8 W; burst: 15–25 W
• Passive cooling only — no fans; heat spreads through chassis; thermal throttling is the norm, not the exception
• Battery thermal limits — Li-ion batteries degrade above 45 °C; charging + compute creates dual heat source
• Sustained ML inference — edge NPUs run at 1–3 W sustained; burst to 8–10 W for <30 s before throttling

Cooling Solutions by TDP

Standards & Specifications

• IEEE 1801-2024 (UPF 4.0) — current Unified Power Format; extends retention modelling and successive refinement semantics
• Si2 CPF — Common Power Format; Cadence-backed alternative to UPF
• JEDEC JESD51 — Thermal measurement standards for semiconductor devices
• Intel SVID Protocol — Serial VID Interface Design Guide
• PMBus (SMBus) — Power management bus protocol specification

Key Papers

• Esmaeilzadeh et al. —"Dark Silicon and the End of Multicore Scaling," ISCA 2011
• Horowitz —"Computing's Energy Problem (and What We Can Do About It)," ISSCC 2014
• Kim et al. —"FIVR — Fully Integrated Voltage Regulators on 4th Generation Intel Core SoCs," ISSCC 2014
• Singh et al. —"Zen 2: The AMD 7nm Energy-Efficient High-Performance x86-64 Microprocessor Core," ISSCC 2020
• Jouppi et al. —"TPU v4: An Optically Reconfigurable Supercomputer," ISCA 2023

Books

• Keating et al. — Low Power Methodology Manual, Springer, 2007
• Rabaey, Chandrakasan, Nikolic — Digital Integrated Circuits, 2nd ed., Pearson
• Weste & Harris — CMOS VLSI Design, 4th ed., Addison-Wesley
• Irwin & De — Low Power Design Methodologies, Springer, 1996
• Calhoun & Chandrakasan — Ultra-Low-Voltage Circuit Design, Springer, 2015

Industry Resources

• Synopsys — Power Compiler, PrimeTime PX, IC Compiler II power optimization user guides
• Cadence — Voltus IC Power Integrity, Joules RTL Power Solution
• NVIDIA — GPU Power Management documentation (NVML API, DCGM)
• Intel — Running Average Power Limit (RAPL) interface specification