Power & Thermal — Delivering 1000 W to a Card and Removing It Again

Topics We'll Cover

The GPU is increasingly an electrical and thermal engineering problem before it is a software one. This deck follows a single watt from the wall socket, through the PSU and 12V-2x6 cable, into the on-card VRMs and rails, into the die — and then back out as heat through the heat-spreader, cold plate, coolant loop, and ultimately the facility's chilled-water plant.

• Power in 2026 — Why It Matters
• Power Delivery — From PSU to Die
• Multi-Rail Design
• Transient Response — The Hard Part
• GPU Boost & Clock/Voltage Curves
• Power States (P-states) and Idle Power
• Power Capping
• Cooling Spectrum — Air to Immersion
• Thermal Throttling — The Last Defence
• Datacenter Engineering — From Card to Row
• Interactive: Power & Cooling Calculator

Power in 2026 — Why It Matters

The headline numbers tell you the story before any of the engineering does. A consumer flagship now draws what an entire workstation drew a decade ago, and a single rack of accelerators draws what a small office building draws.

Scaling out to a rack

An NVL72 rack is 36 GB200 superchips packaged with NVLink switches, PSUs, and a closed water loop in a single 19-inch frame. It draws roughly 120 kW. A datacenter row of eight NVL72 racks draws nearly 1 MW (about 580 kW of it the 576 GPUs themselves) — before networking outside the racks, storage, or the cooling and power-conversion overhead.

What that means for the engineering

• Per-die heat flux is around 0.6 W/mm² on a B200 die (1000 W over ~1,600 mm²) — comparable to a small soldering iron tip, applied uniformly across the package.
• Per-rack heat flux at 120 kW in 0.5 m² of frontal area is about 240 kW/m² — roughly the heat flux of a domestic gas hob held continuously against a wall of equipment.
• Air-cooling has run out at this scale. Liquid is no longer optional for top-tier accelerators.

Power Delivery — From PSU to Die

The journey from the wall socket to a transistor switching at 2.5 GHz crosses six or seven distinct voltage domains. Most of them are invisible to the user but each one is engineered to a tight specification.

The legacy ATX path versus 12V-2x6

For a long time consumer PSUs delivered 12 V on three or more separate 8-pin EPS/PCIe connectors. Each 8-pin is rated at 150 W — so a 450 W card needed three of them, plus an adapter. The 12V-2x6 connector (a quiet revision of the original 12VHPWR shipped with the RTX 4090) consolidates that into a single 16-pin header rated for 600 W on one cable, with four sense pins that let the card negotiate its allowance with the PSU.

Inside the on-card VRM

A single 12 V → 1 V buck converter at 700 A is impossible — the inductor would saturate, the MOSFETs would melt, and the ripple would be unusable. So the design splits the conversion across multiple parallel phases, each handling roughly 50–80 A. A B200 board carries on the order of 20 phases for the GPU core alone.

• Each phase consists of a high-side MOSFET, a low-side MOSFET, an inductor (~150 nH), a driver IC, and the local sense network. Many designs now use DrMOS integrated power stages to combine all of this into one package.
• Phases are interleaved: their switching cycles are evenly spaced in time, so that ripple currents partially cancel at the output and the bulk capacitor sees only a small fraction of the per-phase ripple. This is the same trick a polyphase rectifier uses on the AC side.
• The switching frequency sits around 500 kHz. Higher frequencies shrink the inductors but push switching losses up; lower frequencies reduce loss but demand bigger magnetics. Datacenter GPUs trend lower (300–500 kHz) for efficiency; tightly-packed laptops and cards trend higher.

Multi-Rail Design

A modern datacenter card is not a single voltage domain. It is six to twelve separate regulated supplies, each tuned for a specific consumer on the package, each with its own controller IC, sequencing requirement, and tolerance budget.

Why split the rails at all?

• The GPU core rail varies wildly with workload. Idle to a tensor-core matmul is a 100× current swing in microseconds. If the HBM IO sat on the same rail, every clock cycle of computation would inject ripple straight into the memory interface.
• HBM IO needs a stable, clean +0.4 V regardless of what the core is doing. Eye margins on a 9.2 Gbps DRAM bus are measured in tens of millivolts; rail noise that would not register as noise on the core would close the HBM eye entirely.
• NVLink PHYs need their own clean rail to keep eye margins under aggressor noise from the DRAM bus and the core. SerDes operating at 200 Gbps per lane have less voltage budget than HBM does, not more.
• Sequencing. Cores expect their PLL rail and analog rail to come up before the digital core; the wrong order can latch up the chip permanently. A bad sequence in manufacturing test is the kind of bug that destroys silicon.

Representative rail map for a Hopper-class card

The controllers

Each rail has its own digital PWM controller talking to its own array of power stages. Common silicon vendors: Renesas (formerly Intersil), Monolithic Power Systems (MPS), Infineon (formerly International Rectifier). All speak PMBus/I²C to the GSP firmware on the card, which orchestrates startup sequencing, telemetry, and per-rail margining.

Transient Response — The Hard Part

The hardest electrical problem on a modern GPU board is not steady-state efficiency; it is the transient. A single tensor-core matmul issued by the scheduler can take the chip from a few amps of leakage to several hundred amps of switching activity in a clock cycle or two — well under 100 nanoseconds.

Why VRMs cannot keep up

A buck phase switching at 500 kHz has a control loop bandwidth of roughly 50–100 kHz. That means the VRM cannot react to changes faster than ~10 microseconds, and certainly not on a 100-nanosecond timescale. If the load steps before the VRM can respond, the rail voltage droops until the inductor current ramps up to meet the demand. Droop the rail too far and the chip's minimum-operating-voltage spec is violated; logic glitches; PLLs lose lock; the workload hangs.

The decoupling network bridges the gap

A tiered capacitor network lives between the VRM phases and the package, each tier handling a different time-constant of disturbance:

The cost of getting it wrong

Insufficient decoupling shows up as voltage droop that pushes the rail under its minimum-operating-voltage line. The chip responds in one of two ways:

• Clock undershoot or PLL unlock — rare, dramatic, the workload hangs and the driver issues an Xid event.
• Adaptive voltage/frequency throttling kicks in: GPU Boost detects the droop event and pulls clock down to give itself more margin. The workload runs — but ~10–20% slower, and the user blames the model instead of the board.

GPU Boost & Clock/Voltage Curves

Every modern NVIDIA GPU runs at variable frequency. Idle, an H100 might sit at 800 MHz drawing 50 W; under sustained load, it reaches 1.98 GHz drawing its full 700 W. A consumer 4090 will idle at ~210 MHz and boost to 2.5 GHz or more.

What GPU Boost actually does

GPU Boost is a closed-loop firmware algorithm running inside the on-board GSP (GPU System Processor). Each die is binned at the factory with a per-part frequency-versus-voltage curve measured at production test. At runtime the GSP picks the highest stable clock that satisfies all of these constraints simultaneously:

• Power budget. Total board draw must stay under TGP (or the operator's -pl override).
• Temperature. Die T_j, HBM hotspot, VRM hotspot must all stay under their throttle thresholds.
• Voltage stability. Adaptive droop monitors trip if the rail collapses under a transient.
• Per-die binning. A "golden" die might hold 2.6 GHz at 1.05 V; a tail-of-distribution die might need 1.10 V for the same clock. The curve is per-part.

Because the curve is per-part, two physically identical 4090s in the same chassis will boost differently. The luckier silicon does more work for the same watts.

Inspecting it on a live system

# --- Selected output, annotated ---
Clocks
    Graphics                          : 1980 MHz   # current SM clock under load
    SM                                : 1980 MHz
    Memory                            : 1593 MHz   # HBM clock (fixed)
    Video                             : 1740 MHz

Max Clocks
    Graphics                          : 1980 MHz   # max permitted

Default Applications Clocks
    Graphics                          : 1755 MHz   # conservative default

Power Readings
    Power Management                  : Supported
    Power Draw                        : 689.42 W
    Current Power Limit               : 700.00 W   # TGP cap from firmware
    Min Power Limit                   : 200.00 W
    Max Power Limit                   : 700.00 W

Temperature
    GPU Current Temp                  : 71 C
    GPU Shutdown Temp                 : 95 C    # hard cutoff
    GPU Slowdown Temp                 : 90 C    # aggressive throttle
    GPU Max Operating Temp            : 88 C
    Memory Current Temp               : 78 C    # HBM hotspot
    Memory Max Operating Temp         : 95 C

Datacenter vs consumer tuning

The two tiers tune their boost behaviour very differently:

Power States (P-states) and Idle Power

NVIDIA exposes a discrete ladder of performance states, P0 through P12, that the GSP firmware moves between based on observed workload. P0 is full boost; P12 is the deepest idle. The transitions are fast (microseconds for clock changes, milliseconds for the voltage rail to settle), but with one important caveat that bites operators.

Persistence mode — the production-server lever

By default on consumer cards, when no CUDA process holds the device the driver unloads and the card drops to P8 or P12. Re-loading the driver on the next process start takes around 10 seconds. For a server that handles spiky inference traffic, that delay is wholly unacceptable: the first request after a quiet period stalls for ten seconds, the user sees a timeout, the autoscaler panics.

The fix is persistence mode:

# one-shot
$ sudo nvidia-smi -pm 1
Enabled persistence mode for GPU 00000000:01:00.0.

# systemd unit (the proper way)
$ sudo systemctl enable --now nvidia-persistenced

# verify
$ nvidia-smi --query-gpu=persistence_mode --format=csv
persistence_mode
Enabled

With persistence on, the driver stays loaded permanently, the card holds at P2 or P0 instead of dropping to P12, and process startup is back to milliseconds. This is mandatory on production inference hosts. Datacenter cards typically have it on by default; consumer cards do not.

Power Capping

Operator-level power capping lets you trade a small amount of throughput for a large amount of power. It is one of the most useful operational knobs on the platform, and it is criminally under-used.

# Cap RTX 4090 from 450 W to 350 W
$ sudo nvidia-smi -pl 350
Power limit for GPU 00000000:01:00.0 was set to 350.00 W from 450.00 W.
All done.

# Verify the floor and ceiling
$ nvidia-smi --query-gpu=power.limit,power.min_limit,power.max_limit --format=csv
power.limit [W], power.min_limit [W], power.max_limit [W]
350.00 W, 100.00 W, 450.00 W

# Recent driver versions also accept a floor:
$ sudo nvidia-smi --power-limit-min=200 --power-limit=350

The economics

Power scales roughly with V²f, while throughput scales linearly with f. So pulling clock down a small amount drops power a large amount — the relationship is supra-linear. Empirical numbers for a 4090:

The PL is enforced inside the GSP firmware. When the operator sets a 350 W cap, the GSP's closed-loop algorithm dynamically scales clocks down to keep total board draw within budget. The user space sees nothing — the workload just runs slightly slower with no other behavioural change.

When to use it

• Dense racks at the edge of the cooling envelope. Eight 4090s in a 4U at 350 W each removes 800 W of heat compared to stock — sometimes the difference between fitting in the chassis and not.
• Brown-out conditions. Datacenter operators integrate nvidia-smi -pl into emergency response: cap the entire fleet by 30% rather than shed load.
• Off-peak workloads. Overnight batch training where wall-clock doesn't matter — reduce energy bill by 20% at 5% throughput cost.
• Per-MIG-instance capping. Recent drivers (2024+) let you cap individual MIG slices independently, which is essential when one tenant runs an FP8 attention kernel and another runs a small embedding workload.

Cooling Spectrum — Air to Immersion

There are roughly five practical approaches to extracting the heat that a power-delivery network has just deposited into the silicon. They span four orders of magnitude in cost and two in capability. The right choice depends almost entirely on the watts-per-card you're trying to remove.

Where the line falls

Thermal Throttling — The Last Defence

If power-delivery is the offensive line, throttling is the goal-keeper of last resort. It exists because cooling can fail in many ways — a fan dies, an inlet temp rises, a coolant-loop CDU loses pressure — and the silicon must protect itself before any of those become permanent.

Sensors and thresholds

The GSP firmware runs a control loop sampling on the order of kilohertz across multiple thermal sensors:

• T_j (junction) across the die — multiple sensors, the maximum is what matters. Throttle threshold ~85–95 °C depending on family.
• HBM hotspot — per-stack thermal diode. Throttle threshold ~95 °C. HBM is the most fragile thermal element on the package; once HBM throttles, memory bandwidth collapses faster than compute.
• VRM phase temperature — at the inductor and at the DrMOS. Threshold ~110 °C. Less commonly hit, but a hot VRM means the power-delivery efficiency is collapsing under you.
• Inlet air or coolant temp on the chassis side — not a throttle source directly, but feeds the policy.

Two failure modes operators see

The diagnostic incantation

$ nvidia-smi --query-gpu=clocks.current.graphics,temperature.gpu,clocks_throttle_reasons.active --format=csv -l 1

clocks.current.graphics [MHz], temperature.gpu [C], clocks_throttle_reasons.active
1980, 71, 0x0000000000000001     # 0x1 = idle, no throttle
1980, 82, 0x0000000000000001
1755, 88, 0x0000000000000004     # 0x4 = SW thermal slowdown
1410, 91, 0x000000000000000c     # 0xc = HW thermal + SW thermal
1245, 94, 0x000000000000000c     # dropping 30%, near shutdown

The throttle-reason bitmap

Datacenter Engineering — From Card to Row

Once you put eight cards in a chassis and 18 chassis in a rack, the engineering changes character. The card-level problems all still exist; on top of them sit airflow, power distribution, plumbing, and PUE.

From card heat to chassis heat

A 700 W card removes ~2390 BTU/hr. (One watt is 3.412 BTU/hr; the conversion is mechanical.) A 4U HGX chassis with eight H100s at 700 W each is 5.6 kW of heat — ~19,100 BTU/hr — that has to leave the rack into the hot aisle.

For air-cooled removal at a typical 15 °C aisle delta-T, the rule of thumb is roughly 0.026 CFM per watt:

• Single 700 W card: ~18 CFM front-to-back.
• Eight-card HGX chassis at 5.6 kW: ~150 CFM, sustained.
• Whole rack at 30 kW (legacy air-cooled limit): ~780 CFM — near the practical limit of perforated-tile delivery.

For closed-loop water at 25 °C inlet and 35 °C return, a similar rule of thumb is ~0.85 GPM per kW (using water's specific heat). So:

• Eight-card HGX at 5.6 kW: ~5 GPM, with ~2 m of head loss across the cold plates.
• NVL72 at 120 kW: ~100 GPM, supplied by a row-level CDU off the facility chilled-water main.

Power distribution at rack scale

A 30–120 kW rack does not run on a household plug.

• The facility delivers three-phase 380 V or 415 V AC to the rack PDU (Power Distribution Unit). Single-phase 230 V can't physically carry the current at this load.
• For dense modern racks (NVL72 and beyond), NVIDIA pushes a 48 V DC rack-internal bus: each compute tray pulls 48 V instead of 12 V, then steps it down on-tray. Why: at 120 kW, 48 V means 2500 A on the rack busbar; 12 V would mean 10000 A. The I²R losses scale as current squared, so the 48 V architecture cuts busbar copper losses by 16×.
• Each compute tray then runs an on-tray 48 V → 12 V intermediate bus converter feeding the GPU board's VRM stage from slide 02.

PUE — Power Usage Effectiveness

PUE is the ratio of total facility power to useful IT power: a PUE of 1.0 means every watt drawn from the grid ends up as compute. Real datacenters live between these poles:

PUE is dominated by cooling overhead, with power-conversion efficiency a distant second. Moving from air-cooled HGX to D2C-cooled NVL72 typically buys 0.10–0.15 of PUE. At hyperscale with a multi-MW deployment, that pays for the entire D2C plumbing capex within the first year.

Interactive: Power & Cooling Calculator

Pick a GPU, a chassis density, a cooling approach, and an inlet temperature. The calculator estimates total heat load, the airflow or coolant flow you'd need, the thermal margin you have, and whether the configuration is feasible. All maths is inlined in the page so you can read it.

How the maths works

// Per-card TGP table (W)
const tgp = {
    rtx4090: 450, rtx5090: 575, rtxpro6000bw: 600,
    a100: 400, h100: 700, h200: 700,
    b200: 1000, gb200: 2700
};

// Cooling capability per card (W) at 25 C inlet
const cap = {
    axial: 450, blower: 400, passive: 700,
    d2c: 1500, immersion: 3000
};

// Inlet derating (warm/hot air loses cooling capability)
const inletScale = (T) => T <= 18 ? 1.10 :
                          T <= 25 ? 1.00 :
                                       0.78;  // 35 C: ~22% capability loss

// Required airflow (CFM) at 15 C delta-T: 0.026 CFM per watt
const cfm = (W) => W * 0.026;

// Required water flow (GPM) at 10 C delta-T: ~0.85 GPM per kW
const gpm = (W) => (W / 1000) * 0.85;

// Facility overhead: each cooling type carries a PUE-like factor
const pue = {
    axial: 1.45, blower: 1.40, passive: 1.30,
    d2c:   1.12, immersion: 1.06
};