Ampere — A100, RTX 30, and the Birth of the LLM Era

Topics We'll Cover

A focused tour of the NVIDIA Ampere generation: the chips, the SM, the tensor-core format revolution, the partitioning story, and the practical numbers that matter for fine-tuning and inference today.

• Ampere in One Page
• GA100 — The Datacenter Die
• The Ampere SM
• 3rd-Gen Tensor Cores — The Format Revolution
• MIG — Hardware Partitioning
• GA10x — The Consumer Family
• NVLink 3 + DGX A100
• A100 80 GB Upgrade — Why HBM Matters
• Performance for Modern LLMs
• Software Stack on Ampere
• Interactive: Ampere SKU Picker

Ampere in One Page

Ampere was NVIDIA's first architecture designed with transformer training as a primary workload. Volta (2017) had introduced tensor cores; Turing (2018) brought them to consumer cards; Ampere (2020) is where the format choices, capacity targets, and partitioning story finally fit the shape of a transformer pretraining run.

The headline trick: 3rd-gen tensor cores with TF32. TF32 is a 19-bit format with FP32-style 8-bit exponent and a 10-bit mantissa. It is a drop-in replacement for FP32 matmul — existing pretraining code “just works” at roughly 8× the throughput, with no autocast and no loss-scaling dance. The other formats — BF16 and INT8 with 2:4 structured sparsity — are what unlocked the GPT-3 era.

Two physical chips, very different worlds:

• GA100 — TSMC 7N, 826 mm², 54.2 billion transistors, HBM2e, datacenter only. Lives in the A100 SXM4 / PCIe and a handful of professional cards.
• GA10x — Samsung 8N, GDDR6X, consumer/workstation. GA102 powers the RTX 3090 / 3090 Ti / 3080 Ti / A40 / RTX A6000; smaller dies (GA104, GA106, GA107) cover the rest of the RTX 30 lineup.

GA100 — The Datacenter Die

The A100 is the binned-down product. The full GA100 is bigger than what ships — NVIDIA leaves headroom for yield. Knowing the full layout helps explain odd numbers like “108 SMs”.

The Ampere SM

An Ampere SM is structured into 4 partitions. Each partition has its own warp scheduler, register file slice, and tensor core.

Async copy: registers stop being a bottleneck

Pre-Ampere, every shared-memory load relayed through registers: global → reg → shared. On a transformer GEMM that means the register file becomes a strict gatekeeper for occupancy.

Ampere introduces cp.async in PTX: the SM issues an asynchronous copy that goes L2 → shared memory directly, bypassing the register file. Combined with a multi-stage pipeline (typically 3 stages), it lets a CUDA programmer prefetch the next tile while the current one is being multiplied — without burning registers on the in-flight data.

Cooperative groups become first-class

CUDA 11 + Ampere promote cooperative thread arrays from a curiosity to a routine. cooperative_groups::thread_block_tile<32>, multi-block sync via the grid group, and async memcpy with cuda::memcpy_async in libcu++ all assume Ampere as the floor.

3rd-Gen Tensor Cores — The Format Revolution

Volta tensor cores did FP16 with FP32 accumulation. Turing added INT8 / INT4. Ampere kept all that and added three things that genuinely changed how people train transformers.

The Ampere throughput table

A100 SXM4 80 GB, dense unless noted. Numbers from the NVIDIA A100 datasheet.

MIG — Hardware Partitioning

Multi-Instance GPU is Ampere's most underrated feature. An A100 can be carved at the hardware level into up to seven isolated GPU instances, each with its own dedicated SMs, L2 slice, and HBM region. Not virtualisation — isolation is enforced by the memory crossbar, not by software.

Granularities

The partition unit is one GPC (which on A100 corresponds to ~14 SMs, ~5 GB HBM, ~5 MB L2). Available shapes:

Why this matters in shared clusters

• Per-team isolation. A 1g.5gb slice cannot interfere with the L2 or HBM of any other slice. A noisy embedding service won't tank your training run.
• Predictable QoS. Each instance gets a guaranteed slice of bandwidth.
• Higher utilisation. A small inference workload that would “reserve” a whole A100 can now share with six others.

One A100 carved up — a typical mixed layout

# enable MIG mode (drains running work; needs reset)
nvidia-smi -mig 1

# list available profiles
nvidia-smi mig -lgip

# create the slices on GPU 0
nvidia-smi mig -cgi 9,14,19,19 -C
# 9 = 3g.20gb, 14 = 2g.10gb, 19 = 1g.5gb (profile IDs vary by SKU)

# each slice gets its own UUID; pin a container to one
docker run --gpus '"device=MIG-GPU-..."' vllm/vllm-openai ...

GA10x — The Consumer Family

The other half of Ampere lives on Samsung 8N. Same 3rd-gen tensor cores, same TF32/BF16 story, but no HBM, no NVSwitch, and a quietly different SM that has been the source of years of confusion.

Why the “CUDA core” count is misleading

On GA100, a partition has 16 FP32 cores and 16 INT32 cores; an SM totals 64 FP32. On GA10x, NVIDIA doubled the FP32 count per partition: each partition has 16 FP32 + 16 dual-issue (FP32 or INT32) cores. An SM totals 128 FP32 when running pure FP32, but only 64 FP32 if INT32 is also live (because the dual-issue cores get co-opted).

This is why an RTX 3090 advertises 10496 CUDA cores when GA100 with the same silicon area would have ~5000. The marketing number is the FP32-only peak; real shader workloads alternate FP32 and INT32 (address arithmetic, indexing) and rarely sustain it. The tensor-core count is unchanged — 4 per SM, just like GA100.

Workstation cards are the sleeper picks

The RTX A6000 and A40 are Ampere's 48 GB GDDR6 ECC options. Same GA102 die as a 3090, but with the full memory complement, ECC enabled, and a datacenter-blessed driver. The A6000 supports an NVLink bridge between two cards (112 GB/s) — not switch-class, but fine for 2-way TP on 70B models. Used 48 GB Ampere workstation cards remain a 2026 sweet spot for serious local work.

NVLink 3 + DGX A100

NVLink 3.0 doubles the per-lane signalling rate from NVLink 2.0 (50 vs 25 Gb/s) but halves the lanes per link, so each link stays at 25 GB/s per direction. A100 doubles the link count to 12 → 600 GB/s aggregate bidirectional. NVSwitch 2.0 is the fabric chip that ties them together.

The reference platform for the GPT-3 era

Meta's OPT-175B (2022), BLOOM (2022), Megatron-Turing NLG (2022), and most large public LLMs from this period were trained on DGX A100 or HGX A100-derived systems. The 3D-parallelism recipe (TP × PP × DP) was tuned around exactly this baseboard.

Worked example: TP=8 all-reduce on a 70B model

Per layer, a tensor-parallel all-reduce needs to ship one full layer's hidden state. For Llama-2-70B with hidden dim 8192 and BF16: 8192 × seq × 2 bytes per token. At seq=4096 that's 67 MB per layer.

• Over NVSwitch (600 GB/s): 67 MB / 600 GB/s → ~110 µs per all-reduce, about 75 µs for the smaller per-layer tensors typical in attention.
• Over PCIe 4.0 x16 (~25 GB/s effective): same 67 MB → ~2.7 ms, or ~1 ms for the smaller tensors.

That's a 10–20× latency gap. With 80 transformer layers, each step accumulates ~80 × 75 µs = 6 ms on NVSwitch vs 80 ms on PCIe. NVLink is why TP=8 works on a single A100 node.

A100 80 GB Upgrade — Why HBM Matters

In November 2020, NVIDIA announced the A100 80 GB. Same die. Same SMs. Same NVLink. The change was on the memory side: doubled HBM capacity (40 GB HBM2 → 80 GB HBM2e) and a 31% bandwidth bump (1.5 → 2.0 TB/s) thanks to faster HBM2e clocks.

Why this was a big deal for LLMs

40 GB held 13B in BF16 fine, struggled at 30B, and was helpless at 70B+ without sharding. 80 GB removed sharding from the workflow for an entire class of model sizes:

• Llama-65B / Llama-2-70B in BF16 — needs ~140 GB of weights; with TP=2 across two A100 80 GB, this fits comfortably with KV-cache headroom.
• OPT-66B fine-tuning — entire optimiser state plus weights fit on 8× A100 80 GB without ZeRO offload to CPU.
• Single-card 30B — full BF16 inference, no quantisation needed.

2026 used market

A100 80 GB SXM4 modules are still going strong on the second-hand market. They are the price-performance sweet spot for fine-tuning Llama-3-70B, Mixtral, and similar mid-tier models. Hopper and Blackwell are faster per-dollar for new builds, but a used A100 80 GB cluster remains a credible 2026 platform — especially for INT4 inference, where the bottleneck is bandwidth and not FP8 compute.

Performance for Modern LLMs

Where does Ampere sit in 2026? Pretraining at frontier scale has moved decisively to H100 / H200 / B200 — FP8 doubles compute and the Transformer Engine's per-tensor scaling is genuinely difficult to give up. But Ampere remains unbeatable in three places:

FP8 emulation on Ampere

You can “run” FP8 on A100 via libraries like vLLM's FP8-W8A8 path, but be clear about what's happening: the weights are stored as FP8 (saving HBM bandwidth on weight load), then dequantised into BF16 for tensor-core math. There is no compute uplift — throughput is the same as plain BF16. The win is purely a smaller working set.

For decode (memory-bound), this is still useful: you cut weight bandwidth in half and tok/s improves by 30–50%. For prefill (compute-bound), it's a wash. Hopper's native FP8 doubles compute as well; that's the meaningful gap.

Worked numbers: Llama-3-70B AWQ-INT4 inference

Two A100 80 GB SXM4 with TP=2 over NVLink, vLLM 0.5+, batch sizes from solo user to mid-batch:

A single A100 80 GB at INT4 holds Llama-3-70B with ~40 GB headroom for KV-cache. Two cards via NVLink approximately double throughput on prefill and add ~70–80% on decode (TP overhead).

Software Stack on Ampere

Ampere has the most mature software stack of any NVIDIA generation simply because it has been in production the longest. Five years of bug fixes, kernel tuning, and library work all assume CC 8.0 / 8.6 as the baseline.

The TF32 vs BF16 trap

If you're writing new transformer code, the right default is BF16 explicitly, not TF32. Reason: cuDNN/cuBLAS schedule a different (and faster) tensor-core kernel for BF16 matmuls than for TF32. TF32 is the “don't break my old code” format; BF16 is the “I'm writing this from scratch” format. On A100, BF16 matmul is faster than TF32 matmul (312 TFLOPS vs 156 TFLOPS) and accuracy is essentially identical for training transformers.

Inference / serving

• vLLM — first-class Ampere support since v0.1. PagedAttention exploits the 40 MB L2.
• TensorRT-LLM — Ampere kernels for AWQ, GPTQ, BF16, INT8 SmoothQuant.
• SGLang — supports Ampere, though some kernels are tuned for Hopper.
• Flash-Attention v2 — supports Ampere fully. Flash-Attention v3 is Hopper-only (uses TMA and warpgroup async); on A100, v2 remains the best option.
• llama.cpp / Ollama — CUDA backend works on Ampere; q4_K_M and q5_K_M are well-tuned.

import torch
# keep TF32 enabled for any leftover FP32 codepaths
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

# but actually train in BF16 explicitly
with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
    out = model(x)
    loss = criterion(out, y)
loss.backward()

Interactive: Ampere SKU Picker

Pick an Ampere SKU; see the headline numbers, what it means for LLM capacity at FP16 and INT4, and what it's actually good for.

“Largest LLM” is a back-of-envelope estimate: VRAM ÷ 2 (B params) for FP16 weight-only fit, VRAM ÷ 0.6 for INT4. Real workloads need additional headroom for KV-cache, activations, and CUDA overhead — subtract 10–20%.