Optical Networks for AI Clusters

Topics We'll Cover

• Scale-Up, Scale-Out and Scale-Across
• The Traffic Collectives Create
• Scale-Up Domains: Copper Today
• Scale-Out Fabrics in Practice
• Fat-Trees, Rails and Rail-Only
• Interactive: How Many Transceivers a Cluster Needs
• Tori and Optical Switches
• Copper Against Optics
• Pluggable, Linear or Co-Packaged in the Fabric
• How Links Fail
• Scale-Across: Between Data Centres
• What to Take Away

Scale-Up, Scale-Out and Scale-Across

• Each level has its own bandwidth, reach and medium, and so its own optical question: whether optics can replace copper in scale-up (optical I/O), how many transceivers scale-out needs and what they draw (slide 06), and how much a training run can tolerate a slow link (scale-across).
• Parallelism strategies map onto the levels: tensor parallelism within scale-up, data and pipeline parallelism across scale-out (the traffic of each kind of parallelism).

The Traffic Collectives Create

• All-reduce sums gradients across data-parallel replicas. The ring algorithm sends and receives 2(N − 1)/N times the data per participant, which meets the bandwidth lower bound for the operation (Patarasuk and Yuan, J. Parallel Distrib. Comput. 69, 117, 2009): for large N, each accelerator moves about twice its gradient size every step, whatever the cluster's size (ring all-reduce).
• All-to-all in mixture-of-experts models sends tokens to the accelerators holding their experts: every participant talks to every other, with data-dependent sizes (MoE all-to-all).
• Tensor-parallel traffic is the heaviest and most latency-sensitive, so it stays inside the scale-up domain.
• Synchronous training waits for the slowest link: collectives finish when the last participant finishes, so a congested or failing link slows everything (slide 10). This is why AI fabrics are built non-blocking, and why congestion control matters more than in general-purpose networks.

Scale-Up Domains: Copper Today

• Why copper: inside a rack the links are short enough for passive cable, which draws no power and has no lasers to fail. Copper's reach at 200 Gb/s per lane is an objective of at least 1 m (deck 01, slide 10), so a copper scale-up domain is about the size of a rack.
• Where optics comes in: a scale-up domain larger than a rack, or one rack's GPUs spread out to cool them, needs reach that copper does not have at these rates. That is the case for optical I/O chiplets in the accelerator package (deck 02, slide 06), with energies that must approach copper's to be chosen.

Scale-Out Fabrics in Practice

• The common shape is a non-blocking folded Clos (fat-tree) of two or three tiers, every link optical except, sometimes, the shortest; the interactive on slide 06 counts what that costs in transceivers.
• The Llama 3 training run used Meta's RoCE cluster of 24,000 GPUs on a three-layer Clos network (Llama Team, arXiv:2407.21783); its failure statistics are on slide 10.

Fat-Trees, Rails and Rail-Only

• Fat-tree: Al-Fares, Loukissas and Vahdat built non-blocking data-centre networks from identical commodity switches arranged as a folded Clos; with k-port switches, three tiers join up to k³/4 hosts (ACM SIGCOMM 2008). With radix 64 that is 65,536 endpoints; with radix 128, 524,288.
• Rail-optimised: GPU i of every server connects to rail switch i (NVIDIA's SuperPOD, slide 04). Collectives between same-numbered GPUs stay within one rail, one hop away; traffic between rails crosses the server's own scale-up domain.
• Rail-only: Wang et al. observe that LLM training traffic between different rails is small and can be forwarded through the scale-up domain, and remove the spine altogether, reporting 38–77% lower network cost and 37–75% lower network power at the same training performance (arXiv:2307.12169; IEEE HOTI 2024). The saving is mostly the spine's switches and transceivers.

Interactive: How Many Transceivers a Cluster Needs

A non-blocking fat-tree of radix-k switches, sized for the number of endpoints (one scale-out port per accelerator), with a transceiver at each end of every optical link. The arithmetic: a t-tier tree has one link per endpoint at each of t tier boundaries, so 2·E·t transceivers if all are optical. Module power presets are cited (deck 01, slide 05); switch-box and accelerator powers are illustrative.

Limits: one scale-out port per accelerator at the switches' port rate (real clusters often have one 400G NIC per GPU into 800G switch ports split in two, or several NICs per GPU); no oversubscription; no spare ports; front-end, storage and scale-up networks excluded. A rail-only design (slide 05) removes the top tier's links.

Tori and Optical Switches

• A torus connects each accelerator directly to its neighbours in two or three dimensions, with no switches between them. It suits the nearest-neighbour and ring patterns of dense training and costs one link per neighbour, not a tree of switches; its weakness is many hops between distant chips and sensitivity to any failed node.
• Google adds optical circuit switches between 64-chip cubes: TPU v4 assembles 4,096 chips into tori of the shape each job asks for and routes around failures, with the OCSes under 5% of system cost and 3% of power (Jouppi et al., ISCA 2023; deck 03, slide 06).
• Ironwood, Google's 2025 TPU, scales a pod to 9,216 chips linked by its inter-chip interconnect, drawing nearly 10 MW (Google blog); a later post gives 9.6 Tb/s of ICI per chip and describes OCS rerouting around faults (Google Cloud blog). Vendor statements.
• Torus against fat-tree is a choice between switches and flexibility: a fat-tree serves any traffic pattern, including all-to-all, at the cost of switches and transceivers; a torus with OCS is cheaper per accelerator and reshapes per job, but suits patterns that fit its geometry.

Copper Against Optics

• The boundary moves inward with each lane-rate doubling, because copper's reach roughly halves. Systems therefore pack accelerators as densely as cooling allows, to keep scale-up links on copper.
• The next step for scale-up is optics that costs close to copper in energy and reliability: co-packaged optics in switches and optical I/O in accelerators (deck 02).

Pluggable, Linear or Co-Packaged in the Fabric

These choices multiply: the interactive on slide 06 shows that the transceiver count is set by topology (tiers × endpoints) and the power by module choice, so a design can attack either factor. The device-level reasons each module draws what it does are in OptSiP 03.

How Links Fail

Source: Llama Team, “The Llama 3 Herd of Models” (arXiv:2407.21783), Table 5: 466 job interruptions in a 54-day snapshot, 47 planned and 419 unexpected, about 78% of the unexpected ones attributed to confirmed or suspected hardware issues. Only the three largest named causes are listed here.

• One link stops the job. A synchronous training step needs every participant, so any failed link or flapping transceiver interrupts the whole run until it restarts from a checkpoint. 35 network interruptions in 54 days is more than one every two days.
• Scale multiplies rare failures. At a trade-press figure of about 200 FIT per transceiver (Herrick, Compound Semiconductor), 100,000 transceivers would see about 0.02 failures an hour, one every two days or so (arithmetic: 10⁵ × 200 failures per 10⁹ device-hours). The interactive's 3-tier, 16,384-endpoint fabric has nearly 100,000 transceivers.
• Remedies: more reliable lasers, or lasers kept outside packages where they can be replaced (deck 02, slide 07); spare capacity and fast rerouting (OCS in TPU v4, slide 07); fewer transceivers per accelerator (rail-only, copper in scale-up); and checkpointing that makes restarts cheap.

Scale-Across: Between Data Centres

• Power and buildings now limit cluster size as much as networks do, so some training runs span sites. Gemini Ultra was trained on TPU v4 across multiple data centres, with model parallelism inside superpods and data parallelism across them (Gemini Team, arXiv:2312.11805).
• Long-haul links carry far less than a fabric, so the algorithm must communicate less. DiLoCo trains replicas locally for many steps and synchronises rarely; with eight workers on C4 it matches fully synchronous training while communicating 500 times less (Douillard et al., arXiv:2311.08105).
• Optics here is coherent DWDM (direct and coherent detection), on the dense grid of OptSiP 01, slide 11.
• The break-even bandwidth between sites, and frequency-domain compression of what is sent, are worked through in the sister series: cross-data-centre training, DiLoCo and DeMo.

What to Take Away

• Three levels, three media: scale-up on copper inside a rack (NVL72: 72 GPUs, 130 TB/s), scale-out on optics across a hall (non-blocking fat-trees of 24,000 to 100,000 GPUs), scale-across on long-haul links with algorithms that communicate rarely.
• Collectives set the traffic: ring all-reduce moves about twice the gradient per accelerator per step; all-to-all and tensor parallelism need the most bandwidth.
• Transceiver count is topology: 2 × endpoints × tiers in a non-blocking fat-tree, about 100,000 for 16,384 accelerators in three tiers. Rail-only, OCS and copper attack the count; LPO and CPO attack the watts per transceiver.
• Links fail: 8.4% of Llama 3's unexpected interruptions were network switches or cables, and one failure stops a synchronous job. Reliability per transceiver matters as much as power.
• The optical future of AI networks is lower-power modules, co-packaged switches, optical switching of slowly changing layers and, eventually, optics in the scale-up domain. The devices beneath all of it are in Silicon Photonics Fundamentals; light used for computing rather than carrying is the subject of Optical Technology for AI Processing.