Optical Technology — Optical Technology for Data Centres

Optical Technology for Data Centres

How light carries data centre and AI cluster traffic, and what it costs: pluggable modules and the linear-drive optics that remove their DSPs, co-packaged optics on the switch package, optical circuit switches that replace packet spines, and the scale-up and scale-out networks of AI clusters. Device physics comes from Silicon Photonics Fundamentals and is linked, not repeated.

PluggablesLPOCo-packaged opticsOCSFat-treesAI clusters4 decks live

Presentations in This Series

  1. Pluggable Transceivers to Linear-Drive Optics →
    What is inside a pluggable optical module and where its power goes: form factors and MSAs, the Ethernet optical interfaces from 400G to 1.6T, host electrical lanes, the retiming DSP, and what linear-drive (LPO) and linear-receive (LRO) modules save by removing it and what they give up.
    liveQSFP-DD / OSFP · IEEE 802.3 · DSP retimer · LPO
  2. Co-Packaged Optics and Optical I/O →
    Moving the optics from the faceplate onto the switch or processor package: why the electrical channel is the problem, optical engines and external lasers, the OIF agreements, what vendors have announced and claim, serviceability, heat and fibre, and optical I/O chiplets for compute.
    liveCPO · Optical engine · ELSFP · XSR
  3. Optical Circuit Switching →
    Switching light without converting it: packet against circuit switching, MEMS, piezo, liquid-crystal, silicon-photonic and wavelength-routed switches, Google's optical spine and TPU v4 interconnect, reconfiguration time against traffic, and what an optical switch costs in the link budget.
    liveOCS · 3D MEMS · Topology engineering · TPU v4
  4. Optical Networks for AI Clusters →
    The networks that join accelerators: scale-up domains and scale-out fabrics, the traffic collectives create, fat-trees, rail-optimised and rail-only designs, tori with optical switches, copper against optics, how many transceivers a cluster needs and what they draw, and how links fail.
    liveScale-up · Scale-out · Fat-tree · Rail-optimised

How to read this series. Deck 01 is the module at the faceplate and where its watts go, and what linear drive saves. Deck 02 moves the optics onto the switch package. Deck 03 switches light itself, as Google does in its spine and TPU interconnect. Deck 04 puts them together in the networks of AI clusters, with a calculator for how many transceivers a cluster needs. Each deck has one interactive.

Where the numbers come from. This is mostly a literature series. Every number is cited on its slide to a paper, a standard (IEEE 802.3, OIF, ITU-T), an MSA or a datasheet, with a link, or computed live by a deck's interactive. Much of what is public about data-centre optics comes from vendors: their press releases and datasheets are cited and labelled as vendor claims, and the interactives' coefficients are labelled as illustrative where no source fixes them. Standards in progress (IEEE P802.3dj) are called drafts.

Built on Silicon Photonics Fundamentals. Modulators, photodiodes, lasers, WDM, packaging and the link budget are explained in Silicon Photonics Fundamentals; concept chips and glossary entries link to its glossary. The traffic these networks carry is worked through in Compute in Transit.

Glossary: Concepts and Where They Are Explained

Every data-centre concept the decks rely on, with a short explanation and links to the slides that explain it. Device concepts are linked to the Silicon Photonics Fundamentals glossary, and the traffic of training and inference to Compute in Transit, rather than repeated.

Pluggable optics

Pluggable transceiver and MSA
A self-contained optical link end that slides into a faceplate cage: electrical lanes on one side, fibre on the other. A multi-source agreement fixes the cage, connector, management interface and power envelope, so switches and optics come from different suppliers and failed modules are swapped per port.Explained in: OptDC 01 · what a pluggable module is
QSFP-DD and OSFP
The two eight-lane form factors of 400G and 800G pluggables. QSFP-DD's power rating rose from 20 W (2019) to 25 W (2021); OSFP has an integrated heat sink for up to 15 W with conventional airflow; OSFP-XD doubles the lanes to 16.Explained in: OptDC 01 · what a pluggable module is · OptDC 01 · faceplates, heat and the switch
Retiming DSP
The digital signal processor in a module that recovers clock and data from the lossy host trace and re-times them for the optics, and the reverse on receive. It splits one hard link into two separately specified ones, at a cost of several watts per module.Explained in: OptDC 01 · inside a DSP-retimed module · OptDC 01 · where a module's power goes
Ethernet optical interfaces: SR, DR, FR, LR
The IEEE 802.3 and MSA optical interfaces. SR: multimode VCSELs, about 100 m. DR: one wavelength per single-mode fibre pair, 500 m. FR and LR: four or eight wavelengths on a duplex fibre, 2 and 10 km. 200 Gb/s-per-lane versions are in the IEEE P802.3dj draft.Explained in: OptDC 01 · the optical interfaces · OptDC 01 · the reach ladder
Host electrical interface (VSR)
The channel from a switch's SerDes to a pluggable module: package, board trace and connector. OIF's CEI-112G-VSR allows up to 16 dB of loss at the Nyquist frequency, including one connector, at 100 Gb/s per lane.Explained in: OptDC 01 · the host electrical interface · OptDC 02 · why move the optics into the package
Module power
What a pluggable draws: an 800G datasheet gives 17 W maximum; DSP vendors claim 800G modules below 11–12 W. At 800 Gb/s, 11–17 W is about 14–21 pJ/bit, most of it DSP, drivers, TIAs and lasers.Explained in: OptDC 01 · where a module's power goes · OptDC 01 · interactive module power · OptDC 04 · interactive transceiver count
Linear-drive pluggable optics (LPO)
A pluggable with no DSP: linear drivers and TIAs pass the waveform through, and the switch's SerDes equalises the whole link. Vendors claim roughly half the module power; the LPO MSA (formed 2024) released its first specification, for 100 Gb/s-per-lane single-mode modules, in 2025.Explained in: OptDC 01 · linear-drive pluggable optics · OptDC 01 · interactive module power · OptDC 01 · what linear drive gives up
Linear receive optics (LRO)
A half-way module: retimed transmit, linear receive. The launched optical signal stays standard and interoperable while the receive side goes linear; described by one vendor as saving roughly half as much as LPO.Explained in: OptDC 01 · linear receive
Copper reach
How far passive twin-axial copper carries a lane: at least 2 m at 100 Gb/s per lane (IEEE 802.3ck) and an objective of at least 1 m at 200 Gb/s (P802.3dj). It roughly halves with each lane-rate doubling, which pushes optics towards the chip.Explained in: OptDC 01 · the reach ladder · OptDC 04 · copper against optics
Switch capacity and radix
A switch chip's total bandwidth and its number of ports: 51.2 Tb/s is 64 × 800G, 102.4 Tb/s is 128 × 800G or 64 × 1.6T. Radix sets how many endpoints each tier of a fat-tree can join.Explained in: OptDC 01 · faceplates, heat and the switch · OptDC 04 · fat-trees, rails and rail-only · OptDC 04 · interactive transceiver count

Co-packaged optics

Co-packaged optics (CPO)
Optical engines on the switch or processor package substrate, millimetres from the die, instead of pluggable modules at the faceplate. It removes the host trace and the module DSP; switch vendors claim 50–70% lower optical interconnect power, at the cost of field replacement of optics.Explained in: OptDC 02 · why move the optics into the package · OptDC 02 · anatomy of a co-packaged switch · OptDC 02 · what vendors have announced
Optical engine
The silicon photonic chip, with its drivers and TIAs, that converts a group of electrical lanes to light on the package: for example 6.4 Tb/s per engine, eight around a 51.2 Tb/s switch.Explained in: OptDC 02 · anatomy of a co-packaged switch
XSR and XSR+ electrical lanes
OIF's extra-short-reach lanes from a die to another die or to an optical engine: CEI-112G-XSR reaches up to 50 mm; XSR+ allows up to 13 dB bump to bump, for multi-chip modules, CPO and near-packaged optics.Explained in: OptDC 02 · why move the optics into the package · OptDC 02 · the OIF agreements · OptDC 02 · interactive energy along the path
OIF co-packaging agreements
The Co-Packaging Framework (2022), the 3.2 Tb/s co-packaged module (2023: 8 × 400G FR4 or DR4 optics behind 32 XSR lanes), the external laser module ELSFP (2023, revised 2025) and the XSR+ lane (2024).Explained in: OptDC 02 · the OIF agreements
Optical I/O chiplet
An optical engine inside a processor package, for scale-up and memory traffic, typically with many wavelengths at modest rates per wavelength: Intel's prototype runs 64 lanes at 32 Gb/s on 8 wavelengths per fibre; vendors claim about 5 pJ/bit.Explained in: OptDC 02 · optical I/O for compute · OptDC 04 · scale-up domains
Near-packaged optics
Optical engines on the board next to the package rather than on it: a shorter electrical channel than pluggables, with some separation of optics and switch. One step on the path from faceplate to die.Explained in: OptDC 02 · near-packaged optics
Serviceability of co-packaged optics
What CPO gives up and how designs recover it: lasers kept in front-panel pluggable modules, known-good engines before assembly, spare lanes, and detachable fibre connectors.Explained in: OptDC 02 · serviceability and reliability · OptDC 02 · fibre at scale
Fibre count
Data fibres needed for a given bandwidth: lanes ÷ wavelengths per fibre, doubled for two directions. 51.2 Tb/s of 100G lanes needs 1,024 fibres with one lane per fibre, 256 with four wavelengths per fibre, plus laser feeds.Explained in: OptDC 02 · fibre at scale

Optical circuit switching

Optical circuit switch (OCS)
A switch that steers light from input to output fibres without detecting it: no transceivers inside, under 1 W per port for Google's Palomar, and agnostic to line rate. It sets circuits, not packets, and reconfigures in nanoseconds to milliseconds depending on technology.Explained in: OptDC 03 · packet and circuit switching · OptDC 03 · how a MEMS OCS works · OptDC 03 · the technologies compared
3D MEMS optical switch
Two arrays of two-axis micro-mirrors that aim each input beam at any output fibre. Hundreds of ports at under 2 dB typical loss, switching in milliseconds: Google's Palomar is 136 × 136.Explained in: OptDC 03 · how a MEMS OCS works · OptDC 03 · the technologies compared
Reconfiguration time and duty cycle
How long a circuit switch takes to change its connections, against how long a configuration is used: the fabric carries traffic for 1 − treconfigure/tperiod of the time. Milliseconds suit per-job topologies; per-packet switching needs nanoseconds.Explained in: OptDC 03 · interactive reconfiguration · OptDC 03 · reconfiguration time against traffic
Direct connect and topology engineering
Wiring aggregation blocks to each other through an OCS layer instead of through a packet spine, and choosing that wiring to match the traffic matrix. Google reports 30% lower capital cost and 41% lower power for its Jupiter fabric.Explained in: OptDC 03 · Google's optical spine
Optical circulator
A non-reciprocal device that lets one fibre carry both directions of a link, halving OCS ports and fibres; it makes the switch's return loss matter, since reflections reach the same port's receiver.Explained in: OptDC 03 · lightwave fabrics
Optically reconfigurable torus (TPU v4)
4,096 TPU v4 chips in 64-chip cubes joined through 3D MEMS OCSes into tori of the shape each job asks for, routing around failed racks; the OCSes are under 5% of system cost and 3% of power.Explained in: OptDC 03 · TPU v4 · OptDC 04 · tori and optical switches
Fast optical switching (RotorNet, Sirius)
Research designs that switch light far faster than MEMS: RotorNet cycles rotor switches through fixed matchings with no central scheduler; Sirius uses tunable lasers and passive gratings to reconfigure end to end in 3.84 ns.Explained in: OptDC 03 · reconfiguration time against traffic · OptDC 03 · the technologies compared
OCS insertion loss in the link budget
The loss every link pays to cross an optical switch: typically under 2 dB for MEMS switches, several dB for integrated silicon switches. With no regeneration inside, it comes straight out of the transceivers' margin.Explained in: OptDC 03 · an optical switch in the link budget · OptDC 03 · interactive reconfiguration

AI cluster networks

Scale-up domain
Accelerators joined by a very high-bandwidth, low-latency fabric with memory semantics: NVLink (NVL72: 72 GPUs, 130 TB/s), UALink (up to 1,024 accelerators at 200 Gb/s per lane), Google's ICI. Mostly copper inside a rack today.Explained in: OptDC 04 · scale-up, scale-out and scale-across · OptDC 04 · scale-up domains
Scale-out fabric
The NIC-based network joining thousands to hundreds of thousands of accelerators: InfiniBand or Ethernet (RoCE, Ultra Ethernet), built as a non-blocking fat-tree, mostly optical.Explained in: OptDC 04 · scale-up, scale-out and scale-across · OptDC 04 · scale-out fabrics in practice
Fat-tree (folded Clos)
A non-blocking network of identical switches in tiers; with k-port switches, three tiers join up to k3/4 endpoints (65,536 at radix 64). Every tier boundary carries one link per endpoint, so a t-tier tree needs 2·E·t transceivers if all links are optical.Explained in: OptDC 04 · fat-trees, rails and rail-only · OptDC 04 · interactive transceiver count
Rail-optimised and rail-only networks
Rail-optimised: GPU i of every server connects to rail switch i, so same-numbered GPUs are one hop apart. Rail-only removes the spine and forwards the little cross-rail traffic through the scale-up domain, reporting 38–77% lower network cost and 37–75% lower power.Explained in: OptDC 04 · fat-trees, rails and rail-only · OptDC 04 · pluggable, linear or co-packaged
Torus interconnect
Accelerators linked directly to their neighbours in two or three dimensions with no switches between: cheap per chip and good for ring and neighbour patterns, poorer for all-to-all and sensitive to failed nodes unless optical switches reroute.Explained in: OptDC 04 · tori and optical switches
Transceiver count of a cluster
Two per optical link: 2 × endpoints × tiers for a non-blocking fat-tree, about 98,000 for 16,384 accelerators in three tiers. Topology (rail-only, OCS, copper) sets the count; module type (DSP, LPO, CPO) sets the watts each.Explained in: OptDC 04 · interactive transceiver count · OptDC 04 · pluggable, linear or co-packaged
Link failures in training
A synchronous training job stops when any participant's link fails. In Llama 3's 54-day snapshot on 16,384 GPUs, 35 of 419 unexpected interruptions (8.4%) were network switches or cables; at a trade-press 200 FIT per transceiver, 100,000 transceivers fail about once every two days.Explained in: OptDC 04 · how links fail · OptDC 02 · serviceability and reliability
RoCE and Ultra Ethernet
Ethernet for AI fabrics: RDMA over Converged Ethernet carries GPU-to-GPU transfers on Ethernet switches (Meta's 24,576-GPU cluster); the Ultra Ethernet Consortium's 1.0 specification (2025) defines a new transport and congestion control for AI and HPC.Explained in: OptDC 04 · scale-out fabrics in practice
Scale-across
Training one model across several data centres over long-haul coherent links, with algorithms that synchronise rarely (DiLoCo: 500 times less communication on its benchmark).Explained in: OptDC 04 · scale-up, scale-out and scale-across · OptDC 04 · scale-across

Explained in the sister series

NRZ and PAM4 · Mach–Zehnder modulator (MZM) · Transimpedance amplifier and receiver sensitivity · CWDM, LAN-WDM and DWDM grids · Wavelength-division multiplexing (WDM) · External laser source and ELSFP · Electronic–photonic integration · Fibre attach and photonic packaging · Thermal tuning and wavelength locking · Thermo-optic effect · Multi-wavelength sources and combs · Wall-plug efficiency · Energy per bit · Optical link budget · Direct and coherent detection · Bandwidth scaling: lanes, wavelengths and symbol rate · Dark current, capacitance and bandwidth · Photonic link module · Ring all-reduce · MoE all-to-all · The traffic of each kind of parallelism · Low-communication training (DiLoCo family) · DeMo (decoupled momentum) · Cross-data-centre training and break-even bandwidth.