Optical Circuit Switching

Topics We'll Cover

• Packet Switching and Circuit Switching
• How a MEMS Optical Circuit Switch Works
• The Technologies Compared
• Google's Optical Spine
• Lightwave Fabrics: Circulators and WDM
• TPU v4: an Optically Reconfigurable Supercomputer
• Interactive: Reconfiguration and Spine Power
• Reconfiguration Time Against Traffic
• An Optical Switch in the Link Budget
• What Circuit Switching Cannot Do
• What to Take Away

Packet Switching and Circuit Switching

• So an OCS replaces the layer of a network whose connections change slowly: the spine between aggregation blocks, or the wiring between racks of accelerators. Packet switches stay where traffic must be sorted packet by packet.
• A survey of the earlier optical data-centre interconnect proposals is Kachris and Tomkos (IEEE Commun. Surveys Tuts. 14, 1021, 2012).

How a MEMS Optical Circuit Switch Works

• Free-space beam steering: each input fibre's beam hits its own micro-mirror, which tilts in two axes to aim it at the mirror of the chosen output, which aims it into that fibre. Any input can reach any output; switching is the time for the mirrors to move and settle.
• Google's Palomar is such a switch, with 136 × 136 ports; its MEMS package has 176 mirrors of which 40 are disabled for yield, and the whole switch draws at most 108 W (Liu et al., “Lightwave Fabrics”, ACM SIGCOMM 2023). Its insertion loss is typically under 2 dB and it switches in milliseconds (Urata et al., “Mission Apollo”, arXiv:2208.10041).
• 108 W across 136 ports is under 1 W per port, against a packet switch port plus its transceiver at the other end of a fibre (slide 07).

The Technologies Compared

• The trade is speed against loss and scale: free-space MEMS gives hundreds of low-loss ports but switches in milliseconds; integrated silicon switches reach hundreds of nanoseconds at several dB of loss; wavelength routing switches in nanoseconds but needs fast tunable lasers at every endpoint.

Google's Optical Spine

• Jupiter Evolving: Google moved its data-centre fabric from a Clos with a packet spine to a direct-connect topology among aggregation blocks, through a datacenter interconnection layer of MEMS optical circuit switches that allows dynamic reconfiguration (Poutievski et al., ACM SIGCOMM 2022).
• Reported results: 5× higher speed and capacity, 30% lower capital cost and 41% lower power; an average block-level path length of 1.4; and 3× faster fabric reconfiguration than the earlier patch-panel interconnect.
• Topology engineering: because the OCS layer decides which blocks are wired to which, the topology itself is tuned to the measured traffic matrix, alongside traffic engineering of the paths over it. The spine's packet switches, and the transceivers that terminated links on them, are gone.
• Rate agnosticism pays over time: blocks of different generations and line rates connect through the same OCS layer, so the fabric is upgraded block by block rather than all at once.

Lightwave Fabrics: Circulators and WDM

• Circulators halve the switch. An optical circulator lets one fibre carry both directions of a link, so the OCS needs one port per link instead of two: Liu et al. use circulators “to have a bi-directional link, which halves the number of OCS ports and fiber cables” (ACM SIGCOMM 2023).
• Reflections then matter: on a bidirectional fibre, light reflected back towards the transmitter lands on the receiver of the same port. Apollo specifies the switch's return loss at −38 dB (−46 dB typical) for this reason (arXiv:2208.10041).
• WDM transceivers put several wavelengths on each fibre (four channels, in the snippets of the paper available for this deck), multiplying what each OCS port carries (WDM).
• The payoff reported for the TPU system: up to 3× better system availability and up to 3.3× model-dependent performance improvement over a static fabric, with the optical circuit switching under 6% of total system cost (Liu et al. 2023).

TPU v4: an Optically Reconfigurable Supercomputer

Jouppi et al. describe a 4,096-chip TPU v4 machine whose chip-to-chip interconnect runs through optical circuit switches (ISCA 2023; arXiv:2304.01433).

• Three uses of one switch: carve the machine into slices of the shape a job needs; route around failed racks without recabling; and choose topologies (twisted tori) that suit the job's communication.
• Milliseconds is fast enough because the topology changes per job or per failure, not per packet. Packets inside a slice travel over fixed circuits between chips.
• How the torus compares with fat-trees for AI traffic is in deck 04.

Interactive: Reconfiguration and Spine Power

Two questions about putting an OCS in a network. How often can it reconfigure? The fabric carries traffic for the fraction 1 − t_reconfigure/t_period of the time. What does it save? Replacing a packet spine of N ports removes a switch port and a transceiver per spine port and adds the OCS's own power. Presets set the reconfiguration time from the sources of slide 03 (MEMS: Calient's under 50 ms per port, vendor; silicon photonic MEMS: under 400 ns; tunable lasers: 3.84 ns). The OCS's 0.8 W per port is Palomar's 108 W over 136 ports, and the transceiver's 17 W an 800G datasheet maximum (deck 01). The packet-switch power per port is illustrative.

Limits: the dark time assumes all circuits reconfigure together and carry nothing meanwhile; real systems drain traffic first and reconfigure part of the fabric at a time. The power comparison counts only the spine layer, not the aggregation blocks either side, and ignores the cost of circulators and of transceivers that tolerate the OCS's extra loss.

Reconfiguration Time Against Traffic

• Hybrid designs added an OCS beside a packet network to carry large, stable flows: Helios (Farrington et al., ACM SIGCOMM 2010) and c-Through (Wang et al., ACM SIGCOMM 2010).
• RotorNet makes switching traffic-independent: rotor switches cycle through a fixed set of matchings, so no central scheduler is needed, with switches that scale to thousands of ports (Mellette et al., ACM SIGCOMM 2017).
• Sirius goes to nanoseconds: tunable lasers with passive gratings, reconfiguring end to end in 3.84 ns on 50 Gb/s channels, with up to 74–77% lower power than an electrical network in simulation (Ballani et al., ACM SIGCOMM 2020).
• The rule, which the interactive makes concrete: the switch's reconfiguration time must be small against how long a configuration is used. MEMS suits topologies that change per job; per-packet switching needs nanoseconds.

An Optical Switch in the Link Budget

• Every OCS adds its insertion loss to every link through it: typically under 2 dB for Palomar, 1.8 dB typical for Calient's S320, up to 9.8 dB on chip for Seok et al.'s silicon switch (slide 03). The transceivers must be specified to tolerate it, and every extra decibel is paid in laser power at the wall (optical link budget; try it in the OptSiP 03 interactive).
• Return loss matters on bidirectional circulator links (slide 05), and connectors either side of the switch add their own loss and reflections.
• The light must survive without regeneration: an OCS has no receiver to clean the signal up, so several in series, or a lossy integrated switch, quickly use up a short-reach module's margin. This favours low-loss free-space switches in the spine, and single stages.
• Wavelength and polarisation: MEMS mirrors reflect all wavelengths alike, so WDM transceivers pass through unchanged; integrated switches can be wavelength- and polarisation-dependent (OptSiP 01, slide 03).

What Circuit Switching Cannot Do

• No per-packet decisions and no buffering. An OCS cannot absorb bursts or mix many small flows onto one port; that still needs packet switches at the edges, and a topology whose circuits match the traffic.
• It needs a control plane that knows the traffic. Topology engineering (Jupiter) or per-job slicing (TPU v4) works because traffic is predictable over the switch's reconfiguration time. Unpredictable all-to-all traffic, such as mixture-of-experts dispatch, suits a full packet fabric better (MoE all-to-all).
• Reconfiguring drops traffic on the circuits that move, so it must be drained first: the dark time of the interactive.
• Mechanical parts in MEMS and robotic switches raise reliability questions that solid-state switches do not; the published systems report gains in availability, because routing around failures outweighs the switch's own failure rate (slide 06).

What to Take Away

• An OCS steers light without detecting it: no transceivers inside, under 1 W per port (Palomar: 108 W for 136 ports), and agnostic to line rate, so it survives optics upgrades.
• Technologies trade speed for loss and scale: MEMS gives hundreds of ports at under 2 dB in milliseconds; silicon photonic switches reach under 400 ns at several dB; tunable lasers switch in nanoseconds.
• Google's evidence: an OCS spine gave 30% lower capital cost and 41% lower power (Jupiter); in TPU v4 the OCSes cost under 5% of the system and under 3% of its power while slicing the machine and routing around failures.
• The rule: reconfiguration time must be small against how long a topology is used; milliseconds suit per-job topologies, not per-packet switching.
• Every OCS adds loss to every link through it, paid for in the transceivers' budget.
• Next: deck 04 puts pluggables, CPO and OCS into the networks of AI clusters.