How light can do the arithmetic of neural networks: the three physical facts it rests on, and the families built on them: free-space and diffractive optics, coherent Mach–Zehnder meshes, incoherent wavelength-multiplexed crossbars and photoelectric multiplication. With the results each has published, the nonlinearity problem, and training as well as inference.
Concepts used here, and where they are explained. Each links to a glossary entry: this series glossary, or the glossaries of Fourier Optics for Inference and Training, Silicon Photonics Fundamentals and other sister series for concepts they already explain.
Light doing part of a neural network's arithmetic: multiplying, summing or transforming values encoded in an optical field, with electronics around it to convert data in and out, apply nonlinearities and store results. Most of a neural network's operations are matrix multiplications, and a matrix–vector product is exactly what interference and propagation compute for free, so that is where almost every approach aims.
Optical matrix multiplication, optical convolution and transforms, diffractive networks, and the precision, energy and scale arguments around them. Decks 02 and 03 of this series.
Fourier optics for inference and training: Fourier Optics for Inference and for Training. The devices: Silicon Photonics Fundamentals. Linked, not repeated.
Optics that only moves bits between chips. That is interconnect, the subject of Optical Technology for Data Centres (slide 12).
Reviews: Wetzstein et al., “Inference in artificial intelligence with deep optics and photonics” (Nature 588, 39, 2020); Shastri et al., “Photonics for artificial intelligence and neuromorphic computing” (arXiv:2011.00111; Nature Photonics 15, 102, 2021); McMahon, “The physics of optical computing” (Nat. Rev. Phys. 5, 717, 2023).
Coherent fields add as complex amplitudes. Split an input, scale and phase-shift the copies, and recombine them: the output field is a weighted sum. A grid of such operations is a matrix–vector product (coherent and incoherent light).
Passing through a lens, a diffractive layer or a waveguide mesh applies a fixed linear transform to the whole field at once. A lens's is the Fourier transform (the lens as a Fourier transformer).
Photodiodes measure power, |E|2, losing sign and phase unless a reference beam is mixed in. Detection also multiplies, which photoelectric multiplication exploits (the phase problem; slide 09).
| Family | How it multiplies | Representative work | Slide |
|---|---|---|---|
| Free-space Fourier (4f) | A lens transforms, a mask multiplies in the Fourier plane: convolution | Covered in Fourier Optics for Inference | 04 |
| Diffractive networks | Light passes several engineered diffractive layers; the layers are the weights | Lin et al. 2018; Zhou et al. 2021 | 05 |
| Coherent MZI meshes | A programmable grid of interferometers applies a unitary (and, with attenuation, any) matrix | Shen et al. 2017 | 06 |
| Incoherent WDM crossbars | Each wavelength carries one input; attenuators or rings weight it; a photodiode sums | Feldmann et al. 2021; Tait et al. 2017; Xu et al. 2021 | 08 |
| Photoelectric multiplication | Input and weight both encoded in light; homodyne detection multiplies them | Hamerly et al. 2019; Sludds et al. 2022 | 09 |
| Hybrid diffractive–interference chips | Diffractive and interferometric stages on one chip, with analogue electronics | Taichi (Xu et al. 2024); ACCEL (Chen et al. 2023) | 05 |
A lens computes a two-dimensional Fourier transform in one pass, and a 4f system with a mask in the Fourier plane computes a convolution. The physics, the spatial light modulators that write data and masks, the precision and the conversion energy are worked through in FOptInf 01, Fourier Optics for Engineers, and are not repeated here.
A diffractive network is a stack of thin layers, each a grid of pixels that delays or attenuates the light passing through. Free-space propagation between layers connects every pixel to many in the next layer, so the stack is a fixed, trained linear network that computes as light passes (diffractive deep neural networks).
| Work | What it showed | Source |
|---|---|---|
| Lin et al. (UCLA) | 3D-printed layers at 0.4 THz; a five-layer network classifies handwritten digits at 91.75% in numerical blind testing | Science 361, 1004, 2018 |
| Zhou et al. (Tsinghua), diffractive processing unit | A reconfigurable unit with millions of neurons; a prototype from off-the-shelf parts reported several times the speed and over ten times the energy efficiency of GPUs, per the authors | Nature Photon. 15, 367, 2021 |
| ACCEL (Tsinghua) | An all-analogue photoelectronic chip for vision: 4.6 peta-operations per second (over 99% by optics), 74.8 peta-operations per second per watt, 72 ns per frame, as the authors report | Chen et al. (Nature 623, 48, 2023) |
| Taichi (Tsinghua) | A diffractive–interference photonic chiplet reporting 160 TOPS/W, and 91.89% accuracy on the 1,623-category Omniglot set | Xu et al. (Science 384, 202, 2024) |
A Mach–Zehnder interferometer with two phase shifters is a programmable 2 × 2 unitary. Arranged in a triangle (Reck) or a rectangle (Clements), N(N−1)/2 of them implement any N × N unitary; two meshes and a row of attenuators implement any matrix by its singular value decomposition (MZI meshes; devices in OptSiP 01, slide 08).
A rectangular mesh of N modes with random phases (a random matrix), run on random signed inputs twice: once ideal, once with Gaussian phase errors, splitter errors and loss in every MZI. The error is the RMS difference between the two outputs after removing the average loss; “equivalent bits” is −log2 of it. The error and loss values are inputs, not measurements; Fang et al.'s 0.01 rad and 1% are useful reference points (slide 06).
Limits: the phases are uniform random, not trained or calibrated, and the errors are static. A splitter error of 10 mrad moves its split ratio by about one percentage point, from 50:50 to 49:51. Real meshes are calibrated after fabrication (slide 06), which removes much of the static error and leaves drift and noise. Detector noise and conversion are in deck 02.
Put each input on its own wavelength, weight each wavelength with an attenuator, and sum all of them on one photodiode: a dot product with no interference and no phase to stabilise. Many photodiodes, each with its own row of weights, make a matrix–vector product (WDM).
| Work | What it showed | Source |
|---|---|---|
| Tait et al., silicon photonic weight banks | Microring resonators weight each wavelength; the circuit is shown to be isomorphic to a continuous-time neural network, with a simulated 294-fold speed-up over a conventional benchmark | Sci. Rep. 7, 7430, 2017 |
| Feldmann et al., photonic tensor core | Phase-change-material weights and a chip-scale frequency comb as the many-wavelength source; trillions of multiply–accumulates per second, bandwidth above 14 GHz limited by modulators and detectors | Nature 589, 52, 2021 |
| Xu et al., convolutional accelerator | A microcomb interleaving time, wavelength and space: more than 10 TOPS (11 in the title) on 250,000-pixel images, and 88% accuracy on handwritten digits with an optical convolutional network | Nature 589, 44, 2021 |
| Ríos et al., phase-change photonic memory | Up to eight transmission levels (3 bits) in one Ge2Sb2Te5 cell, non-volatile, switched with as little as 13.4 pJ | Nature Photon. 9, 725, 2015 |
Encode both the input and the weight in light, and interfere them on a balanced detector: the photocurrent is proportional to their product, and integrating over time sums many products. Hamerly et al. proposed this as a route to very large networks and derived a “standard quantum limit” set by detector shot noise, which “can be as low as 50 zJ/MAC”, below the Landauer limit for irreversible digital computing (Phys. Rev. X 9, 021032, 2019; a theoretical result).
| Experiment | Optical energy per multiplication | Accuracy | Source |
|---|---|---|---|
| Free-space optical network (Wang et al.) | About 3.1 detected photons; about 0.66 photons (2.5 × 10−19 J) | About 99%; about 90% (handwritten digits) | Nature Commun. 13, 123, 2022 |
| Netcast: weights broadcast over 86 km of deployed fibre to an edge device (Sludds et al.) | 40 aJ per multiply (fewer than one photon) | 98.8% (93%) | arXiv:2203.05466; Science, 2022 |
A neural network needs a nonlinearity between its linear layers. Optics is linear at the intensities of these systems, so most designs detect the light, apply the nonlinearity electronically, and convert back: an optical–electrical–optical round trip per layer, with its conversion energy and latency.
| Work | What it showed | Source |
|---|---|---|
| Ashtiani, Geers and Aflatouni | An on-chip photonic deep network classifying images directly from light in under 570 ps, “comparable with a single clock cycle” of digital platforms; above 93.8% (two-class) and 89.8% (four-class) on handwritten letters | Nature 606, 501, 2022 |
| Bandyopadhyay et al. | A three-layer, six-neuron network computing linear and nonlinear functions on one chip with 410 ps latency; above 92% on a vowel task | Nature Photon. 18, 1335, 2024 |
| Optics that moves data | Where it is covered |
|---|---|
| Pluggable and co-packaged optics, optical I/O chiplets | OptDC 01, OptDC 02 |
| Optical circuit switching | OptDC 03 |
| AI cluster networks | OptDC 04 |
| Computing on data while it moves through a link | Compute in Transit (a photonic link module that also computes) |