Optical Technology for AI Processing — Presentation 03
Where Optics Could Win, and the Limits
Putting decks 01 and 02 to work: the conditions under which optical arithmetic beats digital, a map of where it wins, which AI operations fit, memory and Amdahl, precision for inference and training, the scaling argument for very large models, and the niches (latency, sensing, transforms) where the case is strongest.
Win conditions
Amdahl
Memory wall
1/d scaling
Latency
Scorecard
Large N
+
Low precision
+
Static weights
+
High utilisation
=
a chance
00
Topics We'll Cover
Concepts used here, and where they are explained. Each links to a glossary entry: this series glossary, or the glossaries of Fourier Optics for Inference and Training, Silicon Photonics Fundamentals and other sister series for concepts they already explain.
01
A Caution to Start
McMahon's review of the physics of optical computing argues that “the speed of light is not a key differentiating physical property of optics for computing”, and sets out eleven features of optics (parallelism, passive linear operations, low-loss data movement and others) from which an advantage could be built, if a design uses several of them and avoids the costs of conversion and precision (Nat. Rev. Phys. 5, 717, 2023; arXiv:2308.00088).
- So the question is never “is light fast?” but whether, for a given workload, the energy saved inside the optics exceeds the energy spent converting into and out of it, holding weights in it, and keeping it busy.
- The Fourier Optics series asked that question for Fourier engines in LLM inference and training and found the answer mostly negative (Fourier Optics for Inference; for Training). This deck asks it for general optical matrix multiplication, which covers far more of the work.
02
The Conditions for a Win
| Condition | Why | Where it was shown |
|---|
| Large matrices | Each conversion is shared by N MACs | OptAI 02, slide 01; N > 500 in Nahmias et al. |
| Low precision | Converter energy doubles and photon count quadruples per bit | OptAI 02, slide 05; ≤ 4 bits in Nahmias et al. |
| Static, reused weights | Changing weights costs a conversion per weight | OptAI 02, slide 01 |
| Low static power per weight | Paid at every use; 1 mW at 1 GHz is 1 pJ per MAC | OptAI 02, slide 08 |
| High utilisation | Lasers and bias burn power when idle | OptAI 02, slide 08 |
| Weights near the optics | Fetching weights from DRAM costs more than the MACs save | slide 05 |
Nahmias et al. summarise the first two: photonic MACs have their advantages for large processors, large vector sizes (N > 500) and low noise precision (≤ 4 bits) (IEEE JSTQE 26, 2020). The interactive on the next slide maps the first two conditions directly.
03
Interactive: Where Optics Wins
Every cell is the energy-per-MAC model of OptAI 02 at one matrix size and precision, compared with a digital reference. Green cells beat it, orange cells lose. The ADC figure of merit (with the DAC at half of it), laser efficiency, static power and reuse are illustrative inputs; the default digital reference is Horowitz's 8-bit MAC at 45 nm (OptAI 02, slide 11).
Try: set the digital reference to 0.05 pJ (a newer node), or a static power of 0.1 mW per weight, and watch the green region shrink; set weights to reload every vector, as attention scores do, and it retreats to the lowest precisions. The model's limits are listed under OptAI 02's interactive; memory is not in it (slide 05).
04
Which AI Operations Fit
| Operation | Optical fit | Why |
|---|
| Weight × activation matmuls (projections, MLP, convolutions) | Good | Static weights, reused over every token or batch element |
| Attention scores QKT and scores × V | Poor | Both operands are activations: the “weights” change every token, so every value needs a conversion (data-by-data products) |
| Softmax, normalisation, activations | Digital | Nonlinear, and small in FLOPs but on the critical path (OptAI 01, slide 10) |
| Fourier transforms and long convolutions | Good, where they exist | A lens computes them in one pass; rare in LLMs (slide 10) |
| Training's backward pass | Harder | Transposed weights, gradient precision, weight updates every step (slide 06) |
- Batching changes the picture: in a decode step with batch size 1, every weight is used once per token, so a weight-stationary optical engine's reuse must come from keeping weights in place across tokens; with larger batches or prefill, each weight serves many vectors.
- The equivalent analysis for a Fourier engine, with its much smaller share, is FOptInf 02's Amdahl analysis.
05
Memory and Amdahl
- Data movement dominates digital AI energy. At 45 nm an 8-bit multiply costs 0.2 pJ, a 64-bit SRAM access 10 pJ and a 64-bit DRAM access 1.3–2.6 nJ (Horowitz, ISSCC 2014). Making the MAC cheaper changes little if the weights still arrive from DRAM.
- So optics must hold its weights, in phase-change cells or programmed phases, for long enough to use them many times: a weight-stationary design. Model sizes far exceed what one optical array holds, so large models are tiled, and every reload is a conversion per weight (the interactive's reload setting).
- Amdahl's law applies at system level: if a fraction f of the energy is optical MACs and they become free, total energy falls by at most f. With memory, nonlinearities and control left digital, f is well below 1 (optical share and the Amdahl bound).
- Architecture models say the same: Andrulis et al. find conversion and DRAM can take a significant portion of a photonic accelerator's energy (arXiv:2405.07266); the classic survey of where accelerator energy goes is Sze et al. (Proc. IEEE 105, 2295, 2017).
06
Precision for Inference and for Training
Inference
- INT8 and FP8 inference is standard; matching INT8's error needs about 8 ENOB (the noise rule).
- Some tasks tolerate far less: about 90% digit accuracy at about 0.66 photons per multiplication (Wang et al., Nature Commun. 13, 123, 2022).
- Precision can vary by layer (Garg et al., arXiv:2102.06365).
Inference is the easier target: lower precision, static weights and a forward pass only. Most of what the companies in deck 04 describe is inference.
07
The Scaling Argument: 1/d
Anderson et al. studied Transformer inference with optical matrix multiplication and modelled how its energy scales with model size. Their key finding is that “optical energy per multiply-accumulate (MAC) scales as 1/d where d is the Transformer width”: wider models share each conversion and each photon among more MACs (arXiv:2302.10360; TMLR 2024).
| What the paper projects | Against |
|---|
| About 100× energy advantage for some of the largest current Transformers (about 500 billion parameters) | Digital processors at 300 fJ/MAC |
| More than 8,000× for models of a quadrillion parameters | The same |
- These are modelled projections, not measurements, and they rest on assumptions about converters, memory access and precision that the paper states; a further >100,000× case assumes cheaper memory access, better conversion and 4-bit precision.
- The argument is the reuse argument of OptAI 02, slide 01 taken to model scale: if an optical array were as wide as the model, conversions per MAC would fall as 1/d. The practical limit is how wide an array can be built (OptAI 02, slide 09), and whether weights can be held in it.
08
Latency Niches
| Result | Latency | Source |
|---|
| On-chip photonic network classifying images | Under 570 ps | Ashtiani et al. (Nature 606, 501, 2022) |
| Three-layer network with on-chip nonlinearities | 410 ps | Bandyopadhyay et al. (Nature Photon. 18, 1335, 2024) |
| Integrated photonic accelerator solving Ising problems, more than 16,000 photonic components, up to 1 GHz | As small as 3 ns per cycle | Hua et al., Lightelligence (Nature 640, 361, 2025) |
- Where latency is worth paying for: iterative solvers that loop many times through one matrix (Ising and combinatorial optimisation), control loops, signal processing on data that arrives fast, and decisions that must be taken within a clock cycle or two.
- Large-model serving is rarely in this class: its latency is dominated by memory bandwidth and batching, not by the time light takes to cross a chip (LLM Inference Simulators).
09
When the Data Is Already Light
- Imaging and sensing skip the input conversion. A scene arrives as light, so a diffractive or photonic front-end processes it before any detector and need only digitise a compressed result. ACCEL reports 4.6 peta-operations per second, over 99% of them optical, 74.8 peta-operations per second per watt and 72 ns per frame on vision tasks (Chen et al., Nature 623, 48, 2023; the authors' figures).
- Data arriving over fibre can be computed on before detection: Netcast multiplied at 40 aJ per multiply at an edge device whose weights arrived over 86 km of deployed fibre (Sludds et al., arXiv:2203.05466; Science, 2022). Compute on data in a link is the subject of Compute in Transit.
- Why this matters: removing the input DAC removes half of the conversion bill of OptAI 02, slide 07, and the scene's own pixels give an enormous N for free.
10
Transforms, Convolutions and Encryption
- Fourier transforms and convolutions are what a lens computes natively. For LLMs the Fourier Optics series found the transform share of prefill FLOPs small, even in FFT-mixing models (FOptInf 02), and the simulated optical prefill pools broke even only under favourable assumptions (FOptInf 03, the break-even point).
- Convolutional vision networks are a better fit: convolution is the bulk of their work and their weights are small and static (Xu et al.'s 11 TOPS convolutional accelerator, OptAI 01, slide 08).
- Fully homomorphic encryption spends much of its time in number-theoretic transforms, the finite-field cousin of the FFT; optical engines for it are analysed in FHESim 04, Optical NTT Engines, where exact arithmetic makes the precision demands severe. A company working on this is in deck 04.
11
A Scorecard
| Workload | N and reuse | Precision | Weights | Verdict |
|---|
| Vision on light that arrives optically | Very large (pixels) | Low | Static | strongest case |
| CNN inference | Large | INT8 or lower | Static, small | plausible |
| Iterative solvers (Ising, optimisation) | One matrix, many passes | Low | Static per problem | plausible, latency-driven |
| LLM inference: projections and MLP | Large in prefill, batch-dependent in decode | INT8/FP8 | Huge; must be tiled and reloaded | conditional on memory |
| LLM attention | Activation × activation | Higher | Change every token | poor |
| Training | Large | High for gradients | Change every step | hardest |
These verdicts are this series' engineering judgement from the conditions of slide 02 and the model of slide 03, not measurements; deck 04 sets them beside what companies say they have built.
12
What to Take Away
- Speed of light is not the advantage; shared conversions, passive linear operations and parallelism are, when a workload can use them.
- Optics wins at large N and low precision with static weights, low static power and high utilisation; the interactive shows how fast the winning region shrinks when any of these slips.
- Memory decides: a DRAM access costs thousands of MACs, so an optical engine must hold its weights; for large models that means tiling and reloads.
- Inference before training; weight × activation before attention.
- The scaling argument (energy per MAC as 1/d) projects 100× for today's largest Transformers and far more beyond, as a model, not a measurement.
- The clearest niches: data that is already light (vision, sensing, fibre), latency-bound iterative solvers, and convolution-heavy networks.
- Next: deck 04 surveys who is building what, in their own words.