The question turned around: how is optical hardware itself trained, and what does that teach about putting optics in an LLM training loop? In-silico training and the reality gap, the adjoint method, physics-aware training, direct feedback alignment, forward-only methods, analogue noise and a measured case of it inside DeMo's feedback loop, the ADC precision a training format asks for, and a live experiment that trains one distorted filter three ways.
Concepts used here, and where they are explained. Each links to a glossary entry: this series glossary, or the Fourier Optics for Inference glossary for the physics and precision concepts that series already explains.
Deck 01 asked whether a Fourier-optical engine could take over part of an LLM training step, and found the share small: 0.15% of a step for a Hyena-2 model at 4,096 tokens, 12.19% with block-circulant weights (results.md §2). In that picture the LLM's weights live in digital memory, the optimizer is digital, and the engine computes fixed transforms on values it is handed.
This deck asks the other question. The optics has parameters of its own: phase shifters in a mesh, pixels on a mask, the alignment of a lens. How are they set, when no digital model of the hardware is exact? Photonic-computing research has answered this in several ways, each with a different answer to “who computes the gradient”:
| Approach | Who computes the gradient | Slides |
|---|---|---|
| In-silico | A digital model, before the hardware exists | 03 |
| In-situ backpropagation | The hardware, by interfering forward and adjoint fields | 04–05 |
| Physics-aware training | A digital twin, fed the hardware's real outputs | 06 |
| Direct feedback alignment | A fixed random projection of the error (done optically) | 07 |
| Forward-only methods | The hardware, from forward passes alone | 08 |
The lesson for an LLM engine comes at the end (slide 14): these methods matter for calibrating the engine's own parameters, not for training the LLM. The physics of meshes, SLMs and detectors is in FOptInf 01 and is not repeated here.
Backpropagation on a GPU rests on three assumptions that optical hardware breaks: the forward function is known exactly, every intermediate value can be stored and read back, and a weight can be written to the precision the optimizer asks for.
| Digital training | Training optical parameters | |
|---|---|---|
| Forward function | Exactly the code | A physical process: fabrication error, crosstalk, loss, temperature drift. A model of it is approximate |
| Intermediate values | Stored activations, read back for the backward pass | Fields inside a mesh or at a Fourier plane, visible only through detectors that measure intensity, not phase (the phase problem) |
| Gradient | Computed by the chain rule | Must be measured on the hardware, modelled by a twin, or avoided |
| Parameter writes | Any float, any time | A heater current or SLM pixel with finite resolution and settling time |
| Read-out | Exact to the number format | An ADC at finite ENOB, plus noise |
Two kinds of error result, and they behave differently in training. Systematic error (the model is wrong) biases every gradient the same way; random error (read-out noise) mostly averages out over steps. The rest of the deck is about which method copes with which, and the experiment on slide 12 shows both.
The simplest method: train a digital model of the optics, then build or program the hardware with the result. Lin et al.'s diffractive deep neural network (D²NN) did exactly this: layers of diffractive elements whose transmission was trained in simulation, then 3D-printed and run at terahertz frequencies as a handwritten-digit classifier and an imaging lens (Lin et al., Science 361, 1004, 2018). Training was entirely digital; the optics only ran inference.
It works when the model is good, and it is the only option for a passive, fabricated device. Its weakness is the reality gap: the trained parameters are optimal for the model, not for the hardware. Misalignment between layers, fabrication tolerance, material absorption and detector response all move the real device away from the model, and nothing in the training loop sees it.
In-silico training converges to the model's optimum. More steps, a better optimizer or a bigger batch only get closer to that point; the error from the mismatch stays. The only fixes are a better model or putting the hardware in the loop. The experiment on slide 12 shows the plateau.
Hughes, Minkov, Shi and Fan showed that a reciprocal photonic circuit can measure its own gradient (Optica 5, 864, 2018). The adjoint method, long used for inverse design, says the derivative of a loss with respect to a phase shifter depends only on the forward field and the adjoint field at that shifter. The adjoint field is what you get by sending the output error backwards through the same circuit; reciprocity makes that the transpose the chain rule needs.
Because detectors measure intensity, the cross term is recovered from three intensity measurements at each phase shifter: the forward field, the adjoint field, and the two interfering. The gradient for every phase shifter comes out of the same three passes, in parallel, as the chain rule's dot product does on a GPU. What the method asks for is reciprocity, coherent sources that can inject a chosen adjoint field, and an intensity monitor beside every parameter.
The identity |a + b*|² = |a|² + |b|² + 2 Re(a·b) is what turns three intensities into the product of two fields. Sign and scale constants are in the paper.
Pai et al. implemented the scheme on a programmable silicon-photonic MZI mesh and trained small classifiers with gradients measured on the chip (Science 380, 398, 2023). The hardware computed its own gradient, so whatever the chip actually does, the gradient is of that, not of a model of it.
In a 4f engine the parameters are the Fourier-plane mask pixels. For y = h ∗ u, the gradient with respect to the mask H is G·conj(U), the filter gradient of deck 01's correlation rule. An adjoint pass would put the error spectrum and the input spectrum at the same plane, so a coherent camera there could measure the data-by-data product that deck 01 said could not sit on a mask. It needs phase-sensitive detection at every pixel (the phase problem) and is speculation: no such measurement at LLM scale has been published.
Wright et al. proposed a hybrid: run the forward pass on the physical system, and the backward pass through a differentiable digital model of it (Nature 601, 549, 2022). They trained three different physical systems this way: broadband optical second-harmonic generation, a mechanical oscillating plate, and an analogue electronic circuit.
For a linear filter the 90° condition is per frequency: physics-aware training converges if the twin's phase error stays below 90° at every frequency, and stalls or diverges where it does not. The experiment on slide 12 computes the worst phase error of its hidden distortion live; push the distortion until it passes 90° and watch the physics-aware curve fail while the in-situ curve does not.
Direct feedback alignment (DFA) drops the backward pass altogether. Each hidden layer receives the output error multiplied by its own fixed random matrix, instead of the error backpropagated through the transposed weights; the forward weights learn to align with the feedback (Nøkland, NeurIPS 2016, arXiv:1609.01596).
δl = (Wl+1T δl+1) ⊙ f′(al): sequential, layer by layer, through the transposed forward weights.
δl = (Bl e) ⊙ f′(al): every layer at once, from one fixed random Bl. No weight transport, no model of the forward path.
The fixed random projection B e is a large dense matrix multiply that never changes, which is what light through a scattering medium computes: a fixed random transmission matrix, applied in one pass, read out by a camera. Launay et al. built a photonic co-processor that does this projection and trained networks with DFA on it (arXiv:2012.06373, 2020).
The projection needs no calibration because it only has to be fixed, not known: an elegant fit for optics. But DFA is a different learning rule, and its accuracy trails backpropagation on harder tasks; published evidence that it matches backpropagation when training large transformers is lacking. For LLM training, the optical part is a random matrix multiply, not a Fourier transform, and the method changes the algorithm the model is trained with.
Two 2023–2024 results train physical systems without any digital backward pass and without a model of the hardware:
Momeni et al. train each physical layer with a local objective computed from forward passes alone, in the spirit of Hinton's forward-forward algorithm, and demonstrate it on acoustic, microwave and optical systems (Science 382, 1297, 2023). No error is propagated between layers; no layer needs a model of the others.
Xue et al. use spatial symmetry and Lorentz reciprocity so that the backward propagation of the error can itself be carried out as a forward propagation through the physical system, measured rather than modelled, in free-space and integrated optical networks (Nature 632, 280, 2024).
Stochastic gradient descent already lives with noise: a minibatch gradient is a noisy estimate of the full one. What matters about analogue error is not its size alone but whether it is zero-mean.
| Error | Example | In the forward pass | In the gradient |
|---|---|---|---|
| Zero-mean, independent | ADC quantisation with dither, shot and thermal noise | Acts like injected noise; training tolerates it, and noise injection is a known regulariser | Adds variance to the minibatch noise; averages out over steps, at a cost in convergence speed |
| Biased, systematic | Gain and phase error, saturation, offset, crosstalk, a mis-modelled twin | The model computes a different function; in-silico training never sees it | Shifts every gradient the same way; does not average out; moves the converged point |
| Slow drift | Temperature, laser power, SLM ageing | Biased within a window, changing across windows | Tracked by in-loop training; must be recalibrated otherwise |
The same distinction appears in low-precision digital training: rounding every update to nearest loses updates smaller than half a step, a biased error, while stochastic rounding is zero-mean and keeps them in expectation (Gupta et al., ICML 2015, arXiv:1502.02551). For analogue parameter writes, the same holds: an SLM pixel or heater with finite resolution silently drops updates below its step unless the master copy of the parameter is kept digitally.
A real case. In the Compute in Transit experiment (a 15.8M-parameter GPT on TinyStories, 8 simulated workers), emulated analogue optics computed DeMo's fixed DCT band while error feedback removed the exact band. All six runs of that first design diverged, to final losses of 11.153–558.834 against 3.124 for the band computed exactly, and ENOB 12 ended worse than ENOB 6. The cause was bias: quantisation and clipping errors that repeated every step and were never fed back. Feeding back the band actually sent fixed it. The details are on the next slide.
Read-out noise sets a floor on what the deployed engine can deliver (the dashed line on slide 12), but within a training loop it mostly washes out. Bias is what training on the wrong model cannot fix. Repeated passes cut random noise by √k (averaging) and do nothing for bias.
The Compute in Transit experiment (CiT 04) put an emulated analogue transform inside a real training loop. It trained a 15.8M-parameter GPT (15,833,472 parameters, 8 layers, width 384) on TinyStories with 8 simulated workers for 3,000 steps, using DeMo (glossary): each worker DCT-transforms its update in 256-point chunks and sends only a fixed frequency band, one coefficient in sixteen. Variant E computed that band with an emulated optical transform modelled on a photonic link module (Kundu et al., arXiv:2608.21536): each value quantised against a full scale into P binary bit planes, each plane through the transform with read-out noise at a given ENOB, the planes recombined. Variant C computed the same band exactly.
Error feedback is what makes a compressed update work: the delta Δt = mt + et is the momentum plus everything not yet sent, the band is removed, and the rest stays in the residual, decaying only by 0.999 per step. DeMo then applies the sign of the update. The optics' input range was set as full scale = κ·RMS(Δ) with κ = 4, measured over calibration steps 200–299 and then either held fixed or re-set every 100 steps from the delta's mean square. As first designed (full scale fixed, the exact band fed back), all six E runs diverged while C trained:
| E run | First design: final loss (clip rate) | Band sent fed back: final loss (clip rate) |
|---|---|---|
| ENOB 6, 8 planes | 11.153 (22.59%) | 3.423 (0.28%) |
| ENOB 8, 8 planes | 214.647 (45.80%) | 3.017 (0.29%) |
| ENOB 10, 8 planes | 42.183 (24.93%) | 2.878 (0.30%) |
| ENOB 12, 8 planes | 336.981 (57.50%) | 2.905 (0.30%) |
| ENOB 8, 4 planes | 344.388 (56.52%) | 3.137 (0.22%) |
| ENOB 8, 12 planes | 558.834 (54.57%) | 3.024 (0.30%) |
| C, exact band (3 seeds) | 3.124 ± 0.008 | |
Final validation loss at step 3,000, clip rate over the whole run; one seed per E run. Source: CiT experiment results.md §1 and §3, plotted on the results site; the repaired runs re-set the full scale every 100 steps.
The diagnosis was isolated on the CPU with TinyShakespeare runs (4 layers, width 128, 600 steps): the first design reached 8.809 against 2.464 for the exact band; re-setting the full scale with the exact band still fed back gave 3.404; removing the read-out noise entirely still left 2.967; feeding back the band sent gave 2.520–2.535. Quantisation and clipping errors that were never fed back were enough to break training. Source: CiT experiment optics_investigation.md §1–4.
This is the “biased, systematic” row of slide 09, measured: more ENOB did not help because the damage was a repeated error, not a random one. The cure was not better optics but closing the loop on what the optics actually did, the same lesson as hardware-in-the-loop training (slide 06). Precision then matters again, on the next slide.
Mixed-precision LLM training keeps activations and weights in BF16 or FP8 E4M3 and gradients in FP8 E5M2, which trades mantissa bits for range (Micikevicius et al., “FP8 formats for deep learning”, arXiv:2209.05433: E4M3 reaches 448, E5M2 57,344). An analogue engine in that loop need not be exact; its read-out should add no more error than the format's own rounding. The inference series' noise rule, relative error = crest × 2−ENOB × 2/√12, applied to a 4,096-sample long convolution and its two gradients:
| Signal | Crest factor | ENOB for BF16 | FP8 E4M3 | FP8 E5M2 |
|---|---|---|---|---|
| Forward y | 3.55 | 10.3 | 6.3 | 5.3 |
| Gradient dU (input) | 3.72 | 10.3 | 6.3 | 5.3 |
| Gradient dH (filter) | 4.92 | 10.7 | 6.7 | 5.8 |
Source: results.md, §5. The gradient signal carries rare spikes (1% of samples ×4, an illustrative model of gradient outliers), which raise the crest factor.
Measured in a training loop. The repaired runs of slide 10 (band sent fed back, full scale re-set every 100 steps) show what ENOB and how many bit planes an LLM training loop needed, at one size: a 15.8M-parameter GPT on TinyStories, 8 simulated workers, one seed per optics run.
| Run | Final loss | Steps to 3.158 |
|---|---|---|
| ENOB 6, 8 planes | 3.423 | never |
| ENOB 8, 8 planes | 3.017 | 1,800 |
| ENOB 10, 8 planes | 2.878 | 1,600 |
| ENOB 12, 8 planes | 2.905 | 1,800 |
| ENOB 8, 4 planes | 3.137 | 2,600 |
| ENOB 8, 12 planes | 3.024 | 2,000 |
| Perfect optics (control: no quantisation, no noise) | 3.141 | 2,800 |
| C, exact band (3 seeds) | 3.124 ± 0.008 | 2,667 |
Source: CiT experiment results.md §1 (final loss; steps to 3.158, the 2-D fixed band's final loss, evaluated every 200 steps) and §3; CiT simulation results.md §7c. On the results site: ENOB and bit planes, repaired.
From ENOB 8 up, the repaired optics band ends below the exact band it emulates: 2.878 at ENOB 10 against 3.124 ± 0.008. A hypothesis from the experiment's notes, untested: quantisation and clipping are non-linear, so the values sent carry a little of the large out-of-band residual, and feeding them back lets some out-of-band information reach the model. Against it: every optics run is a single seed, and on TinyShakespeare the repaired optics ended slightly above the exact band (2.520–2.535 against 2.464) (results site). Either way it does not rescue the fixed band: the best optics run, 2.878, is still 1.09 above top-k's 1.785.
The “hardware” is a 1-D causal convolution with a hidden fixed distortion: a short kernel (equivalently a per-frequency gain and phase error) and a tanh saturation, read out with zero-mean noise at the chosen ENOB. The task: learn a 16-tap target filter by gradient descent. Three trainers see the same inputs:
Illustrative toy, computed live in your browser with a seeded PRNG (results are reproducible). Defaults: distortion 0.5, ENOB 6, learning rate 0.1, 200 steps, seed 7. Setup: 64-sample sequences of Gaussian input, 8 per step, plain SGD from a zero filter; ADC full scale ±4× the target's RMS; the in-situ gradient read-out is auto-ranged to its own peak. Metrics are the hardware loss (normalised MSE) averaged over the last 10 steps, on 4 fixed evaluation sequences with the noise-free hardware; the dashed line is the read-out noise alone, which deployment adds on top.
| Method | Who computes the gradient | Needs a model? | Hardware passes per step | Scaling evidence |
|---|---|---|---|---|
| Digital backprop (reference) | The chain rule, on stored activations | The code is the model | None; backward costs about 2× forward (the 6N rule) | Every LLM |
| In-silico (D²NN) | A computer, on a model of the optics | Yes, and it must be accurate | None while training; the hardware only runs the result | Fabricated diffractive networks on small vision tasks (Lin 2018) |
| In-situ backprop (adjoint) | The hardware, by interference | No; needs reciprocity, adjoint injection, monitors at every parameter | Forward, adjoint and interference passes per sample | Small MZI meshes (Pai 2023) |
| Physics-aware | A digital twin, on the hardware's real outputs | Yes, differentiable but approximate | One forward pass per sample | Three different physical systems on small tasks (Wright 2022) |
| DFA, optical projection | Fixed random projection of the output error | No: the projection must be fixed, not known | One projection per sample, plus the forward pass | Photonic co-processor (Launay 2020); lags backprop on harder tasks |
| Forward-only | The hardware, from forward passes (local objectives, or a reciprocal mirror of the backward pass) | No | Two or more forward passes per sample | Optical, acoustic, microwave demonstrations (Momeni 2023; Xue 2024) |
Two axes separate them: how much of the gradient comes from the real device (in-silico none, physics-aware the error only, in-situ and forward-only all of it), and whether the learning rule is still backpropagation (DFA and local learning change it). Only methods that keep backpropagation and take the gradient from the device remove both the reality gap and the algorithm risk; they are also the ones that ask the most of the hardware.
The engine of deck 01 computes fixed transforms (FFTs and Fourier-domain products) on values it is handed. The LLM's weights, master copies and optimizer stay digital; a weight spectrum is written to the mask once per step and read back by nothing. So the training methods in this deck are not a way to train the LLM. They matter for the engine's own parameters: the transfer function of its lenses, mask pixels and detectors.
If the engine computes the forward and backward transforms of an LLM layer and the digital side computes the chain rule as if they were exact, the LLM itself is being trained physics-aware style: real (distorted) outputs, idealised gradients. The experiment shows that converges while the per-frequency phase error stays well inside 90°, and that the residual error is set by the bias, not the noise. That gives a calibration target for an engine in a training loop. This mapping is the author's own analysis, attributed to no company.