Fourier Optics for Inference 02 — Transforms in Inference Workloads

00

Topics We'll Cover

Where a Transform Could Hide in an LLM
A Standard Decoder Has No FFT
FNet: Fourier Token Mixing
Long Convolutions: S4, H3 and Hyena
Causality at Prefill and at Decode
The Amdahl Analysis: Prefill FLOP Shares
Interactive: Amdahl Explorer
Structured Weights: Circulant and Monarch
Transform-Domain Compression of the KV Cache
Precision and Energy per Token
Holding the Weights: Fourier-Plane Mask Capacity
What Prefill Hands to Decode
Why Disaggregation Fits
What Does Not Map
What to Take Away

Concepts used here, and where they are explained. Each links to a glossary entry: this series glossary, or the glossaries of LLM Inference Simulators, FHE Accelerator Simulators and the Simulation Engineering Toolkit for concepts those series already explain.

01

Where a Transform Could Hide in an LLM

Deck 01 showed what a Fourier-optical engine does well: long transforms and convolutions, with the weights held on a mask, at a cost per converted value. The question for inference is whether the work contains such transforms, and how much of the work they are. Three places to look, and two things that are photonics but not Fourier optics:

Where	What the transform does	Slides
Token mixing	FNet's Fourier mixing; long convolutions (S4's convolution mode, H3, Hyena) computed by FFT	03–07
Structured weights	Circulant, block-circulant or Monarch matrices, whose products are transforms	08
Transform-domain compression	DCT/Fourier coefficients of the KV cache or hidden states along the sequence	09
Optical MACs (not Fourier optics)	Matrix multiplies done optically: the simulator's hypothetical optical part	02
Photonic interconnect (not Fourier optics)	Moving the KV cache between pools over optical links	13

Every number from here on comes from analysis/flop_share.py, which counts FLOPs for a model of Llama-3-8B's shape (Llama 3 report) in five variants and writes results.md.

02

A Standard Decoder Has No FFT

A Llama-style decoder is matrix multiplies (projections, the SwiGLU MLP, the LM head) plus attention (QK^T and AV), softmax, norms and activations. None of it is a Fourier transform. The analysis reproduces the companion simulator's FLOP count exactly (Disaggregated_Inference_Sim's cost model, checked to the FLOP), and splits it by class:

Llama-3-8B prefill	GFLOP	Dense matmul	Attention
512 tokens	7,753.6	99.1%	0.9%
2,048 tokens	31,839.1	96.5%	3.5%
32,768 tokens	773,308.9	63.6%	36.4%
131,072 tokens	6,470,935.2	30.4%	69.6%

Source: results.md, section “2. Prefill”. Lengths beyond the model's trained context are shape-only extrapolations.

An optical MAC would help this model

The simulator's HYPOTHETICAL_OPTICAL device is a “what if compute were nearly free” probe: it speeds up every dense FLOP, and, as its comment in hardware.py says, decode barely moves because decode is bandwidth-bound.

A transform engine would not

A Fourier transform engine accelerates only FFT and convolution work and leaves matmuls to a digital part. For this model its share is zero. The rest of the deck asks which other models give it something to do.

03

FNet: Fourier Token Mixing

FNet replaces each self-attention sublayer of a Transformer encoder with an unparameterised 2-D discrete Fourier transform over the sequence and hidden dimensions, keeping the real part. Lee-Thorp et al. report 92–97% of BERT's GLUE accuracy while training 80% faster on GPUs and 70% faster on TPUs at 512 tokens (arXiv:2105.03824).

Why it looks made for optics

A 2-D DFT over (sequence × hidden) is exactly what a lens computes.
No weights in the mixer: no mask to rewrite.

Why it does not help an LLM

It is not causal. Every output mixes every token, including later ones. In the analysis, perturbing the last token changes the first token's mixed output by 64.0 (a 256×64 block): a decoder cannot use it as it stands.
It is a tiny share of the work. In a model of Llama-3-8B's shape with FNet mixing, the transform is 0.09% of prefill FLOPs at 2,048 tokens: the MLP dominates.

The FFT is what makes FNet cheap. Making a cheap operation free buys little: the theme of this deck.

04

Long Convolutions: S4, H3 and Hyena

Causal sequence models that mix tokens with a long convolution (a filter as long as the sequence) can compute it by FFT in O(L log L):

Model	Mixer	Convolution or recurrence?
S4 (Gu, Goel, Ré, arXiv:2111.00396)	A structured, time-invariant state-space model	Both: a long convolution for training, a recurrence for generation (reported 60× faster generation)
H3 (Fu, Dao et al., arXiv:2212.14052)	Two SSMs with multiplicative gating	FFT convolution at training and prefill
Hyena (Poli et al., arXiv:2302.10866)	N+1 projections, a short depthwise conv, N implicit long convolutions with gating (order 2 for language)	FFT convolution, with input and filter zero-padded to 2L − 1 for causality (the paper, section 3)
Mamba (Gu, Dao, arXiv:2312.00752)	A selective SSM: parameters depend on the input	Recurrence only: selectivity “prevents the use of efficient convolutions” (the abstract)

On GPUs the FFT is the awkward part. FlashFFTConv finds that the FFT does not use matrix-multiply units well and incurs I/O between memory levels; recasting it as matrix multiplies speeds exact FFT convolutions up to 7.93× over PyTorch (arXiv:2311.05908).
Hyena's claim to fame is speed at length: the paper reports its operator twice as fast as optimised attention at 8K tokens and 100× faster at 64K.
Hybrids interleave a few attention layers with these mixers, e.g. Jamba with Mamba (arXiv:2403.19887) and the striped Hyena hybrids studied by Poli et al. (arXiv:2403.17844). The analysis includes a 1-in-4 attention hybrid (an illustrative ratio).

05

Causality at Prefill and at Decode

Prefill: one long FFT per channel

The whole prompt is known, so each channel's long convolution is one FFT, a multiply and an inverse FFT, zero-padded to 2L so it is linear and causal.
Checked in NumPy: padded FFT against direct, 5.6e-16 of the peak; perturbing the last input moves earlier outputs by 7.1e-15. Unpadded, the wrap leaks the future by 2.75.

Decode: one token at a time

Direct: cache past inputs, take a dot product with the filter: O(context) per token, no transform. It equals the prefill output (5.8e-16).
Distilled: fit each filter with a small SSM and run it as a recurrence, O(1) per token (Laughing Hyena, arXiv:2310.18780).
Relaxed tiling: convolve each completed block of 2^l inputs by FFT and add it to future outputs: exact, O(L log² L) in total (Flash Inference, arXiv:2410.12982). Checked: 4.5e-16.

Mixer	Context	GFLOP / token	Mixer FLOPs / token	Transform share
transformer	2,048	16.084	1,074.3 M	0.00%
transformer	32,768	32.190	17,180.4 M	0.00%
hyena_direct	2,048	17.694	1,074.3 M	0.00%
hyena_direct	32,768	33.800	17,180.4 M	0.00%
hyena_distilled	2,048	16.653	33.6 M	0.00%
hyena_distilled	32,768	16.653	33.6 M	0.00%
hyena_tiled	2,048	16.877	256.9 M	1.40%
hyena_tiled	32,768	17.046	425.7 M	2.34%
hyena_recompute	2,048	162.650	146,030.5 M	85.82%
hyena_recompute	32,768	3,040.278	3,023,658.5 M	96.06%

Source: results.md, section “4. Decode”

So the hypothesis needs care: decode can use FFTs, but only in small passes on the latency-critical path. Conversions per token per channel (DAC+ADC pairs): a prefill long convolution converts each input once and each output once, 1 pair per token. Relaxed tiling at decode converts, at each of its tile levels, a block of B inputs and B outputs every 2B steps, 0.5 pairs per token per level: 6 pairs at context 2,048 (12 levels); 8 pairs at context 32,768 (16 levels), in small passes on the latency-critical path.

06

The Amdahl Analysis: Prefill FLOP Shares

Optical share f = FFT FLOPs plus the pointwise multiplies in the Fourier domain (what a 4f mask does), over all prefill FLOPs. If the engine made that work free, prefill would speed up by at most 1/(1−f) (Amdahl's law).

Variant	512 tokens	2,048 tokens	8,192 tokens	32,768 tokens	131,072 tokens
Transformer (Llama-3-8B)	0.00%	0.00%	0.00%	0.00%	0.00%
FNet-shaped (non-causal)	0.08%	0.09%	0.10%	0.11%	0.12%
Hyena-2	0.17%	0.20%	0.23%	0.26%	0.29%
Hybrid 1:3 attention:Hyena	0.13%	0.15%	0.17%	0.18%	0.15%
Hyena-2 + block-circulant weights	13.63%	13.99%	14.35%	14.70%	15.05%

Source: results.md, section “2. Prefill”

Hyena-2 at 2,048 tokens: f = 0.20%, an Amdahl bound of 1.002x. The projections and the MLP, which stay digital, are over 99% of the work.
The share barely grows with length, because FFT work is O(L log L) per channel against O(L) for the matmuls: log L grows slowly.
The count is not the issue. The Hyena operator (one layer's mixer, without the MLP) counted here against the Hyena paper's own formula (appendix A.2, s = 2,048): this script 277.16 GFLOP, of which 0.76% is FFT convolution; the paper's formula 276.87 GFLOP, of which 0.67%. The difference is padding (2L here) and the FFT constant; the conclusion does not depend on it.
Time, not FLOPs, is what an engine removes. If GPUs ran FFTs at 1/16 of their matmul rate (an assumption), Hyena-2's share of prefill time at 2,048 tokens would be 3.08% and the bound 1.032x.

07

Interactive: Amdahl Explorer

The FLOP ledger of flop_share.py, ported to JavaScript and tested to give identical numbers (js/flop_model.js). Choose a model, a prompt length, where the LM head runs, and how fast a GPU runs FFTs relative to matmuls (r, an assumption).

Model (Llama-3-8B shape)

Prompt tokens2,048

LM head

GPU FFT rate, relative to matmul

Prefill GFLOP

—

Optical share (FLOPs)

—

Amdahl bound (FLOPs)

—

Share of time / bound at r

—

Things to try: switch the block-circulant model's LM head to the last token only, then set r = 1/16. That is the only corner of this space where a transform engine has most of the work.

08

Structured Weights: Circulant and Monarch

If token mixing is too small a share, the transforms would have to come from the weights: matrices whose product with a vector is a transform.

Circulant and block-circulant matrices: a circulant block is defined by one vector, and multiplying by it is an FFT, a pointwise product and an inverse FFT. Cheng et al. used circulant projections in fully connected layers (arXiv:1502.03436); CirCNN built block-circulant CNNs with FFT-based hardware (arXiv:1708.08917).
Butterfly and Monarch matrices: products of sparse or block-diagonal factors and permutations, a family that contains the FFT (Dao et al., arXiv:1903.05895; arXiv:2204.00595). Monarch Mixer uses them along both sequence and hidden dimensions and has a causal variant that matched GPT-style Transformers at 360M parameters in pretraining perplexity (arXiv:2310.12109).

The analysis makes every projection and MLP matrix block-circulant (block 256, illustrative) in the Hyena-2 model. The LM head then dominates, so where it runs decides the answer:

Variant	Prompt	GFLOP	Optical share	Amdahl bound
Hyena-2	2,048	31,959.9	0.21%	1.002x
Hyena-2	32,768	511,686.3	0.28%	1.003x
Hyena-2 + block-circulant weights	2,048	426.6	84.54%	6.469x
Hyena-2 + block-circulant weights	32,768	7,152.8	85.47%	6.883x

Source: results.md, section “2. Prefill”

Speculative

This is the only variant where a transform engine has most of the work, and it is the least established: phase A of this work found no published LLM of this size with block-circulant weights throughout. Quality at that scale is unknown, and the weights live on the Fourier-plane mask (slide 11).

09

Transform-Domain Compression of the KV Cache

A third place transforms appear: compressing what the model stores, along the sequence, by keeping a few frequency coefficients. This literature exists and is recent:

Work	What it transforms	Reported result
Fourier Transformer (He et al., arXiv:2305.15099)	Hidden states, with a DCT computed by FFT, progressively removing sequence redundancy	Inherits pretrained weights (BART); strong long-range benchmark results
FreqKV (Kai et al., arXiv:2505.00570)	The KV cache, iteratively compressed in the frequency domain, keeping low frequencies	Extends LLaMA-2-7B's context to 256K tokens with minimal training at 8K
FourierAttention (Liu et al., arXiv:2506.11886)	The long-context-insensitive head dimensions, projected onto fixed-length Fourier bases	Training-free; a fused Triton kernel

What it saves is memory and bandwidth, which is what decode is short of (LLM Inference Simulators 03, “KV-Cache Arithmetic”). The transforms themselves are small next to the model's matmuls.
A possible optical role (speculation): the prefill pool compresses the KV cache before it crosses the link, so the transform runs where optics is strong and shrinks the hand-off (slide 12). Whether these methods' accuracy survives analogue noise is untested.

10

Precision and Energy per Token

Precision: float workloads need the analogue error to match the number format's own rounding (deck 01, “Precision: ENOB, Noise and Crosstalk”): about 8.0 ENOB for INT8-like error, 6.3 for FP8 E4M3 and 10.3 for BF16. Below that, averaging costs 4× the passes per bit.

Energy: DAC+ADC pairs per prompt token, and their energy at each ENOB, against the digital energy of the transform work they replace, in µJ per prompt token (Walden FoMs 10/20 fJ and 1 pJ/FLOP, both illustrative):

Variant	Pairs / token	Digital uJ	ENOB 8 uJ	ENOB 12 uJ	Break-even ENOB
FNet-shaped (non-causal)	131,072	11.1	1.0	16.1	11.5
Hyena-2	262,144	33.0	2.0	32.2	12.0
Hybrid 1:3 attention:Hyena	196,608	24.8	1.5	24.2	12.0
Hyena-2 + block-circulant weights	2,162,688	176.1	16.6	265.8	11.4

Source: results.md, section “7. Conversion energy”

The break-even ENOB is about 12 for every variant, because each output sample of an FFT convolution replaces a similar number of FLOPs whatever the model. At ENOB 8, Hyena-2's conversions cost 2.0 µJ per token against 33.0 µJ for the same work done digitally.
But that work is small. For scale: the whole Hyena-2 prefill costs 16,656 uJ per prompt token of dynamic FLOP energy at 1 pJ/FLOP; the transform work is a small slice of it.

11

Holding the Weights: Fourier-Plane Mask Capacity

A 4f pass multiplies by one mask. Each Hyena channel's filter spectrum (or each circulant block) must be on the mask while its inputs pass, so the mask is the weight store, and rewriting it costs a frame of the spatial light modulator (deck 01, “Spatial Light Modulators”). Time to rewrite a 2-megapixel mask enough times for one forward pass, at three device rates: a micromirror device at 1-bit depth (20 kHz) and 8-bit depth (1.03 kHz), and a liquid-crystal SLM (30 Hz):

Variant	Prompt	Rewrites	DMD 20 kHz	DMD 1.03 kHz	LC 30 Hz
Hyena-2	2,048	269	13.4 ms	261 ms	9.0 s
Hyena-2	32,768	4,296	214.8 ms	4,167 ms	143.2 s
Hyena-2 + block-circulant weights	2,048	277	13.8 ms	269 ms	9.2 s
Hyena-2 + block-circulant weights	32,768	4,303	215.2 ms	4,174 ms	143.4 s

Source: results.md, section “8. Holding the filters”

Filter spectra are big: 2 × 4,096 channels × (L+1) frequency bins per layer, for 32 layers. A 2-megapixel mask holds a fraction of one forward pass's worth.
Rewrites are amortised over the batch. Every sequence in a prefill batch uses the same filters, so one pass over the mask serves all of them: large prefill batches help, and token-by-token decode cannot amortise at all.
For comparison, the simulator prefills a 2,048-token Llama-3-8B prompt on one H100 in 59.033 ms (its results.md, §1). At the 8-bit device rate the mask alone takes longer, unless batches are large or the mask is much bigger or faster. The device figures are illustrative, from one paper (Miscuglio et al., arXiv:2008.05853).

12

What Prefill Hands to Decode

Disaggregated serving moves state from the prefill pool to the decode pool once per request (LLM Inference Simulators 05, “What the KV Transfer Costs”). What that state is depends on the mixer:

Decode style	Bytes per prompt token	Bytes for a 2,048-token prompt	Grows with the prompt
Transformer KV cache (GQA)	131,072	268.4 MB	yes
Hyena direct: cached projection inputs	524,288	1,073.7 MB	yes
Hyena distilled recurrence state	n/a	16.8 MB	no

Source: results.md, section “9. What prefill hands to decode”

Direct long-convolution decode is heavier than attention. Hyena-2 caches 2 × d values per layer per token against GQA's 2 × 8 heads × 128: four times the bytes to store, to read every decode step, and to transfer.
Distillation changes the hand-off completely: a constant-size state, independent of prompt length. That lever is larger than any choice of link.
The phase-B simulator will measure both levers: the link (electrical or photonic) and what crosses it.

13

Why Disaggregation Fits

The phases want different hardware. Prefill is compute-bound, decode memory-bound (LLM Inference Simulators 03, “Two Phases: Prefill and Decode”); Splitwise proposes different machines for each (arXiv:2311.18677), and DistServe provisions them separately for goodput (arXiv:2401.09670).
Prefill is where a transform engine could work: long transforms, and filters held on the mask across a whole batch. Decode keeps its token-by-token recurrence on digital parts. If the transforms were a large share, disaggregation would let the optical engine join only the pool that can use it.
Photonic interconnect is a separate topic: optical I/O chiplets (Wade et al., TeraPHY, IEEE Micro 2020) and data-centre optics (Cheng et al., Optica 2018) move the KV cache; they compute nothing.

Public material, and this deck's speculation

Optical-computing companies publish little about inference mapping. Several publicly describe photonic computing systems for AI and for FHE, and published optical computing for FHE centres on Fourier transforms (FHESim 04). The mapping in this deck is the author's own analysis and speculation, attributed to no company.

14

What Does Not Map

Work	Why a Fourier transform engine does not help
Attention (QK^T, softmax, AV)	A data-dependent, causal, softmax-normalised product: not a convolution. At long context it is the biggest term (69.6% of Llama-3-8B prefill at 131,072 tokens), and none of it is a transform
Dense projections, MLP, LM head	General matmuls: work for an optical MAC, not a Fourier engine, unless the weights are structured (slide 08)
Mamba's selective scan	Input-dependent parameters rule out the convolution form; it is a recurrence
Decode, generally	One token per step: direct dot products or recurrences, memory-bound; relaxed tiling uses FFTs only in small latency-critical passes (slide 05)
Norms, activations, softmax, sampling, MoE routing	Non-linear or data-dependent; stay digital
FNet mixing in a decoder	Not causal (slide 03)
KV-cache reads	A bandwidth problem; optics could shrink it only through compression (slide 09) or move it over photonic links

And the hidden costs that apply even where it maps: conversions per value, precision passes, mask rewrites, sign recovery at the detector, and static power whether or not work arrives.

15

What to Take Away

Standard decoders have no Fourier transforms. Optical MACs and Fourier engines are different bets; this series is about the second.
Where FFTs do appear, they are what makes a model cheap, so they are a small share. A Llama-3-8B-shaped Hyena-2 model spends 0.20% of its prefill FLOPs on transform work at 2,048 tokens: if the optics made it free, prefill would be at most 1.002x faster (Amdahl). The FFT is what makes Hyena cheap; the MLP and projections stay digital and dominate.
Only when the weights themselves are structured (block-circulant, speculative at LLM scale) does transform work matter: 13.99% of a far smaller total (Amdahl bound 1.16x) with the LM head charged for every prompt token, because the dense LM head is then the bulk of the work; with the LM head on the last token only, 84.54% and 6.5x.
Causality splits the phases: prefill has long, padded FFTs with weights held across the batch; decode is token by token, and its FFT option (relaxed tiling) uses small passes on the critical path. That is why, if anything, a transform engine belongs in a prefill pool.
The bill: about 8 ENOB for INT8-like error and 10 for BF16-like, break-even with digital FFT energy near 12 ENOB, and mask rewrites that can exceed a GPU's whole prefill time.
Next (planned): deck 03 puts an optical transform device into the prefill pool of Disaggregated_Inference_Sim and measures TTFT, TPOT, goodput, joules per token and PPA against all-GPU pools, with the break-even point.