Optical Prefill Pools: Simulated Results

Topics We'll Cover

• The Question, and the Short Answer
• What Was Added to the Simulator
• The Transform Engine Model
• Heterogeneous Pools: H100 Prefill, A100 Decode
• Optical Prefill Against All-GPU Baselines
• Interactive: The Simulator, Live
• Where the Time Goes: Mask Rewrites and Passes
• Static Power and Energy per Token
• The Break-Even Point
• Area, Cost and Performance per Dollar
• The KV Hand-off: Link and Payload
• Compute in the Transport, Not the Model
• Compressing the Hand-off: In Transit or at the GPU
• Beyond Compression: Resharding, Confidential KV, FEC and Collectives
• What This Does Not Show
• What to Take Away

The Question, and the Short Answer

Deck 02 found by counting FLOPs that a Fourier-optical engine has little to do in LLM inference: 0.20% of a Hyena-2 model's prefill FLOPs at 2,048 tokens, and a large share only when the weights themselves are block-circulant, which is speculative at this scale. This deck runs the full serving system: a disaggregated cluster in Disaggregated_Inference_Sim, with an optical transform device in the prefill pool, against all-GPU baselines on the same request stream.

What Was Added to the Simulator

Four extensions, each with every existing number unchanged (the 36 earlier tests pass, and sections 1–9 of results.md regenerate identically). The Python package and its JavaScript port (web/sim_engine.js, which this deck runs) agree bit for bit on every new configuration; the Rust port carries heterogeneous pools only.

The Transform Engine Model

Per prefill step (a batch of prompts), following the phase-A design. All integer counts, so Python, JavaScript and any later port agree exactly:

Source: hardware.py (TransformEngine, CostModel.optical_terms). Every coefficient on this slide is illustrative.

Heterogeneous Pools: H100 Prefill, A100 Decode

The generally useful part of this work, and the part ported to Rust. Prefill is compute-bound and decode memory-bound (InfSim 03), so each pool can use the part that suits it:

Source: results.md, section “10. Heterogeneous”

• Setting --prefill-device h100 --decode-device h100 explicitly reproduces the --device h100 run bit-identically (every request's timestamps and every summary number): True.
• An A100 decode pool costs TPOT but still meets the 25 ms SLO here, at lower J/token. An A100 prefill pool alone cannot keep up with 8 req/s (SLO 0.0%); two of them nearly can (96.9%).
• The Rust port reproduces each of these pool mixes with identical timestamps and summaries (Rust_DES_Kernel results, section 9).

Optical Prefill Against All-GPU Baselines

Two models from results.md section 11 (the others are there too): Hyena-2, where transforms are 0.2% of prefill, and the most transform-heavy variant, block-circulant weights with the LM head on the last token only (84.5%).

Hyena-2

Source: results.md, section “11. All-GPU”

Hyena-2 + circulant 256, last-token head

Source: results.md, section “11. All-GPU”

• Defaults collapse. 64 averaging passes (BF16 at ENOB 8) and hundreds of mask rewrites at 1,031 Hz per step (slide 07) make every prefill step hundreds of milliseconds: the queue never drains.
• Hyena's TPOT on GPUs is the real problem, and optics does not touch it: its direct-decode cache is 4× the GQA KV cache, so decode is slower than attention's. A distilled recurrence fixes it (section 11, “distilled decode”).
• With GPU FFTs at their matmul rate (an optimistic assumption for GPUs), even the optimistic engine loses TTFT on the circulant variant.

Interactive: The Simulator, Live

The simulator itself, inlined from web/sim_engine.js (bit-exact with the Python package, tested), running the same 800 requests results.md used. The defaults reproduce section 11's circulant last-token row with the optimistic engine. Every control is a real simulator input; the coefficients are illustrative.

Where the Time Goes: Mask Rewrites and Passes

One 2,048-token prefill step, closed form. The transform share sets what the engine could take; the mask and the passes set what it costs:

Source: results.md, section “12. Sweeps”

Mask rate (circulant 256, last-token head, ENOB 11, overlapped):

Source: results.md, section “12. Sweeps”

• The circulant variant's forward pass needs 552,456,192 filter-spectrum values on a 2-million-value mask: 277 rewrites per step, whatever the batch. At today's 8-bit DMD rate that alone is longer than the whole GPU prefill.
• The GPU's own prefill of the circulant variant is tiny (a few milliseconds) if its FFTs run at the matmul rate. That assumption, not the optics, decides most of the comparison.

Static Power and Energy per Token

ENOB (circulant 256, last-token head, 20 kHz mask, overlapped). Below 11 bits the averaging passes multiply the conversions; above it each conversion costs twice as much per extra bit:

Source: results.md, section “12. Sweeps”

Lasers and thermal tuning, charged for the whole run (optimistic engine, full runs):

Source: results.md, section “12. Sweeps”

For comparison, all-GPU 1P1D: 0.185 J/token with GPU FFTs at the matmul rate, 0.210 at 1/16 (section 13).

The Break-Even Point

Where does an optical prefill pool stop paying? Bisection on full simulator runs (as the sister series' search.py finds the maximum sustainable load), for the most transform-heavy variant and three assumptions about how fast GPUs run FFTs:

Source: results.md, section “13. Break-even”

Source: results.md, section “13. Break-even”

• GPU FFT efficiency below which the optimistic optical pool wins TTFT p99: 0.032 (1/30.9).
• Energy: the optical pool's J/token matches the GPU's only if the lasers and tuning stay below the static power in the table: a few watts against GPUs that run FFTs well, tens of watts against GPUs that do not.
• Latency: precision cannot rescue it (lowest ENOB that matches the GPU's TTFT p99: never, because one pass at ENOB 11 is already the floor); the mask rewrite rate is what has to improve.

Area, Cost and Performance per Dollar

The PPA view reuses brief 03's method rather than inventing new coefficients: published GPU die areas, Murphy yield at D₀ = 0.1/cm², $10,000 per 300 mm wafer, and the speculative optical-engine areas of FHE_Accelerator_Sim's area model (SimEng 13). Silicon only: no HBM, packaging or test.

Source: results.md, section “10. Heterogeneous”

Throughput per dollar of silicon, from section 11 (circulant 256, last-token head):

Source: results.md, section “11. All-GPU”

• The engine adds about 100 mm² and about 5% of silicon cost per prefill device (a separate, small photonic die yields well); that is small next to the latency and static-power verdicts.
• perf/W here is tokens per joule of the whole cluster, static power included, so it ranks configurations by energy alone.

The KV Hand-off: Link and Payload

Photonic interconnect is not Fourier optics, but it moves the hand-off between pools. Two levers: the link, and what is handed off (deck 02: Hyena-2's direct-decode cache is 4× the GQA KV cache; a distilled state is a constant).

Source: results.md, section “14. The KV hand-off”

Hyena direct cache (the biggest hand-off):

Source: results.md, section “14. The KV hand-off”

• On 25 GbE the 1.07 GB hand-off saturates the link; on every faster link the hand-off is a few percent of end-to-end time or less.
• The choice of what to hand off moves more than the choice of link: a distilled state is under 1% of end-to-end time even on 25 GbE (section 14).
• cpo-optical is an illustrative co-packaged-optics link (round numbers; optical I/O chiplets reach multi-Tb/s, Wade et al., IEEE Micro 2020).

Compute in the Transport, Not the Model

A different place for optics: computing in the link, on data that must move anyway. Published 2026 compute-in-transit prototypes put operations in an optical transceiver's data path. The number that decides what such a stage can do is its compute budget per byte at line rate: about 1–2 operations per byte. This simulator uses 1.6 (illustrative).

Compressing the Hand-off: In Transit or at the GPU

The same compression done in two places: in the link (no GPU time, +1 pJ/bit, +1 µs, limited by the budget) or on the prefill GPU (an elementwise pass: read the KV, write the compressed copy). TTFT does not include the hand-off (the first token leaves the prefill pool), so the effect shows in the hand-off latency (first token to KV on the decode pool), in TPOT and in the SLO. The transport-bound cases:

Source: results.md, section “15. Compute in the transport”

Source: results.md, section “15. Compute in the transport”

• Where the link is the bottleneck, compression helps a lot, wherever it is done. GQA KV over 25 GbE at 14 req/s: hand-off p99 9,657.8 ms uncompressed, 166.6 ms with fp8 in transit, 166.6 ms at the GPU; SLO met 41.0% against 100.0%.
• The GPU does it nearly free. An elementwise pass over the KV (read it, write half of it) is short next to a prefill step, so in-transit and at-GPU hand-off latencies match for fp8, and J/token is the same to three decimals. The in-transit stage saves GPU time, not latency.
• The budget bites for fp4. At 3.8 operations per line byte the stage is transit-bound: on 25 GbE at 8 req/s its hand-off p99 is 196.6 ms in transit against 81.0 ms at the GPU, worse than fp8 in transit.
• Where the link is not the bottleneck (every faster link in section 15), neither placement changes the SLO. The full grid (5 links × 3 hand-offs × 4 presets × 2 placements, and the load sweep) is in results.md section 15.

Beyond Compression: Resharding, Confidential KV, FEC and Collectives

Discussion only: not simulated, and all speculative. Other functions a compute-in-transit stage on the hand-off path could take, roughly from most to least promising:

What This Does Not Show

• Illustrative coefficients throughout. Converter figures of merit, 20 W of lasers and tuning, 10¹² samples/s, the 2-million-value mask and its rewrite rates, the in-transit budget, energy and latency, the co-packaged-optics link, the GPU power coefficients, and the PPA's wafer price and defect density.
• No optical hardware data. Nothing here was measured on an optical engine; the device model is first-order (noise as ENOB, ideal gain control, no crosstalk, laser noise or calibration drift; intensity detection as a doubling of passes).
• A speculative model variant. No published LLM of this size uses block-circulant weights throughout, as far as phase A found; Hyena-2 and the hybrid are shape-matched to Llama-3-8B, not trained models.
• GPU FFT efficiency is an assumption (1, 1/4 or 1/16 of the matmul rate), and it moves the circulant verdict more than any optical parameter.
• Not simulated: relaxed-tiling decode (FFTs at decode), accuracy of compressed KV, decode-side handling of compressed KV, tensor-parallel all-reduce, chunked prefill.
• Speculative mapping. How any real photonic system would serve LLM inference is this series' speculation, attributed to no company.

What to Take Away

• Heterogeneous pools are worth having, optics or not: an A100 decode pool behind an H100 prefill pool keeps every SLO here at lower J/token, and the Rust port carries the feature bit-exactly.
• An optical transform engine does not pay for attention or Hyena models. The transforms are too small a share (Amdahl), and the engine's passes, mask rewrites and static power cost more than the FLOPs it saves.
• Only structured weights give it a window, and only if GPUs run FFTs poorly and the mask rewrites far faster than today's 8-bit devices; static power must stay low for the energy to break even.
• The hand-off is a better home for optics than the model: at 1–2 operations per byte, an in-transit stage suits requantising or frequency-compressing data that must move anyway. It matters only where the link is the bottleneck, and there a GPU can do the same compression nearly free: the stage's gain is GPU time and energy, not latency.
• Every number came from a recorded run; the simulator on slide 06 is the same code.

Back to the series hub, or the LLM Inference Simulators, where the simulator is built.