Neural Sound Synthesis · Part 12 — Real-Time, Control & Frontiers

1Real-time & streaming — the RAVE turn

Almost every model in this series generates offline: a WaveNet grinds out 16,000 samples one at a time (Part 2), a diffusion model runs dozens of denoising passes over a whole clip (Part 8, Part 11). You wait, then you listen. A musician on a stage cannot wait. The frontier question is whether a neural synthesiser can behave like an instrument — responding to a control gesture within a few milliseconds, continuously, on ordinary hardware.

RAVE (Caillon & Esling, 2021) is the system that made this practical. It is a variational autoencoder (Part 1's fifth family) whose encoder maps audio into a very low-dimensional continuous latent, and whose decoder turns a latent trajectory back into a 44.1 kHz waveform — fast enough to run faster than real time on a laptop CPU. Because the latent is only a handful of dimensions per frame, you can expose those dimensions as knobs and play the timbre.

Try it — a latent you can hear move

The demo below is a live, continuously-running synthesis graph, in the RAVE spirit: a small latent vector $\mathbf{z}=(z_1,z_2,z_3)$ drives the harmonic spectrum of the sound in real time. Press Start, then sweep the sliders — brightness, harmonicity and register change the timbre the instant you move them, with no re-render and no offline pass. That immediacy is the whole point: contrast it with the "generate, wait, listen" loop of every earlier part.

z₁ brightness 0.60 z₂ harmonicity 0.50 z₃ register 0.40

Press Start, then move the three latent knobs. The bars are the live harmonic spectrum; the sound updates continuously.

What you are hearing. This is a hand-built additive stand-in, not a trained decoder — but the interaction model is exactly RAVE's: a tiny latent, mapped continuously to timbre, driving an always-on audio graph. A real RAVE decoder replaces the harmonic bank with a learned convolutional network, so the same three knobs would glide between the timbres of whatever corpus it was trained on (voices, strings, ambience) rather than a fixed harmonic series.

How RAVE earns its speed

Two ideas do the work. First, a two-stage training recipe. Stage one trains the VAE with a plain reconstruction + KL objective to learn a compact, well-behaved latent — stable but slightly dull. Stage two freezes the encoder and fine-tunes only the decoder adversarially, with a discriminator (Part 3, Part 7) pushing the output toward realism. You get the VAE's smooth, controllable latent and the GAN's crisp high frequencies, without the training instability of doing both at once.

Second, causal, cached convolutions. A streaming model may only look backwards: the output at frame $$t$$ can depend on inputs up to $$t$$ but never after. That is the same causal constraint WaveNet obeyed (Part 1, Part 2), but here it is exploited for latency: each layer caches the tail of its previous input so a new block of samples costs only the incremental convolution, not a recomputation of the whole receptive field.

$$\hat{y}_t = f\big(x_{t-r+1},\,\dots,\,x_t\big),\qquad \text{latency} \;\approx\; \underbrace{\tfrac{L_{\text{block}}}{f_s}}_{\text{block}} \;+\; \underbrace{\tfrac{r-1}{f_s}}_{\text{receptive tail}}$$

With a causal receptive field $$r$$ and block length $L_{\text{block}}$ , the algorithmic latency is just the block plus the causal tail — a few milliseconds, not the length of the clip. This is what lets RAVE ship as nn~, an external for Max/MSP and Pure Data: a trained model drops into a patch as a single object, and a performer bends its latent with a MIDI controller live.

the constraint in one line · why non-causal padding leaks the future

A symmetric ("same") convolution centred on sample $$t$$ reads $x_{t+1},\dots,x_{t+(k-1)/2}$ — samples that, in a live stream, have not happened yet. Training that way and then deploying streaming produces a model that hallucinates or clicks at every block boundary. Causal convolution shifts the kernel so it reads only $x_{t-(k-1)},\dots,x_t$ ; the cost is that the receptive field extends only backwards, so the same reach $r=1+(k-1)\sum_\ell d_\ell$ now costs pure latency you must budget for. Every real-time neural synthesiser lives inside this trade between reach and delay.

2Controllability — from black box to instrument

A model that sounds perfect but only ever produces one thing is a recording, not a synthesiser. The value of these systems is in the handles they give you — and the series has quietly been a tour of ever-better handles. This section names the families of control and where each one lives.

Four ways to steer a generator

Control surface	What you turn	Where in the series
Interpretable / physical	Parameters that mean something: pitch, loudness, harmonic distribution, filter cutoff. DDSP exposes an oscillator's own controls, so a gradient can tune a synth you already understand.	DDSP Part 6
Latent / attribute	Continuous learned dimensions (RAVE's $\mathbf{z}$ ), or explicit conditioning tags — speaker, instrument, style — concatenated to the model's input.	RAVE, GANSynth Part 7 · 12
Symbolic / score	Notes, phonemes, F0 contours: the generator renders a structured plan you wrote. TTS and singing models are conditioned this way.	TTS, singing Part 4 · 5
Text / prompt	Natural-language description, embedded by a joint audio–text model and used as a diffusion or LM condition. Flexible, but coarse and hard to make repeatable.	Text-to-audio Part 10 · 11

Read down that list and you are reading a tension. The top is precise and real-time but narrow; the bottom is expressive and general but sloppy and slow. DDSP's cutoff knob does exactly one thing every time; a text prompt does roughly the right thing, differently on each seed. The open problem of controllability is to get prompt-level generality with knob-level precision and repeatability — ideally at the real-time latency of Section 1.

The best performance interfaces disentangle. If moving one control changes one perceptual thing — brightness without pitch, effort without vowel — the model is playable. If every knob smears across everything, it is a slot machine with faders.

Disentanglement is not free. RAVE's latent dimensions are ordered by explained variance, so the first few are the most audibly meaningful — but they are not guaranteed to be perceptually independent. Achieving clean, labelled, orthogonal controls (this axis is brightness, that one is attack) remains an active research target rather than a solved feature.

3Evaluation, revisited — measuring a distribution

Part 1 introduced the metrics ladder and promised this part would dissect it. Here is the debt paid. The central idea is one Part 1 stated and we now make concrete: for generative audio you cannot score sample-against-target, because there is no single correct output. You must compare the distribution of what the model makes against the distribution of real audio.

Fréchet Audio Distance, in full

FAD (Kilgour et al., 2019) embeds a set of real and a set of generated clips with a pretrained audio network, fits a Gaussian to each cloud of embeddings, and reports the 2-Wasserstein (Fréchet) distance between those two Gaussians:

$$\mathrm{FAD} = \lVert \mu_r - \mu_g \rVert_2^{2} + \operatorname{tr}\!\Big(\Sigma_r + \Sigma_g - 2\big(\Sigma_r \Sigma_g\big)^{1/2}\Big)$$

The first term punishes a mismatch in the average embedding; the second punishes a mismatch in the spread and correlation. Crucially it is zero only when the two Gaussians coincide — so a model is rewarded for covering the real corpus's whole range, not for nailing any one target. That is exactly why it succeeds where a per-sample loss (which would demand the model reproduce a specific waveform) is meaningless for generation.

Feel the two terms move

Below are two clouds of embeddings in a toy 2-D space: real (fixed) and generated (yours). Drag the generated cloud's mean and reshape its covariance, and watch FAD — and its mean-term / covariance-term split — update live. Press Match real to slide the generated distribution onto the real one and see the distance collapse toward zero.

μ_g x 1.70 μ_g y -1.30 spread 2.40 shape° 115

FAD = ‖μ_r−μ_g‖² + tr(Σ_r+Σ_g−2(Σ_rΣ_g)^½). Move the generated cloud onto the real one to drive it toward 0.

the matrix square-root, closed-form for 2×2

The intimidating term is $\operatorname{tr}\big((\Sigma_r\Sigma_g)^{1/2}\big)$ . For symmetric positive-definite $\Sigma_r,\Sigma_g$ the product $\Sigma_r\Sigma_g$ has real non-negative eigenvalues $\lambda_1,\lambda_2$ , and $\operatorname{tr}\big((\Sigma_r\Sigma_g)^{1/2}\big)=\sqrt{\lambda_1}+\sqrt{\lambda_2}$ . Using $(\sqrt{\lambda_1}+\sqrt{\lambda_2})^2=\lambda_1+\lambda_2+2\sqrt{\lambda_1\lambda_2}$ with $\lambda_1+\lambda_2=\operatorname{tr}(\Sigma_r\Sigma_g)$ and $\lambda_1\lambda_2=\det\Sigma_r\det\Sigma_g$ ,

$$\operatorname{tr}\big((\Sigma_r\Sigma_g)^{1/2}\big)=\sqrt{\operatorname{tr}(\Sigma_r\Sigma_g)+2\sqrt{\det\Sigma_r\,\det\Sigma_g}}\,.$$

That is precisely the formula the demo evaluates each frame from the empirical covariances of the two clouds — no eigensolver required.

The rest of the ladder — and why it wobbles

CLAP score

For text-conditioned audio, embed the output and the prompt with a joint CLAP model and take their cosine similarity — a proxy for "does this sound match those words?".

$$\text{CLAP} = \frac{\mathbf{e}_{\text{audio}}\cdot \mathbf{e}_{\text{text}}}{\lVert\mathbf{e}_{\text{audio}}\rVert\,\lVert\mathbf{e}_{\text{text}}\rVert}$$

Human tests — still the ceiling

MOS (mean opinion score, 1–5) and MUSHRA (hidden reference + anchor, 0–100) remain the only measures that directly ask "does this sound real / good?". Expensive, slow, and irreplaceable.

Every objective metric is gameable — and FAD especially so. Its score depends heavily on the embedding network and the reference set; a model can lower FAD by matching corpus statistics while still producing artefacts, and small evaluation sets give unstable, biased estimates. Newer kernel-based distances — KAD, built on the Maximum Mean Discrepancy — drop the Gaussian assumption and are unbiased for small samples, but they too are proxies. The honest paper still runs a listening test.

4Safety, provenance & ethics — who made this sound?

A model that can clone a voice from seconds of audio, or generate a song in a named artist's style, is a capability and a hazard in the same weights. This is not an appendix to the engineering — it increasingly is the deployment question.

The hazards, plainly

Voice cloning & consent. Zero-shot speaker cloning enables accessibility and localisation — and impersonation, fraud, and non-consensual synthetic speech. Consent of the voice owner is the ethical fault line.
Deepfakes. Synthetic audio of real people saying things they never said, at a quality that defeats casual listeners.
Copyright & datasets. Training corpora scraped from copyrighted recordings raise unresolved questions of consent, licensing, and style-appropriation that the law is still catching up to.

Two technical responses: watermarking and provenance

Audio watermarking embeds an imperceptible, robust signal into generated audio that a detector can later read — ideally surviving compression, resampling, and re-recording. AudioSeal (San Roman et al., 2024) is the state of the art: it trains a generator/detector pair jointly and, unusually, detects the mark sample-by-sample, so it can flag which segments of a longer clip are synthetic — proactive detection built into the model, not bolted on after.

Provenance attacks the problem from the metadata side. C2PA (the Coalition for Content Provenance and Authenticity) defines a cryptographically-signed manifest — "content credentials" — that travels with a file and records how it was made and edited. Watermarking survives the metadata being stripped; C2PA carries rich, verifiable history while it is intact. They are complementary, not rivals.

Watermark. In the signal itself; robust to re-encoding; low bandwidth (a flag, maybe an ID). Answers "is this synthetic?" even after the file has been mangled.

Provenance (C2PA). In signed metadata; rich and human-readable; brittle — a screenshot or a re-upload can strip it. Answers "where did this come from?" while the manifest survives.

Neither is a solution alone, and both are subject to an adversary. But the direction is clear: attribution is becoming a first-class design requirement for generative audio, alongside quality and speed.

5On-device & efficiency — shrinking the giant

The models that top the FAD leaderboards are large and slow. Getting them onto a phone, a pedal, or a browser tab — running with low latency and no cloud round-trip — is its own research frontier, and it leans on three levers.

Fewer steps — consistency models

Diffusion's cost is its dozens of denoising steps (Part 8, Part 11). Consistency models (Song et al., 2023) train a network so that any point on a noising trajectory maps directly to the clean sample — enabling one- or few-step generation. Applied to audio, they collapse a 50-step sampler toward a single pass.

Smaller weights — distillation & quantization

Distillation trains a compact student to mimic a large teacher. Quantization stores and computes in 8-bit (or lower) instead of 32-bit floats, shrinking memory and speeding inference — the same μ-law intuition from Part 1, now applied to the weights rather than the signal.

RAVE's whole design (Section 1) is an existence proof that this can be done: a compact VAE, a causal decoder, and enough engineering to beat real time on a CPU. The frontier is to bring the quality of the big text-to-audio systems into that same budget.

The efficiency triangle mirrors the family trilemma. Part 1 framed quality vs. speed vs. coverage; on-device work is the same trade under a hard compute ceiling. A consistency-distilled, quantized model buys speed and size — and the open question is always how much quality and diversity it quietly spends to get there.

6The whole series, on one map

Part 1 drew a map: four domains a model can work in, five families that place a distribution over them. Every part since was a point on that map. Here is the map again, filled in — and the honest reading is that modern systems no longer sit on a single point. They compose families across domains.

Where each part sat

Part	Domain	Dominant family	Modern combination it points to
2 AR waveform	Raw waveform	Autoregressive	The fidelity ceiling; now a teacher for faster students.
3 Neural vocoders	Spectrogram → wave	GAN	The real-time inverse of the STFT; the last mile of most pipelines.
4 End-to-end TTS	Spectrogram	AR / flow / VAE	Text → mel → wave, then fused (VITS = flow + VAE + GAN).
5 Singing voice	Spectrogram	Diffusion	Score + F0 control meets diffusion refinement.
6 Differentiable DSP	Interpretable params	Autoencoder	Neural control of a classical synth — the cleanest handles in the series.
7 GAN instruments	Spectrogram / latent	GAN	Adversarial timbre; attribute-conditioned generation.
8 Diffusion & flow	Waveform / spectrogram	Diffusion / flow	The quality engine, now being distilled toward few-step.
9 Neural codecs	Discrete tokens	VAE + VQ + GAN	The tokeniser that made audio language models possible.
10 LM music generation	Discrete tokens	Autoregressive	A transformer over codec tokens — MusicGen, AudioLM.
11 Latent diffusion	Continuous latent	Diffusion + VAE	Text-to-audio in a compressed latent — Stable Audio, AudioLDM.
12 This part	Continuous latent	VAE + GAN, streaming	Real-time, controllable, accountable — RAVE and beyond.

Navigate the field

The map below places all twelve parts on a timeline, 2016 → now → frontier. Pick a lens — one of the open problems that runs through the whole series — to see which parts advanced it, and click any node for a summary and a link back to that part.

Click a node, or step with the arrows. Choose a lens to highlight the parts that advanced that open problem.

Select a node above to see its thread and a link to that part.

7Open problems & the road ahead

The neural era went from a network that could barely babble a vowel to text-prompted music in under a decade. The remaining problems are not small refinements; they are the axes the next decade will be argued along.

Real-time, high fidelity

Streaming, causal models that match offline quality on a CPU or an edge device. RAVE proved the shape of the answer; closing the quality gap to the big systems is open. §1 · §5

Controllability

Prompt-level generality with knob-level precision, disentanglement, and repeatability — controls a performer can trust the same way twice. §2 · Part 6

Long-form structure

Coherent minutes of music or speech with global form — verses, returns, argument — not just locally plausible texture. Still largely unsolved. Part 10 · 11

Evaluation

Metrics that track human judgement, resist gaming, and stay stable on small sets. FAD, CLAP and KAD are proxies; the gap to MUSHRA is real. §3 · Part 1

Watermarking & attribution

Imperceptible, robust marks (AudioSeal) and verifiable provenance (C2PA) as default, not opt-in — so synthetic audio is accountable. §4

On-device & efficiency

Consistency models, distillation, quantization — the frontier of shrinking a leaderboard-topping model to a phone without spending its soul. §5 · Part 8

The families are primitives; the art is in how you compose them. Twelve parts on, that line from Part 1 is the whole thesis. There is no winning architecture — there is a palette of domains and families, and a set of hard trade-offs, and the systems that matter are the ones that combine them well enough to be fast, controllable, and honest about what they are.

↺Close the loop

You have the whole arc now: the map, every technique that fills it in, and the frontier where they meet. The natural last move is to go back to the beginning and read the map with everything you now know — the domains, the families, and the metrics will look different on the second pass.

Series hubAll twelve parts · the interactive map Part 11 ←Latent Diffusion & Text-to-Audio Part 1 ↺Foundations & the Landscape — revisit the map CollectionDSP & Music

References

Caillon, A. & Esling, P. (2021). RAVE: A variational autoencoder for fast and high-quality neural audio synthesis. arXiv:2111.05011.
Kilgour, K. et al. (2019). Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. Interspeech.
Wu, Y. et al. (2023). Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (CLAP). ICASSP.
San Roman, R. et al. (2024). Proactive Detection of Voice Cloning with Localized Watermarking (AudioSeal). ICML.
Song, Y. et al. (2023). Consistency Models. ICML.
Gretton, A. et al. (2012). A Kernel Two-Sample Test (MMD). JMLR — the basis for kernel audio distances (KAD).
Engel, J. et al. (2020). DDSP: Differentiable Digital Signal Processing. ICLR.
Coalition for Content Provenance and Authenticity (2023). C2PA Technical Specification. (Content Credentials.)