How machine learning learned to make sound — the voice, music, and instruments — from WaveNet's sample-by-sample gamble to diffusion models and language models that compose. Deep theory, live in-browser demos, and the maths worked all the way through. Every sound on these pages is synthesised live in the Web Audio API; every diagram computes in real time.
For seventy years, synthesising sound meant hand-designing a signal-processing algorithm — an oscillator bank, a filter, a physical model of a string. You wrote the equations; the computer executed them. Neural sound synthesis inverts that: you supply examples of the sound you want, and a network learns the generating process itself, discovering structure no engineer wrote down.
This series is a comprehensive, technical tour of that revolution — the models, the mathematics, and the ideas that connect them. It is built for people who want to understand, not just use: every part derives its core equations, shows the architecture as an interactive diagram, and lets you hear the technique running live. It sits alongside the classical-DSP synthesis work in the DSP & Music collection — where neural methods stand on the shoulders of the vocoder, the spectrogram, and differentiable filters.
The central question of the field: an audio second is 16,000–48,000 numbers, each depending on all the others. How do you learn a probability distribution over that — and sample from it fast enough to be useful? — the thread running through all twelve parts
Every generative-audio system is a choice along two axes: what domain it models the sound in (raw samples, a spectrogram, discrete tokens, or a learned latent), and what generative family it uses to place a distribution over that domain. Hover the map to trace where each landmark model sits — the whole series is a walk across this plane.
Nothing on this map is purely one thing. VITS is a flow and a VAE and a GAN; MusicGen is a language model over a codec's tokens. The families are lenses, not boxes — and the most capable modern systems deliberately combine them.
The series covers the whole of synthetic sound, but three application pillars recur. Each imposes different constraints, and the techniques cross-pollinate constantly.
| Pillar | What is hard about it | Signature techniques & parts |
|---|---|---|
| The Voice speech & singing |
Intelligibility is unforgiving; humans are expert critics of voices. Prosody, pitch, vibrato and phoneme timing must all be controllable and natural. | Vocoders (P3), TTS acoustic models & VITS (P4), singing-voice synthesis & voice conversion (P5) |
| Music full mixes & songs |
Structure over minutes, many simultaneous sources, and text/style control. The long-range-dependency problem at its most extreme. | Neural codecs (P9), language-model music (P10), latent-diffusion text-to-audio (P11) |
| Instruments single-note timbre |
Continuous, meaningful control of timbre and pitch; faithful attack transients; interpolation between instruments that never existed. | DDSP (P6), GAN instrument synthesis / GANSynth (P7), real-time timbre transfer (P12) |
Read in order for a complete education, or jump to a technique. Each part is a standalone interactive presentation with live audio, derived mathematics, and an animated architecture.
All twelve parts are live. Read them in order for a complete education, or jump straight to a technique — each stands alone.
A generative-audio model cannot be scored by a single loss. The field uses a ladder of metrics — from human listening tests to embedding-space distances — and each part reports the ones that matter for its task. The two you will meet everywhere:
Human listeners rate naturalness from 1 (bad) to 5 (excellent); the MOS is the average. Its rigorous cousin MUSHRA hides a reference and anchors among the candidates. Slow and expensive, but the only ground truth for "does this sound real?"
FAD mirrors the FID of image generation: embed real and generated audio with a pretrained network, fit a Gaussian to each set, and measure the distance between them —
where $(\mu_r,\Sigma_r)$ and $(\mu_g,\Sigma_g)$ are the mean and covariance of the real and generated embeddings. Lower is better; unlike a per-sample loss it rewards matching the distribution of a corpus. Part 12 dissects it, and CLAP score — how well generated audio matches a text prompt — in full.
Every part is a single self-contained web page — no installation, no accounts, no audio files downloaded. Open it and the sound is generated on your machine.
| Turn the volume up | The demos are the argument. Reading about a vocoder is nothing like hearing Griffin–Lim and HiFi-GAN invert the same spectrogram. |
| Open the derivations | Collapsed ▸ derivation panels hold the full maths. Skip them for the intuition; open them for the proof. |
| Follow the cross-links | Each part ends with links to its neighbours. Diffusion (P8) makes more sense after codecs (P9) which makes more sense after vocoders (P3). |
| These are teaching models | The in-browser networks are small, real implementations of the ideas — enough to hear the mechanism — not the billion-parameter production systems. The maths is the real thing. |