Before any model can generate a voice or a violin, it has to solve one brutal
problem: an audio second is tens of thousands of numbers, every one entangled with
the rest. This part builds the vocabulary of the whole field — the four domains a
model can work in, the five ways to place a distribution over them, and how we tell
whether the result is any good. Four live demos throughout.
1The problem — audio is enormous, and every sample is entangled
A neural network that generates sound must learn a probability distribution
$p(\mathbf{x})$ over waveforms $\mathbf{x}$ and then draw samples from it. That sounds
like image generation, and much of the machinery is shared — but audio has two
properties that make it uniquely hard.
First: the dimensionality is staggering
Compact-disc audio is 44,100 samples per second per channel. One second of mono audio
is a point in a 44,100-dimensional space; a three-minute song lives in nearly eight
million dimensions. Compare a $256\times256$ image at 65,536 pixels — a single second
of speech at 16 kHz already exceeds a quarter of that, and songs dwarf it.
$$\dim(\text{1 s @ }16\text{ kHz}) = 16{,}000 \qquad \dim(\text{3 min song @ }44.1\text{ kHz, stereo}) \approx 2.4\times10^{7}$$
Second: the correlations span every time scale at once
In a spoken sentence, the sign of sample $n$ is set by the pitch period (a few
milliseconds), the phoneme (tens of milliseconds), the word (hundreds), and the
sentence prosody (seconds) — simultaneously. A model that captures pitch but
not prosody produces robotic monotone; one that captures words but not pitch periods
produces buzzing mush. This is the long-range-dependency problem, and
it is the reason the field fractured into so many approaches.
Drag across the demo below to hear it. The same 0.4-second clip is generated three
ways; watch how structure at one scale (the harmonic comb, the formant envelope, the
decay) is a different-looking pattern in each domain — yet it is one sound.
Why not just predict the next sample and be done? You can — that is
Part 2's WaveNet. At 16 kHz a one-second clip needs 16,000 sequential forward
passes, and each must "remember" thousands of samples back to keep the pitch stable.
Every technique in this series is, at heart, a different bargain with that arithmetic.
2Four domains — where a model chooses to work
You almost never generate raw samples directly from scratch. Instead you pick a
representation that makes the distribution easier to learn, generate in that
domain, and convert back. There are four dominant choices, in increasing order of
abstraction.
The same 0.4 s of audio, four ways. All four panels are computed live from one generated clip.
Domain
What it is
Dimensionality / step
Who works here
Raw waveform
The samples themselves, $x_n\in[-1,1]$. Maximum fidelity, maximum burden — the model owns phase, pitch, timbre, everything.
1 number / sample · 16k–48k per second
WaveNet, SampleRNN, DiffWave (P2, P8)
Spectrogram
Short-time Fourier magnitudes, usually mel-scaled. Throws away phase; compresses time ~256×. A vocoder must invert it.
~80 mel bins / frame · ~86 frames/s
Tacotron, FastSpeech, DiffSinger (P4, P5)
Discrete tokens
A neural codec quantizes audio into a small vocabulary of integers — like text. Enables transformer language models over sound.
~4–8 codes / frame · ~50 frames/s
EnCodec, AudioLM, MusicGen (P9, P10)
Continuous latent
A learned, compressed vector space from a VAE. Smooth and low-dimensional — ideal for diffusion and interpolation.
~64 dims / frame · ~20–40 frames/s
Stable Audio, RAVE, AudioLDM (P11, P12)
The trend of the last decade reads straight down that table: the field kept climbing
the ladder of abstraction, letting a separately trained decoder shoulder the
fidelity while the generative model works in an ever-friendlier space. The cost is a
two-stage pipeline and a decoder that can hallucinate — a tension every later part
returns to.
derivation · why the mel spectrogram is ~256× smaller yet keeps what matters
A mel spectrogram with STFT hop $H$ and $M$ mel bands stores $M$ numbers every
$H$ samples, versus $H$ raw samples over the same span. The compression ratio is
$$\rho = \frac{H}{M}.$$
Typical speech settings $H=256,\,M=80$ give $\rho = 3.2$ per channel — but the
raw signal also carries phase, which the magnitude spectrogram discards entirely.
Counting the real+imaginary STFT that the magnitude replaces, a length-$N$ frame's
$N$ real samples map to $N/2{+}1$ complex bins ($N{+}2$ reals) then collapse to $M$
magnitudes. With $N=1024$: $1024 \to 80$, about $13\times$ per frame, and because
frames overlap by $H/N$ the effective per-sample saving compounds. The
perceptual justification is that the mel warping and magnitude-only
encoding discard exactly the information the ear is least sensitive to — absolute
phase and fine high-frequency resolution — which is why a vocoder can plausibly
invent it back (Part 3).
3Quantization — how autoregressive models turn a knob into a word
A model that predicts the next sample as a classification problem (WaveNet
does exactly this) needs the continuous amplitude turned into a finite alphabet. Naive
linear quantization wastes its bits: the ear hears loudness logarithmically, so quiet
passages — where most of the signal lives — get almost no levels. The fix is
μ-law companding, borrowed from 1960s telephony.
Compress with $F$, quantize the result uniformly to 256 levels (8 bits), and WaveNet's
original 16-bit-quality audio becomes a 256-way softmax — small enough to be tractable,
and because the companding curve is steep near zero, the quiet detail survives.
Left: the companding curve. Right: quantization error on the signal. Drop to 3–4 bits and toggle μ-law to hear why telephony chose it.
Try this: set 3 bits and play with μ-law off — the quiet tail
of the decaying tone turns to gravelly noise. Switch μ-law on at the same 3
bits: the tail is recognisable again. Same number of levels; smarter placement.
derivation · quantization noise and the 6 dB-per-bit rule
Uniform quantization to $b$ bits over a full-scale range $2A$ has step
$\Delta = 2A/2^{b}$. Modelling the rounding error as uniform on
$[-\Delta/2,\Delta/2]$ gives noise power
Each bit buys ~6 dB — but only on average across the range. The catch
that μ-law fixes: a signal at $-40$ dB full scale only exercises the lowest few
codes, so its effective $b$ collapses. Companding warps the step sizes to be small
where the amplitude is small, holding SQNR roughly constant across loudness. That
constant-SQNR property is exactly what a generative model wants: it must model quiet
and loud passages with equal relative precision.
4Long context — the dilated-convolution trick
To keep pitch stable a waveform model must "see" thousands of past samples, but a plain
stack of convolutions with kernel $k$ over $L$ layers only reaches back
$L(k-1)+1$ samples — linearly. WaveNet's decisive idea was the dilated causal
convolution: double the gap between kernel taps each layer, so the receptive
field grows exponentially while the compute grows only linearly.
With kernel $k=2$, ten layers reach back $2^{10}=1024$ samples; stack a few such blocks
and you cover the hundreds of milliseconds a pitch contour needs — from a network only
tens of layers deep. Drag the slider and watch the cone of dependency widen.
Each layer doubles the dilation. The lit edges trace which inputs the rightmost output actually depends on.
Exponential reach. Receptive field — — that many input samples feed the newest prediction, at a cost of only — layers.
The catch. This buys context, not speed. Generation is still one sample at a time: 16,000 forward passes per second. Parts 3, 8 and 9 are largely about escaping that sequential bottleneck.
5Five model families — five ways to place a distribution
Given a domain, how do you actually model $p(\mathbf{x})$ and sample from it? Five
families dominate audio. Each is a different answer to the same trilemma —
quality, sampling speed, and mode coverage (does it capture
the full diversity of the data, or collapse to a few safe outputs?). No family wins all
three; the history of the field is largely the search for better trade-offs.
A schematic of how each family turns noise / context into a sample. Press Generate to animate the process.
The five, and their bargain
Family
Core idea & objective
Quality
Speed
Coverage
Autoregressive
Factor $p(\mathbf{x})=\prod_n p(x_n\mid x_{
Excellent
Very slow (sequential)
Excellent
GAN
A generator maps noise to audio; a discriminator learns to spot fakes; they play a minimax game. No likelihood. MelGAN, GANSynth.
High
Very fast (1 pass)
Can collapse
Diffusion
Learn to reverse a gradual noising process; sample by denoising from static. Score matching. DiffWave.
Excellent
Slow→fast (steps)
Excellent
Normalizing flow
An invertible network with tractable Jacobian; exact likelihood, parallel sampling. WaveGlow, the flow inside VITS.
High
Fast (1 pass)
Good
VAE
Encode to a latent, decode back; maximise the evidence lower bound. Rarely generative alone — the compressor behind codecs & latent diffusion.
Good (blurry solo)
Fast
Good
the maths, in one line each
Autoregressive — maximise the exact log-likelihood factorised over time:
Read the modern systems as combinations. VITS is a flow, a VAE, and a GAN at
once. MusicGen is an autoregressive transformer over a VAE-like codec's tokens. Stable
Audio is diffusion inside a VAE latent. The families are primitives; the art is in how
you compose them.
6Evaluation — you cannot optimise what you cannot measure
Generated audio has no single "correct" target, so the field uses a ladder of metrics:
subjective listening tests at the top, objective embedding distances below, and
signal-level measures at the bottom. Every later part reports the ones that matter for
its task.
Subjective — the ground truth
MOS: listeners rate naturalness 1–5; report the mean with a
confidence interval. MUSHRA: a rigorous multi-stimulus test with a
hidden reference and a hidden low-quality anchor, scored 0–100. Expensive and
slow, but the only true measure of "does this sound real?"
Objective — the proxy
FAD (Fréchet Audio Distance) compares the distribution of
generated vs real audio in a pretrained embedding space. PESQ and
STOI score speech quality/intelligibility against a reference.
CLAP score measures how well audio matches a text prompt.
with $(\mu_r,\Sigma_r)$ and $(\mu_g,\Sigma_g)$ the mean and covariance of embeddings of
a real and a generated set. It is the audio twin of the FID used for images: lower is
better, and unlike a per-sample loss it rewards matching the corpus as a whole rather
than any one target. Part 12 dissects FAD and CLAP in full.
Every objective metric is gameable. A model can score well on FAD and
still sound wrong to a human; a vocoder can ace PESQ and buzz audibly. This is why the
honest papers still run listening tests — and why the demos in this series ask
you to be the judge.
7A decade in one column
The neural era is astonishingly compressed — from the first raw-audio network to
text-prompted music in under seven years. The rest of the series expands each of these
lines into a full part.
2016
WaveNet generates raw audio sample-by-sample with dilated causal convolutions — proof that a network can model a waveform directly. Part 2
2017
Tacotron and Tacotron 2 make text-to-mel-spectrogram end-to-end with attention; NSynth releases a huge instrument-note dataset. Part 4 · 7
2018–19
GANSynth shows adversarial instrument timbre; MelGAN and Parallel WaveGAN make GAN vocoders fast and real-time. Part 3 · 7
2020
DDSP fuses classical synthesis with autograd; HiFi-GAN sets the vocoder standard; DiffWave & WaveGrad bring diffusion to audio; VITS unifies TTS end-to-end. Part 3 · 4 · 6 · 8
2021–22
SoundStream & EnCodec turn audio into discrete tokens; AudioLM generates audio as a language; Riffusion diffuses spectrograms from text. Part 9 · 10 · 11
2023–24
MusicGen, AudioLDM, Stable Audio deliver controllable text-to-music; RAVE and consistency models push toward real-time. Part 10 · 11 · 12
→Where to go next
You now have the whole map: the domains a model can work in, the families that place a
distribution over them, and the metrics that judge the result. The natural next step is
the technique that started the neural era — predicting a waveform one sample at a time.