In 2016 WaveNet did the thing everyone thought was too expensive: it modelled raw
audio one sample at a time. This part unpacks how — the factorisation that makes it
exact, the dilated-convolution stack that gives it memory, the gated residual block
at its heart, and the μ-law softmax at its mouth — and confronts the price of doing
it: one strictly-ordered forward pass per sample. Three live demos throughout.
1Autoregression — modelling a waveform one sample at a time
Any joint distribution over a sequence can be factored, exactly and without
approximation, into a product of conditionals by the chain rule of probability. An
autoregressive (AR) model simply parameterises each factor with a neural network
and reads them left to right in time.
Every sample $x_n$ is predicted from the samples before it, and only those — the model
never peeks at the future. Because the factorisation is exact, training is exact too:
maximise the log-likelihood of real audio, which for a sequence is just the sum of the
per-sample log-probabilities. There is no adversary, no lower bound, no reconstruction
surrogate — only a single, honest number to climb.
Teacher forcing — why training is fast but sampling is not
During training the true $x_{every conditional can be
evaluated in parallel across the whole clip in one forward pass — this is
teacher forcing. Generation is the opposite: the model's own outputs
become its next inputs, so the $N$ conditionals must be evaluated in sequence.
That asymmetry — parallel to train, serial to sample — is the defining tension of the
entire family, and Section 6 measures its cost.
Lineage. WaveNet is the audio member of a family born in vision:
van den Oord's PixelRNN / PixelCNN modelled an image pixel-by-pixel with exactly
this factorisation and a categorical output. WaveNet transplanted the idea to the time
axis and swapped the masked 2-D convolutions for dilated causal 1-D ones. The
softmax-over-intensities trick came straight across too.
derivation · why the chain rule is exact and what "teacher forcing" costs
For any random vector the joint always equals the product of conditionals:
$p(x_1,\dots,x_N)=p(x_1)\,p(x_2\mid x_1)\cdots p(x_N\mid x_{
At training time the conditioning context $x_{sampling time $x_{
2Dilated causal convolutions — memory without recurrence
To keep the pitch of a voice stable, the prediction of $x_n$ must depend on samples
thousands back — a full pitch period and more. WaveNet buys that reach without an RNN by
stacking dilated causal convolutions: the kernel taps skip over a gap
that doubles every layer.
Causal means the convolution at time $n$ reads only $x_{\le n}$ (implemented by
left-padding), so the AR ordering is never violated. Dilated means layer
$\ell$ spaces its $k$ taps $d_\ell = 2^{\ell}$ samples apart. Stacking $L$ such layers
with kernel $k=2$ gives a receptive field that grows exponentially in depth while the
compute grows only linearly:
Ten layers already reach back $2^{10}=1024$ samples ($64$ ms at 16 kHz); WaveNet
repeats several such dilation stacks (each resetting $1,2,4,\dots,512$) to reach
hundreds of milliseconds. Part 1's receptive-field playground visualises this
dependency cone; here we take it as given and build the block that lives on top of it.
No recurrence, on purpose. SampleRNN (Section 7) reaches the same
context with a hierarchy of RNNs; WaveNet's convolutional route is fully parallel at
train time and has no hidden-state bottleneck — the trade is a fixed, finite receptive
field rather than an unbounded one.
3The residual block — gated activations, skips, and conditioning
WaveNet is a stack of identical residual blocks. Each one takes the residual stream,
runs a dilated causal convolution, passes it through a gated activation
unit, and splits the result two ways: back into the residual stream, and out to
a global skip sum. Step through the diagram — or hover a stage — to see the signal flow.
Hover a stage or press Step to read what it does. Press Flow to send a signal pulse through the block.
The gated activation unit
Instead of a ReLU, WaveNet borrows the multiplicative gate of the LSTM (via PixelCNN): a
$\tanh$ "content" branch is modulated element-wise by a $\sigma$ "gate" branch, both fed
by their own convolution of the same input.
The sigmoid, bounded in $(0,1)$, acts as a learned valve deciding how much of each
$\tanh$ feature survives — empirically this outperformed a plain rectifier for modelling
audio. The gated output is then projected by a $1\times1$ convolution (a per-timestep
linear layer) into two tensors: one added back to the block input (the residual),
one accumulated into the skip path. After the final block all skip
contributions are summed and passed through ReLU · 1×1 · ReLU · 1×1 · softmax.
Conditioning — how a waveform model becomes a voice
An unconditioned WaveNet babbles plausible-but-meaningless speech. To say words,
the gate is conditioned on side information $\mathbf{h}$. WaveNet distinguishes two kinds:
Global conditioning. A single vector for the whole clip — e.g. a speaker embedding. It is broadcast to every timestep: it biases who is speaking, the timbre and register.
Local conditioning. A time series $\mathbf{h}_n$ — e.g. linguistic features or a mel spectrogram — upsampled to the audio rate. It steers what is said, moment to moment. This is the hook that makes WaveNet a neural vocoder (Part 3).
Two connections do two jobs. The residual path (adding the block's
output back to its input) is the He-et-al. identity shortcut: it keeps gradients healthy
through 30–40 layers so the deep dilation stack is trainable at all. The
skip path is a read-out: every block writes a summand into a
shared bus that feeds the output head, so the final prediction can draw on features from
every dilation scale at once — fine pitch structure from the shallow, low-dilation layers
and slow prosody from the deep, high-dilation ones. Residual carries information
forward through depth; skip carries it sideways to the output.
4The output — a softmax over μ-law levels
WaveNet does not regress a real-valued amplitude; it classifies. The next sample
is drawn from a categorical distribution over a finite alphabet of levels — a choice that
lets the model represent arbitrarily multi-modal, skewed conditionals instead of being
pinned to a single Gaussian mean.
Raw 16-bit audio has 65,536 levels — a hopeless softmax. WaveNet first companded the
signal with the telephony μ-law curve and quantised to just
256 levels (8 bits). Because the curve is steep near zero, the ear's
logarithmic loudness sensitivity is respected and the quiet detail — where most of speech
lives — survives the coarse quantisation (Part 1 derives the constant-SQNR property).
So the head emits a 256-way softmax; the loss is ordinary cross-entropy against the
μ-law bin of the true sample. It is worth pausing on what this buys: the conditional
$p_\theta(x_n\mid x_{bimodal — "this could be the top of the pitch
period or the start of a new one" — something a mean-squared-error regressor can never
express without averaging the two into mush.
Later refinements. The 256-way softmax ignores the ordinal structure of
amplitude (bin 130 is "near" bin 131). WaveRNN and the follow-up WaveNets replaced it with
a discretised mixture of logistics or a dual 8-bit (coarse+fine) split, cutting
the output width while restoring ordinality. The categorical idea, though, is the through-line.
5Sampling — sequential, stochastic, and prone to drift
Here is autoregression made audible. The sampler below seeds a few cycles of a tone, then
generates every subsequent sample in order: a low-order linear predictor proposes
a mean, and the next sample is a temperature-controlled categorical draw over 64 μ-law
bins centred on it. Then it is fed back in and the loop repeats — exactly WaveNet's
generation loop, in miniature.
Temperature 0 = greedy (a rock-steady pitch); raise it and the categorical draw takes over until the tone dissolves into noise.
Temperature is the softmax sharpness. At $T=0$ the draw collapses to the
predictor's argmax — deterministic, a pure resonator, a stable pitch. As $T$ rises the
distribution flattens, draws stray from the prediction, and by $T\!\approx\!1$ the output
is broadband noise. This is the same knob that trades a language model's coherence for its
creativity; in audio it trades tone for hiss.
Teacher-forcing vs free-running exposes the family's Achilles heel.
Free-running (the box unchecked) feeds the model its own outputs, so at any
temperature above zero small errors compound — the pitch wanders and can collapse. Tick
teacher-forcing and the predictor is fed the clean reference instead: the same
temperature now only sprinkles noise on a pitch that never drifts. That gap is
exposure bias — the model is trained on ground-truth context it never
sees at generation time.
Try it. Set $T\approx0.6$ and toggle the box. Free-running: the tone
degrades and destabilises as errors accumulate. Teacher-forcing: rock-steady pitch with a
little grain. Same model, same temperature — the only difference is whether the context is
honest.
Dilation makes context almost free: each added layer doubles the receptive field.
But it does nothing for speed. Producing one second of 16 kHz audio requires
16,000 forward passes that cannot be reordered or batched across time — each needs the
sample the previous one just made. Slide the depth and watch the asymmetry.
Left: receptive field, doubling per layer. Right: the generation cost — fixed at the sample rate, no matter how deep.
Context is cheap. Receptive field — — reachable from a network only tens of layers deep, because depth buys context exponentially.
Time is not. Generation cost — sequential forward passes — independent of depth. The original WaveNet took minutes to synthesise a second of audio. Escaping this is the story of Parts 3 and 8.
The bottleneck is fundamental to the factorisation, not the architecture: sample $n$ is a
function of sample $n-1$, so the $N$ passes form a dependency chain of length $N$. You can
make each pass cheaper (WaveRNN), cache its redundant work (Fast WaveNet), or replace the
whole serial loop with a parallel model (Parallel WaveNet, GAN vocoders, diffusion) — but
you cannot parallelise the chain itself while keeping the exact AR likelihood.
7Escaping the bottleneck — Fast WaveNet to Parallel WaveNet
WaveNet's quality was immediately state of the art and its speed immediately unusable. The
two years that followed were a concentrated assault on the sequential cost — three
distinct strategies, each trading something for throughput.
Method
Idea
What it attacks
Cost / catch
Fast WaveNet
Cache each layer's intermediate activations so a new sample reuses prior convolutions instead of recomputing them.
Redundant compute per pass — $O(2^{L})\!\to\!O(L)$ per sample.
Still strictly one sample at a time; exact same model, just no wasted work.
SampleRNN
A hierarchy of RNNs at different clock rates: slow top tiers set coarse structure, a fast bottom tier emits samples.
Long-range context, cheaply — via recurrence instead of deep dilation.
Still autoregressive at the sample tier; sequential sampling remains.
WaveRNN
A single sparse GRU with a dual (coarse+fine) 8-bit softmax; weight pruning + subscaling to batch and fold the loop.
Per-sample compute and memory — small enough to run faster-than-real-time on a phone CPU.
Sparsity and subscaling add engineering complexity; still fundamentally AR.
Parallel WaveNet
Distil the trained AR "teacher" into an inverse-autoregressive-flow "student" that generates all samples in one parallel pass.
The sequential chain itself — sampling becomes $O(1)$ passes.
Two-stage training; needs probability density distillation to match the teacher.
Parallel WaveNet & probability density distillation
The cleverest of these deserves a closer look. An inverse autoregressive flow
(IAF) can sample in parallel — every output is an invertible transform of an
independent noise sample — but it cannot cheaply score arbitrary data, the mirror
image of WaveNet's problem. Parallel WaveNet trains an IAF student to imitate the
slow AR teacher by minimising the KL divergence between them, evaluated on the
student's own samples (which it can score, and the teacher can too):
The result generates audio faster than real time at the teacher's quality — the pattern
of "train a slow exact model, distil into a fast parallel one" that later reappears
throughout the field (consistency models, distilled diffusion in Part 8). The GAN
vocoders of Part 3 take the more direct route: skip the AR teacher entirely and learn
a one-pass generator adversarially.
Read the sequence as a single arc: WaveNet proved raw-audio modelling is possible;
Fast WaveNet and WaveRNN made it cheaper without changing the maths; SampleRNN found
context through hierarchy; and Parallel WaveNet finally broke the serial chain by
distillation. Everything after is a different bargain with the same $\prod_n p(x_n\mid x_{
8Two years that reshaped audio, 2016 → 2018
The autoregressive era of raw-audio synthesis is astonishingly compressed: from the first
working model to real-time distillation in barely two years.
2016 · early
PixelRNN / PixelCNN establish autoregressive generation of images with a categorical output and masked convolutions — the direct ancestor of WaveNet. lineage
2016 · Sept
WaveNet (van den Oord et al.) models raw audio sample-by-sample with dilated causal convolutions, gated activations, and a 256-way μ-law softmax — a leap in naturalness for TTS. this part
2016 · late
Fast WaveNet caches convolutional activations, cutting per-sample generation from $O(2^{L})$ to $O(L)$ without changing the model. §7
2017 · Feb
SampleRNN reaches long context with a multi-tier hierarchy of RNNs running at different clock rates. §7
2017 · Nov
Parallel WaveNet distils the AR teacher into an inverse-autoregressive-flow student via probability density distillation — faster-than-real-time, and shipped in Google Assistant. §7
2018
WaveRNN compresses everything into a single sparse GRU with a dual softmax and subscaling — high-quality neural audio on a mobile CPU. The AR vocoder becomes practical. §7
→Where to go next
You now have the autoregressive playbook: the exact factorisation, the dilated stack that
gives it memory, the gated residual block, the μ-law softmax, and the sequential-sampling
tax that made everyone want out. The next part is the first clean escape — adversarial
vocoders that generate a whole waveform in a single parallel pass.