A GAN learns to fake a sound in one leap; a diffusion model learns something humbler
and stronger — how to remove a little noise. Stack a thousand such steps and
static resolves into a violin. This part derives denoising diffusion from the forward
noising process to the reverse sampler, connects it to score matching and continuous
flows, and confronts its one real weakness — speed. Three live demos throughout.
1The forward process — destroy a signal on a schedule
Diffusion turns generation inside out. Instead of learning to build a sound from
nothing, it defines a fixed forward process that takes a clean signal
$\mathbf{x}_0$ and, over $T$ steps, drowns it in Gaussian noise until nothing but static
remains. The generative model never has to learn this — it is hand-designed. All the
model learns is how to walk it backwards.
Each forward step adds a pinch of noise and shrinks the signal slightly, governed by a
variance schedule $\beta_1,\dots,\beta_T$ with $0<\beta_t\ll 1$:
The magic is that you never need to iterate. Because a composition of Gaussians is
Gaussian, the marginal at any step has a closed form in terms of
$\mathbf{x}_0$ alone. Writing $\alpha_t = 1-\beta_t$ and
$\bar\alpha_t=\prod_{s=1}^{t}\alpha_s$, a single reparameterisation samples $\mathbf{x}_t$ directly:
Drag the timestep in the demo. At $t=0$ you hear and see the clean tone; slide right and
watch $\sqrt{\bar\alpha_t}$ fade the signal while $\sqrt{1-\bar\alpha_t}$ raises the
noise floor, until the waveform is indistinguishable from white noise. Then press
Denoise and watch the process run in reverse — from pure static back to
the tone, one small step at a time — and hear the recovered sound.
Forward: slide t to add noise per the cosine schedule. Reverse: press Denoise to sample x₀ back from pure noise, then hear it.
Why this is a good idea. The forward process manufactures, for free, an
endless supply of (noisy input, known clean target) pairs at every noise level.
That converts an impossible density-estimation problem into a mundane supervised
regression: given a noisy signal and its noise level $t$, predict what was added. No
adversary, no likelihood integral — just a squared error.
derivation · the closed form q(x_t | x_0)
Apply the recursion and the reparameterisation trick twice. With
$\boldsymbol\epsilon_{t-1},\boldsymbol\epsilon'\sim\mathcal N(0,\mathbf I)$ independent,
The two noise terms are independent Gaussians; their sum has variance
$\alpha_t(1-\alpha_{t-1})+(1-\alpha_t)=1-\alpha_t\alpha_{t-1}$, so they merge into one
Gaussian of that variance. Unrolling all the way to $\mathbf{x}_0$ telescopes the
product of $\alpha$'s into $\bar\alpha_t$:
This is why diffusion training is cheap: to make a sample at level $t$ you draw one
$\boldsymbol\epsilon$ and one multiply-add — never $t$ sequential steps. As
$\bar\alpha_t\!\to\!0$ the signal term vanishes and $\mathbf{x}_T\!\approx\!\boldsymbol\epsilon$,
a standard normal with no dependence on the data — which is exactly why we can
start reverse sampling from pure noise.
2The reverse process — one network, every noise level
Reversing the chain means sampling from $q(\mathbf{x}_{t-1}\mid\mathbf{x}_t)$. That true
reverse is intractable — it depends on the whole data distribution — but for small
$\beta_t$ it is provably close to Gaussian, so we approximate it with a learned Gaussian
$p_\theta(\mathbf{x}_{t-1}\mid\mathbf{x}_t)=\mathcal N(\mathbf{x}_{t-1};\ \boldsymbol\mu_\theta(\mathbf{x}_t,t),\ \Sigma_t)$.
The only thing the network has to produce is that mean.
Ho et al. (2020) found the parameterisation that makes it work. Rather than predict
the mean directly, predict the noise $\boldsymbol\epsilon$ that was
added — a network $\boldsymbol\epsilon_\theta(\mathbf{x}_t,t)$. The posterior mean then
falls out in closed form, and the training objective collapses to a plain denoising
regression, weighted uniformly across timesteps (the celebrated "simple" loss):
Train one shared network on this single objective — pick a random clean clip, a random
$t$, a random $\boldsymbol\epsilon$; corrupt; ask it to name the noise. At sampling time,
start from $\mathbf{x}_T\sim\mathcal N(0,\mathbf I)$ and step down using its predictions.
The score-matching view — the same thing from physics
Song & Ermon arrived at diffusion from a different door: score matching. The
score of a distribution is the gradient of its log-density,
$\nabla_{\mathbf x}\log p(\mathbf x)$ — a vector field pointing "uphill" toward likely
data. If you knew it, you could sample by following it (Langevin dynamics). The link to
DDPM is exact: predicting the noise is estimating the score of the
noise-perturbed data, up to a scale set by the noise level.
Song's 2021 synthesis recast the whole thing as a stochastic differential
equation: the forward process is an SDE that drifts the data toward noise, and
generation solves the corresponding reverse-time SDE, whose drift is exactly the
score. DDPM's discrete chain is one particular discretisation of that SDE. This is the
unifying picture the rest of the field builds on.
derivation · from the variational bound to the simple loss
The true training target is the variational (ELBO) bound on
$-\log p_\theta(\mathbf x_0)$, a sum of KL terms
$D_{\mathrm{KL}}\!\big(q(\mathbf x_{t-1}\mid\mathbf x_t,\mathbf x_0)\,\|\,p_\theta\big)$.
Bayes' rule makes the forward posterior Gaussian with a closed-form mean
Each KL between two Gaussians of equal covariance is just a scaled squared
difference of means, $\propto\lVert\tilde{\boldsymbol\mu}_t-\boldsymbol\mu_\theta\rVert^2$.
Substituting $\mathbf x_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon)/\sqrt{\bar\alpha_t}$
and matching terms shows the optimal $\boldsymbol\mu_\theta$ is the sampler formula
above, and the residual is precisely
$\lVert\boldsymbol\epsilon-\boldsymbol\epsilon_\theta\rVert^2$ times a
$t$-dependent weight. Ho et al.'s decisive empirical finding was that
dropping that weight — training all timesteps equally — gives better samples.
That is the "simple" loss: a maximum-likelihood objective wearing the clothes of a
denoiser.
The oracle reverse trajectory in Demo 1 uses exactly
$\tilde{\boldsymbol\mu}_t$ with the true $\mathbf x_0$ — a perfect
$\boldsymbol\epsilon_\theta$ — which is why it converges cleanly back to the tone. A
trained network only approximates it.
3Noise schedules — the dial that controls destruction
The schedule $\{\beta_t\}$ is the one free design choice in the forward process, and it
matters more than it looks. It decides how fast information is destroyed — and
therefore how the model's limited capacity is spread across the noise levels it must
learn. Get it wrong and you either waste steps on near-clean signals or annihilate the
structure too early.
The original DDPM used a linear schedule, $\beta_t$ rising linearly from
$10^{-4}$ to $0.02$. Nichol & Dhariwal showed it destroys information too quickly near
the end: the last quarter of steps operate on what is already almost pure noise, doing
little useful work. Their cosine schedule fixes this by defining
$\bar\alpha_t$ directly so it decays gently and spends more steps in the perceptually
critical middle range:
Switch the schedule below and watch all three views update. The key diagnostic is the
log-SNR, $\log\!\big(\bar\alpha_t/(1-\bar\alpha_t)\big)$: it measures how
much signal survives relative to noise at each step. The cosine curve keeps log-SNR
changing smoothly and lingers longer near the informative zero-crossing, whereas the
linear schedule plunges through it.
Top: per-step variance β_t. Middle: cumulative signal retention ᾱ_t. Bottom: log-SNR — the schedule's true signature.
Read the middle panel. $\bar\alpha_t$ is the fraction of the clean
signal's amplitude that survives to step $t$. Where the cosine curve stays high for
longer, the model gets many training examples that are only mildly corrupted — the regime
that fixes fine detail. The linear curve dumps that budget into near-pure-noise steps
that teach the model almost nothing new.
4Diffusion for audio — DiffWave, WaveGrad and guidance
The 2020 image results landed on audio within months. Two vocoders arrived almost
together — DiffWave and WaveGrad — both generating a raw
waveform by denoising, both conditioned on a mel-spectrogram. They are diffusion's
answer to the GAN vocoders of Part 3: given the mel that a TTS acoustic model
produced, invent the matching audio.
Model
What it does
Architecture
Note
DiffWave
Denoising diffusion on the raw waveform, conditioned on mel. Also works unconditionally and for class-conditional generation.
Bidirectional dilated-conv stack (WaveNet-like), no autoregression — the whole clip is denoised in parallel per step.
Estimates the data gradient (score) to refine noise into audio, conditioned on mel; conditions on a continuous noise level rather than a discrete index.
UNet-style with FiLM conditioning on the mel and on $\sqrt{\bar\alpha}$.
Trades steps for quality on a single trained model.
Conditioning — how the mel gets in
The mel-spectrogram enters the denoiser as a side input at every layer, typically
upsampled to the audio rate and injected through FiLM / bias-scale modulation. Formally
the model learns the conditional score
$\boldsymbol\epsilon_\theta(\mathbf x_t,t,\mathbf c)$ with $\mathbf c$ the mel. Everything
from Sections 1–2 carries over unchanged; the condition just rides along. This is why the
same machinery serves TTS (mel from text), singing (Part 5's DiffSinger), and
bandwidth extension.
Classifier-free guidance — trading diversity for fidelity
To make a conditional model follow its condition more tightly, classifier-free
guidance (Ho & Salimans) trains one network to do both conditional
and unconditional denoising (by randomly dropping $\mathbf c$), then at sampling time
extrapolates away from the unconditional prediction:
The guidance weight $w>1$ sharpens adherence to the prompt at some cost in diversity — the
same knob that powers text-to-image and, in Parts 11–12, text-to-audio. It needs no
separate classifier, which is why it became universal.
The cost that defines the rest of this part. Every DiffWave sample is
many forward passes through a large network — DiffWave's parallelism across time is a huge
win over WaveNet's per-sample autoregression, but it is still tens of network evaluations
where a GAN vocoder needs one. Section 5 is about closing that gap.
5The step-count problem — and how to sample fast
Diffusion's Achilles heel is sampling speed. Quality is superb and mode coverage is
excellent — it does not collapse the way a GAN can — but a naive DDPM sampler needs
hundreds to a thousand sequential network evaluations. That is the same
"sequential-slowness" theme that dogged the autoregressive models of Part 2, in a new
costume. The last five years of diffusion research are largely one question: fewer
steps, same quality.
Move the step count in the demo. With a good sampler, reconstruction error falls
steeply for the first handful of steps and then flattens — a strong signal that most of
the thousand-step budget is wasted. The wall-clock scales linearly with steps, so the
engineering prize is obvious.
Left: reconstruction error vs steps (DDIM-style deterministic sampler). Right: the recovered signal at the chosen budget, over the clean target.
Four ways out
Method
Idea
Typical steps
DDIM
A deterministic, non-Markovian sampler sharing the same trained network. Removes the injected noise $\sigma_t\mathbf z$, so you can skip timesteps and take a straight ODE path.
10–50
Progressive distillation
Train a student to match two teacher steps in one, then halve again, repeatedly — each round doubles speed.
4–8
Consistency models
Learn a network that maps any point on a noise trajectory directly to its origin, enabling one- or two-step generation.
1–2
Rectified flow / flow matching
Train straight-line probability paths so an ODE solver needs very few steps; "reflow" straightens them further.
1–4
DDIM is the workhorse — same weights, a caption-worthy trick. The rest retrain or
distil for extra speed. Consistency models and rectified flow are the current
frontier, pushing audio diffusion toward the single-pass latency that GANs enjoyed all
along, without the mode collapse.
derivation · why DDIM can skip steps
DDIM defines a family of non-Markovian forward processes that all share the same
marginals $q(\mathbf x_t\mid\mathbf x_0)$ as DDPM — so a network trained with the
simple loss is already a valid DDIM model, no retraining. Its update first
forms the predicted clean signal (Tweedie's formula),
With the stochastic term set to zero this is a deterministic map — an Euler step of
the probability-flow ODE. Because there is no re-injected randomness to integrate out,
you can evaluate it on any decreasing subset of timesteps and still land near the data
manifold; the discretisation error, not lost noise, is all that grows as you take fewer,
larger steps. That is exactly the error curve the demo traces.
6Flows — exact likelihood and the road to flow matching
The "& Flow" in the title is not an afterthought — normalizing flows are diffusion's
close cousin, and the two are converging. A flow is an invertible neural network
$f$ that maps data to a simple base distribution (a Gaussian) and back. Because it is
bijective with a tractable Jacobian, it gives something diffusion and GANs cannot: the
exact log-likelihood, via the change-of-variables formula.
WaveGlow (NVIDIA, 2019) is the canonical audio flow: a stack of
invertible affine coupling layers, conditioned on the mel-spectrogram, that turns
a Gaussian into a waveform in a single parallel forward pass — no autoregression, no
iterative denoising. Trained purely by maximising the exact likelihood above, it sidesteps
the adversarial instability of GAN vocoders and the many-step cost of diffusion. The price
is architectural: invertibility with a cheap Jacobian is a real constraint, so flows tend
to need many parameters for a given quality, and the affine-coupling trick limits how the
network can mix its inputs.
Continuous flows meet diffusion
Push the number of coupling layers to a continuum and a flow becomes an ODE — a
continuous normalizing flow — that transports the Gaussian to the data along a
smooth trajectory. This is the bridge: diffusion's probability-flow ODE and a CNF are the
same kind of object. Flow matching (Lipman et al., 2023) trains such
an ODE directly by regressing its velocity field onto a chosen target path, and
rectified flow chooses straight paths so the ODE integrates in a
handful of steps:
Notice the family resemblance to the diffusion loss of Section 2 — both regress a network
onto a simple target under a linear interpolation of data and noise. Flow matching is
increasingly the training objective of choice for new audio and music generators precisely
because it inherits diffusion's stability and coverage while sampling in far fewer steps.
Diffusion vs flow, one line. Diffusion learns to reverse a stochastic corruption step by step; a flow learns a single deterministic invertible map. Flow matching unifies them: a deterministic ODE trained the diffusion way.
Where flows lost. Pure normalizing flows never matched diffusion's peak quality at equal size — the invertibility constraint is expensive. Their ideas survived not as standalone vocoders but folded into VITS (Part 6) and into the flow-matching objective now training the latest systems.
7Four years, static to state of the art
Diffusion went from an image-generation curiosity to the dominant paradigm for
high-fidelity audio and music in about four years. Each line below is a step on the road
from "beautiful but slow" to "beautiful and fast."
2020
DDPM (Ho et al.) makes denoising diffusion practical with the ε-prediction simple loss; DiffWave & WaveGrad bring it to audio as mel-conditioned vocoders. Part 8
2021
DDIM enables deterministic few-step sampling on trained models; Song et al.'s score SDE unifies diffusion and score matching; cosine schedules and classifier-free guidance sharpen quality. Part 8
2022
Progressive distillation compresses samplers to a handful of steps; latent diffusion moves the process into a VAE space, powering Riffusion and AudioLDM. Part 11
2023
Consistency models reach one/two-step generation; flow matching & rectified flow straighten the paths; Stable Audio diffuses long musical structure in a latent. Part 11 · 12
→Where to go next
You now hold the full diffusion toolkit: the forward noising process and its closed form,
the ε-prediction reverse sampler and its score-matching twin, the schedules that shape
destruction, the audio models that put it to work, the many-step problem and the
DDIM/distillation/consistency/flow-matching answers to it. Diffusion shines in a
latent space rather than on raw samples — which is exactly why the next parts
build the codecs and VAEs that give it one.