Diffusion made stunning images by denoising in a compressed latent, not on raw pixels.
Audio borrowed the trick wholesale: run the reverse process inside a learned, tiny
latent from a VAE, steer it with a shared text–audio embedding, and a plain
prompt becomes a barking dog, a rainstorm, or a full stereo track. Three live demos
build the picture from the spectrogram up.
Part 8 ran diffusion directly on audio: start from noise, denoise toward a waveform or a
full-resolution spectrogram. It works, but it is punishingly expensive — every one of the
dozens-to-thousands of reverse steps evaluates a network over a signal with tens of
thousands of dimensions. Latent diffusion is the fix that unlocked
text-to-image (Rombach et al., 2022) and then text-to-audio: don't diffuse the signal,
diffuse a compressed code of it.
The recipe is two-stage. First, a variational autoencoder (Part 9) learns
to squeeze audio into a small continuous latent $\mathbf{z}$ and reconstruct it faithfully —
a perceptual compressor that throws away what the ear won't miss. Then a diffusion model is
trained to generate in that latent, and the VAE decoder turns each generated
$\mathbf{z}$ back into sound. The generative network never touches a raw sample; it works in
a space that may be dozens of times smaller.
This is the ordinary denoising objective from Part 8 — predict the noise $\epsilon$ added
at level $t$ — with two changes: it lives on the latent $\mathbf{z}_t = \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$
instead of the signal, and it takes a conditioning $\mathbf{c}$ — usually a
text embedding — injected through cross-attention. Cut the dimensionality and you cut both
the memory and the number of steps needed for a given quality; add the conditioning and the
same machine becomes a prompt-driven instrument.
Why it is cheap. A U-Net or DiT denoiser costs roughly the
square of the token count in its attention. Compress time–frequency by, say, 8× along
each axis before diffusing and you shrink that quadratic cost by orders of magnitude — the whole
reason latent diffusion runs on a single GPU.
The catch, inherited from Part 9. Quality is now capped by the
autoencoder. If the VAE can't represent a transient or a sibilant, no amount of diffusion in its
latent will recover it — the decoder can only ever hand back sounds it was trained to make.
The rest of this part is the anatomy of that machine, one component per section: the image the
diffusion sees (§2), the latent it actually lives in (§3), and the text embedding that steers it (§4).
2Spectrograms as images — the Riffusion trick
The fastest route to text-to-audio, historically, skipped the custom latent entirely.
Riffusion (Forsgren & Martiros, 2022) fine-tuned Stable Diffusion
itself — an image model — on pictures of mel spectrograms paired with text. A spectrogram
is, after all, just a greyscale image: time on one axis, frequency on the other, brightness for
energy. Diffuse the image, then invert it back to sound.
Inversion is the only audio-specific step. The magnitude spectrogram has thrown away phase, so a
vocoder must invent it: Riffusion used Griffin–Lim, which iterates
STFT ↔ inverse-STFT to hallucinate a phase consistent with the given magnitudes.
Because Stable Diffusion's latent is spatially smooth, two prompts can be interpolated and the
audio morphs seamlessly — and a spectrogram can be generated to loop, giving endless variations
of a groove.
The demo below is a teaching model of the reverse process operating in the image domain.
On the left is the target spectrogram of a short clip we generated. On the right, we corrupt it to
pure noise and then run reverse-diffusion steps that progressively denoise it back. To make a clean,
convergent trajectory you can watch, the demo uses the known target as an oracle — a real
trained $\epsilon_\theta$ estimates the same denoising direction, imperfectly. Set the step count,
press Denoise, then Invert & play to hear the current spectrogram reconstructed
with oracle phase.
Left: target spectrogram. Right: a fully-noised spectrogram. Press Denoise to run the reverse trajectory.
Try this: denoise with just 4 steps, invert and listen — the reconstruction
is grainy, the harmonics smeared. Now push the slider to 60 steps and repeat: each reverse
step removes a slice of noise variance, the harmonic combs sharpen, and the clip rings clean. That
step–quality trade is the single dial every diffusion sampler exposes.
the reverse step, and why more steps means cleaner audio
Forward diffusion adds Gaussian noise on a schedule $\bar\alpha_t$, so a level-$t$ latent is
$\mathbf{z}_t=\sqrt{\bar\alpha_t}\,\mathbf{z}_0+\sqrt{1-\bar\alpha_t}\,\epsilon$. The trained network
predicts $\epsilon$, which gives an estimate of the clean signal
$\hat{\mathbf{z}}_0=(\mathbf{z}_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}$, and
each reverse step nudges $\mathbf{z}_t$ toward $\mathbf{z}_{t-1}$ using it:
Each step is a linear approximation to a curved probability-flow trajectory. Few steps means
large jumps that overshoot the curve — audibly, smeared transients and rough noise floors. Many
steps hug the curve and land closer to the true data manifold. Fast samplers (DDIM, DPM-Solver)
buy back most of the quality at low step counts by using higher-order updates; the demo's linear
oracle blend is the intuition, not the production sampler.
3The VAE latent — a small, smooth space to walk in
Riffusion diffuses full spectrogram images; modern audio systems diffuse a purpose-built
continuous latent from an audio VAE (Part 9). Two properties make that latent the
right place to work. It is compact — Stable Audio's autoencoder compresses 44.1 kHz
stereo by a large factor, so the diffusion model sees a short sequence of fat vectors instead of a
huge spectrogram. And it is smooth: the VAE's KL prior packs the codes into a well-behaved
region where nearby points decode to perceptually nearby sounds.
Smoothness is what makes generation and interpolation work. If a straight line between two latents
passed through "holes" that decode to garbage, diffusion — which spends its whole trajectory moving
through that space — would keep falling into them. Instead you can take two codes and walk between
them, and the sound morphs continuously. Drag the slider to walk the latent path below; the
spectrogram at the current point is decoded live, and you can hear either endpoint.
t = 0.00 — endpoint A. Drag the slider to interpolate the latent; the spectrogram is decoded at each point.
Endpoint A is a low, dark, near-harmonic tone; B is a high, bright, slightly inharmonic and noisier
one. Notice the morph is monotone — brightness, pitch and noisiness slide together with no
sudden jumps. That is the geometric property a diffusion model relies on, and the reason latent
interpolation is a headline feature of these systems rather than an accident.
Latent vs pixel diffusion, in one line. The VAE decoder shoulders fidelity; the
diffusion model only has to place a distribution over a friendly low-dimensional code. The same
division of labour recurs across the field — codecs+LMs (Part 10), latent+diffusion here.
4Text conditioning — CLAP and classifier-free guidance
A latent diffusion model generates something from noise; text conditioning decides
what. The bridge is a shared text–audio embedding space. CLAP
(Contrastive Language–Audio Pretraining) trains an audio encoder and a text encoder together
so that a clip and its caption land at nearly the same point — the audio twin of CLIP. AudioLDM
(Liu et al., 2023) leaned on this so heavily it called the CLAP embedding the "language of
audio": at training time it conditions on the audio's CLAP embedding, and at inference
swaps in the text's — the shared space makes them interchangeable.
The contrastive objective is the InfoNCE loss: pull matched text/audio pairs together, push
mismatched ones apart, across a batch of $N$.
with $\mathbf{t}_i,\mathbf{a}_i$ the L2-normalised text and audio embeddings and $\tau$ a temperature.
The map below is a cartoon of the result: four prompts (squares) and their matching audio clips
(circles) sit close in a 2-D joint space. Pick a prompt and press Generate — a sample is
pulled from the prior mean toward the prompt's region. The strength of that pull is
classifier-free guidance.
■ text embedding ● audio embedding ○ generated sample. Pick a prompt and press Generate.
Classifier-free guidance (Ho & Salimans, 2021) is the workhorse that makes prompts bite.
You train one network to predict noise both with the condition and without it (dropping $\mathbf{c}$
some fraction of the time), then at sampling time extrapolate away from the unconditional prediction:
At $w=0$ the sample ignores the prompt (the mean of the prior); at $w=1$ it is the plain conditional;
at $w>1$ it is pushed past the conditional, harder onto the prompt. Slide the guidance up in
the demo and the generated point marches out toward — and then overshoots — the prompt cluster, exactly
as the equation says. Overshoot is the double edge: high $w$ nails the prompt but flattens diversity and
can distort, which is why real systems tune $w$ to a sweet spot around $3$–$7$.
The guidance you hear. Press Generate on each prompt — the demo synthesises a crude
representative sound for that label. In a real system the same guidance vector isn't a toy 2-D arrow but
a high-dimensional cross-attention conditioning that steers every reverse step of the latent diffusion.
5The systems — from spectrogram hack to stereo tracks
In two years the field went from repurposing an image model to purpose-built stereo music generators.
The through-line is constant — diffusion in a latent, conditioned on text — and the differences are in
the autoencoder, the backbone, and the conditioning.
System
Year
What it diffuses
Conditioning & backbone
Riffusion
2022
Mel-spectrogram images in Stable Diffusion's VAE latent
A mel-spectrogram VAE latent, trained on audio only
Conditioned on CLAP embeddings ("language of audio"); U-Net; HiFi-GAN vocoder
AudioLDM 2
2023
Same latent, unified across speech / music / SFX
A "language of audio" (AudioMAE) target predicted by a GPT front-end; more general conditioning
Make-An-Audio · Tango
2023
Spectrogram / mel latents
CLAP or an instruction-tuned LLM (FLAN-T5, in Tango) as the text encoder
MusicLDM
2023
Music-domain latent
CLAP conditioning + beat-synchronous mixup to fight training-data memorisation
Stable Audio
2023–24
A highly-compressed waveform autoencoder latent — full 44.1 kHz stereo
Text + timing embeddings for variable-length tracks; DiT (transformer) backbone
Stable Audio (Evans et al., 2023/24) is the clearest departure. It abandons the
spectrogram entirely for a heavily-compressed autoencoder that maps stereo waveform straight to latent
and back, so there is no separate vocoder and no phase to hallucinate. It replaces the U-Net with a
diffusion transformer (DiT), and — crucially — adds a timing-conditioning
embedding that tells the model the desired duration and offset, letting one model produce
variable-length clips up to minutes long instead of a fixed window. That combination — waveform latent,
DiT, timing control — is what turned text-to-audio from short sound-effects into full musical structures.
Every one of these is the same sentence with different nouns: encode audio to a latent, learn to
denoise that latent, steer the denoising with a text embedding, decode back to sound. The research is
in choosing each noun well.
6Diffusion vs the token-LM route
Part 10 generated audio the other way: quantise it into discrete tokens with a codec, then predict
the next token with a transformer language model. Latent diffusion and token-LMs are the two dominant
paradigms for text-to-audio, and they make opposite bets on almost every axis.
Diffusion (this part)
Continuous latent; non-autoregressive — the whole clip is
refined in parallel over reverse steps.
Strong on texture and timbre: dense, stationary content like pads, rain, or
rich instrumental mixes.
Quality–speed dial is the step count; fast samplers shrink it.
Global coherence comes for free (all positions denoised together); very long-range musical
form is harder.
Token language model (Part 10)
Discrete tokens; autoregressive — one token at a time,
left to right.
Strong on long structure: verse–chorus form, lyrics, sequences that
must stay consistent over time.
Quality–speed dial is sequence length × codebooks; slow to sample.
The lines are blurring. MusicGen (Part 10) is an AR transformer over codec tokens; Stable Audio is a
diffusion transformer over a continuous latent — both use a transformer backbone, differing
only in whether it predicts discrete tokens autoregressively or denoises a continuous latent in
parallel. Hybrids exist too: diffusion over continuous codec features, or LM-planned coarse
structure refined by a diffusion decoder. The primitives from Part 1 are being recombined, not chosen
between.
Rule of thumb. If the task is a texture, a stem, or a short richly-timbred clip, reach
for diffusion. If it needs to hold a form or follow lyrics over tens of seconds, an autoregressive LM
still has the edge — and increasingly the answer is one transformer wearing whichever hat the task needs.
7Latent diffusion for audio, 2022 → 2024
A compressed three-year arc: from bolting text onto an image model to native, minutes-long stereo
generation. Part 12 picks up where this leaves off — real-time control and the frontier.
2022
Latent Diffusion / Stable Diffusion (Rombach et al.) establishes diffusion in a VAE latent for images — the template the whole field copies. Part 8 · 9
Dec 2022
Riffusion fine-tunes Stable Diffusion on spectrogram images with text; seamless looping and prompt interpolation, Griffin–Lim to audio. §2
early 2023
AudioLDM, Make-An-Audio and Tango bring CLAP-/LLM-conditioned latent diffusion to general text-to-audio; MusicLDM targets music. §4 · 5
mid 2023
AudioLDM 2 unifies speech, music and sound effects under a single "language of audio" target. §5
2023–24
Stable Audio & Stable Audio 2 deliver variable-length 44.1 kHz stereo via a waveform-latent DiT with timing conditioning. §5 · Part 12
→Where to go next
You now have the second great text-to-audio paradigm end to end: a VAE latent to work in, a diffusion
model to place a distribution over it, and a CLAP embedding with classifier-free guidance to steer it
from a prompt. The finale asks the questions this all raises — how do we judge the output, and
can we run it live?