Larger $w$ makes the output cling harder to the prompt at some cost to diversity and
naturalness — the same quality/adherence dial you feel in any text-to-image system, now over
audio tokens.
derivation · why temperature is a precision knob on the categorical
Write the softmax with an inverse-temperature $\beta = 1/\tau$. Then
$p_i \propto \exp(\beta u_i)$, which is exactly a Boltzmann distribution at "temperature"
$\tau$. Two limits fall straight out. As $\beta\to\infty$ ($\tau\to 0$) the ratio of any
non-maximal probability to the maximal one,
$$\frac{p_i}{p_{\max}} = \exp\!\big(\beta\,(u_i - u_{\max})\big) \longrightarrow 0,$$
so all mass concentrates on $\arg\max_i u_i$ — greedy decoding. As $\beta\to 0$
($\tau\to\infty$) every $\exp(\beta u_i)\to 1$ and $p_i \to 1/V$ — uniform. The demo works
on counts rather than logits, which is the same statement with $u_i = \log(\text{count}_i)$:
raising a probability to the power $1/\tau$ and renormalising sharpens ($\tau<1$) or
flattens ($\tau>1$) it identically. That equivalence is why a bigram over seed melodies is
a faithful, if tiny, stand-in for a transformer over codec tokens: only the source of the
logits differs.
3AudioLM — the semantic + acoustic hierarchy
A single codec-token LM has a problem: EnCodec-style tokens are optimised for
reconstruction fidelity, not linguistic structure, so a raw LM over them drifts —
it keeps the timbre but loses the tune over a few seconds. AudioLM (Borsos et al., 2022)
solved this by predicting audio in two kinds of token, in coarse-to-fine order.
Semantic tokens — the plan
Discretised (k-means) activations of a self-supervised model, w2v-BERT.
They are sampled slowly (~25 Hz) and capture long-term structure: phonetics,
melody, rhythm, prosody. They sound terrible if decoded alone — they carry the plan, not the
audio.
Acoustic tokens — the render
The RVQ codes of a SoundStream codec, several per frame across quantiser
levels. They carry fidelity: speaker identity, timbre, recording conditions, fine
spectral detail — everything needed to reconstruct a convincing waveform.
Generation runs as a cascade of three transformer stages. First a semantic stage
models $p(s_{1:T})$ — the long-horizon plan. Then a coarse acoustic stage predicts the
first few RVQ levels conditioned on the full semantic sequence, and finally a fine
acoustic stage fills the remaining RVQ levels. Coarse-to-fine, plan-then-render:
$$p(s,a) = \underbrace{p(s_{1:T})}_{\text{semantic}} \; \underbrace{p\!\left(a^{\le q_c}_{1:T}\mid s_{1:T}\right)}_{\text{coarse acoustic}} \; \underbrace{p\!\left(a^{> q_c}_{1:T}\mid a^{\le q_c}_{1:T}\right)}_{\text{fine acoustic}}$$
Because the slow semantic stream commits to the structure first, the acoustic stages only
have to render it — and the result stays coherent for far longer than a flat
codec-token LM. AudioLM could continue a piano piece or a spoken sentence for many seconds
while keeping the melody, speaker and syntax intact, all trained with no transcripts or labels.
Explore the two-stage stack below.
Read the order. The top row fills completely before any acoustic token
appears — the model decides what happens across the whole clip before deciding
how it sounds. Within the acoustic block, level 1 (coarsest, most perceptually
significant) is committed before the fine levels refine it. That is the RVQ residual ordering
of Part 9, now used as a generation schedule.
4MusicLM — text conditioning by borrowing an embedding
AudioLM is unconditional: it continues audio, it does not follow a prompt. MusicLM (Agostinelli
et al., 2023) turned it into a text-to-music system with one addition — a joint
text–audio embedding space that lets a caption stand in for the missing audio context.
That embedding is MuLan, a contrastive model (a CLIP for music) trained on
millions of tracks paired with free-form text so that a caption and a matching clip land near
each other. At training time MusicLM conditions on MuLan's audio embedding of the
target; at inference it swaps in the MuLan text embedding of the prompt — the shared
space makes the substitution nearly seamless, and no captioned music is needed to train the
generative stages at all.
$$M_{\text{text}}(\text{"a calm violin melody"}) \;\approx\; M_{\text{audio}}(\text{matching clip}) \;\Rightarrow\; p\big(\text{semantic},\text{acoustic}\mid M\big)$$
The generative core is AudioLM's hierarchy, extended to a three-level conditioning cascade:
the MuLan token conditions a semantic stage, which conditions a coarse acoustic
stage, which conditions a fine acoustic stage. MusicLM also shipped MusicCaps, a
benchmark of expert-written captions, and demonstrated melody conditioning — hum a tune, describe
a style, get the tune rendered in that style.
| System | Conditioning | Token hierarchy | Signature idea |
|---|
| AudioLM | none (continuation) | semantic → coarse → fine | self-supervised semantic tokens for long-term coherence |
| MusicLM | text / melody via MuLan | MuLan → semantic → coarse → fine | contrastive text–audio embedding as the prompt bridge |
| MusicGen | text (T5) + melody | single stage, parallel codebooks | codebook-interleaving pattern + classifier-free guidance |
Notice the compression happening across these three rows. AudioLM needed three transformers and a
hand-built semantic/acoustic split. MusicLM added a fourth conditioning stage. MusicGen — next —
threw the hierarchy out and did it with one transformer. The field's arc in this part is
the steady collapse of a cascade into a single model.
5MusicGen — one transformer over parallel codebooks
EnCodec emits $K$ codebook tokens per frame (the RVQ levels). A naive LM must
somehow model these parallel streams. MusicGen (Copet et al., 2023) does it with a
single autoregressive transformer and a codebook-interleaving pattern
that decides the order in which the $K\times T$ tokens are unrolled into one sequence. The
choice of pattern is the whole efficiency story.
Three ways to unroll a $K\times T$ grid
- Flatten — emit all $K$ codebooks of frame 1, then all $K$ of frame 2, … A perfect autoregressive factorisation, but the sequence is $K{\times}T$ long: $K$ times slower to generate.
- Parallel — predict all $K$ codebooks of a frame in one step. Only $T$ steps, but it wrongly assumes the $K$ codes of a frame are conditionally independent — and they are not, because they are RVQ residuals of one another.
- Delay — MusicGen's answer. Offset codebook $k$ by $k$ steps so that, at any decoding step, the coarse codes a frame depends on are already known. It costs only $T + K - 1$ steps yet respects the residual dependency.
Formally, if $z_{k,t}$ is the token at codebook $k$, frame $t$, the pattern is just a schedule
$s(k,t)$ giving the decoding step at which that token is produced:
$$s_{\text{flatten}}(k,t) = tK + k,\qquad s_{\text{parallel}}(k,t) = t,\qquad s_{\text{delay}}(k,t) = t + k$$
The delay pattern turns a $K{\times}T$ problem that "should" take $KT$ steps into one that takes
$T{+}K{-}1$ — for $T=1500,\,K=4$ that is ~1503 steps instead of 6000, a near-$4\times$ speedup, with
almost no quality loss because the diagonal schedule still lets each fine code see the coarse codes
of its frame. Step through all three below.
Rows = RVQ codebooks, columns = frames. Bright cells are being predicted at this step; filled cells are already decoded. Watch the total step count change with the pattern.
MusicGen conditions on text through a frozen T5 encoder (cross-attention or
prefix), supports melody conditioning via a chromagram, and uses the classifier-free guidance of
§2 to trade prompt-adherence against diversity. One model, one stage, competitive with the
cascades — which is why it became the default open text-to-music baseline.
derivation · the delay pattern's step count and why it is nearly lossless
Under the delay schedule $s(k,t) = t + k$ with $k\in\{0,\dots,K-1\}$ and
$t\in\{0,\dots,T-1\}$, the largest step index is $s_{\max} = (T-1) + (K-1)$, so the number of
decoding steps is
$$N_{\text{delay}} = T + K - 1,$$
versus $N_{\text{flatten}} = TK$ and $N_{\text{parallel}} = T$. The speedup over flatten is
$TK/(T{+}K{-}1)\to K$ as $T\gg K$. The key property: at step $n$ the tokens emitted are
$\{z_{k,t}: t+k=n\}$, and each such token's frame-$t$ coarse neighbours $z_{k'
6Jukebox and VALL-E — the extremes of the same idea
Two systems bracket the token-LM family: one that predated the codec era and paid dearly for it,
and one that carried the recipe into speech and made voice cloning trivial.
Jukebox (Dhariwal et al., 2020) — the ambitious ancestor
Two years before AudioLM, OpenAI's Jukebox generated whole songs with singing directly
in raw audio. It built its own tokens with a multi-scale VQ-VAE — three levels
compressing raw audio by 8×, 32× and 128× — then trained sparse transformers as
autoregressive priors over each level, upsampling coarse tokens to fine. It conditioned on artist,
genre and even time-aligned lyrics, and it worked: recognisable songs in the style of
named artists, with words. The cost was brutal — coherence wandered over a song's length, and
rendering a minute of audio took hours, because its codec tokens were far less compressed and less
structured than the semantic/acoustic split that came later. Jukebox is the proof-of-concept whose
pain motivated everything in §§3–5.
VALL-E (Wang et al., 2023) — the speech cousin
VALL-E applied the MusicGen-shaped recipe to text-to-speech: an EnCodec token LM, with a text
prompt and a 3-second enrolment clip of a target speaker as acoustic context. Because the
acoustic tokens carry timbre and the model does in-context learning, it reproduces an unseen
speaker's voice, prosody and even acoustic environment from those three seconds — zero-shot
voice cloning. It uses a hybrid schedule: an autoregressive model for the coarse codebook and a
non-autoregressive model for the rest, a close relative of MusicGen's interleaving. VALL-E is the
clearest evidence that "audio is a language" is a claim about all audio, not just music.
The dual-use edge. Zero-shot cloning from three seconds of audio is also a
deepfake engine. Every serious release in this line now ships with a watermarking or detection
plan; the capability and its misuse are the same model read two ways.
7From Jukebox to MusicGen in three years
The token-LM approach to music went from a compute-devouring curiosity to an open, single-model
baseline in the span of three years — the moment good neural codecs (Part 9) arrived.
2020
Jukebox generates raw-audio songs with lyrics via a multi-scale VQ-VAE and sparse-transformer priors — proof that a token LM can sing, at ruinous cost. Part 10
2021–22
SoundStream & EnCodec deliver compact, structured RVQ tokens; AudioLM introduces the semantic + acoustic hierarchy for long-term-coherent, label-free continuation. Part 9 · 10
Jan 2023
MusicLM adds MuLan text–audio embeddings for text-to-music; VALL-E ports the recipe to zero-shot text-to-speech voice cloning. Part 10
Jun 2023
MusicGen collapses the cascade into a single transformer over EnCodec's parallel codebooks using the delay interleaving pattern, T5 text and melody conditioning, classifier-free guidance — open-sourced. Part 10
2023→
Attention turns to the diffusion alternative for text-to-audio — continuous latents instead of discrete tokens, non-autoregressive, faster at length. Part 11
→Where to go next
You have seen the second great paradigm of neural audio: once a codec discretises sound, a
transformer models it exactly like language — factorised left to right, trained by cross-entropy,
sampled with temperature, steered by guidance. AudioLM's hierarchy bought coherence, MusicLM bought
prompts, MusicGen bought a single efficient model. But discrete tokens are not the only game. Part 11
takes the other road entirely: diffusion in a continuous latent, generating audio in
parallel rather than one token at a time.
References
- Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., Sutskever, I. (2020). Jukebox: A Generative Model for Music. arXiv:2005.00341.
- Borsos, Z. et al. (2022). AudioLM: A Language Modeling Approach to Audio Generation. arXiv:2209.03143.
- Agostinelli, A. et al. (2023). MusicLM: Generating Music From Text. arXiv:2301.11325.
- Copet, J. et al. (2023). Simple and Controllable Music Generation (MusicGen). arXiv:2306.05284.
- Wang, C. et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E). arXiv:2301.02111.
- Défossez, A. et al. (2022). High Fidelity Neural Audio Compression (EnCodec). arXiv:2210.13438.
- Ho, J. & Salimans, T. (2022). Classifier-Free Diffusion Guidance. arXiv:2207.12598.
- Holtzman, A. et al. (2020). The Curious Case of Neural Text Degeneration. ICLR. (top-p / nucleus sampling.)