Neural Audio Codecs

1The discrete bottleneck — one architecture, two payoffs

Part 8 built the variational autoencoder: an encoder $E$ that maps audio $\mathbf{x}$ to a continuous latent $\mathbf{z}$, and a decoder $G$ that reconstructs it. A neural audio codec keeps that skeleton but forces the latent through a discrete gate — every frame is snapped to one of a finite set of learned vectors. That single change buys two things a continuous VAE cannot give you at once.

2Vector quantization — snapping a vector to its nearest word

The discrete gate is vector quantization, introduced to deep learning by the VQ-VAE (van den Oord et al., 2017). Hold a codebook — a learned table of $K$ vectors $\{\mathbf{e}_k\}_{k=1}^{K}$, each the same dimension as the encoder output. To quantize an encoder vector $\mathbf{z}_e$, find the nearest codebook entry and pass its index forward; the decoder reads back that entry $\mathbf{e}_k$.

The problem: you cannot differentiate an $\arg\min$

Nearest-neighbour lookup is a step function — its gradient is zero almost everywhere and undefined at the boundaries. Back-propagation would have no signal to train the encoder. The VQ-VAE's trick is the straight-through estimator: on the forward pass use the quantized $\mathbf{z}_q$; on the backward pass simply copy the gradient from $\mathbf{z}_q$ straight onto $\mathbf{z}_e$, as if quantization were the identity.

where $\operatorname{sg}[\cdot]$ is the stop-gradient operator (identity forward, zero derivative backward). The forward value is exactly $\mathbf{e}_{k^\star}$; the backward value pretends the codebook was transparent. It is biased, but it works.

The losses that move the codebook

Straight-through trains the encoder and decoder, but the codebook vectors themselves get no gradient from reconstruction. Two extra terms fix that. The codebook loss pulls each chosen entry toward the encoder output; the commitment loss pushes the encoder output toward its chosen entry so it does not wander between codes.

The stop-gradients split the responsibility: the codebook term only moves $\mathbf{e}$, the commitment term only moves $\mathbf{z}_e$, and $\beta\approx0.25$ balances how tightly the encoder is held to the table. In practice the codebook term is usually replaced by an exponential moving average (EMA) update — treat each entry as the running mean of the encoder vectors that selected it, which is more stable than an SGD step.

3Residual VQ — a stack of codebooks refining the leftover

A single codebook of $K$ entries carries only $\log_2 K$ bits per frame. To reach the fidelity a codec needs you would want an enormous $K$ — but a million-entry codebook is impossible to train (collapse) and slow to search. Residual vector quantization sidesteps this: quantize coarsely with codebook 1, take the residual — what the first code missed — and quantize that with codebook 2, and so on down a stack of $N_q$ codebooks.

Each stage chips away at the error the previous stages left behind, so the residual magnitude $\lVert\mathbf{r}_i\rVert$ shrinks geometrically and the reconstruction $\hat{\mathbf{z}}$ marches toward the true encoder vector. With $N_q$ codebooks of $K$ entries each, the effective vocabulary is $K^{N_q}$ — the reach of a giant codebook — at the training cost and search cost of $N_q$ small ones.

4SoundStream & EnCodec — the full pipeline

SoundStream (Zeghidour et al., 2021) and EnCodec (Défossez et al., 2022) wrap RVQ in a fully convolutional, streaming encoder/decoder and train the whole thing end-to-end with a mixture of reconstruction and adversarial objectives. The result is a real-time neural codec that runs on a laptop CPU. Hover the blocks below to walk the signal from waveform to tokens and back.

The four pieces

• Convolutional encoder. A cascade of strided 1-D convolutions with residual blocks. Each strided layer downsamples in time (strides like $2,4,5,8$), so ~$24\,000$ samples per second collapse to ~$75$ latent frames per second — the total stride is the product, e.g. $320\times$.
• RVQ bottleneck. Each latent frame is quantized by the stack of Section 3 into $N_q$ integer codes. These codes are the transmitted bitstream and the tokens Part 10 models.
• Convolutional decoder. A mirror of the encoder using transposed convolutions that upsample back to the waveform rate.
• Streaming. All convolutions are causal and padding is cached, so the codec runs frame-by-frame with a fixed, small latency — essential for live calls.

The training objective — reconstruct, and fool a critic

An $\ell_2$ waveform loss alone gives muffled audio (it averages away phase detail, exactly the Part 1 lesson). Neural codecs instead combine three ingredients, and it is the combination that yields fidelity.

A feature-matching loss (match the discriminator's internal activations for real vs fake) usually joins them to stabilise the GAN. EnCodec adds a small entropy-coding stage — an arithmetic coder over the code indices — that squeezes a further ~25 % off the bitrate for free, since the codes are not uniformly used.

5Bitrate & quality — hear the codebooks add up

The bitrate of an RVQ codec is a single tidy product: the number of frames per second, times the number of codebooks, times the bits per codebook. Nothing else. That is what makes RVQ so controllable — every knob in the formula is an integer you can dial.

With EnCodec's $f_r=75$ frames/s, $K=1024$ (10 bits) and $N_q=8$ codebooks you get $75\times8\times10 = 6000$ bits/s $= 6$ kbps; drop to $N_q=2$ and you are at $1.5$ kbps — near-telephone quality from 1500 bits a second. Because the stack is ordered, changing $N_q$ needs no retraining: you just transmit fewer codes.

The demo below runs a residual quantizer over an actual synthesised clip and lets you hear the trade. Add stages and the reconstruction climbs the SNR curve toward the original; strip them away and the quantization error becomes audible hiss and grain. Press original then quantized to A/B them at the current setting.

6DAC — closing the codebook-usage gap

The Descript Audio Codec (Kumar et al., 2023) took the SoundStream/EnCodec template and fixed its weakest link: codebook collapse. EnCodec's codebooks were often only partly used, wasting bitrate. DAC's changes push utilisation close to 100 % and let a single model compress 44.1 kHz audio — speech, music, environmental sound — at ~8 kbps with quality that beats EnCodec at 3× the rate.

The three key improvements

• Factorized, L2-normalized codes. The encoder output is projected to a low-dimensional space (e.g. 8-D) for the lookup, and both the code and the query are L2-normalized so nearest-neighbour becomes a cosine search on a hypersphere. Low dimension + normalization dramatically raises how many codes get used.
• Snake activation. Borrowed from BigVGAN, the periodic activation $x + \tfrac{1}{\alpha}\sin^2(\alpha x)$ gives the decoder a built-in inductive bias toward periodic (i.e. pitched, harmonic) signals — a better fit for audio than ReLU.
• Quantizer dropout, fixed. DAC noted that naïve quantizer dropout slightly hurts full-bitrate quality and applies it more carefully, keeping the bitrate-scalability benefit without the fidelity tax.

The lesson is general: the bottleneck was never the RVQ idea but the statistics of the codebook. Force the codes onto a sphere in a low-dimensional space and the $\arg\min$ spreads its mass, so the bits you paid for are actually spent. DAC is now the default codec behind many 2024–25 audio language models.

A neural codec is judged by a strange double standard: it must be a great compressor (low bitrate, high fidelity) and a great tokenizer (codes a language model can predict). DAC's high codebook usage helps both — every token carries information, so the downstream transformer never wastes capacity on dead symbols.

7Why discretization unlocks language models

Everything in this part has been building one object: a sequence of integers that stands in for a waveform. Once audio is a sequence of tokens over a fixed vocabulary, the entire machinery of language modelling applies unchanged — and that is the subject of Part 10.

A transformer models $p(\text{token}_t \mid \text{token}_{

Contrast with classical codecs

Classical codecs — MP3, AAC, Opus — also compress audio hard, but by hand-designed signal processing: an MDCT into the frequency domain, a psychoacoustic model that discards what the ear cannot hear, then entropy coding. They are superb compressors and terrible tokenizers — their bitstream is a tangle of scale factors and Huffman codes with no learnable, model-friendly structure. See the DSP & Music collection's MDCT guide for how the classical side works.

Neither replaces the other: Opus still wins on CPU cost and guaranteed low latency for a phone call. But only the neural codec doubles as a generative substrate, and that is why the field adopted it wholesale for synthesis.

8From VQ-VAE to DAC in six years

The discrete-bottleneck idea arrived for images and speech, then was engineered into a real-time audio compressor in a remarkably tight arc.

→Where to go next

You now have the tokenizer of the neural-audio era: an autoencoder whose discrete bottleneck compresses sound to kilobits and, in the same breath, emits a sequence of integers. The obvious next question is what to do with that sequence — and the answer is to model it exactly as a language.