Differentiable DSP

1The idea — put the synthesiser inside the network

Every model in this series so far spends most of its capacity re-learning something physics already knows: that a pitched sound is a sum of harmonics, that breath is filtered noise, that a room smears the result with a decaying tail. Differentiable Digital Signal Processing (DDSP) takes the opposite bet — it hands those facts to the network as fixed, differentiable DSP modules, and asks the network only to drive them.

A DDSP model is an encoder–decoder. The decoder is not a stack of transposed convolutions painting a spectrogram; it is a literal synthesiser — an additive harmonic oscillator bank plus a time-varying filtered-noise generator (and, optionally, a convolutional reverb). Because every one of those operations is written in terms of differentiable primitives (sums, sines, multiplies, FFTs), the whole synthesiser sits inside the autograd graph, and the training loss can back-propagate through the act of making sound into the small network that predicts the controls.

That single architectural choice buys three things at once. The inductive bias is enormous, so the model is small (hundreds of thousands of parameters, not hundreds of millions) and learns from minutes of audio, not thousands of hours. The controls are interpretable and disentangled: pitch is a number, loudness is a number, timbre is a spectral shape you can read off and edit. And because the oscillators are exact, there is no vocoder to hallucinate phase — the output is as clean as the harmonic model allows.

DDSP is best understood not as a new generative family but as a differentiable re-implementation of fifty years of synthesis. The novelty is not the oscillator; it is that you can now train it with a gradient. after Engel, Hantrakul, Roberts & Gu, ICLR 2020

2The decoder — a harmonic oscillator bank plus filtered noise

The heart of DDSP is the harmonic-plus-noise synthesiser. A deterministic, pitched part is a bank of sinusoids locked to integer multiples of a fundamental $f_0$; a stochastic, un-pitched part is white noise pushed through a time-varying filter. Sum them and you can build almost any sustained instrument or voiced sound. The demo below is that decoder — every knob is a control the network would otherwise predict.

The harmonic oscillator bank

The pitched part is a direct additive synthesiser. Each harmonic $k$ has a slowly varying amplitude $A_k(t)\ge 0$ and an instantaneous phase obtained by integrating its frequency — $k$ times the fundamental — so that all harmonics stay phase-locked to $f_0$ even as it glides:

Two constraints make it trainable and clean. Amplitudes must be non-negative, so the network's raw outputs $h_k$ pass through a modified sigmoid that is smooth, bounded below by zero, and saturates gently:

And any harmonic whose frequency $k f_0$ crosses the Nyquist limit is forced to zero — a one-line, differentiable anti-aliasing mask that a raw waveform model has to learn the hard way.

The filtered-noise (subtractive) branch

Bowed hiss, breath, the chiff of a flute, fricatives — none of that is harmonic. DDSP adds a subtractive branch: white noise $u(t)$ shaped, frame by frame, by a linear filter whose magnitude response $H_\ell$ the network predicts directly. In practice this is a frequency-sampling FIR: take the predicted magnitudes, treat them as the DFT magnitude of an impulse response, window it, and convolve — all differentiable.

Try this: push brightness up and hear the timbre open as energy climbs into the upper harmonics; then raise noise amt to blow breath across the tone without changing its pitch. Slide harm↔noise fully right and the pitched comb dissolves into pure shaped noise — the same synthesiser now voicing a fricative.

3The training objective — multi-scale spectral loss

Here is the subtlety that makes DDSP work at all. You cannot train an oscillator bank by comparing its waveform to the target sample-by-sample: two sinusoids of identical frequency and amplitude but different phase are perceptually identical yet have a huge $L_2$ waveform error. The fix is to compare in the magnitude spectral domain, which throws phase away — and to do it at several time–frequency resolutions at once.

Each $S_i$ is the STFT taken with FFT size $i$. A small window has sharp time resolution but coarse frequency resolution; a large window is the reverse. Summing $L_1$ magnitude distances across sizes means the loss cannot be fooled by getting one resolution right — a transient must line up in the short window and a steady pitch must line up in the long one. The optional log term adds sensitivity to quiet detail.

4The controls — loudness, pitch, and the encoder

If the decoder is a synthesiser, the controls are the performance. DDSP's original autoencoder drives the synth from just two interpretable signals plus a learned residual: a fundamental frequency $f_0$, a loudness $\ell$, and (optionally) a small latent $z$ for timbre that the pitch and loudness do not explain.

The encoder is a small recurrent or convolutional network. Feed it audio; it emits, per frame, the harmonic amplitudes $\{A_k\}$, the noise magnitudes $\{H_\ell\}$, and (with the latent variant) a compact $z$. Because those are the only outputs, and they feed a fixed synthesiser, the network has nowhere to hide fidelity — it must genuinely explain the sound in synthesiser terms. That is the discipline behind DDSP's data efficiency.

5Timbre transfer — re-decode the same performance

Because pitch and loudness are cleanly separated from timbre, DDSP does something that used to require a full re-recording: it plays one performance — the $f_0$ and loudness envelopes from a singer or a saxophone — through a different learned spectral shape. The melody is preserved; the instrument changes. And because the timbre is just a harmonic distribution plus a noise ratio, you can interpolate smoothly between two instruments.

The two endpoints are not recordings — they are learned spectral envelopes, one per instrument, exactly the $\{A_k\}$ a trained DDSP encoder would emit. The morph is a straight linear interpolation of those envelopes and of the harmonic-to-noise ratio. That a convex blend of two timbres is itself a plausible timbre is a direct consequence of working in the interpretable synthesiser space rather than in raw samples or entangled latents.

This is the DDSP promise in one gesture: the model never memorised a violin waveform. It learned how a violin distributes a given pitch and loudness across a spectrum — and a learned function like that can be blended, extrapolated, and re-driven at will.

6Lineage — the neural heir to source-filter and SMS

None of DDSP's components are new; their differentiability is. The harmonic bank is the additive synthesis of the 1970s; the harmonic-plus-noise split is Serra and Smith's Spectral Modeling Synthesis (SMS, 1990); the excitation-times-filter factoring is the source-filter model that underpins every vocoder and the formant synthesis of Part 5. DDSP re-expresses all of them in an autograd framework so a network can learn their controls end-to-end.

The difference that matters: SMS analysed a sound with hand-built peak-tracking and residual estimation. DDSP learns the analysis by pushing a spectral loss back through the very same synthesis equations. Fifty years of DSP become the network's architecture rather than its competition — the throughline of the DDSP · Differentiable DSP repository and the wider DSP & Music collection.

7DDSP in context

A short arc: from spectral analysis, through neural pitch and vocoders, to the moment the synthesiser itself became differentiable — and what it seeded afterwards.

→Where to go next

DDSP reached expressive, controllable timbre by keeping the physics and learning the controls. The next part takes the opposite route to the same goal — throwing the DSP away and letting an adversary learn instrument timbre from noise. Part 5, just behind us, built voices from formant and glottal models; the contrast frames why DDSP's inductive bias is such a bargain.