A GAN can paint a whole photograph in one shot — so why not a whole musical note?
The obstacle was phase: adversarial audio buzzed and smeared until GANSynth found the
trick — predict instantaneous frequency, not raw phase, grow the network
resolution-by-resolution, and condition on pitch so timbre lives in a smooth,
interpolable latent space. Three live demos let you morph timbres, see why phase is
unlearnable, and walk the generator.
Part 2 generated a waveform one sample at a time; Part 6 handed a differentiable
oscillator the physics for free. This part asks a blunter question: can an adversarial
network — the same machinery that generates a face in a single forward pass — generate
a whole four-second instrument note in one shot, with a latent space you can
interpolate? The answer is GANSynth, and getting there meant confronting the
one thing images never have to worry about: phase.
NSynth — a dataset that made the question answerable
In 2017 the Magenta team released NSynth: 305,979 musical notes from
1,006 commercial instruments, each a four-second monophonic sample at 16 kHz,
annotated with pitch, velocity, instrument family and acoustic qualities. Crucially it
is aligned — every note starts at $t=0$, one pitch, one instrument — so a model
can learn the manifold of timbre with pitch held as an explicit, separable variable.
The accompanying WaveNet autoencoder encoded each note into a temporal
embedding and decoded it with an autoregressive WaveNet: high quality, but hundreds of
thousands of sequential samples per note to generate, and a latent that was temporal
rather than a single global timbre knob.
The prize. If a single latent vector $z$ could stand for "this
instrument's timbre" and pitch could be dialled in separately, then the straight line
between two $z$'s would be a smooth morph from, say, a flute to a mallet — a control no
sample library can offer. That is exactly what GANSynth delivers, and the last demo on
this page lets you drive it.
Why a GAN, and why it is hard
A GAN pits a generator $G$ that maps a noise vector $z$ to a signal against a
discriminator $D$ trained to tell real from fake. Sampling is a single forward
pass — orders of magnitude faster than autoregression — and the latent space tends to
be smooth and interpolable. But audio is not an image: it is a quasi-periodic signal
whose perceptual identity lives partly in phase relationships across time, and
a naïve GAN has no reason to get those coherent. The next three sections are the story
of how that was fixed.
Before GANSynth, Donahue, McAuley & Puckette (2019) asked the field's first clean
question: where should an audio GAN operate — on the raw waveform, or on a
spectrogram? They built one of each. Both worked, both had a characteristic flaw, and
the flaws are exactly the two forces GANSynth had to reconcile.
WaveGAN — adversarial in the waveform
WaveGAN takes the DCGAN image architecture and flattens it to one
dimension: 2-D $5\times5$ convolutions become length-25 1-D convolutions, and the
generator uses transposed convolutions to upsample noise to 16,384 samples
(~1 s). It owns phase implicitly — the waveform is phase — but the
transposed convolutions leave a checkerboard fingerprint: a periodic
artifact at the stride frequency, audible as a pitched buzz.
SpecGAN — adversarial in the spectrogram
SpecGAN instead generates a log-magnitude spectrogram image with a
standard 2-D DCGAN, then reconstructs a waveform with Griffin–Lim.
The magnitude image is easy for a GAN — but magnitude alone throws phase away, and
Griffin–Lim must guess it by iterated STFT projection. The guess is
never quite right, so SpecGAN audio has a smeared, watery quality.
The checkerboard artifact
A transposed convolution with kernel size $k$ and stride $s$ where $k$ is not divisible
by $s$ deposits overlapping, unevenly-summed contributions — the same effect that
dapples upsampled images with a grid. WaveGAN's fix was phase shuffle:
randomly shifting each discriminator activation by a few samples so $D$ cannot lock onto
the artifact's fixed phase and force $G$ to hide it. It is a patch, not a cure.
The takeaway that set up GANSynth. Waveform GANs get phase "for free"
but fight upsampling artifacts and struggle with pitch periodicity; spectrogram GANs get
a friendly image target but discard phase and pay for it at inversion. Neither gives you
coherent phase and an easy 2-D target. GANSynth's contribution is a
representation that does both.
why Griffin–Lim cannot rescue a magnitude-only GAN
Griffin–Lim iterates: given a target magnitude $|S|$, start from random phase,
take the inverse STFT, re-take the STFT, keep the new phase but reset the magnitude to
$|S|$, and repeat. It converges to a signal whose STFT magnitude is close to $|S|$ — but
the fixed point is not unique, and for a magnitude that a GAN hallucinated
(rather than one that came from a real, consistent signal) there may be no
waveform with exactly that magnitude. The algorithm then lands on a compromise:
phase-incoherent between frames, which the ear hears as the characteristic smear.
GANSynth sidesteps the whole problem by predicting phase information directly, so no
guessing is ever required.
3Instantaneous frequency — the key insight
Here is the crux of the whole part. The reason you cannot just add a "phase channel" to
SpecGAN is that raw phase, bin by bin, is essentially random — there is no
smooth image for the generator to learn. But the time-derivative of
unwrapped phase — the instantaneous frequency — is smooth and near-constant along a
harmonic. Predict that, and phase becomes a learnable target.
For a single sinusoid at frequency $f$, the phase of its STFT bin advances by a fixed
amount every hop: it wraps around the circle over and over, so a plot of the raw phase
looks like noise. The rate of advance, however, is constant. Instantaneous frequency is
defined as the time derivative of the (unwrapped) phase, which on a frame grid is simply
the phase difference between successive frames:
where $\phi(t,k)$ is the phase of frequency bin $k$ in frame $t$. Along a stable partial
the per-frame phase advance is the same every frame, so IF is a flat, coherent
band — a target with structure a convolutional generator can produce. Toggle the
three views below: the magnitude has clean harmonic stripes, the raw phase is visual
static, and the IF image recovers coherent horizontal bands at each harmonic.
Same harmonic tone, three representations. Raw phase is near-random static; its per-frame difference — the instantaneous frequency — is coherent, and that is what GANSynth predicts.
Try this: switch to Raw phase and drag the hop slider — the
static reshuffles but never organises. Now switch to Instantaneous freq: the
harmonic bands snap into place and stay put as you change the hop, because the phase
advance per hop is a well-defined, smooth quantity. That stability is precisely
why a GAN can learn it.
GANSynth generates a two-channel "image": one channel is the log-magnitude, the other is
the instantaneous frequency (Engel et al. test both linear-frequency and
mel-frequency variants; the mel IF version scored best). At synthesis time the
IF is cumulatively summed back into phase, combined with the magnitude, and inverted by a
single STFT — no Griffin–Lim, no phase guessing.
derivation · why unwrapping makes phase learnable but raw phase does not
Take a pure tone $x(n)=\cos(2\pi f n / f_s)$. In STFT bin $k$ its phase at frame $t$
(hop $H$) is $\phi(t,k) = 2\pi f H t / f_s + \phi_0 \pmod{2\pi}$. As $t$ increases this
wraps through $2\pi$ repeatedly, so sampled on $(-\pi,\pi]$ it is a fast sawtooth that,
once the bin is not exactly aligned to a frame-periodic frequency, looks pseudo-random
to a convolution. The difference $\phi(t,k)-\phi(t-1,k) = 2\pi f H / f_s$ is a
constant independent of $t$ — a flat band. That is the instantaneous frequency (up to a
$2\pi$ wrap, which is removed by taking the principal value). So the same information,
expressed as a derivative rather than an absolute, changes from unlearnable static into
a smooth low-entropy image. This is the entire representational trick of GANSynth, and
it is why the demo's IF panel has structure the phase panel lacks.
With a learnable target in hand, GANSynth borrows two ideas from the image world and one
of its own. From Karras et al.'s Progressive GAN it takes
resolution growing and the WGAN-GP objective; its own contribution is to
condition on pitch with a one-hot label so that timbre — carried by a
single global latent — is disentangled from the note being played.
Hover the blocks in the diagram to inspect each stage, and press Grow to watch
the generator's output resolution climb the way it does during training.
hover a block for detail
z (timbre) is concatenated with a one-hot pitch label and pushed through a progressively grown generator to a log-magnitude + IF image, then inverse-STFT to audio in one pass.
Pitch conditioning disentangles timbre from note
The latent $z$ is a single vector held constant across the whole note, so it
encodes global timbre rather than any moment. Alongside it, the generator receives a
one-hot pitch label; the discriminator is likewise conditioned and also
tasked with classifying pitch. Because pitch is supplied explicitly, $z$ is freed from
having to encode it — the network learns a timbre manifold that is
pitch-invariant. You can hold $z$ fixed and sweep pitch (same instrument up a
scale), or hold pitch fixed and move $z$ (morph instruments at constant pitch). The last
demo does exactly the latter.
Progressive growing stabilises training
Training starts at a tiny resolution — a coarse $4\times8$ magnitude/IF image — where the
adversarial game is easy, then fades in successively higher-resolution layers,
doubling the frequency and time axes each stage up to the full spectrogram. Low-frequency
structure (the fundamental, gross envelope) is settled before the network is ever asked
to render fine harmonics, which tames the instability that plagues high-resolution GANs.
GANSynth trains with the Wasserstein GAN objective plus a gradient penalty (WGAN-GP): the
critic estimates an earth-mover distance rather than a classifier probability, and the
penalty enforces the 1-Lipschitz constraint on $D$ by pushing the gradient norm toward 1
at points $\hat x$ interpolated between real and fake. This is what makes progressive
growing converge reliably.
Result. On NSynth, GANSynth generated notes far faster than the WaveNet
autoencoder (a single forward pass versus tens of thousands of sequential steps) and, in
listening tests, matched or beat it on quality — while offering the smooth, interpolable
latent the autoregressive model never had.
5The timbre manifold — hear the latent space
This is the pay-off. Because pitch is factored out and $z$ is a single global vector,
the space of timbres is smooth: a straight line between two instruments passes through
plausible in-between instruments that were never recorded. Below is a two-dimensional
slice of that manifold. Drag the latent point and play — the note you hear is an additive
resynthesis whose harmonic envelope is the interpolation of four instrument presets.
Corners are Brass · String · Reed · Mallet. The bars show the morphed harmonic spectrum at the current latent; the square is the 2-D timbre slice. Same pitch everywhere — only timbre moves.
The four corners are chosen to be perceptually distinct: brass (a full,
bright harmonic stack), string (rich but rolled-off), reed
(odd-harmonic-dominant, clarinet-like), and mallet (sparse partials with a
fast percussive decay). Bilinear interpolation of their harmonic-amplitude vectors is a toy
stand-in for GANSynth's learned manifold — but it captures the essential phenomenon: the
between is meaningful, and pitch stays put while timbre travels.
The honesty note. This demo interpolates hand-designed spectra, not a
trained network's latent. GANSynth's real manifold is learned and far richer — but its
defining property, a smooth walk from one instrument to another at fixed pitch, is exactly
what you are steering here. Press Sweep morph to hear a continuous glide across
the diagonal.
why interpolation is a property of the model, not the audio
Averaging two waveforms does not average their timbres — it produces
destructive interference and a comb-filtered mess, because phase relationships collide.
Averaging two spectra is better but still ignores how partials should track. The
value of a learned latent is that the model has organised timbre into a space where the
arithmetic mean of two codes decodes to a coherent sound. GANSynth earns that space by
training a generator whose output is always a valid note; the straight line between two
$z$'s therefore stays on the manifold of valid notes. Our additive toy fakes it by
interpolating in a hand-chosen harmonic basis where linear blends happen to sound
reasonable — a reminder that the representation is what makes interpolation musical.
6Why GANs here — the bargain and its bill
GANSynth is the high-water mark of adversarial instrument synthesis, and reading its
trade-offs is the cleanest way to understand why the field's centre of gravity later
shifted to diffusion (Part 8).
Property
GANSynth (adversarial)
WaveNet-AE (autoregressive)
Generation speed
One forward pass — thousands of notes/sec on a GPU
Sequential — tens of thousands of steps per note
Latent space
Smooth, global, interpolable; morph & resynth
Temporal embedding; no single timbre knob
Phase / fidelity
Coherent via predicted IF; strong on pitched, harmonic sound
Training instability: the minimax game has no single loss that both
players descend, so training can oscillate or diverge; progressive growing and the
Lipschitz-constrained WGAN-GP critic are precisely the scaffolding that keeps GANSynth on
its feet. Mode collapse: because the generator is rewarded only for
fooling $D$, not for covering the data, it can retreat to a handful of safe timbres and
ignore rare ones — a failure a likelihood model structurally cannot have.
The GAN bargain is speed and a beautiful latent space, paid for with training fragility
and no likelihood. Diffusion models — Part 8 — keep the fast-ish sampling and the smooth
latent while restoring a stable, likelihood-like training objective. Read GANSynth as the
idea that proved instrument timbre has a learnable manifold; later families
inherited the goal and swapped the engine.
— the through-line into Part 8
Contrast with Part 6. DDSP reaches a smooth, controllable timbre space by
a different route: it bakes in a differentiable harmonic-plus-noise synthesiser, so the
model's priors carry phase and periodicity for free. GANSynth learns the
representation from scratch and pays for it in training difficulty; DDSP engineers the
representation and pays for it in generality. Two answers to one question — how to make
timbre interpolable.
72017 → 2019, in one column
Adversarial instrument synthesis is a short, sharp arc: a dataset, two proof-of-concept
GANs, and the representation that made them sing — all inside three years.
2017
NSynth (Engel et al.) releases 305k aligned instrument notes and a WaveNet autoencoder — high quality, but slow and without a global timbre latent. Part 7
2018
Progressive GAN (Karras et al.) grows image GANs resolution-by-resolution with WGAN-GP — the training recipe GANSynth adopts wholesale. image world
2019
WaveGAN & SpecGAN (Donahue et al., ICLR) establish adversarial audio in the waveform and spectrogram domains — and expose the checkerboard and phase problems. Part 7
2019
GANSynth (Engel et al., ICLR) predicts log-magnitude + instantaneous frequency, grows progressively, and conditions on pitch — fast, high-quality, interpolable instrument notes. Part 7
2020–22
DDSP offers a DSP-prior alternative for timbre; diffusion models begin displacing GANs on stability and coverage. Part 6 · 8
→Where to go next
You have seen adversarial synthesis solve instrument timbre: the instantaneous-frequency
representation that makes phase learnable, the progressive growing and pitch conditioning
that make training work, and the smooth latent that makes morphing possible — paid for
with training fragility. The next part keeps the goal and swaps the engine, trading the
minimax game for a stable denoising process.