Neural Sound Synthesis · Part 3 — Neural Vocoders (GAN)

1The vocoder problem — invert a spectrogram that threw its phase away

Modern text-to-speech and singing systems do not predict a waveform. They predict a mel spectrogram — a compact time×frequency picture of magnitudes. Something must then turn that picture back into 22,050 samples per second of moving air. That something is a vocoder, and its job is harder than it looks, because the spectrogram is missing exactly the thing that makes a waveform a waveform: its phase.

What the mel spectrogram keeps, and what it discards

The short-time Fourier transform (STFT) slides a window across the signal and takes a DFT of each frame, producing a complex value $S_{t,f}=|S_{t,f}|\,e^{j\angle S_{t,f}}$ for every time–frequency bin. A mel spectrogram keeps only the magnitude $|S_{t,f}|$, warped onto ~80 perceptual mel bands and log-compressed. The phase $\angle S_{t,f}$ — which encodes where each sinusoid sits in time — is thrown on the floor.

$$S_{t,f} = \sum_{n} x[n]\,w[n-tH]\,e^{-j 2\pi f n / N}, \qquad \text{mel keeps only } |S_{t,f}|.$$

Reconstruction must therefore invent a phase field that is (a) consistent with the kept magnitudes and (b) consistent with itself across overlapping frames — because the same sample is covered by several windows, and they must agree. That second constraint is what makes the problem non-trivial: an arbitrary phase gives a spectrogram whose overlapping frames contradict each other, and the ear hears the contradiction as a metallic, watery, "phasey" buzz.

Why work in mel at all, then? Because it is roughly $256\times$ smaller than the waveform and perceptually smooth, so the acoustic model (Part 4) that maps text to it has a far easier target. The field's bargain is a two-stage pipeline: a slow, easy acoustic model plus a fast, hard vocoder. This part is about making that second stage fast and faithful.

derivation · why magnitude-only is missing "half" the information

A length-$N$ real frame has $N$ real degrees of freedom. Its DFT has $N/2{+}1$ complex bins, i.e. $N{+}2$ real numbers, but conjugate symmetry and the real DC and Nyquist bins remove the redundancy back down to $N$. Splitting into magnitude and phase,

$$\underbrace{|S_f|}_{N/2+1\text{ reals}} \;\cup\; \underbrace{\angle S_f}_{N/2-1\text{ reals}} = N \text{ reals}.$$

Keeping magnitude alone discards the ~$N/2$ phase reals — very nearly half the frame's information. What rescues inversion is overlap: with hop $H \ll N$ each sample sits under many windows, so the magnitudes are massively redundant across frames. That redundancy is a set of consistency equations the true phase satisfies — and which Griffin–Lim, and later a neural network, exploit to solve for a plausible phase.

2Griffin–Lim — the classical answer, and why it is not enough

Before neural vocoders, phase was recovered by iteration. Griffin & Lim (1984) start from random phase, run an inverse STFT to get a candidate waveform, take its STFT, throw away the phase they just produced but keep the true target magnitude, and repeat. Each round the overlapping frames agree a little more. It works — slowly, and never perfectly.

$$x^{(i+1)} = \operatorname{ISTFT}\!\Big(\, |S|\;\odot\;\exp\!\big(j\,\angle\,\operatorname{STFT}(x^{(i)})\big)\Big)$$

The update is an alternating projection: onto the set of signals with the correct magnitude, then onto the set of consistent STFTs (those that are the STFT of an actual time signal). It provably never increases the spectral inconsistency, but it converges to a local minimum whose invented phase is only plausible, not correct — and audibly so at low iteration counts.

iterations 6

A sung vowel is generated in-browser, its magnitude STFT computed, then phase is rebuilt from random by real Griffin–Lim iteration. Few iterations = phasey; many = clean.

Try this: set 1–3 iterations and press Griffin–Lim — the vowel is recognisable but robotic and buzzy, the invented phase fighting itself across frames. Push to 40–50 and it cleans up, but every one of those iterations is a full ISTFT+STFT pass. A neural vocoder learns to jump straight to good phase in one feed-forward pass — no iteration at all.

The three limits that motivated everything after

Slow. Dozens of STFT/ISTFT round-trips per clip; unusable for real-time synthesis.
Only plausible phase. It minimises spectral inconsistency, not perceptual quality — the result is intelligible but characteristically artificial.
Magnitude must be exact. It assumes the target magnitude is the true STFT magnitude. A predicted mel is lossy and mel-warped, so inverting it back to a linear STFT (a pseudo-inverse of the mel filterbank) adds error before Griffin–Lim even starts.

derivation · Griffin–Lim as alternating projection & the consistency measure

Let $\mathcal{A}$ be the set of complex STFTs with the target magnitude $|S|$, and $\mathcal{C}$ the set of consistent STFTs — those equal to $\operatorname{STFT}(\operatorname{ISTFT}(\cdot))$ for some time signal. Griffin–Lim alternately projects onto each:

$$\hat S^{(i+1)} = P_{\mathcal{A}}\big(P_{\mathcal{C}}(\hat S^{(i)})\big),$$

where $P_{\mathcal{C}}$ is exactly the ISTFT-then-STFT round trip and $P_{\mathcal{A}}$ replaces each magnitude with the target while keeping the phase. The quantity it monotonically decreases is the spectral inconsistency

$$\mathcal{I}(\hat S) = \big\lVert \hat S - \operatorname{STFT}(\operatorname{ISTFT}(\hat S)) \big\rVert^2,$$

and the demo's error meter reports the closely related spectral convergence $\lVert\,|\operatorname{STFT}(x^{(i)})| - |S|\,\rVert / \lVert |S| \rVert$. Because both projections are non-expansive but $\mathcal{A}$ is non-convex (a magnitude constraint), convergence is only to a local optimum — the reason the invented phase is plausible rather than true.

3Why GANs — trade likelihood for a single fast pass

Autoregressive vocoders like WaveNet (Part 2) invert a mel with superb quality but emit one sample at a time — thousands of sequential passes per second. The generative adversarial network offers the opposite bargain: a convolutional generator maps the whole mel to the whole waveform in one parallel forward pass, and a discriminator — trained to tell real audio from generated — supplies the only "loss" rich enough to police fine waveform detail.

$$\min_G \max_D \; \mathbb{E}_{x}\big[\log D(x)\big] + \mathbb{E}_{s}\big[\log\!\big(1 - D(G(s))\big)\big],\qquad s=\text{mel},\; x=\text{waveform}$$

Why does the discriminator matter so much? Because a plain regression loss — mean-squared error between predicted and true samples — is blind to phase in exactly the wrong way. Two waveforms can sound identical yet differ sample-by-sample (a shifted pitch pulse), so MSE punishes correct-sounding audio and rewards the blurry, over-smoothed average of all valid phases: a muffled buzz. The discriminator instead learns a perceptual notion of "realness" that tolerates phase shifts but flags the metallic textures Griffin–Lim leaves behind.

The prize. HiFi-GAN synthesises 22 kHz audio hundreds of times faster than real time on a GPU — and faster than real time on a single CPU thread — at quality rivalling autoregressive WaveNet. Feed-forward, no iteration, no per-sample loop.

The peril. GANs have no likelihood and can mode-collapse or hallucinate artefacts a regression model never would. The entire craft below — feature matching, mel loss, multiple discriminators — exists to tame that instability without giving up the speed.

4MelGAN & Parallel WaveGAN — making GAN vocoders actually work

The first GAN vocoders that held together arrived in 2019. Two ideas from them survive in every model since: feature-matching (MelGAN) and a multi-resolution STFT loss (Parallel WaveGAN). Both are ways to give the generator a stable, informative signal that raw adversarial loss alone cannot.

MelGAN — window-based discriminators + feature matching

MelGAN's generator is a stack of transposed convolutions with residual blocks — fully feed-forward, non-autoregressive. Its key contributions are on the discriminator and loss side. It uses several identical discriminators operating on the raw waveform at different downsampling rates (a multi-scale critic), and it adds a feature-matching loss: match the discriminator's intermediate feature maps for real and generated audio, not just its final real/fake verdict.

$$\mathcal{L}_{\text{FM}}(G;D) = \mathbb{E}_{x,s}\left[\sum_{i=1}^{T}\frac{1}{N_i}\big\lVert D^{(i)}(x) - D^{(i)}(G(s))\big\rVert_1\right]$$

Here $D^{(i)}$ is the $i$-th of $T$ layers and $N_i$ its element count. This is a learned perceptual loss: the discriminator's own hierarchy of features becomes a yardstick for similarity, far more informative than a single scalar and far more stable than pure adversarial gradient. It is the single most important trick for keeping GAN vocoder training from diverging.

Parallel WaveGAN — non-autoregressive, multi-resolution STFT loss

Parallel WaveGAN (PWG) pairs a small WaveNet-style generator (dilated convs, non-autoregressive, conditioned on the mel and driven by noise) with a multi-resolution STFT loss: compute STFT losses at several window/hop/FFT settings and average them, so the model is supervised on both fine time resolution (short windows) and fine frequency resolution (long windows) at once.

$$\mathcal{L}_{\text{aux}} = \frac{1}{M}\sum_{m=1}^{M}\Big(\underbrace{\tfrac{\lVert |S_m| - |\hat S_m| \rVert_F}{\lVert |S_m| \rVert_F}}_{\text{spectral convergence}} + \underbrace{\tfrac{1}{K}\big\lVert \log|S_m| - \log|\hat S_m|\big\rVert_1}_{\text{log-magnitude}}\Big)$$

The two terms are complementary: spectral convergence emphasises large spectral peaks (formants, harmonics), while the log-magnitude term emphasises the quiet valleys the ear still hears. Averaging over $M$ resolutions prevents the model from over-fitting artefacts tuned to any single STFT grid — a direct antidote to the single-grid brittleness that plagues Griffin–Lim.

Model	Generator	Discriminator	Signature loss
MelGAN (2019)	Transposed-conv + residual stacks, feed-forward	Multi-scale on raw / ×2 / ×4 audio	Adversarial + feature matching
Parallel WaveGAN (2019)	Non-AR dilated WaveNet, noise + mel input	Single waveform critic	Adversarial + multi-resolution STFT
HiFi-GAN (2020)	Transposed-conv upsampling + MRF blocks	MPD (primes) + MSD (pooled)	Adversarial (LSGAN) + FM + L_mel

5The HiFi-GAN generator — transposed-conv upsampling + multi-receptive-field fusion

HiFi-GAN (Kong et al., 2020) became the default neural vocoder, and its generator is a model of clarity. It takes an 80-band mel at frame rate and upsamples it stage by stage — each stage a transposed 1-D convolution — until the time axis reaches the waveform sample rate. After every upsample it applies a multi-receptive-field fusion (MRF) block: several residual blocks with different kernel sizes and dilations, summed, so the network sees many temporal scales at once.

Hover a stage to read what it does. Mel frames (80×T) are upsampled ×8·8·2·2 = 256 to waveform rate, each stage followed by an MRF block fusing three dilated residual paths.

The upsampling factors $[8,8,2,2]$ multiply to $256$ — precisely the STFT hop, so a mel at ~86 frames/s becomes audio at $86\times256 \approx 22{,}050$ samples/s. Transposed convolution is learnable upsampling: it inserts zeros and convolves, letting the network synthesise the fine structure between mel frames rather than merely interpolate. But transposed convs alone cause the notorious "checkerboard" artefacts and see only a narrow context — which is what the MRF fixes.

Multi-receptive-field fusion (MRF)

An MRF block runs $|k_r|$ residual blocks in parallel — in HiFi-GAN V1 with kernel sizes $\{3,7,11\}$ and dilations $\{1,3,5\}$ — and sums their outputs. A short kernel captures pitch-period detail; a long, dilated kernel captures formant-scale envelope. Their sum lets one stage attend to fine and coarse structure simultaneously, which is exactly what a waveform demands (recall Part 1's long-range-dependency problem).

$$\text{MRF}(x) = \sum_{r} \text{ResBlock}_{k_r,\,D_r}(x),\qquad k_r \in \{3,7,11\},\; D_r \in \{[1,3,5],\dots\}$$

Design knob. V1 (large) maximises quality; V2/V3 shrink channel counts and MRF widths for speed. The architecture degrades gracefully — the same block structure, fewer parameters — which is why HiFi-GAN spans from server-grade fidelity to on-device real-time from one design.

6The discriminators — periods the 1-D critic cannot see

A generator is only as good as the critic that trains it. HiFi-GAN's decisive idea is a pair of discriminator ensembles. The Multi-Period Discriminator (MPD) reshapes the 1-D waveform into 2-D grids indexed by several prime periods, exposing periodic structure a flat 1-D convolution smears over. The Multi-Scale Discriminator (MSD) looks at the raw, ×2- and ×4-pooled waveform to police longer-range, smoother consistency.

The MPD reshape — why prime periods

Speech and singing are quasi-periodic: a pitch period repeats every few milliseconds. A 1-D convolution with a modest kernel cannot span a whole period, so period-length artefacts (a subtly wrong pitch pulse, a buzzy sub-harmonic) slip past it. The MPD's trick: take the length-$T$ signal and fold it into a 2-D array of width $p$, so every column collects samples one period apart. When $p$ matches the signal's period, each column lines up in phase and vertical structure snaps into view — and a 2-D convolution can now catch it. Drag the reshape period below.

signal period P₀ reshape p

Top: a quasi-periodic 1-D signal. Bottom: it folded into rows of width p. Set p = P₀ and the columns align into clean vertical bands — that is what the 2-D discriminator sees.

Try this: leave the signal period at 7 and step the reshape p through 2, 3, 5, 7, 11. Only at p = 7 do the columns become coherent vertical stripes (the alignment meter spikes). Since you never know an artefact's period in advance, HiFi-GAN runs several sub-discriminators at periods $\{2,3,5,7,11\}$ — coprime, so together they cover a wide range of period lengths with minimal overlap.

The MSD — smoothed, downsampled views

The MPD sees fine periodic detail but each of its sub-discriminators only touches disjoint samples (every $p$-th one). The Multi-Scale Discriminator complements it by operating on contiguous audio at three scales — raw, ×2-pooled, ×4-pooled — the same multi-scale idea MelGAN introduced. Coarser scales judge long-range envelope and rhythm; the raw scale judges high-frequency crispness. Together, MPD + MSD leave few places for an artefact to hide.

derivation · what the MPD reshape does to a periodic component

Write the 1-D signal as $x[n]$ and reshape it row-major into a 2-D array $X[r,c] = x[r p + c]$ of width $p$. Column $c$ therefore samples $x[c], x[c+p], x[c+2p], \dots$ — points spaced exactly $p$ apart. If $x$ has period $P_0$ and $p = P_0$, then $x[rP_0 + c] = x[c]$ for all $r$: every row is identical and each column is constant, so the array is pure vertical stripes. A 2-D convolution over $X$ then trivially detects the periodicity as a horizontal-invariance pattern.

If instead $\gcd(p, P_0) = g < p$, column $c$ cycles through $P_0/g$ distinct phase values and the vertical coherence is partial or absent. This is precisely why the chosen periods are prime: distinct primes share no common factor, so their reshapes probe genuinely different periodicities and a periodic artefact of almost any length trips at least one of them.

7The objective — three losses in one balance

HiFi-GAN's training loss is a weighted sum of three ingredients, applied against every sub-discriminator $D_k$ in $\{$MPD periods$\} \cup \{$MSD scales$\}$. Each term does a distinct job; drop any one and quality or stability suffers.

1 · Adversarial loss (least-squares GAN)

HiFi-GAN uses the LSGAN form — squared error to the labels $1$ (real) and $0$ (fake) — which gives smoother, non-saturating gradients than the original log-loss and is far more stable for audio.

$$\mathcal{L}_{\text{adv}}(D_k)=\mathbb{E}_{x,s}\big[(D_k(x)-1)^2 + D_k(G(s))^2\big],\qquad \mathcal{L}_{\text{adv}}(G;D_k)=\mathbb{E}_{s}\big[(D_k(G(s))-1)^2\big]$$

2 · Feature-matching loss

The MelGAN idea, applied to every sub-discriminator: match intermediate feature maps of real and generated audio. This is the perceptual backbone that keeps training on the rails.

$$\mathcal{L}_{\text{FM}}(G;D_k)=\mathbb{E}_{x,s}\left[\sum_{i=1}^{T}\frac{1}{N_i}\big\lVert D_k^{(i)}(x)-D_k^{(i)}(G(s))\big\rVert_1\right]$$

3 · Mel-spectrogram reconstruction loss

An $\ell_1$ distance between the mel of the generated waveform and the target mel. It anchors the generator to the correct content — the right words, pitch, and timbre — while the adversarial and FM terms handle realism. It also sharply accelerates early training.

$$\mathcal{L}_{\text{mel}}(G)=\mathbb{E}_{x,s}\big[\lVert \phi(x)-\phi(G(s))\rVert_1\big],\qquad \phi=\text{mel-spectrogram operator}$$

The full objective

Sum the adversarial and feature-matching terms over all $K$ sub-discriminators, add the mel loss with its large weight, and you have HiFi-GAN's generator and discriminator objectives:

$$\mathcal{L}_G = \sum_{k=1}^{K}\Big[\mathcal{L}_{\text{adv}}(G;D_k) + \lambda_{\text{fm}}\,\mathcal{L}_{\text{FM}}(G;D_k)\Big] + \lambda_{\text{mel}}\,\mathcal{L}_{\text{mel}}(G), \qquad \mathcal{L}_D = \sum_{k=1}^{K}\mathcal{L}_{\text{adv}}(D_k)$$

with $\lambda_{\text{fm}} = 2$ and $\lambda_{\text{mel}} = 45$. That lopsided weighting is telling: the mel loss dominates numerically, pinning down what to say, while the adversarial and feature-matching terms — small in magnitude but rich in gradient — decide how real it sounds. The GAN supplies the phase and micro-texture that no reconstruction loss can, at the speed no autoregressive model can match.

Read the whole architecture as a division of labour: the mel loss says the words, the feature-matching loss says them cleanly, and the discriminators — periodic and multi-scale — make sure they sound like a person and not like Griffin–Lim.

8Beyond HiFi-GAN & a five-year timeline

HiFi-GAN's template — feed-forward upsampling generator, ensemble of discriminators, three-term loss — proved so durable that later work mostly refined the generator's inductive bias rather than replacing the recipe.

UnivNet added a multi-resolution spectrogram discriminator for broadband fidelity. BigVGAN (2022) scaled the generator up and, crucially, replaced plain activations with a periodic Snake activation plus anti-aliased up/down-sampling, giving a genuinely universal vocoder that generalises across unseen speakers, languages, and even non-speech audio from a single model — the current strong baseline. The through-line is inductive bias: bake periodicity and anti-aliasing into the generator so the adversarial loss has less to fix.

2019

MelGAN — feed-forward GAN vocoder with multi-scale discriminators and the feature-matching loss. GAN-TTS and Parallel WaveGAN land the same year, PWG adding the multi-resolution STFT loss. GAN vocoders arrive

2020

HiFi-GAN sets the standard: transposed-conv + MRF generator, MPD (prime periods) + MSD, LSGAN + FM + L_mel. Becomes the default vocoder across the field. The standard

2021

UnivNet and Fre-GAN push spectral discriminators and multi-resolution critics for broadband quality. Refinement

2022

BigVGAN — Snake activation, anti-aliased sampling, and scale deliver a universal vocoder that generalises far beyond its training distribution. Universal

2020+

In parallel, diffusion vocoders — DiffWave, WaveGrad — trade single-pass speed for iterative denoising and their own quality frontier. Part 8

→Where to go next

You now have the fast half of the two-stage pipeline: how a magnitude-only mel becomes a waveform in one adversarial forward pass, why Griffin–Lim was not enough, and how HiFi-GAN's generator, discriminators, and losses fit together. The obvious question is where those mels come from — the acoustic model of Part 4 — and the alternative inversion route of Part 8's diffusion vocoders. And it all stands against Part 2's autoregressive vocoders, the slow, high-fidelity baseline this whole part was built to out-run.

Series hubAll twelve parts · the interactive map Part 2 ←Autoregressive Waveform Models Part 4 →End-to-End TTS & Acoustic Models

References

Griffin, D. & Lim, J. (1984). Signal Estimation from Modified Short-Time Fourier Transform. IEEE Trans. ASSP, 32(2).
Kumar, K. et al. (2019). MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. NeurIPS.
Yamamoto, R., Song, E., Kim, J.-M. (2020). Parallel WaveGAN: A Fast Waveform Generation Model Based on GAN with Multi-Resolution Spectrogram. ICASSP.
Kong, J., Kim, J., Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS.
Mao, X. et al. (2017). Least Squares Generative Adversarial Networks. ICCV.
Jang, W. et al. (2021). UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators. Interspeech.
Lee, S. et al. (2022). BigVGAN: A Universal Neural Vocoder with Large-Scale Training. arXiv:2206.04658.