Neural Sound Synthesis · Part 4 — End-to-End Text-to-Speech

1The pipeline — from a string of letters to a moving voice

Text-to-speech is a translation between two wildly mismatched sequences: a handful of discrete characters, and hundreds of thousands of continuous audio samples with their own pitch, rhythm and timbre. The modern solution splits the work into two learned stages joined by a spectrogram, and the story of this part is how that seam was progressively erased.

$$\underbrace{\text{text}}_{\sim 20\ \text{chars}} \;\xrightarrow{\ \text{acoustic model}\ }\; \underbrace{\text{mel spectrogram}}_{80\times\sim 800}\;\xrightarrow{\ \text{vocoder}\ }\; \underbrace{\text{waveform}}_{\sim 3\times10^{5}\ \text{samples}}$$

The acoustic model (this part) maps text to a mel spectrogram — an 80-band, ~86-frames-per-second picture of the sound. The vocoder (Part 3) inverts that mel into an audible waveform. The mel is the interface: it throws away phase and fine detail so the acoustic model can work in a compact, friendly space, and lets the vocoder be trained and swapped independently.

The hard part is that nobody tells the model the alignment. A word may map to three frames or thirty; the network must discover, unsupervised, which stretch of audio each letter is responsible for. Everything below is a different answer to that one question: soft attention (Tacotron), an explicit duration (FastSpeech), or a monotonic search inside a VAE (VITS).

System	Alignment mechanism	Decoder	Speed	Robustness
Tacotron 2	Learned soft attention (location-sensitive)	Autoregressive, one frame at a time	Slow	Fragile — can babble / loop
FastSpeech 2	Explicit duration per phoneme + length regulator	Non-autoregressive, all frames at once	Very fast	Robust by construction
VITS	Monotonic Alignment Search (MAS) inside a CVAE	Flow + HiFi-GAN, text → waveform	Fast	Robust, fully end-to-end

Two stages or one? "End-to-end" gets used loosely. Tacotron 2 is end-to-end text → mel but still needs a separate vocoder. VITS is end-to-end text → waveform with no mel at inference at all. This part walks that line from left to right.

2Tacotron & the fragility of attention

Tacotron (2017) was the first system to go from characters to a spectrogram in a single trained network. Its heart is a sequence-to-sequence model with attention: an encoder reads the text, and at every decoder step the model computes a weighted average over the encoded text — a soft pointer to "which letter am I speaking now?".

Encoder, attention, autoregressive decoder

The encoder is a CBHG stack — a 1-D convolution bank, highway layers, and a bidirectional GRU — turning character embeddings into a sequence of context vectors $h_1,\dots,h_T$. The decoder is autoregressive: it emits one mel frame at a time, feeding each predicted frame back as the input to the next, and it decides where to look with attention weights

$$\alpha_{i} = \operatorname{softmax}(e_{i}),\qquad e_{i,j} = \mathbf{v}^{\!\top}\tanh\!\big(W\,\mathbf{s}_{i-1} + V\,h_j + U\,f_{i,j} + b\big)$$

where $\mathbf{s}_{i-1}$ is the decoder state, $h_j$ the $j$-th text vector, and the context fed to the decoder is $\mathbf{c}_i = \sum_j \alpha_{i,j}\,h_j$. Tacotron 2 replaced the CBHG with plain convolutions + LSTM and swapped the vocoder for WaveNet, reaching near-human MOS — but the attention is the fragile organ. A separate stop token is predicted each step so the decoder knows when to halt.

Content vs location-sensitive attention

Plain content-based attention scores a text vector purely on how well it matches the current state — it has no notion that speech marches forward. It can jump backwards, stick, or attend to two places at once, producing the notorious failure modes: repeated syllables, skipped words, or a collapse into babble. The $U f_{i,j}$ term above is location-sensitive attention: $f_{i,j}$ is convolved from the previous attention weights, biasing the model to advance smoothly and monotonically. It is the single most important trick for stable Tacotron.

Rows = text tokens, columns = decoder mel frames. A healthy alignment is a clean monotonic diagonal. Switch to failure to see stalls, skips and split attention.

Read the diagonal. Each column is one decoder step's distribution over the text. A crisp, steadily-rising ridge means the model knows exactly where it is. That is what "good alignment" looks like — and why researchers plot attention matrices to debug TTS.

What "the model lost alignment" means. In the failure mode the ridge stalls on one token (a repeat), then leaps ahead skipping several (a skip), and mass splits across rows (a collapse). Audibly: stuttered or dropped words, or noise. Location-sensitivity exists to prevent exactly this.

derivation · why soft attention drifts, and what location-sensitivity fixes

Content attention chooses the alignment for frame $i$ from the energies $e_{i,j}=g(\mathbf{s}_{i-1},h_j)$ alone. Nothing in $g$ references the previous alignment $\alpha_{i-1}$, so the argmax over $j$ can move anywhere between steps — including backwards or nowhere. Over a long utterance small errors compound: once the decoder feeds back a slightly-wrong frame, the state drifts, the next attention is worse, and the loop diverges.

Location-sensitive attention adds a term computed from the cumulative attention. Let $F = \mathrm{conv1d}\!\big(\sum_{i'after where the model has already attended, the energy is biased to advance by roughly one token per step — encoding the prior that speech is monotonic and surjective. FastSpeech and VITS take this to its logical end: stop softening the alignment and make it a hard, monotonic map.

3FastSpeech — make alignment explicit, generate in parallel

FastSpeech (2019) asked: if the failure is the soft attention, why keep it? If we simply knew how many frames each phoneme lasts, we could copy each phoneme encoding that many times and decode every frame in parallel — no recurrence, no drift, no runaway loops. That is the length regulator, and it makes TTS both fast and robust.

The length regulator

Given phoneme encodings $H=[h_1,\dots,h_K]$ and predicted integer durations $d=[d_1,\dots,d_K]$, the length regulator expands $H$ into a frame-level sequence by repeating each $h_k$ exactly $d_k$ times:

$$\mathrm{LR}(H,d) = \big[\,\underbrace{h_1,\dots,h_1}_{d_1},\ \underbrace{h_2,\dots,h_2}_{d_2},\ \dots,\ \underbrace{h_K,\dots,h_K}_{d_K}\,\big],\qquad \text{length} = \sum_{k=1}^{K} d_k$$

A duration predictor — a small conv network on top of the encoder — estimates $d_k$ (in log-space) from the phonemes alone, so at inference no target audio is needed. Multiply all durations by a constant $1/\alpha$ and the whole utterance speeds up or slows down while keeping pitch; that is the speaking-rate knob in the demo.

Where the durations come from — FS1 vs FS2

FastSpeech 1 distils them from an autoregressive teacher (a Tacotron-like model): it runs the teacher, reads out its attention matrix, and takes the number of frames each phoneme "won" as the duration label. Clever, but it inherits the teacher's alignment errors. FastSpeech 2 drops the teacher and supervises durations directly from forced alignment (Montreal Forced Aligner), and — crucially — adds variance adaptors: it also predicts per-frame pitch ($F_0$) and energy, injecting them before decoding. This fixes the one-to-many problem (the same text can be said many ways) that made FS1 sound flat.

speaking rate 1.00

Four phonemes of "hello". Drag a duration slider: the length regulator widens that phoneme's block, the formant contour holds longer on it, and the synthesised sound lengthens audibly.

Try this: push OW's slider to the maximum and press Synthesise — the final vowel drawls. Now pull it to 1 and stretch EH instead. Same phonemes, same pitch; only the timing changed, exactly as the length regulator dictates. The speaking-rate control scales every duration at once.

derivation · the duration loss and why log-domain

The duration predictor $\hat d = \mathrm{DP}(H)$ is trained with a mean-squared error against alignment-derived targets $d^\star$, but in the log domain:

$$\mathcal{L}_{\text{dur}} = \big\lVert \log(1+\hat d) - \log(1+d^\star)\big\rVert_2^2.$$

Durations are positive and heavy-tailed — a stressed vowel can last ten times a plosive. Regressing in log-space makes the loss scale-invariant (a factor-of-two error costs the same for a short and a long phoneme) and keeps the predictor from being dominated by the longest sounds. At inference $\hat d = \mathrm{round}(\exp(\cdot)-1)$, clamped to be non-negative, and the length regulator uses these integer counts. Because every frame is produced independently once $d$ is fixed, decoding is a single parallel pass — the whole reason FastSpeech is an order of magnitude faster than Tacotron.

4VITS — one network, text to waveform

Tacotron and FastSpeech both stop at the mel and hand off to a vocoder — two models, two training runs, and a mismatch between the mel the acoustic model produces and the mel the vocoder was trained on. VITS (2021) removes the seam entirely. It is a conditional variational autoencoder whose decoder is a HiFi-GAN, trained adversarially, with a normalizing-flow prior and a monotonic alignment search standing in for attention. Text goes in; a waveform comes out; there is no mel at inference.

hover / tap a block ▸

VITS as a conditional VAE: posterior q(z|x) (bottom) and prior p(z|c) (top) meet at the latent z; a normalizing flow enriches the prior, MAS aligns text to frames, and a HiFi-GAN decoder + discriminators produce the waveform. Hover any block.

The pieces, and how they combine

Posterior encoder $q(z\mid x)$ — non-causal WaveNet blocks read a linear spectrogram of the target audio and output the mean/variance of the latent $z$. Used in training only.
Prior $p(z\mid c)$ — the text encoder predicts a Gaussian per phoneme; a normalizing flow $f_\theta$ then warps it into a far more expressive distribution over $z$.
Monotonic Alignment Search (MAS) — dynamic programming finds the monotonic text-to-frame alignment that maximises the likelihood, replacing attention with a hard, non-skipping map. A stochastic duration predictor (itself a flow) lets rhythm vary run to run.
HiFi-GAN decoder — transposed convolutions upsample $z$ straight to a waveform, trained with multi-period + multi-scale discriminators plus feature-matching and mel losses.

The objective — an ELBO with a flow

VITS maximises a conditional evidence lower bound: reconstruct the audio from $z$, while keeping the posterior close to the text-conditioned prior.

$$\log p(x\mid c) \;\ge\; \mathbb{E}_{q(z\mid x)}\big[\log p(x\mid z)\big] \;-\; D_{\mathrm{KL}}\!\big(q(z\mid x)\,\big\|\,p(z\mid c)\big)$$

The flexible prior is what the flow buys. With $z' = f_\theta(z)$ an invertible map into the space where the text-side Gaussian $\mathcal{N}(\mu_\theta,\sigma_\theta)$ lives, the change-of-variables formula gives

$$p(z\mid c) = \mathcal{N}\!\big(f_\theta(z);\,\mu_\theta(c,A),\,\sigma_\theta(c,A)\big)\,\left|\det\frac{\partial f_\theta(z)}{\partial z}\right|$$

where $A$ is the monotonic alignment chosen by MAS. In practice the reconstruction term is not a raw-waveform likelihood but a mel-reconstruction loss plus the adversarial + feature-matching losses of HiFi-GAN — so VITS is, literally, a VAE and a flow and a GAN optimised together. That triple is why its samples are both diverse (VAE/flow) and crisp (GAN).

derivation · why the KL term needs the flow's log-determinant

The KL between the Gaussian posterior $q(z\mid x)=\mathcal{N}(\mu_q,\sigma_q)$ and a plain Gaussian prior would be closed-form but far too rigid — real latent distributions of speech are multi-modal. VITS instead defines the prior implicitly through an invertible flow. Writing the KL as an expectation,

$$D_{\mathrm{KL}} = \mathbb{E}_{q}\big[\log q(z\mid x) - \log p(z\mid c)\big],$$

and substituting the change-of-variables density,

$$\log p(z\mid c) = \log \mathcal{N}\!\big(f_\theta(z);\mu_\theta,\sigma_\theta\big) + \log\left|\det \tfrac{\partial f_\theta}{\partial z}\right|,$$

the Jacobian log-determinant is exactly the correction that keeps $p(z\mid c)$ a valid normalised density after the warp. VITS builds $f_\theta$ from affine coupling layers whose Jacobian is triangular, so $\log|\det|$ is just the sum of the log-scale outputs — cheap to compute and to backprop. The alignment $A$ enters through $\mu_\theta(c,A),\sigma_\theta(c,A)$: MAS picks the monotonic $A$ that maximises this very prior likelihood, so alignment and density estimation are optimised by the same objective — no external aligner, no attention to lose.

Every earlier idea survives inside VITS, hardened. Location-sensitive attention becomes a hard monotonic search; FastSpeech's deterministic duration becomes a stochastic, flow-sampled one; the mel-then-vocoder split collapses into a single latent. It is the clearest example in this series of composing primitives — VAE, flow, GAN — into one system. — the through-line from §2 to §4

5Glow-TTS, and the token-LM frontier

VITS did not appear from nowhere. Two threads fed it, and one thread leads out of it toward the language-model era that Part 10 explores.

Glow-TTS — the missing link

Glow-TTS (2020) is the model that made monotonic alignment search practical. It is a flow-based, non-autoregressive text-to-mel model: a flow maps the mel to a latent whose prior is a per-phoneme Gaussian, and MAS finds the monotonic alignment between them by dynamic programming — no attention, no external aligner, exact likelihood. VITS takes Glow-TTS's MAS and prior flow, swaps text-to-mel for text-to-waveform by bolting on a HiFi-GAN decoder and adversarial training, and wraps the whole thing in a VAE. Read Glow-TTS as VITS minus the vocoder and the adversary.

Beyond spectrograms — TTS as a language model

Everything above generates continuous acoustic features. A different lineage treats speech as a sequence of discrete tokens from a neural codec (Part 9) and models them with a transformer language model — literally next-token prediction over sound. VALL-E (2023) reframed TTS this way: condition the LM on a 3-second enrolment clip and it performs zero-shot voice cloning, reproducing a speaker's timbre and prosody from that single prompt, because in-context learning transfers from text LMs to audio tokens. This is the bridge to prompt-based, zero-shot TTS — and the subject of Part 10.

Two philosophies, one goal. VITS pours everything into a single end-to-end network trained from scratch on one corpus. Token-LM TTS (VALL-E, and successors) leans on massive codec-token pretraining and a prompt. The field now runs on both rails; which wins depends on data, latency and how much control you want.

6Five years, end to end

The seam between text and waveform was erased in stages, each dissolving one more hand-built component. In 2017 you needed a vocoder, an aligner and a lot of luck; by 2021 a single network did the lot.

2017

Tacotron makes character-to-spectrogram end-to-end with seq2seq attention. Tacotron 2 adds location-sensitive attention + a WaveNet vocoder and reaches near-human MOS — but the attention is fragile. §2

2019

FastSpeech ditches soft attention for an explicit duration predictor and a length regulator — non-autoregressive, parallel, robust — with durations distilled from an AR teacher. §3

2020

Glow-TTS introduces monotonic alignment search with a flow prior; FastSpeech 2 supervises duration from forced alignment and adds pitch/energy variance adaptors. §3 · §5

2021

VITS unifies text encoder, MAS, flow prior, stochastic duration and a HiFi-GAN decoder into one adversarially-trained CVAE — text to waveform, no mel at inference. §4

2023

VALL-E recasts TTS as a language model over neural-codec tokens, unlocking zero-shot voice cloning from a 3-second prompt — the road into Part 10. §5 · Part 10

→Where to go next

You now have the acoustic-model half of speech synthesis: how text is aligned to time (attention, durations, MAS), how a spectrogram is predicted, and how VITS folds even the vocoder into the same network. The mel these models emit still has to become sound — that inversion is Part 3. Extending the same machinery with an explicit pitch target turns speech into song in Part 5, and treating audio as tokens for a language model is Part 10.

Series hubAll twelve parts · the interactive map Part 3 ←Neural Vocoders — the mel inverter Part 5 →Neural Singing Voice — TTS + pitch CollectionDSP & Music

References

Wang, Y. et al. (2017). Tacotron: Towards End-to-End Speech Synthesis. Interspeech.
Shen, J. et al. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2). ICASSP.
Chorowski, J. et al. (2015). Attention-Based Models for Speech Recognition. NeurIPS. (Location-sensitive attention.)
Ren, Y. et al. (2019). FastSpeech: Fast, Robust and Controllable Text to Speech. NeurIPS.
Ren, Y. et al. (2021). FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. ICLR.
Kim, J. et al. (2020). Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search. NeurIPS.
Kim, J., Kong, J., Son, J. (2021). Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (VITS). ICML.
Wang, C. et al. (2023). Neural Codec Language Models are Zero-Shot Text-to-Speech Synthesizers (VALL-E). arXiv:2301.02111.