Neural Sound Synthesis · Part 5 — Neural Singing Voice

1Why singing is harder than speech

A text-to-speech model (Part 4) is handed words and asked to infer a natural prosody — where the pitch rises, how long each phoneme lasts. A singing-voice-synthesis (SVS) model is handed the same words plus a score: an exact melody, exact note durations, dynamics, articulation. That extra conditioning sounds like it should make the job easier. It makes it far harder, because now every one of those quantities must be rendered precisely — a singer who is 30 cents flat is "out of tune", while a talker 30 cents off pitch is simply talking.

Six demands separate singing from speech:

Continuous, accurate melodic F0. Pitch is not decoration — it is the tune. The model must track a target contour to within a few cents and glide (portamento) between notes rather than stepping.
Vibrato. A quasi-periodic 5–7 Hz modulation of F0, tens of cents deep, that human ears read as expressive warmth. Absent it, a sustained note sounds synthetic; wrong, it sounds seasick.
Sustained vowels. Speech vowels last ~80 ms; a held note can last several seconds. The vocal-tract filter must stay stable and alive (breath, micro-drift) over a span an order of magnitude longer than any spoken phone.
Wide pitch range & register. A trained voice spans two octaves or more, crossing the chest/head register break where timbre changes abruptly.
Dynamics & expression. Crescendo, decrescendo, onset attack, release — the intensity envelope carries as much musicality as the notes.
Phoneme-to-note timing alignment. The lyric's phonemes must be laid onto the score so that the vowel — which carries the pitch — lands on the beat, with consonants clustered just before it. This alignment is given by the score, not inferred, and it is the single biggest structural difference from TTS.

Axis	Text-to-speech (Part 4)	Singing-voice synthesis
Pitch	Inferred prosody; tolerance is loose	Prescribed melody; cents-level accuracy, plus vibrato
Timing	Model predicts phone durations	Durations fixed by note values in the score
Vowel length	~50–120 ms	Up to several seconds, sustained & stable
Conditioning	Phonemes (+ optional style)	Phonemes + F0 + duration + dynamics
Failure mode	Unnatural rhythm	Out of tune / off the beat — immediately obvious

Everything that follows is a way of meeting those demands. We start with the physics the demos are built on — the source-filter model — because it is both the grandparent of every parametric SVS system and the thing you will hear yourself controlling in a moment.

2The source-filter model — and a sung vowel you can shape

A voiced sound is a source passed through a filter. The vocal folds chop the breath into a periodic glottal pulse train at the fundamental frequency $F_0$ — a buzzy sawtooth-like spectrum with a $\sim\!-12$ dB/octave tilt and harmonics at every multiple of $F_0$. The vocal tract above the folds is a resonant tube whose resonances — the formants $F_1, F_2, F_3\ldots$ — boost the harmonics near them. Change the tube's shape (jaw, tongue, lips) and you move the formants; that, and only that, is what turns one buzz into /a/ versus /i/.

$$X(f) = \underbrace{G(f)}_{\text{glottal source }(F_0)} \cdot \underbrace{V(f)}_{\text{vocal-tract filter (formants)}} \cdot \underbrace{R(f)}_{\text{lip radiation}}$$

The demo synthesises exactly this, additively. It builds a harmonic series at a time-varying $F_0$ and weights each harmonic $h$ by a source tilt $1/h$ times the sum of the selected vowel's formant resonances evaluated at that harmonic's frequency:

$$x(t) = \sum_{h=1}^{H} a_h(t)\,\sin\!\big(h\,\phi(t)\big),\qquad a_h = \frac{1}{h}\sum_{k}\frac{g_k}{1+\big((hF_0 - f_k)/b_k\big)^2},\qquad \phi(t)=2\pi\!\int_0^t \!F_0(\tau)\,d\tau$$

with $(f_k, g_k, b_k)$ the centre frequency, gain and bandwidth of the $k$-th formant. The melody comes from the pitch contour you draw; the vibrato adds a slow modulation of $F_0$ measured in cents:

$$F_0(t) = f_{\text{note}}(t)\cdot 2^{\,(d/1200)\,\sin(2\pi r t)}$$

where $d$ is the vibrato depth in cents and $r$ its rate in Hz. Drag on the canvas to reshape the tune, pick a vowel, and push the vibrato controls until it stops sounding like an organ and starts sounding like a voice.

vowel vibrato depth 35¢ vibrato rate 5.5

Drag to draw an F0 curve over 2 s (grid = semitones). The faint wiggle is the vibrato that will actually be sung.

Try this. Draw a long flat held note, set vibrato depth to 0 and press play — flat, dead, electronic. Now push depth to ~40 ¢ and rate to ~5.5 Hz. The same pitch, the same vowel, suddenly reads as sung. That one control is most of what separates a note from a voice.

Why additive, not a filter? A time-varying $F_0$ makes a classic biquad formant filter ring and click at note boundaries. Re-deriving each harmonic's amplitude from the formant curve every sample — as above — keeps the vowel identity fixed while the pitch glides, which is exactly the stability a held note needs.

Vowel	F1 (Hz)	F2 (Hz)	F3 (Hz)	Tongue / lips
`/a/`	~700	~1220	~2600	open, low tongue
`/e/`	~530	~1840	~2480	mid front
`/i/`	~270	~2290	~3010	high front (F1 low, F2 high)
`/o/`	~570	~840	~2410	back, rounded
`/u/`	~300	~870	~2240	high back, rounded (both low)

derivation · why F1 and F2 alone almost fix the vowel

Model the vocal tract as a uniform tube of length $L\approx17$ cm, closed at the glottis and open at the lips. Its resonances fall at

$$f_k = \frac{(2k-1)\,c}{4L},\qquad k=1,2,3,\ldots$$

with $c\approx343$ m/s, giving $\sim500, 1500, 2500$ Hz — the neutral schwa. Constricting the tube at different points shifts these resonances predictably: a constriction near the lips lowers $F_1$; a front-tongue constriction raises $F_2$. Plot any speaker's vowels on the $(F_1, F_2)$ plane and they separate into the familiar vowel quadrilateral — which is why the demo only needs to move a handful of formant peaks to swing convincingly between /a/, /i/ and /u/. The higher formants ($F_3$ up) carry speaker identity and "ring" more than vowel identity; the singer's formant — a cluster around 2.8–3.4 kHz that lets an operatic voice cut through an orchestra — lives there too.

3Score alignment — lyrics pinned to notes

In TTS the model predicts, for each phoneme, how many acoustic frames it should occupy — a learned duration model. In SVS the note values in the score already fix those durations: a half note is a half note. The system's job is the reverse — to distribute each syllable's phonemes within its allotted note so that the vowel, which carries the pitch, lands on the beat and sustains, while the leading consonants are compressed into the moments just before.

The piano roll below makes the point physically. Each column is a time step; each row a pitch. Click to place one note per column — a monophonic melody. The syllables of the lyric are consumed left to right and assigned to the notes you place, so the same formant synth now has to (a) hit each note's pitch and (b) switch vowels at each note onset. That coupling of a phoneme stream to a note stream is the essence of an SVS front end.

note length 0.22s vibrato 30¢

Click cells to build a melody. Syllables (mapped to vowels) are aligned to the notes in time order — that alignment is what SVS is given and TTS must guess.

The front end, in miniature. A real system (NNSVS, OpenUtau/ENUNU) runs a lyric through a grapheme-to-phoneme step, then a note-to-phoneme aligner that honours the score's timing, before any acoustic model runs. The piano roll collapses all of that into one visible rule: syllable $n$ sings on note $n$, for exactly the note's duration.

4Acoustic models — from HMM and VOCALOID to DiffSinger

Given the aligned score, an acoustic model maps (phoneme, note/F0, duration) to a mel-spectrogram, which a vocoder (Part 3) turns into waveform. The history of that mapping runs from hand-crafted statistics to diffusion.

The concatenative and parametric heritage

VOCALOID (Yamaha, 2003–) — the technology behind Hatsune Miku — is concatenative: it stitches together small recorded diphone samples from a singer database, pitch-shifting and time-stretching them to the score. It is controllable and characterful but audibly "spliced", and every new voice needs a laborious recorded library. In parallel, the speech world's HMM/statistical parametric approach was adapted for singing — Sinsy (2010) and Merlin-style DNNs modelled formant and F0 trajectories with hidden Markov models and, later, feed-forward and recurrent nets. Parametric systems are smooth and tiny but over-smooth: the mels are averaged, so the result is intelligible yet buzzy and lifeless.

Neural parametric & the move to spectrograms

NPSS (Blaauw & Bonada, 2017) — "A Neural Parametric Singing Synthesizer" — replaced the HMM with WaveNet-style autoregressive networks predicting vocoder features, keeping the source-filter decomposition but learning the trajectories. The open NNSVS framework and its ENUNU /OpenUtau bridge made this recipe reproducible for hobbyist voicebanks: a front-end aligner, a neural acoustic model (feed-forward, autoregressive, or diffusion), and a neural vocoder.

DiffSinger — diffusion, with a shallow-diffusion shortcut

DiffSinger (Liu et al., AAAI 2022) casts the acoustic model as a denoising diffusion (Part 8) over the mel-spectrogram, conditioned on the score. A network $\epsilon_\theta$ is trained to predict the noise added to a clean mel $m_0$ at every corruption level $t$, conditioned on $c=\{\text{phoneme},\text{duration},F_0\}$:

$$\mathcal{L} = \mathbb{E}_{t,\,m_0,\,\epsilon}\big\|\epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\,m_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\; t,\; c\big)\big\|^2$$

Pure diffusion starts sampling from Gaussian noise at $t=T$ and denoises through every step — high quality but slow, and the early steps waste effort inventing coarse structure a cheaper model already knows. DiffSinger's shallow diffusion mechanism exploits that: it first runs a simple auxiliary decoder (an FFT/FastSpeech-style regression model) to produce an approximate mel $\tilde m$, then starts the reverse process not from noise but from $\tilde m$ corrupted only to an intermediate step $k \ll T$:

$$m_k = \sqrt{\bar\alpha_k}\,\tilde m + \sqrt{1-\bar\alpha_k}\,\epsilon,\qquad k \ll T,$$

and denoises from $k$ down to $0$. Because $\tilde m$ already has the right global shape, a shallow $k$ suffices — far fewer steps, while the diffusion refines the fine detail (harmonic sharpness, breathiness) that the regression model over-smoothed. The chosen $k$ is set by measuring where the auxiliary output's noise level intersects the diffusion schedule. It is a clean example of the field's recurring trick: let a fast model do the easy part, and spend the expensive generative machinery only where it pays.

The conditioning is the point. Everything that makes singing hard enters DiffSinger as the condition $c$: an explicit $F_0$ contour (so the tune is exact), explicit durations from the score (so the rhythm is exact), and phoneme identity (so the vowels are right). The diffusion only has to render timbre, not decide the music.

Read DiffSinger as the singing counterpart to the diffusion vocoders and TTS of Part 8: the same denoising objective, but conditioned on a score instead of free prosody, and accelerated by starting the walk partway along the corruption schedule.

5Singing voice conversion — and the ethics of cloning

Synthesis builds a voice from a score. Conversion takes an existing performance and re-timbres it: keep the melody, the lyrics, the phrasing of a source recording, but render them in a target singer's voice. The community systems so-vits-svc and RVC made this astonishingly accessible — and, with it, forced a reckoning about consent. The engine is a clean factorisation of what a voice signal actually contains.

Factorise: content · speaker · pitch

The core idea is disentanglement. A content encoder — a self-supervised speech model such as HuBERT or its speaker-scrubbed variant ContentVec — maps the source audio to frame-level units $z$ that encode what is being said/sung (linguistic/phonetic content, sometimes called PPGs or discrete units) while discarding who is saying it. A separate speaker embedding $s$ carries the target identity, and an $F_0$ track — extracted with a pitch estimator — carries the melody. A decoder/vocoder recombines them:

$$\hat x = \mathcal{D}\big(\underbrace{E_c(x_{\text{src}})}_{\text{content } z},\; \underbrace{s_{\text{tgt}}}_{\text{speaker}},\; \underbrace{F_0(x_{\text{src}})}_{\text{pitch}}\big)$$

Training pushes $E_c$ to be speaker-invariant — its output must reconstruct the audio when combined with the correct speaker embedding, but must not itself leak identity. One way to state that objective is a reconstruction loss under an information-bottleneck / adversarial constraint that strips speaker information from the content stream:

$$\min_{E_c,\mathcal{D}}\; \mathbb{E}\,\big\|x - \mathcal{D}(E_c(x),s,F_0)\big\| \;\;\text{s.t.}\;\; I\big(E_c(x);\, s\big)\to 0$$

At inference you simply swap $s_{\text{src}}\!\to\! s_{\text{tgt}}$ (and optionally shift $F_0$ for range). Hover the pipeline to see each stage; toggle between the so-vits-svc and RVC variants to see where they differ.

hover a block ↑

Source audio is factorised into speaker-invariant content, a pitch track, and a target speaker embedding, then re-synthesised by a neural vocoder.

so-vits-svc vs RVC

Both share the content + speaker + $F_0$ recipe and both typically decode with an NSF-HiFiGAN vocoder — a HiFi-GAN (Part 3) fitted with a neural source-filter excitation so it can be explicitly driven by the $F_0$ track, keeping pitch rock-steady. Where they diverge: so-vits-svc (SoftVC + VITS) is a full VITS-style conditional VAE/flow model trained per target voice. RVC — Retrieval-based Voice Conversion — adds a retrieval step: at inference it replaces each content feature with (a blend toward) its nearest neighbours in a FAISS index of the target singer's training features, pulling the timbre closer to the target and suppressing source leakage with far less training data. That retrieval index is the block that appears only in the RVC pipeline above.

Estimating F0: the pitch-tracker lineage

Every one of these systems stands on an $F_0$ estimator, and its accuracy caps the whole result — a wrong octave or a cent of jitter is audible. The lineage runs from autocorrelation and YIN, to its probabilistic successor pYIN (Mauch & Dixon, 2014) with a Viterbi-smoothed pitch track, to the CNN pitch tracker CREPE (Kim et al., 2018) which regresses F0 directly from the waveform, to RMVPE (2023), now the default in RVC for its robustness on polyphonic and noisy singing.

Ethics — consent is not optional. Singing-voice conversion can clone a recognisable artist from minutes of audio, and in 2023 a viral track using cloned pop-star voices ("Heart on My Sleeve") made the stakes concrete. A voice is biometric and identity-bearing: cloning one without the singer's informed consent can violate publicity/likeness rights, mislead listeners, and cause real harm, regardless of how the model was trained. Treat these tools as you would any identity-replicating technology — consent first, disclosure always, and no impersonation. This series demonstrates the mechanics on synthetic voices precisely to keep that line bright.

the maths · why F0 is passed in explicitly, not left to the decoder

If the decoder $\mathcal{D}$ had to infer pitch from content units, any ambiguity in $z$ would smear the melody — the exact failure mode singers cannot tolerate. By extracting $F_0$ separately and feeding it as an explicit conditioning signal (and, in NSF-HiFiGAN, literally as the excitation source's frequency), the model factorises pitch out of the learning problem entirely:

$$p(\hat x \mid x_{\text{src}}, s) = p\big(\hat x \mid z, s, F_0\big),\qquad z \perp F_0 \perp s.$$

The three streams are made as independent as possible so each can be edited alone: transpose by scaling $F_0$, change singer by swapping $s$, change the lyric only by changing the source. That editability — melody, identity and content on separate knobs — is exactly what makes conversion useful, and exactly what makes misuse easy.

6Singing synthesis in one column

The pre-neural era is longer here than in most of the series — singing had commercial concatenative and statistical systems years before deep learning — but the neural acceleration since 2017 is just as steep.

2003–07

VOCALOID ships concatenative singing synthesis; Hatsune Miku (2007) turns a diphone-database voicebank into a cultural phenomenon.

2010

Sinsy brings HMM statistical-parametric singing synthesis; smooth, tiny, but over-smoothed and buzzy. heritage

2017

NPSS (Blaauw & Bonada) — a WaveNet-style neural parametric singing synthesiser — begins the neural era for SVS.

2018–19

CREPE sets a new bar for CNN pitch tracking; the neural source-filter (NSF) model gives vocoders an explicit, controllable F0 excitation. Part 3

2020–21

HiFi-GAN becomes the vocoder standard; NNSVS/ENUNU open up reproducible voicebanks; HuBERT supplies self-supervised content features. Part 3 · 4

2022

DiffSinger (AAAI) brings diffusion + the shallow-diffusion mechanism to SVS; ContentVec scrubs speaker identity from content units; so-vits-svc emerges. Part 8

2023

RVC (retrieval-based conversion) + RMVPE pitch make voice conversion mainstream — and cloned-voice tracks force the consent debate into the open. ethics

→Where to go next

You have shaped a sung vowel from a source and a filter, aligned lyrics to a score, and traced how modern systems factorise a voice into content, identity and pitch. The natural next step is to make that synthesis differentiable end-to-end — folding the very source-filter oscillators you just drove into the training loop itself.

Series hubAll twelve parts · the interactive map Part 4 ←End-to-End Text-to-Speech Part 6 →Differentiable DSP Part 3Vocoders — NSF-HiFiGAN Part 8The diffusion machinery

References

Liu, J. et al. (2022). DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism. AAAI. arXiv:2105.02446.
Blaauw, M. & Bonada, J. (2017). A Neural Parametric Singing Synthesizer (NPSS). Interspeech / Applied Sciences.
Wang, X., Takaki, S., Yamagishi, J. (2019). Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis. IEEE/ACM TASLP.
Kong, J., Kim, J., Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS.
Hsu, W.-N. et al. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction. IEEE/ACM TASLP.
Qian, K. et al. (2022). ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers. ICML.
so-vits-svc contributors (2023). SoftVC VITS Singing Voice Conversion. github.com/svc-develop-team/so-vits-svc.
RVC-Project (2023). Retrieval-based-Voice-Conversion-WebUI. github.com/RVC-Project.
Kim, J.W. et al. (2018). CREPE: A Convolutional Representation for Pitch Estimation. ICASSP.
Mauch, M. & Dixon, S. (2014). pYIN: A Fundamental Frequency Estimator Using Probabilistic Threshold Distributions. ICASSP.