NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

1 Introduction

• NaturalSpeech 2 encodes speech into latent vectors, then uses a phoneme-conditioned latent diffusion model with duration and pitch predictors to generate vectors decoded into speech.At inference, the diffusion model generates latent vectors from text or phonemes, and the codec decoder reconstructs the waveform.Source: §1 · p. 2
• Continuous vectors avoid the long flattened sequences created by residual quantizers, shortening acoustic-model sequences while retaining fine-grained information for speech reconstruction.The introduction notes that eight residual quantizers per frame would make the flattened token sequence eight times longer.Source: §1 · p. 2
• Non-autoregressive diffusion models learn complex continuous-vector distributions without the error propagation that can destabilize autoregressive speech generation.The paper frames this as an alternative to large-scale systems that quantize speech into discrete tokens and model them autoregressively [13] [14] [15].Source: §1 · pp. 2–3
• Speech prompting encourages the diffusion model and pitch and duration predictors to follow prompt characteristics, facilitating in-context learning and strengthening zero-shot synthesis.The approach is designed to reproduce characteristics of the prompt in generated speech.Source: §1 · p. 3
• Together, non-autoregressive generation and duration and pitch prediction support greater stability, a single diffusion acoustic model instead of two-stage token prediction, and styles such as singing.The introduction contrasts this design with two-stage token prediction [21] [13].Source: §1 · p. 3
• At 400M parameters and 44K hours of training data, NaturalSpeech 2 generates natural zero-shot speech, with CMOS naturalness comparable to or better than ground truth on LibriSpeech and VCTK.It also reports more similar prosody to the prompt and ground-truth speech, and zero-shot singing synthesis from either a singing prompt or only a speech prompt.Source: §1 · p. 3

2.1 TTS for Natural Voice: Quality and Diversity

• Studio-trained neural TTS can reach human-level quality (NaturalSpeech [11]) but lacks speaker, prosody, and style diversity, motivating large-scale, in-the-wild multi-speaker systems [13] [14] [15].Single-token VQ eases generation but can reduce reconstruction quality, whereas multi-token RVQ improves fidelity but lengthens sequences; AudioLM [21] uses a complex two-stage language-model approach that can incur cascaded errors.Source: §2.1 · p. 3

2.2 Neural Audio Codec

• Residual vector quantizers support low-bitrate, high-fidelity audio compression, but their long discrete sequences can cause prediction errors, word skipping, repetition, or speech collapse.NaturalSpeech 2 instead uses a codec that maps speech to continuous vectors, preserving fine-grained reconstruction detail without increasing sequence length; a neural codec encodes and decodes compact audio representations.Source: §2.2 · pp. 3–4

2.3 Generative Models for Speech Synthesis

• Autoregressive models are sensitive to sequence length and error propagation, so NaturalSpeech 2 uses diffusion with duration prediction and length expansion to address robustness issues in monotonic text-to-speech alignment.Autoregressive and diffusion models have both been used for speech synthesis, including waveform and acoustic-feature generation.Source: §2.3 · p. 4

3.1 Neural Audio Codec with Continuous Vectors

• The neural audio codec uses residual vector quantization to convert frame-level speech representations into quantized vectors that serve as diffusion training targets.Each frame corresponds to a 12.5 ms segment at 16 kHz; many quantizers and codebook entries approximate continuous vectors while enabling compact storage and discrete-classification regularization.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Latent Diffusion Model with Non-Autoregressive Generation

• The diffusion model predicts codec latents from text-conditioned features, using phoneme encoding and duration/pitch prediction to form frame-level conditioning.A WaveNet-based network predicts clean latents from noisy inputs, and training adds an RVQ cross-entropy loss to data and score losses.Source: §3.2 · pp. 5–6

3.3 Speech Prompting for In-Context Learning

• Speech prompting trains the duration/pitch predictor and diffusion model to follow prompt information, including speaker identity, for in-context zero-shot generation.A Transformer prompt encoder processes a randomly selected speech segment; the predictors attend to its hidden sequence, while the diffusion model compresses prompt information through two attention blocks before FiLM conditioning.Source: §3.3 · p. 7, Figure 3 · p. 7

3.4 Connection to NaturalSpeech

• Compared with NaturalSpeech [11], NaturalSpeech 2 targets diversity and zero-shot synthesis on large, multi-speaker, in-the-wild datasets, adding diffusion, residual vector quantization, and speech prompting.NaturalSpeech [11] focused on speech quality using single-speaker studio data; NaturalSpeech 2 retains waveform encoder/decoder and prior components while adding mechanisms for complex data distributions and zero-shot capability.Source: §3.4 · p. 7

4.1 Datasets

• Training uses 44K hours of 16 kHz MLS English audiobook speech from 5,490 speakers, with text converted to phonemes, aligned for duration, and analyzed for pitch.The training set contains 2,742 male and 2,748 female speakers.Source: §4.1 · p. 8
• Evaluation uses 600 LibriSpeech test-clean utterances and 540 VCTK utterances from speakers unseen in training, each synthesized using a different same-speaker utterance as a σ-second prompt.The samples are drawn from 40 LibriSpeech speakers and 108 VCTK speakers.Source: §4.1 · p. 8

4.2 Model Configuration and Comparison

• The system uses a 6-layer phoneme Transformer, convolutional duration and pitch predictors with attention for in-context learning, a 6-layer prompt encoder, and a 40-layer WaveNet diffusion model conditioned through FiLM.The diffusion model uses 32 prompt query tokens; the passage specifies 512-dimensional hidden representations for the encoder, predictors’ attention, and WaveNet.Source: §4.2 · p. 8
• Comparisons use YourTTS with its official code and pretrained checkpoint, plus VALL-E samples collected from its demo page.YourTTS was trained on VCTK, LibriTTS, and TTS-Portuguese; VALL-E represents a discrete-codec autoregressive baseline.Source: §4.2 · p. 8

4.3 Model Training and Inference

• The audio codec is trained for 440K steps on 8 V100 GPUs, then supplies quantized latents for diffusion-model training, which runs for 300K steps on 16 V100 GPUs.The authors note that the diffusion model is still underfitting and longer training may improve performance; inference uses temperature 1.22 and 150 Euler ODE diffusion steps.Source: §4.3 · p. 9

4.4 Evaluation Metrics

• Objective evaluation measures prompt prosody similarity, ground-truth prosody correlation and RMSE, and WER from a HuBERT-based ASR system.Prompt prosody compares distribution statistics of pitch and duration; WER uses a CTC HuBERT model pretrained on Libri-light and fine-tuned on LibriSpeech’s 960-hour training set.Source: §4.4 · p. 9
• Human evaluation measures robustness on 50 hard sentences by counting repeated words, skipped words, and error sentences, and rates naturalness and speaker similarity with CMOS and SMOS.CMOS uses 12 native-speaker judges, while SMOS uses 6 native-speaker judges to compare synthesized speech with the prompt.Source: §4.4 · p. 9

5.1 Generation Quality

• NaturalSpeech 2 gains 0.65 CMOS on LibriSpeech and 0.58 on VCTK over YourTTS, and is comparable to ground truth on LibriSpeech.The paper reports a +0.04 difference as on par with ground truth and a −0.30 gap on VCTK.Source: §5.1 · p. 10, Table 3 · p. 10

5.2 Generation Similarity

• NaturalSpeech 2 outperforms YourTTS on every reported pitch and duration prosody-similarity metric for LibriSpeech and VCTK.This also holds on VCTK speakers seen by YourTTS but unseen by NaturalSpeech 2.Source: §5.2 · p. 10, Table 4 · p. 10
• NaturalSpeech 2 exceeds YourTTS by 1.25 SMOS on LibriSpeech and 0.77 on VCTK.The paper concludes that NaturalSpeech 2 is significantly better in speaker similarity.Source: Table 5 · p. 10, §5.2 · p. 11

5.3 Robustness

• NaturalSpeech 2 significantly outperforms YourTTS in WER on LibriSpeech and VCTK, with results comparable to ground truth on LibriSpeech and better on VCTK.The paper suggests VCTK’s higher WER may reflect noisy recording conditions and the lack of ASR fine-tuning for that dataset.Source: §5.3 · p. 11, Table 6 · p. 11
• NaturalSpeech 2 and other non-autoregressive models have no intelligibility issues on 50 hard sentences, unlike autoregressive systems with high error rates.The comparison includes FastSpeech [6], NaturalSpeech [11], Tacotron [3], Transformer TTS [5], and VALL-E [13].Source: §5.3 · p. 11, Table 7 · p. 11

5.4 Comparison with Other TTS Systems

• NaturalSpeech 2 outperforms VALL-E by 0.3 in SMOS and 0.31 in CMOS.The paper interprets these results as better speaker similarity, naturalness, and audio quality.Source: §5.4 · p. 11, Table 8 · p. 11

5.5 Ablation Study

• Ablations show that removing the diffusion prompt worsens pitch-mean similarity from 10.11 to 21.69 or prevents convergence, while removing CE loss or query attention also degrades similarity.The paper attributes CE-loss benefits to regularization of RVQ latent representations and notes that direct cross-attention to prompt hidden states can leak details and mislead generation.Source: §5.5 · p. 12, Table 9 · p. 12
• Longer prompts improve NaturalSpeech 2’s prosody similarity to the prompt.The tested prompt lengths are σ = 3, 5, and 10 seconds; the paper attributes the improvement to longer prompts revealing more prosody details.Source: §5.5 · p. 12

5.6 Zero-Shot Singing Synthesis

• The singing dataset contains around 30 hours of data, collected from web songs and paired lyrics, processed to remove accompaniment and filter misaligned samples, then upsampled and mixed with speech data.Source: §5.6 · p. 12
• NaturalSpeech 2 can generate a novel singing voice using a speech prompt, and singing prompts can specify different singer timbres.Synthesis uses ground-truth pitch and duration from another singing voice.Source: §5.6 · p. 12

5.7 Extension to Voice Conversion and Speech Enhancement

• NaturalSpeech 2 extends beyond text-to-speech and singing synthesis to zero-shot voice conversion and speech enhancement.Source: §5.7 · p. 13

5.7.1 Voice Conversion

• Voice conversion diffuses source audio to an informative starting point, then denoises it into target audio conditioned on source phonemes, duration, predicted pitch, and the prompt.The starting point retains some source information rather than directly adding Gaussian noise to the source.Source: §5.7.1 · p. 13
• NaturalSpeech 2 produces speech with prosody similar to the source while replicating the timbre specified by the prompt in zero-shot voice conversion.Source: §5.7.1 · p. 13

5.7.2 Speech Enhancement

• Speech enhancement assumes noisy source and prompt audio share background noise, uses the noisy prompt for source-aware diffusion, and uses a clean prompt for target-aware denoising.The procedure uses source duration and pitch during diffusion, then source phonemes, duration, and pitch during denoising.Source: §5.7.2 · p. 14
• NaturalSpeech 2 can eliminate background noise while preserving prosody and timbre in speech enhancement.The method uses source-aware diffusion followed by target-aware denoising with a clean prompt.Source: §5.7.2 · p. 14

Conclusion

• At 400M parameters, 44K hours of speech, and 5K speakers, NaturalSpeech 2 delivers expressive, robust, faithful speech with strong zero-shot ability, outperforming previous TTS systems.The system uses continuous latent vectors from a neural audio codec, non-autoregressive latent diffusion, and speech prompting in the duration/pitch predictor and diffusion model to facilitate zero-shot in-context learning.Source: §6 · p. 14

Future Work

• Future work will explore consistency models [52] [53] to speed diffusion and large-scale speaking and singing voice training for more powerful mixed speaking/singing.Source: §6 · p. 14

Broader Impacts

• Because NaturalSpeech 2 can maintain speaker identity, misuse could enable voice-identification spoofing or speaker impersonation; real-world use should require speaker approval and synthesized-speech detection.The experiments assumed users agreed to be the target speaker; the proposed safeguards apply if the model is generalized to unseen speakers in the real world.Source: §6 · p. 14

• Table 11 presents detailed model configurations for NaturalSpeech 2.Source: Table 11 · p. 19

B The Details of WaveNet Architecture in the Diffusion Model

• The diffusion-model WaveNet has 40 blocks, each combining a kernel-size-3 dilated CNN, Q-K-V attention, and a FiLM layer.The architecture is shown in Figure 4.Source: B The Details of WaveNet Architecture… · p. 19, Figure 4 · p. 20
• Each block attends to keys and values from the speech-prompt encoder’s first Q-K-V attention module, then uses attention results to generate FiLM scale and bias terms.Source: B The Details of WaveNet Architecture… · p. 19
• The final WaveNet output is calculated by averaging the skip outputs from each layer.Source: B The Details of WaveNet Architecture… · p. 19

C The 50 Particularly Hard Sentences

• The opening examples include isolated tokens, numeric sequences, status strings, error messages, and technical identifiers.They include hexadecimal and UUID-like strings, debugging labels, database connector names, standards, and configuration values.Source: C The 50 Particularly Hard Sentences · pp. 20–21
• Later examples combine filenames, deployment-related text, a phone-number-heavy request, and a log error with longer paths and URLs.The list also includes a sentence about the distribution of Xbox 360 consoles.Source: C The 50 Particularly Hard Sentences · p. 21
• The final examples span product specifications, a technical explanation, forum-style exchanges, an operational description, a press release, and a news caption.They include an explanation of Extended MAPI and C++, an employment-ID registry process, a Microsoft–Shell announcement, and a Cannes screening caption.Source: C The 50 Particularly Hard Sentences · pp. 21–22

D Prosody Similarity with Ground Truth

• NaturalSpeech 2 outperforms YourTTS by a large margin in prosody matching against ground-truth speech, evaluated with Pearson correlation and RMSE for pitch and duration.Source: Table 12 · p. 22, D Prosody Similarity with Ground Truth · p. 22

E Ablation Study

• The speech prompt is the most important component for generation quality in the ablation study.The study compares prosody similarity between ablation-model audio and ground-truth speech in correlation and RMSE for pitch and duration (Table 13).Source: E Ablation Study · p. 22, Table 13 · p. 22
• Cross-entropy and query attention also help produce high-quality speech in the ablation study.The reported comparison concerns prosody similarity between synthesized and ground-truth speech.Source: E Ablation Study · p. 22, Table 13 · p. 22