Source-linked AI summary
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian
TL;DR
Large-scale TTS needs to model diverse speakers, prosodies, and styles, while discrete-token autoregressive systems can suffer from instability and degraded acoustic detail. NaturalSpeech 2 uses continuous codec latents, non-autoregressive latent diffusion, and speech prompting, and scales this approach to zero-shot speech and singing synthesis. It reports strong prosody, robustness, fidelity, and voice-quality results, including singing synthesis from only a speech prompt.
Problem
Large-scale TTS must capture diverse speaker identities, prosodies, and styles, but discrete-token autoregressive pipelines suffer from unstable outputs and acoustic information loss.
Method
NaturalSpeech 2 uses a neural audio codec with continuous latent vectors, a non-autoregressive latent diffusion model, and speech prompting for in-context zero-shot synthesis.
Results
NaturalSpeech 2 scales to 400M parameters and 44K hours of speech data, outperforming previous TTS systems and enabling zero-shot singing synthesis from only a speech prompt.
Takeaways & Limitations
The system synthesizes expressive, robust, and faithful speech across diverse speakers, prosodies, and styles in zero-shot scenarios.
Takeaways & Limitations
Real-world deployment should require speaker approval protocols and synthesized-speech detection to address spoofing and impersonation risks.
Abstract
from arXiv · showhide
Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.
1 Introduction
NaturalSpeech 2 addresses the diversity and robustness limits of large-scale TTS by combining continuous latent vectors, non-autoregressive diffusion, and speech prompting. Scaled to 44K hours, it supports strong zero-shot speech and singing synthesis across diverse speakers, prosodies, and styles.
- Motivation: Large-scale TTS is needed to capture diverse speaker identities, prosodies, and styles, but prior token-based systems can produce unstable outputs and lose fine acoustic detail.Autoregressive generation over long discrete-token sequences is vulnerable to error propagation, while low-bitrate codecs can lose high-frequency information.
- System design: NaturalSpeech 2 converts speech waveforms into continuous latent vectors and reconstructs them with a neural audio codec.The codec encoder supplies latent-vector targets for the diffusion model, while the decoder converts generated vectors back into speech.
- Design choices: Continuous vectors reduce sequence length and preserve more fine-grained information than flattened residual-quantizer token sequences.Using eight residual quantizers would make the flattened sequence eight times longer.
- Design choices: Diffusion models generate continuous vectors non-autoregressively, avoiding the error propagation associated with autoregressive acoustic models.This design targets more stable and robust speech generation.
- Design choices: Speech prompting guides the diffusion model and duration/pitch predictors toward prompt characteristics, strengthening zero-shot in-context synthesis.The mechanism is designed to preserve speaker and stylistic information from short speech prompts.
2 Background
The background reviews neural codec and generative-model choices underlying large-scale TTS. It motivates continuous representations and diffusion-based generation as responses to the limitations of discrete-token autoregressive pipelines.
- TTS background: Single-speaker recording-studio datasets do not adequately represent the diversity of speaker identities, prosodies, and speaking or singing styles.Large-scale corpora are therefore used to model broader speech variation and support few-shot or zero-shot generalization.
- Neural audio codecs: Discrete codecs face a bitrate-quality dilemma: short low-bitrate sequences ease language-model generation but can damage waveform reconstruction quality.Multiple tokens per frame improve representation detail while increasing sequence length and prediction difficulty.
- Neural audio codecs: A neural audio codec encodes waveforms into compact representations and decodes those representations back into audio.SoundStream and Encodec use residual vector quantizers to compress speech into multiple discrete tokens.
- Neural audio codecs: Residual vector quantizers create sequences R times longer when R quantizers are used, making discrete-token prediction difficult for language models.Prediction errors can cause word skipping, repetition, or speech collapse during waveform reconstruction.
- Generative models: Autoregressive models are sensitive to sequence length and error propagation, causing unstable prosody and robustness problems such as skipping, repetition, and collapse.NaturalSpeech 2 instead uses diffusion with duration prediction and length expansion for text-to-speech alignment.
3 NaturalSpeech 2
NaturalSpeech 2 combines a neural audio codec with latent diffusion and speech prompting for natural, zero-shot synthesis across diverse speech scenarios. Its design uses continuous quantized vectors, non-autoregressive generation, and prompt-conditioned prediction to address large-scale speech modeling.
- System overview: NaturalSpeech 2 uses a neural audio codec and diffusion model with a phoneme encoder and duration/pitch predictor for natural, zero-shot voice synthesis.The codec provides latent targets, while the prior supplies conditioning information for diffusion.
- Continuous latent representation: The codec encoder extracts frame-level representations, RVQ quantizes them through multiple codebooks, and the decoder reconstructs the waveform from the quantized vectors.The quantized vectors serve as the latent diffusion model’s training targets.
- Continuous latent representation: Continuous vectors reduce sequence-length pressure while preserving more information for high-quality audio reconstruction than discrete-token representations.Each audio frame has one vector rather than multiple discrete tokens; the codec uses RVQ to approximate continuous vectors efficiently.
- Latent diffusion: Diffusion models generate continuous latent vectors non-autoregressively, avoiding the error propagation associated with autoregressive models.Sampling starts from Gaussian noise and reverses the diffusion process using a learned neural network conditioned on text-derived information.
- Speech prompting: Speech prompting trains the duration/pitch predictor and diffusion model to follow speaker characteristics and other diverse information for stronger zero-shot generation.Training masks a random speech-latent segment; inference uses a reference speech latent as the prompt.
- Connection to NaturalSpeech: Compared with NaturalSpeech, NaturalSpeech 2 emphasizes diversity and zero-shot synthesis on large-scale, multi-speaker, in-the-wild data rather than single-speaker studio quality.It retains core encoder, decoder, and prior components while adding diffusion, RVQ, and speech prompting mechanisms.
4 Experimental Settings
NaturalSpeech 2 is evaluated on large-scale speech data using zero-shot synthesis, objective metrics, and human judgments. The experiments compare it with prior zero-shot systems and assess naturalness, similarity, intelligibility, and robustness.
- Datasets: The training data comprise 44K hours of English speech from Multilingual LibriSpeech, with 5,490 distinct speakers.The evaluation uses LibriSpeech test-clean and VCTK, containing 40 and 108 distinct speakers, respectively.
- Model configuration: NaturalSpeech 2 uses a Transformer phoneme encoder, in-context pitch and duration predictors, a speech-prompt encoder, and a WaveNet-based diffusion model.The diffusion model contains 40 WaveNet layers, while the pitch and duration predictors incorporate Q-K-V attention layers.
- Baselines: YourTTS and VALL-E serve as comparison systems for zero-shot synthesis.YourTTS uses VCTK, LibriTTS, and TTS-Portuguese, while VALL-E uses a discrete audio codec and autoregressive language model.
- Training and inference: Training first fits a neural audio codec, then uses its extracted quantized latent vectors as targets for diffusion-model training.The codec is trained for 440K steps, and the diffusion model is trained for 300K steps.
- Training and inference: Inference uses temperature-controlled sampling, an Euler ODE solver, and 150 diffusion steps to balance generation quality and latency.The temperature is set to τ = 1.22.
- Metrics: Evaluation combines objective measures with human judgments of naturalness, speaker similarity, and intelligibility.Objective measures include prompt and ground-truth prosody similarity and WER; subjective measures include intelligibility, CMOS, and SMOS.
5 Results on Natural and Zero-Shot Synthesis
NaturalSpeech 2 is evaluated across naturalness, prompt similarity, speaker similarity, robustness, and ablations. It consistently improves over zero-shot baselines, while speech prompting and auxiliary design choices materially affect performance.
- Generation Quality: NaturalSpeech 2 achieves CMOS gains of 0.65 on LibriSpeech and 0.58 on VCTK over YourTTS.It is comparable to ground-truth quality on LibriSpeech and has much better quality than ground truth on VCTK under the reported CMOS comparison.
- Generation Similarity: NaturalSpeech 2 consistently outperforms YourTTS on all prompt-prosody metrics across LibriSpeech and VCTK.This remains true even though YourTTS has seen 97 of VCTK’s 108 speakers during training.
- Generation Similarity: NaturalSpeech 2 improves speaker similarity over YourTTS by 1.25 SMOS points on LibriSpeech and 0.77 on VCTK.The evaluation uses 3-second speech prompts.
- Robustness: NaturalSpeech 2 significantly outperforms YourTTS in WER on LibriSpeech and VCTK, with performance comparable to ground truth on LibriSpeech and higher than ground truth on VCTK.The authors attribute the higher VCTK WER partly to noisy conditions and limited ASR fine-tuning for that dataset.
- Robustness: NaturalSpeech 2 and other non-autoregressive systems remain robust on 50 particularly hard sentences without intelligibility issues.The test targets repeating words, skipping words, and error sentences associated with autoregressive alignment mismatch.
- Comparison with Other TTS Systems: Against VALL-E, NaturalSpeech 2 gains 0.3 SMOS and 0.31 CMOS.The reported gains indicate higher speaker similarity and naturalness in this comparison.
- Ablation Studies: Removing diffusion-model speech prompting worsens prosody similarity from 10.11 to 21.69 for pitch mean or prevents convergence.Removing the RVQ cross-entropy loss or query-attention strategy also degrades performance.
- Ablation Studies: Longer speech prompts improve similarity between generated and prompt speech under the reported prompt-length experiment.The study varies prompt length across 3, 5, and 10 seconds.
5.6 Zero-Shot Singing Synthesis
NaturalSpeech 2 is evaluated for zero-shot singing synthesis using speech and singing data, including the ability to generate singing voices from a speech prompt. The same framework also supports zero-shot voice conversion and speech enhancement while preserving selected speech characteristics.
- Zero-Shot Singing Synthesis: Around 30 hours of singing data are crawled, cleaned, and mixed with speech data for singing experiments.Backing vocals and accompaniment are removed, while ASR filters samples with lyric-audio misalignments.
- Zero-Shot Singing Synthesis: NaturalSpeech 2 synthesizes singing voices with different singer timbres using singing prompts and can generate a novel singing voice from a speech prompt.Inference uses another singing voice’s ground-truth pitch and duration, with 1000 diffusion steps for better performance.
- Voice Conversion: Zero-shot voice conversion starts from source-aware diffusion output rather than random Gaussian noise, then performs target-aware denoising conditioned on source content and a voice prompt.The source-aware process retains source information, while denoising uses phonemes, duration, predicted pitch, and the prompt audio.
- Voice Conversion: NaturalSpeech 2 produces speech with source-like prosody while replicating the prompt-specified timbre in zero-shot voice conversion.This connects source-content preservation with target-speaker characteristics.
- Speech Enhancement: In speech enhancement, the model uses noisy source and prompt audio for source-aware diffusion, then a clean prompt for target-aware denoising to remove background noise.The procedure uses source phonemes, duration, and pitch while preserving prosody and other crucial speech aspects.
6 Conclusion and Future Work
NaturalSpeech 2 combines continuous latent vectors, latent diffusion, and speech prompting to synthesize expressive, robust, high-fidelity speech with strong zero-shot ability. The paper also identifies voice-identity misuse risks and proposes consent and detection measures for real-world deployment.
- Conclusion: NaturalSpeech 2 uses continuous latent vectors and non-autoregressive latent diffusion for natural and zero-shot text-to-speech synthesis.Speech prompting is added to the duration/pitch predictor and diffusion model to facilitate in-context learning.
- Conclusion: 400M parameters, 44K hours of speech, and 5K speakers support synthesis with high expressiveness, robustness, fidelity, and strong zero-shot ability.The paper reports that NaturalSpeech 2 outperforms previous TTS systems.
- Future Work: Future work will explore consistency models for faster diffusion and larger-scale speaking-and-singing training for mixed speaking/singing capability.These directions are stated as plans for improving efficiency and mixed vocal-style modeling.
- Broader Impacts: Maintaining speaker identity creates potential misuse risks including spoofing voice identification and impersonating a specific speaker.The paper recommends speaker approval protocols and synthesized-speech detection for generalized real-world use.
A Model Details
Table 11 provides the detailed model configurations of NaturalSpeech 2.
- Model Details: Table 11 contains the detailed model configurations of NaturalSpeech 2.It is presented as the model-details reference for the system.
- Model Details: The table is part of the appendix’s model-details material for NaturalSpeech 2.Its scope is the system’s configuration rather than an experiment result.
- Model Details: Use Table 11 to locate the configuration specifications associated with NaturalSpeech 2.The passage identifies the table’s purpose but does not state individual parameter values.
B The Details of WaveNet Architecture in the Diffusion Model
The diffusion model uses a 40-block WaveNet architecture combining dilated convolutions, Q-K-V attention, and FiLM conditioning. Attention incorporates information from the speech-prompt encoder to generate FiLM scale and bias terms.
- WaveNet Architecture: The WaveNet in the diffusion model consists of 40 blocks.Each block contains a dilated CNN, Q-K-V attention, and a FiLM layer.
- WaveNet Architecture: Each block uses a kernel-3 dilated CNN with dilation 2, followed by Q-K-V attention and FiLM conditioning.These components define the repeated architecture shown in Figure 4.
- WaveNet Architecture: Q-K-V attention attends to key/value representations from the speech-prompt encoder’s first attention module.The resulting attention outputs generate the FiLM scale and bias terms used as conditional information.
C The 50 Particularly Hard Sentences
This section lists 50 particularly hard sentences used to evaluate generation quality, including short tokens, repetitions, codes, technical text, and announcements.
- 50 particularly hard sentences are listed for evaluation.
- The list includes short alphabetic and numeric items, repeated phrases, and sequences containing pass, fail, cancellation, and totals.
- It also includes hexadecimal-like strings, Boolean values, system errors, process commands, and database connector references.
- The final entries include registry and employment-ID instructions, a corporate partnership announcement, and an entertainment-news sentence.
D Prosody Similarity with Ground Truth
The study evaluates prosody similarity between synthesized and ground-truth speech using pitch and duration correlation and RMSE, finding a large-margin advantage for NaturalSpeech 2 over YourTTS.
- Prosody similarity is measured with Pearson correlation and RMSE for pitch and duration.The comparison uses synthesized speech against ground-truth speech.
- NaturalSpeech 2 outperforms the YourTTS baseline by a large margin in prosody similarity.The authors report this as evidence that NaturalSpeech 2 better matches ground-truth prosody.
E Ablation Study
The ablation study evaluates prosody similarity against ground-truth speech and identifies the speech prompt as the most important component, with cross-entropy and query attention also helping.
- The ablation study compares synthesized and ground-truth prosody using pitch and duration correlation and RMSE.A dash indicates that a model failed to converge.
- The speech prompt is the most important factor for generation quality in the ablation results.
- Cross-entropy and query attention also help achieve high-quality speech synthesis.