Source-linked AI summary
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao
TL;DR
Large-scale zero-shot TTS still struggles with speech quality, similarity, and prosody because speech combines complex attributes. NaturalSpeech 3 factorizes those attributes in a neural codec and generates them with prompted factorized diffusion, achieving state-of-the-art results across quality, similarity, prosody, and intelligibility, including human-level performance on multi-speaker LibriSpeech.
Problem
Large-scale zero-shot TTS remains limited in voice quality, similarity, and prosody, while speech’s intertwined attributes complicate generation.
Method
NaturalSpeech 3 uses FACodec factorized vector quantization and factorized diffusion to represent and generate speech attributes from corresponding prompts.
Results
NaturalSpeech 3 outperforms state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, with human-level performance on multi-speaker LibriSpeech.
Takeaways & Limitations
Factorizing speech attributes enables attribute manipulation through customized prompts and supports improved performance when scaling to 1B parameters and 200K hours of training data.
Takeaways & Limitations
The factorized attributes do not cover all speech aspects, including background sounds, and the model carries misuse risks such as voice spoofing or impersonation.
Abstract
from arXiv · showhide
While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech in a zero-shot way. Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt. With this factorization design, NaturalSpeech 3 can effectively and efficiently model intricate speech with disentangled subspaces in a divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, and achieves on-par quality with human recordings. Furthermore, we achieve better performance by scaling to 1B parameters and 200K hours of training data.
1 Introduction
NaturalSpeech 3 addresses persistent zero-shot TTS shortcomings by factorizing speech attributes and generating them with corresponding prompts. The system reports gains across quality, similarity, prosody, and intelligibility.
- Large-scale zero-shot TTS systems still produce unsatisfactory voice quality, similarity, and prosody despite expanded data and model sizes.
- Speech generation is difficult because content, prosody, timbre, and acoustic detail are intricately embedded in speech.
- NaturalSpeech 3 uses FACodec with factorized vector quantization to decompose speech into distinct attribute subspaces.
- Its factorized diffusion model generates duration, content, prosody, and acoustic detail from corresponding prompts, enabling attribute-level control.
- 0.64 → 0.67 on Sim-O and 3.69 → 4.01 on SMOS mark improved similarity, alongside gains in prosody and intelligibility.The reported prosody changes include −0.16 average MCD and +0.21 SMOS, while WER improves from 1.94 to 1.81.
2 Background
The background organizes zero-shot TTS by representation and modeling strategy, reviews speech representations and generation paradigms, and motivates attribute disentanglement. It positions NaturalSpeech 3 as combining attribute-based factorization with factorized diffusion.
- Zero-shot TTS: Zero-shot TTS synthesizes speech for unseen speakers using speech prompts.
- Zero-shot TTS: Zero-shot TTS systems are categorized by discrete or continuous representations and autoregressive or non-autoregressive modeling.
- Speech Attribute Disentanglement: NaturalSpeech 3 extends these directions by disentangling speech waveforms into attribute subspaces and generating them with a factorized diffusion model.
- Speech Representations in TTS: Prior raw-waveform, mel-spectrogram, and data-driven representations do not explicitly address speech’s multiple complex attributes.
- Generation Methods in TTS: Non-autoregressive models offer robustness and generation speed, whereas autoregressive models provide greater diversity, prosody, expressiveness, and flexibility.
- Speech Attribute Disentanglement: Earlier disentanglement methods separate attributes such as content, fundamental frequency, or timbre, but reported speech quality remains unsatisfying.
3 NaturalSpeech 3
NaturalSpeech 3 factorizes speech into disentangled attribute subspaces with FACodec and generates duration, prosody, content, and acoustic details through factorized diffusion. The design targets more efficient zero-shot speech modeling while preserving controllable attribute generation.
- Overall Architecture: NaturalSpeech 3 uses FACodec to represent speech as disentangled content, prosody, timbre, and acoustic-detail subspaces.FACodec combines a speech encoder, timbre extractor, three factorized vector quantizers, and a decoder that reconstructs the waveform.
- Attribute Disentanglement: FACodec promotes disentanglement through information bottlenecks, auxiliary supervision, gradient reversal, and detail dropout.The bottleneck limits code capacity; supervision targets pitch, phonemes, and speaker identity, while gradient reversal removes undesired information from latent spaces.
- Factorized Diffusion: The factorized diffusion model sequentially generates duration, prosody, content, and acoustic details from corresponding prompts and conditions.Duration generation is followed by length regulation to produce frame-level phoneme conditioning, after which the remaining attributes are generated with shared discrete-diffusion formulations.
- Diffusion Formulation: The diffusion forward process progressively masks target discrete tokens, while the reverse model predicts masked tokens conditioned on prompts and other conditions.Inference begins from a fully masked sequence and iteratively remasks low-confidence predictions; classifier-free guidance can steer conditional generation.
4 Experiments and Results
Experiments evaluate NaturalSpeech 3 across quality, speaker and prosody similarity, robustness, ablations, codec reconstruction, extensibility, and scaling. The results consistently support factorization, with human-level or competitive performance and further gains from larger models and datasets.
- Generation Quality: NaturalSpeech 3 nearly matches ground-truth speech quality and substantially outperforms baselines in CMOS tests on LibriSpeech test-clean and RAVDESS.It scores −0.08 on LibriSpeech test-clean and −0.17 on RAVDESS relative to ground truth.
- Robustness: NaturalSpeech 3 achieves high intelligibility and robustness, with WER better than ground truth and close to human recordings.The authors report considerable margins over other baselines on the LibriSpeech test-clean benchmark.
- Ablation Study: Removing factorization causes degradation in similarity, robustness, and quality, while removing classifier-free guidance also reduces similarity and quality.Without factorization, Sim-O drops 0.12 and CMOS drops 0.25; without guidance, Sim-O drops 0.03 and CMOS drops 0.06.
- Codec and Scaling: FACodec improves reconstruction over SoundStream at the same bandwidth, and scaling the model from 500M to 1B parameters improves Sim-O by 0.05 and WER by 0.40.FACodec gains 0.44 PESQ, 0.05 STOI, 0.14 MSTFT, and 0.79 MCD over SoundStream; the model-scaling comparison uses 200K hours of data.
5 Conclusion
NaturalSpeech 3 combines FACodec with factorized diffusion to generate speech attributes in disentangled subspaces. It outperforms state-of-the-art systems, reaches human-level performance on LibriSpeech, and improves with larger models and datasets.
- 5 Conclusion: NaturalSpeech 3 combines FACodec for attribute decomposition with factorized diffusion for discrete subspace generation.The system factorizes speech into content, prosody, acoustic details, and timbre.
- 5 Conclusion: NaturalSpeech 3 outperforms state-of-the-art TTS systems on speech quality, similarity, prosody, and intelligibility.
- 5 Conclusion: NaturalSpeech 3 achieves human-level performance on multi-speaker LibriSpeech and improves when scaled to 1B parameters and 200K hours of training data.
6 Boarder Impact
The paper identifies misuse risks from high speaker similarity, including voice-identification spoofing and speaker impersonation. It assumes user consent and recommends detection and reporting mechanisms.
- 6 Boarder Impact: High speaker similarity creates potential misuse risks, including spoofing voice identification and impersonating a specific speaker.
- 6 Boarder Impact: The experiments assume that users agree to be the target speaker in speech synthesis.
- 6 Boarder Impact: The paper recommends robust synthesized-speech detection and reporting systems to help prevent misuse.
A.2 Training and Inference Details
Training uses 60K hours of unlabeled Librilight speech from around 7000 speakers with phoneme and duration preprocessing. Inference runs four iterations per diffusion process, totaling 60 forward passes.
- A.2 Training and Inference Details: Training uses Librilight, containing 60K hours of 16KHz unlabeled speech and around 7000 distinct speakers.Transcriptions, phonemes, and durations are obtained with internal ASR, grapheme-to-phoneme conversion, and alignment tools.
- A.2 Training and Inference Details: Training runs for 1M steps on 8 A100 80GB GPUs with 10K latent-vector frames per GPU per batch.The optimizer is AdamW with a learning rate of 1e-4 and β1 = 0.9.
- A.2 Training and Inference Details: Inference performs four iterations in each diffusion process and totals 60 forward passes because classifier-free guidance doubles computation for several processes.
A.3 Evaluation Baselines
NaturalSpeech 3 is evaluated against strong zero-shot TTS baselines spanning discrete-token and continuous-vector generation with autoregressive or non-autoregressive models. Evaluation includes LibriSpeech test-clean results for WER and UTMOS.
- A.3 Evaluation Baselines: Baseline results come from papers, authors, official checkpoints, or reproduced experiments, with HierSpeech++ evaluated without its super-resolution model.
- A.3 Evaluation Baselines: The evaluation reports WER inferred by an advanced ASR system and UTMOS, a surrogate objective metric of MOS.Results are presented on the LibriSpeech test-clean set.
A.5 Latency Analysis
NaturalSpeech 3 is evaluated for inference latency against autoregressive and non-autoregressive baselines, including a reduced-step diffusion variant.
- 15.27× speedup over VALL-E and 1.24× over NaturalSpeech 2, while surpassing both baselines on all metrics.The evaluation uses speaker similarity and quality metrics on LibriSpeech test-clean.
- 4.41× faster inference with one diffusion step while maintaining robust performance, with changes of −0.01 Sim-O, −0.01 Sim-R, and −0.29 UTMOS.
- NaturalSpeech 3 is reported as both effective and efficient relative to VALL-E and NaturalSpeech 2.
A.6 Ablation Study on Duration Diffusion Model
The duration diffusion ablation examines generation, objective, conditioning, and prompting choices, while additional evaluations assess prosody and codec reconstruction. The reported ablations support the contribution of these design choices to performance and robust prosody similarity.
- Duration predictor ablation: Four duration-predictor factors are ablated: multi-step versus one-step generation, cross-entropy versus L2 objective, phoneme-level prosody conditioning, and duration prompting.The evaluation covers speaker similarity, robustness, and quality on LibriSpeech test-clean.
- Duration predictor ablation: Each design aspect of the duration predictor contributes to performance improvement.
- Prosody evaluation: NaturalSpeech 3 demonstrates robust performance across eight emotions on RAVDESS, verifying effectiveness and robustness for prosody similarity.The comparison uses MCD scores against baseline methods.
B.4 Ablation Study
The ablation study tests information bottlenecks, gradient reversal, acoustic-detail quantizers, and prosody representations as components of FACodec disentanglement and reconstruction.
- Information bottleneck: Without an information bottleneck, FACodec can exhibit incomplete disentanglement in zero-shot voice conversion.The bottleneck is contrasted with quantization in the original rather than a low-dimensional space.
- Gradient reversal: Removing gradient reversal allows content and prosody information to leak into the acoustic-detail representation.This is observed when reconstructing speech using detail codes and timbre embedding.
- Acoustic details quantizer: Acoustic-detail quantizers improve FACodec speech reconstruction quality, although content, prosody, and timbre already contain most speech information.
C Limitation and Future Works
The paper identifies limitations in attribute coverage, training-data coverage, and FACodec supervision and task validation. Future work targets additional attributes, more diverse data, unsupervised disentanglement, and broader codec applications.
- Attribute Coverage: The factorization does not cover all speech aspects, including background sounds, and future work will explore energy and background-sound attributes.
- Data Coverage: Training on English LibriVox audiobook speech limits coverage of real-world voice diversity and multilingual TTS.The authors propose collecting speech data with larger diversity.
- Neural Speech Codec: FACodec requires phoneme transcription for content supervision and has only been verified for disentanglement in zero-shot TTS.Future work includes unsupervised disentanglement and tasks such as zero-shot voice conversion and automatic speech recognition.