Source-linked AI summary
NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Frank Soong, Tao Qin, Sheng Zhao, Tie-Yan Liu
TL;DR
The paper asks how to define, judge, and achieve human-level quality in TTS, where current systems still differ from human recordings. It introduces NaturalSpeech, a VAE-based end-to-end text-to-waveform system with modules that strengthen the text prior and simplify the speech posterior. On LJSpeech, it reports −0.01 CMOS versus recordings with p ≫ 0.05, meeting its test-set criterion for human-level quality.
Problem
TTS systems have advanced but still show a quality gap from human recordings, leaving human-level quality insufficiently defined, judged, and achieved.
Method
NaturalSpeech uses a VAE for end-to-end text-to-waveform generation, with phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory mechanism.
Results
−0.01 CMOS versus human recordings on LJSpeech, with p ≫ 0.05 under a Wilcoxon signed rank test, indicating no statistically significant difference.
Takeaways & Limitations
NaturalSpeech achieves the paper’s statistical definition of human-level quality on the evaluated LJSpeech test set.
Takeaways & Limitations
The human-level claim is limited to statistical indistinguishability from human recordings on the tested dataset and does not imply replacing or surpassing humans.
Abstract
from arXiv · showhide
Text to speech (TTS) has made rapid progress in both academia and industry in recent years. Some questions naturally arise that whether a TTS system can achieve human-level quality, how to define/judge that quality and how to achieve it. In this paper, we answer these questions by first defining the human-level quality based on the statistical significance of subjective measure and introducing appropriate guidelines to judge it, and then developing a TTS system called NaturalSpeech that achieves human-level quality on a benchmark dataset. Specifically, we leverage a variational autoencoder (VAE) for end-to-end text to waveform generation, with several key modules to enhance the capacity of the prior from text and reduce the complexity of the posterior from speech, including phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory mechanism in VAE. Experiment evaluations on popular LJSpeech dataset show that our proposed NaturalSpeech achieves -0.01 CMOS (comparative mean opinion score) to human recordings at the sentence level, with Wilcoxon signed rank test at p-level p >> 0.05, which demonstrates no statistically significant difference from human recordings for the first time on this dataset.
1 Introduction
The paper frames human-level TTS as a measurable goal and develops NaturalSpeech, a fully end-to-end text-to-waveform system designed to close the gap with human recordings. On LJSpeech, NaturalSpeech reaches statistically indistinguishable quality from human recordings under the reported CMOS evaluation.
- Motivation: Human-level TTS requires defining, judging, and engineering quality relative to human recordings.The paper addresses these questions through a statistical definition, evaluation guidelines, and the NaturalSpeech system.
- System: NaturalSpeech uses a VAE to generate waveforms end-to-end by matching a text-conditioned prior with a speech-derived posterior.Speech is compressed into frame-level latent representations, while text produces the prior used for waveform synthesis.
- System: Phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory-based VAE enhance representations and simplify latent generation.These modules improve the prior from text and reduce posterior complexity from speech.
- Design advantages: NaturalSpeech reduces training-inference mismatch and alleviates text-to-speech one-to-many mapping through end-to-end generation, differentiable duration modeling, memory, and bidirectional prior/posterior modules.The system also targets representation capacity limitations in earlier TTS models.
- Evaluation: −0.01 CMOS versus human recordings was achieved on LJSpeech, with p ≫ 0.05 under a Wilcoxon signed rank test.The result indicates no statistically significant difference from the recordings in the reported evaluation.
2 Definition and Judgement of Human-Level Quality in TTS
The paper defines human-level TTS statistically and specifies subjective side-by-side evaluation guidelines for determining whether generated speech differs from human recordings. It applies these guidelines to prior systems on LJSpeech, while clarifying that statistical indistinguishability is test-set specific.
- Definition: Human-level quality means no statistically significant difference between generated-speech and corresponding human-recording quality scores on a test set.This is the paper’s formal operational definition.
- Definition: The paper explicitly distinguishes test-set human-level quality from surpassing or replacing human speakers.The claim is limited to statistical indistinguishability from human recordings on that test set.
- Judgement guidelines: Subjective evaluation is preferred because objective metrics such as PESQ, STOI, and SI-SDR are described as unreliable for TTS perception quality.MOS is also described as insufficiently sensitive for comparing small voice-quality differences.
- Judgement guidelines: The guidelines require side-by-side listening by more than 20 native-language judges and at least 50 test utterances from each system.Human-level quality is assessed using average CMOS near 0 together with the Wilcoxon test’s p-level criterion.
- Prior-system evaluation: The study evaluates FastSpeech 2 plus HiFiGAN, Glow-TTS plus HiFiGAN, Grad-TTS plus HiFiGAN, and VITS on LJSpeech.The reproduced evaluations use 50 test utterances and 20 judges for MOS and CMOS assessments.
3 Description of NaturalSpeech System
NaturalSpeech is a fully end-to-end text-to-waveform TTS system that uses a VAE to align text-derived priors with speech-derived posteriors. Its modules enhance the prior, simplify the posterior, and address training–inference mismatch through differentiable duration modeling, bidirectional flow mappings, memory, and end-to-end optimization.
- Design principle: NaturalSpeech uses a VAE to compress speech into frame-level representations for waveform reconstruction while predicting the corresponding prior from phoneme sequences.The system is designed for conditional waveform generation from text.
- Phoneme encoder: Large-scale mixed-phoneme pre-training enhances phoneme-sequence representations for prior prediction.The pre-training uses phoneme and sup-phoneme representations to avoid character/word inconsistency and limited phoneme-vocabulary capacity.
- Differentiable durator: The differentiable durator predicts phoneme durations and uses learnable upsampling to extend phoneme-level hidden sequences to frame-level prior distributions.It also applies additional linear layers to the expanded hidden sequence.
- Bidirectional prior/posterior: A bidirectional flow module enhances the text prior through forward mapping and simplifies the speech posterior through backward mapping.Backward and forward losses train both flow directions and reduce the training–inference mismatch of prior flow models.
- VAE with memory: The memory-based VAE uses posterior latents as queries to attend to a memory bank, using the attention result for waveform reconstruction.The memory mechanism reduces the complexity required of the posterior, with memory and attention parameters included in the waveform decoder.
- Training and advantages: NaturalSpeech combines backward, forward, reconstruction, and end-to-end losses, then generates waveforms directly from text without cascaded acoustic and vocoder stages.The stated advantages include reduced training–inference mismatch, non-autoregressive inference, no cascaded errors, and unified variance modeling.
4 Experiments and Results
Experiments on LJSpeech evaluate NaturalSpeech against human recordings, prior TTS systems, and ablated variants, while also measuring training setup and inference latency. NaturalSpeech matches human-recording quality statistically, outperforms the compared systems in voice quality, and benefits from its design modules.
- Experimental Settings: LJSpeech contains 13,100 single-speaker English audios totaling nearly 24 hours; experiments use 12,500 training, 100 validation, and 500 test samples.Phoneme pre-training additionally uses 200 million news-crawl sentences without extra paired text-speech data.
- Experimental Settings: NaturalSpeech is trained for 15k epochs on 8 NVIDIA V100 GPUs with 32G memory and a dynamic batch size of 8,000 speech frames per GPU.The optimizer is AdamW with initial learning rate 2 × 10^-4 and decay factor γ = 0.999875 per epoch.
- Comparison with Human Recordings: −0.01 CMOS versus human recordings, with Wilcoxon p ≫ 0.05, indicates no statistically significant difference in the sentence-level comparison.The evaluation uses 50 test utterances and 20 judges per utterance; MOS and CMOS scores are similar to human recordings.
- Comparison with Previous TTS Systems: NaturalSpeech achieves better voice quality than FastSpeech 2 + HiFiGAN, Glow-TTS + HiFiGAN, Grad-TTS + HiFiGAN, and VITS in both MOS and CMOS.The compared systems’ results were reproduced by the authors, with HiFiGAN fine-tuned on predicted mel-spectrograms.
- Ablation Studies and Method Analyses: Removing phoneme pre-training causes a −0.09 CMOS drop, while removing the memory mechanism causes a −0.06 CMOS drop.These ablations support the effectiveness of phoneme pre-training and memory-based VAE design, respectively.
- Ablation Studies and Method Analyses: NaturalSpeech has 28.7M parameters and achieves faster or comparable inference speed while also achieving better voice quality than previous systems.Latency is measured on an NVIDIA V100 with batch size 1 sentence and averaged over the test sentences.
5 Conclusions and Discussions
The paper defines and evaluates human-level TTS quality through statistical comparison with human recordings, then develops NaturalSpeech to meet that criterion on LJSpeech. The authors restrict the claim to statistical indistinguishability on that dataset and identify more challenging voices and scenarios as future targets.
- Conclusions: The paper systematically studies how to define, judge, and build TTS systems with human-level quality.It develops NaturalSpeech as a fully end-to-end text-to-waveform model with phoneme pre-training, a differentiable durator, bidirectional prior/posterior modeling, and a memory mechanism in VAE.
- Conclusions: NaturalSpeech achieves human-level quality on LJSpeech, with CMOS evaluations showing no statistically significant difference from human recordings for the first time on this dataset.The claim concerns the evaluated LJSpeech benchmark rather than superiority to or replacement of human speech.
- Discussions: The authors do not claim that NaturalSpeech can surpass or replace humans; the claim is limited to statistical indistinguishability from human recordings on LJSpeech.They plan to evaluate more challenging settings such as expressive voices, longform audiobook voices, and singing voices.
A Study of the Quality Gap of Previous TTS System
The study decomposes the quality gap between a previous cascaded TTS system and human recordings across its phoneme encoder, variance adaptor, mel-spectrogram decoder, and vocoder. It identifies training-inference mismatch and one-to-many text-to-speech mapping as key sources of degradation.
- Systematic study: The evaluated FastSpeech 2 and HiFiGAN system contains a phoneme encoder, variance adaptor, mel-spectrogram decoder, and vocoder.These components provide the basis for measuring quality loss against component-specific upper bounds.
- Component quality gaps: 0.15 CMOS is the mel-spectrogram decoder’s drop relative to ground-truth mel-spectrograms.The comparison uses ground-truth pitch and duration as decoder inputs, then evaluates both outputs through the vocoder.
- Component quality gaps: 0.14 CMOS is the variance adaptor’s drop when predicted pitch and duration replace ground-truth values.The mel-spectrogram decoder and vocoder are used to convert both settings into waveforms for evaluation.
- Study assumptions: The phoneme-hidden-sequence comparison is approximate because ground-truth hidden sequences cannot be obtained.This limits the interpretation of that particular upper-bound comparison.
- Component quality gaps: 0.12 CMOS is gained by large-scale phoneme pre-training, indicating remaining improvement space in the phoneme encoder.The encoder is pretrained and then fine-tuned with the FastSpeech 2 training pipeline.
- Sources of degradation: Training-inference mismatch arises because ground-truth mel-spectrograms, pitch, and duration are used during training while predictions are used during inference.The paper identifies fully end-to-end text-to-waveform optimization as helpful for eliminating this mismatch.
B Differentiable Durator
The differentiable durator expands phoneme-level hidden representations into frame-level representations while enabling end-to-end duration optimization. It combines duration prediction with learnable, flexible upsampling rather than hard repetition.
- Differentiable Durator: The differentiable durator upsamples H_n×h into O_m×h, where h is hidden dimension, n phoneme length, and m frame length.It consists of a duration predictor and a learnable upsampling layer for phoneme-to-frame expansion.
- Duration Predictor: The duration predictor maps the phoneme hidden sequence H_n×h to estimated phoneme durations d̂_n×1.It uses three one-dimensional convolutional layers with ReLU, layer normalization, and dropout.
- Learnable Upsampling Layer: The learnable upsampling layer takes phoneme duration d and hidden sequence H as inputs to produce frame-level sequence O.It first calculates duration start and end matrices S_m×n and E_m×n, then constructs attention and context representations.
- Learnable Upsampling Layer: The upsampling computation uses projection, convolution, expansion, concatenation, and multilayer perceptron operations to form frame-level hidden outputs.The resulting representation is mapped to the prior distribution parameters μ(y; θ_pri) and σ(y; θ_pri).
- Differentiable Durator: Learnable upsampling provides flexible phoneme-duration adjustment and makes phoneme-to-frame expansion differentiable for joint optimization.This contrasts with simply repeating each phoneme representation according to predicted duration.
C Alternative Formulation of Bidirectional Prior/Posterior
The alternative bidirectional prior/posterior formulation expresses both directions as direct KL-divergence matching. Forward mapping enhances the text-conditioned prior, while backward mapping simplifies the speech-conditioned posterior.
- Alternative formulation: The bidirectional formulation therefore replaces the original flow-based loss expressions with direct KL losses between the corresponding transformed distributions.Both directions retain the appropriate change-of-variable terms when distributions are mapped between spaces.
- Backward loss: The backward loss directly matches posterior q(z|x; φ) to prior p(z|y; θ_pri) using KL divergence.It is written as KL[q(z|x; φ)||p(z|y; θ_pri)].
- Backward loss: The backward formulation applies the inverse flow mapping f^-1 to express the prior in the posterior’s transformed space.The change-of-variable rule contributes the determinant term involving ∂f^-1(z; θ_bpp)/∂z.
- Forward loss: The forward loss directly matches prior p(z′|y; θ_pri) to posterior q(z′|x; φ) using KL divergence.It is written as KL[p(z′|y; θ_pri)||q(z′|x; φ)].
- Forward loss: The forward formulation uses the flow mapping f to transform the prior and includes its change-of-variable determinant.The transformed prior is matched to the speech-derived posterior in z space.
D Soft Dynamic Time Warping in KL loss
Soft-DTW extends the bidirectional KL losses to frame-level prior and posterior distributions with different lengths. It recursively finds a soft best alignment while allowing insertions, deletions, and diagonal matches through a warp penalty.
- Motivation: Standard KL loss cannot be applied directly because frame-level prior and ground-truth speech distributions usually have different lengths.The paper uses Soft-DTW KL loss to handle this length mismatch.
- Backward loss: Soft-DTW recursively computes the backward KL loss by combining predecessor costs, local KL divergence, and warp penalties.The recurrence supports alternative alignment paths between simplified posterior and prior frames.
- Alignment cost: The local cost r_i,j is the KL divergence between aligned frame prefixes of the simplified posterior and prior under the best alignment.The formulation compares q(z′|x; φ, θ_bpp) with p(z′|y; θ_pri).
- Soft-DTW details: The soft-min operator replaces a hard minimum, while γ = 0.01 controls its softness and warp = 0.07 penalizes non-diagonal paths.These settings are specified for the Soft-DTW calculation.
- Forward loss: The forward Soft-DTW KL loss uses an analogous recurrence to align enhanced prior frames with posterior frames.Its recurrence includes the same three predecessor choices with local KL terms and warp penalties.
E Waveform Decoder Loss
NaturalSpeech replaces waveform reconstruction and prediction negative log-likelihoods with adversarial, feature-mapping, and mel-spectrogram losses. These losses compare generated speech with real waveforms at waveform, discriminator-feature, and mel-spectrogram levels.
- GAN Loss: GAN loss replaces negative log-likelihood for waveform reconstruction and prediction, following LS-GAN.The generator minimizes the loss while the discriminator maximizes it.
- Feature Mapping Loss: Feature mapping loss uses L1 distances between real and generated samples across discriminator intermediate features.It aggregates feature differences over discriminator layers and their feature counts.
- Mel-Spectrogram Loss: Mel-spectrogram loss computes the L1 distance between ground-truth and generated waveform mel-spectrograms.A function S(·) converts each waveform into its corresponding mel-spectrogram.
F Training Details of NaturalSpeech
Training uses staged procedures for duration prediction, bidirectional prior/posterior optimization, and VAE memory initialization. These stages transition from alignment- or posterior-based initialization toward learned end-to-end components.
- Duration Predictor: During warmup, monotonic alignment search estimates phoneme-to-frame durations by finding the optimal alignment between prior and posterior distributions.The alignment is searched with dynamic programming and recovered by backtracking the highest-probability path.
- Duration Predictor: After warmup, durations come from the duration predictor instead of MAS, while gradients are stopped at its input.Warmup lasts the first 1k epochs; the tuning stage uses only Le2e and updates the durator.
- Bidirectional Prior/Posterior: Bidirectional prior/posterior training uses only Lbwd during warmup, then adds Lfwd for bidirectional optimization.The warmup stage first establishes a reasonable prior distribution.
- VAE with Memory: The VAE memory bank is disabled during warmup and later initialized with 1K K-means cluster centers from posterior frame representations.After initialization, the memory bank is jointly trained with the model.
G Hyper-Parameters of NaturalSpeech
The section lists NaturalSpeech hyper-parameters and separates training-time components from the parameters retained for inference.
- Hyper-parameters: Table 8 provides the hyper-parameters used by NaturalSpeech.The supplied passage identifies the table but does not enumerate its entries.
- Model Parameters: Inference uses only θpho, θdur, θbpp, and θdec, totaling 28.7M model parameters.The posterior encoder and discriminators contribute additional training-time parameters but are not used during inference.