Source-linked AI summary
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu
TL;DR
FastSpeech’s teacher-student pipeline complicates training and loses information while non-autoregressive TTS must model varied speech from text. FastSpeech 2 trains on ground-truth targets with speech-variation conditioning, and FastSpeech 2s generates waveforms directly; experiments report faster training or inference and better voice quality.
Problem
Non-autoregressive TTS must predict varied speech from text alone, while FastSpeech’s distillation pipeline is complicated and its teacher-derived duration and targets lose information.
Method
FastSpeech 2 directly uses ground-truth mel-spectrograms and conditions on duration, pitch, and energy, while FastSpeech 2s directly generates speech waveforms from phoneme sequences in parallel.
Results
FastSpeech 2 achieves a 3x training speed-up over FastSpeech and better voice quality, while FastSpeech 2s further simplifies inference and maintains high voice quality.
Takeaways & Limitations
The proposed systems provide faster non-autoregressive TTS while addressing variation modeling through richer speech information and fully parallel waveform generation.
Takeaways & Limitations
FastSpeech 2 uses MFA trained on the paper’s training set for speech-text alignment, and the authors leave non-autoregressive TTS without external alignment models for future work.
Abstract
from arXiv · showhide
Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.
1 INTRODUCTION
FastSpeech addresses slow and fragile autoregressive TTS with non-autoregressive synthesis, but its teacher-student pipeline introduces complexity and information loss. FastSpeech 2 simplifies training, adds speech-variation inputs, and FastSpeech 2s extends the approach to parallel text-to-waveform generation.
- Non-autoregressive TTS generates mel-spectrograms quickly and avoids robustness issues while maintaining comparable voice quality.Previous autoregressive systems commonly suffer from slow inference and word skipping or repeating.
- FastSpeech alleviates TTS’s one-to-many mapping by distilling teacher-generated mel-spectrograms and expanding text using teacher-extracted durations.These mechanisms reduce target-side variance and match text and mel-spectrogram sequence lengths.
- FastSpeech’s two-stage teacher-student pipeline complicates training, while teacher-derived targets and durations can limit information quality.The supplied passage introduces these disadvantages but is truncated before detailing all of them.
- FastSpeech 2 directly trains on ground-truth targets and conditions synthesis on speech variation information including pitch, energy, and more accurate duration.During training, these features are extracted from target speech; during inference, predictors provide their values.
2 FASTSPEECH 2 AND 2S
FastSpeech 2 removes FastSpeech’s teacher-student distillation pipeline, trains on ground-truth targets, and adds duration, pitch, and energy conditioning to address one-to-many speech variation. FastSpeech 2s extends this design to fully parallel text-to-waveform synthesis.
- Motivation: FastSpeech 2 addresses FastSpeech’s complicated pipeline, distilled-target information loss, and inaccurate teacher-derived duration.FastSpeech’s design also involved reducing target variance through knowledge distillation and using teacher attention for duration extraction.
- FastSpeech 2: FastSpeech 2 directly trains on ground-truth mel-spectrograms instead of simplified teacher outputs.This removes teacher-student distillation and avoids information loss in distilled mel-spectrograms.
- Variance Adaptor: The variance adaptor adds duration, pitch, and energy information to the phoneme hidden sequence before parallel mel-spectrogram decoding.During training, extracted ground-truth values condition prediction and supervise the corresponding predictors.
- Variance Adaptor: FastSpeech 2 uses extracted recording duration as the duration-prediction target rather than duration extracted from a pretrained autoregressive TTS model.The duration predictor represents how many mel frames correspond to each phoneme and is optimized with MSE loss in logarithmic space.
- Assumption and Scope: FastSpeech 2’s alignment uses Montreal Forced Aligner trained on the paired text-audio training set.The authors state that future work will investigate non-autoregressive TTS without external alignment models.
- FastSpeech 2s: FastSpeech 2s directly generates waveform from text without cascaded mel-spectrogram generation and waveform generation.It conditions waveform generation on intermediate hidden representations and discards the mel-spectrogram decoder during inference.
- FastSpeech 2s: FastSpeech 2s addresses waveform-generation challenges with adversarial training and full-sequence mel-spectrogram decoding for text-feature extraction.Waveforms contain additional variance such as phase, while full-text audio clips are difficult to train because of their length and GPU-memory demands.
3 EXPERIMENTS AND RESULTS
Experiments evaluate audio quality, speed, variance modeling, duration accuracy, and ablations for FastSpeech 2 and 2s. The models show faster training or inference, closer pitch and energy characteristics, and measurable contributions from pitch, energy, CWT, and the mel-spectrogram decoder.
- Training and Inference Speedup: FastSpeech 2 reduces total training time by 3.12× compared with FastSpeech.The comparison includes FastSpeech teacher and student training, but excludes vocoder training.
- Training and Inference Speedup: FastSpeech 2 and 2s speed up waveform synthesis by 47.8× and 51.8× respectively over Transformer TTS.FastSpeech 2s is faster than FastSpeech 2 because it generates speech fully end-to-end.
- Variance Information: FastSpeech 2 and 2s produce pitch distributions closer to ground truth than other methods, with smaller average DTW distances.Their pitch standard deviation, skewness, and kurtosis are also closer to the ground-truth audio.
- Variance Information: FastSpeech 2 and 2s achieve lower energy MAE than FastSpeech, indicating energy more similar to ground-truth audio.The comparison uses ground-truth MFA duration for both systems.
- Ablation Study: CWT-based pitch prediction causes CMOS drops of 0.185 and 0.201 when replaced by direct contour fitting in FastSpeech 2 and 2s.The results support CWT for better pitch modeling and prosody.
- Ablation Study: Removing the FastSpeech 2s mel-spectrogram decoder causes a 0.285 CMOS drop.The ablation identifies the decoder as essential to high-quality waveform generation.
4 CONCLUSION
FastSpeech 2 simplifies training and improves variation modeling, while FastSpeech 2s extends the approach to parallel text-to-waveform generation. Experiments report better voice quality than FastSpeech, with FastSpeech 2 surpassing autoregressive models in voice quality.
- FastSpeech 2: FastSpeech 2 trains directly on ground-truth mel-spectrograms and adds improved duration, pitch, energy, and CWT-based pitch modeling.These changes target FastSpeech’s training complexity, information loss, and one-to-many mapping problem.
- FastSpeech 2s: FastSpeech 2s generates waveforms from text non-autoregressively with fully end-to-end inference and faster inference speed.It is developed as an extension of FastSpeech 2.
LICENSE AND AGREEMENT
The paper prohibits using its technologies to synthesize someone’s speech without consent and prohibits uses involving specified illegal activities.
- Usage restrictions: Speech synthesis requires the subject’s consent under the paper’s stated usage agreement.The agreement also prohibits child pornography, terrorism, politics, and other illegal acts.
A MODEL CONFIGURATION
FastSpeech 2 uses feed-forward Transformer blocks in its encoder and mel-spectrogram decoder, with configuration details compared across Transformer TTS, FastSpeech, and FastSpeech 2/2s.
- A MODEL CONFIGURATION: Table 7 lists hyperparameters for Transformer TTS, FastSpeech, and FastSpeech 2/2s.
B TRAINING AND INFERENCE
FastSpeech 2 is trained with Adam on one NVIDIA V100 GPU, while FastSpeech 2s uses separate training hardware and settings described in the implementation details.
- B TRAINING AND INFERENCE: FastSpeech 2 is trained on one NVIDIA V100 GPU with batches of 48 sentences.Training uses Adam with β1 = 0.9, β2 = 0.98, ε = 10^-9, and the Vaswani et al. learning-rate schedule.
- B TRAINING AND INFERENCE: FastSpeech 2 trains for 160k steps until convergence.
- B TRAINING AND INFERENCE: During inference, FastSpeech 2 converts predicted mel-spectrograms into audio with a pre-trained Parallel WaveGAN.
C.1 CONTINUOUS WAVELET TRANSFORM
The method converts a continuous pitch contour into a pitch spectrogram with CWT and reconstructs the contour with iCWT, using ten wavelet scales for decomposition and recomposition.
- C.1 CONTINUOUS WAVELET TRANSFORM: CWT converts the continuous pitch contour F0 into a pitch spectrogram W(τ, t).The wavelet representation uses scale and position variables τ and t.
- C.1 CONTINUOUS WAVELET TRANSFORM: iCWT recovers the original pitch contour F0 from the wavelet representation W(τ, t).
- C.1 CONTINUOUS WAVELET TRANSFORM: The pitch contour is decomposed into 10 separate wavelet components.The scales use i = 1, ..., 10 and τ0 = 5ms.
- C.1 CONTINUOUS WAVELET TRANSFORM: Ten wavelet components are recomposed into the pitch contour ˆF0.
C.2 IMPLEMENTATION DETAILS
Pitch processing extracts and normalizes the pitch contour, converts it into a CWT pitch spectrogram, and trains a predictor whose outputs are inverted and denormalized during inference.
- C.2 IMPLEMENTATION DETAILS: Pitch extraction fills unvoiced frames by linear interpolation, applies a logarithmic transform, and normalizes each utterance.The original utterance-level mean and variance are retained for reconstruction before CWT conversion.
- C.2 IMPLEMENTATION DETAILS: The pitch predictor uses two 1D-convolutional layers with ReLU, layer normalization, dropout, and a linear projection.A separate linear layer predicts the mean and variance of the recovered pitch contour from a global vector.
- C.2 IMPLEMENTATION DETAILS: During training, the pitch predictor is optimized with mean square error against the ground-truth pitch spectrogram and contour statistics.
- C.2 IMPLEMENTATION DETAILS: During inference, predicted pitch spectrograms are converted to contours with iCWT and denormalized using predicted mean and variance.Figure 2 presents the pitch predictor details; CWT and iCWT denote continuous and inverse continuous wavelet transforms.
D CASE STUDY ON PITCH CONTOUR
FastSpeech 2 and FastSpeech 2s capture pitch variations better than FastSpeech, attributed to using pitch information as an input.
- FastSpeech 2 and 2s capture pitch variations better than FastSpeech.The comparison uses pitch contours from generated and ground-truth audio samples.
E VARIANCE CONTROL
FastSpeech 2 makes synthesized speech controllable through variance inputs, including pitch, and retains high-quality mel-spectrograms after pitch manipulation.
- FastSpeech 2 generates high-quality mel-spectrograms after adjusting F̂0 from 0.75 to 1.50 times.The manipulation changes the pitch input, with corresponding mel-spectrograms shown before and after adjustment.
- FastSpeech 2 and 2s allow manual control of pitch, duration, and energy in synthesized audio.The models introduce variance information as inputs, enabling control over these speech attributes.