Source-linked AI summary
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, Yonghui Wu
TL;DR
Text-to-speech systems still struggled to produce natural speech, while WaveNet required elaborate linguistic features. Tacotron 2 combines mel-spectrogram prediction with a modified WaveNet vocoder, achieving sound quality close to natural human speech without complex feature engineering.
Problem
Text-to-speech systems remained challenging, and WaveNet conditioning required domain-intensive linguistic, pitch, duration, and pronunciation features.
Method
Tacotron 2 predicts mel spectrograms from character sequences with an attention-based sequence-to-sequence network, then synthesizes waveforms using a modified WaveNet vocoder.
Results
Tacotron 2 achieves state-of-the-art sound quality close to natural human speech and significantly outperforms the evaluated alternative TTS systems.
Takeaways & Limitations
The system can be trained directly from data without complex feature engineering while combining Tacotron-level prosody with WaveNet-level audio quality.
Takeaways & Limitations
The evaluation set shares recurring patterns and common words with training data, which could potentially inflate MOS despite preserving relative comparisons.
Abstract
from arXiv · showhide
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text. The system is composed of a recurrent sequence-to-sequence feature prediction network that maps character embeddings to mel-scale spectrograms, followed by a modified WaveNet model acting as a vocoder to synthesize timedomain waveforms from those spectrograms. Our model achieves a mean opinion score (MOS) of $4.53$ comparable to a MOS of $4.58$ for professionally recorded speech. To validate our design choices, we present ablation studies of key components of our system and evaluate the impact of using mel spectrograms as the input to WaveNet instead of linguistic, duration, and $F_0$ features. We further demonstrate that using a compact acoustic intermediate representation enables significant simplification of the WaveNet architecture.
1. INTRODUCTION
The introduction frames natural TTS as challenging and identifies the expertise-heavy feature engineering and artifacts limiting existing approaches. It presents Tacotron 2 as a unified neural system combining character-to-mel-spectrogram prediction with a modified WaveNet vocoder.
- Natural speech generation from text remains challenging despite decades of investigation and successive dominant synthesis techniques.
- WaveNet produces audio quality rivaling human speech, but its linguistic, F0, and duration inputs require elaborate text analysis, domain expertise, and a robust pronunciation lexicon.
- Tacotron simplifies synthesis by predicting spectrograms from characters, but its Griffin-Lim vocoder introduces characteristic artifacts and lower quality than WaveNet.
- The paper proposes an entirely neural pipeline that predicts mel spectrograms from normalized characters and uses a modified WaveNet vocoder to synthesize natural-sounding speech.
2. MODEL ARCHITECTURE
Tacotron 2 combines an attention-based recurrent sequence-to-sequence network that predicts mel spectrograms from characters with a modified WaveNet vocoder that generates time-domain waveforms. Mel spectrograms bridge the components as a compact, smoother acoustic representation that supports separate training and simplifies waveform generation.
- System overview: The system has two components: an attention-based recurrent sequence-to-sequence network predicts mel spectrogram frames from characters, and modified WaveNet generates waveforms from them.The mel spectrogram frames are the conditioning representation passed between the components.
- Acoustic representation: Mel spectrograms enable separate training because they are easily computed from waveforms, smoother than samples, and phase-invariant under squared-error training.They also provide a simpler, lower-level acoustic representation than the linguistic and acoustic features used in WaveNet.
- Spectrogram representation: The mel spectrogram uses a 50 ms STFT frame size and 12.5 ms hop, then an 80-channel mel filterbank spanning 125 Hz to 7.6 kHz with log compression.A 5 ms hop increased temporal resolution but caused significantly more pronunciation issues.
- Spectrogram prediction network: The spectrogram predictor uses character embeddings, a convolutional encoder, location-sensitive attention, and an autoregressive recurrent decoder that predicts one frame at a time.The decoder’s pre-net acts as an information bottleneck that was essential for learning attention.
- Spectrogram prediction network: Unlike original Tacotron, the model uses vanilla LSTM and convolutional layers instead of CBHG and GRU blocks, with no reduction factor.Each decoder step therefore corresponds to a single spectrogram frame.
- Waveform synthesis: Modified WaveNet inverts mel spectrograms using 30 dilated convolution layers in 3 dilation cycles and 2 conditioning-stack upsampling layers.It generates 16-bit samples at 24 kHz with a 10-component mixture of logistic distributions.
3. EXPERIMENTS & RESULTS
Tacotron 2 achieves near-ground-truth speech quality across in-domain and out-of-domain evaluations, while mel spectrogram conditioning enables a compact WaveNet vocoder. Ablations show that matching training and inference features matters, and sufficient waveform-scale context remains necessary.
- Overall evaluation: 4.53 MOS is comparable to ground-truth audio, and the proposed system significantly outperforms prior TTS systems.The system is compared with linguistic-feature WaveNet, original Tacotron with Griffin-Lim, concatenative, and parametric baselines.
- Overall evaluation: −0.270 ± 0.155 indicates a small but statistically significant preference for ground truth over synthesized speech.The side-by-side evaluation used 800 ratings on 100 items.
- Error analysis: 4.354 MOS was obtained on a separate 100-sentence test set, with 23 sentences judged to have unnatural prosody and 6 containing mispronunciations.The manual analysis found 1 skipped-word case, 0 repeated-word cases, and 1 end-point prediction failure.
- Generalization: 4.148±0.124 MOS on 37 news headlines is nearly tied with linguistic-feature WaveNet at 4.137 ± 0.128.The side-by-side comparison showed a statistically insignificant preference toward the proposed system of 0.142 ± 0.338.
- Feature ablations: Matching WaveNet training features to inference features gives the best performance; ground-truth training features perform worse when synthesizing predicted spectrograms.Predicted spectrograms are oversmoothed and less detailed because the feature network optimizes squared error.
- WaveNet ablations: 12 WaveNet layers with a 10.5 ms receptive field achieve high-quality audio versus 30 layers and 256 ms in the baseline, but removing dilation degrades quality.These results support compact mel conditioning while showing that sufficient waveform-sample-scale context remains necessary.
4. CONCLUSION
Tacotron 2 is a fully neural TTS system that predicts mel spectrograms with an attention-based recurrent sequence-to-sequence network and synthesizes waveforms with a modified WaveNet vocoder. It achieves Tacotron-level prosody, WaveNet-level audio quality, and state-of-the-art sound quality close to natural human speech without complex feature engineering.
- 4. CONCLUSION: Tacotron 2 combines an attention-based recurrent sequence-to-sequence network for mel spectrogram prediction with a modified WaveNet vocoder.The system is fully neural and trained directly from data.
- 4. CONCLUSION: The resulting system synthesizes speech with Tacotron-level prosody and WaveNet-level audio quality.
- 4. CONCLUSION: Tacotron 2 achieves state-of-the-art sound quality close to that of natural human speech.Its direct-from-data training avoids reliance on complex feature engineering.