Source-linked AI summary
Tacotron: Towards End-to-End Speech Synthesis
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, Rif A. Saurous
TL;DR
Traditional TTS systems require complex, expertise-intensive pipelines, while end-to-end TTS must map compressed text to variable, continuous audio. Tacotron uses character-to-spectrogram sequence-to-sequence modeling trained from text–audio pairs, achieving 3.82 MOS on US English and outperforming a production parametric system in naturalness. Its frame-based inference is substantially faster than sample-level autoregressive methods, though several components remain open to improvement.
Problem
Traditional TTS pipelines require multiple complex components built with extensive domain expertise and brittle design choices, while text-to-audio mapping involves substantial variation.
Method
Tacotron is an integrated attention-based sequence-to-sequence model that takes characters as input and produces spectrogram frames from text–audio pairs.
Results
3.82 MOS was achieved on US English, outperforming a production parametric system in naturalness.
Takeaways & Limitations
Frame-based generation makes Tacotron substantially faster in inference than sample-level autoregressive methods, while avoiding hand-engineered linguistic features and complex alignment components.
Takeaways & Limitations
The output layer, attention module, loss function, and Griffin-Lim waveform synthesizer remain open to improvement, with Griffin-Lim known to produce audible artifacts.
Abstract
from arXiv · showhide
A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these components often requires extensive domain expertise and may contain brittle design choices. In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters. Given <text, audio> pairs, the model can be trained completely from scratch with random initialization. We present several key techniques to make the sequence-to-sequence framework perform well for this challenging task. Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness. In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.
1 INTRODUCTION
Traditional TTS pipelines rely on complex, independently trained components that require extensive expertise, while end-to-end TTS must handle long continuous outputs and substantial variation. Tacotron addresses this challenge with character-to-spectrogram sequence modeling trained from text–audio pairs.
- 1 INTRODUCTION: Traditional TTS pipelines combine text analysis, duration modeling, acoustic prediction, and vocoding, requiring extensive expertise and engineering effort.Their independently trained components can compound errors.
- 1 INTRODUCTION: End-to-end TTS can reduce feature engineering, support richer conditioning, ease adaptation, and potentially improve robustness by avoiding separately trained components.The passage identifies these as advantages of integrated modeling.
- 1 INTRODUCTION: TTS is a large-scale inverse problem because compressed text must be decompressed into audio with varied pronunciations and speaking styles.The same text can produce substantially different signal-level outputs.
- 1 INTRODUCTION: Tacotron uses an attention-based sequence-to-sequence model that accepts characters and produces raw spectrograms from text–audio pairs.The model is trained from scratch with random initialization.
- 1 INTRODUCTION: 3.82 MOS was achieved on a US English evaluation set, outperforming a production parametric system in naturalness.The result uses a simple waveform synthesis technique.
2 RELATED WORK
Prior neural TTS systems either replace only selected pipeline components, train components separately, or require alignment aids and vocoder parameters. Tacotron instead modifies sequence-to-sequence modeling for character inputs and trains the system from scratch.
- 2 RELATED WORK: WaveNet improves audio generation but remains slow and dependent on linguistic features from an existing TTS frontend.It replaces the vocoder and acoustic model rather than providing an end-to-end system.
- 2 RELATED WORK: DeepVoice replaces traditional TTS components with neural networks, but its components are independently trained and are not straightforward to optimize end to end.The passage contrasts this with Tacotron’s integrated training goal.
- 2 RELATED WORK: Earlier attention-based end-to-end TTS required a pretrained HMM aligner, used training tricks that affected prosody, predicted vocoder parameters, and relied on phoneme inputs.The reported experimental results were also limited.
- 2 RELATED WORK: Char2Wav accepts characters but predicts vocoder parameters and requires separate pretraining of its sequence-to-sequence and SampleRNN models.Tacotron instead directly predicts raw spectrograms and can be trained from scratch.
3 MODEL ARCHITECTURE
Tacotron uses an attention-based sequence-to-sequence architecture that maps character inputs to spectrogram frames before waveform synthesis. Its encoder, decoder, CBHG modules, and post-processing network address representation, alignment, and synthesis requirements.
- Overall architecture: The model encodes characters, decodes spectrogram frames with attention, and converts them into waveforms through post-processing.The encoder produces sequential text representations, while the decoder predicts mel-scale spectrogram targets.
- CBHG module: CBHG combines 1-D convolution banks, highway networks, and a bidirectional GRU to extract local, contextual, and high-level sequence representations.Convolution outputs are max-pooled, processed with residual convolutions, and passed through highway layers before bidirectional recurrence.
- Encoder: The encoder embeds one-hot character vectors, applies a dropout bottleneck pre-net, and uses CBHG to form the attention representation.The authors report that this CBHG-based encoder reduces overfitting and makes fewer mispronunciations than a standard multilayer RNN encoder.
- Decoder: The attention decoder uses a stateful recurrent query, concatenated context, stacked GRUs, and vertical residual connections to predict mel-scale spectrograms.The mel-scale target is less redundant for learning text-speech alignment than directly predicting raw spectrograms.
- Decoder: Predicting multiple non-overlapping frames per decoder step reduces decoding steps and improves convergence speed and attention alignment stability.At inference, the last frame among each group is fed into the next decoder step; during training, every r-th ground-truth frame is used.
- Post-processing and synthesis: The post-processing CBHG converts decoded targets into linear-frequency spectral magnitudes, which Griffin-Lim synthesizes into waveforms.Because it sees the full decoded sequence, the post-processing network can use forward and backward information to correct individual-frame errors.
4 MODEL DETAILS
The model uses fixed spectrogram preprocessing, scheduled optimization, equal-weight L1 losses, and padding reconstruction during training. These details specify the signal representation, optimization schedule, and stopping behavior.
- Signal representation: The experiments use log-magnitude spectrograms with Hann windowing, 50 ms frames, 12.5 ms shifts, 2048-point Fourier transforms, and 24 kHz sampling.Pre-emphasis with coefficient 0.97 was also found helpful.
- Optimization: Training uses Adam with learning-rate decay from 0.001 to 0.0001 across 2M global steps and equal-weight L1 losses for mel- and linear-scale spectrograms.The intermediate learning rates are 0.0005 after 500K steps and 0.0003 after 1M steps.
- Sequence padding: With batch size 32, reconstructing zero-padded frames helps the model learn when to stop emitting outputs and prevents repeated sounds near sequence ends.The authors found that masking padded-frame loss alone caused stopping failures.
5 EXPERIMENTS
Experiments evaluate Tacotron through ablations, spectrogram comparisons, attention alignments, and mean opinion scores. The results indicate cleaner alignments, improved spectrogram structure, and higher naturalness than a production parametric system.
- Experimental Setup: Tacotron was trained on about 24.6 hours of text-normalized North American English speech from a professional female speaker.The dataset phrases were text normalized, such as converting “16” to “sixteen”.
- Ablation Analysis: Vanilla seq2seq attention can stall for many frames, damaging intelligibility, naturalness, and duration, whereas Tacotron learns clean, smooth alignments.Figure 3 compares the learned attention alignments; Tacotron’s decoder is shorter because its output reduction factor is r=5.
- Ablation Analysis: The CBHG encoder produces less noisy alignments than a two-layer residual GRU encoder and generalizes well to long, complex phrases.Noisy alignments were associated with mispronunciations in synthesized signals.
- Ablation Analysis: The post-processing net yields better-resolved harmonics and high-frequency formant structure, reducing synthesis artifacts.The comparison kept other components unchanged except that the decoder predicted linear-scale spectrograms without the post-processing net.
- Mean Opinion Score Tests: 3.82 MOS: Tacotron outperformed a production LSTM-based parametric system in naturalness on tests using 100 unseen phrases and eight ratings per phrase.Native speakers rated naturalness on a 5-point Likert scale, with headphone ratings included in MOS computation.
6 DISCUSSIONS
Tacotron is an integrated character-to-spectrogram TTS model that can be trained from scratch and runs faster through frame-based inference. The authors identify several model components and design choices as open areas for improvement.
- Discussion: Tacotron takes character sequences as input and outputs spectrograms for waveform synthesis, without hand-engineered linguistic features or an HMM aligner.The model can be trained from scratch with random initialization and uses a simple waveform synthesis module.
- Discussion: 3.82 MOS: Tacotron outperformed a production parametric system in US English naturalness.Its frame-based inference is substantially faster than sample-level autoregressive methods.
- Limitations: The output layer, attention module, loss function, and Griffin-Lim waveform synthesizer remain open areas for improvement.The authors specifically note that Griffin-Lim may produce audible artifacts and are investigating neural-network-based spectrogram inversion.