Source-linked AI summary
Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
Jaehyeon Kim, Sungwon Kim, Jungil Kong, Sungroh Yoon
TL;DR
Parallel TTS models are faster but traditionally require external autoregressive aligners for training. Glow-TTS learns monotonic alignments internally with dynamic programming and generative flows, achieving 15.7 times faster synthesis than Tacotron 2 with comparable performance and additional controllability.
Problem
Parallel TTS models require external aligners, typically pretrained autoregressive TTS models, to obtain text–speech alignments during training.
Method
Glow-TTS combines a flow-based generative model with dynamic programming to learn the most probable monotonic alignment internally.
Results
15.7 times faster than Tacotron 2, Glow-TTS synthesizes mel-spectrograms with comparable performance and significantly better robustness on long utterances.
Takeaways & Limitations
Glow-TTS supports fast, diverse, controllable, robust, and multi-speaker speech synthesis without external aligners.
Takeaways & Limitations
Glow-TTS assumes the alignment is monotonic and surjective so text is neither skipped nor repeated.
Abstract
from arXiv · showhide
Recently, text-to-speech (TTS) models such as FastSpeech and ParaNet have been proposed to generate mel-spectrograms from text in parallel. Despite the advantage, the parallel TTS models cannot be trained without guidance from autoregressive TTS models as their external aligners. In this work, we propose Glow-TTS, a flow-based generative model for parallel TTS that does not require any external aligner. By combining the properties of flows and dynamic programming, the proposed model searches for the most probable monotonic alignment between text and the latent representation of speech on its own. We demonstrate that enforcing hard monotonic alignments enables robust TTS, which generalizes to long utterances, and employing generative flows enables fast, diverse, and controllable speech synthesis. Glow-TTS obtains an order-of-magnitude speed-up over the autoregressive model, Tacotron 2, at synthesis with comparable speech quality. We further show that our model can be easily extended to a multi-speaker setting.
1 Introduction
Autoregressive TTS offers high quality but can be slow and error-prone, while parallel TTS requires external aligners. Glow-TTS removes that dependency by learning monotonic alignments internally and synthesizes quickly with comparable quality.
- Motivation: Autoregressive TTS inference slows with utterance length and can produce attention errors on repeated words.These models generate outputs sequentially, creating deployment delays for long utterances.
- Motivation: Parallel TTS models synthesize mel-spectrograms faster and reduce some failures through monotonic alignment constraints.However, their training requires well-aligned text–speech attention maps.
- Motivation: Parallel TTS performance depends critically on external aligners, typically pretrained autoregressive TTS models.The aligners provide the attention maps needed to address the length mismatch between text and speech.
- Glow-TTS: Glow-TTS is a flow-based parallel TTS model that internally learns its alignment and eliminates external aligners.This simplifies the training procedure of parallel TTS models.
- Glow-TTS: Glow-TTS combines flows with dynamic programming to search for the most probable monotonic alignment and maximize speech log-likelihood.Hard monotonic alignments support robust synthesis, while flows support fast, diverse, and controllable speech generation.
- Results: 15.7 times faster than Tacotron 2, Glow-TTS generates mel-spectrograms with comparable performance and greater robustness on long utterances.The model also supports controllable intonation and pitch and can extend to multi-speaker synthesis.
2 Related Work
Prior alignment methods use dynamic programming, while parallel TTS systems commonly depend on autoregressive aligners. Glow-TTS instead combines internal hard monotonic alignment with generative flows for robustness, diversity, and controllability.
- Alignment Estimation: HMMs and CTC estimate text–speech alignments efficiently using forward-backward dynamic programming.Glow-TTS introduces a related dynamic-programming approach for finding probable alignments.
- Text-to-Speech Models: FastSpeech and ParaNet generate mel-spectrogram frames in parallel but require pretrained autoregressive models to extract alignments.Their external aligners address the length mismatch between text and speech.
- Flow-based Generative Models: Flow-based generative models provide exact likelihood estimation through invertible transformations and enable efficient sampling.These properties have also been applied to speech synthesis to address slow autoregressive vocoder sampling.
- Related Models: AlignTTS, Flowtron, and Flow-TTS pursue related alignment-free, flow-based, or controllable TTS capabilities.The cited comparison distinguishes their architectures and attention mechanisms from Glow-TTS.
- Glow-TTS: Glow-TTS combines hard monotonic alignments with generative flows to target robustness, diversity, and controllability.This design contrasts with related TTS models using soft attention modules.
3 Glow-TTS
Glow-TTS combines a flow-based decoder with monotonic alignment search to train and synthesize speech without an external aligner. Dynamic programming finds the most probable monotonic alignment, while duration prediction enables parallel inference.
- 3.1 Training and Inference Procedures: Glow-TTS transforms a conditional prior distribution into mel-spectrograms with a flow-based decoder and computes exact data log-likelihood using change of variables.The text encoder produces the prior statistics from the text condition.
- 3.1 Training and Inference Procedures: The alignment function maps latent speech positions to text-encoder statistics, and training seeks parameters and alignments that maximize data log-likelihood.The prior uses isotropic multivariate Gaussian statistics produced by the text encoder.
- 3.1 Training and Inference Procedures: Monotonic alignment search decomposes optimization by finding the most probable monotonic alignment for current parameters before updating the model parameters.The modified objective does not guarantee the global solution but provides a lower bound.
- 3.1 Training and Inference Procedures: During inference, the text encoder and duration predictor estimate prior statistics and alignment, a latent variable is sampled, and the decoder synthesizes mel-spectrograms in parallel.The architecture uses a flow-based decoder composed of repeated normalization, invertible convolution, and affine coupling blocks.
- 3.2 Monotonic Alignment Search: MAS recursively caches partial-alignment log-likelihoods, then backtracks from the final text position to recover the complete alignment efficiently.The recurrence considers whether the previous latent frame aligned to the previous or current text statistics.
- 3.2 Monotonic Alignment Search: O(Ttext × Tmel) is MAS’s time complexity, while each iteration takes less than 20 ms and less than 2% of total training time in experiments.MAS runs on CPU and is unnecessary during inference because the duration predictor estimates alignment.
4 Experiments
The experiments evaluate Glow-TTS against Tacotron 2 in single-speaker and multi-speaker settings using established speech datasets and a shared vocoder. Training and conditioning procedures are adapted for the multi-speaker case.
- 4 Experiments: The single-speaker evaluation uses LJSpeech, containing 13,100 clips totaling approximately 24 hours, split into 12,500 training, 100 validation, and 500 test samples.The multi-speaker evaluation uses the train-clean-100 subset of LibriTTS.
- 4 Experiments: Glow-TTS is compared with the best publicly available autoregressive Tacotron 2 model using phoneme tokens and WaveGlow to convert generated mel-spectrograms into waveforms.Both systems use the same mel-spectrogram configuration.
- 4 Experiments: Glow-TTS training required 240K iterations and 3 days with mixed precision on two NVIDIA V100 GPUs.Optimization used Adam with the Noam learning-rate schedule.
- 4 Experiments: The multi-speaker model adds speaker embeddings and a larger hidden dimension, applying speaker conditioning in every decoder affine coupling layer.Tacotron 2 instead concatenates the speaker embedding with encoder outputs at each time step.
5 Results
Glow-TTS achieves comparable speech quality to Tacotron 2 while substantially improving synthesis speed and robustness on long utterances. Its flow-based latent representation also supports controllable variation in intonation, pitch, and speaking rate, with multi-speaker extension demonstrated.
- Sampling Speed: 15.7 times faster synthesis was achieved by Glow-TTS than Tacotron 2 on average, with inference time remaining almost constant at 40ms regardless of utterance length.Tacotron 2 inference time increases linearly because it samples sequentially.
- Sampling Speed: 1.5 seconds was sufficient to synthesize 1-minute speech end-to-end with Glow-TTS and WaveGlow.Glow-TTS and WaveGlow accounted for 4% and 96% of total inference time, respectively; Glow-TTS took 55ms for the mel-spectrogram.
- Robustness: Glow-TTS remained robust on long texts unseen during training, whereas Tacotron 2’s CER began increasing beyond approximately 260 input characters.Robustness was evaluated on long utterances from Harry Potter and the Philosopher’s Stone using Google Cloud Speech-To-Text.
- Diversity and Controllability: Varying ϵ produces diverse stress and intonation patterns, varying T controls pitch while maintaining similar intonation, and scaling predicted durations controls speaking rate.The evaluated duration multipliers were 1.25, 1.0, 0.75, and 0.5.
- Audio Quality: 3.45 MOS was achieved by the best Glow-TTS configuration, with performance comparable to Tacotron 2.The evaluation used 50 speakers and one randomly sampled test utterance per speaker.
- Multi-Speaker TTS: Different speaker identities lead Glow-TTS to predict different durations for tokens in the same sentence.The speaker identity was the only input difference in this comparison.
6 Conclusion
Glow-TTS is a parallel, flow-based TTS model that learns the most probable monotonic alignment between text and speech latents without external aligners. It synthesizes 15.7 times faster than Tacotron 2 with comparable performance and supports robustness, controllability, and multi-speaker extension.
- Glow-TTS is a flow-based parallel TTS model that learns the most probable monotonic alignment between text and speech latents.
- Glow-TTS simplifies training by finding the alignment itself, eliminating the necessity of external aligners.
- 15.7 times faster than Tacotron 2, Glow-TTS achieves comparable performance while synthesizing mel-spectrograms.
- Glow-TTS provides control over speaking rate and pitch, robustness, and extensibility to a multi-speaker setting.
Broader Impact
Glow-TTS and related neural TTS models can support applications requiring naturally synthesized speech. The listed applications include voice assistants, audiobooks, advertising, automotive navigation, and automated answering services.
- Neural TTS models can support applications that require naturally synthesized speech.
- Potential applications include AI voice assistants, audiobooks, advertisements, automotive navigation systems, and automated answering services.
Supplementary Material of Glow-TTS: A Generative Flow for Text-to-Speech via
The supplementary material includes a section on Monotonic Alignment Search.
- Monotonic Alignment Search is presented as a supplementary-material topic.
A.1. Details of the Model Architecture
The supplementary architecture material describes Glow-TTS's encoder and decoder, including text processing, duration and prior prediction, and grouped 1x1 convolutions. It also lists model hyperparameters and reports a 28.6M-parameter model.
- Grouped 1x1 convolutions mix channels across the feature-map halves separated by coupling layers.
- The encoder processes text through an encoder pre-net and Transformer encoder before predicting prior statistics and durations from hidden representation h.
- The decoder architecture and its implementation details are provided in the supplementary material.
- Glow-TTS has 28.6M parameters, fewer than FastSpeech's 30.1M parameters.
- Table 3 lists Glow-TTS hyperparameters for the LJ dataset.
B.1. Attention Error Analysis
The analysis compares attention errors across TTS models on 100 test sentences, including long utterances. Glow-TTS performs slightly worse than Tacotron 2 on these sentences but retains robustness on extremely long inputs.
- 100 test sentences were used to measure attention alignment errors, with average and maximum lengths of 59.65 and 315.DeepVoice 3 and ParaNet results are not directly comparable because different grapheme-to-phoneme conversion tools were used.
- Monotonic attention masking reduces attention errors by restricting computation to a fixed window around the target position.
- Glow-TTS performs slightly worse than Tacotron 2 on the test sentences but remains robust to extremely long sentences, unlike Tacotron 2.
B.2. Side-by-side Evaluation between Glow-TTS and Tacotron 2
The side-by-side evaluation compares Glow-TTS with Tacotron 2 using CMOS ratings and also describes latent-space speaker conversion. Glow-TTS is preferred in the reported evaluation, while its flow decoder supports conversion between source and target speaker identities.
- Glow-TTS wins against Tacotron 2 by a CMOS gap of 0.934 in 500 ratings across 50 items.Both models were trained on LJSpeech and evaluated with sampling temperature 0.333.
- Table 5 reports the Comparative Mean Opinion Score of single-speaker TTS models.
- Speaker conversion maps a source mel-spectrogram into a target speaker’s mel-spectrogram by inverting and then forwarding the flow decoder.The inverse pass uses the source speaker identity, while the forward pass uses the target identity.