Source-linked AI summary
One TTS Alignment To Rule Them All
Rohan Badlani, Adrian Łancucki, Kevin J. Shih, Rafael Valle, Wei Ping, Bryan Catanzaro
TL;DR
Autoregressive TTS alignments can be brittle on long or out-of-domain text, and parallel systems often rely on externally sourced durations. This paper applies a shared alignment-learning framework using forward-sum, Viterbi, and diagonal-prior guidance; experiments report improvements across both autoregressive and parallel models, including speech quality and robustness.
Problem
Autoregressive TTS alignments can fail on long or out-of-domain text, while parallel models commonly rely on external durations that complicate end-to-end training.
Source: §1 · p. 1Method
The framework applies alignment learning across TTS architectures using forward-sum likelihood, model-specific alignment distributions, Viterbi paths, and a static prior.
Source: §2.1 · p. 2, §2.2 · p. 2, §2.3 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Speech-to-text alignment is a critical component of neural textto-speech (TTS) models. Autoregressive TTS models typically use an attention mechanism to learn these alignments on-line. However, these alignments tend to be brittle and often fail to generalize to long utterances and out-of-domain text, leading to missing or repeating words. Most non-autoregressive endto-end TTS models rely on durations extracted from external sources. In this paper we leverage the alignment mechanism proposed in RAD-TTS as a generic alignment learning framework, easily applicable to a variety of neural TTS models. The framework combines forward-sum algorithm, the Viterbi algorithm, and a simple and efficient static prior. In our experiments, the alignment learning framework improves all tested TTS architectures, both autoregressive (Flowtron, Tacotron 2) and non-autoregressive (FastPitch, FastSpeech 2, RAD-TTS). Specifically, it improves alignment convergence speed of existing attention-based mechanisms, simplifies the training pipeline, and makes the models more robust to errors on long utterances. Most importantly, the framework improves the perceived speech synthesis quality, as judged by human evaluators.
1. Introduction
Autoregressive TTS alignments can fail on long or out-of-domain text, while parallel models often depend on external durations. The paper applies a shared alignment-learning framework across TTS models and reports improved robustness and speech quality.
1. Introduction
- Autoregressive TTS models can produce missing or repeated words on out-of-domain text, especially in long utterances.Even content- and location-sensitive attention can still have alignment problems on long utterances.Source: §1 · p. 1
- Parallel TTS models often require external durations, which can complicate training and may be unavailable or unsuitable for some languages.The paper motivates end-to-end alignment learning as a way to simplify the training pipeline and achieve rapid, stable convergence.Source: §1 · p. 1
- The framework extends RAD-TTS alignment learning to autoregressive and parallel models, simplifying their pipelines and improving convergence and robustness on long utterances.It adds a constraint that directly maximizes the likelihood of text given speech mel-spectrograms.Source: §1 · p. 1
- Models trained with the framework have fewer repeated and missing words, greater stability on long sequences, and improved overall speech quality in human evaluation.Source: §1 · p. 1
2. Alignment Learning Framework
The framework aligns text representations with mel-spectrograms and applies a shared alignment-learning objective to autoregressive and parallel TTS models. It combines monotonic likelihood maximization, model-specific alignment distributions, and a static prior to guide learning.
2. Alignment Learning Framework
- The framework aligns encoded text inputs with mel-spectrograms across autoregressive and parallel TTS architectures.The section introduces the alignment objective and its applications to both model types.Source: §2 · p. 1
2.1. Unsupervised alignment learning objective
- The objective maximizes text likelihood over valid monotonic alignments, using the forward-sum algorithm and a CTC implementation.The formulation does not depend on how the token-to-frame likelihood is obtained, so it applies to both autoregressive and parallel models.Source: §2.1 · pp. 1–2
2.2. Autoregressive TTS Models
- Autoregressive models use attention weights as the token-to-mel-frame likelihood distribution, incorporating that distribution into the alignment loss.The described setup uses stateful content-based attention for Flowtron and hybrid content- and location-based attention for Tacotron 2.Source: §2.2 · p. 2
2.3. Parallel TTS Models
- Parallel models can use a standalone aligner based on learned text–mel affinities, then convert soft alignments to hard monotonic paths with Viterbi and a KL-divergence loss.The framework uses simple convolutional encoders for text and mel-spectrograms; the resulting hard alignments address the train-test gap between soft maps and required durations.Source: §2.3 · pp. 2–3
2.4. Alignment Acceleration
- A static 2D prior, wider near the center and narrower near the corners, makes far-off-diagonal alignments less probable to accelerate learning.The prior is applied during training, when mel-spectrogram length is known; other priors can also serve this acceleration goal.Source: §2.4 · p. 3
3. Experiments
Experiments compare the framework with existing alignment methods on LJ Speech, assessing convergence, alignment quality, human-rated speech quality, and long-utterance robustness. Results show faster or comparable convergence in tested models, sharper alignments, closer duration estimates, improved human preference, and fewer errors as utterances lengthen.
3. Experiments
- The LJ Speech experiments compare convergence, distance from annotated durations, and speech quality against architecture-specific baseline alignment methods.Baselines use existing methods for Flowtron and Tacotron 2, Tacotron 2-derived durations for FastPitch, and MFA durations for FastSpeech 2 and RAD-TTS.Source: §3 · p. 3
3.1. Convergence Rate
- The static prior significantly improves Tacotron 2 convergence; Flowtron benefits most, while parallel models converge at their forced-aligner baseline rate.The prior allows at least two Flowtron autoregressive flows to train simultaneously; adding Lalign further improves convergence and reduces training time.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2. Alignment Sharpness
- Lalign produces sharper attention distributions with more connected paths, suggesting more confident, continuous alignments.The visual inspection covers a specific validation sample.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
3.3. Duration Analysis
- Framework-derived phoneme durations converge faster and lie closer to manually annotated ground-truth durations than baseline durations.The comparison uses 10 LJ test samples and average L1 distance between extracted and annotated durations.Source: §3.3 · p. 4, Figure 4 · p. 4
3.4. Pairwise Opinion Scores
- Framework-trained models consistently improve over all baselines in human pairwise preference scores.Listeners compared same-architecture samples and rated overall quality by text accuracy, pleasantness, and naturalness.Source: Table 1 · p. 4, §3.4 · p. 4
3.5. Robustness to Errors on Long Utterances
- Autoregressive models with Lalign have lower character error rates on long utterances, while parallel models have much lower rates than autoregressive models.The evaluation synthesizes LJ Speech-trained models on 14,045 full LibriTTS sentences and uses Jasper for recognition.Source: §3.5 · p. 4, Figure 5 · p. 4
4. Conclusion
The framework applies to autoregressive and parallel TTS architectures, using forward-sum, Viterbi, and diagonal priors to support stable, fast-converging online alignment learning. Experiments report better human-rated speech quality, fewer alignment failures, faster convergence, and improved robustness on long text sequences.
4. Conclusion
- The framework combines forward-sum, Viterbi, and diagonal priors to make online alignment learning stable and fast-converging across autoregressive and parallel TTS architectures.It eliminates the need for forced aligners, which are expensive and not readily available for some languages.Source: §4 · p. 4