Source-linked AI summary
Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, Tom Bagby
TL;DR
Attention-based end-to-end TTS can produce natural speech, but alignment failures remain a challenge, particularly for longer utterances. The paper compares location-relative GMM and additive-energy mechanisms, modifies GMM attention, and introduces DCA. It reports that DCA and GMMv2b generalize to utterances much longer than those seen in training while preserving naturalness on shorter utterances.
Problem
Attention-based end-to-end TTS can produce human-rivaling naturalness in some domains, but alignment failures can cause missing or repeated words, incomplete synthesis, and difficulty with longer utterances.
Source: §1 · p. 1Method
The paper compares content-based and location-sensitive attention with location-relative GMM mechanisms and introduces DCA, a purely location-based additive-energy mechanism with dynamic filters and a prior filter.
Source: §1 · p. 1, §2.3 · p. 2Results
DCA and GMMv2b generalize to utterances much longer than those seen during training while preserving naturalness on shorter utterances.
Source: §4 · p. 4Takeaways & Limitations
The authors suggest these mechanisms may support paragraph- or long-sentence synthesis with improved naturalness and continuity compared with stitching separately synthesized sentences or clauses.
Source: §4 · p. 4GMM attention's infinite-support mixture distributions are not necessarily monotonic and may allow backward emphasis or expanded variance, potentially hurting alignment stability.
Source: §2.4 · p. 2
Abstract
from arXiv · showhide
Despite the ability to produce human-level speech for in-domain text, attention-based end-to-end text-to-speech (TTS) systems suffer from text alignment failures that increase in frequency for out-of-domain text. We show that these failures can be addressed using simple location-relative attention mechanisms that do away with content-based query/key comparisons. We compare two families of attention mechanisms: location-relative GMM-based mechanisms and additive energy-based mechanisms. We suggest simple modifications to GMM-based attention that allow it to align quickly and consistently during training, and introduce a new location-relative attention mechanism to the additive energy-based family, called Dynamic Convolution Attention (DCA). We compare the various mechanisms in terms of alignment speed and consistency during training, naturalness, and ability to generalize to long utterances, and conclude that GMM attention and DCA can generalize to very long utterances, while preserving naturalness for shorter, in-domain utterances.
1. INTRODUCTION
Attention-based end-to-end TTS can produce highly natural speech in some domains, but alignment failures threaten completeness and long-utterance synthesis. This paper compares location-relative attention mechanisms and introduces modifications intended to improve alignment and long-utterance generalization.
1. INTRODUCTION
- Attention-based sequence-to-sequence models predominate in end-to-end TTS and can achieve human-rivaling naturalness for certain domains, but alignment failures cause missing or repeated words and incomplete synthesis.Such failures also include difficulty generalizing to longer utterances.Source: §1 · p. 1
- Tacotron 1 uses purely content-based attention, while Tacotron 2 adds location-sensitive features and can generalize beyond training utterance lengths.The paper characterizes the original content-based mechanism as among the least stable choices because it does not exploit TTS alignment's monotonicity and locality.Source: §1 · p. 1
- Monotonic alignment methods have shown faster training alignment, improved stability and naturalness, and fewer synthesis errors, but may require recursive computation, training hacks, or lower-quality hard-alignment inference.The reported trade-offs include reduced efficiency from marginalizing alignments recursively and decreased quality in hard alignment mode.Source: §1 · p. 1
- Prior work explored purely location-based GMM attention and softplus stabilization, but no systematic comparison of these design choices had been made.The paper frames its comparison as covering both Tacotron mechanisms and location-relative alternatives.Source: §1 · p. 1
- The paper compares Tacotron attention mechanisms with location-relative alternatives, modifying GMM attention and introducing Dynamic Convolution Attention (DCA) as a purely location-based mechanism.The comparison covers alignment speed and consistency, speech naturalness, and long-utterance generalization.Source: §1 · p. 1
2. TWO FAMILIES OF ATTENTION MECHANISMS
The section describes how attention alignments produce decoder context, then presents GMM-based and additive energy-based mechanisms, including Dynamic Convolution Attention (DCA).
2.1. Basic Setup
- The decoder uses alignment weights to form a context vector from encoder states, then combines that context with its attention-RNN state to produce an output.The attention mechanism computes the alignment at each decoder step.Source: §2.1 · p. 1
2.2. GMM-Based Mechanisms
- GMM attention forms encoder weights from an unnormalized mixture of K Gaussians whose means advance recurrently, potentially monotonically when offsets are positive.The mixture parameters are computed from the attention-RNN state.Source: §2.2 · p. 2
- V1 normalizes mixture weights and components and uses exponentials; V2 uses softplus for offset and standard deviation, with tested biases targeting Δ_i=1 and σ_i=10.The biases are applied to intermediate parameters to encourage useful final values at initialization.Source: §2.2 · p. 2, Table 1 · p. 2
2.3. Additive Energy-Based Mechanisms
- Additive energy mechanisms compute attention weights by softmaxing MLP energies; location-sensitive attention uses convolutional features from the previous alignment.The generalized formulation also includes content-based query/key comparisons.Source: §2.3 · p. 2, Table 2 · p. 2
2.4. Dynamic Convolution Attention
- DCA removes content-based query/key terms, adds state-conditioned dynamic filters, and uses a causal prior filter to bias alignment toward short forward steps.The dynamic filters adjust alignment relative to the previous step, while the prior filter allows only forward progression.Source: §2.4 · p. 2, §2.3 · p. 2
- DCA's beta-binomial prior uses α=0.1 and β=0.9 on length-11 taps, encouraging average forward movement of one encoder step per decoder step as uncertainty grows.The experiments use the same prior parameters across datasets.Source: §2.4 · p. 3
3. EXPERIMENTS
The experiments compare attention mechanisms on training alignment, in-domain naturalness, and long-utterance robustness using the Lessac and LJ datasets.
3.1. Experiment Setup
- Tests vary only attention in Tacotron across Lessac (49,852 training utterances, 37 hours, up to 5 seconds) and LJ (12,764 utterances, 23 hours, up to 10 seconds).Models trained for 300,000 steps; DCA used eight static and eight dynamic filters of length 21 and a length-11 prior filter.Source: §3.1 · p. 3
3.2. Alignment Speed and Consistency
- On Lessac, DCA and GMMv2b align fastest; on harder LJ, LSA succeeds in all 10 trials but is slower, while DCA and GMMv2b each fail once.On Lessac, adding a bias hurts GMMv1 consistency but helps GMMv2.Source: Figure 2 · p. 3, §3.2 · pp. 3–4
3.3. In-Domain Naturalness
- In-domain MOS is around 4.3 for LSA, DCA, and GMMv2b, with equivalent scores above CBA, which scores lower due to occasional catastrophic attention failures.The scores are crowd-sourced ratings on a 1–5 scale, where 5 represents completely natural speech.Source: Table 3 · p. 4, §3.3 · p. 4
3.4. Generalization to Long Utterances
- DCA and GMMv2b generalize across the tested 10–299-word utterances, while CBA degrades beyond training lengths and LSA rises around 3x the maximum training length.The models were trained on utterances up to 5 seconds for Lessac and 10 seconds for LJ.Source: Figure 3 · p. 4, §3.4 · p. 4
4. DISCUSSION
DCA and GMMv2b generalize to utterances much longer than those seen in training while preserving naturalness on shorter utterances. Their practical appeal includes fast alignment and DCA-specific advantages for bounded receptive fields and alignment stability.
4. DISCUSSION
- DCA and GMMv2b generalize to much longer-than-training utterances while preserving naturalness on shorter utterances, supporting paragraph- or long-sentence synthesis.The authors say this could improve naturalness and continuity over synthesizing and stitching together separate sentences or clauses.Source: §4 · p. 4
- The two location-relative mechanisms avoid dynamic programming and tend to align quickly, making occasional failures easy to detect and training restarts feasible.LSA seemed more consistent despite slower alignment; the authors note that slower alignment can sometimes lead to worse-quality models, probably because other components remain unaligned longer.Source: §4 · p. 4
- Compared with GMMv2b, DCA more easily bounds its receptive field for hard-windowing optimizations and uses normalized weights that help stabilize coarse-grained alignment.Its prior filter numerically disallows backward or excessive forward movement.Source: §4 · p. 4
- The authors conclude that location-relative attention merits further study for monotonic alignment tasks such as TTS and speech recognition..Source: §4 · p. 4