Source-linked AI summary
Attention-Constrained Inference for Robust Decoder-Only Text-to-Speech
Hankun Wang, Chenpeng Du, Yiwei Guo, Shuai Wang, Xie Chen, Kai Yu
TL;DR
Decoder-only TTS models can generate natural-sounding speech but may lack explicit monotonic alignment constraints, posing a robustness challenge. ACI identifies alignment-bearing attention maps and constrains them during inference without changing training. On VALL-E, it reduced WER by up to 20.5% relatively while naturalness and speaker similarity were comparable.
Problem
Decoder-only TTS models lack explicit monotonic alignment constraints, and the gap between teacher-forced training and autoregressive inference makes robustness challenging.
Source: §1 · p. 1Method
ACI uses Attention Sweeping to identify alignment-emerged attention maps and applies constraining masks to them at inference without changing training.
Source: §1 · p. 2Results
On VALL-E models of various configurations, ACI reduced synthesized-speech WER by up to 20.5% relatively while naturalness and speaker similarity were comparable.
Source: §1 · p. 2Takeaways & Limitations
Decoder-only TTS models can learn alignment-like structures that can be located and exploited, providing a basis for potential work such as streaming generation.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Recent popular decoder-only text-to-speech models are known for their ability of generating natural-sounding speech. However, such models sometimes suffer from word skipping and repeating due to the lack of explicit monotonic alignment constraints. In this paper, we notice from the attention maps that some particular attention heads of the decoder-only model indicate the alignments between speech and text. We call the attention maps of those heads Alignment-Emerged Attention Maps (AEAMs). Based on this discovery, we propose a novel inference method without altering the training process, named Attention-Constrained Inference (ACI), to facilitate monotonic synthesis. It first identifies AEAMs using the Attention Sweeping algorithm and then applies constraining masks on AEAMs. Our experimental results on decoder-only TTS model VALL-E show that the WER of synthesized speech is reduced by up to 20.5% relatively with ACI while the naturalness and speaker similarity are comparable.
1. INTRODUCTION
Decoder-only TTS can lack explicit monotonic alignment constraints, making robustness challenging. The paper identifies alignment-bearing attention heads and constrains them at inference without changing training.
1. INTRODUCTION
- Decoder-only TTS lacks explicit monotonic alignment constraints, and the gap between teacher-forced training and autoregressive inference makes robustness challenging.Source: §1 · p. 1
- Some heads in a particular early decoder layer show diagonal speech-to-text attention patterns that indicate alignment; the paper names their maps AEAMs.Source: §1 · pp. 1–2
- ACI uses Attention Sweeping to identify AEAMs, then applies constraining masks at inference to guide monotonic synthesis without changing the model or training process.Source: §1 · p. 2
- On VALL-E models of various configurations, ACI reduced synthesized-speech WER by up to 20.5% relatively while naturalness and speaker similarity remained comparable.Source: §1 · p. 2
2. ATTENTION-CONSTRAINED INFERENCE
ACI identifies attention heads whose maps track text–speech alignment, then constrains those maps during inference to guide monotonic synthesis without changing or retraining the model.
2.1. Notations
- The notation distinguishes discrete speech tokens y from text tokens x, usually phonemes, and defines E over equal-length sequences.The supplied definition requires sequence length T to be at least t.Source: §2.1 · p. 3
2.2. Attention Sweeping
- Attention Sweeping runs ground-truth text–speech pairs through the decoder, then scores normalized attention maps for concentration and monotonic alignment to identify AEAMs.The alignment cost uses a forced-alignment reference during detection; that reference is not needed for subsequent attention-constrained inference.Source: §2.2 · p. 3
2.3. Attention Constraining
- When synthesized speech diverges from text, AEAMs can misalign repeated phonemes and steer later generation nonmonotonically, motivating masks around the preceding token’s attention center.The mask leaves a neighborhood around the selected center unmasked; its radius is correlated with the map’s entropy cost.Source: §2.3 · p. 4
- The dynamic-programming center strategy uses information from preceding rows and is more stable than argmax at preserving monotonicity when locating an attention center.Argmax can fall back when preceding attention values are diffuse; the DP strategy selects the position with minimum DP cost.Source: §2.3 · p. 4
- Combining argmax or DP center selection with last-row or history-kept masking yields four attention-constraining strategies.The last-row strategy masks only the current row, while history-kept retains the mask across rows.Source: §2.3 · p. 4
3. EXPERIMENT
Across four VALL-E models, DP attention-constraining strategies reduce WER while preserving reported speech-quality metrics. The experiments identify early-layer AEAMs and find that masking all AEAMs below the threshold is the most robust ablation choice.
3.1. Setup
- Four VALL-E models span three sizes and LibriTTS/LibriLight training sets, using Encodec at 16 kHz with 8 RVQ indices per frame.Source: §3.1 · pp. 4–5, Table 1 · p. 5
3.2. Attention Sweeping
- Each model has 1–4 AEAMs in its second or third layer, below τ = 1 on CE/CA plots; these shallow alignments may guide deeper inference.Source: §3.2 · p. 5, Figure 4 · p. 5
3.3. Robust Generation
- 13.4%–20.5% WER reductions across each model with DP strategies; deletions, insertions, and substitutions also fall versus baseline.Source: §3.3 · p. 5, Table 2 · p. 6
- ACI with DP shows no degradation in MCD, P-MOS, or SECS; subjective naturalness is slightly higher or on par with baseline.Source: §3.3 · p. 6, Table 3 · p. 6
3.4. Ablation Study
- For STD-585h with DP/history kept, masking all AEAMs below τ = 1 is more robust than masking only the lowest-cost head or all second-layer maps.Source: §3.4 · p. 6, Table 4 · p. 6
4. CONCLUSIONS
The paper introduces training-free Attention-Constrained Inference (ACI) to improve decoder-only TTS robustness by identifying alignment-related attention maps and constraining them. It reports that decoder-only TTS models can learn alignment-like structures, with potential uses including streaming generation and adding signals to attention during training.
- ACI is a training-free inference method introduced to improve VALL-E synthesis robustness.Source: §4 · p. 6
- Attention Sweeping identifies and filters alignment-related AEAMs before attention constraints mitigate synthesis errors.Source: §4 · p. 6
- Decoder-only TTS models can unsupervisedly learn alignment-like structures, with potential applications including streaming generation or adding signals to attention during training.Source: §4 · p. 6