Eliminating stability hallucinations in llm-based tts models via attention guidance

1. INTRODUCTION

• LLM-based TTS lacks explicit text–speech alignment and can repeat speech endlessly or omit intended segments on long or hard texts.These failures occur despite stable synthesis in common text scenarios.Source: §1 · p. 1
• Earlier alignment constraints rely on cross-attention, hard stepwise monotonic attention can degrade naturalness, and accurate forced-alignment labels are difficult to obtain at scale.The paper notes these approaches are incompatible with decoder-only LLM-based TTS, or limited in application by label availability.Source: §1 · p. 1
• OAS and attention-guided training are proposed to reduce stability hallucinations in LLM-based TTS without forced-alignment labels or compromised naturalness.The paper reports that OAS is integrated into CosyVoice2 training to learn continuous, stable alignment; pre-trained attention values also guide student training.Source: §1 · pp. 1–2
• The Viterbi algorithm takes an alignment-probability matrix as input and returns an optimal alignment path using dynamic programming and backtracking.The algorithm initializes scores, selects the maximum-scoring endpoint, and traces back the path.Source: §1 · p. 2

2.1. Attention analyses based on CosyVoice2

• Middle-layer attention heads focus on text–speech alignment, with global forward paths resembling cross-attention; the authors hypothesize these heads most affect alignment.The analysis uses CosyVoice2 with a 24-layer, 14-head-per-layer Qwen-0.5B backbone.Source: §2.1 · p. 2

2.2. Metric for Alignment Quality: OAS

• OAS measures attention probability along a Viterbi-selected optimal alignment path relative to total attention scores in the alignment region.The path maximizes the sum of attention probabilities and OAS is intended to reflect global alignment continuity and monotonicity.Source: §2.2 · p. 2
• 0.638 is the reported correlation coefficient between OAS and word error rate in synthesized speech.The authors report that OAS reflects alignment continuity and monotonicity across text and speech tokens.Source: §2.2 · p. 2

2.3. Incorporate OAS into the training of CosyVoice2

• The authors infer alignment heads are primarily in layers 8 and 9, then restrict assigned heads to the alignment region and apply OAS regularization.OAS retains forward gradients, but head-function randomness prevents directly applying its loss to alignment heads without this selection and masking strategy.Source: §2.3 · pp. 2–3

2.4. Attention-Guided Training for Student CosyVoice2

• For student training, the teacher’s highest-OAS head supplies pseudo-alignment labels; sparse text-token targets and progress-bar prediction provide CoT supervision.Each text token is marked once in the sparse target, avoiding boundaries when possible; progress-bar loss combines L1 and first-order difference losses at unmasked positions.Source: §2.4 · p. 3

3.1. Experimental Setup

• The evaluation used Seed-TTS-Eval and CV3-Eval subsets for hard and common text scenarios and compared standard CosyVoice models with OAS and attention-guided variants.WenetSpeech4TTS was used to scratch-train the LLM; the evaluated variants included CV2 M, CV2 OAS, and CV2 AG.Source: §3.1 · p. 3

3.3. Effectiveness of Incorporating OAS into CosyVoice2

• CV2 OAS applying LOAS lowered WER by 2.1% and 1.6% on hard Seed-TTS-Eval and CV3-Eval, and by 0.1% and 0.2% on common scenarios.SIM and UTMOS showed no decrease. Table 1 reports objective metrics on Seed-TTS-Eval and CV3-Eval.Source: Table 1 · p. 4, §3.3 · p. 4
• Under common text scenarios, CV OAS slightly improved subjective speech naturalness in the MOS test.The test used 30 randomly selected sentences from each dataset and 15 native Mandarin speakers; Table 2 reports MOS with 95% confidence.Source: Table 2 · p. 4, §3.3 · p. 4

3.2. Analysis of the OAS metric

• OAS correlated with WER at 0.638, supporting its use to supervise text-speech alignment; the analysis averaged the five highest-scoring attention heads.The analysis used 400 hard text sentences synthesized by CV2.Source: §3.2 · p. 4

3.4. Effectiveness of Attention-Guided Training

• Sparse text-token guidance outperformed full-token guidance, while adding progress-bar supervision further reduced misalignment and stability hallucinations.Table 3 reports text-token supervision accuracy on the training and validation sets; sparse tokens address pseudo-alignment accuracy and boundary ambiguity.Source: §3.4 · p. 4, Table 3 · p. 4

• OAS loss and attention-guided training reduce CosyVoice2 stability hallucinations under hard text scenarios without additional negative effects.Source: §4 · p. 4