Source-linked AI summary
Taming Long-form Text-to-Speech
Rongxiang Wang, Berkin Durmus, Aysegul Orhon, Eduardo Pacheco, Atila Orhon
TL;DR
Long-form TTS reliability declines even when models perform well on short prompts, limiting their use with extended text and reference audio. LACI detects alignment failures during generation and recovers through rollback and temporary attention guardrails. It substantially reduces long-prompt WER, while wSIM measures speaker instability across local windows.
Problem
Qwen3-TTS and VoxCPM2 have high content accuracy on short prompts but deteriorate on long prompts, limiting the use of long reference audio for speaker similarity.
Source: §1 · p. 1Method
LACI monitors attention alignment to detect skips and hallucinations, then rolls back to the failure onset and regenerates with temporary attention guardrails.
Source: §1 · p. 1, §3.2 · p. 2, §3.3 · p. 2Results
Qwen3-TTS-0.6B worst-of-N WER reaches 35.2% beyond 1500 words without guardrails, while LACI keeps it between 3.3% and 5.4% across prompt lengths.
Source: §5 · p. 3Takeaways & Limitations
LACI closes the long-form reliability gap for Qwen3-TTS-0.6B and carries over to Qwen3-TTS-1.7B and VoxCPM2, enabling voice cloning to benefit from long reference audio.
Source: §7 · p. 4On VoxCPM2 prompts beyond 1300 words, skip and hallucination failures resist regeneration; LACI reduces their fraction from 70% to 57%.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
1. INTRODUCTION
Autoregressive neural codec language models have made state-of-the-art TTS widely useful, but their reliability declines on long prompts. The paper attributes the short-to-long-form gap to attention misalignment and proposes detecting and recovering from errors during generation.
Motivation
- Autoregressive neural codec language models power recent state-of-the-art TTS systems [2] [3] [4].Source: §1 · p. 1
- Qwen3-TTS and VoxCPM2 lose content reliability as prompts grow, limiting the benefit of long reference audio for speaker similarity.The paper links reliable long-form generation to consistent timbre and prosody in use cases such as audiobook narration and voice agents.Source: §1 · p. 1
Contributions
- LACI detects failures from attention weights during streaming, then rolls back and applies temporary attention guardrails to recover across model sizes and families.The paper attributes the reliability gap to audio-text attention misalignment in Qwen3-TTS-0.6B, Qwen3-TTS-1.7B, and VoxCPM2.Source: §1 · p. 1
2. RELATED WORK
Prior work controls TTS alignment during training or uses inference-time candidate selection and always-on attention masks. LACI builds on attention-constrained inference by detecting failures and applying guardrails selectively.
Training-time methods
- Training-time approaches bias attention toward monotonic progress or supply explicit text-audio alignment, but require training intervention and may challenge training stability.Examples include Tacotron 2 [5], duration models and forced-alignment objectives [6], Kokoro, and neural codec language models [8] [9] [10].Source: §2 · pp. 1–2
Inference-time methods
- Fig. 2 depicts emergent alignment heads mapping corresponding audio and text tokens, with patterns consistent across text and speaker prompts.Source: Figure 2 · p. 2
- Inference-time seed selection scales cost linearly with candidates and cannot fix failures that persist across seeds.Always-on ACI [13] can introduce errors where the baseline is correct and cannot target persistent failures without detection.Source: §2 · p. 2
3. METHOD
The method identifies skip and hallucination failures through deviations in audio-text alignment, then detects them from attention coverage during streaming. It recovers by rolling back and temporarily constraining attention.
3.1. Failure Definition
- A skip is a contiguous deletion of at least 10 words, while a hallucination is a contiguous mix of substitutions and insertions of at least 20 words.These are the two dominant failure modes identified in the long-form reliability gap.Source: §3.1 · p. 2
3.2. Failure Detection
- Alignment heads track corresponding text and audio positions; their reading position departs from its expected pace at skip or hallucination onset.The alignment pattern is consistent across prompts and speakers during high-accuracy sessions and breaks during low-accuracy sessions.Source: §3.2 · p. 2
- The detector bins prompt tokens, flags skips when a passed bin received too little attention and hallucinations when reading stalls, and ends generation once all bins receive their share.Detection occurs within a few seconds of output audio; reported latency is independent of hardware.Source: §3.2 · p. 2
3.3. Failure Recovery
- LACI rolls back to the detected failure onset, reseeds, and temporarily applies a hard attention mask until generation passes the detection point.Persistent failures can trigger repeated recovery attempts, capped to bound computational overhead.Source: §3.3 · p. 2, Figure 3 · p. 3
4. EXPERIMENTS
The experiments evaluate content accuracy and speaker similarity across controlled prompt lengths, RNG seeds, and reference-audio durations. They also introduce a sliding-window speaker metric to reveal local instability that aggregate similarity can miss.
Content-accuracy evaluation
- The 144-prompt evaluation uses 10 RNG seeds per system and extends AppTek call sessions into length buckets while keeping domain and prompt difficulty constant.The evaluated systems are Qwen3-TTS-0.6B-CustomVoice, Qwen3-TTS-1.7B-CustomVoice, and VoxCPM2.Source: §4 · p. 3
Voice-cloning evaluation
- Voice-cloning tests vary reference audio up to 120 seconds while holding generated text fixed and evaluating Qwen3-TTS-0.6B-Base across 10 seeds.Source: §4 · p. 3
Evaluation metrics
- wSIM uses 8-second windows with a 4-second stride to expose short spans of speaker instability that global SIM can miss.Worst-of-N wSIM is the lowest score across windows and seeds; the lowest scores were manually verified as identity switches or severe distortions.Source: §4 · p. 3
Content-accuracy evaluation
- Table 2 evaluates mean and worst-of-N WER for Qwen3-TTS-0.6B on AppTek Call Center prompts across 10 RNG seeds, comparing baseline, ACI, and LACI.The baseline is the reference Qwen3-TTS implementation without guardrails; LACI improves mean and worst-case reliability at every length.Source: Table 2 · p. 3
5. RESULTS
LACI closes the long-prompt reliability gap for Qwen3-TTS-0.6B and improves reliability across additional models and long-reference voice cloning. Always-on ACI leaves an upper tail and can worsen short-prompt results.
Long-form reliability
- For Qwen3-TTS-0.6B, beyond 1500 words baseline worst-of-N WER reaches 35.2%, while LACI keeps worst-of-N WER between 3.3% and 5.4% at every prompt length.Source: §5 · p. 3
- On 146 baseline failure cases, ACI lowers median WER but retains a high upper tail, while LACI brings every case into a low, tight band.LACI matches the baseline on short prompts and matches or improves on ACI at every prompt length; ACI’s always-on mask raises short-prompt worst-of-N WER from 5.4% to 6.0%.Source: §5 · pp. 3–4, Figure 4 · p. 4
Generalization
- Qwen3-TTS-1.7B worst-of-N WER falls below 4.5% in both longest prompt buckets with LACI, but VoxCPM2 failures beyond 1300 words remain largely unrepaired.For VoxCPM2 beyond 1300 words, the failure fraction falls from 70% to 57% with LACI; the passage identifies a model-coherence limit rather than a detection gap.Source: §5 · p. 4
Voice cloning
- At 120 seconds of reference audio, baseline worst-of-N wSIM falls to 0.01 while LACI reaches 0.47, and catastrophic failure rates fall from 26% to below 1%.The passage notes that whole-clip SIM can hide short spans of identity switching or distortion that wSIM exposes.Source: §5 · p. 4
6. FUTURE WORK
The paper proposes using LACI-generated high- and low-quality speech as reinforcement-learning training data. Its results also identify settings where guardrails generalize and where reliability remains limited.
Future direction
- LACI could supply high- and low-quality speech rollouts for reinforcement-learning environments, with the aim of training more reliable TTS models.The authors invite further exploration of this direction.Source: §6 · p. 4
Generalization boundary
- Beyond 1300 words, neither guardrail fully repairs VoxCPM2, which the authors attribute to a model-coherence limit rather than a detection gap.Both guardrails repair the 1000–1300-word range.Source: Table 3 · p. 4
Voice cloning
- At reference lengths up to 120 seconds, LACI keeps catastrophic failure rates below 5%, while the baseline rate grows from 5% to 26%.The figure also reports improvements in worst-of-N SIM and sustained high worst-of-N wSIM with LACI.Source: Figure 5 · p. 4
7. CONCLUSION
The conclusion attributes long-form reliability failures to audio-text misalignment on emergent alignment heads. LACI monitors those heads and repairs detected failures through rollback and temporary guardrails.
Conclusion
- Long-form reliability gaps in state-of-the-art autoregressive TTS models are explained by audio-text misalignment on emergent alignment heads.The conclusion describes this as the explanation for the gap.Source: §7 · p. 4
- LACI detects skips and hallucinations within seconds by monitoring alignment heads, then repairs them through rollback and regeneration under temporary guardrails.The conclusion reports negligible overhead for this process.Source: §7 · p. 4
- The conclusion reports that LACI closes the reliability gap for Qwen3-TTS-0.6B, carries over to Qwen3-TTS-1.7B and VoxCPM2, and supports long-reference voice cloning.Long reference audio can then benefit voice cloning rather than being a source of failure.Source: §7 · p. 4
9. COMPLIANCE WITH ETHICAL STANDARDS
The study uses only the publicly released AppTek Call Center dataset and collects no new human-subjects data.