Source-linked AI summary
AlignDPO: Preference-Gated Alignment for Reducing Hallucination in Decoder-Only TTS
Xiao Zhou, Oisín Turbitt, Kit Bower-Morris, Jonathan Carlton, Jamie Stacey, Kris Y. Hong
TL;DR
Decoder-only TTS models can hallucinate content when weak text–speech alignment undermines autoregressive generation. AlignDPO folds a preference-gated CTC alignment term into DPO, and on Seed-TTS-Eval English it reduces hallucination and word error rates relative to DPO while lowering severe hallucinations to ∼0.6%.
Problem
Decoder-only TTS models remain prone to omissions, repetitions, and fabricated content words associated with weak text–speech alignment during autoregressive generation.
Source: §1 · p. 1Method
AlignDPO identifies naturally emerged alignment heads with a CTC criterion and folds a lightweight CTC term into DPO for chosen samples only.
Source: §1 · p. 1, §3.1 · p. 2Results
∼0.6%: AlignDPO lowers severe hallucination from 4.4% and significantly improves hallucination and word error rates over the DPO baseline on Seed-TTS-Eval English.
Source: §5.2 · p. 5, §6 · p. 6Takeaways & Limitations
Hallucination robustness is best at moderate alignment-head sharpness; over-sharpening is no better than the backbone and is less robust to ACI.
Source: §1 · p. 1, §5.4 · p. 5
Abstract
from arXiv · showhide
Decoder-only text-to-speech (TTS) models scale efficiently but remain prone to content hallucinations that arise from weak text-speech alignment during autoregressive generation. We find that robustness is governed by a non-monotone relation to the sharpness of the alignment-bearing attention heads: a moderate degree is best, whereas over-sharpening is no better than the unaligned backbone and even less robust. Guided by this, we present AlignDPO, a post-training method that reaches this moderate regime by folding a lightweight connectionist-temporal-classification (CTC) alignment term into Direct Preference Optimization (DPO), applied only to the chosen samples, with no architectural or inference-time change. On the Seed-TTS-Eval English set, this significantly reduces the content-hallucination and word error rates relative to a strong DPO baseline and lowers the severe content-hallucination rate to ~0.6% (from 4.4%); a listening study further finds it preferred for naturalness over both the backbone and that baseline. Alignment is thus best learned and kept moderate rather than maximized or imposed at decoding. Audio samples are available at https://align-dpo-demo.vercel.app.
1. INTRODUCTION
Decoder-only TTS models can hallucinate content words when autoregressive generation has weak text–speech alignment. AlignDPO strengthens alignment during preference optimization and improves hallucination and word-error outcomes over a strong DPO baseline.
1. INTRODUCTION
- Decoder-only TTS models are efficient and scalable but can omit, repeat, or fabricate content words because of weak text–speech alignment during autoregressive generation.These content errors are distinguished from natural variation in fillers or function words.Source: §1 · p. 1
- Prior remedies add alignment through architectural changes, external aligners, training-time guidance, or inference-time constraints, while DPO leaves alignment structure implicit.The paper confirms that inference-time ACI helps an unaligned model.Source: §1 · p. 1
- AlignDPO post-trains a from-scratch backbone by strengthening naturally emerged alignment-bearing heads with a lightweight CTC term during preference optimization.The experiments use a 530M backbone trained on 980 hours of supervised data and post-trained on 1,600 hours of preference data.Source: §1 · p. 1
- 15.0% to 11.3%: AlignDPO reduces hallucination relative to a strong DPO baseline, lowers severe hallucination to ∼0.6%, and improves word error rate.The chosen-sample CTC term is applied at a moderate weight.Source: §1 · p. 1
2. RELATED WORK
Prior work addresses decoder-only TTS hallucinations through alignment interventions, inference-time constraints, and preference optimization. AlignDPO adds an explicit alignment constraint to preference optimization while aiming to avoid external alignment dependencies and decode-time machinery.
2. RELATED WORK
- Decoder-only codec language models generate speech tokens autoregressively but can hallucinate through skips, repetitions, and run-ons; flow-matching alternatives are outside this decoder-only focus.The related-work discussion notes that flow-matching systems achieve high robustness but abandon the target paradigm.Source: §2 · p. 2
- Alignment remedies include architectural changes, forced aligners, teacher guidance, and inference-time constraints, often adding components or external dependencies.ACI reshapes attention at decoding time, while other approaches may reduce deployment simplicity or modeling flexibility.Source: §2 · p. 2
- Existing preference-optimization methods improve TTS qualities but do not add an explicit alignment constraint targeting hallucination; AlignDPO inserts preference-gated CTC into DPO.The paper uses automated preferences and one teacher-forced criterion to identify alignment heads.Source: §2 · p. 2
3. ALIGNDPO
AlignDPO identifies naturally emerged near-monotonic attention heads with a single CTC criterion, then folds that criterion into DPO for chosen samples. This applies alignment refinement during preference optimization rather than uniformly during supervised training or at inference.
3. ALIGNDPO
- AlignDPO applies a CTC alignment constraint during preference optimization, producing DPO+M directly from the backbone rather than through an intermediate supervised-refinement model.Alternative placements considered include supervised refinement, plain DPO, and inference-time ACI.Source: §3 · p. 2
3.1. Alignment-Emerged Heads and Single-Criterion Identification
- A small set of from-scratch backbone heads develops near-monotonic speech-to-text maps, termed Alignment-Emerged Attention Maps (AEAMs).A length-normalized CTC score of each head’s row-normalized map identifies them.Source: §3.1 · p. 2
- Heads with LCTC below τ=2 are selected from a teacher-forced pass; the same CTC score identifies alignment heads and supervises refinement without an external aligner.Teacher forcing uses ground-truth text–speech pairs, preventing hallucinations from corrupting the scored maps.Source: §3.1 · p. 2
3.2. Alignment Refinement via Preference-Gated DPO
- Preference data pairs chosen and rejected generations, and the CTC term is applied only to chosen samples so rejected-sample maps receive no alignment gradient.The optimization favors generations that are both perceptually preferred and more monotonically aligned.Source: §3.2 · p. 3
4. EXPERIMENTAL SETUP
The experiments evaluate hallucination, intelligibility, speaker similarity, and perceptual quality across a from-scratch decoder-only backbone and alignment variants. They compare training placements and preference gating on Seed-TTS-Eval English and a separate hard set.
4.1. Evaluation Metrics
- Hallucination metrics count skipped, repeated, or fabricated words, with severe hallucination defined as at least four hallucinated content words per utterance.A human-validated set found ∼95% recall and ∼49% precision for HAL; human labels and WER also supported the system ordering.Source: §4.1 · p. 3
4.2. Base Model
- The 530M-parameter decoder-only Transformer is trained from scratch with cross-entropy on 980 hours of filtered GigaSpeech-M data and WavTokenizer acoustic tokens.Text is phonemized with espeak-ng; speech uses a single-codebook tokenizer at 40 tokens per second.Source: §4.2 · p. 3
4.3. Comparison Systems
- The primary comparison includes the supervised backbone, supervised CTC refinement, plain DPO, chosen-only CTC inside DPO, and inference-time ACI.The study also tests CTC-weight variants, ungated training, and initialization from the supervised-refinement model.Source: §4.3 · pp. 3–4
4.4. Training Configuration
- DPO and DPO+M are post-trained from the backbone on 1,600 hours of preference pairs, with DPO+M’s CTC weight annealed to 0.1.Supervised Base+M instead adds CTC during continued supervised training; the preference pool includes normal and challenging texts.Source: §4.4 · p. 4
4.5. Evaluation Protocol
- Evaluation uses 706 non-overlapping Seed-TTS-Eval English sentences and a separate 500-utterance hard set to stress-test preference gating and ACI.The comparisons ask where to apply alignment and how much to sharpen heads, including gated versus ungated application.Source: §4.5 · p. 4
5. RESULTS
DPO+M outperformed DPO on hallucination and WER, while moderate sharpening—not maximal sharpness—was associated with the best robustness. The results also show improved naturalness preference and changes to the model’s alignment heads.
5.1. Native Alignment Heads
- The CTC term sharpened three native alignment heads and pulled a fourth head, L4H10, toward the alignment-emerged set.DPO+M also reduced the targeted heads’ residual attention diffuseness relative to DPO.Source: §5.1 · pp. 4–5
5.2. Hallucination and Intelligibility
- DPO+M significantly outperformed DPO on hallucination and WER and reduced severe hallucinations to ~0.6%.SIM and pMOS were comparable across systems; human labels confirmed the hallucination ordering.Source: §5.2 · p. 5, Table 1 · p. 5
5.3. Applying the Constraint
- Starting DPO from an alignment-refined Base+M gave no significant hallucination improvement, and applying CTC from that over-sharpened initialization forfeited the DPO+M gain.The from-Base+M DPO+M variant reverted to DPO-level hallucination.Source: §5.3 · p. 5, Table 2 · p. 5
5.4. Increasing Sharpness
- Hallucination was lowest at moderate sharpening (λ≈0.1–0.3), while over-sharpened models hallucinated more despite lower attention entropy.The results associate hallucination with free-running alignment loss rather than attention entropy alone.Source: §5.4 · p. 5, Figure 3 · p. 5
- Applying CTC to rejected as well as chosen samples preserved in-domain HAL but regressed to DPO-level performance on the hard set; chosen-only gating performed best there.The authors attribute the regression to spurious alignments encoded from rejected samples on long, hard inputs.Source: §5.4 · p. 6, Table 3 · p. 6
5.5. Subjective Evaluation
- In a listening study of hallucination-free utterance pairs, DPO+M was preferred over both Base and DPO for naturalness.The study included 15 listeners and 30 utterances.Source: §5.5 · p. 6, Figure 4 · p. 6
6. CONCLUSION
AlignDPO uses preference-gated CTC alignment in DPO to reduce decoder-only TTS content hallucinations. The conclusion reports improved Seed-TTS-Eval English outcomes and argues alignment should remain moderate, while limiting evidence to one backbone, codec, and corpus.
6. CONCLUSION
- The severe hallucination rate falls from 4.4% to ∼0.6%, with word error and naturalness improving over a DPO baseline.Source: §6 · p. 6
- AlignDPO folds a preference-gated CTC alignment term into DPO, applied only to chosen samples without architectural change, an external aligner, or a teacher.Source: §6 · p. 6
- Chosen-only gating preserves the gain out of domain, and the paper concludes alignment is best kept moderate rather than maximized or imposed at decoding.Source: §6 · p. 6
- The reported scope is one backbone, codec, and corpus.Source: §6 · p. 6