Source-linked AI summary
Likelihood-Constrained Acoustic Reranking for Training-Free Hallucination Mitigation in LLM-Based ASR
Jiasheng Kuang, Linru Zheng, Hongjin Song, Zhaoqi Cui, Song Li
TL;DR
LLM-based ASR can let linguistic and task priors override acoustic evidence, producing several hallucination failure modes. The paper proposes LCAR, which likelihood-constrains acoustic reranking to plausible alternatives, and evaluates it across four systems and two human-audited suites. LCAR reduces detector-identified hallucinations while largely preserving standard-test recognition performance.
Problem
LLM-based ASR linguistic and task priors can override available acoustic evidence, causing translation, instruction execution, repetition, or catastrophic deletion.
Method
LCAR first restricts candidates by base-model likelihood and then selects among them using attention-derived acoustic compatibility from the existing LM head.
Results
LCAR reduces detector-identified hallucinations across four LLM-based ASR systems and largely preserves WER/CER on standard open-source test sets.
Takeaways & Limitations
Likelihood-constrained acoustic intervention can improve acoustic grounding while retaining support from the base model without inference-time detector, reference, auxiliary model, or additional training.
Takeaways & Limitations
Counted hallucination events must differ from the same-run greedy-decoding output after text normalization.
Abstract
from arXiv · showhide
Large language model (LLM)-based automatic speech recognition (ASR) systems achieve strong performance on conventional speech data by leveraging powerful linguistic priors and multilingual capabilities. However, under challenging conditions, these priors can override acoustic evidence, resulting in unintended translation, instruction execution, repetition, or catastrophic deletion. We propose Likelihood-Constrained Acoustic Reranking (LCAR), a training-free decoding method that improves acoustic grounding while preserving support from the base model. At each decoding step, LCAR first retains tokens whose base-model likelihood falls within a margin of the greedy token, then reranks them using an acoustic compatibility score computed from attention-pooled audio embeddings and the existing LM head. By restricting acoustic intervention to plausible, model-supported alternatives, LCAR requires no additional training, external detector, reference transcript, or auxiliary model at inference. We evaluate LCAR on four LLM-based ASR systems using human-audited TTS and open-source speech challenge suites. At $δ=0.60$, LCAR removes 38.8--57.1\% of detector-identified hallucination failures while largely maintaining WER/CER on standard open-source test sets.
1. INTRODUCTION
LLM-based ASR systems can let linguistic and task priors override acoustic evidence, producing translation, instruction execution, repetition, or deletion failures. LCAR addresses these failures by constraining acoustic reranking to model-supported alternatives without additional inference-time training or auxiliary components.
- LLM-based ASR linguistic priors can override transcription, causing code-switched phrases to be translated or audible commands to be executed.
- LCAR admits only tokens within δ nats of the greedy token before using acoustic compatibility to rerank them.At δ = 0, it exactly matches greedy decoding.
- LCAR requires no detector, reference transcript, auxiliary model, or additional training at inference.
- The paper contributes a grounding-based account of translation, instruction execution, unsupported repetition, and catastrophic deletion in LLM-based ASR.
- The evaluation uses two human-audited 500-utterance challenge suites: one TTS-based and one based on open-source speech.Both suites are publicly available on Hugging Face.
2. BENCHMARK CONSTRUCTION
The benchmark defines acoustic-grounding hallucinations as decoding trajectories where linguistic or task priors override available acoustic evidence, and evaluates them through human-audited challenge suites and semantic screening. The construction distinguishes failure categories and applies explicit exclusion and adjudication rules.
- 2.1. Hallucination Categories: Acoustic-grounding hallucinations are defined as decoding trajectories in which linguistic or task priors override available acoustic evidence.Categories include translation or transliteration, instruction execution, unsupported repetition, and catastrophic deletion.
- 2.2. Challenge Suites: The suites combine IndexTTS2 Mandarin-English utterances with OpenSpeech code-switched audio and text-based or spoken instruction attacks.Each released suite contains 500 human-audited source utterances.
- 2.3. Semantic Screening: Semantic screening uses two-stage adjudication to align references and hypotheses, assign provisional categories, verify criteria, and assess Prompt-HAL attribution.
- 2.3. Semantic Screening: Translation and transliteration require meaning-bearing spans to cross language or script boundaries, while instruction execution requires following an attack instead of transcribing it.
- 2.3. Semantic Screening: Formatting differences, conventional names or acronyms, and local sound-alike substitutions are excluded from counted hallucination events.An event must also differ from the same-run greedy-decoding output after text normalization.
- 2.3. Semantic Screening: Human review found 93.3% precision among 400 detector positives, with 27 rejected cases classified as ordinary ASR errors.Stratum-weighted precision was 93.9% with a 95% confidence interval of 93.1–94.7%.
3. METHOD
LCAR combines base-model likelihood filtering with acoustic reranking, using attention-pooled audio representations and the existing LM head to select among plausible tokens. The method constrains intervention relative to the greedy token while preserving the option of unchanged greedy decoding.
- Acoustic compatibility: LCAR uses the existing LM head both for base-model next-token probabilities and for acoustic compatibility scores from pooled audio context.The two operations share output-projection weights but operate on decoder states and attention-pooled acoustic context, respectively.
- Acoustic compatibility: The audio context is formed by averaging attention from the generated position to audio positions across retained final layers and heads.Attention weights are renormalized over audio positions before producing the stepwise pooled vector.
- Likelihood-constrained decoding: LCAR first forms C_t(δ) from tokens whose base-model log likelihood is within δ nats of the greedy token.The candidate set limits acoustic intervention to alternatives with model support.
- Likelihood-constrained decoding: It then selects the acoustically most compatible token from the admitted candidate set.At δ = 0, the candidate set contains only the greedy token, so LCAR reduces to greedy decoding.
- Implementation: LCAR reuses encoder states and adds attention averaging over n layers plus one LM-head projection per decoding step.EOS is never overridden, and deletion is repaired only through an earlier prefix change.
4. EXPERIMENTAL SETUP
The evaluation uses four LLM-based ASR systems, matched challenge examples, standard speech sets, and detector-transition metrics to assess hallucination mitigation and recognition costs.
- Four systems are evaluated with official prompts and preprocessing, using n = 4 and eager attention extraction in the main sweep.
- Each track contains 400 matched-greedy positives and 400 negative controls for paired evaluation.
- Normal-speech evaluation uses 2,620 LibriSpeech test-clean and 5,000 AISHELL2 utterances, scored by WER and CER.
- The setup defines mixed-language units from Han characters, casefolded English words, and digit strings, with substitutions, deletions, and insertions measured against reference units.
- HIR-Fix and HIR-Induce measure positive-to-negative and negative-to-positive detector transitions, respectively.
- At δ = 0.60, the likelihood margin retains tokens with at least e−0.60 = 54.9% of the greedy token probability.
5. RESULTS AND DISCUSSION
LCAR produces positive hallucination-reduction gains across model–suite pairs while keeping conservative-margin recognition changes small, and ablations identify the likelihood constraint as essential.
- 5.1. Main Results: At δ = 0.60, NetHIR ranges from 5.7 to 43.7 points on IndexTTS2 and from 52.8 to 70.9 points on OpenSpeech.
- 5.1. Main Results: NetHIR is positive for every model–suite pair, generally increasing as larger margins admit more near-greedy alternatives.
- 5.2. Normal-Speech Evaluation: At δ = 0.60, WER/CER changes are +0.002%/−0.034% for Qwen3, −0.032%/−0.154% for Kimi, +0.105%/+0.129% for Qwen2 Base, and +0.007%/+0.085% for GLM.
- 5.3. Ablation and Repair Analysis: Last-4 aggregation attains 53.1% HIR-Fix, versus 51.6% for Last-2, 48.5% for mean pooling, and 41.0% for the final layer.
- 5.3. Ablation and Repair Analysis: The likelihood constraint is essential: LC-DoLa reaches 46.0% HIR-Fix, whereas acoustic-only selection causes more than 600/1,000 percentage-point WER/CER increases.
- 5.3. Ablation and Repair Analysis: Among 3,997 detected events, 28.8% yield faithful recovery and 52.0% ordinary ASR errors, so detector exits are not synonymous with exact recovery.
6. CONCLUSION
The paper identifies acoustic-grounding hallucination patterns and proposes LCAR, a training-free decoder that reduces detector-identified hallucinations while largely preserving standard recognition performance.
- The conclusion identifies unintended translation or transliteration, instruction execution, unsupported repetition, and catastrophic deletion as acoustic-grounding hallucination patterns.
- LCAR restricts candidates using base-model likelihood and selects among them with attention-derived acoustic compatibility.
- Across four LLM-based ASR systems and two human-audited challenge suites, LCAR reduces detector-identified hallucinations while largely preserving standard open-source test-set recognition performance.