Source-linked AI summary
TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models
Heyu Chang, Nianwen Si, Hao Zhang, Wenlin Zhang, Dan Qu
TL;DR
LALMs can answer yes to absent sound events, a failure linked to strong language priors. TAD contrasts real-audio logits with a silent reference and gates affirmative-token penalties using first-step audio–silent margin evidence. Across two datasets, the authors report improved hallucination robustness over prior methods with generally stable AQA performance.
Problem
LALMs can give linguistically plausible but audio-unsupported answers, including yes responses when queried sounds are absent.
Source: §1 · p. 1Method
TAD contrasts real-audio and silent-reference predictions and applies a first-step, confidence-guided penalty to affirmative tokens when audio does not sufficiently increase the YES–NO margin.
Source: §1 · p. 1Results
Takeaways & Limitations
TAD offers a training-free, decision-aware adjustment intended to reduce prior-driven affirmative bias while preserving stable responses in binary AQA.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events, thus undermining reliability in audio question answering. We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation that grounds the initial yes/no decision by contrasting logits under real audio with a matched silent reference. TAD introduces a token-adaptive, confidence-guided gate that is decision-critical at the first decoding step and class-conditional on affirmative tokens, using the audio-silent margin to avoid overcorrection when evidence is weak or already sufficient. Experiments on AudioCaps-Hallucination show that, relative to Audio-Aware Decoding (AAD), a contrastive baseline with fixed contrast strength, TAD improves F1 for Qwen2 by 0.059 to 0.117 across Popular, Adversarial, and Random splits, and for Gemma by 0.025 to 0.064, while on Clotho-AQA it raises F1 from 0.810 to 0.816 on Qwen2 and remains comparable to AAD on Gemma.
1. Introduction
LALMs can answer yes to binary audio questions even when the queried sound is absent, and fixed-strength contrastive methods do not adapt to varying audio evidence. TAD applies a confidence-guided adjustment to affirmative tokens at the first decoding step; experiments report improved hallucination robustness with generally stable AQA performance.
1. Introduction
- LALMs can produce plausible but audio-unsupported answers, particularly affirmative responses to absent sounds in binary presence/absence questions.The cited failure is associated with strong language priors and spurious training-data co-occurrences.Source: §1 · p. 1
- TAD contrasts real-audio and silent-reference logits, then penalizes affirmative tokens at the first step only when the audio-induced YES–NO margin gain falls below a threshold.This confidence-guided, class-conditional intervention is designed to avoid overcorrection under weak evidence.Source: §1 · p. 1
- Prior contrastive approaches use fixed contrast strength across decoding, limiting adaptation to varying audio evidence and hallucination suppression.Audio-Aware Decoding and Adaptive Vector Steering are examples of training-free inference-time interventions.Source: §1 · p. 1
- Experiments on AudioCaps-Hallucination and Clotho-AQA show improved hallucination-oriented robustness across diverse settings while maintaining generally stable AQA performance.The method is presented as a training-free plug-in logits processor.Source: §1 · p. 1
2. Method
TAD compares predictions from real audio with a length-matched silent reference, maps vocabulary logits to YES/NO evidence, and applies a first-step gate based on the audio-induced margin gain.
2.1. Contrastive Decoding with Silent Reference
- The silent-reference contrast compares real-audio logits with logits conditioned on an all-zero waveform of the same length, whose predictions approximate text and language priors with minimal effective audio evidence.The contrastive formulation combines the two streams, with α controlling contrast strength.Source: §2.1 · p. 2
2.2. Yes/No Token Set Construction
- YES and NO token sets cover multiple surface forms and subword variants so the binary decision is not tied to a single token ID.The sets are built by tokenizing affirmative and negative forms and taking the deduplicated union of their token IDs.Source: §2.2 · p. 2
2.3. Log-Sum-Exp Pooling over Subword Variants
- Log-sum-exp pooling aggregates vocabulary logits across token IDs for each semantic class, yielding YES and NO scores for both audio and silent conditions.These pooled scores summarize the model’s class preference under each condition.Source: §2.3 · p. 2
2.4. Confidence-Margin-Guided Gated Decoding
- At the first decoding step, TAD compares the real-audio and silent YES–NO margins and penalizes affirmative tokens only when the audio-induced margin gain is below threshold.The gate leaves the AAD-combined logits unchanged when it is not triggered; subsequent decoding uses those logits alone.Source: §2.4 · p. 2
3.1. Experimental Setups
The evaluation uses two audio question-answering benchmarks, two audio-language models of different scales, and standard binary classification metrics with selected decoding hyperparameters.
3.1. Experimental Setups
- AudioCaps-Hallucination uses binary questions generated from five templates and RANDOM, ADVERSARIAL, and POPULAR object-sampling strategies; Clotho-AQA evaluation uses 1,991 yes/no samples.The AudioCaps-Hallucination split sizes are 30,220, 31,047, and 31,376 QA pairs, respectively.Source: §3.1 · p. 2
- Evaluation covers Qwen2-Audio-7B-Instruct and the lightweight Gemma-3n-E4B-it model with approximately 4B parameters.The models provide evaluation at different scales.Source: §3.1 · p. 3
- The study reports accuracy, precision, recall, and F1 with NO treated as the positive class, and uses α typically at 0.5 or 1.0, τ=0.2, and γ=2.5.The authors state that they do not exhaustively ablate α, τ, and γ.Source: §3.1 · p. 3
3.2. Main Results
TAD achieves the highest F1 on several AudioCaps-Hallucination splits across Qwen2 and Gemma, while Clotho-AQA results show a Qwen2 F1 gain and Gemma performance close to AAD.
3.2.2. Results on Clotho-AQA
- On Clotho-AQA, Qwen2 TAD achieves the best F1 at 0.816 with α=1.0 while keeping accuracy competitive.Both AAD and TAD improve over default decoding mainly through higher recall and F1.Source: §3.2.2 · p. 3
- On Clotho-AQA, Gemma AAD and TAD raise recall from 0.470 to at least 0.844, while F1 is 0.658 for AAD and 0.650 for TAD.The increased conservativeness comes with reduced precision and accuracy.Source: §3.2.2 · p. 3
3.2.1. Results on AudioCaps-Hallucination
- Qwen2 TAD achieves the best F1 on AudioCaps-Hallucination RANDOM (0.853) and POPULAR (0.558), while AVS leads ADVERSARIAL at 0.524.TAD remains competitive and improves over AAD on ADVERSARIAL.Source: Table 1 · p. 3, §3.2.1 · p. 3
- Gemma TAD achieves the highest F1 on AudioCaps-Hallucination ADVERSARIAL (0.611) and POPULAR (0.582), while AVS leads RANDOM at 0.727.Increasing α generally raises TAD recall on harder settings, but can slightly reduce RANDOM F1 when contrast is too strong.Source: Table 2 · p. 3, §3.2.1 · p. 3
3.3. Further Analysis
The analyses show TAD shifts binary decisions toward rejecting unsupported objects, improves first-step separability, and selectively preserves affirmative answers when audio evidence is strong.
3.3.1. Confusion-matrix Perspective
- True-NO recall rises from 0.266 under default decoding to 0.655 with AAD and 0.858 with TAD, while TAD’s true-YES recall is 0.844.The normalized confusion matrices show a larger gain in rejecting unsupported objects than the loss in true positives.Source: §3.3.1 · p. 3, Figure 2 · p. 4
3.3.2. ROC and AUC
- TAD achieves higher ROC and AUC than default decoding and AAD using first-step logits and NO as the positive class.The thresholded P(no) score reflects early-stage discriminability independent of later decoding.Source: Figure 3 · p. 4, §3.3.2 · p. 4
3.3.3. Quantitative Behavior of Audio Evidence Scores
- Silent-reference margins overlap across labels, but with-audio margins separate them; δ < 0 skews true-NO and δ > 0 true-YES, supporting selective suppression when audio gains are weak.TAD preserves affirmative answers when audio provides a clear margin gain.Source: Figure 4 · p. 4, §3.3.3 · p. 4
4. Conclusion
The conclusion presents TAD as a training-free, first-step intervention for audio-object hallucination and reports improved robustness across two datasets.
4. Conclusion
- TAD is a training-free, decision-aware method for mitigating audio-object hallucination in LALMs.It targets prior-driven affirmative bias in binary audio question answering.Source: §4 · p. 4
- At the first decoding step, TAD estimates audio-supported evidence and confidence-gates adjustments to downweight unsupported affirmative outputs while preserving stable responses.The conclusion describes this as a targeted, class-conditional intervention.Source: §4 · p. 4
- Experiments on AudioCaps-Hallucination and Clotho-AQA show improved robustness over prior methods, while early-step evidence and separability analyses help explain the gains.Source: §4 · p. 4
6. Generative AI Use Disclosure
Generative AI tools assisted only with language polishing and readability; the authors take responsibility for the research and its scientific conclusions.
6. Generative AI Use Disclosure
- Generative AI tools were used solely for language polishing and readability improvement.Source: §6 · p. 5
- The authors take responsibility for the research design, experimental process, and data analysis.Source: §6 · p. 5
- The authors take responsibility for interpreting results and the manuscript’s scientific conclusions.Source: §6 · p. 5