Source-linked AI summary
Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition
YoungChae Kim, Da-Hee Yang, Joon-Hyuk Chang
TL;DR
The paper addresses the trade-off caused by fixed-strength contrastive decoding in noisy AVSR, where stronger intervention can help severe noise but distort reliable clean predictions. It introduces token-level reliability scaling from attention and predictive-divergence cues, reporting consistent improvements across clean and noisy conditions without additional training. The approach is practical as a plug-and-play inference strategy, although extreme inter-model disagreement can destabilize contrastive scoring through rank distortion.
Problem
Fixed contrastive strength cannot adapt to token-level acoustic reliability across noise conditions, causing miscalibrated intervention in noisy AVSR.
Method
The method dynamically scales contrastive decoding strength using attention-derived reliability signals and inter-model predictive divergence during inference.
Results
The method consistently improves AVSR performance from clean to severely noisy conditions and generalizes across model scales and evaluation domains.
Takeaways & Limitations
Reliability-aware contrastive decoding provides a training-free, plug-and-play route to improving robustness while preserving clean-speech accuracy without architectural modification.
Takeaways & Limitations
Extreme inter-model disagreement can destabilize contrastive decoding because collapsed Amateur probabilities create token-dependent offsets that distort ranking.
Abstract
from arXiv · showhide
Large language model (LLM)-based audio-visual speech recognition (AVSR) systems are robust under noise. Contrastive decoding (CD), originally introduced to stabilize LLM generation by contrasting a weaker model against a stronger one at inference time, adjusts predictions without additional training. In this work, we apply CD to AVSR by contrasting audio-only conditioning with full audio-visual conditioning within the same underlying model. However, using a fixed contrastive strength introduces a trade-off across noise levels: stronger intervention helps under severe noise but may over-correct reliable predictions in clean conditions. We propose reliability-aware scaling of CD for AVSR. Instead of using a fixed strength, we adaptively modulate the contrastive influence at each token based on reliability signals derived from attention dynamics and inter-model predictive divergence. Experiments on LRS3 show consistent improvements across clean and low-SNR conditions.
1. Introduction
LLM-based AVSR improves noisy-speech recognition, but fixed contrastive decoding can over-correct reliable tokens because acoustic reliability varies with noise. The proposed attention-guided scaling adapts contrastive strength during inference without additional training or architectural changes.
- Existing noise-robust AVSR methods modify decoder structure, requiring additional fine-tuning and increasing model complexity, training costs, and deployment inefficiency.
- Contrastive decoding applies a training-free inference-time comparison between full audio-visual and audio-only conditioning within the same LLM-based AVSR model.
- Fixed contrastive weighting treats tokens uniformly despite changing acoustic reliability, improving severe-noise robustness but potentially distorting reliable predictions in cleaner conditions.
- Attention-guided reliability scaling dynamically modulates token-level contrastive strength using attention behavior and inter-model predictive divergence.
- Experiments show consistent gains from clean to severely noisy conditions, with generalization across model scales and evaluation domains.
2. Method
The method replaces fixed-strength contrastive decoding with token-level scaling driven by attention-based reliability and predictive disagreement. It combines energy, entropy, and Jensen–Shannon divergence signals to gate intervention and suppress unstable contrastive effects.
- Motivation: Fixed contrastive strength is suboptimal because token-level acoustic reliability and noise conditions vary during decoding.Uniform intervention can improve severe-noise robustness while distorting reliable predictions under cleaner conditions.
- Contrastive decoding: Contrastive decoding adjusts token log-probabilities by contrasting full audio-visual Expert predictions with audio-only Amateur predictions.The two conditions use the same underlying AVSR model, avoiding architectural mismatch.
- Token-level gating: The effective intervention strength is λ(t)_eff = w_t · λ, where w_t continuously interpolates between pure AVSR decoding and full contrastive decoding.w_t = 0 gives pure AVSR, w_t = 1 gives standard CD, and intermediate values provide selective intervention.
- Reliability signals: The gate w_t multiplicatively fuses energy, entropy, and JS-divergence reliability metrics, activating CD conservatively only when all signals support intervention.The metrics are derived from attention behavior and predictive disagreement between the two conditioning settings.
- Attention extraction: Audio attention is analyzed from the current decoding token at the final Transformer layer, with values averaged across attention heads.This setup targets modality interaction most directly connected to the next-token probability distribution.
- Audio entropy: Audio entropy distinguishes concentrated attention associated with clear signals from dispersed attention associated with noisy or uncertain signals.Audio-specific attention is renormalized per head, and entropy is normalized by log N_a to obtain a length-invariant value in [0, 1].
- JS-divergence filtering: The JS-divergence filter emphasizes informative disagreement near μ_sweet while suppressing nearly identical or excessively divergent predictions.This prevents extreme disagreement from causing rank distortion through unstable token-wise offsets.
3. Experiments and Results
Experiments on LRS3 evaluate reliability-scaled contrastive decoding across clean and noisy conditions, model scales, and in-domain and out-of-domain test sets. The results show consistent gains, while ablations explain how reliability cues complement one another and why fixed contrastive weights are condition-dependent.
- Datasets and Metrics: The evaluation corrupts LRS3 test utterances with MUSAN noise at 0, -5, -10, and -15 dB and measures transcription quality using WER.The test set contains 1,327 utterances.
- Analysis and Ablation: Figure 2 shows that decreasing SNR shifts JS divergence rightward, while ASR token error rises sharply with JS and AVSR remains relatively stable across disagreement levels.At -15 dB, the JS distribution becomes bimodal, motivating a Gaussian filter centered at µsweet = 0.35.
- Model Architectures and Training Setup: The framework is evaluated across Llama-AVSR, Omni-AVSR, and Qwen-AVSR, spanning Llama-3.1-8B, Llama-3.2-1B, and Qwen2.5-0.5B systems.The base LLM and encoders remain frozen, with LoRA applied to projection layers.
- Main Results: The proposed method generalizes across model scales and LRS3/LRS2 domains, improving clean-condition WER while maintaining gains under noisy conditions.The reported decoding latency increases from 1577.9 ms to 1714.3 ms per utterance, a +8.6% overhead.
- Analysis and Ablation: Fixed-weight CD is condition-dependent: stronger contrast helps severe-noise robustness but can penalize reliable clean-condition predictions when incoming SNR varies.The base contrastive weight is set to λ = 0.3 as the best overall value across the explored range.
- Analysis and Ablation: JS divergence stabilizes clean and mildly noisy decoding, whereas energy and entropy cues contribute more under moderate noise; multiplicative combination balances performance across SNRs.The combined gating activates CD only when the reliability signals jointly indicate unreliability.
4. Conclusion
The paper proposes reliability-aware contrastive decoding that scales intervention using attention and divergence cues. It aims to improve severe-noise robustness while preserving clean accuracy through a plug-and-play, training-free approach.
- Reliability-aware contrastive decoding adaptively scales intervention strength using attention and divergence cues.
- The method is presented as a plug-and-play, training-free solution for robust AVSR without typical denoising–clean-accuracy trade-offs.