PaperlayerView exact revision on arXiv

Source-linked AI summary

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro

arXiv:2601.21181v1cs.AI

TL;DR

MLLMs suffer from cross-modal hallucinations because modality-agnostic decoding does not account for task-specific modality requirements. MAD addresses this gap with training-free, self-assessed modality weighting for adaptive contrastive decoding, improving hallucination results across CMM and AVHBench. The findings support explicit modality-aware fusion for more reliable multimodal reasoning.

  • Problem

    MLLMs face cross-modal hallucinations, while existing contrastive decoding methods lack awareness of task-specific modality requirements.

  • Method

    MAD queries the model to self-assess modality relevance, then uses task-aware weights to adaptively fuse modality-specific contrastive distributions.

  • Results

    MAD significantly improves cross-modal hallucination performance on both CMM and AVHBench while maintaining standard evaluation performance.

  • Takeaways & Limitations

    Explicit modality-aware fusion can reduce cross-modal hallucinations by emphasizing relevant information and suppressing interference from irrelevant modalities.

  • Takeaways & Limitations

    Existing modality-agnostic methods provide the constrained comparison scope motivating MAD, which specifically targets task-specific modality requirements.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at \href{https://github.com/top-yun/MAD}{https://github.com/top-yun/MAD}

1. Introduction

MLLMs can suffer cross-modal hallucinations when one modality improperly influences generation about another. MAD addresses this with task-aware, self-assessed modality weighting for adaptive contrastive decoding.

  • Challenge: Cross-modal hallucinations occur when one modality improperly influences content generation about another, revealing a failure of modality separation.Audio-driven visual hallucinations can arise when auditory signals cause the model to invent visual events.
  • Challenge: Existing contrastive decoding methods are modality-agnostic and apply uniform distortions without considering task-specific modality requirements.This static strategy cannot dynamically block irrelevant inputs from causing cross-modal corruption.
  • Method: MAD is a training-free method that explicitly determines each task’s modality requirements and dynamically adapts contrastive decoding.It introduces task-driven modality weighting without requiring model retraining.
  • Method: MAD self-assesses modality relevance, extracts task-aware weights, and adaptively fuses modality-specific contrastive distributions.This task-conditioned fusion emphasizes relevant modalities while suppressing inappropriate cross-modal interference.
  • Results: MAD improves cross-modal hallucination performance on both CMM and AVHBench while maintaining standard-task performance.The evaluation covers multiple audio-visual language models and two dedicated hallucination benchmarks.

3. Method

MAD extends contrastive decoding with task-specific modality weighting, using self-assessed modality relevance to adapt suppression across audio, visual, and joint branches.

  • 3.1.1. Multimodal Input and Generation Process: An AV-LLM processes video, audio, and textual question inputs through modality-specific encoders and predicts tokens autoregressively.
  • 3.1.2. Contrastive Decoding for Vision Language Models: Contrastive decoding contrasts clean and degraded inputs, amplifying modality-grounded logits while down-weighting tokens supported mainly by language priors.
  • 3.2.1. Weighted Contrastive Decoding: Fixed contrastive strength is unsuitable across tasks because relevant modalities differ between questions, such as audio for sound identification and vision for car color.
  • 3.2.1. Weighted Contrastive Decoding: MAD sets each modality’s contrastive strength as αm = γ · wm, where γ is shared and wm is a task-specific relevance weight.
  • 3.2.2. Modality-Adaptive Weight Extraction: The model extracts logits for video, audio, and both from a modality query, then converts them into normalized self-assessed weights used to scale contrastive branches.
  • 3.2.2. Modality-Adaptive Weight Extraction: Weight distributions align with question requirements: visual questions favor wv, audio questions wa, and audio-visual questions wav with balanced wv and wa.
  • 3.2.3. Modality-Adaptive Generation: MAD replaces fixed branch strengths with γ · wav, γ · wv, and γ · wa, adaptively suppressing irrelevant-modality hallucinations while preserving relevant information.

4. Experiments

MAD is evaluated on cross-modal hallucination benchmarks and general audio-visual QA, where it consistently improves over existing decoding methods. Ablations indicate that adaptive, modality-specific weighting and all three modality weights are important for robust performance.

  • Evaluation Setup: MAD is evaluated on CMM and AVHBench using multiple state-of-the-art audio-visual language models against existing contrastive decoding methods.
  • Main Results: MAD consistently outperforms existing decoding methods across models and datasets, mitigating cross-modal hallucinations.
  • AVHBench Results: On AVHBench, video-driven audio hallucination accuracy improves by +4.0% for VideoLLaMA2-AV and +5.7% for Qwen2.5-Omni, with Qwen2.5-Omni gaining +3.7% on audio-driven video hallucination.
  • Ablations: Uniform and argmax weighting degrade performance, while adaptive weighting achieves balanced fusion and consistently outperforms both baselines across models and benchmarks.
  • Ablations: Removing either audio or visual weights lowers accuracy and dominance performance, while using all three weights achieves the highest accuracy of 81.3%.
  • General AVQA: MAD achieves comparable or slightly improved performance on general AVQA benchmarks beyond explicit cross-modal hallucination tests.

5. Conclusion

The paper concludes that MAD is a simple, training-free strategy for aligning contrastive signals with each question’s modality requirements. Self-assessed modality relevance and modality-specific weights suppress interference from irrelevant modalities and mitigate hallucination.

  • MAD aligns contrastive signals with question-specific modality requirements through self-assessed relevance and modality-specific weights.
  • The method adaptively modulates multiple modalities to suppress interference from irrelevant modalities and mitigate cross-modal hallucination.

6. Impact of γ

The study analyzes γ as the temperature controlling contrastive strength between modality-specific and full distributions. It selects γ = 2.5 as the default because performance remains robust across models and benchmarks.

  • γ controls the contrastive strength between modality-specific distributions and the full distribution in MAD’s modality-adaptive weighting mechanism.
  • The analysis varies γ from 0.5 to 3.0 in 0.5 intervals on AVHBench and CMM using VideoLLaMA2-AV and Qwen2.5-Omni.
  • γ = 2.5 is used as the default because it demonstrates consistently robust performance across different models and benchmarks.

7. Qualitative Analysis of Modality-Adaptive Weights

The qualitative analysis tests whether extracted modality weights reflect question-specific modality dependence. The weights assign greater emphasis to the modality or modality combination required by each question type.

  • The analysis samples 100 videos from VideoMME and constructs 300 questions divided into visual, audio, and audio-visual categories.
  • Table 5 pairs representative questions from each category with their computed modality weights.
  • Visual questions receive higher wv, audio questions higher wa, and audio-visual questions the highest wav weight.

8. Modality Weight Distribution Analysis Across Tasks

MAD assigns modality weights according to task requirements, emphasizing the modality needed for each question while reducing misleading cross-modal influence. This pattern holds across AVHBench and CMM task categories.

  • MAD assigns higher weights to the modality relevant to each question type: audio for audio questions, video for visual questions, and both for audio-visual questions.Table 5 reports higher wv for visual questions, higher wa for audio questions, and the highest wav for audio-visual questions.
  • AVHBench: In V→A tasks, MAD prioritizes audio despite potentially misleading visual information.The highest wa proportion indicates that the model identifies audio comprehension as the task requirement.
  • AVHBench: In A→V tasks, MAD prioritizes video when audio could interfere with video understanding.The predominant wv proportion indicates recognition of video as the essential modality.
  • AVHBench: The symmetric weighting behavior across V→A and A→V tasks validates the adaptability of MAD’s weighting mechanism.
  • CMM: For visual dominance, MAD increases audio weight to counter visual over-reliance and prevent audio hallucinations.
  • CMM: For audio dominance, MAD increases video weight to suppress video hallucinations caused by audio interference.
  • CMM: For language-dominant questions, MAD substantially increases video weight to encourage reliance on visual evidence over linguistic shortcuts.
  • Across both benchmarks, the weighting mechanism captures task-specific modality requirements and emphasizes relevant information while suppressing misleading cross-modal influence.

9. Robustness Analysis of Modality Query Prompts

MAD’s modality-query weighting remains stable across semantically similar prompt formulations. The reported variation is small on both AVHBench and CMM, supporting robustness to prompt phrasing.

  • Robustness Analysis of Modality Query Prompts: MAD’s robustness analysis tests whether alternative modality-query prompts affect its ability to generate appropriate modality weights.
  • 9.1. Alternative Prompt Formulations: Four alternative prompts preserve the same semantic intent while varying the wording used to identify the modality required for reasoning.
  • 9.2. Experimental Results: Figure 8 reports performance statistics across modality-query prompts on both AVHBench and CMM.
  • 9.2. Experimental Results: 0.26% standard deviation on AVHBench and 0.31% on CMM indicate highly consistent performance across prompt variations.
  • 9.2. Experimental Results: The minimum-to-maximum performance ranges are 0.59% on AVHBench and 0.83% on CMM.
  • 9.2. Experimental Results: The consistency across diverse phrasings suggests modality-weight generation relies on semantic understanding rather than prompt engineering or surface-level pattern matching.

10. Computational Analysis

The computational analysis compares decoding latency among contrastive-decoding methods for MAD and two audio-visual language models.

  • MAD’s computational overhead is analyzed by comparing decoding latency in milliseconds per token with other contrastive-decoding methods.Experiments use VideoLLaMA2-AV and Qwen2.5-Omni on 8 NVIDIA RTX A6000 GPUs.
  • Table 6 presents the computational-efficiency comparison among contrastive-decoding methods.

11. Qualitative Results

The qualitative-results section presents examples illustrating how MAD mitigates cross-modal hallucinations in VideoLLaMA2-AV and Qwen2.5-Omni.

  • The section provides qualitative examples showing how MAD mitigates cross-modal hallucinations.
  • Figure 9 reports qualitative results for VideoLLaMA2-AV.
  • Figure 10 reports qualitative results for Qwen2.5-Omni.
Loading 2601.21181v1…