PaperlayerView exact revision on arXiv

Source-linked AI summary

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

Yuchen Deng, Chang Sun, Hai-Tao Zheng, Feidiao Yang, Yuxing Han

arXiv:2609.04247v1eess.AScs.SD

TL;DR

Omnimodal models remain vulnerable to cross-modal hallucinations, while existing training-free decoders do not assess predictive compatibility within the joint audio-visual branch. CAD estimates potential conflict and whether intervention is warranted, then selectively reallocates decoding weight. It consistently outperforms the base decoder and competitive training-free methods across evaluated benchmarks and backbones, with overall-accuracy gains of 14.1 points on CMM and 8.0 points on AVHBench for Qwen2.5-Omni-7B.

  • Problem

    Existing training-free decoders modulate modality influence but do not assess predictive compatibility within the joint audio-visual branch, where discrepancies may reflect harmful interference or useful complementarity.

    Source: Introduction · p. 1

  • Method

    CAD uses PCME to estimate potential conflict and CAA to assess whether task-space evidence warrants intervention, selectively reallocating weight from the joint branch to unimodal branches.

    Source: Introduction · p. 1, Overview of CAD · p. 3

  • Results

    CAD consistently achieves higher overall accuracy than the base decoder and competitive training-free methods across evaluated backbones; on Qwen2.5-Omni-7B, overall accuracy rises by 14.1 points on CMM and 8.0 points on AVHBench.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The findings support selective intervention on audio-visual discrepancies, since discrepancy magnitude alone does not establish that fusion is harmful.

    Source: Introduction · p. 1, Potential Conflict Magnitude Estimation · p. 5

    PCME's unsigned score measures potential conflict magnitude, not whether the interaction is harmful or the joint prediction is erroneous, so it cannot alone establish grounds for intervention.

    Source: Potential Conflict Magnitude Estimation · p. 5

Abstract

from arXiv · show

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.

Introduction

CAD distinguishes harmful audio-visual interference from useful fusion and intervenes only when conflict is actionable; experiments show accuracy gains without model retraining.

Introduction

  • Cross-modal hallucinations arise when one modality improperly influences content about another, such as visual cues prompting nonexistent sounds or acoustic cues prompting nonexistent visual events.Poorly coordinated fusion can introduce harmful interference, whereas effective fusion can exploit complementary audio and visual evidence.Source: Introduction · p. 1
  • AVCD uses uniform perturbations and MAD adaptively weights branches, but neither assesses audio-visual predictive compatibility within the joint branch (Jung, Jang, and Chung 2025; Chung et al. 2026).Thus, relevance or perturbation alone does not distinguish useful complementarity from harmful interference.Source: Introduction · p. 1
  • Joint inference is incorrect in 4.5% of cases where both unimodal branches are correct, but recovers the correct answer in 8.7% where both are incorrect.The results show that fusion can produce either destructive interference or complementary recovery.Source: Introduction · p. 1
  • CAD combines PCME’s unsigned potential-conflict score with CAA’s task-space answer relations, using query relevance and answer decisiveness to assess actionability.CAA applies Dempster-Shafer reliability discounting; when conflict exceeds a threshold and evidence supports intervention, CAD shifts weight from the joint branch to unimodal branches.Source: Introduction · pp. 1–2
  • CAD raises overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, for Qwen2.5-Omni-7B, and by 4.1 and 6.9 points for OmniVinci.It consistently outperforms Base, AVCD, and MAD across the two backbones and achieves these gains without retraining.Source: Figure 1 · p. 2, Introduction · p. 2
  • Experiments cover CMM, AVHBench, WorldSense, and VideoMME, and report CAD’s highest overall accuracy across evaluated backbone–benchmark pairs, with ablations isolating component contributions.The evaluation includes four audio-visual backbones and comparisons against Base, AVCD, and MAD.Source: Introduction · pp. 1–2

Related work

The related work traces the shift to unified audio-visual reasoning and identifies a gap: existing decoding methods do not determine whether cross-modal disagreement is complementary or harmful.

Omnimodal Large Language Models

  • Omni-LLMs such as Qwen2.5-omni and OmniVinci unify text, image, video, and audio representations and interleave audio-visual tokens, enabling joint reasoning but risking harmful fusion conflicts.Visual signals provide spatial, object-state, and motion information, while audio contributes semantic, sound-event, and temporal cues (AI et al. 2025; Tong et al. 2025; Xie and Wu 2024).Source: Omnimodal Large Language Models · p. 2

Mitigating Hallucinations in MLLMs

  • Vision-language training-free decoders use perturbations, descriptions, instruction contrasts, or layerwise differences, while AVCD (Jung, Jang, and Chung 2025) and MAD (Chung et al. 2026) adapt audio-visual decoding without distinguishing complementarity from harmful conflict.CAD instead assesses audio-only, visual-only, and joint predictions, then adaptively reweights decoding branches to reduce conflict-induced interference.Source: Mitigating Hallucinations in MLLMs · p. 2

Preliminaries

MAD adapts decoding weights to query-level modality relevance, but the joint audio-video prediction can diverge from unimodal predictions through either destructive interference or useful complementarity. This motivates CAD’s separation of predictive discrepancy measurement from intervention assessment.

Modality-Adaptive Decoding

  • MAD uses a fixed modality-query prompt and softmaxes next-token logits for “audio,” “video,” and “both” into relevance weights.These category-level weights represent the query’s reliance on audio, video, or their joint use.Source: Modality-Adaptive Decoding · p. 3
  • MAD evaluates joint, audio-only, video-only, and question-only branches, then uses relevance-dependent coefficients to select the next token by greedy decoding.The joint coefficient increases with joint relevance; unimodal coefficients reflect relevance relative to the joint configuration, while the negative question-only coefficient suppresses language-prior-driven tokens.Source: Modality-Adaptive Decoding · p. 3

Motivation

  • On CMM, AV error was 4.5% when both unimodal branches were correct, while AV accuracy was 8.7% when both were incorrect.The diagnostic evaluated Qwen2.5-Omni-7B on 800 questions; Figure 2 illustrates destructive fusion and complementary recovery.Source: Motivation · p. 3, Figure 2 · p. 3
  • MAD’s relevance-based coefficients do not test joint-unimodal prediction compatibility, so CAD separates unsigned discrepancy measurement (PCME) from assessment of whether the answer relation warrants intervention (CAA).Relevance weights alone cannot distinguish conflicts that warrant suppression from complementarity that should be preserved.Source: Motivation · p. 3

Method

CAD is a training-free decoder that estimates potential audio–visual conflict, assesses whether reliable evidence warrants intervention, and selectively reallocates decoding weights when conflict is actionable. Its two stages are Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA).

Potential Conflict Magnitude Estimation

  • PCME averages normalized Jensen–Shannon discrepancies between audio and video predictions and between the joint prediction and a relevance-weighted unimodal reference, then scales by relevance balance.The relevance-balance factor decreases when either modality is less relevant, and the resulting unsigned score measures potential conflict rather than whether the interaction is harmful.Source: Potential Conflict Magnitude Estimation · pp. 4–5
  • PCME’s unsigned conflict score does not establish that the joint prediction is erroneous; intervention eligibility requires C ≥ τ, and magnitude alone is insufficient to warrant action.Discrepancy may reflect either beneficial cross-modal interaction or harmful interference, motivating CAA’s separate assessment of reliable intervention evidence.Source: Potential Conflict Magnitude Estimation · p. 5

Conflict Actionability Assessment

  • CAA projects audio, video, and joint predictions into task-answer space, then discounts answer evidence using each modality’s query relevance and answer decisiveness.Their product is an uncalibrated reliability proxy, not a probability of correctness; CAA uses Dempster–Shafer reliability discounting and adapts aggregation conservatively because the model branches share evidence.Source: Conflict Actionability Assessment · p. 5
  • CAA sets actionability to zero when all branches agree, uses shared reliability when unimodal answers agree against the joint answer, and otherwise bases support on reliable opposition.When the joint branch follows one disagreeing unimodal branch, the competing branch must have stronger reliability; a third joint answer uses the strongest unimodal opposition as a heuristic. The resulting signal requires both conflict magnitude and evidential support, without identifying the ground-truth answer.Source: Conflict Actionability Assessment · p. 5
  • For actionable conflicts, CAD shifts decoding influence from the joint branch to unimodal branches in proportion to their positive coefficients, preserving the coefficient sum and leaving the text coefficient unchanged.If C < τ or B = 0, CAD reduces to the base MAD decoder; coefficients are computed from initial next-token distributions and remain fixed during generation.Source: Conflict Actionability Assessment · pp. 5–6

Experiments

CAD improves cross-modal hallucination accuracy across backbones and also performs best on the evaluated general audio-visual QA benchmarks. Ablations show that relevance weighting and conflict-aware intervention contribute complementary benefits.

Experimental Setting

  • Experiments cover four hallucination and general audio-visual QA benchmarks, four audio-visual LLM backbones, and comparisons with AVCD and MAD.CMM and AVHBench target cross-modal hallucinations; WorldSense and VideoMME assess general audio-visual QA. AVCD uses uniform modality perturbations, while MAD adjusts predictions by modality relevance.Source: Experimental Setting · p. 6

Main Results

  • On AVHBench, Uniform reaches 78.0% and Argmax 79.6% overall accuracy, while CAD reaches 84.1%.Uniform misses query-specific evidence requirements, while Argmax discards complementary information from non-selected modalities; the results support combining soft relevance-aware weighting with conflict-aware reallocation.Source: Main Results · pp. 6–7, Table 2 · p. 7
  • On Qwen2.5-Omni-7B, CAD raises overall accuracy by 14.1 points to 85.0% on CMM and by 8.0 points to 84.1% on AVHBench.On CMM, visual- and audio-dominance accuracy increase by 13.8 and 14.5 points; on AVHBench, video-driven audio and audio-driven video accuracy increase by 7.5 and 9.0 points. CAD also improves results across the other evaluated backbones.Source: Main Results · p. 6, Table 1 · p. 7
  • Removing any relevance weight degrades AVHBench performance; removing rV lowers audio-driven video accuracy from 85.9% to 80.0% and overall accuracy from 84.1% to 80.9%.Removing rA mainly reduces video-driven audio accuracy, while removing rAV causes smaller consistent drops in both categories; the full model performs best.Source: Table 3 · p. 7, Main Results · p. 7
  • On AVHBench, adding conflict magnitude C raises overall accuracy from 80.7% to 82.7%, using gate B alone reaches 83.3%, and using both reaches 84.1%.C identifies potentially problematic fusion, while B assesses whether the discrepancy warrants correction; together they improve both hallucination categories.Source: Table 4 · p. 7, Main Results · pp. 7–8
  • On CMM, mean potential conflict magnitude C is 0.100 for wrong Qwen2.5-Omni-7B predictions versus 0.033 for correct predictions.Wrong predictions are more concentrated in the high-C region, indicating that larger discrepancies are associated with cross-modal reasoning errors.Source: Figure 4 · p. 8, Main Results · p. 8
  • CAD achieves the best performance on both WorldSense and VideoMME (w/o Sub.), outperforming the base model and MAD.The reported results suggest that conflict-aware decoding generalizes from hallucination mitigation to general audio-visual question answering.Source: Table 5 · p. 8, Main Results · p. 8

Conclusion

CAD is a training-free framework that estimates potential cross-modal conflicts and selectively reallocates decoding weights to mitigate hallucinations. Experiments show consistent improvements over strong training-free baselines, while manually tuned hyperparameters remain a limitation.

Conclusion

  • CAD assesses conflict magnitude and actionability across audio, visual, and joint audio-visual predictions, then reallocates decoding weights to suppress unreliable fusion.The framework targets cross-modal hallucinations in Omni-LLMs without training.Source: Conclusion · p. 8
  • Experiments across four representative audio-visual models show consistent improvements over strong training-free baselines on CMM and AVHBench for both video-driven audio and audio-driven video hallucinations.Source: Conclusion · p. 8
  • CAD relies on manually tuned hyperparameters, which may limit adaptability across models and datasets; adaptive parameter selection is a future-work direction.Source: Conclusion · p. 8
Loading 2609.04247v1…