PaperlayerView exact revision on arXiv

Source-linked AI summary

Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs

Tianle Chen, Chaitanya Chakka, Arjun Reddy Akula, Xavier Thomas, Deepti Ghadiyaram

arXiv:2511.22826v2cs.CV

TL;DR

The paper asks whether MLLMs can reason reliably when modalities are missing or conflict, a concern given training that often assumes aligned inputs. It introduces MMA-Bench and interpretability analyses, then reports that modality-aware tuning improves grounding and robustness across aligned and misaligned settings.

  • Problem

    The paper examines whether MLLMs can handle missing, noisy, or conflicting modalities, given that many models are trained on datasets assuming aligned inputs.

    Source: §1 · p. 2

  • Method

    The authors use MMA-Bench to control modality presence and alignment, analyze model behavior with black-box and white-box techniques, and apply modality-aware fine-tuning.

    Source: §1 · p. 2

  • Results

    Modality-aware tuning yields 20–40% absolute gains on visual- and audio-focused prompts and improves robustness across aligned and misaligned settings.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    The authors report stronger cross-modal grounding and attention redistribution from modality-aware tuning, with improved robustness under audio-visual conflict.

    Source: §1 · p. 2, §5.2 · p. 8

    The evaluation is currently limited to Qwen2.5-Omni-7B and classification-style QA; broader architectures and open-ended reasoning remain future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.

1. Introduction

The paper asks whether MLLMs can handle missing, noisy, or conflicting modalities, then introduces a benchmark and interpretability analyses to examine that question. It reports modality over-reliance and proposes modality-aware tuning to improve cross-modal grounding.

1. Introduction

  • MLLMs are often trained on aligned modalities, raising concerns about brittleness to missing, noisy, or conflicting inputs and potential poisoning or prompt attacks.The authors note possible risks in robotics and medical diagnostics.Source: §1 · p. 2
  • MMA-Bench controls the presence or alignment of video, audio, and text, while black-box and white-box analyses probe model behavior under those conditions.The analyses find over-reliance on text and visual modalities, with models collapsing when modalities conflict.Source: §1 · p. 2
  • The proposed modality-aware tuning aims to instill modality grounding, and fine-tuned models show stronger cross-modal grounding, interpretable attention redistribution, and measurable gains on in- and out-of-domain benchmarks.The stated contributions include a benchmark, diagnostic framework, and fine-tuning approach.Source: §1 · p. 2

2. Related Work

The section distinguishes MMA-Bench from prior audio-visual benchmarks and situates the paper’s prompt-conditioned supervision among approaches to modality bias and misalignment.

Audio–Visual Evaluation Benchmarks

  • AVTrustBench [5] and AVHBench [24] assess global audio-visual consistency or hallucination, while VGGSounder [40], DAVE [21], and AURA [22] assume aligned sight and sound.MMA-Bench instead pairs visual- and audio-focused questions whose correct answers may diverge under semantic conflict.Source: §2 · p. 2
  • DAVE, AURA, and OmniVideoBench [17] offer compositional or logical reasoning tests but seldom require distinct answers for visual and auditory content.MMA-Bench is designed to diagnose visual and auditory reasoning separately within natural multimodal context.Source: §2 · p. 2

Mitigating Modality Bias and Misalignment

  • Prior mitigation approaches include inference-time steering such as AutoSteer [28] and MC2 [36], training objectives such as Arrow-of-Time [33], decoding methods AVCD [16] and Fork-Merge [15], and distillation such as Bridging Ears and Eyes [14].The paper contrasts its supervised fine-tuning on aligned and deliberately misaligned pairs with these approaches.Source: §2 · p. 2
  • Prompt-conditioned supervision is intended to redistribute attention toward the queried modality, rather than statically suppressing modality bias.The method uses semantically aligned and deliberately misaligned video-audio pairs.Source: §2 · p. 2

3. Multi-Modal Alignment (MMA) Bench: Debugging Alignment Gaps in MLLMs

MMA-Bench is designed to probe modality-specific reasoning when video and audio may conflict. Its curation pipeline filters noisy source data, and paired questions evaluate visual and auditory content separately.

3. Multi-Modal Alignment (MMA) Bench: Debugging Alignment Gaps in MLLMs

  • The automated pipeline simplifies the ontology and retains clips only when an audible event is clearly produced by a visible object.Aligned-sample criteria require a visible sounding object, corresponding audio, and no distracting or composite audio sources.Source: Figure 2 · p. 3, §3 · p. 3
  • MMA-Bench addresses a gap in benchmarks that provide binary alignment labels rather than separate semantic annotations for video and audio.Each video receives a visual-focused and an audio-focused question, allowing modality-specific reasoning to be assessed separately.Source: §3 · pp. 3–4
  • AudioSet’s evaluation split has missing sounding objects, overly fine-grained classes, noisy labels, and multiple labels that may not correspond to one clear audio-visual event.The source split contains 20,371 videos annotated with 527 audio event classes.Source: §3 · p. 3
  • Curation prunes abstract or overly fine-grained classes, filters multi-event clips and missing sounding objects with four questions, then manually checks retained samples.The aligned set contains 658 single-label videos; each receives a misaligned counterpart by swapping its audio track with a different event.Source: §3 · p. 3
  • The benchmark’s controlled scenarios selectively perturb or remove one modality while holding the others unchanged to isolate sources of multimodal brittleness.Table 2 summarizes these misalignment and ablation scenarios.Source: Table 2 · p. 4
  • The final benchmark contains 1,316 videos, half aligned and half misaligned, spanning 49 representative sound categories.The dataset is created through ontology pruning, automated cross-modal verification, and manual inspection; Table 1 summarizes retained samples across curation stages.Source: §3 · pp. 3–4, Table 1 · p. 4

4. Are modern MLLMs Truly “Multi-modal”?

The section probes how MLLMs use visual, audio, and text cues under controlled modality conditions, combining performance tests with attention analysis. The results show brittle grounding when cues are absent or conflict, alongside prompt-sensitive attention shifts.

4. Are modern MLLMs Truly “Multi-modal”?

  • The study combines black-box performance probing with white-box inspection of intermediate attention layers and heads across controlled misalignment scenarios.Its stated goal is to understand how visual, text, and audio representations interact during reasoning.Source: §4 · p. 4

4.1.2. Performance when one modality is removed

  • With frames zeroed, most models' audio-task accuracy drops significantly, while VideoLLaMA2 improves slightly; with audio removed, visual-task accuracy declines only slightly, by 4–5% for Qwen2.5-Omni-7B and VideoLLaMA2.Under visual-focused prompts with no visual input, all models except PandaGPT still return high accuracy using audio cues.Source: Figure 4 · p. 4, §4.1.2 · p. 5

4.1.3. Video

  • Under semantic misalignment, Qwen2.5-Omni-7B and VideoLLaMA2 lose accuracy on both prompt types, while Gemini-2.0-Flash-Lite and PandaGPT audio accuracy falls to 1–4%.The first pair still maintains reasonable performance; the latter pair's visual accuracy remains nearly unchanged or slightly improves.Source: Figure 6 · p. 5, §4.1.3 · p. 6

4.1.1. Experiment setup

  • The experiments use three open-sourced models and one closed-sourced model, average results across three random seeds, and target one modality per prompt while generally providing both modalities.The models are VideoLLaMA2, PandaGPT, Qwen2.5-Omni-7B, and Gemini-2.0-Flash-Lite.Source: §4.1.1 · p. 5

4.2. White-box Interpretability and Findings

  • Text tokens receive an average of 81.29% of total attention weights in Qwen2.5-Omni-7B and 56.29% in VideoLLaMA2.The analysis links this textual attention dominance to the models' sharp performance decline under textual misalignment.Source: §4.2 · p. 7
  • Misaligned samples show higher Cohen’s-D magnitudes for visual and audio tokens, indicating stronger prompt-driven attention shifts; magnitudes peak around layer 17 for video and 22 for audio.The authors interpret deeper-layer shifts as likely reflecting decisions about which modality to trust, rather than raw fusion.Source: Figure 9 · p. 7, §4.2 · p. 7

5. Experiments

Experiments evaluate modality-aware fine-tuning on MMA-Bench, comparing it with other models and testing robustness across modality conflicts, missing inputs, misleading text, attention patterns, and AVHBench. Results show stronger robustness and grounding, with a long-context limitation for audio-focused prompts.

5.1. Implementation Details

  • Fine-tuning uses visual- and audio-focused classification tasks with both modalities provided, and LoRA adapters on Qwen2.5-Omni-7B’s attention and feed-forward projections.Source: §5.1 · p. 7

5.2. Quantitative Results

  • 79.79% versus 24.95% (Gemini-2.5-Pro) and 14.58% (Qwen3-Omni-30B): tuned Qwen2.5-Omni-7B achieves the highest overall robustness, especially under audio–visual conflict.Source: Table 3 · p. 8, §5.2 · p. 8
  • Modality-aware fine-tuning yields 20–40% absolute gains on visual- and audio-focused prompts across aligned and misaligned settings.Source: §5.2 · p. 8
  • Visual-aligned accuracy remains 94.68% for tuned Qwen2.5-Omni-7B, compared with 97.90% for Gemini-2.5-Pro and 92.88% for Qwen3-Omni-30B.Source: §5.2 · p. 8
  • With one modality removed, fine-tuning raises visual accuracy by over 20% with silent audio and audio accuracy by nearly 50% when video frames are zeroed.Source: Table 4 · p. 8, §5.2 · p. 8

5.2.1. Attentions After Modality-Aware Tuning

  • After tuning, absolute Cohen’s-D increases across layers, especially deeper ones, indicating more decisive attention shifts toward the modality emphasized by the prompt.Source: Figure 10 · p. 9, §5.2.1 · p. 9

5.2. Quantitative Results

  • Under misleading captions, tuning gains exceed 50% for visual prompts and reach 16% for audio prompts, while long irrelevant text still mildly reduces audio-prompt accuracy.Source: Table 5 · p. 8, §5.2 · p. 9

5.3. Zero-shot Evaluation on AVHBench

  • On AVHBench, zero-shot accuracy improves by +8.2% on V2A and +4.7% on A2V; AV Matching is unchanged and AV Captioning rises by +0.1 METEOR.Source: Table 6 · p. 9, §5.3 · p. 9

6. Limitations and Future Work

The paper frames its analysis as an initial step toward systematic interpretation of MLLM robustness to semantic misalignment. Its evaluation is limited to one model and classification-style QA, with broader architectures and tasks left for future work.

6. Limitations and Future Work

  • The evaluation is limited to Qwen2.5-Omni-7B and classification-style QA; planned extensions include larger Qwen models, other architectures, open-ended reasoning, and finer-grained temporal misalignment.Source: §6 · p. 9
  • The paper positions its analysis as a first step toward systematic interpretation of MLLM robustness to semantic misalignment.Source: §6 · p. 9

Appendix Table of Contents

  • Appendices document dataset curation, implementation, interpretability analyses, prompt templates, generalization, additional baselines, abstention, Chain-of-Thought, and qualitative examples.Source: Appendix Table of Contents · p. 12

A. AudioSet Ontology Details and Dataset Simplification

AudioSet’s hierarchical labels and sample annotations create redundancy and ambiguity for audio–visual classification, motivating ontology pruning and staged filtering. The resulting dataset is simplified, checked for cross-modal grounding, and standardized for training.

A. AudioSet Ontology Details and Dataset Simplification

  • AudioSet’s hierarchical labels can duplicate one event across parent and child classes, allow cross-branch ambiguity, and average 2.7 labels per sample, complicating precise misalignment analysis.Source: A. AudioSet Ontology Details and… · p. 12

A.1. Stage 1: AudioSet Ontology Simplification

  • Ontology pruning absorbs descendants below depth three, removes actionless, ambiguous, multiply-parented, and restricted classes, and reduces 527 classes to 61 manually verified audible and visually identifiable classes.Source: §A.1 · p. 13

A.2. Class guided video filtering

  • Sample filtering retains the most specific annotations, excludes labels spanning disjoint ontology roots, and removes restricted or residual multi-label examples to form a single-label dataset.Source: §A.2 · p. 13

B. Sample Curation and Quality Verification Protocol

  • The curation protocol combines automated consistency checks with human verification to ensure each retained video has a clear signal grounded in both modalities.Source: B. Sample Curation and Quality… · p. 13

B.1. Automated Consistency Checks

  • Automated filtering retains a candidate only if it passes four visual, auditory, and cross-modal existence checks; failure on any check leads to rejection.Source: §B.1 · p. 14

B.2. Human Verification Protocol

  • Two researchers independently assess visibility, audio purity, and action dynamics; only unanimous passes are included, yielding 658 aligned videos.Source: §B.2 · p. 14

How did we choose the frame size to resize videos to?

D. Finetuning Details

  • Aligned samples are duplicated ten times to balance their distribution with generated misaligned examples during fine-tuning.Source: D. Finetuning Details · p. 15

E. White-box analysis: Additional Cohen’s-Dtrends

The white-box analyses find that modality selectivity is organized more clearly by layer than by attention head, while attention remains strongly text-centric. Additional tests show limited visual-task disruption but substantial audio-task degradation under conflicting modalities.

E.1. Headwise Statistics

  • Modality selectivity is clearer across layers than across heads, where Cohen’s-D trends remain noisy.The reported selectivity has positive magnitudes for video tokens and negative magnitudes for audio tokens.Source: §E.1 · p. 15, Figure 16 · p. 15

E.2. VideoLLaMA2trends

  • VideoLLaMA2 shows nearly identical Cohen’s-D magnitudes for aligned and misaligned samples, indicating little attention reallocation under conflict.This pattern is consistent with its weaker black-box performance relative to Qwen2.5-Omni.Source: §E.2 · pp. 15–16, Figure 18 · p. 15

F. Attention heatmaps

  • Text tokens receive most attention mass, while video and audio tokens receive very little across output steps.The analysis excludes text-token attention when measuring relative allocation between audio and video streams.Source: F. Attention heatmaps · p. 16, Figure 19 · p. 17

G. Misleading Text Prompt

H. Cross-Class Generalization and Compositionality

I.1. Semantic Misalignment Experiments

  • Under conflicting modalities, visual-focused prompts remain comparatively robust, while audio-focused accuracy drops significantly across additional baseline models.For example, Qwen3-Omni drops approximately 9% on visual-focused prompts, while Gemini-2.0-Flash and Qwen3-Omni fall below 15% on audio-focused prompts.Source: §I.1 · p. 16

I.2. Unimodal Experiments

Unimodal ablations show that visual reasoning barely depends on audio, whereas audio-task performance draws on visual context. Misleading captions and long irrelevant text interfere especially with audio-focused reasoning, with long-context resilience differing between closed- and open-source models.

I.2. Unimodal Experiments

  • Removing audio barely changes visual-task accuracy, but removing video lowers audio-task accuracy, indicating that audio predictions partly rely on visual context.Examples include Gemini-2.5-Pro falling from 97.90% to 97.23% without audio, and Qwen3-Omni falling from 57.39% to 44.68% with frames zeroed.Source: §I.2 · p. 17, Figure 21 · p. 17
  • With audio-focused prompts, visual-only inputs can outperform the multimodal baseline: Gemini-2.5-Pro rises from 60.37% to 95.55%.Gemini-2.0-Flash rises from 57.21% to 91.79%; the authors attribute this pattern to visual-to-audio inference in a forced-choice setting.Source: §I.2 · pp. 17–18, Figure 22 · p. 18

I.3. Misalignment via text

  • Misleading captions leave stronger models’ visual-prompt accuracy comparatively stable but sharply reduce audio-prompt accuracy across architectures.Under visual prompts, Gemini-2.5-Pro and Qwen3-Omni-30B drop 1.9% and 0.6%, while Gemini-2.0-Flash and ChatBridge drop 16.9% and 29.9%.Source: §I.3 · p. 18, Figure 23 · p. 18

I.4. Irrelevant long context caption

  • Long irrelevant context causes moderate degradation for Gemini models, severe declines for Qwen3-Omni-30B, and complete collapse for ChatBridge.The largest drop across the Gemini models and prompt settings is 10.4%.Source: §I.4 · pp. 18–19, Figure 24 · p. 18

J. Unimodal Abstention Evaluation: The “None of the Above” Experiment

The abstention test adds “None of the above” when one modality is missing, addressing forced-choice evaluations that can prompt guessing or hallucination. The tuned model shows strong zero-shot abstention for missing visual evidence, but not for missing audio.

Zero-Shot Abstention Test

  • The original forced-choice unimodal setup is ill-posed because, without the relevant modality, models must guess or rely on the remaining modality.The abstention option makes rejecting the semantic classes the correct response when the queried content is missing.Source: J. Unimodal Abstention Evaluation: The… · p. 19
  • With frames zeroed, the tuned model reaches 90.27% accuracy, versus 10.94% for base Qwen2.5-Omni-7B and 15.05% for Qwen3-Omni.The test adds “None of the above” and the tuned model had not encountered that label or unimodal data during training.Source: J. Unimodal Abstention Evaluation: The… · p. 19
  • With audio removed, the tuned model scores 0%, indicating that the visual-to-audio shortcut remains despite improved abstention when visual evidence is absent.The authors suggest visible objects can still prompt prediction of associated sounds.Source: J. Unimodal Abstention Evaluation: The… · p. 19

K. Can Reasoning Traces Fix Misalignment? (Chain-of-Thought Evaluation)

The CoT evaluation tests whether inference-time reasoning can resolve sensory conflict without parameter updates. It instead reports a visual-task regression for the base model and a sharp audio-performance drop for the tuned model.

CoT prompting

  • CoT lowers the base Qwen model’s Visual Base accuracy from 76.68% to 66.18%.The authors suggest reasoning steps may describe conflicting priors rather than filter noise.Source: K. Can Reasoning Traces Fix… · p. 20, Table 11 · p. 20
  • For the tuned model, CoT reduces aligned-sample audio-prompt accuracy from 88.14% to 58.46%, while visual performance remains stable.The authors hypothesize that reasoning reintroduces dominant visual input and overrides learned audio alignment.Source: K. Can Reasoning Traces Fix… · p. 20, Table 11 · p. 20
  • The authors conclude that robust modality alignment requires parameter optimization and that CoT might not help perceptual grounding.This conclusion is drawn from the reported CoT regressions under the tested conditions.Source: K. Can Reasoning Traces Fix… · p. 20

L. Qualitative Analysis of Improved Modality Grounding

Qualitative examples show alignment-aware fine-tuning improving grounding when visual, auditory, and textual cues conflict. The model follows the requested sensory evidence where baselines are misled by conflicting modalities.

L. Qualitative Analysis of Improved Modality Grounding

  • In a dog-video/phone-ringing example, the tuned model answers from the auditory signal despite the visual contradiction, while baselines predict barking.This illustrates decoupling conflicting sensory streams.Source: L. Qualitative Analysis of Improved… · p. 20
  • With misleading captions that contradict sensory content, the tuned model ignores the linguistic hallucination and grounds answers in audio-visual evidence.The passage attributes this resilience to reduced over-reliance on visual and textual priors.Source: L. Qualitative Analysis of Improved… · p. 20
  • Figures 26–28 mark incorrect baseline predictions in red and correct model predictions in green, illustrating improved grounding in the requested sensory input.The captions characterize baseline errors as hallucinations driven by conflicting modalities.Source: Figure 26 · p. 21, Figure 27 · p. 22, Figure 28 · p. 23
Loading 2511.22826v2…