PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

arXiv:2606.00959v2cs.AI

TL;DR

Multimodal evaluations reveal what models predict more readily than how video and audio contribute to decisions, motivating a language-conditioned analysis of sensory information. The paper introduces Sensory PID and PID-guided sample reweighting, finding a sensory synergy bottleneck with visual-first, instruction-conditioned processing and initial adaptation gains, while identifying a profile-dependent boundary.

  • Problem

    Current multimodal evaluations provide limited insight into how different modalities contribute to predictions, an important gap for understanding reliable multimodal reasoning.

  • Method

    The paper introduces Sensory PID, a conditional decision-level decomposition of language-conditioned video–audio information, and a PID-guided policy that converts local diagnostics into sample-reweighting signals.

  • Results

    Sensory PID reveals a sensory synergy bottleneck: even on audio–visual fusion tasks, decisions are dominated by modality-unique information, with visual dominance, visual-first dynamics, and instruction-conditioned fusion.

  • Takeaways & Limitations

    Local PID diagnostics provide initial evidence for improving reasoning and grounding through targeted reweighting, while reference analyses broadly recover prior vision–language PID patterns.

  • Takeaways & Limitations

    The reweighting experiments do not establish that the policy can create fusion capacity when baseline decisions remain strongly language-prior-reliant.

Abstract

from arXiv · show

Understanding how multimodal large language models use different modalities is important for reliable reasoning. We employ Partial Information Decomposition (PID) as a decision-level lens and introduce Sensory PID, a conditional formulation that conditions on language and separates unique, redundant, and synergistic contributions from video and audio. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck: even on audio-visual fusion tasks, decisions remain dominated by modality-unique information, with stronger reliance on vision. Modality-shuffling interventions support this asymmetry, while layer-wise analysis reveals a visual-first computation pattern and instruction perturbations show that late-stage sensory fusion is conditioned by language. Beyond diagnosis, PID-guided sample reweighting provides initial evidence that local diagnostic signals can improve multimodal reasoning and grounding performance. As reference validation, our vision-language analysis broadly corroborates previously reported decision-level PID patterns across tasks, models, interventions, and layers.

1. Introduction

The paper addresses limited insight into how modalities contribute to multimodal decisions by introducing Sensory PID for language-conditioned video–audio interaction. Applied to omni-modal models, it finds modality-unique information dominates even fusion tasks, with visual-first and instruction-conditioned sensory processing; PID-guided reweighting offers initial adaptation evidence.

  • Current evaluations mainly measure predictions, providing limited insight into how different modalities contribute to those decisions.
  • Sensory PID conditions on language and separates video-unique, audio-unique, redundant, and synergistic sensory information.
  • Omni-modal sensory interaction findings: Even on joint audio–visual questions, decision-relevant information remains dominated by modality-unique contributions, with interventions supporting visual dominance.
  • Omni-modal sensory interaction findings: Layer-wise analysis finds visual-unique information becomes decision-relevant before auditory information and audiovisual synergy, while late sensory fusion varies with linguistic task demand.
  • Reference validation: A reference analysis across vision–language models broadly corroborates previously reported task-, model-, intervention-, and layer-dependent PID patterns.
  • PID-guided sample reweighting: PID-guided reweighting provides initial evidence of reasoning and grounding gains in joint-modality-reliant models, while exposing a profile-dependent boundary case.

2. Methodology

The methodology combines a conditional, language-fixed Sensory PID decomposition with a PID-guided sample-reweighting intervention. A BATCH-based, training-free pipeline estimates sensory contributions from shared representations and text-conditioned full or partial-sensory predictions.

  • The method has two components: Sensory PID characterizes language-conditioned video–audio contributions, while sample reweighting converts local diagnostics into adaptation signals.
  • Conditional Sensory PID: Sensory PID decomposes I(Y ; V, A | T) into video-unique, audio-unique, redundant, and jointly available synergistic information.
  • Conditional Sensory PID: The instruction remains fixed as task context, separating task specification from sensory evidence rather than treating language as a symmetric PID source.
  • Estimation pipeline: BATCH estimates the decomposition from mean-pooled video and audio representations and text-conditioned decision distributions, using full, video-only, and audio-only conditions.
  • Estimation pipeline: Partial-sensory conditions are approximated by replacing the masked modality’s embeddings while retaining the other modality and the instruction.
  • Estimation pipeline: The estimator includes stabilization because partial-sensory candidate distributions can become unstable, and continuous confidence-dependent stabilization avoids abrupt replacement changes.
  • Estimator scope: BATCH, pooling, corruption, confidence regularization, and soft aggregation are implementation choices rather than the paper’s methodological contributions.
  • PID-guided reweighting: The reweighting policy targets samples with unrealized fusion potential or language-side shortcuts, upweighting selected synergy-gap examples and downweighting shortcut examples.

3. Experimental Setup

The experiments primarily evaluate language-conditioned sensory interaction in omni-modal models, with a separate vision–language analysis serving as reference validation. The setup combines zero-shot MUSIC-AVQA testing with modality interventions, layer-wise analysis, instruction perturbations, and broad LVLM benchmarking.

  • Primary experiments study Sensory PID in omni-modal models, while vision–language evaluation checks whether the decision-level PID pipeline recovers prior patterns.
  • Omni-modal evaluation: The omni-modal study evaluates Qwen2.5-Omni 3B and 7B and VITA-1.5 7B across two model families and multiple parameter scales.
  • Omni-modal evaluation: MUSIC-AVQA is divided into balanced Audio-Focused, Visual-Focused, and AV-Fusion subsets, with AV-Fusion serving as the primary synergy testbed.
  • Sensory protocol: Models are evaluated zero-shot with standardized prompts and deterministic greedy decoding, retaining the same instruction across full and partial-sensory conditions.
  • Sensory protocol: Video or audio streams are independently shuffled across samples, and Sensory PID components are traced across transformer depth to probe aligned evidence and computation timing.
  • Vision–language reference: The reference evaluation covers 20 vision–language models from 7 families across six benchmarks and includes a text-only condition for measuring image-related accuracy change.

4. Sensory Interaction in Omni-Modal Models

Sensory PID finds that omni-modal decisions on audio-visual fusion questions remain dominated by modality-unique information, especially vision, while audiovisual synergy emerges later and depends on language-specified task demands.

  • Sensory synergy bottleneck: Unique visual information generally exceeds unique auditory information, indicating stronger reliance on visual evidence.The visual skew motivates modality-shuffling tests of asymmetric sensitivity.
  • Sensory synergy bottleneck: Sensory synergy remains limited on AV-Fusion questions, so evaluated models derive most decision-relevant sensory information from modality-unique signals.This pattern is termed a sensory synergy bottleneck.
  • Functional validation: Shuffling video causes substantially larger performance drops than shuffling audio, including on AV-Fusion questions.The intervention preserves each modality’s marginal distribution while breaking its correspondence with the question.
  • Visual-first dynamics: Visual-unique information rises earlier and more sharply than auditory-unique information, becoming dominant through much of intermediate computation.Auditory-unique information remains comparatively limited during this stage.
  • Visual-first dynamics: Sensory synergy increases mainly in later layers but generally remains below visual-unique information, with model architecture and scale changing timing rather than ordering.Across evaluated architectures and scales, visual-unique information emerges first.
  • Instruction-conditioned fusion: Changing the instruction while holding sensory inputs fixed alters late-stage audiovisual synergy while leaving earlier unimodal dynamics comparatively stable.Less fusion-dependent instructions attenuate late-layer synergy, showing that language conditions sensory integration.

5. Evaluating PID-Guided Reweighting

PID-guided sample reweighting improves several reasoning and grounding outcomes over matched alternatives in joint-modality-reliant settings, but its benefits are selective and do not extend to the evaluated strongly language-prior-reliant model.

  • Main results: LoRA-PID improves MMBench by +1.1 pp, MMStar by +2.3 pp, and POPE by +1.3 pp relative to uniform fine-tuning.The MMStar improvement has a 95% paired-bootstrap confidence interval of [1.4, 3.2] pp.
  • Main results: On MMStar, LoRA-PID remains positive relative to random reweighting (+2.8 pp), accuracy-based mining (+1.8 pp), and ablation-based selection (+1.3 pp).These matched comparisons test whether gains exceed generic non-uniform weighting, difficulty mining, or modality sensitivity.
  • Where reweighting helps: Relative to uniform LoRA, LoRA-PID decreases MMMU by 0.5 pp and PMC-VQA by 0.3 pp.This selective trade-off is consistent with downweighting examples exhibiting high text-unique reliance.
  • Where reweighting helps: The largest MMBench gains occur in Spatial Reasoning (+3.5 pp), Attribute Comparison (+2.8 pp), and Action Recognition (+2.5 pp), while OCR/Text Recognition and Commonsense each decline by −0.3 pp.Improvements are smaller for Object Localization (+1.1 pp) and Scene Classification (+0.5 pp).
  • Post-tuning information profile: LoRA-PID raises held-out MMStar synergy from Svl = 1.20 to Svl = 1.36 and increases the synergy share from 67.5% to 73.9%.Utxt decreases from 0.56 to 0.46, while matched alternatives produce smaller profile shifts.
  • Cross-model behavior: The intervention improves LLaVA-OneVision-7B but not the strongly language-prior-reliant Gemma3-12B, whose MMStar score decreases from 56.2 to 54.3.For Gemma3-12B, post-tuning Svl changes only from 0.20 to 0.21, and MMMU decreases from 59.3 to 54.1.

6. Related Work

The paper situates its decision-level PID analysis within broader multimodal studies using attention, representation alignment, modality ablation, and prior PID-based characterizations.

  • Related work: Existing multimodal analyses examine attention, representation alignment, and modality ablation, while prior work uses PID to characterize multimodal information structure.The paper builds on these complementary perspectives rather than treating PID as the only analysis framework.

7. Conclusion

The paper introduces Sensory PID for language-conditioned video–audio interaction and uses PID-derived local diagnostics to guide vision–language sample reweighting. The analyses identify modality-unique dominance, visual-first dynamics, and instruction-conditioned fusion while extending decision-level PID evaluation.

  • Sensory PID conditions on language and decomposes video–audio decision information into video-unique, audio-unique, redundant, and synergistic components.
  • PID separates decision-relevant information into unique, redundant, and synergistic components whose sum equals joint mutual information.
  • BATCH makes PID estimation tractable by learning source encoders and Sinkhorn-normalized couplings under preserved source-marginal constraints.
  • The vision–language reference analysis uses mean-pooled modality representations and calibrated embedding masking to approximate partial-input predictive distributions.
  • Local sample-level mutual-information contributions are retained to construct synergy and text-unique reliance scores for PID-guided reweighting.

A.4. BATCH-Induced Local Diagnostic Scores: Full Derivation

The appendix derives BATCH-induced local PID diagnostics from sample-level mutual-information terms and evaluates their stability across alternative coupling anchors. These diagnostics support sample ranking for PID-guided reweighting.

  • The four PID mutual-information quantities are implemented as sample-level expectations before aggregation into local diagnostic contributions.
  • The row-anchored coupling contribution fixes one visual sample and marginalizes over textual samples through the learned Sinkhorn coupling.
  • Across 3,000 profiled samples, row- and column-anchored rankings correlate at ρ = 0.87 with 95% CI [0.83, 0.91].
  • Averaging scores over K = 50 random batch draws with batch size 256 reduces dependence on mini-batch composition.

C. Limitations and Future Directions

The study is limited by discrete decision spaces, approximated unimodal conditionals, statistical rather than causal interpretation, and proof-of-concept reweighting. Future work targets open-ended outputs, training-time diagnostics, and PID-guided data construction.

  • Limitations: The estimation framework primarily supports discrete multiple-choice decisions, leaving open-ended generation and sequence-level information estimation unresolved.
  • Limitations: Calibrated embedding masking approximates unimodal conditionals but may produce hidden states different from native partial-modality inference, biasing unique-information estimates.
  • Limitations: PID measures statistical dependence in model-induced predictive distributions and does not identify the internal mechanisms responsible for observed behavior.
  • Limitations: The reweighting experiment is a proof of concept using lightweight LoRA and limited policies, not a complete recipe for larger-scale or open-ended adaptation.
  • Future Directions: Future directions include extending PID beyond multiple-choice outputs, using it during training, and constructing benchmarks that isolate cross-modal reasoning.

E.1. Vision–Language Benchmarks

The evaluated vision–language benchmarks separate into synergy-dominant tasks, where visual and textual evidence are jointly used, and text-unique or prior-dominant tasks, where language-side information is more prominent.

  • Synergy-Dominant Reasoning and Grounding Benchmarks: MMBench, MMStar, and POPE are characterized as synergy-dominant benchmarks in the PID analysis.MMBench spans fine-grained abilities, MMStar is designed to require visual information, and POPE probes object-level grounding.
  • Text-Unique / Prior-Dominant Knowledge Benchmarks: MMMU and PMC-VQA tend toward text-unique profiles despite containing heterogeneous visual or medical inputs.Their questions also require substantial domain or medical knowledge, which contributes to language-side predictive information.
  • Benchmark contrasts: Reefknot can differ from POPE despite both addressing hallucination-related behavior, showing that relational questions may elicit stronger text-unique reliance.The passage attributes this contrast to relational patterns being partially supported by textual priors or common visual-language associations.
  • Tri-modal evaluation: MUSIC-AVQA is used for tri-modal evaluation because its audio-visual questions support separate audio-focused, visual-focused, and jointly requiring subsets.The balanced evaluation set contains 1,800 samples in each subset, including an AV-Fusion subset for testing sensory synergy.

F.1. Benchmark-Level Interaction Profiles

Across six benchmarks and 20 models, PID profiles distinguish synergy-dominant from text-unique-dominant tasks, with model-family tendencies, behavioral correlations, and layer-wise dynamics depending on the benchmark.

  • Benchmark-level separation: MMBench, MMStar, and POPE have Svl > Utxt, whereas MMMU, PMC-VQA, and Reefknot have Utxt > Svl with Utxt exceeding 60%.The expanded evaluation broadly reproduces the reported task-level separation between synergy-heavy and text-unique profiles.
  • Model-family profiles: Qwen2/2.5-VL, Qwen3-VL, InternVL3, and LLaVA-OneVision generally show more joint-modality reliance, while Gemma3 and Cambrian show stronger language-prior reliance.Qwen3-VL preserves its profile from 2B to 32B and Gemma3 remains comparatively text-unique-heavy from 4B to 27B, but profiles vary by benchmark.
  • Behavioral validity: On MMBench, MMStar, and POPE, ρ(Svl, ∆vision) ≥0.798 and ρ(Svl, Acc) ≥0.718 (p < 0.001).By contrast, Utxt is negatively associated with ∆vision, with ρ ≤−0.502 (p < 0.05), on the reported benchmarks.
  • Behavioral validity: Total mutual information correlates with accuracy at ρ = 0.418–0.593 but shows little association with vision-removal sensitivity at |ρ| ≤0.118.The decomposition therefore captures behavioral distinctions that total decision information does not capture by itself.
  • Layer-wise dynamics: Layer-wise analysis shows weak early decision-level information, middle-layer unimodal accumulation, and late-stage growth in Svl accompanied by declining Utxt.The late transition is stronger on MMStar than MMMU, where text-unique information remains comparatively stronger.

G. Ablation Studies and Sensitivity Analysis

Sensitivity and implementation analyses test whether PID-guided estimates and reweighting procedures remain stable across estimator choices, confidence thresholds, and controlled training configurations.

  • Estimator sensitivity: PID estimates remain stable across Mean Pooling, Last Hidden State, Max Pooling, and τ ∈{0.3, 0.4, 0.5}, with typical variation below 0.01 bits.The reported MMBench ablation treats this stability as evidence that the interaction profiles are not artifacts of these estimation choices.
  • PID-guided sample selection: The reweighting procedure selects language-shortcut and under-synergized samples separately, assigning weights of 0.5 and 3.0, respectively.Samples with high fusion potential but limited current synergy receive priority among the upweighted group.
  • Training configuration: All conditions use identical LoRA settings, including rank r = 16, scaling factor α = 32, dropout 0.05, and learning rate 2 × 10^-5.Adapters target the last 20% of transformer layers, motivated by late-layer emergence of cross-modal synergy.
  • Evaluation controls: The evaluation excludes samples from the named benchmark test sets during PID profiling, hyperparameter selection, and LoRA training.Post-tuning diagnostics use a held-out MMStar evaluation subset and results are averaged over three seeds.

H.4. Supplementary Results

Supplementary experiments characterize uncertainty, category-level behavior, cross-model variation, and hyperparameter sensitivity for PID-guided reweighting and its benchmark outcomes.

  • Subset results: PID-guided gains are largest on POPE-Adversarial, reaching +1.9 over Uniform.The result is reported as consistent with reducing shortcut-driven co-occurrence hallucination priors.
  • Cross-model results: The Gemma3-12B negative case, at +0.3 within seed variation, suggests reweighting amplifies existing fusion capacity rather than creating it.This result reinforces the diagnostic interpretation of modality-use profiles.
  • Upweight sensitivity: Performance peaks at a 3× upweight factor and plateaus at 5×, while MMMU performance decreases monotonically across the tested factors.The sensitivity study varies the synergy-gap upweight while fixing the shortcut downweight at 0.5×.
  • Selection-count sensitivity: Selecting shortcuts too aggressively at 40% harms knowledge benchmarks, whereas selecting only 10% limits the method’s impact.The result indicates a sensitivity to the shortcut-selection count rather than a uniformly beneficial increase or decrease.
  • Per-seed results: LoRA-PID achieves the highest MMStar accuracy across all three seeds, with a worst-case advantage of +1.3 over LoRA-Ablation.The comparison is reported for Qwen2.5-VL-7B, with Condition D averaging over three random assignments per seed.
Loading 2606.00959v2…