PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

Jungyeon Lee, Yejin Yoon, Taeuk Kim

arXiv:2609.00550v1cs.CL

TL;DR

MLLMs increasingly process equivalent evidence as text, rendered images, or both, but their consistency under conflicts with parametric knowledge is unclear. This paper evaluates modality robustness across controlled conflict settings and finds unstable modality preferences, practical consequences, and only partial mitigation through fine-tuning.

  • Problem

    It remains unclear whether MLLMs process semantically equivalent textual and visual evidence consistently when that evidence conflicts with parametric knowledge.

  • Method

    The paper evaluates single- and multi-evidence knowledge conflicts across 13 MLLMs and two datasets while controlling evidence semantics and modality.

  • Results

    Modality reliance is unstable: models often prefer conflicting image evidence over equivalent text, while joint text-image preferences vary with input order, model, and dataset.

  • Takeaways & Limitations

    This instability affects multimodal RAG and enables adversarial attacks; among tested remedies, conflict-aware fine-tuning provides only partial mitigation.

  • Takeaways & Limitations

    The study examines only textual evidence and rendered-image equivalents, not naturally occurring visual evidence or other modalities, and its mechanistic account remains incomplete.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered image of the same passage, or as both together. However, it remains unclear how consistently these surface forms are processed, especially when the evidence conflicts with the model's parametric knowledge. We study modality robustness under knowledge conflict across 13 MLLMs and two datasets, and find them far from robust. (1) Contrary to common belief, models favor a context that contradicts parametric knowledge more readily in image form than in text form; (2) when a contradicting text and image are presented together, the preferred modality is essentially arbitrary, varying with input order, model, and dataset. We further demonstrate that this instability has practical consequences: it degrades performance in multimodal RAG and can be exploited by adversarial attacks. To alleviate this brittleness, we examine several simple techniques---prompting, steering, supervised fine-tuning (SFT), and direct preference optimization; the majority prove ineffective, whereas SFT achieves moderate success. We therefore call for greater awareness of this inconsistency and argue that it is fundamental, demanding attention at multiple training stages.

1 Introduction

This paper examines whether MLLMs process semantically equivalent evidence consistently when it conflicts with parametric knowledge. Across controlled modalities and downstream tasks, it finds substantial instability and only partial mitigation through conflict-aware fine-tuning.

  • In single-evidence conflicts, models tend to prefer a rendered image over text conveying the same external evidence.
  • In multimodal RAG, changing fixed-content evidence from text to image alters evidence use, while image-rendered harmful requests increase attack success rates by 6.99 percentage points on average.Image evidence also makes models less distracted by irrelevant context when salient knowledge is represented visually.
  • Simple prompting is insufficient, whereas conflict-aware fine-tuning partially mitigates modality inconsistency and balances modality preference.
  • The study formalizes modality robustness as an evidence-authority problem under knowledge conflict.
  • The evaluation disentangles semantic content, presentation modality, and parametric knowledge using single- and multi-evidence conflict settings.
  • Across 13 MLLMs and two datasets, modality reliance varies with model family, input order, and evidence composition, affecting multimodal RAG and refusal behavior.

2 Related Work

Prior work generally studies knowledge conflict with fixed text modality or reports a text-over-vision bias. This paper extends the analysis to controlled multimodal knowledge conflict and finds that the apparent bias can reverse and become arbitrary.

  • Knowledge-conflict research has generally assumed a fixed modality, typically text, while multimodal extensions have not explicitly controlled conflicts with parametric memory.
  • Prior studies commonly report that MLLMs favor text over visual evidence, a pattern summarized as modality collapse.
  • The paper studies modality robustness in knowledge-conflict scenarios where consistent handling of input context is required.
  • In this setting, the reported textual bias often reverses, with MLLMs favoring images over text.
  • Figure 1 illustrates that input order can flip which modality the model follows when textual and visual evidence support different answers.

3 Problem Formulation

The paper defines modality robustness by comparing responses to controlled textual and rendered-image evidence under conflicts between parametric and external knowledge. It evaluates single- and multi-evidence settings across datasets, models, ordering, and evidence-following metrics.

  • 3 Problem Formulation: Modality robustness asks whether MLLMs resolve knowledge conflicts consistently across different numbers and orderings of textual and visual inputs.
  • 3 Problem Formulation: The setup holds evidence semantics fixed while presenting the same external evidence as text eT or a rendered image eI, with differing responses indicating a robustness failure.
  • 3.1 Categories of Knowledge Conflict: The study covers single-evidence conflicts, multi-evidence conflicts in both input orders, and an extended three-way conflict with two competing sources.
  • 3.1 Categories of Knowledge Conflict: Parametric knowledge is estimated through unanimous agreement across five closed-book QA samples, then aligned against factual and counterfactual evidence answers.
  • 3.2 Other Experimental Factors: Experiments use CONFLICTQA’s naturally written counter-memory passages and NQ-SWAP’s retrieval-like substituted passages.
  • 3.2 Other Experimental Factors: The evaluation tests modality robustness across 13 proprietary and open-source MLLMs, model families, and scales.
  • 3.2 Other Experimental Factors: External-source following rate credits a model only when all three generated responses match the external answer, making the metric conservative.
  • 3.2 Other Experimental Factors: The normalized image-over-text gap compares modality-specific following rates, with positive values indicating greater image reliance and a range of [-100, 100].

4 MLLMs Lack Modality Robustness

Across models and datasets, modality changes how MLLMs resolve knowledge conflicts: images can be favored over equivalent text, while conflicting text–image preferences shift with order, model family, and dataset.

  • Overall finding: The dominant pattern is instability: modality is not merely a container for evidence but influences how models resolve conflicts.The normalized gap G measures image-over-text reliance, with positive values indicating greater reliance on images and larger magnitudes indicating wider differences.
  • Single-evidence conflict: Single-evidence gaps show a generally weak image preference, but GPT-4o and LLaVA provide notable exceptions.Claude Sonnet 4.5 reaches +42.86% on CONFLICTQA and +22.63% on NQ-SWAP; GPT-4o reverses across datasets, while LLaVA remains text-dominant.
  • Multi-evidence conflict: In multi-evidence conflicts, reversing input order can reverse modality reliance: 8 of 13 CONFLICTQA models and 6 of 13 NQ-SWAP models change the sign of G.The same models often shift toward image-supported answers when the order changes from xI→T to xT→I.
  • Dataset variation: Dataset differences are substantial: CONFLICTQA produces larger modality gaps than NQ-SWAP, with OmniVinci exceeding 86% on CONFLICTQA but nearly neutral on NQ-SWAP.The paper relates this contrast to broader semantic divergence in CONFLICTQA versus largely overlapping passages in NQ-SWAP.
  • Model-family variation: Modality preference also varies by model family: proprietary models are more consistently image-positive, LLaVA is text-dominant, and Qwen and InternVL are order-sensitive.Qwen and InternVL become strongly image-positive under xT→I but negative under xI→T, especially on CONFLICTQA.

5 When Modality Instability Matters

The paper tests whether modality instability affects multimodal RAG and safety. Changing a passage or instruction from text to an image alters attention and increases vulnerability to harmful-content attacks.

  • 5.1 Impact of Gold-Passage Modality in RAG: Image-rendering the gold passage can mitigate lost-in-the-middle effects in multimodal RAG when relevant evidence appears among distractors.The evaluation fixes query, passage content, and distractors while varying only the gold passage’s modality across gold-only, First, Middle, and Last settings.
  • 5.1 Impact of Gold-Passage Modality in RAG: +6.59 ROUGE-L is the gain for Gemini 2.5 Pro when the image-rendered gold passage occupies the middle position.Performance is measured on 170 MS MARCO v2.1 query–passage instances using ROUGE-L.
  • 5.2 Vulnerability to Image-Rendered Attacks: Rendering the full harmful instruction as an image increases attack success rates across the evaluated safety settings.The condition moves the entire instruction into the visual channel while leaving its semantics unchanged; ASR measures harmful responses instead of refusals.
  • 5.2 Vulnerability to Image-Rendered Attacks: 6.99 percentage points is the average ASR increase across 156 evaluation settings, with positive changes in 126/156 comparisons (80.8%).The average increase is positive across all model × baseline cells, ranging from +3.74% for Qwen to +9.30% for LLaVA.
  • 5.2 Vulnerability to Image-Rendered Attacks: The ASR increase is largest against the Text-only baseline (+11.60%) and smallest against Stable Diffusion (+4.97%).These results indicate an additional vulnerability beyond attacks that place only part of the instruction in an image.

6 Mitigating Modality Instability

The paper evaluates prompting, post-training, and inference-time interventions for modality instability. Prompting is unreliable, conflict-aware SFT moderately improves balance, while DPO and steering can trade image dominance for text dominance rather than robustly correcting both.

  • 6.1 Prompting is Not Sufficient: Prompting does not reliably control modality reliance across models or requested modalities.It partially reduces imbalance for Qwen2.5-VL-7B but amplifies bias for GPT-5.4.
  • 6.2 Conflict-Aware Fine-Tuning: Conflict-aware SFT consistently moves the normalized modality gap closer to zero across evaluated settings.In the multi-evidence xT→I condition, the image-dominant gap decreases from 73.82% to 52.04%.
  • 6.2 Conflict-Aware Fine-Tuning: Conflict-aware SFT reduces misleading image following while preserving, and slightly increasing, following of reliable image evidence.Thus, improved balance is not explained solely by suppressing image evidence.
  • 6.3 Additional Mitigation Baselines: Stronger representation steering decreases image-dominant gaps under xT→I but increases text-dominant gaps under xI→T.At stronger scales, modality preference shifts toward text under both input orders.
  • 6.3 Additional Mitigation Baselines: Effective mitigation must reduce imbalance in both directions rather than simply shifting reliance toward a fixed modality.The paper contrasts this criterion with the directional tradeoffs observed for DPO and steering.

7 Further Analysis: Input-Side Sensitivity

Input-side processing and visual rendering substantially affect modality reliance even when model weights or evidence content remain fixed. Reasonable global rendering changes have small effects, whereas answer-span highlighting produces larger shifts, while preprocessing effects are asymmetric and not fully isolating.

  • 7.1 Influence of Visual Preprocessing: Visual preprocessing alone can shift a model’s modality profile despite frozen model weights.The analysis compares Qwen2.5-VL and LLaVA-v1.6-7B, which have opposing native tendencies.
  • 7.1 Influence of Visual Preprocessing: Under LLaVA-style preprocessing, Qwen’s Gsingle changes from 9.71% to -37.23%, near LLaVA’s native -34.92%.The reverse intervention shifts LLaVA only modestly, from -34.92% to -30.91%.
  • 7.1 Influence of Visual Preprocessing: Preprocessing is one factor in modality reliance, but the setup cannot isolate which downstream components interact with it.The paper identifies matched-model training that differs only in preprocessing as a more comprehensive test.
  • 7.2 Influence of Image Presentation: Modest global changes to rendered evidence produce small shifts, whereas answer-span highlighting produces much larger shifts.The diagnostic changes image appearance while leaving query and evidence content untouched.
  • 7.2 Influence of Image Presentation: Reasonable font, color, and background variations leave gain magnitudes largely intact, but image evidence adds visual variability beyond content.Targeted highlighting is an upper-bound salience diagnostic rather than a deployable setting because it presupposes the answer.

8 Conclusion

The paper concludes that MLLMs do not resolve knowledge conflicts consistently across modalities. Modality preference shifts with dataset, input order, and evidence composition, producing practical consequences while resisting simple remedies.

  • 8 Conclusion: The same knowledge conflict can yield different answers depending on how evidence is delivered.Preference shifts with dataset, input order, and evidence composition.
  • 8 Conclusion: No single examined factor reliably predicts which modality a model will favor.The paper therefore characterizes modality preference as an evaluation-setup artifact rather than a fixed model property.
  • 8 Conclusion: Modality instability has beneficial consequences in RAG and harmful consequences through more potent adversarial attacks.The paper cites salient-passage highlighting as a beneficial example and adversarial exploitation as a harmful one.
  • 8 Conclusion: The explored remedies are largely ineffective, while conflict-aware fine-tuning offers only partial mitigation.The origin of the instability remains an open question requiring analysis across multiple training stages.

Limitations

The study examines only textual evidence and its rendered-image form, uses non-exhaustive MLLM coverage, and leaves a fuller mechanistic account for future work.

  • Limitations: The study does not examine naturally occurring visual evidence, audio, or other modalities where modality dynamics may differ.Its evidence is limited to text-modality content and rendered-image form.
  • Limitations: Although the study evaluates 13 MLLMs across proprietary and open-source families, its model coverage is not exhaustive.
  • Limitations: A more thorough mechanistic account of why modality reliance shifts remains undeveloped.The paper identifies several contributing factors but does not fully explain the documented shifts.

Ethical Statement

The safety experiments diagnose modality-dependent weaknesses in refusal behavior without releasing harmful instructions or unsafe generations. Released artifacts are restricted to evaluation metadata and non-sensitive analysis code.

  • The safety experiments diagnose modality-dependent weaknesses in multimodal refusal behavior rather than introduce new attack recipes.
  • The study reports aggregate attack-success-rate results without including concrete harmful instructions or unsafe generations.
  • Any released artifacts exclude harmful rendered prompts and are limited to evaluation metadata and non-sensitive analysis code.

A.2 Dataset Preprocessing and Instance Selection

The paper constructs controlled knowledge-conflict instances from CONFLICTQA and NQ-SWAP, applies preprocessing and model-specific filtering, and analyzes evidence divergence and order sensitivity. The datasets differ substantially in how their paired passages diverge beyond the answer strings.

  • Dataset construction: CONFLICTQA contains 9,544 GPT-4-generated instances, while NQ-SWAP contains 4,746 variants with substituted answer entities.
  • Instance selection: Model-specific filtering retains instances with consistent five-run closed-book predictions and valid opposing answer–context pairs, so usable counts differ by model.
  • Multi-evidence construction: Multi-evidence pairs use distinct external sources and evaluate both modality assignments by swapping which context is textual or visual.
  • Dataset-level evidence divergence: 97.9% of NQ-SWAP passage pairs become identical after masking answer strings, compared with 0.0% of CONFLICTQA pairs.
  • Dataset-level evidence divergence: CONFLICTQA has a larger relative token-length gap but lower token-LCS and TF–IDF cosine similarity than NQ-SWAP: 0.372 vs. 0.010, 0.188 vs. 0.942, and 0.126 vs. 0.922.
  • Order sensitivity: A high flip ratio indicates that reversing evidence order changes the model’s preferred evidence source.

C.2 Preference-based post-training

The paper compares preference-based post-training with representation-level steering for controlling modality reliance. DPO reduces misleading-image following overall but fails to produce consistent modality balance across input orders, while steering likewise changes preference directionally.

  • Preference-based post-training: DPO constructs preference pairs that select the factual answer over the answer supported by a misleading image-rendered passage.
  • DPO results: After DPO, the normalized gap changes from -68.89% to -88.08% under xI→T and from 68.47% to 65.57% under xT→I.
  • DPO results: DPO lowers average misleading-image following from 41.11% to 38.15%, a reduction of 2.96 percentage points, but does not consistently improve modality balance.
  • Representation-level steering: Increasing steering strength moves modality preference toward text, while stronger steering reduces average misleading-image following from 42.28% to 30.15%.
  • Representation-level steering: Steering is directional: reducing image preference under one order can amplify existing text preference under the other, unlike the bidirectional balance required for robustness.
  • Visual evidence processing: Near-perfect image-to-text recovery shows that models can read rendered evidence, implicating cross-modal integration rather than perception as the problem’s locus.
  • Safety implications: Image-rendered harmful instructions increase attack success rates by 6.99 percentage points on average across the reported safety evaluation.
Loading 2609.00550v1…