PaperlayerView exact revision on arXiv

Source-linked AI summary

Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts

Athulith Paraselli, Etha Tianze Hua, Ellie Pavlick

arXiv:2609.00293v1cs.CL

TL;DR

The paper asks how VLMs reconcile contextual information that conflicts with parametric knowledge, a question motivated by multimodal retrieval settings and limited evidence across modalities. Using a controlled fact-retrieval benchmark, it finds that VLMs favor context for textual entities but parametric knowledge for visual entities, linking this gap to delayed visual resolution and insufficient suppression of factual recall.

  • Problem

    Little is known about how VLMs recognize and reconcile context-memory conflicts when conflicting information crosses text and image modalities.

  • Method

    The study uses a controlled fact-retrieval setting with mixed text and image context, a 37K-instance dataset across three domains, and a uniform modality-comparison prompt.

  • Results

    VLMs prefer in-context information for textually presented entities but parametric information for visually presented entities, because slower visual resolution delays suppression of parametric retrieval.

  • Takeaways & Limitations

    Conflict resolution is not consistent across modalities; chain-of-thought has little effect, while more image-grounded context reduces visual entities’ parametric bias.

  • Takeaways & Limitations

    The controlled mock-RAG setup does not capture diverse real-world multimodal usage, and the study focuses on open-source VLMs whose mechanisms may not scale to substantially larger proprietary systems.

Abstract

from arXiv · show

We investigate how vision-language models (VLMs) handle context-memory conflicts; that is, situations in which the model is given information in context that differs from what was stored parametrically during training. We document asymmetric biases: models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images. We relate this asymmetry to the late representational alignment across modalities, showing that the longer processing time associated with resolving visual entities prevents the suppression of the model's usual factual recall mechanism, thus resulting in more parametric answers. Chain-of-thought reasoning does not appear to resolve the gap, but increasing the amount of visual information in the context does show an effect. These results illustrate the complexity of ensuring consistent behavior as models become increasingly multimodal and retrieval-augmented.

1 Introduction

The paper studies multimodal context-memory conflicts because contextual information may be more current or reliable than parametric knowledge, while cross-modal conflict resolution remains poorly understood. It introduces a controlled VLM study showing modality-dependent preferences and a mechanism for the asymmetry.

  • Multimodal retrieval and tool-use settings make it important for models to use contextual information when it is more current, task-specific, or reliable than stored knowledge.
  • Little is known about how models recognize and reconcile conflicting information across modalities.
  • The study curates a conflicting fact-retrieval dataset with 37K instances across three domains.
  • VLMs prefer in-context information for textually presented entities but parametric information for visually presented entities.
  • Entity-resolution-conditioned factual-recall suppression explains the modality asymmetry: textual conflict suppresses parametric retrieval, whereas slower visual resolution fails to suppress it sufficiently.
  • Chain-of-thought prompting has little effect, while increasing image-grounded contextual information reduces visual entities’ parametric bias.

2 Related work

Prior work studied factual retrieval and context-memory conflicts in language and vision-language models, but did not directly compare textual and visual presentations of the same entity under controlled conflict.

  • Knowledge retrieval commonly involves MLP enrichment at entity positions followed by attention-based extraction of associated attributes at generation tokens.
  • The benchmark represents each contradiction as an entity paired with a context statement that conflicts with a known fact.
  • Existing work reported delayed visual processing and modality-dependent factual-recall circuits in VLMs.
  • Prior benchmarks examined conflicting evidence for visual entities but primarily compared different VLM architectures rather than modalities directly.

3 Experimental Setup

The study constructs controlled multimodal conflicts by presenting the same entity textually or visually alongside a counterfactual textual statement and factoid query. Its benchmark spans three domains and uses a uniform prompt structure across multiple VLMs.

  • The experimental sample requires an entity with textual and visual forms, a factoid with a known parametric answer, and a plausible contradictory in-context statement.
  • The benchmark covers celebrities, buildings, and artworks.
  • The uniform prompt combines supplied textual context, an entity shown as an image or text, and a query to isolate entity-modality effects.
  • The evaluation uses Gemma, Qwen2.5-VL, and Ministral models and retains instances where entities and parametric facts were correctly identified without conflict.

4 Asymmetric Conflict Resolution Between Modalities

Under context-memory conflict, most tested VLMs produce more parametric answers for visual than textual entities, though the size of this modality gap varies across models and datasets. The proposed explanation is that slower visual resolution delays suppression of parametric enrichment.

  • Visual entities yield substantially more parametric responses than textual counterparts under context-memory conflict.
  • The modality gap is especially pronounced for relations concerning essential entity traits, such as celebrity careers.
  • For textual entities, attention to conflicting context suppresses downstream MLP-mediated parametric enrichment, whereas slower visual resolution allows enrichment to proceed largely unaffected.
  • The degree of parametric bias varies across models and datasets; for Gemma-3-12B, building locations show an 80% parametric bias gap.

5 Mechanism Underlying Modality Asymmetry

Causal interventions indicate that modality asymmetry arises because visual entities retain parametric enrichment more strongly than textual entities when context conflicts with stored knowledge. Early-to-middle entity-token MLPs promote visual parametric answers, while contextual attention suppresses these contributions more effectively for textual entities.

  • 5.1 MLPs Retrieve Parametric Information: MLP ablation narrows the modality gap by shifting visual entities toward contextual answers while leaving textual entities relatively unaffected.The intervention targets entity-token MLPs, which contribute additively across early-to-middle layers to entity enrichment and factual recall.
  • 5.1 MLPs Retrieve Parametric Information: Across models and datasets, ablating early-to-middle MLPs reduces visual parametric bias toward the textual baseline more than textual bias.The largest visual shift occurs when early MLPs are ablated, indicating their substantial contribution to parametric output.
  • 5 Mechanism Underlying Modality Asymmetry: The mechanism contrasts with standard factual recall, where Entity MLPs promote parametric information for both visual and textual entities.The competing explanations are weaker baseline promotion for text versus context-sensitive suppression of textual Entity MLPs.
  • 5.2 Attention Suppresses MLPs When Sources Conflict: Attention masking causes a large parametric shift for textual entities, and restoring clean downstream MLP activations largely neutralizes that shift.This pattern supports context-sensitive suppression of entity-token MLP contributions rather than inherently weaker textual parametric promotion.
  • 5.2 Attention Suppresses MLPs When Sources Conflict: For visual entities, attention masking produces only a small shift and clean-MLP restoration adds little, indicating weak contextual suppression of parametric enrichment.Textual entities depend more strongly on context-attention for suppressing downstream Entity MLP contributions.
  • 5.3 Slow Image Resolution Prevents MLPs From Being Suppressed: Visual parametric preference decreases when later-layer visual representations are back-patched into the first layer, eventually approaching the textual baseline.The result suggests that behavioral divergence stems from insufficient early alignment between visual and textual representations.

6 Prompt-Based Methods For Mitigating the Modality Gap

The paper evaluates prompting strategies to reduce modality asymmetry in context-memory conflicts. Chain-of-thought prompting produces inconsistent model-specific changes and does not reliably close the gap, whereas visual-context prompting changes parametric reporting in modality-dependent ways.

  • 6 Prompt-Based Methods For Mitigating the Modality Gap: The paper tests Chain-of-Thought prompting and visual-context simulation as black-box interventions for mitigating the modality gap.These strategies seek convergence across modalities without directly modifying internal activations.
  • 6.1 Chain-of-Thought Does Not Bridge the Modality Gap: Chain-of-thought prompting may help resolve visual entities before reasoning, but its effects are inconsistent across models.The proposed rationale is to make visual-entity resolution more similar to textual processing.
  • 6.1 Chain-of-Thought Does Not Bridge the Modality Gap: Across Qwen, Ministral, and Gemma, CoT leaves a similar behavioral divergence between visual and textual entities.Qwen changes little; Ministral reduces parametric reporting for both modalities; Gemma slightly narrows the gap through more text-case parametric reporting.
  • 6.1 Chain-of-Thought Does Not Bridge the Modality Gap: CoT prompting cannot be reliably leveraged to enforce task consistency across modalities.The persistent divergence also supports an account based on insufficient parametric suppression rather than weakened contextual binding.
  • 6.2 Visual Context Can Prevent Textual Suppression: Replacing textual context with an image increases parametric response rates for text entities across all models.The intervention is designed to require image resolution before processing the contextual statement.
  • 6.2 Visual Context Can Prevent Textual Suppression: Using the same image for visual entities lowers parametric rates relative to baseline, with rates approaching the textual-entity baseline.This supports the role of close representational alignment between the queried entity and contextual information.
  • 6.2 Visual Context Can Prevent Textual Suppression: Switching to a different image of the same entity weakens the visual-context effect.The authors connect this fragility to the need for precise representational alignment for context-driven suppression.

7 Conclusion

Across models and datasets, VLMs rely more heavily on parametric knowledge for visual entities than textual counterparts. Mechanistic interventions link this divergence to incomplete suppression of parametric retrieval when representations do not align early across modalities.

  • VLMs rely on parametric knowledge at a noticeably higher rate for visual entities than for textual counterparts across models and datasets.
  • Early-layer attention suppresses downstream parametric promotion for textual entities, allowing conflicting contextual information to prevail.
  • For visual entities, the parametric signal remains relatively invariant to suppression, producing the observed modality divergence.
  • Back-patching fully resolved late-layer visual features into initial layers induces behavior more consistent with textual counterparts.
  • These findings are a first step toward accounting for modal biases when integrating external knowledge into multimodal systems.

Limitations

The study uses a controlled mock-RAG setting to reduce confounding variables, but this limits how directly its findings represent real-world multimodal usage. Its scope also excludes substantially larger proprietary systems.

  • The controlled mock-RAG setup limits confounding variables for mechanistic analysis but does not capture the diversity of real-world usage.
  • Real-world inputs may prepend visual entities or interleave multimodal information across complex, multiimage document layouts.
  • The study focuses on open-source VLMs, so whether the same mechanisms scale to substantially larger proprietary systems remains unclear.

Ethical Considerations

The study constructs controlled multimodal conflict data from public entities and factual attributes across celebrities, buildings, and artworks. Its prompts pair ground-truth knowledge with counterfactual alternatives and evaluate open-ended responses.

  • Ethical Scope: The study uses publicly available entities and factual attributes and does not introduce new private or sensitive personal information.
  • Domain Curation: The benchmark covers celebrities, buildings, and artworks using publicly available entity metadata and images.
  • Conflict Format: Each evaluation block combines ground-truth knowledge, counterfactual mis-knowledge, and queries testing reliance on context versus internal memory.
  • Conflict Format: Conflicting attributes are presented in consistent domain-specific formats for celebrities, buildings, and artworks.
  • Conflict Format: Entity identifiers are formatted with structural context for buildings and artworks, while celebrity queries use the individual’s full name.
  • Conflict Format: Artwork text includes the completion year because factual recall performance otherwise suffered compared with the visual case.

A.3 Model Specific Dataset Filtering

The filtering procedure retains only cases where models both recognize the visual entity and retrieve its ground-truth fact parametrically. An LLM judge then classifies conflict responses semantically into contextual, parametric, or neither outcomes.

  • Two-stage filtering: The two-stage filter first verifies that each model can identify the entity in the image.
  • Two-stage filtering: The second stage tests whether the model recalls the relevant internal fact for both textual and visual entity presentations.
  • Response classification: The LLM judge classifies responses as supporting the parametric answer, the conflicting contextual answer, or neither.
  • Response classification: Semantic judging accommodates paraphrases and formatting variation rather than relying on exact string matching.
  • Two-stage filtering: Only entities passing visual recognition and correct factual recall are retained for final evaluation datasets.
  • Interpretation: Retention rates provide insight into parametric knowledge strength, while the strongest modal effects occur for core entity relations.

B Effect of Window Size on MLP Ablations

The study ablates MLPs at entity positions using multiple window sizes and measures shifts between parametric and contextual answers. Larger windows produce the clearest behavioral changes, while margin shifts can also reflect unrelated outputs.

  • Ablation setup: MLPs are ablated over entity positions with window sizes of 1, 3, 5, and 8.Visual entity positions are image patches; textual positions include the entity’s full name and contextualizing tokens.
  • Behavioral effects: Window sizes of 5 and 8 produce the most apparent shifts for both modalities.
  • Measurement: The margin metric measures models’ probabilities of producing the parametric and contextual answer sequences as continuous signals.
  • Interpretation: Margin changes may reflect interventions inducing outputs that are neither the parametric nor contextual answer.Such outputs may occupy an irrelevant part of the model’s vocabulary space.

C.1 MLP Ablations

The ablation analyses examine whether interventions shift responses between parametric and contextual answers without generally producing unrelated outputs. They also compare MLP, attention-masking, and back-patching behavior across modalities and model families.

  • MLP ablations: MLP ablation primarily shifts responses between parametric and contextual behavior because the neither rate remains relatively low.The result is shown for the ablation experiments in Figure 10.
  • Attention masking: Attention masking keeps the neither rate below 10% across all models, indicating that it generally does not induce out-of-distribution outputs.
  • Back-patching: Back-patching produces near-zero neither rates until the last layer for Qwen and Gemma, while Ministral’s neither rate begins climbing around layer 30.The layer-29 intervention appears to lessen the modality gap, while later representations may no longer relate to the queried entity.
  • Visual context: Providing a visual reference in context reverses the behavioral pattern, with visual entities favoring contextual information and textual entities reporting parametric knowledge more often.The authors infer that early entity alignment mediates MLP suppression, with model-specific differences for Ministral.
  • Dataset controls: Across dataset splits, the mechanistic trends remain similar, although Gemma shows a small amount of suppression in the celebrity dataset.The suppression remains substantially smaller than in the text condition and does not meaningfully affect the conclusions.
Loading 2609.00293v1…