PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, Vésteinn Snæbjarnarson

arXiv:2607.26326v1cs.CV

TL;DR

MLLMs may fail when visual evidence conflicts with pretrained knowledge, but it is unclear whether they fail to perceive that evidence or to use it. This paper separates these possibilities with reconstruction and context-sensitivity diagnostics, finding that coarse visual evidence is preserved while reliance on vision versus prior knowledge remains unstable but steerable.

  • Problem

    It is unclear whether MLLM failures on conflicting visual evidence reflect degraded visual representations or the model’s failure to use available evidence.

  • Method

    The study reconstructs images from frozen MLLM image tokens and introduces WhatIfVis to measure whether models follow visual context or language priors under controlled instructions.

  • Results

    Coarse visual evidence remains encoded, while vanilla models inconsistently follow or disregard it; activation patching and learned steering vectors improve controllability across models.

  • Takeaways & Limitations

    For the studied coarse attributes, MLLM bottlenecks lie in controlling visual-evidence use rather than in preserving visual information.

  • Takeaways & Limitations

    The study distinguishes perception from utilization only for coarse attributes, not fine-grained information that vision encoders may discard.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

1 Introduction

This work separates visual perception from visual-evidence utilization in MLLMs, testing whether coarse attributes are encoded but inconsistently used against pretrained priors. It introduces reconstruction and context-sensitivity analyses to diagnose and control this vision-versus-prior trade-off.

  • Motivation: The paper distinguishes perception failure, where visual evidence is degraded, from utilization failure, where evidence remains available but lacks a consistent policy for use.Standard multimodal benchmarks conflate these stages, preventing them from isolating whether downstream errors arise from visual blind spots or utilization errors.
  • Contributions: A generative probe reconstructs counterfactual images from final-layer image tokens, finding coarse content remains encoded across three model families and implicating visual-evidence use rather than perception.The analysis focuses on object-level attributes such as identity, color, size, count, and weight, which vision encoders capture more reliably than fine-grained details.
  • Contributions: WhatIfVis uses counterfactual images and matched textual controls to expose bidirectional context-sensitivity failures that marginal accuracy can hide.Vanilla models over-trust visual evidence for some attributes while disregarding it for others, revealing the utilization failure hypothesis.
  • Contributions: Activation patching identifies architecture-specific layers that resolve the vision-versus-prior trade-off and reveals a compact one-dimensional subspace that functions as a controllable scalar knob.Manipulating this subspace improves controllability over the vanilla model, even without explicit instruction.
  • Contributions: Across six models spanning three attention architectures, the vision-versus-prior trade-off resolves within a narrow, architecture-specific band of layers.The examined families include standard-transformer, linear-attention-hybrid, and per-layer-embedding designs.

2 Background

The paper builds on the standard MLLM architecture, where a vision encoder and projector connect image features to a pretrained language model. It also adopts subspace interventions that encode and control context-following intent in the residual stream, extending this question to visual context.

  • Multimodal language models: MLLMs couple a pretrained language model with a vision encoder and lightweight projector that maps image patch features into the language model’s embedding space.Given an image, the encoder produces patch features that the projector converts into visual tokens for multimodal conditioning.
  • Multimodal language models: The study covers MLLM families using language-supervised vision transformers such as CLIP or SigLIP and projectors ranging from MLPs to resamplers.The named model families include Qwen2.5-VL and Qwen3.5.
  • Subspace intervention: Prior work shows that context-use instructions can be encoded in a residual-stream vector, enabling context sensitivity to be controlled by editing that vector while largely preserving other information.The method identifies a critical layer and decomposes the representation into an intent-encoding component and its orthogonal complement using a rank-1 projection.
  • Static steering: The paper learns a unit steering vector with other parameters frozen, then asks whether a scalar-controlled subspace can similarly set visual-context intent without prompt instructions.The scalar takes model-specific values corresponding to following the visual context or the prior.

3 Image Reconstruction from MLLMs

The reconstruction probe tests whether coarse visual evidence is lost before reaching the language backbone or instead remains available but unused. Across three architecturally distinct MLLMs, counterfactual attributes remain decodable from final-layer image tokens, indicating a post-perceptual utilization bottleneck.

  • Motivation: Task accuracy alone cannot distinguish degraded visual encoding from visual information arriving but going unused, because language priors can supply answers.The reconstruction probe separates H1, information loss before language-backbone use, from H2, information arriving but remaining unused.
  • Probing Design: The reconstruction setup uses frozen MLLMs’ final-layer image tokens with a trainable connector and decoder trained only on real-world images.Because the connector and decoder never see counterfactuals during training, recovery reads retained representation content rather than decoder-generated prior knowledge.
  • Results: Reconstructions are metrically closer to counterfactual references than paired real-world images across all models and metrics, with mIoU exceeding 0.86.The metrics assess complementary fidelity scales, while mIoU indicates that identity, boundaries, and layout survive the backbone.
  • Results: 93.4%, 97.8%, and 97.8% of reconstructions from counterfactual tokens correctly recover the anomalous count across three models, according to human evaluation.Two annotators evaluated object, color, and leg count, with Cohen’s κ=0.84; the counterfactual attribute was reproduced only when carried by the tokens.
  • Takeaway: At the tested coarse-grained level, counterfactual evidence survives in final-layer image tokens and is decodable across three architecturally distinct MLLMs, yet models cannot reliably use it.The findings establish that the content is present while leaving its utilization as the bottleneck.

4 Finding the Knob Behind Visual Context Sensitivity

The section establishes that visual context controllability is measurable with matched visual-prior conflicts, improves substantially through SFT, and reflects a narrow, architecture-specific vision-versus-prior trade-off in the network.

  • Benchmark and evaluation: The controlled setup holds the image and question fixed while changing only whether the model should follow visual evidence or rely on prior knowledge.Pair-accuracy requires correct answers under both context and prior intents on the same example; probability scoring reduces sensitivity to output formatting.
  • Benchmark and evaluation: WhatIfVis contains 3,049 manually inspected counterfactual samples spanning five conflict types: Spatial-Temporal, Color, Count, Size, and Weight.Perception tasks test directly observable attributes, whereas Perception + Reasoning tasks require multi-hop reasoning beyond directly visible features.
  • Adapt to the task: Vanilla models show poor and task-dependent controllability: they favor priors on Spatial-Temporal, Count, and Weight, but rely more on images for Size and Color.Baseline pair-accuracy approaches zero for Weight and Count, while the strongest models approach 50 on Color.
  • Adapt to the task: SFT raises macro pair-accuracy from 14.7 to 52.7 across six models, with Color and Spatial-Temporal approaching 80, while gains transfer across held-out tasks.Transfer is weaker for Count and Weight, which require inference beyond directly visible features.
  • Localize the trade-off: Activation patching localizes the vision-versus-prior trade-off to a narrow, recurring mid-to-upper layer band at architecture-specific depths across all six models.Patching important layers increases the probability of the corresponding source answer, whether the source instruction favors context or prior knowledge.

5 A Controllable Subspace for Visual Context Sensitivity

The vision-versus-prior trade-off is controllable along a single learned vector, enabling instruction-free steering that improves controllability over vanilla models, though it remains below SFT.

  • 5 A Controllable Subspace for Visual Context Sensitivity: A unit vector spanning the patching-identified trade-off subspace provides a one-dimensional control knob for steering toward visual context or the prior.The intervention uses scalar c(w) values applied at the identified layers without intent instructions.
  • 5 A Controllable Subspace for Visual Context Sensitivity: Mean PairAcc rises from 14.7 to 37.7 with instruction-free subspace steering, versus 52.7 for SFT (Figure 7, Table 6).The scalar c(w) replaces verbal intent instructions during inference.
  • 5 A Controllable Subspace for Visual Context Sensitivity: The improvement depends on the learned direction: a matched random rank-1 vector reaches only 5.9, below the 14.7 vanilla baseline.The random intervention disrupts the model rather than steering it, showing that the effect is not due merely to rank-1 expressivity.

6 Comparison with Textual Context

Text is substantially more controllable than vision in MLLMs, and the widening gap with model scale is chiefly attributable to difficulty using visual evidence rather than ignoring the prior.

  • Comparison with Textual Context: Text provides an instruction-following upper bound rather than a fair head-to-head comparison because each statement explicitly asserts the counterfactual image content.The paired text channel is used as a same-mechanism reference for how much harder the visual channel is to control.
  • Comparison with Textual Context: 14.7 image versus 43.4 text at baseline, with modality gaps widening across all three model families as model size increases.The gaps grow from 9.1 to 20.6 pp in Qwen2.5-VL, 18.0 to 55.8 pp in Qwen3.5, and 26.8 to 41.5 pp in Gemma-4.
  • Comparison with Textual Context: Separating context-intent and prior-intent accuracy localizes the modality gap to using image evidence, while ignoring evidence remains nearly modality-invariant.At baseline, prior-intent accuracy differs by only 0.6 pp between image and text: 68.9 image versus 69.5 text; it stays within a few points under interventions.

7 Related Work

Prior work frames MLLMs as systems that integrate heterogeneous modalities while balancing immediate visual context against pretrained parametric knowledge. This work extends visual-information analysis beyond the vision encoder by evaluating pixel-level reconstruction after propagation through the projector and LLM backbone.

  • Visual information in MLLMs: MLLMs are typically pretrained with next-token prediction over images, text, and videos, with early models such as BLIP (Li et al., 2022) and Flamingo (Alayrac et al., 2022) using cross-attention for visual–linguistic interaction.
  • Visual information in MLLMs: Visual-information quality can be evaluated through pixel-level reconstruction after visual representations propagate through both the projector and the LLM backbone, rather than only at the vision-encoder level.
  • Context sensitivity: MLLMs must balance immediate visual context against parametric knowledge acquired during large-scale pretraining, which supports factual recall and sequence memorization but can create knowledge conflicts.

8 Conclusions … C Implementation

The study argues that failures on coarse visual attributes usually arise after perception: visual evidence remains represented, but models do not reliably use it. WhatIfVis evaluates this mismatch with curated image–prior conflicts, while the paper documents benchmark scope and implementation choices.

  • 8 Conclusions: Visual evidence is rarely the main failure source for the studied coarse attributes because contested content remains legible in frozen MLLM representations, shifting the bottleneck to how models use encoded evidence.The conclusion explicitly separates seeing from using and locates the problem downstream of representation encoding.
  • A Limitations and Future work: The conclusions caution that the architectural evidence is not exhaustive, since designs such as mixture-of-experts routing remain untested.The stated scope covers a linear-attention hybrid and a per-layer-embedding design, rather than all MLLM architectures.
  • B Benchmark Composition and Curation: WhatIfVis contains 3,049 examples spanning five conflict types across perception and perception-plus-reasoning families.The benchmark repurposes and extends prior datasets and adds generated images, as summarized in Table 3 and Figure 4.
  • B Benchmark Composition and Curation: Gold answers are inherited from source annotations, responses use category-specific templates, and scoring compares greedy completions or model likelihoods against context and prior answers.The protocol is designed to reduce response-style effects while distinguishing image-matching from world-knowledge answers.
  • B Benchmark Composition and Curation: Train/test source diversity is deliberately maintained to limit leakage, with the Spatial-Temporal split used for SFT, activation patching, and steering.Its test set is primarily drawn from ViLP and VLind-Bench, whereas training is dominated by VLind-Bench and generated images.
  • B Benchmark Composition and Curation: Examples retain only cases where human reviewers agree that the question has different answers with and without the image, while generated images undergo independent counterfactual-faithfulness verification.For non-generated subsets, two annotators assess answerability in both settings; generated images are rated against the target counterfactual prompt.
  • C Implementation: The reconstruction setup uses three MLLM backbones, 256 image tokens, a six-layer Encoder–Projector connector, and Sana-0.6B as the image-generation head, trained on ImageNet-1K.The backbones span Qwen2.5VL-3B, Qwen3.5-4B, and Gemma-4-E2B-IT.
  • C Implementation: SFT adapts six instruction-tuned MLLMs across three model families using the 1,136-example Spatial & Temporal training set, standard next-token loss, and LoRA on attention projections.The models include two size tiers from each of Qwen2.5VL, Qwen3.5, and Gemma-4 families.

D Extra reconstruction results and exact-match results

Additional reconstruction analyses show that coarse attributes beyond count remain recoverable, while exact-match evaluation preserves the paper’s qualitative ordering despite lower absolute scores. Gemma-4-E2B produces more faithful reconstructions than Qwen models, and SFT substantially improves exact-match performance over vanilla models.

  • Reconstruction results: Reconstructions recover coarse structure across Color, Size, and Weight, with moderate SSIM and PSNR and low LPIPS relative to counterfactual references.The analysis evaluates 50 samples per subset using complementary pixel-level agreement measures.
  • Reconstruction results: Gemma-4-E2B is uniformly more faithful than the Qwen models, matching their ordering on Count; fidelity is highest for Color, while Size and Weight score lower.The lower Size and Weight fidelity broadly matches the difficulty ordering reported throughout the paper.
  • Exact-match results: SFT raises macro exact-match PairAcc from 8.2% for vanilla models to 35.9%, while exact-match preserves the qualitative ordering despite penalizing formatting mismatches.The comparison uses the image channel and the stricter greedy-decoded exact-match scorer for both intents.

E Steering: Per-Model Results and the Specificity Control · F Prompts

A single learned steering vector improves visual-context controllability across all six models without intent instructions, while matched prompt templates specify whether to follow or ignore image or textual evidence.

  • E Steering: Per-Model Results and the Specificity Control: 14.7% to 37.7% macro pair-accuracy: a single learned vector improves steering without intent instructions across all six models.Per-model gains range from +14.4 pp for Qwen3.5-4B to +34.8 pp for Qwen2.5-VL-3B, with final values spanning 33.9%–45.9%.
  • E Steering: Per-Model Results and the Specificity Control: Exact-match scoring reproduces the SFT lift qualitatively, with larger gains on perceivable attributes and marginal gains on reasoning-heavy tasks.The comparison uses a Spatial-Temporal-only LoRA SFT model evaluated on other tasks out of distribution, alongside baseline intent instructions.
  • E Steering: Per-Model Results and the Specificity Control: The modality gap reflects difficulty following visual evidence: context-intent accuracy is 19.0–30.8 percentage points lower for images than matched text, while prior-intent accuracy differs by at most 4.5 points.Results are probability-scorer, macro-averaged over six models, with PairAcc separated into its two intent components.
  • E Steering: Per-Model Results and the Specificity Control: 5.9%: a random rank-1 projection underperforms the 14.7% baseline, showing that gains come from the learned vector rather than rank-1 expressivity.The random projection uses the same layers and per-model multipliers, with three seeds and per-model standard deviation ≤3.5 pp.
  • E Steering: Per-Model Results and the Specificity Control: Per-model steering remains highly stable across three refitting seeds, with macro pair-accuracy standard deviation ≤0.12 pp on five models.The seed-stability sweep covers the five models for which the sweep completed.
  • F Prompts: Image prompts instruct the model to consider only the image or ignore it, while matched-text prompts analogously direct attention to or away from the statement.The prompt templates use concise one-sentence answers and explicitly state the relevant evidence source.

G Activation Patching: Full Results

The full activation-patching analysis localizes the vision-versus-prior trade-off by testing cross-direction substitutions across layers. It applies both patching directions to each SFT model and identifies decision-flipping layer bands with a binary search.

  • Method: For each SFT model, cross-direction patching substitutes image-channel multi-head attention outputs at the target pass’s last-token position in both ctx→pri and pri→ctx directions.The source pass is the prior-intent or context-intent pass, while the target pass uses the opposite intent.
  • Full Results: Figure 10 reports Qwen3.5-4B image-channel cross-direction patching curves, showing source-minus-target answer-probability changes across layers with patched layers shaded.These curves complement the main results in Figure 6.
  • Method: A binary range search with threshold 0.85 identifies the contiguous layer band whose substitutions flip the decision.

H Comparison with Textual Context: Full Results

Matched text statements provide an instruction-following upper bound rather than a fair modality comparison. Image–text gaps vary sharply by task, with Color and Spatial-Temporal closer to text performance than Count and Weight.

  • Comparison setup: Text statements assert each counterfactual fact outright, making the text condition an instruction-following upper bound rather than a fair head-to-head comparison.The text channel nevertheless runs the same base, SFT, and steering pipeline on matched content.
  • Per-task results: 32.8% and 20.8% image-only accuracy on Color and Spatial-Temporal approaches text-only performance of 49.1% and 50.0%, respectively.These tasks show the smallest modality gaps among the reported comparisons.
  • Per-task results: 2.8% and 0.5% image-only accuracy on Count and Weight contrasts with text-only performance of 60.4% and 29.2%, respectively.Language can state these counterfactual facts directly, whereas images require exhaustive enumeration or inference.
  • Cross-task analysis: The task divergence suggests that visual access to evidence, rather than downstream decision-making, varies across tasks.This interpretation is consistent with the attribute ordering observed in the SFT and steering lifts.
Loading 2607.26326v1…