PaperlayerView exact revision on arXiv

Source-linked AI summary

Tinted Frames: Question Framing Blinds Vision-Language Models

Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher, Leonid Sigal, Ritwik Gupta

arXiv:2603.19203v2cs.CV

TL;DR

VLMs may underuse visual evidence, and this blindness varies with linguistic framing even when the underlying visual reasoning is unchanged. The paper probes attention mechanistically, finds that constrained framings misallocate it and degrade predictions, and introduces learnable-token prompt tuning that restores more robust attention and improves consistency.

  • Problem

    VLMs can generate plausible answers from language priors, leaving it unclear whether visual reasoning remains grounded when question framing changes.

    Source: §1 · pp. 1–2

  • Method

    The paper uses attention rollout and intervention to connect framing with visual attention and prediction, then applies lightweight prompt tuning with learnable tokens.

    Source: §1 · p. 2, §5 · p. 5, Realigning attention via prompt tuning · p. 9

  • Results

    Constrained framings reduce and redirect visual attention, while attention realignment restores grounding and yields consistent improvements across models and benchmarks.

    Source: §1 · p. 2, Models and benchmarks · p. 11, §7 · p. 12

  • Takeaways & Limitations

    Visual blindness in VLMs is a dynamic, framing-dependent behavior that can be realigned without modifying model weights.

    Source: §1 · p. 2, §7 · p. 12

    The hypothesis assumes that framing-induced failures originate at the prompt level rather than from a fundamental model deficiency.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.

1 Introduction

VLMs can be selectively blind: linguistic framing changes visual attention even when framings require the same visual reasoning. Constrained formats reduce and misallocate attention, degrading accuracy, while lightweight prompt tuning realigns attention and improves robustness.

  • VLMs modulate image attention according to open-ended, Yes/No, or multiple-choice framing despite identical underlying visual concepts.Source: §1 · p. 2
  • Cross-framing evaluation tests whether models preserve correct open-ended answers after semantically equivalent constrained reformulations.Source: §1 · p. 2
  • Constrained framings reduce overall image attention and redirect focus away from task-relevant regions.Source: §1 · p. 2
  • Attention intervention confirms that framing-related prediction failures are induced by shifts in visual attention.Source: §1 · p. 2
  • Learnable soft tokens realign constrained-framing attention with open-ended patterns, improving visual grounding and performance without modifying model weights.Source: §1 · p. 2

2 Related Work

Prior work shows that VLMs may answer correctly without grounding responses in visual evidence, while prompt sensitivity has mainly been studied through output changes. This paper examines framing as a structural source of attention and prediction variability.

3 Hypothesis on Framing-Attention Influence

The paper hypothesizes that framing affects predictions through visual attention: framing reshapes attention, and altered attention degrades prediction quality. It investigates these pathways before using prompt tuning for realignment.

  • The ideal VLM processing chain assumes question semantics, rather than framing, determines visual attention and prediction.Source: §3 · p. 4
  • The proposed pathway is F→A→Y, where framing changes visual attention and attention affects predictions.Source: §3 · p. 4
  • The study separately tests framing effects on predictions, framing effects on attention, and prompt-tuning-based attention realignment.Source: §3 · p. 4

4 Cross-Framing Inconsistency (F→Y)

The paper measures cross-framing inconsistency by comparing semantically equivalent open-ended, Yes/No, and MCQ questions. Models often fail to preserve correct open-ended answers under constrained framing, especially for object-grounding tasks.

  • The protocol retains correctly answered open-ended samples and tests whether equivalent Yes/No or MCQ reformulations preserve those answers.Source: Evaluation Protocol · p. 4
  • Cross-framing inconsistency is defined by correct open-ended answers followed by incorrect constrained-framing answers.Source: Evaluation Protocol · p. 4
  • 26% inconsistency on GQA and 38% on SeedBench are reported for Qwen2.5-VL-7B.Source: Figure 3 · p. 5
  • More than 15% cross-framing inconsistency occurs for Qwen2.5-VL, Gemma3, and GLM4.1V on GQA.Source: Evaluation Protocol · p. 5
  • Object-grounding tasks, especially spatial relation and counting with multiple objects, have the highest inconsistency rates.Source: Evaluation Protocol · p. 5
  • These findings establish that framing alters model predictions.Source: Evaluation Protocol · p. 5

5 Impact of Framing on Visual Attention (F→A→Y)

Question framing reshapes visual attention: constrained formats reduce visual reliance, redirect attention away from task-relevant regions, and produce more diffuse attention. Steering attention toward open-ended patterns recovers accuracy, with spatial allocation helping broadly and visual-energy magnitude helping grounding-heavy tasks.

Attention measurement

  • Attention rollout quantifies visual information propagation from input visual tokens to generated output tokens while accounting for residual connections and receptive-field normalization.Visual Energy aggregates rollout probability mass connecting output queries to visual keys.Source: Visual attention aggregation · p. 6

Framing reshapes attention

  • Constrained framings consistently lower visual energy and redirect attention from task-relevant image regions toward semantically uninformative sink tokens.These effects occur for Yes/No and MCQ framings compared with open-ended generation across GQAF and V*F.Source: Visual energy and spatial allocation · p. 7
  • Box attention falls from 19% for open-ended questions to around 12% for Yes/No and 13% for MCQ, a 40% relative drop on GQAF.On V*F, the corresponding relative drop is roughly 50%.Source: Visual energy and spatial allocation · p. 7
  • Constrained framings increase attention entropy, leaving visual attention more diffuse and less focused on any specific region.The divergence emerges in approximately layers 12–22 and persists through later layers, where visual energy and box attention drop significantly.Source: Visual energy and spatial allocation · p. 7, Layer-wise analysis · p. 7

Attention predicts accuracy

  • Attention steering recovers accuracy: spatial allocation yields universal gains, while visual-energy magnitude primarily benefits grounding-heavy tasks.On V*F, VE steering improves Yes/No and MCQ by +2.7 and +1.6 points, while Box steering improves them by +2.0 and +2.9 points; on GQAF, Box steering gives approximately +1.3 points for both framings.Source: Results · p. 8, Takeaways · p. 9

6 Attention Realignment via Prompt Tuning

The paper uses learnable soft tokens and attention alignment to make constrained framings resemble open-ended visual attention, improving consistency and benchmark performance without changing model weights.

Attention Realignment via Prompt Tuning

  • The method treats open-ended attention as supervision for realigning constrained framings through prompt-level intervention.The hypothesis is that constrained-framing failures originate at the prompt level rather than from a fundamental model limitation.Source: §6 · p. 9, Realigning attention via prompt tuning · p. 9
  • Training uses triplets of equivalent open-ended, yes/no, and multiple-choice questions, appending K learnable tokens only to constrained inputs.The open-ended input remains unchanged while yes/no and multiple-choice inputs receive the learnable tokens.Source: Realigning attention via prompt tuning · p. 9
  • The objectives combine next-token cross-entropy with attention alignment between constrained and open-ended framings.The alignment loss uses L2 visual-energy matching and KL divergence between normalized visual-attention distributions.Source: Realigning attention via prompt tuning · pp. 9–10

6.1 Experiments

  • Evaluation covers five VLMs and seven benchmarks spanning general reasoning, alignment, and fine-grained grounding tasks.The study also revisits GQA, SeedBench, GQAF, and V*F to measure attention recovery and cross-framing inconsistency.Source: Models and benchmarks · p. 10
  • Learnable tokens substantially restore constrained-framing visual energy, increase bounding-box attention, and reduce cross-framing inconsistency across models.On SeedBench with Qwen2.5-VL-7B, inconsistency reductions reach 20% for instance interaction and 15% for counting.Source: Models and benchmarks · p. 11
  • Performance improves across most model–benchmark combinations, with the largest gains on fine-grained grounding tasks.On V*, Qwen2.5-VL-7B improves by 2.5pp, while HRBench8k improves steadily for Gemma3 and GLM4.1V.Source: Improving visual attention and… · p. 11, Table 1 · p. 11
  • Ablation results identify explicit attention realignment as the primary improvement driver, while cross-entropy preserves normal question-answering capability.Removing attention alignment yields only marginal gains, whereas removing cross-entropy causes significant performance drops.Source: Model ablation · p. 11, Table 2 · p. 12

7 Conclusion

The conclusion presents framing-dependent visual attention as a dynamic source of VLM blindness and describes prompt tuning as a weight-preserving mitigation. The evaluation pipeline preserves semantic content across open-ended, yes/no, and multiple-choice variants.

7 Conclusion

  • Visual attention is significantly affected by question framing, making visual blindness a dynamic behavior rather than a static architectural limitation.The proposed prompt-tuning method realigns attention under constrained framings without modifying model weights.Source: §7 · p. 12
  • The reframing prompts require yes/no and MCQ variants to test the same knowledge and remain answerable from the same visual information.MCQ outputs use four options, while open-ended outputs use WH-questions without options.Source: §A.1 · p. 17
  • The reframing procedure creates semantically equivalent open-ended, yes/no, and multiple-choice variants while keeping the underlying visual reasoning constant.GQAF and V* are curated with three framing variants per semantic query.Source: §A.2 · p. 18
  • The curated datasets contain 10k GQA queries and 300 V* samples, yielding 30k and 900 examples across three framings.The final datasets are produced after filtering samples that do not follow the required JSON format.Source: §A.2 · p. 18
  • MCQ questions can be converted to open-ended form by removing answer options before further reframing.This rule-based step is used because an MCQ without options is naturally open-ended.Source: §A.2 · p. 18, Figure A1 · p. 18
  • Human evaluation finds consistently high reframing correctness, with spatial consistency of 96.9% and 95.4% and exclusivity of 93.7% and 95.7%.The evaluation includes 500 paired samples and 197 English-speaking evaluators from six countries.Source: §A.2 · p. 19

A.5 Evaluations

The evaluation spans benchmarks with different question formats and visual demands, using benchmark-specific scoring procedures. Additional ablations examine token positioning and weighting strategies.

A.5 Evaluations

  • GQA uses official scoring with simple string matching for yes/no and open-ended answers.GQA contains both open-ended and yes/no questions.Source: §A.5 · p. 20
  • SeedBench evaluation focuses on single-image questions and ranks answer options by their likelihood.The study subsamples the single-image portion of this video, single-image, and multi-image benchmark.Source: SeedBench · p. 20
  • RealWorldQA evaluation uses its MCQ subset and matches predicted letters against ground-truth letters.The benchmark assesses basic real-world spatial understanding.Source: RealWorldQA · p. 20
  • MME, HallusionBench, and POPE use yes/no formats with appended output instructions and string-matching evaluation.HallusionBench evaluation here focuses on its VQA split.Source: RealWorldQA · p. 20, HallusionBench · p. 21, POPE · p. 21
  • MMMU-Pro evaluation uses its vision split, where the question and image are presented together in a screenshot.The benchmark includes standard four-option, ten-option, and purely vision-based formats.Source: MMMU-Pro · p. 20
  • The appendix includes ablations of learnable-token positioning and weighting strategies across benchmark settings.The weighting ablation compares confidence-based weighting with equal weighting.Source: Table A3 · p. 21, Table A4 · p. 21
  • HRBench8k and V* emphasize high-resolution inputs with small target regions requiring strong visual grounding.V* includes OCR and GPT-hard splits and has around 300 samples.Source: POPE · p. 21

B.1 Ablation Study

The ablation study examines where learnable tokens are placed and how training samples are weighted, finding that confidence-based filtering improves performance over equal weighting.

B.1 Ablation Study

  • The token-position ablation studies placements within sequences ordered as image tokens, question tokens, and instruction tokens.Learnable tokens are appended to yes/no and MCQ questions to encourage image attention.Source: Position of the Learnable Tokens · p. 21
  • Confidence weighting filters low-confidence samples because incorrect open-ended answers can provide inaccurate and noisy attention supervision.The framework compares hard and soft confidence thresholds with equal weighting.Source: Position of the Learnable Tokens · p. 21
  • Confidence weighting consistently improves performance across benchmarks compared with equal weighting.The result supports using confidence to select more reliable attention maps during alignment training.Source: Weighting Strategy for Alignment Loss… · p. 22

B.2 Visual Attention Analysis for Other VLMs

Across additional VLMs, visual-attention behavior varies by framing in a pattern similar to Qwen2.5-VL-7B, with open-ended questions receiving higher bounding-box attention. Visual energy also drops on non-open-ended framings.

  • Multiple models show visual-attention behavior across framings similar to Qwen2.5-VL-7B on V*F and GQAF.Source: §B.2 · p. 22
  • Open-ended questions receive higher bounding-box attention than constrained framings in the additional-model analysis.Source: §B.2 · p. 22
  • Visual energy drops significantly on non-open-ended framings.Source: Figure A3 · p. 23

C Dataset Examples and Qualitative Results

The appendix provides reframed GQAF examples across open-ended and Yes/No formats, alongside qualitative attention-rollout comparisons between the baseline and the learned-token approach. The learned tokens produce attention maps that focus more on task-relevant regions than the baseline.

D Limitation

The study analyzes five diverse VLM families, but the rapidly evolving multimodal architecture landscape leaves emerging model types for future investigation. Proposed extensions include Mamba-based and Mixture-of-Experts VLMs.

  • The study provides a comprehensive mechanistic analysis across five diverse and prominent VLM families.Source: D Limitation · p. 23
  • Future research could examine Mamba-based VLMs and Mixture-of-Experts models for similar framing-induced attention shifts.Source: D Limitation · p. 23

Related papers

Loading 2603.19203v2…