Source-linked AI summary
Tinted Frames: Question Framing Blinds Vision-Language Models
Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher, Leonid Sigal, Ritwik Gupta
TL;DR
VLMs may underuse visual evidence, and this blindness varies with linguistic framing even when the underlying visual reasoning is unchanged. The paper probes attention mechanistically, finds that constrained framings misallocate it and degrade predictions, and introduces learnable-token prompt tuning that restores more robust attention and improves consistency.
Problem
VLMs can generate plausible answers from language priors, leaving it unclear whether visual reasoning remains grounded when question framing changes.
Method
The paper uses attention rollout and intervention to connect framing with visual attention and prediction, then applies lightweight prompt tuning with learnable tokens.
Results
Constrained framings reduce and redirect visual attention, while attention realignment restores grounding and yields consistent improvements across models and benchmarks.
Takeaways & Limitations
Visual blindness in VLMs is a dynamic, framing-dependent behavior that can be realigned without modifying model weights.
Takeaways & Limitations
The hypothesis assumes that framing-induced failures originate at the prompt level rather than from a fundamental model deficiency.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
1 Introduction
VLMs can be selectively blind: linguistic framing changes visual attention even when framings require the same visual reasoning. Constrained formats reduce and misallocate attention, degrading accuracy, while lightweight prompt tuning realigns attention and improves robustness.
- VLMs modulate image attention according to open-ended, Yes/No, or multiple-choice framing despite identical underlying visual concepts.
- Cross-framing evaluation tests whether models preserve correct open-ended answers after semantically equivalent constrained reformulations.
- Constrained framings reduce overall image attention and redirect focus away from task-relevant regions.
- Attention intervention confirms that framing-related prediction failures are induced by shifts in visual attention.
- Learnable soft tokens realign constrained-framing attention with open-ended patterns, improving visual grounding and performance without modifying model weights.
2 Related Work
Prior work shows that VLMs may answer correctly without grounding responses in visual evidence, while prompt sensitivity has mainly been studied through output changes. This paper examines framing as a structural source of attention and prediction variability.
- VLM visual grounding is implicit rather than a primary training objective, so correct answers can arise from attention to irrelevant regions.
- The paper links question framing, visual attention, and output quality mechanistically.
- VLMs often attend less to visual content and disproportionately weight semantically meaningless visual tokens.
- Existing prompt-sensitivity studies examine wording changes within fixed formats, whereas framing changes the format while preserving the semantic question.
3 Hypothesis on Framing-Attention Influence
The paper hypothesizes that framing affects predictions through visual attention: framing reshapes attention, and altered attention degrades prediction quality. It investigates these pathways before using prompt tuning for realignment.
- The ideal VLM processing chain assumes question semantics, rather than framing, determines visual attention and prediction.
- The proposed pathway is F→A→Y, where framing changes visual attention and attention affects predictions.
- The study separately tests framing effects on predictions, framing effects on attention, and prompt-tuning-based attention realignment.
4 Cross-Framing Inconsistency (F→Y)
The paper measures cross-framing inconsistency by comparing semantically equivalent open-ended, Yes/No, and MCQ questions. Models often fail to preserve correct open-ended answers under constrained framing, especially for object-grounding tasks.
- The protocol retains correctly answered open-ended samples and tests whether equivalent Yes/No or MCQ reformulations preserve those answers.
- Cross-framing inconsistency is defined by correct open-ended answers followed by incorrect constrained-framing answers.
- 26% inconsistency on GQA and 38% on SeedBench are reported for Qwen2.5-VL-7B.
- More than 15% cross-framing inconsistency occurs for Qwen2.5-VL, Gemma3, and GLM4.1V on GQA.
- Object-grounding tasks, especially spatial relation and counting with multiple objects, have the highest inconsistency rates.
- These findings establish that framing alters model predictions.
5 Impact of Framing on Visual Attention (F→A→Y)
Question framing reshapes visual attention: constrained formats reduce visual reliance, redirect attention away from task-relevant regions, and produce more diffuse attention. Steering attention toward open-ended patterns recovers accuracy, with spatial allocation helping broadly and visual-energy magnitude helping grounding-heavy tasks.
- Attention measurement: Attention rollout quantifies visual information propagation from input visual tokens to generated output tokens while accounting for residual connections and receptive-field normalization.Visual Energy aggregates rollout probability mass connecting output queries to visual keys.
- Framing reshapes attention: Constrained framings consistently lower visual energy and redirect attention from task-relevant image regions toward semantically uninformative sink tokens.These effects occur for Yes/No and MCQ framings compared with open-ended generation across GQAF and V*F.
- Framing reshapes attention: Box attention falls from 19% for open-ended questions to around 12% for Yes/No and 13% for MCQ, a 40% relative drop on GQAF.On V*F, the corresponding relative drop is roughly 50%.
- Framing reshapes attention: Constrained framings increase attention entropy, leaving visual attention more diffuse and less focused on any specific region.The divergence emerges in approximately layers 12–22 and persists through later layers, where visual energy and box attention drop significantly.
- Attention predicts accuracy: Attention steering recovers accuracy: spatial allocation yields universal gains, while visual-energy magnitude primarily benefits grounding-heavy tasks.On V*F, VE steering improves Yes/No and MCQ by +2.7 and +1.6 points, while Box steering improves them by +2.0 and +2.9 points; on GQAF, Box steering gives approximately +1.3 points for both framings.
6 Attention Realignment via Prompt Tuning
The paper uses learnable soft tokens and attention alignment to make constrained framings resemble open-ended visual attention, improving consistency and benchmark performance without changing model weights.
- Attention Realignment via Prompt Tuning: The method treats open-ended attention as supervision for realigning constrained framings through prompt-level intervention.The hypothesis is that constrained-framing failures originate at the prompt level rather than from a fundamental model limitation.
- Attention Realignment via Prompt Tuning: Training uses triplets of equivalent open-ended, yes/no, and multiple-choice questions, appending K learnable tokens only to constrained inputs.The open-ended input remains unchanged while yes/no and multiple-choice inputs receive the learnable tokens.
- Attention Realignment via Prompt Tuning: The objectives combine next-token cross-entropy with attention alignment between constrained and open-ended framings.The alignment loss uses L2 visual-energy matching and KL divergence between normalized visual-attention distributions.
- 6.1 Experiments: Evaluation covers five VLMs and seven benchmarks spanning general reasoning, alignment, and fine-grained grounding tasks.The study also revisits GQA, SeedBench, GQAF, and V*F to measure attention recovery and cross-framing inconsistency.
- 6.1 Experiments: Learnable tokens substantially restore constrained-framing visual energy, increase bounding-box attention, and reduce cross-framing inconsistency across models.On SeedBench with Qwen2.5-VL-7B, inconsistency reductions reach 20% for instance interaction and 15% for counting.
- 6.1 Experiments: Performance improves across most model–benchmark combinations, with the largest gains on fine-grained grounding tasks.On V*, Qwen2.5-VL-7B improves by 2.5pp, while HRBench8k improves steadily for Gemma3 and GLM4.1V.
- 6.1 Experiments: Ablation results identify explicit attention realignment as the primary improvement driver, while cross-entropy preserves normal question-answering capability.Removing attention alignment yields only marginal gains, whereas removing cross-entropy causes significant performance drops.
7 Conclusion
The conclusion presents framing-dependent visual attention as a dynamic source of VLM blindness and describes prompt tuning as a weight-preserving mitigation. The evaluation pipeline preserves semantic content across open-ended, yes/no, and multiple-choice variants.
- 7 Conclusion: Visual attention is significantly affected by question framing, making visual blindness a dynamic behavior rather than a static architectural limitation.The proposed prompt-tuning method realigns attention under constrained framings without modifying model weights.
- 7 Conclusion: The reframing prompts require yes/no and MCQ variants to test the same knowledge and remain answerable from the same visual information.MCQ outputs use four options, while open-ended outputs use WH-questions without options.
- 7 Conclusion: The reframing procedure creates semantically equivalent open-ended, yes/no, and multiple-choice variants while keeping the underlying visual reasoning constant.GQAF and V* are curated with three framing variants per semantic query.
- 7 Conclusion: The curated datasets contain 10k GQA queries and 300 V* samples, yielding 30k and 900 examples across three framings.The final datasets are produced after filtering samples that do not follow the required JSON format.
- 7 Conclusion: MCQ questions can be converted to open-ended form by removing answer options before further reframing.This rule-based step is used because an MCQ without options is naturally open-ended.
- 7 Conclusion: Human evaluation finds consistently high reframing correctness, with spatial consistency of 96.9% and 95.4% and exclusivity of 93.7% and 95.7%.The evaluation includes 500 paired samples and 197 English-speaking evaluators from six countries.
A.5 Evaluations
The evaluation spans benchmarks with different question formats and visual demands, using benchmark-specific scoring procedures. Additional ablations examine token positioning and weighting strategies.
- A.5 Evaluations: GQA uses official scoring with simple string matching for yes/no and open-ended answers.GQA contains both open-ended and yes/no questions.
- A.5 Evaluations: SeedBench evaluation focuses on single-image questions and ranks answer options by their likelihood.The study subsamples the single-image portion of this video, single-image, and multi-image benchmark.
- A.5 Evaluations: RealWorldQA evaluation uses its MCQ subset and matches predicted letters against ground-truth letters.The benchmark assesses basic real-world spatial understanding.
- A.5 Evaluations: MME, HallusionBench, and POPE use yes/no formats with appended output instructions and string-matching evaluation.HallusionBench evaluation here focuses on its VQA split.
- A.5 Evaluations: MMMU-Pro evaluation uses its vision split, where the question and image are presented together in a screenshot.The benchmark includes standard four-option, ten-option, and purely vision-based formats.
- A.5 Evaluations: The appendix includes ablations of learnable-token positioning and weighting strategies across benchmark settings.The weighting ablation compares confidence-based weighting with equal weighting.
- A.5 Evaluations: HRBench8k and V* emphasize high-resolution inputs with small target regions requiring strong visual grounding.V* includes OCR and GPT-hard splits and has around 300 samples.
B.1 Ablation Study
The ablation study examines where learnable tokens are placed and how training samples are weighted, finding that confidence-based filtering improves performance over equal weighting.
- B.1 Ablation Study: The token-position ablation studies placements within sequences ordered as image tokens, question tokens, and instruction tokens.Learnable tokens are appended to yes/no and MCQ questions to encourage image attention.
- B.1 Ablation Study: Confidence weighting filters low-confidence samples because incorrect open-ended answers can provide inaccurate and noisy attention supervision.The framework compares hard and soft confidence thresholds with equal weighting.
- B.1 Ablation Study: Confidence weighting consistently improves performance across benchmarks compared with equal weighting.The result supports using confidence to select more reliable attention maps during alignment training.
B.2 Visual Attention Analysis for Other VLMs
Across additional VLMs, visual-attention behavior varies by framing in a pattern similar to Qwen2.5-VL-7B, with open-ended questions receiving higher bounding-box attention. Visual energy also drops on non-open-ended framings.
- Multiple models show visual-attention behavior across framings similar to Qwen2.5-VL-7B on V*F and GQAF.
- Open-ended questions receive higher bounding-box attention than constrained framings in the additional-model analysis.
- Visual energy drops significantly on non-open-ended framings.
C Dataset Examples and Qualitative Results
The appendix provides reframed GQAF examples across open-ended and Yes/No formats, alongside qualitative attention-rollout comparisons between the baseline and the learned-token approach. The learned tokens produce attention maps that focus more on task-relevant regions than the baseline.
- Randomly sampled GQAF questions are shown in original open-ended and Yes/No framings and then reframed into the other formats.
- The qualitative comparison uses Qwen2.5-VL-7B inference with and without extra learned tokens, visualizing attention rollout during output generation.
- The learned-token approach focuses significantly more on task-relevant regions, while the baseline often spreads attention to irrelevant background areas.
D Limitation
The study analyzes five diverse VLM families, but the rapidly evolving multimodal architecture landscape leaves emerging model types for future investigation. Proposed extensions include Mamba-based and Mixture-of-Experts VLMs.
- The study provides a comprehensive mechanistic analysis across five diverse and prominent VLM families.
- Future research could examine Mamba-based VLMs and Mixture-of-Experts models for similar framing-induced attention shifts.