PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas

Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li

arXiv:2503.01773v3cs.CL

TL;DR

VLMs often fail at even simple spatial reasoning, and the paper investigates whether their internal visual-textual attention explains these errors. It traces attention across layers and introduces confidence-guided ADAPTVIS, which adaptively sharpens or smooths image attention. Across spatial reasoning benchmarks, the method reports improvements of up to 50% points with negligible computational cost.

  • Problem

    VLMs struggle with basic spatial relationships, while how image and text tokens interact to construct geometric understanding remains underexplored.

  • Method

    The paper analyzes attention distributions across layers and uses generation confidence to adaptively sharpen or smooth image-token attention at inference time.

  • Results

    Up to 50% points of improvement are reported across WhatsUp and VSR on synthetic and real-world images.

  • Takeaways & Limitations

    Successful spatial reasoning is associated with attention aligned to actual object locations, with confidence indicating when the model should trust or broaden its focus.

  • Takeaways & Limitations

    Effective intervention requires devising attention adjustments without prior knowledge of the attention scores before a single inference run produces the attention map.

Abstract

from arXiv · show

Large Vision Language Models (VLMs) have long struggled with spatial reasoning tasks. Surprisingly, even simple spatial reasoning tasks, such as recognizing "under" or "behind" relationships between only two objects, pose significant challenges for current VLMs. In this work, we study the spatial reasoning challenge from the lens of mechanistic interpretability, diving into the model's internal states to examine the interactions between image and text tokens. By tracing attention distribution over the image through out intermediate layers, we observe that successful spatial reasoning correlates strongly with the model's ability to align its attention distribution with actual object locations, particularly differing between familiar and unfamiliar spatial relationships. Motivated by these findings, we propose ADAPTVIS based on inference-time confidence scores to sharpen the attention on highly relevant regions when confident, while smoothing and broadening the attention window to consider a wider context when confidence is lower. This training-free decoding method shows significant improvement (e.g., up to a 50 absolute point improvement) on spatial reasoning benchmarks such as WhatsUp and VSR with negligible cost. We make code and data publicly available for research purposes at https://github.com/shiqichen17/AdaptVis.

1. Introduction

VLMs struggle with basic spatial relationships, motivating an examination of how visual and textual tokens interact. ADAPTVIS intervenes in image-token attention using generation confidence, improving focus and spatial reasoning performance.

  • 1. Introduction: VLMs frequently misidentify simple relationships such as “behind,” revealing a persistent bottleneck in vision-centric processing.The task may involve only two objects, yet models can substitute one spatial relation for another.
  • 1. Introduction: Spatial reasoning provides a lens for studying how models identify objects while maintaining broader geometric context.The paper links geometric understanding to how attention is distributed across visual tokens.
  • 1. Introduction: Image tokens comprise about 90% of the input sequence but receive only about 10% of total attention, suggesting textual priors can overshadow visual evidence.The authors use this imbalance to motivate analysis of internal token interactions.
  • 1. Introduction: ADAPTVIS sharpens image attention when generation confidence is high and smooths it when confidence is low.The intervention trusts confident attention patterns but broadens exploration when the model is uncertain.
  • 1. Introduction: Up to 50% points of improvement is reported across WhatsUp and VSR, spanning synthetic and real-world images.The reported gains are described as computationally efficient and robust across scene types.
  • 1. Introduction: Visualizations indicate that dynamically adjusted attention better aligns model focus with actual object locations and spatial relationships.The paper connects successful reasoning with knowing when to trust or question the initial spatial understanding.

2. Preliminary on VLMs

The preliminary analysis describes VLM components and examines attention over image tokens. It also introduces the WhatsUp evaluation structure and reports a strong imbalance favoring textual attention.

  • 2. Preliminary on VLMs: VLMs combine a visual encoder, a projector, and a pretrained language model built from stacked transformer layers.The visual encoder processes the image, while the projector maps image information into token space.
  • 2. Preliminary on VLMs: Multi-head self-attention computes attention across heads using query, key, and value projections, then concatenates the head outputs.The formulation uses H attention heads over token representations.
  • 2. Preliminary on VLMs: A causal mask prevents tokens from attending to future tokens in the sequence.The mask is defined as zero for permitted positions and −∞ otherwise.
  • 2. Preliminary on VLMs: WhatsUp organizes subsets by choice counts, object counts, and synthetic or real data sources, with evaluation prompts shown separately.The figure distinguishes the dataset composition from the prompts used for evaluation.
  • 2. Preliminary on VLMs: Image tokens occupy approximately 90% of the sequence but receive only about 10% of total attention on WhatsUp.The reported allocation indicates substantially greater attention to textual tokens despite their smaller count.

3. Text-Vision Attention Interactions

The analysis finds that VLMs allocate far less attention to image tokens than text tokens during spatial reasoning, and tests whether increasing visual attention improves accuracy. Uniformly increasing attention across all image tokens does not improve spatial reasoning, motivating analysis of attention distribution.

  • Experiment setup: WhatsUp combines synthetic and realistic images, including controlled subsets that vary object size and support contrastive pair and set accuracy evaluation.Controlled A contains one large and one small object, while Controlled B contains two small objects.
  • Experiment setup: The study reformats WhatsUp image-caption pairs into generative question-answering examples for tracing VLM internal states.Questions ask for spatial labels such as left, right, on, and under, using four- or six-option answer spaces depending on the subset.
  • Experiment setup: The analysis evaluates LLaVA-1.5 using exact-match accuracy across standardized spatial-label settings.The Controlled Image and COCO subsets use four labels, while VG uses six labels including behind and front.
  • Attention imbalance: Image tokens receive substantially less attention than text tokens despite comprising many more input tokens.In the analyzed setting, image sequences contain 576 tokens versus typically 30–40 text tokens, while text receives approximately nine times more attention.
  • Uniform visual scaling: Uniformly adding positive constants to image attention logits does not improve spatial reasoning accuracy.This intervention increases attention to the entire image rather than changing its spatial distribution.

4. Visual Attention Distribution

The paper links spatial-reasoning errors to where visual attention is distributed rather than merely how much image attention is present. It uses overlap-based diagnostics and temperature scaling to intervene on attention, reporting substantial gains across synthetic and real-image benchmarks.

  • Layerwise visual attention: Attention overlap with YOLO-annotated entities peaks in middle layers, suggesting these layers are most informative for image processing.Early layers receive more image attention but show low AUROC, whereas middle layers refine local visual understanding and achieve higher overlap AUROC.
  • Attention alignment: Correct answers coincide with attention focused on referenced entities, while incorrect answers show insufficient or misplaced attention.The qualitative examples use 17th-layer attention and contrast relevant-object focus with attention on irrelevant image regions.
  • Attention alignment: Overlap between YOLO annotations and attention patterns can serve as a metric for detecting answer correctness.Figure 6 reports AUROC for this overlap at the 17th and 18th layers in Controlled A.
  • ScalingVis: ScalingVis dynamically changes the temperature of image attention so confident patterns become sharper and incorrect patterns become more dispersed.Multiplying by α greater than 1 emphasizes the original distribution, whereas α less than 1 broadens it; the intervention targets final-token attention to image tokens.
  • Experiment setup: Experiments evaluate WhatsUp and VSR with accuracy and F1, selecting α on validation data before testing.The baselines include DoLa and VCD, and α is selected from the reported candidate set.
  • Results: 37.2 absolute points is the maximum reported gain from controlling image-attention distribution.Temperature below one generally helps synthetic data, while temperature above one benefits real-image datasets across the reported cases.

5. Adaptively Intervening the Attention Distribution by Model Confidence

ADAPTVIS adaptively intervenes in visual attention according to model confidence, smoothing attention when confidence is low and sharpening it when confidence is high. Experiments report substantial spatial-reasoning gains across synthetic and real-image datasets.

  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: ADAPTVIS addresses the question of when to strengthen or broaden image attention by using model confidence as an adaptive intervention signal.The method is motivated by performance differences between sharpening and smoothing across settings.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: Low confidence triggers smoothing to broaden the attended image context, while high confidence triggers sharpening to concentrate attention on key objects.The intervention applies small α < 1 for low confidence and large α > 1 for high confidence.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: Figure 8 compares accuracy and confidence changes against intervention coefficients for Cont A and Cont B, with greedy decoding at coefficient 1 as the baseline.The figure separates accuracy and confidence for each controlled dataset.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: Figure 9 illustrates smoothing on Cont A and Cont B, whereas Figure 10 illustrates sharpening on COCO and VG.Smoothing changes the focused position substantially in synthetic data, while sharpening largely reinforces existing highlighted areas in real data.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: The intervention targets the last input token’s attention directed toward image tokens.Equations 3a and 3b define the confidence-dependent attention modification.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: The evaluation uses the same setting as SCALINGVIS, with hyperparameter choice and robustness discussed in Appendix E.3.This section does not provide the appendix’s hyperparameter results.
  • 5. Adaptively Intervening the Attention Distribution by Model Confidence: Up to 50 absolute points of improvement are reported, with ADAPTVIS especially outperforming SCALINGVIS on synthetic datasets and remaining slightly better on COCO and VG.The results are reported for Controlled A, Controlled B, COCO, VG, and VSR; Table 1 reports metrics in ×10^-2 and Table 2 reports VSR Exact Match and F1.

6. Related Work

Related work covers biased attention patterns in language models, efficient training-free interventions, and failure analyses of VLM vision processing. ADAPTVIS extends attention intervention to VLMs without manual segment specification or multiple validation runs.

  • 6. Related Work: Prior language-model studies identify biased attention across context windows, including ineffective middle-context use and initial-token attention sinks.Some approaches address these biases through fine-tuning.
  • 6. Related Work: Training-free alternatives include input-adaptive calibration, position-specific interventions, and PASTA’s emphasis on selected segments for specific heads.The paper positions its approach as extending this intervention line to VLMs.
  • 6. Related Work: ADAPTVIS extends attention intervention to VLMs without manual segment specification or multiple validation runs.The related work also connects the paper to VLM hallucination and spurious-correlation analyses in multi-object recognition.

7. Conclusion and Future Work

The paper identifies insufficient image-token attention, the geometric location of attention, and generation confidence as central factors in VLM spatial reasoning. It proposes ADAPTVIS and points toward more complex geometric reasoning as future work.

  • 7. Conclusion and Future Work: VLMs allocate surprisingly insufficient attention to image tokens during spatial reasoning.This is presented as one of the paper’s central mechanistic findings.
  • 7. Conclusion and Future Work: The location of attention on image tokens is more important than attention quantity alone.The conclusion emphasizes geometric allocation rather than simply increasing visual attention.
  • 7. Conclusion and Future Work: Generation confidence serves as an indicator of image familiarity and the correctness of the model’s attention pattern.This finding motivates confidence-based adjustment of attention distribution.
  • 7. Conclusion and Future Work: ADAPTVIS dynamically adjusts attention distribution and significantly improves spatial reasoning performance.Future research is proposed for complicated geometric structures and long-horizon spatial reasoning.

Impact Statement

The paper highlights a gap between VLM object recognition and basic spatial understanding, with implications for applications requiring reliable geometric interpretation. It presents ADAPTVIS and mechanistic interpretability as directions toward broader spatial reasoning capabilities.

  • Impact Statement: Current VLMs can excel at object recognition while struggling with basic spatial relationships.This gap is framed as relevant to real-world systems that must interpret visual geometry.
  • Impact Statement: A robot that identifies objects but cannot understand their spatial relationships may struggle with picking, placing, and navigation.The paper also notes potential traffic-interpretation challenges for autonomous vehicles.
  • Impact Statement: Improved spatial reasoning from ADAPTVIS could support applications such as medical-image interpretation and augmented reality.The paper presents these as potential application opportunities.
  • Impact Statement: Future work on complex geometric structures and long-horizon spatial reasoning could advance understanding of how artificial systems process spatial information.The paper also connects this direction to cognitive science and human spatial cognition.
  • Impact Statement: The authors frame stronger visual spatial understanding as potentially relevant to healthcare, assistive technologies, urban planning, and environmental monitoring.These domains are presented as longer-term societal impact areas.

A. Limitations

ADAPTVIS and SCALINGVIS address model-related spatial hallucinations and self-alignment issues, but not errors beyond the language model’s capabilities. ADAPTVIS also requires validation-based tuning because optimal parameters vary across distributions and prompts.

  • ADAPTVIS and SCALINGVIS do not handle errors outside the language model’s capabilities, including CLIP failures.
  • ADAPTVIS requires a validation set to tune confidence threshold β and attention coefficient α across distributions and prompts.

B. Additional Results on Qwen2-VL

Additional experiments on Qwen2-VL show that attention intervention improves difficult spatial reasoning cases while preserving performance on general benchmarks. The analyses also link lower confidence to weaker spatial-relation accuracy and show that attention distributions differ across synthetic and real-image data.

  • Additional Results on Qwen2-VL: 10+ absolute points: attention intervention improves Qwen2-VL performance on VG two object, where baseline performance is lowest.The gains support generalizability to another VLM and are especially notable in challenging spatial reasoning cases.
  • Additional Results on Qwen2-VL: Attention intervention maintains performance on POPE, GQA, and TextVQA, with smaller gains than on spatial reasoning tasks.The authors attribute the difference to general QA tasks being less sensitive to geometric image-attention structure.
  • Attention analysis: Image attention remains much lower than text-token attention in Qwen2-VL, consistent with the reported sparsity pattern across VLMs.Table 6 is cited as supporting the general validity of this imbalance.
  • Confidence and accuracy: Lower-confidence relations such as “Left” and “Under” tend to have lower accuracy than higher-confidence relations such as “On” and “Right”.After AdaptVis, some relations such as “Left” improve alongside increased confidence.
  • Other metrics: Synthetic datasets have more concentrated and higher-skewness attention distributions than real images, but object distribution confounds both differences as familiarity measures.The passage states that concentration and skewness do not isolate the real factor behind the dataset differences.
  • Label distribution: Controlled Image labels are balanced, whereas real-image datasets contain more “left” and “right” examples and fewer other relations.This imbalance is reported for the real-image datasets, including COCO two and VG two.

E.4. More Attention Analysis

Further attention analyses examine sparsity, confidence calibration, localization overlap, coefficient effects, prompt sensitivity, efficiency, and generalization. The results support attention-based error detection and robust improvements, while also identifying annotation and computational limitations.

  • Attention sparsity: Image attention remains sparse across Cont B and real-image subsets of WhatsUp.
  • Confidence calibration: Model self-confidence distinguishes correct and incorrect generations better than aggregate or maximum image-attention scores in the AUROC calibration experiment.Figure 18 compares attention-based metrics with model self-confidence.
  • YOLO overlap: High AUROC in middle-to-high layers indicates that attention-pattern overlap with YOLO annotations can detect factual errors on Controlled A.
  • YOLO overlap: YOLO annotation errors—missed detection, misclassification, bounding-box error, and ambiguous reference—can make attention-overlap AUROC relatively modest.
  • Coefficient effects: Coefficients above 1 improve familiar “left” and “right” relations, whereas coefficients below 1 improve less familiar “under” and “on” relations.
  • Prompt sensitivity: Reversing entity order causes a significant performance drop, while adaptive attention intervention consistently improves results across prompt orders.This indicates prompt sensitivity in existing attention patterns and generation results.
  • Efficiency: ScalingVis adds negligible computation compared with greedy decoding, whereas AdaptVis incurs higher overhead because it computes a threshold.
  • More benchmarks: The method consistently outperforms baseline across POPE, GQA, and VQAv2, supporting generalization to general question answering.

F. URLs of Code and Data

The supplementary materials provide code and data access information, configuration options for both methods, and additional visual and prompt-variation results.

  • Code and data are provided at the listed AdaptVis research repository URL.
  • The evaluation configuration selects the dataset, model, and method, with separate weight ranges and a threshold parameter for AdaptVis.
  • Additional figures visualize fixed α cases, attention examples, prompt changes on WhatsUp, and performance before and after ScalingVis.
Loading 2503.01773v3…