Source-linked AI summary
Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li, Zhixiang Lu, Haolin Yang, Imran Razzak, Yutong Xie
TL;DR
The paper examines whether VLM spatial reasoning requires precise object localization, a question obscured by benchmarks that conflate localization, relational grounding, and reasoning. Using mechanistic interventions and analyses across LLaVA-1.5 and Qwen2.5-VL, it finds that coarse object-centered grounding supports spatial relations, while exact boundaries are unnecessary and localization and reasoning ultimately use partly distinct pathways.
Problem
It is unclear whether VLMs must precisely localize queried objects to answer spatial relations, because correct predictions may instead use coarse layout, contextual regularities, or language priors.
Method
The study analyzes LLaVA-1.5 and Qwen2.5-VL with object-region interventions, layer-wise probing, attention knockout, causal mediation, and targeted head ablation.
Results
VLMs require object grounding but not precise localization: directional judgments remain robust after major localization losses, while position information emerges before relation answers and the tasks rely on sparsely overlapping circuits.
Takeaways & Limitations
Knowing where objects are is not equivalent to knowing how they relate, because coarse object-centered layout can support spatial relations without accurate bounding-box boundaries.
Takeaways & Limitations
The causal-mediation analysis retains only correctly answered relation pairs for which object removal degrades the answer, limiting its scope to object-dependent relational evidence under this paired design.
Abstract
from arXiv · showhide
Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
Introduction
The paper asks whether VLM spatial reasoning requires precise object localization and investigates this question across model families using token-, layer-, and head-level analyses. It finds that coarse object grounding supports spatial relations, while precise localization is not required.
- Introduction: The study addresses whether precise localization, parallel grounding, or correlated capabilities explain spatial-relation predictions.The motivation distinguishes precise localization, relational grounding, and spatial reasoning as separate processes.
- Introduction: The authors investigate LLaVA-1.5 and Qwen2.5-VL using object-region interventions, layer-wise probing, attention knockout, causal mediation, and targeted head ablation.These analyses examine representations and routing across tokens, layers, and attention heads.
- Introduction: Spatial reasoning relies more on coarse object layout than exact boundary geometry.Relation accuracy remains stable after fine boundary removal but declines when interventions reach surrounding context.
- Introduction: Position becomes decodable before relation answers emerge, without forming a strict serial bottleneck.Relation answers may remain correct despite localization failures.
- Introduction: Localization and spatial reasoning use sparsely overlapping attention circuits, with shared heads and largely task-specific pathways.The tasks share early grounding-related processing but are ultimately mediated through partially distinct pathways.
Background
The background explains how visual patch sequences preserve two-dimensional structure through positional encoding and token interactions. It then motivates testing whether recoverable spatial geometry supports relation reasoning through precise boundaries or coarse object-centered cues.
- Background: Visual encoders partition images into H×W patches and flatten them in raster order, assigning index i = rW + c.The resulting patch sequence is transformed into language-compatible visual embeddings.
- Background: Because rasterization does not preserve two-dimensional organization explicitly, positional encoding makes attention sensitive to spatial structure.Absolute and relative mechanisms restore spatial information through position vectors or token displacements.
- Background: Absolute position embeddings can preserve row and column structure even when image patches are represented in a one-dimensional sequence.Fixed-resolution sequence indices correspond to persistent patch coordinates.
- Background: 2D RoPE represents visual interactions through relative displacement Δs_ij = (x_i − x_j, y_i − y_j).The query and key representations are divided into horizontal and vertical subspaces, supporting varying image resolutions and sequence lengths.
- Background: LLaVA-1.5 uses learnable one-dimensional absolute position embeddings, whereas Qwen2.5-VL uses 2D RoPE for dynamic-resolution inputs.The paper studies these models as representative implementations of the two mechanisms.
- Background: Mean-pooled object-aligned tokens form target-reference direction vectors whose geometry is structured with 2D RoPE but breaks down without it.The direction vector is d_t,r = h̄_t − h̄_r, and the reported geometry includes collinearity and orthogonality.
- Background: Recoverable spatial geometry does not by itself explain whether relation reasoning compares object boundaries or uses coarse object-centered cues.This distinction motivates interventions testing the necessity of precise localization.
Preliminaries
The study builds a curated dataset from What’s Up to isolate object-level evidence and evaluates localization and spatial-relation prediction as complementary tasks.
- Dataset construction: The dataset uses controlled real-world images from What’s Up Subsets A and B, retaining left-right relations in Subset A and all listed relations in Subset B.Target and reference objects are annotated with bounding boxes and segmentation masks.
- Dataset construction: An object-removed control set removes the target object and reconstructs its region through LaMa inpainting to test reliance on contextual cues.The control addresses hallucinated detections driven by background context.
- Dataset construction: The filtering retains image pairs where the model identifies the object in the original but fails to identify it after inpainting, isolating real object evidence.Examples of the inpainted dataset are provided in Figure 2.
- Tasks and evaluation: Models perform two complementary tasks on the same image: localizing target and reference objects and predicting their spatial relation.Different prompts are used for the two tasks.
- Tasks and evaluation: Localization is scored by mean success rate at IoU thresholds of 0.5, 0.7, and 0.9, while spatial reasoning uses four-way relation accuracy.Spatial-relation accuracy is the proportion of predictions matching the ground-truth target-to-reference relation.
Experiments
Across token, layer, and head interventions, the experiments show that localization and spatial reasoning share object-grounded processing but are not identical. Precise localization depends on object-internal evidence, while spatial relations can remain correct from coarser grounding and are mediated by partly specialized pathways.
- Token Interventions: Object-aligned token ablation sharply reduces localization while leaving relation accuracy nearly intact under original masks.Target-token ablation reduces Loc-T by 5.37, 27.90, and 64.93 points across the three models, while relation accuracy drops by at most 1.32 points.
- Layer-wise Analysis: Layer-wise probes decode positional information before spatial-relation answers emerge, supporting a staged grounding-to-reasoning process.Position probes peak around layers 14, 12, and 10 for LLaVA-7B, LLaVA-13B, and Qwen-7B, respectively; Qwen relation accuracy rises sharply after layer 19.
- Attention Knockout: Attention knockout reveals partial layer overlap: localization is sensitive to specific intermediate layers, whereas relation prediction uses a broader processing range.Global object-token knockout drives localization nearly to zero and relation accuracy toward chance, showing that spatial reasoning still uses object-aligned information.
- Causal Mediation: Head-level mediation identifies concentrated causal effects in different heads, despite shared layer ranges in some models.In LLaVA-1.5-13B, both tasks are strongest around layers 11–16 but rely on substantially different heads; most other heads have MF scores near zero.
- Head Ablation: Cumulative ablation confirms that localization-important and relation-important heads are functionally distinct, while both tasks reuse some object-grounded intermediate representations.Removing task-important heads is more damaging than removing low-importance heads, and relation accuracy also declines when localization-important heads are removed.
Related Work
Prior work evaluates spatial reasoning through complementary benchmarks, develops explicit grounding systems, and begins tracing multimodal computation mechanistically. However, jointly introduced grounding supervision and output interfaces leave the causal role of precise localization unresolved.
- Spatial reasoning benchmarks cover broad relations, perception-intensive abilities, and controlled separation of recognition, localization, and relational inference.
- Explicit grounding systems generate coordinates, combine coordinates with region features, and extend grounding toward pixel-level masks and segmentation.
- These methods show that precise localization can be learned, elicited, or decoded, but not that it is causally required for spatial reasoning.
- Mechanistic studies trace visual computation through stages of visual enrichment, semantic refinement, and transitions toward language-dominated predictions.
Conclusion
The results indicate that VLMs require object grounding for spatial relations but not precise localization. Coarse object-centered layout can support relations despite severe localization failures, while generalization beyond the studied settings remains open.
- VLMs require object grounding for spatial reasoning, but precise object localization is not necessary.
- Spatial relations can survive severe localization failures when coarse object-centered layout remains available.
- Whether this mechanism generalizes to crowded scenes, 3D relations, and temporal reasoning remains open.