PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

Cheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, Jaesik Park

arXiv:2605.30161v1cs.CV

TL;DR

It is unclear whether VLM spatial reasoning reflects structured 3D representations or shortcuts from photographic correlations. The paper diagnoses spatial representations with contrastive analyses and SpatialTunnel, finding pervasive vertical-distance entanglement and greater robustness from more structured representations.

  • Problem

    It remains unclear whether VLM spatial reasoning reflects structured 3D understanding or shortcuts based on correlations in photographic data.

  • Method

    The paper analyzes spatial relations in VLM embeddings with minimal contrastive pairs and introduces SpatialTunnel to decouple vertical image position from depth.

  • Results

    Across VLM families, vertical and depth relations are frequently entangled, while models with more structured spatial representations perform better and show greater robustness across diverse benchmarks.

  • Takeaways & Limitations

    Representational structure, rather than benchmark accuracy alone, is a reliable indicator of robust spatial reasoning within the evaluated VLMs.

  • Takeaways & Limitations

    The analysis hypothesizes that VLMs exploit correlations between vertical position and depth as a shortcut for depth-related questions.

Abstract

from arXiv · show

Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM embeddings. Our analysis across multiple model families reveals a consistent vertical-distance entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall benchmark accuracy improves. We further show that models with similar benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks. Code and benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.

1 Introduction

The paper examines whether VLM spatial reasoning reflects structured 3D representations or photographic shortcuts, focusing on vertical–depth entanglement. It introduces contrastive representation analysis and SpatialTunnel to expose these biases and relate representation structure to robustness.

  • Introduction: VLMs achieve strong spatial-benchmark performance despite primarily training on 2D image–text pairs, leaving the source of their spatial reasoning unclear.The paper frames this uncertainty as important for systems deployed in physical environments.
  • Introduction: Photographic perspective creates correlations between vertical image position and depth, enabling models to substitute vertical cues for explicit 3D reasoning.Objects appearing higher in an image are often farther from the camera.
  • Introduction: Behavioral accuracy alone cannot reveal whether models use structured, separable spatial representations or correlated cues, motivating representation-level analysis.Models with similar benchmark performance may rely on different internal mechanisms.
  • Introduction: The framework probes horizontal, vertical, and depth relations with controlled contrastive examples while holding object identity and other confounds fixed.The analysis varies only the spatial relation between objects.
  • Introduction: Across VLM families, horizontal relations form stable opposing directions, while vertical and depth relations are frequently entangled; structured representations predict stronger benchmark performance and robustness.SpatialTunnel decouples vertical image position from depth to reveal shortcut biases hidden under standard evaluations.

2 Related Work

Prior work has established persistent weaknesses in VLM spatial reasoning through controlled and expanded benchmarks, while newer studies probe the internal representations underlying these behaviors. However, existing efforts primarily assess performance or isolated internal signals rather than the paper’s representation-level analysis of spatial organization.

  • Spatial Understanding Datasets and Benchmarks: Controlled benchmarks reveal that VLMs often fail basic positional distinctions and frame-of-reference consistency despite strong semantic performance.What’s Up and COM-FORT provide controlled evaluations of these weaknesses.
  • Spatial Understanding Datasets and Benchmarks: Subsequent benchmarks broaden spatial evaluation to egocentric and cross-video reasoning, 6DoF diagnostics, and multi-step spatial referring.Simulation-based datasets additionally provide large-scale supervision for physical dynamics.
  • Spatial Understanding Datasets and Benchmarks: Simulation-based datasets provide large-scale supervision for physical dynamics, but spatial performance often plateaus with data scaling.These datasets complement benchmark expansions focused on diverse spatial reasoning settings.
  • Probing Internal Representations of Vision-Language Models: Representation-focused studies find that vision encoders encode monocular depth cues and bind geometric coordinates to object activations in early layers.Unified extraction frameworks support systematic comparisons across model families.

3 Perspective Projection Bias in Spatial Understanding

This section shows that VLMs systematically entangle image-plane vertical position with depth, exploiting a perspective-based shortcut rather than reliably inferring 3D relationships. The resulting bias is exposed by consistently higher performance on perspective-consistent examples than on counter-heuristic examples, despite existing benchmarks being heavily skewed toward the former.

  • Perspective projection bias: VLMs frequently use an object’s vertical image position as a proxy for camera distance, a bias termed vertical-distance entanglement.The shortcut corresponds to treating objects that appear higher as farther and objects lower as closer when answering depth questions.
  • Perspective projection bias: For ground-plane objects, perspective makes farther objects appear higher, creating the elevation cue that can support this shortcut.Objects nearer the horizon line are perceived as farther from the observer.
  • Evaluation design: The analysis classifies depth questions as consistent when the farther object is higher and counter otherwise, using queried objects’ vertical center coordinates.Comparable accuracy across groups would indicate no entanglement, whereas an accuracy gap supports reliance on the vertical-position shortcut.
  • Benchmark distribution: 80.9% of EmbSpatial-Bench and 60.5% of CV-Bench-3D depth examples are consistent, while counter examples constitute only about 10% in each benchmark.This imbalance mirrors the perspective statistics of natural photographs, where farther objects commonly appear higher.
  • Counter-example performance: 60.9% versus 24%: Qwen2.5-VL fine-tuned on 2M samples performs substantially better on EmbSpatial-Bench’s consistent split than on counter examples.The reported comparison exemplifies the systematic counter-example failures observed across models, architectures, and training scales.

4 Behavioral Analysis with a Synthetic Dataset

SpatialTunnel is a synthetic benchmark that independently controls image-plane position and 3D depth, enabling matched counterfactual tests of vertical-distance shortcuts. Across models, consistent configurations outperform counter configurations, confirming that vertical-distance entanglement persists under controlled evaluation.

  • Benchmark motivation: SpatialTunnel decouples an object’s vertical image-plane position from its 3D depth, allowing both factors to be manipulated independently.This isolates individual depth cues that are conflated in real photographs.
  • Controlled scene construction: The benchmark places objects at fixed depths while sweeping their angular positions, producing image-plane changes without altering depth ordering.Matched counterfactual pairs can therefore flip vertical arrangement while preserving the ground-truth depth relation.
  • Controlled scene construction: SpatialTunnel discretizes each object’s tunnel position into 16 angular locations, forming a 16 × 16 grid for configuration-level diagnostics.The grid supports heatmap analyses across joint angular configurations.
  • Behavioral results: Across all base and fine-tuned models, consistent-subset accuracy exceeds counter-subset accuracy, producing a positive accuracy gap.This shows that vertical-distance entanglement is model-intrinsic rather than solely caused by natural-image evaluation skew.
  • Behavioral results: Base Qwen2.5-VL-3B achieves vcons = 0.776 but only vctr = 0.360, indicating strong reliance on the vertical-position shortcut.Base NVILA-Lite-2B has a narrower gap, but its sub-0.5 overall accuracy suggests near-random performance.

5 Representation Analysis via Contrastive Probing

The paper probes spatial representations with contrastive question pairs, measuring axis coherence and vertical-distance entanglement in hidden states. Distance coherence predicts counter-example accuracy across benchmarks, while well-separated axes and low entanglement accompany robust spatial reasoning.

  • Contrastive probing: Contrastive queries swap object order so the correct answer becomes the spatial inverse, enabling delta-vector probing at a fixed intermediate layer.The method extracts final-token hidden states and compares representations of paired questions differing only in queried-object ordering.
  • Representation metrics: Axis coherence measures mean pairwise cosine similarity after sign-correcting opposing categories, with high coherence indicating a stable representation direction.Deltas from opposing categories, such as far and close, are negated so vectors align toward a canonical axis direction.
  • Representation metrics: VD-EI quantifies directional coupling between vertical and distance categories, where positive values indicate perspective-predicted entanglement and zero indicates independence.The index contrasts perspective-aligned pairs above↔far and below↔close with perspective-opposing pairs.
  • Coherence findings: Distance coherence is weakest across models and scales, although fine-tuning raises vertical coherence from 0.23 to 0.57 for Molmo and from 0.29 to 0.59 for Qwen.CohD increases by a comparatively smaller margin than vertical coherence.
  • Cross-domain validity: CohD computed on SpatialTunnel correlates with counter accuracy on EmbSpatial-Bench and CV-Bench-3D, with ρ = 0.759 and 0.804, respectively; both p < 10−3.This cross-domain relationship supports CohD as a reusable predictive representation signal rather than a benchmark-specific artifact.
  • Representation structure: RoboRefer and Qwen3 exhibit well-separated spatial axes and strong overall performance, whereas fine-tuned Molmo, NVILA, and Qwen often retain poorly distinguished distance representations.The results associate high CohD and low VD-EI with robust spatial reasoning across benchmarks.

6 Conclusion

The paper introduces a representation-level diagnostic showing that vertical-distance entanglement is a pervasive, model-intrinsic bias across VLM families and scales. More structured spatial representations correspond to stronger counter-heuristic robustness and higher accuracy across diverse spatial reasoning benchmarks.

  • Conclusion: The diagnostic framework identifies vertical-distance entanglement as a pervasive, model-intrinsic bias across VLM families and model scales.This bias is characterized as a recurring property of the models’ internal spatial representations.
  • Conclusion: Models with high distance coherence and low VD-Entanglement Index exhibit stronger counter-heuristic robustness.The conclusion links structured spatial representations to improved performance on examples that oppose perspective-based heuristics.
  • Conclusion: Models with more structured spatial representations achieve higher accuracy across diverse spatial reasoning benchmarks.The identified structural properties are high distance coherence and low VD-Entanglement Index.

Appendices … B.2 Training Data Sources

The appendices explain the geometric origin of vertical-distance cues, experimental setup, evaluated models, and training data assembled from existing spatial reasoning datasets. These materials span synthetic, real-world, embodied, and robotics-oriented sources with varied scales, annotations, tasks, and supervision.

  • Appendices: The appendices provide supplementary material covering ground-plane geometry, experiment setup, SpatialTunnel, and contrastive probing.
  • A Ground-Plane Geometry and Vertical Image Position: Under a zero-tilt pinhole camera, greater depth on a shared ground plane produces a higher image position because points project closer to the horizon.The derivation assumes camera height Hc > 0 and standard image coordinates increasing downward.
  • B Additional Details on Experiment Setup: The experiment setup specifies models, training-data sources, data-mix composition, and benchmarks used in the experiments.
  • B.2 Training Data Sources: Rather than generating data from scratch, the experiments combine existing spatial datasets into training mixes at varying scales.
  • B.2 Training Data Sources: SAT contributes 175K automatically generated question–answer pairs from 22K procedural apartment scenes, including static spatial and dynamic tasks.Its questions derive from perfect 3D geometry and simulator metadata without human annotation.
  • B.2 Training Data Sources: RoboSpatial provides around 3M spatial relations over 1M images and 5K scans from real indoor and tabletop environments.
  • B.2 Training Data Sources: SPAR-7M, RefSpatial, and PRISM broaden supervision across 7M QA pairs, 20M QA pairs, and hundreds of thousands of grasping samples, respectively.They cover multi-view and single-view relations, 31 spatial relations with up to 5 reasoning steps, and task-oriented 6-DoF grasping.

B.3 Data Mix Composition

The study constructs four increasingly large training data mixes from five spatial datasets, using equal dataset allocation up to 800k samples and size-adjusted allocation at 2M. Sampling within each dataset is proportional across its constituent sub-files.

  • B.3 Data Mix Composition: Four training mixes scale from 80k to 2M samples across five spatial datasets, with equal per-dataset allocation through 800k.The 2M mix adjusts allocations for dataset-size differences.
  • B.3 Data Mix Composition: Within each dataset, samples are drawn proportionally across all constituent sub-files and task categories.Examples include SAT’s six task categories and RefSpatial’s seven QA types.
  • B.3 Data Mix Composition: At 2M, smaller datasets such as SAT are included in full, while larger datasets such as RefSpatial are subsampled.RefSpatial is sampled at ∼3.3%, while SAT and PRISM are used in full.

B.4 Benchmarks … C.2 VQA Protocol

The paper evaluates spatial understanding using complementary benchmarks and provides detailed SpatialTunnel generation and VQA protocols. SpatialTunnel controls image-plane layout, depth ordering, and question formulation to isolate spatial reasoning behavior.

  • B.4 Benchmarks: The evaluation uses benchmarks designed to test vision-language models’ spatial understanding ability.
  • B.4 Benchmarks: EmbSpatial-Bench targets embodied spatial understanding with 3D indoor environments, addressing limitations of predominantly 2D, dataset-centric, object-centric benchmarks.
  • B.4 Benchmarks: CV-Bench repurposes ADE20K, COCO, and Omni3D into manually inspected VQA examples covering fundamental 2D and 3D understanding.It contains 2,638 examples with natural-language questions and four-way answer choices.
  • B.4 Benchmarks: BLINK comprises 3,807 visually prompted multiple-choice questions spanning relative depth, spatial reasoning, correspondence, forensics, and multi-view understanding.Its images include abstract diagrams, synthetic scenes, and real-world photographs.
  • B.5 BLINK Confidence Intervals: BLINK subset results are contextualized with Wilson 95% confidence intervals for Rel. Depth (n=124) and Spat. Rel. (n=143).
  • C Additional Details on SpatialTunnel: SpatialTunnel’s additional details cover scene setup, VQA, proprietary-model results, and an object-size variant.
  • C.1 Scene Generation Details: SpatialTunnel scenes are rendered in Blender with two objects at different depths, while independently sweeping their angular positions across 16 discrete tunnel locations.obj1 is always farther from the camera than obj2, producing matched image pairs that vary 2D layout while preserving depth ordering.
  • C.2 VQA Protocol: The VQA protocol renders 3,072 images and 12,288 question-image pairs using four binary depth-comparison templates per image.It averages template-level correctness scores for each angular configuration and evaluates responses with a probability-based protocol.

C.3 Proprietary Model Results

On SpatialTunnel, proprietary models show substantial variation in exact-match accuracy and in the perspective-consistent versus counter-cell gap. Reasoning-enabled GPT-5.2 and Gemini-2.5-Pro achieve high accuracy with reduced or near-zero gaps.

  • Default proprietary models: GPT-5.2 defaults to 0.613 mean exact-match accuracy, with Acc_con = 0.673, Acc_ctr = 0.552, and a Delta = 0.120 gap.Its positive gap indicates better performance on perspective-consistent than counter cells.
  • Reasoning-enabled GPT-5.2: Reasoning improves GPT-5.2 from 0.613 to 0.953 mean accuracy while reducing its gap from Delta = 0.120 to Delta = 0.058.The improvement is measured using final Yes/No outputs and exact-match accuracy.
  • Gemini-2.5-Pro: Gemini-2.5-Pro reaches 0.919 mean accuracy with a slightly negative Delta = -0.028 gap, indicating near-zero perspective bias.The proprietary configurations are evaluated on SpatialTunnel using four question templates per image, with accuracies averaged.

C.4 Extending the Analysis to Object Size … D.5 Cross-Domain Consistency of Distance Coherence

The size-controlled SpatialTunnel intervention shows that apparent size, like vertical position, becomes a depth shortcut, while supplementary probing details establish layer-selection procedures and cross-domain consistency of distance coherence. Distance-coherence rankings are largely preserved between synthetic and real datasets, supporting CohD as a reproducible spatial-representation measure.

  • C.4 Extending the Analysis to Object Size: Object-size manipulations anti-correlate s_1 and s_2 under s_1 + s_2 = 0.4 while holding obj1 farther than obj2, sweeping 11 configurations.The sweep moves from size-consistent to size-conflicting depth cues as the farther object grows and the nearer object shrinks.
  • C.4 Extending the Analysis to Object Size: Models relying on apparent size show lower correctness as the farther object becomes larger, confirming size as a confounding depth cue alongside vertical position.Performance is higher when the cue agrees with true depth and lower when it conflicts; Qwen remains near chance throughout.
  • C.4 Extending the Analysis to Object Size: Fine-tuning raises aggregate accuracy but can amplify size-bias reliance: NVILA-Lite-2B (2M) reaches v = 0.828 with ∆s = +0.207, whereas RoboRefer reaches v = 0.804 with ∆s = +0.061.Qwen variants remain near chance at v ≈0.50 with |∆s| < 0.02, which reflects limited depth discrimination rather than robustness.
  • D.1 Swap Pair Construction: Contrastive probing swaps queried objects for horizontal and vertical relations, while distance pairs reverse target and reference roles within four-choice depth questions.These constructions flip the relational answer while preserving the surrounding visual context.
  • D.2 A Brief Illustration of VD-EI: VD-EI is positive when perspective-aligned pairs are more similar than perspective-opposing pairs and approaches zero when the two effects cancel.It is largest when aligned cosine similarities are high and opposing similarities are negative.
  • D.3 Layer Selection Methodology: Representative layers are selected per model using axis-coherence plateaus first, VD-EI stability second, and avoidance of final output-specialized layers third.Intermediate layers are favored because spatial representations can form and plateau before final layers specialize for output generation.
  • Per-model selection: The selected depths vary by model family, including L∗= 23 (72%) for Molmo-7B-O-0924, L∗= 20 (71%) for NVILA-Lite-2B, L∗= 28 (78%) for Qwen2.5-VL-3B, and L∗= 87 (93%) for Qwen3-VL-235B-A22B-Instruct.Qwen3-VL forms coherent spatial axes unusually late, while smaller models generally use intermediate depths.
  • D.4 Robustness to Alternative Layer Choices.: Distance-coherence rankings remain largely consistent between SpatialTunnel and EmbSpatial-Bench despite different absolute values, including an identical NVILA ordering and increasing Molmo coherence with training scale.Qwen2.5-VL scale variants cluster tightly across domains, whereas Qwen3-235B has a markedly different profile; sampling 1K candidate layers also yields high agreement with the reported CohD ranking.

D.6 Heatmap and PCA Results

Cross-category heatmaps show that models encode opposing directions as antiparallel vectors while separating the horizontal axis from vertical and distance. PCA visualizations further reveal recurring vertical-distance entanglement, with clearer axis separation in RoboRefer and Qwen3-VL-235B.

  • Heatmap results: Opposing categories on the same spatial axis have cosine similarity consistently near −1, indicating antiparallel representations.Similarity is computed between category mean delta vectors.
  • Heatmap results: Horizontal-category similarity is close to zero with vertical and distance categories, suggesting independent horizontal-axis encoding.
  • PCA results: 2D PCA shows opposing categories separating along shared principal components, while far/close overlap with above/below.This overlap reflects vertical-distance entanglement in the Molmo family.
  • PCA results: RoboRefer forms tighter far/close clusters separated from vertical categories, consistent with higher CohD and lower VD-EI.
  • PCA results: Qwen3-VL-235B exhibits cleaner cluster separation across all three axes, including clear three-way separation in 3D space.
  • PCA results: For Molmo, a distinct distance axis does not clearly emerge in 3D PCA, although horizontal and vertical vectors cluster better with data scaling.
Loading 2605.30161v1…