PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective

Songsong Yu, Yuxin Chen, Hao Ju, Lianjie Jia, Fuxi Zhang, Shaofei Huang, Yuhan Wu, Rundi Cui, Binghao Ran, Zaibin Zhang, Zhedong Zheng, Zhipeng Zhang, Yifan Wang, Lin Song, Lijun Wang, Yanwei Li, Ying Shan, Huchuan Lu

arXiv:2509.18905v2cs.AI

TL;DR

Visual Spatial Reasoning is essential for embodied intelligence and autonomous systems, yet VLMs struggle to represent and reason about complex 3D space. This paper reviews VSR methods, organizes capabilities into three levels, and introduces SIBench; experiments reveal a persistent gap between basic perception and higher-order spatial reasoning.

  • Problem

    Achieving human-level Visual Spatial Reasoning in machines remains challenging because representing and reasoning about three-dimensional space is inherently complex.

  • Method

    The paper reviews VSR methodologies, defines three cognitive levels, and curates SIBench from nearly 20 open-source datasets across 23 task settings.

  • Results

    Experiments reveal a critical gap between perceptual and cognitive abilities: models perform well on basic perception but show significant deficiencies in numerical estimation, multi-view reasoning, temporal understanding, and spatial imagination.

  • Takeaways & Limitations

    SIBench provides a comprehensive framework for evaluating VLM spatial intelligence and exposes substantial challenges remaining for artificial agents.

  • Takeaways & Limitations

    VLMs relying primarily on 2D images generally struggle to imagine or reconstruct complete 3D worlds from limited perspectives, especially for occluded geometry and spatial connectivity.

Abstract

from arXiv · show

Visual Spatial Reasoning (VSR) is a core human cognitive ability and a critical requirement for advancing embodied intelligence and autonomous systems. Despite recent progress in Vision-Language Models (VLMs), achieving human-level VSR remains highly challenging due to the complexity of representing and reasoning over three-dimensional space. In this paper, we present a systematic investigation of VSR in VLMs, encompassing a review of existing methodologies across input modalities, model architectures, training strategies, and reasoning mechanisms. Furthermore, we categorize spatial intelligence into three levels of capability, ie, basic perception, spatial understanding, spatial planning, and curate SIBench, a spatial intelligence benchmark encompassing nearly 20 open-source datasets across 23 task settings. Experiments with state-of-the-art VLMs reveal a pronounced gap between perception and reasoning, as models show competence in basic perceptual tasks but consistently underperform in understanding and planning tasks, particularly in numerical estimation, multi-view reasoning, temporal dynamics, and spatial imagination. These findings underscore the substantial challenges that remain in achieving spatial intelligence, while providing both a systematic roadmap and a comprehensive benchmark to drive future research in the field. The related resources of this study are accessible at https://sibench.github.io/Awesome-Visual-Spatial-Reasoning/.

1 INTRODUCTION

Visual Spatial Reasoning is essential for navigating and interacting with the world but remains difficult for VLMs because modeling three-dimensional space is complex. This paper reviews the field, organizes tasks and methods, and introduces SIBench to evaluate broad VSR capabilities.

  • VSR supports navigation and interaction with real-world environments, yet human-level machine performance remains challenging because three-dimensional space is difficult to represent and reason about.
  • Unlike general VQA’s semantic recognition, VSR requires reasoning about relative positions, orientations, distances, object motion, and complex spatial dynamics.
  • The paper reviews VSR methods across input modalities, architectures, training strategies, and reasoning mechanisms while surveying more than 150 related papers.
  • VSR tasks are organized by cognitive levels, providing a systematic categorization of their objectives, challenges, and research progress.
  • SIBench consolidates nearly 20 open-source benchmarks into 23 task settings, exposing deficiencies in numerical estimation, multi-view reasoning, temporal processing, and spatial imagination.

2 BACKGROUND

Spatial intelligence involves perceiving the three-dimensional world, understanding spatial relationships and physical laws, and interacting through navigation or manipulation. The paper focuses on image-, multi-view-, and video-based VLM spatial reasoning while excluding several adjacent modalities and applications.

  • Spatial intelligence spans perception of 3D properties, understanding spatial relationships and physical laws, and interaction through navigation or object manipulation.
  • The paper studies VLM spatial reasoning from images, multi-view inputs, and videos, covering basic perception, spatial relationship understanding, and planning.
  • The scope excludes point-cloud and pure-text multimodal spatial reasoning, generative models, and specialized vision-language action or navigation applications.
  • Prior reviews address VLM evaluation and 3D spatial understanding, but some focus on 3D representations rather than image or video input.

3 METHODOLOGIES

VSR methods enhance VLMs through input modalities, architectures, training, and inference mechanisms to support spatial perception and multi-step reasoning. These approaches add 3D information, specialized processing, external tools or knowledge, and structured reasoning, but face modality and reward-design challenges.

  • VSR methods target input modalities, model architecture, training strategy, and inference methods because spatial tasks require grounding, 3D imagination, and dynamic prediction.
  • A naive VSR model encodes an image and question, aligns visual and text features, and autoregressively generates an answer through an LLM.
  • Depth maps provide a 2.5D compromise between RGB images and point clouds, but their distribution gap from natural images makes effective encoding difficult.
  • Models add specialized visual encoders or symbolic APIs to recover fine-grained spatial information and query numerical depth values when needed.
  • Reinforcement learning is promising for multi-step spatial reasoning, but spatial tasks lack the clear, verifiable reward signals available in mathematics or code generation.
  • Enhancement strategies also use intermediate procedures, multi-agent systems, and retrieval-augmented generation to supply structured reasoning or external spatial knowledge.

4 TASK SETTINGS

The paper organizes VSR tasks into basic perception, spatial understanding, and spatial planning, with understanding further split by static versus dynamic conditions. These settings span object attributes, inter-object relations, viewpoint and temporal reasoning, motion estimation, spatial imagination, and planning under spatial constraints.

  • 4 TASK SETTINGS: VSR task settings are organized into basic perception, spatial understanding, and spatial planning according to increasing reasoning requirements.Basic perception concerns individual-object attributes or states, spatial understanding concerns relationships among multiple objects, and planning seeks solutions under spatial constraints.
  • 4.1 Basic Perception: Basic perception distinguishes relatively stable attributes such as shape, color, size, and quantity from changeable states such as posture, orientation, and open or closed status.State tasks may also require understanding how object states evolve over time.
  • 4.2 Spatial Understanding: Spatial understanding covers static relations, localization, distance, and compatibility reasoning, while dynamic understanding adds viewpoint shifts or temporal components.Dynamic tasks include trajectory description, velocity and acceleration estimation, and spatial imagination.
  • 4.2.2 Dynamic Understanding: Dynamic understanding requires models to infer 3D spatial information from multi-view images or videos, where perspective projection and accumulated sequence errors create major challenges.The paper notes that simple image-feature matching cannot resolve the 2D-to-3D inference problem, while longer sequences require dynamic error correction.
  • 4.2.2 Dynamic Understanding: Spatial imagination asks models to reason from hypothetical viewpoints, including mentally modeling space across limited or changing views.Representative tasks ask what would be farthest from an observer after moving between views.

5 EXPERIMENT

The experiment constructs and evaluates SIBench across 23 task settings and three cognitive levels, using standardized model inputs and task-specific metrics. Results show leading proprietary models outperform open-source and lightweight proprietary models, while VLMs remain weak on several spatial reasoning challenges.

  • 5.1 Benchmark Construction: SIBench consolidates nearly 20 open-source benchmarks into 23 task settings spanning three cognitive levels.The benchmark contains 8.8K samples and covers basic perception, spatial understanding, and planning.
  • 5.2.1 Setup.: The evaluation standardizes image resolution, avoids default Chain-of-Thought prompting, and applies task-specific metrics including MRA for numerical answers.True/False and multiple-choice predictions are compared directly with ground truth, while non-formatted outputs are post-processed.
  • 5.2.2 Main Results.: SIBench-mini uses 40 randomly selected samples per task setting to reduce evaluation cost while preserving the full benchmark’s task-setting coverage.It evaluates Gemini-2.5-Pro, Doubao-Seed-1.6-Vision, GLM4.5-V-106B-A12B, and GPT-5.
  • 5.2.2 Main Results.: GPT-5 achieves the best performance across all three visual-spatial reasoning levels on SIBench and SIBench-mini.Leading proprietary models form the highest performance tier in both benchmarks.
  • 5.2.2 Main Results.: Open-source models outperform lightweight proprietary models overall, although they lag behind leading proprietary models.InternVL-3.5-38B scores 0.5252 and Qwen2.5-VL-72B scores 0.5114, exceeding GPT-4o-mini at 0.4278 and Gemini-2.5-Flash at 0.4389.
  • 5.2.2 Main Results.: Models struggle with quantitative estimation, multi-view reasoning, temporal dynamics, and spatial imagination.They often misestimate distance and size, fail to infer camera-pose transformations, process video frames independently, and fail on tasks requiring mental-model manipulation.

6 DISCUSSION

The discussion identifies four major VSR challenges—fragile foundational perception, poor quantitative precision, weak spatial imagination, and insufficient dynamic-temporal and cross-view reasoning—and outlines data, perception, and unified spatiotemporal architecture as responses.

  • 6.1 Challenges: VLMs struggle with foundational perception, quantitative geometry, 3D spatial imagination, and dynamic-temporal or cross-view reasoning in complex VSR tasks.These weaknesses affect counting, localization, shape recognition, numerical estimation, unseen-object geometry, route planning, maze navigation, and multi-view or temporal integration.
  • 6.1 Challenges: Precise numerical regression remains difficult, restricting applications requiring accurate coordinates or metric units, such as robotic grasping and AR content placement.The discussion specifically identifies Coordinate Conversion, Distance Prediction, and Object Size Estimation as affected tasks.
  • 6.1 Challenges: Limited 3D reconstruction prevents coherent global scene representations, contributing to poor performance on situational geometry, route planning, and maze navigation.The discussion attributes this weakness to reliance on 2D images and limited perspectives.
  • 6.2 Potential Solutions: Higher-quality, diverse, structured training data should span sources and task settings, including relational reasoning, navigation, manipulation, and synthetic data with ground truth.The proposed data strategy links different task settings to distinct spatial abilities.
  • 6.2 Potential Solutions: 3D-aware pre-training and unified spatiotemporal architectures are proposed to improve view-invariant representations and integrate information across time and viewpoints.The architecture direction treats space and time as a continuous four-dimensional stream for dynamic, consistent, ego-centric scene understanding.
  • 6.3 Applications: Improved VSR is positioned as a bridge from language instructions to precise spatial coordinates and action sequences in robotics and autonomous driving.The discussion connects VSR to navigation, manipulation, and planning and execution in embodied systems.

7 CONCLUSION

The paper reviews VSR methods, organizes tasks into three cognitive levels, and introduces SIBench for holistic evaluation. Experiments reveal a gap between basic perception and higher-order spatial reasoning, especially in quantitative, multi-view, temporal, and imaginative tasks.

  • 7 CONCLUSION: The paper conducts a systematic VSR investigation, reviews methods, proposes a three-level cognitive taxonomy, and curates SIBench across 23 task settings.The reviewed dimensions include input modalities, model architectures, training strategies, and inference mechanisms.
  • 7 CONCLUSION: Models perform relatively well on basic perception but show significant deficiencies in higher-order reasoning involving numerical estimation, multi-view reasoning, temporal understanding, and spatial imagination.The experiments use SIBench to evaluate state-of-the-art models across the paper’s spatial-intelligence framework.

APPENDIX A APPENDIX OUTLINE

The supplementary material is organized into six aspects covering SIBench data sources, test-data adaptation, comparisons, failures, benchmark timelines, and benchmark comparisons.

  • APPENDIX A APPENDIX OUTLINE: The supplementary material contains six components: data sources, test-data adaptation, setting comparisons, failure analysis, benchmark timelines, and benchmark comparisons.The outline explicitly lists these aspects as the supplement’s structure.

A.1 Data Source

SIBench’s data source is distributed across three input formats and nearly 20 open-source datasets. The benchmark aggregates these sources to support broad evaluation coverage.

  • A.1 Data Source: SIBench combines approximately 20 open-source datasets into three distinct input formats.The data-source figure presents the distribution across formats and datasets.

A.2 Adaptation of test data

The test-data protocol is adapted to reduce target loss and preserve temporal information when videos vary in duration.

  • Five-minute videos sampled at 30 frames can lose targets, causing mismatches with ground truth in tasks such as object counting.The benchmark therefore restricts selected videos to a maximum duration of one minute.
  • Variable sampling intervals can discard temporal information, especially when estimating velocity and acceleration from videos of different durations.

A.3 Detailed Comparison of Various Settings

SIBench evaluates models across 23 task settings, revealing strong performance on basic perception but substantial difficulty with abstract spatial reasoning.

  • Across 23 SIBench task settings, Gemini-2.5-Pro ranks first overall, while GPT-5 leads on SIBench-mini.The detailed model results are reported in Tables 4 and 5.
  • Models generally identify objects successfully but struggle with spatial imagination, temporal ordering, and precise quantitative estimation.

A.3.1 Areas of Model Strength

Current models perform well on object existence and simple spatial relations, indicating strength in basic recognition and relational perception.

  • Qwen2.5-VL-72B achieves 1.0 on object existence, while many other models score above 0.9.
  • InternVL-3-78B leads Spatial Relation with 0.8402, followed by Qwen2.5-VL-72B at 0.8216.

A.3.2 Areas of Model Weakness

Models remain weak in precise perception, quantitative reasoning, spatial imagination, multi-view inference, and temporal reasoning, with failure cases spanning several benchmarked capabilities.

  • Models struggle with quantitative estimation, spatial imagination, multi-view reasoning, camera pose, geometric reasoning, and temporal appearance order.These weaknesses include reliance on coarse visual clues, inability to manipulate unpictured scenes, and poor interpretation of changing viewpoints.
  • On Spatial Imagination, GPT-4o-mini and InternVL-2.5-78B-MPO receive nearly the lowest scores, indicating difficulty constructing and manipulating mental scenes.
  • LLaVA-OneVision-7B scores 0.0000 on temporal appearance order, highlighting unreliable reasoning about non-static scenes.
  • GPT-5 predicts half the ground-truth number of drawers in a shape-analysis case and substantially misestimates object distances and sizes.It also fails to identify rotated 3D shapes, infer viewpoint-dependent positions, and solve a curtain-opening calculation.
  • The benchmark landscape spans more than 60 VSR benchmarks, while SIBench emphasizes broad input and question-type coverage in a large-scale dataset.The comparison categorizes task settings as concentrated when fewer than 10 are covered and diverse otherwise.
Loading 2509.18905v2…