PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models

Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, Li Yi

arXiv:2506.03135v3cs.CVcs.AIcs.CL

TL;DR

Basic spatial benchmarks increasingly saturate, leaving comprehensive complex spatial reasoning insufficiently evaluated. OmniSpatial addresses this gap with a psychology-grounded benchmark spanning four dimensions and 50 subtasks, alongside PointGraph and SpatialCoT. Experiments report persistent limitations in current VLMs, while both strategies yield consistent gains.

  • Problem

    Existing benchmarks mainly test basic relations and counting, while a unified framework for complex spatial reasoning remains lacking.

  • Method

    OmniSpatial organizes visual–spatial reasoning into four dimensions and 50 subtasks, using 8.4K manually curated QA pairs and exploring PointGraph and SpatialCoT.

  • Results

    Current open- and closed-source VLMs show significant limitations on OmniSpatial, while PointGraph and SpatialCoT yield consistent gains.

  • Takeaways & Limitations

    Comprehensive spatial reasoning evaluation benefits from structured scene-graph cues and viewpoint-aware chain-of-thought reasoning.

  • Takeaways & Limitations

    The visual–spatial reasoning definition excludes non-visual priors, although it remains compatible with multimodal extensions.

Abstract

from arXiv · show

Spatial reasoning is a key aspect of cognitive psychology and remains a bottleneck for current vision-language models (VLMs). While extensive research has aimed to evaluate or improve VLMs' understanding of basic spatial relations, such as distinguishing left from right, near from far, and object counting, these tasks cover only the most elementary layer of spatial reasoning and are largely approaching saturation in the latest reasoning models. In this work, we introduce OmniSpatial, a comprehensive and challenging benchmark for spatial reasoning, grounded in cognitive psychology. OmniSpatial covers four major categories: dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking, with 50 fine-grained subcategories. Through careful manual annotation, we construct over 8.4K question-answer pairs. Extensive experiments show that both open- and closed-source VLMs exhibit significant limitations in comprehensive spatial reasoning. We also explore two strategies-PointGraph (explicit scene graph cues) and SpatialCoT (novel-view chain-of-thought)-to bolster spatial reasoning.

1 INTRODUCTION

Existing benchmarks largely test basic spatial relations and counting, which latest reasoning models handle well. OmniSpatial broadens evaluation to complex spatial cognition through four dimensions, diverse tasks, and auxiliary reasoning strategies.

  • Over 90% accuracy on SpatialBot-Bench and EmbSpatial indicates that basic spatial benchmarks are approaching saturation for recent reasoning models.
  • Complex spatial reasoning requires interpreting ambiguous, dynamic, and context-dependent relationships beyond static object placement.
  • OmniSpatial organizes spatial reasoning into dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking.
  • The benchmark evaluates VLMs on diverse images and video frames spanning scenes, resolutions, lighting, weather, and multiple continents.
  • PointGraph and SpatialCoT are explored as auxiliary chain-of-thought strategies for improving VLM spatial reasoning.

2 PRELIMINARIES: VISUAL–SPATIAL REASONING

The paper defines visual–spatial reasoning as inferring and reasoning about spatial properties from visual observations. Its taxonomy combines cognitive-psychology foundations with four complementary dimensions covering motion, geometry, interaction, and alternative viewpoints.

  • 2 PRELIMINARIES: VISUAL–SPATIAL REASONING: Visual–spatial reasoning maps an RGB observation stream and task-specific query to a verifiable action or answer space.
  • 2 PRELIMINARIES: VISUAL–SPATIAL REASONING: The definition excludes non-visual priors so improvements can be attributed to visual reasoning itself, while remaining compatible with multimodal extensions.
  • 2.1 TAXONOMY OF VISUAL–SPATIAL REASONING: The taxonomy is grounded in cognitive psychology constructs including visualization, mental rotation, perspective taking, and spatial updating.
  • 2.1 TAXONOMY OF VISUAL–SPATIAL REASONING: Together, the four dimensions balance theoretical comprehensiveness with practical applicability across AI, robotics, and human cognition.
  • 2.1 TAXONOMY OF VISUAL–SPATIAL REASONING: Dynamic reasoning targets motion and temporal change, while complex spatial logic targets higher-order relations, transformations, and geometric structures.
  • 2.1 TAXONOMY OF VISUAL–SPATIAL REASONING: Spatial interaction addresses environmental constraints and task goals, whereas perspective taking requires adopting alternative viewpoints.

3 OMNISPATIAL: COMPREHENSIVE SPATIAL REASONING BENCHMARK

OmniSpatial is a comprehensive spatial-reasoning benchmark emphasizing diverse, structured, manually reviewed tasks beyond pattern recognition. It combines heterogeneous visual sources with PointGraph and SpatialCoT strategies to provide explicit relational and multi-view cues.

  • Overview: 8.4K carefully curated question–answer pairs span four categories and 50 fine-grained subtypes requiring reasoning beyond pattern recognition.The benchmark emphasizes diversity, structure, and rigor rather than sheer data volume.
  • Data Collection: The dataset integrates web imagery, cognitive tests, driving questions, and MME and HOI4D images to capture realistic spatial, dynamic, and interaction scenarios.These sources contribute everyday environments, psychology-inspired challenges, safety-critical reasoning, depth information, and human–object interactions.
  • Annotation: Multiple-choice questions use binary or four-option formats and conversational wording to encourage contextual and relational reasoning over template memorization.Questions undergo validation and cross-checking to ensure clarity and answer uniqueness.
  • Enhancement Strategies: PointGraph localizes objects and encodes identities, centers, bounding boxes, and relative positions in a JSON-style scene graph appended to the query.The resulting structured description supplies explicit geometric cues to the VLM.
  • Enhancement Strategies: SpatialCoT synthesizes six additional 3D perspectives with InstantMesh, composes them into a multi-view collage, and feeds the collage with the question into a chain-of-thought pipeline.The strategy is designed to enrich spatial imagination through novel views.

4 EXPERIMENTS

Experiments evaluate proprietary, open-source, reasoning-oriented, and specialized spatial models under a unified multiple-choice protocol. Models perform best on some dynamic and interaction tasks but remain substantially limited on comprehensive, geometric, and perspective-taking reasoning, while PointGraph and SpatialCoT improve performance.

  • Evaluation Setup: The evaluation covers proprietary, open-source, reasoning-oriented, and spatially specialized models using standardized prompts and multiple output-assessment protocols.Reasoning-model outputs are assessed with an automatic judge against ground truth.
  • Main Results: Proprietary reasoning models surpass a 56% overall success rate, yet remain below human-level understanding and require substantial inference time and tokens.ChatGPT o3 and Gemini-2.5-Pro achieve the highest reported overall performance.
  • Category-wise Analysis: Proprietary models perform strongly in Dynamic Reasoning and Spatial Interaction, reflecting proficiency in temporal understanding, spatial relationships, and map-based comprehension.The reported strengths rely on extensive world knowledge and local understanding capabilities.
  • Category-wise Analysis: Pattern Geometric Reasoning reaches only around 30% to 40% accuracy for extended-thinking models, while perspective-taking remains limited by predominantly egocentric analysis.These results identify geometric imagination and non-egocentric viewpoint reasoning as persistent weaknesses.
  • Enhancement Studies: PointGraph produces clear accuracy boosts, particularly in Dynamic Reasoning and Perspective-Taking, whereas traditional textual CoT yields little improvement.The comparison tests PointGraph preprocessing on GPT-4.1, Gemini-2.5-flash, and Qwen-VL2.5-7B.
  • Enhancement Studies: Novel-view synthesis in SpatialCoT significantly improves GPT-4.1 and Qwen-VL2.5-7B on the Perspective-Taking track.The result supports explicit spatial imagination as an effective enhancement.
  • Training Exploration: +7.82 points average gain follows supervised fine-tuning on 6.9K OmniSpatial samples, with consistent improvements across dynamic, interaction, and perspective-taking tracks.This study compares against zero-shot performance.
  • Generalization: Adding OmniSpatial raises SpaceR-7B’s overall VSI-Bench score from 41.68 to 43.68 and improves ordering, counting, and metric or room-size reasoning.The result indicates transfer of complementary supervision to an external spatial benchmark.

5 RELATED WORKS

Prior spatial benchmarks advanced basic spatial relationship reasoning but remain fragmented in coverage and methodology. OmniSpatial addresses the reported lack of a unified framework for complex spatial reasoning with a comprehensive benchmark design.

  • Prior Benchmarks: Earlier benchmarks introduced spatial information and hierarchical task organizations to evaluate fundamental spatial relationships and related capabilities.These efforts established important foundations for spatial reasoning evaluation.
  • Research Gap: Existing benchmarks lack a unified framework covering a wide range of complex spatial reasoning tasks.The stated limitations include reliance on generated images, LLM-generated templates, and domain-specific focus.
  • Research Gap: These design limitations hinder benchmark comprehensiveness and real-world applicability, motivating an integrative spatial reasoning benchmark.The paper positions OmniSpatial as the response to these gaps.

6 CONCLUSION

OmniSpatial benchmarks comprehensive visual–spatial reasoning across diverse tasks and reveals substantial limitations in current VLMs. PointGraph and SpatialCoT provide structured scene-graph and viewpoint-aware reasoning strategies with consistent gains.

  • Conclusion: OmniSpatial contains four categories, 50 subtasks, and 8.4K manually curated question–answer pairs for comprehensive visual–spatial reasoning evaluation.The categories are dynamic reasoning, complex logic, spatial interaction, and perspective-taking.
  • Conclusion: State-of-the-art proprietary and open-source VLMs peak at 57% accuracy, over 30 points below human performance, with particular difficulty in geometric reasoning and non-egocentric perspective-taking.The benchmark exposes limitations not captured by simpler spatial evaluations.
  • Conclusion: PointGraph and SpatialCoT yield consistent gains, underscoring the value of structured and multi-view reasoning.PointGraph provides structured scene-graph reasoning, while SpatialCoT provides viewpoint-aware chain-of-thought.

ETHIC STATEMENT

The authors report compliance with the ICLR Code of Ethics, licensed public datasets, no direct ethical or legal concerns, and a commitment to responsible and fair use.

  • The work complies with the ICLR Code of Ethics and uses publicly available datasets under their respective licenses.
  • The authors report no direct ethical or legal concerns and commit to responsible and fair use of the proposed methods.

REPRODUCIBILITY STATEMENT

The authors support reproducibility through detailed documentation, public datasets, and release of source code in the supplementary materials.

  • The paper and appendix describe the proposed model, implementation details, and evaluation protocols in detail.
  • All datasets used are publicly available and properly referenced.
  • The authors release source code in the supplementary materials to further support reproducibility.

A DETAILED TASK DESIGN

OmniSpatial organizes spatial reasoning into four categories and multiple levels of increasingly detailed tasks, spanning dynamic behavior, complex logic, interaction, and perspective-taking. The task suite connects visual reasoning to practical scenarios including robotics, driving, navigation, and geometric analysis.

  • Category structure: OmniSpatial covers Dynamic Reasoning, Complex Logic, Spatial Interaction, and Perspective Taking, with challenges based on real-world application scenarios.
  • Dynamic Reasoning: Dynamic Reasoning assesses object movement, changing conditions, intent, uniform motion, variable motion, and spatial compatibility.
  • Category structure: The tasks are organized into three levels, with each spatial-ability category containing at least two subtasks and the final level offering finer subdivisions inspired by real-life scenarios.
  • Complex Logic: Complex Logic includes visual pattern style, implicit quantity, geometric attributes, spatial location changes, polyhedron unfolding, projections, mental rotation, assembly, and analytical geometry.
  • Spatial Interaction: Spatial Interaction covers traffic analysis and reasoning about environmental interactions, including anomalies, signs, actions, risks, and terrain.

C.2 EVALUATION ON THE FULL OMNISPATIAL BENCHMARK

Evaluation on the full OmniSpatial benchmark preserves the model ranking seen on smaller splits, while human performance provides an upper-bound reference and combined reasoning components add a modest gain.

  • Full-benchmark evaluation: The full 8.4K test set evaluates model performance across all twelve fine-grained tracks while reducing variance from subset sampling.
  • Model results: Gemini-2.5-Pro ranks first with an average score of 55.05, followed by ChatGPT o3 at 54.52 and Gemini-2.5-Flash at 51.80.
  • Model results: Per-track strengths differ: ChatGPT o3 leads several manipulation, traffic, pattern, egocentric, and hypothetical tasks, while Gemini-2.5-Pro dominates locating, geospatial, geometric, and allocentric tasks.
  • Human baseline: Human annotators achieve approximately 89% average accuracy with substantial agreement of κ/α ≈0.84, while abstract complex reasoning reaches a consistency score of 0.76.
  • Component synergy: Adding PointGraph to SpatialCoT produces an additional gain of roughly 1 percentage point in overall performance for GPT-4.1-mini and Qwen-VL2.5-3B.

F.2 FAILURE CASE ANALYSIS

Failure analyses show that state-of-the-art VLMs struggle with complex 3D scenes, imagined viewpoints, long-horizon dynamics, and difficult spatial logic. SpatialCoT improves viewpoint-transformation and Complex-Logic performance relative to Manual CoT, while model variability differs across model families.

  • Spatial Interaction: State-of-the-art commercial models struggle with complex 3D scenes, orientation, and path analysis under imagined egocentric poses.The task requires imagining a pose and planning a route through a 3D scene graph.
  • Perspective Taking: Top-performing models struggle with frame-of-reference confusion and counterfactual viewpoints in perspective-taking tasks.The example requires imagining facing an object and inferring relative positions under that viewpoint.
  • Dynamic Reasoning: State-of-the-art commercial models struggle to understand long-horizon sequences involving multiple continuous actions.The analysis suggests augmenting temporal and episodic representations of time–space relations as a possible improvement direction.
  • Complex Logic: Reasoning models produce detailed textual analyses for complex paper-folding tasks, but the authors argue these problems are better addressed with vision-grounded reasoning.The example involves tracking how folded corners affect the resulting boundary.
  • Evaluation Variability: Reasoning models exhibit larger variability across five runs, whereas smaller open-source models produce more consistent results.Table 11 reports means and standard deviations for the four coarse-grained categories.
  • Intervention Results: SpatialCoT consistently outperforms Manual CoT on the Complex-Logic track and also improves many viewpoint-transformation problems.The comparison is reported in Table 12 and in perspective-taking analyses.

G LIMITATION & FUTURE WORKS

The paper identifies limits in OmniSpatial’s dynamic-task complexity and in the fundamental nature of its PointGraph and SpatialCoT improvements. Future directions include richer 3D representations, perception, 3D VLMs, reasoning models, and knowledge distillation, alongside potential societal applications.

  • Limitations: OmniSpatial’s operational tasks remain less complex than long videos despite including dynamic image clips from HOI4D.This limits the benchmark’s coverage of long-duration operational dynamics.
  • Limitations: PointGraph and SpatialCoT enhance VLM spatial understanding through point cues, but their improvement is not fundamental in nature.The paper characterizes spatial reasoning as requiring longer and more complex reasoning, similar to mathematics and coding tasks.
  • Future Work: Future work includes introducing 3D representation and perception, 3D VLMs, reasoning models, and knowledge distillation.These directions are presented as responses to the importance of 3D information for spatial reasoning.
  • Potential Impact: OmniSpatial’s motion, geometry, localization, and interaction tasks are described as potentially supporting safer autonomous systems and more efficient design and manufacturing.The stated applications include hazard-aware navigation, packaging, assembly, and reduced material and energy waste.
Loading 2506.03135v3…