PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models

Xinmiao Huang, Qisong He, Zhenglin Huang, Boxuan Wang, Zhuoyun Li, Guangliang Cheng, Yi Dong, Xiaowei Huang

arXiv:2510.13394v4cs.CV

TL;DR

Existing benchmarks inadequately assess spatial reasoning, especially intrinsic-dynamic reasoning. Spatial-DISE combines a cognitive taxonomy with scalable data generation and evaluates current VLMs. The evaluation finds broad cognitive deficits, while the authors identify a synthetic-data boundary that may leave a sim-to-real gap.

  • Problem

    Existing benchmarks inadequately assess spatial reasoning, particularly intrinsic-dynamic reasoning, despite its importance for applications such as robotics, augmented reality, and autonomous navigation.

    Source: §1 · p. 2

  • Method

    Spatial-DISE uses a four-quadrant cognitive taxonomy and an automated pipeline to create a benchmark and a large set of spatial reasoning VQA pairs.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across 32 evaluated VLMs, performance is broadly weak, with most models only slightly above chance and far below the human baseline, especially on cognitive spatial reasoning and mental simulation.

    Source: §1 · pp. 2–3, §4.2 · p. 6

  • Takeaways & Limitations

    The authors present Spatial-DISE as a framework to guide future VLM development toward spatial reasoning.

    Source: §6 · p. 11

    Spatial-DISE-12K consists entirely of synthetic 3D scenes, so a sim-to-real gap may persist in naturalistic settings.

    Source: §6 · p. 11

Abstract

from arXiv · show

Spatial reasoning ability is crucial for Vision Language Models (VLMs) to support real-world applications in diverse domains including robotics, augmented reality, and autonomous navigation. Unfortunately, existing benchmarks are inadequate in assessing spatial reasoning ability, especially the \emph{intrinsic-dynamic} spatial reasoning which is a fundamental aspect of human spatial cognition. In this paper, we propose a unified benchmark, \textbf{Spatial-DISE}, based on a cognitively grounded taxonomy that categorizes tasks into four fundamental quadrants: \textbf{I}ntrinsic-\textbf{S}tatic, Intrinsic-\textbf{D}ynamic, \textbf{E}xtrinsic-Static, and Extrinsic-Dynamic spatial reasoning. Moreover, to address the issue of data scarcity, we develop a scalable and automated pipeline to generate diverse and verifiable spatial reasoning questions, resulting in a new \textbf{Spatial-DISE} dataset that includes Spatial-DISE Bench (559 evaluation VQA pairs) and Spatial-DISE-12K (12K+ training VQA pairs). Our comprehensive evaluation across 28 state-of-the-art VLMs reveals that, current VLMs have a large and consistent gap to human competence, especially on multi-step multi-view spatial reasoning. Spatial-DISE offers a robust framework, valuable dataset, and clear direction for future research toward human-like spatial intelligence. Benchmark, dataset, and code will be publicly released.

1 INTRODUCTION

Spatial-DISE addresses fragmented, static-heavy, and small-scale evaluation of VLM spatial reasoning with a cognitively grounded framework and a scalable dataset. Evaluations reveal broad weaknesses, particularly in cognitive spatial reasoning and mental simulation.

1 INTRODUCTION

  • Spatial-DISE combines a cognitively grounded 2x2 taxonomy with emphasis on dynamic reasoning, covering both 2D and 3D spatial tasks.The dimensions distinguish intrinsic from extrinsic information and static from dynamic tasks requiring mental transformation.Source: §1 · p. 2
  • The dataset includes 559 real-world and synthetic VQA pairs and more than 12,000 verified 3D spatial reasoning pairs generated through an automated pipeline.The larger set spans five 3D tasks; the benchmark set covers 10 tasks across the four DISE quadrants.Source: §1 · p. 2
  • Across 32 VLMs, performance remains low: most models are only slightly above chance and far below the human baseline.The error analysis attributes failures to deficits in rule-based reasoning and mental simulation, rather than simple visual perception.Source: §1 · p. 2
  • The benchmark’s contributions include a reusable, automated Blender pipeline for generating complex 3D spatial reasoning tasks.Seeded randomization and reproducible distractor generation support verifiability.Source: §1 · pp. 2–3
  • The authors frame Spatial-DISE as a unified, verifiable benchmark and dataset for evaluating cognitive spatial reasoning at scale.They use evaluation of 32 state-of-the-art models to characterize current capability boundaries, especially in multi-step mental simulation.Source: §1 · p. 3

2 RELATED WORK

Prior spatial-reasoning benchmarks are described as fragmented, static-focused, and limited in dynamic-task scale. Spatial-DISE combines a unified cognitive taxonomy with a verifiable generation process.

2 RELATED WORK

  • Existing benchmarks provide fragmented coverage, often focusing on narrow tasks rather than a unified cognitive framework for diagnosing spatial-reasoning weaknesses.Table 1 compares benchmark coverage, instance counts, and data sources under the DISE taxonomy.Source: §2 · p. 3, Table 1 · p. 3
  • Many prior benchmarks emphasize fixed scenes and relations, while Spatial-DISE targets dynamic reasoning such as 3D rotation and folding.The passage lists SpatialRGPT, SPARE3D, VSR, CV-Bench, BLINK, What’sUp, and SpatialEval among static-focused benchmarks.Source: §2 · p. 3
  • Although several benchmarks explore dynamic tasks, Spatial-DISE distinguishes itself through a unified cognitive framework and verifiable generation process.The authors argue that simply aggregating existing benchmarks would leave DISE coverage imbalanced and formats heterogeneous, without verifiable training data for underserved dynamic quadrants.Source: §2 · pp. 3–4

3 METHODOLOGY

Spatial-DISE organizes spatial reasoning along intrinsic/extrinsic and static/dynamic dimensions, then designs tasks and a curation pipeline around that taxonomy. Its outputs include a 559-pair benchmark and a 12K-plus dataset.

3 METHODOLOGY

  • The taxonomy defines intrinsic tasks by an object’s internal properties and extrinsic tasks by relations among objects, while dynamic tasks involve movement or transformation.Its four quadrants distinguish unchanged single objects, fixed multi-object scenes, transformations of one object, and changing relations among multiple objects.Source: §3 · p. 4

3.1 TASKS DESIGN

  • Ten cognitively grounded tasks map to all four DISE quadrants and draw on classical psychometric tests to assess abilities such as mental rotation and spatial visualization.Tasks include static shape finding, rotation and folding, 3D projection, and assembling parts into a whole.Source: §3.1 · pp. 4–5, Figure 2 · p. 4

3.2 BENCHMARK CURATION

  • Curation combines real-world problem collection, scalable synthetic generation, and human quality control.The initial collection yielded 1,180 VQA pairs; the Blender pipeline uses reproducible seeding, renders questions and answers, generates distractors, and standardizes outputs before manual checks for unique solutions, clarity, and redundancy.Source: §3.2 · p. 5
  • Spatial-DISE Bench contains 559 evaluation VQA pairs covering all 10 task types and four DISE dimensions.It is designed for model benchmarking.Source: §3.2 · p. 6
  • Spatial-DISE 12K contains over 12,000 verifiable VQA pairs across five 3D tasks, intended for future training and fine-tuning of VLM spatial reasoning.Source: §3.2 · p. 6

4 EVALUATION ON SPATIAL-DISE BENCH

The evaluation compares 32 VLMs with human performance and tests whether fine-tuning on Spatial-DISE-12K improves benchmark and transfer results. Models remain substantially below human performance overall, while fine-tuning yields gains that vary across spatial quadrants and external tasks.

4.2 MAIN RESULTS

  • 28.4% average accuracy across 32 models was only marginally above random chance (25%) and far below the human baseline (76.8%); Doubao1.5-VL-thinking led at 42.0%.The evaluation reports accuracy on Spatial-DISE; Table 2 covers the 32 SOTA models and two fine-tuned models.Source: §4.2 · p. 6, Table 2 · p. 7
  • 25.4% average accuracy on Fold and Punch was near chance, and even Doubao-1.5-thinking reached only 30.8% on the multi-step task.The task requires simulating a fold, punch, and unfold, challenging models to track an object through sequential changes.Source: §4.2 · p. 6
  • Static skill did not consistently precede dynamic skill: Gemini2.0-Flash scored 38.3% on dynamic versus 23.6% on static tasks, while Doubao-1.5-thinking surpassed humans on Extrinsic-Dynamic questions.The authors interpret these results as evidence of fragmented strategies rather than human-like, scaffolded spatial reasoning.Source: §4.2 · p. 8

4.3 FINE-TUNING ON SPATIAL-DISE-12K

  • Fine-tuning raised Qwen2.5-VL-7B accuracy from 26.1% to 47.0% and SpaceOm accuracy from 25.9% to 41.3%, with pronounced gains on dynamic tasks.Qwen2.5-VL-7B Intrinsic-Static accuracy rose from 16.1% to 51.7%.Source: §4.3 · p. 8, Table 3 · p. 8
  • Quadrant-specific fine-tuning mainly improved its matching domain, with minimal or sometimes detrimental off-diagonal transfer and asymmetric effects across quadrants.Transfer was not cleanly factorized by Intrinsic/Extrinsic or Static/Dynamic axes; one example was +3.9 pp transfer from 3D E-D to I-S, contrasted with strongly negative transfer from I-S to E-D.Source: §4.3 · pp. 8–9, Figure 3 · p. 9
  • 3D Extrinsic-Dynamic training transferred positively across varied 2D tasks, whereas Intrinsic-Static and Extrinsic-Static training had constrained or negative transfer.Intrinsic-Dynamic training improved 2D rotation while degrading performance on simpler 2D problems.Source: §4.3 · p. 9
  • Fine-tuning produced selective out-of-domain gains, strongest on SPACE and OmniSpatial, without catastrophic forgetting of general visual capabilities.Benefits were more modest on benchmarks combining spatial reasoning with generalized language or diagrammatic comprehension.Source: §4.3 · p. 9
  • Even after fine-tuning, Qwen2.5-VL-7B-sft remained markedly below the human baseline, while structured 3D training cultivated distinct spatial skill cohorts and selective transfer.The evaluation includes 32 models across 10 model families and uses human performance as a baseline.Source: §4.1 · p. 6, §4.3 · p. 9

5 ERROR ANALYSIS

The error analysis finds that reasoning errors dominate the sampled failures, especially failures to apply rules and mentally simulate transformations. A layout ablation also found near-identical accuracy under merged-image and separate-image presentation.

5.1 ERROR TAXONOMY AND MAIN DEFICITS

  • 72.5% of the 200 analyzed incorrect responses were classified as reasoning errors, compared with 17.5% perceptual and 10% comprehension errors.The analysis used an automated judge with human analysis, examining 50 errors each from four representative models.Source: §5.1 · p. 10
  • Within reasoning errors, failures in rule application accounted for 44.8%, mental simulation 40.0%, and holistic-local processing 15.2%.Examples include disregarding cube-face geometry, miscalculating Fold and Punch state changes, and overlooking structural flaws in favor of superficial similarities.Source: §5.1 · p. 10

5.2 DISENTANGLING PERCEPTUAL CONFOUNDERS

  • Qwen2.5-VL-7B accuracy was 26.1% with merged images and 24.9% with separate images, offering no clear gain from separating the inputs.The authors interpret this layout ablation as support for spatial-logical inference, rather than format parsing, as the dominant difficulty.Source: §5.2 · pp. 10–11, Table 5 · p. 10

6 CONCLUSION AND LIMITATIONS

Spatial-DISE and Spatial-DISE-12K reveal VLM deficits in applying geometric rules and mental simulation. The authors identify limits in error analysis and synthetic-data coverage, and outline directions for future evaluation and development.

6 CONCLUSION AND LIMITATIONS

  • The authors identify universal VLM cognitive deficits, particularly in applying geometric rules and performing mental simulations.They present Spatial-DISE and Spatial-DISE-12K as a framework for evaluating spatial reasoning.Source: §6 · p. 11
  • Future directions include transferring synthetic cognitive concepts to real settings, evaluating interactive tasks, and requiring textual justifications or action plans.The proposed assessments aim to distinguish mental simulation from fragile heuristic matching.Source: §6 · p. 11
  • The error analysis covers 200 errors from four models, and its hybrid LLM-human labels may be biased; Spatial-DISE-12K is entirely synthetic 3D data, so a sim-to-real gap may persist in naturalistic settings.Human quality control samples 1,000 items, while programmatic single-solution checks cover the remainder because full manual verification is infeasible.Source: §6 · p. 11

A.1 TASKS DESIGN DETAILS

The appendix describes a high-level DISE taxonomy that classifies tasks by information type and whether the goal is static or transformative. It also summarizes dataset subsets and real-world data sources.

A.1 TASKS DESIGN DETAILS

  • DISE classifies tasks by intrinsic versus extrinsic information and static analysis versus dynamic transformation, rather than by the lower-level strategies solvers may use.For example, 3D Shape Finding is structurally Intrinsic-Static even if solving it involves mental rotation; classic spatial abilities can span quadrants.Source: §A.1 · p. 15

A.2 DATASET SPLIT DETAILS

  • The task inventory aligns psychometric tests with spatial abilities defined by Linn and Petersen; Table 6 lists tasks and canonical source tests, including Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization.Spatial-DISE Bench includes 2D questions absent from training splits, enabling zero-shot 2D evaluation.Source: Table 6 · p. 15, §A.1 · p. 15, Table 7 · p. 15

A.3 REAL-WORLD DATA COLLECTION DETAILS

  • Real-world items are collected mainly from open-source spatial tests, CEM 11+ non-verbal reasoning tests, and online employment aptitude tests.The listed sources include academic psychometric materials and validated aptitude-test items.Source: §A.3 · p. 15

A.4 SYNTHETIC DATA GENERATION DETAILS

The appendix details automated, Blender-based generation procedures for five synthetic 3D spatial-reasoning tasks. The procedures use task-specific answers and distractors, with reproducible metadata.

A.4 SYNTHETIC DATA GENERATION DETAILS

  • The appendix describes pseudocode and examples for five automated 3D task generators, designed for verifiability and task-specific plausible distractors.The figures illustrate synthetic generation and examples for rotation, projection, combination, folding, and shape folding; code-listing utilities are project-specific.Source: §A.4 · p. 16, Figure 4 · p. 16, Figure 5 · p. 17
  • 3D Rotation asks participants to recognize a generated 3D shape from a different angle; distractors alter the shape and may use varying viewpoints.The answer is the original shape rendered from a new viewpoint, while metadata records the question and answer views and seed.Source: §A.4 · pp. 16–17
  • 3D Projection asks participants to identify the 2D orthographic projection of an isometric 3D shape from the direction marked by an arrow.Distractors include valid projections from the wrong viewpoint and projections of subtly altered shapes.Source: §A.4 · pp. 17–18
  • 3D Combination presents a different target shape and asks which segmented component from a source shape could build it.The source is shown from opposite isometric viewpoints; its components are laid out individually as options.Source: §A.4 · pp. 18–19
  • 3D Folding tests how a textured 2D cube net folds into a 3D cube, with answer options showing folded cubes.Distractors manipulate face orientations, positions, or textures; the net uses common cross or T layouts.Source: §A.4 · pp. 19–20
  • 3D Shape Finding tracks which cube-face texture was replaced in a third view after participants see the cube from other viewpoints.Options show the original face textures, and the correct option is the replaced face.Source: §A.4 · p. 20

B.1 HUMAN PERFORMANCE ASSESSMENT DETAILS

Human performance varied across task categories, with the lowest accuracies and longest response times in 3D Combination and 2D Shape Finding. The appendix also describes the assessment, evaluation implementation, further results, fine-tuning, and error taxonomy.

B.1 HUMAN PERFORMANCE ASSESSMENT DETAILS

  • 54 participants supplied 1,684 valid responses, with each of 559 benchmark items answered by an average of three participants.A matrix-sampling design gave each participant one booklet, with adjacent booklets sharing anchor items; the main paper reports CTT results and uses IRT for cross-validation.Source: §B.1 · p. 22
  • 56.4% for 3D Combination and 61.5% for 2D Shape Finding were the lowest human accuracies; their mean response times were 59.2s and 58.5s, respectively.The authors interpret the inverse relationship between response time and accuracy as evidence that these tasks impose the highest cognitive load.Source: Table 9 · p. 22, §B.1 · p. 22
  • The appendix compares observed CTT accuracy with IRT model-based predictions and item parameters for each DISE category.The authors describe CTT as a direct empirical performance measure and IRT as explanatory; IRT fits a 2PL model to cross-validate CTT findings and place items on a common latent-ability scale.Source: §B.1 · p. 23

B.2 EVALUATION IMPLEMENTATION DETAILS

  • Evaluations used VLMEvalKit v0.2 on three NVIDIA A100-40G GPUs, with low or zero temperatures and do_sample=False for reproducibility.The appendix includes prompt-template listings for evaluated model families.Source: §B.2 · pp. 25–26

B.3 MORE EVALUATION RESULTS

  • Table 13 reports task-specific accuracies on Spatial-DISE Bench and identifies the highest and second-highest values.Its caption names ten tasks, including 2D and 3D combination, folding, rotation, shape finding, projection, and Fold and Punch.Source: Table 13 · p. 27

B.4 SUPERVISED FINE-TUNING HYPERPARAMETERS

  • Supervised fine-tuning used LoRA with Qwen2.5-VL-7B and SpaceOm on the Spatial-DISE-12K training set.All linear layers were targeted for adaptation; Table 14 lists the hyperparameters.Source: §B.4 · p. 27, Table 14 · p. 27

C.1 DEFINITION OF HIGH-LEVEL ERROR

  • The error analysis divides model mistakes into perceptual, comprehension, and reasoning errors.Reasoning errors occur when the model perceives the scene and understands the prompt but fails at logical deduction, including mental rotation, folding, or spatial manipulation.Source: §C.1 · p. 28

C.2 VLM-AS-JUDGE IN ERROR ANALYSIS

This appendix describes a judge-assisted error-analysis workflow and finds that reasoning errors dominate the sampled failures.

C.2 VLM-AS-JUDGE IN ERROR ANALYSIS

  • The authors used Doubao-1.6-thinking as a judge alongside human inspection to analyze sampled wrong responses.Figure 11 depicts the pipeline, and Table 15 lists the distribution of sampled wrong responses.Source: §C.2 · p. 28, Figure 11 · p. 28, Table 15 · p. 28
  • The prompt templates examine individual errors, category-level patterns, and model comparisons, including error types, causes, severity, and improvement suggestions.Source: §C.2 · pp. 28–29
  • 72.5% (145 out of 200) of analyzed mistakes were reasoning errors, compared with 17.5% perceptual errors and 10% comprehension errors.Source: §C.2 · p. 29

C.3 A DEEP DIVE INTO REASONING FAILURES

The deeper analysis groups 145 reasoning errors into three recurring cognitive failures: applying geometric rules, tracking transformations, and balancing holistic and local visual information.

C.3 A DEEP DIVE INTO REASONING FAILURES

  • 44.8% of reasoning failures involved rule application, including failures to apply geometric constraints such as distinguishing adjacent from opposite cube faces.The authors describe a failure to connect visual percepts with abstract geometric principles.Source: §C.3 · pp. 29–30
  • 40.0% of reasoning failures involved mental simulation, with models struggling to track object states through transformations such as folding and unfolding paper.In fold-and-punch tasks, models could miscount layers and consequently predict too few holes after unfolding.Source: §C.3 · p. 30
  • 15.2% of reasoning failures involved holistic-local processing, including choosing visually similar rotation options despite incorrect local component orientations.The authors associate these errors with difficulty allocating attention to task-relevant features.Source: §C.3 · p. 30
  • The authors conclude that these failures reflect missing geometric constraints, dynamic simulation in spatial working memory, and flexible attention, beyond simple pattern matching.Source: §C.3 · p. 30

Related papers

Loading 2510.13394v4…