PaperlayerView exact revision on arXiv

Source-linked AI summary

Holistic Evaluation of Multimodal LLMs on Spatial Intelligence

Zhongang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wanqi Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Oscar Qian, Hui En Pang, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng, Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang

arXiv:2508.13142v5cs.CVcs.CLcs.LGcs.MMcs.RO

TL;DR

Spatial intelligence is important for embodied agents, but advanced multimodal models still struggle with spatial tasks that humans find easy. EASI unifies benchmark tasks and evaluation protocols, and its evaluation finds strong model progress alongside persistent human-performance gaps and no decisive proprietary advantage on the hardest tasks.

  • Problem

    Spatial understanding and reasoning are essential for embodied agents, yet advanced multimodal models often fail at spatial tasks that are easy for humans.

    Source: §1 · p. 1

  • Method

    EASI unifies spatial benchmarks under six fundamental capabilities and evaluates models across eight benchmarks using systematic protocols.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    GPT-5 leads spatial-intelligence benchmarks, but models remain below humans on most benchmarks, perform worse on spatial than non-spatial tasks, and show no decisive proprietary advantage on the hardest tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The EASI codebase and leaderboard provide reproducible evaluation resources and an updated view of model performance across spatial intelligence.

    Source: §1 · p. 3

    GPT-5 still misinterprets camera rotation, indicating that it lacks a solid ability for perspective transformation.

    Source: §4 · p. 8

Abstract

from arXiv · show

Multimodal models have achieved remarkable progress in recent years. Nevertheless, they continue to exhibit notable limitations in spatial understanding and reasoning, the very capability that anchors artificial general intelligence in the physical world. With the recent release of GPT-5, allegedly the most powerful AI model to date, it is timely to examine where the leading models (GPT, Gemini, Grok, Seed, Qwen, and Intern) stand on the path toward spatial intelligence (SI). We thus propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence. EASI conceptualizes a comprehensive taxonomy of spatial tasks that unifies existing benchmarks and a growing collection of newly curated ones, enabling systematic evaluation of state-of-the-art models. In this report, we conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens. Our empirical study then reveals that (1) GPT-5 demonstrates unprecedented strength in SI, yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks. Moreover, we (3) show that SI-tasks expose greater model capability deficiency than non-SI tasks, to the extent that (4) proprietary models do not exhibit a decisive advantage when facing the most difficult ones. In addition, we conduct a qualitative evaluation across a diverse set of scenarios that are intuitive for humans, yet fail the most advanced multimodal models. EASI is an ongoing community effort: we have open-sourced the EASI codebase that provides a one-stop and reproducible solution with standardized interfaces, integrated protocols and prompts that significantly reduce the friction of configuring and running multiple benchmarks; we have also launched an accompanying EASI leaderboard to provide a continually updated snapshot of model performance across the full SI spectrum, accelerating collective progress toward robust SI.

1 Introduction

Spatial intelligence is essential for embodied agents, yet advanced multimodal models still fail at tasks humans find easy. EASI unifies benchmarks and evaluation protocols, then finds that leading models remain substantially weaker on spatial than non-spatial tasks, with persistent difficulties across several capabilities.

1 Introduction

  • EASI consolidates spatial benchmarks into six fundamental capabilities and examines prompts, metrics, and scoring strategies for consistent cross-benchmark evaluation.The report evaluates eight representative benchmarks and presents EASI as extensible to new benchmarks.Source: §1 · p. 2
  • GPT-5 ranks as the new spatial-intelligence state of the art, yet models remain below humans on most benchmarks and spatial tasks expose greater deficiencies than non-spatial tasks.Proprietary models also show no decisive advantage on the most challenging spatial tasks.Source: §1 · p. 2
  • Case studies align with the quantitative pattern: Metric Measurement and Spatial Relations are relatively strong, while four other capabilities retain substantial room for improvement.The analysis identifies perspective effects and integrated, multi-stage reasoning as challenges; it suggests missing spatial representations can undermine otherwise sound strategies.Source: §1 · pp. 2–3
  • EASI releases a reproducible codebase and continually updated leaderboard to support comparisons across the spatial-intelligence spectrum.The codebase consolidates tasks, protocols, prompts, and evaluation pipelines.Source: §1 · p. 3

2 Evaluation Benchmarks

EASI organizes spatial benchmarks around shared capabilities and examines evaluation choices that affect cross-benchmark comparisons. The report uses official benchmark settings across eight benchmarks, with a balanced small version of MindCube and primarily non-circular evaluation.

2 Evaluation Benchmarks

  • EASI is designed to extend beyond the benchmarks in this report while unifying spatial-intelligence tasks and supporting fair cross-benchmark comparison.The report distinguishes its benchmark-specific Official Protocol from a unified EASI Protocol described in the appendix.Source: §2 · p. 3, §2.2 · p. 4

2.1 Six Fundamental Capabilities

  • Six shared capabilities organize benchmark tasks: Metric Measurement, Mental Reconstruction, and Spatial Relations cover scale, object structure, and relative positions or orientations.Mental Reconstruction involves inferring detailed 3D structure from constrained views; Spatial Relations concerns objects within the camera view and does not require imagining a viewpoint transformation.Source: §2.1 · pp. 3–4
  • Perspective-Taking concerns reasoning across viewpoints, while Deformation and Assembly covers structural changes such as unfolding or assembling parts.Perspective-Taking can involve imagined viewpoint changes from one image; Deformation and Assembly addresses transformations beyond shape consistency.Source: §2.1 · p. 4
  • Comprehensive Reasoning requires coordinated spatial abilities, extended memory, and multi-stage reasoning, as in navigation or long-horizon puzzles.The six capabilities are distilled to accommodate benchmarks with different taxonomies.Source: §2.1 · pp. 3–4

2.2 Evaluation Protocols

  • For direct comparison with existing baselines, the report uses each benchmark’s official prompt and native metric rather than a single uniform setting.Official prompts include Direct QA, Zero-shot CoT, and Manual CoT; SITE uses Chance-Adjusted Accuracy and VSI-Bench uses Mean Relative Accuracy for numerical answers.Source: §2.2 · pp. 4–5
  • Answer extraction uses two rule-based passes, and standard non-circular evaluation is used because its rankings broadly agree with circular strategies.CoreCognition is the exception: it uses soft-circular evaluation to match its original paper.Source: §2.2 · p. 5

2.3 Benchmark Statistics

  • The eight-benchmark evaluation covers about 31K images, 4.5K videos, and 24K questions; MindCube-Tiny is used to avoid the imbalance of the larger MindCube subsets.MindCube-Tiny contains 1,050 QA pairs across among, around, and rotation subsets, with 428 unique images.Source: §2.3 · p. 5, Table 2 · p. 5

3 Results

GPT-5 ranks first overall in the eight-benchmark evaluation, with strong results on Metric Measurement and Spatial Relations. Yet substantial model–human gaps remain, spatial tasks are harder than non-spatial tasks, and proprietary models lack a decisive edge on the hardest tasks.

3 Results

  • GPT-5 ranks first overall and leads on all benchmarks except MindCube, reaching human-level performance on Metric Measurement and Spatial Relations tasks.The SITE human baseline is noticeably lower than on other benchmarks, and challenging variants such as Attribute (Measurement) and Block Moving remain insufficiently addressed.Source: §3 · p. 6
  • Human–model gaps often exceed 30 points across Perspective-Taking, Comprehensive Reasoning, and Deformation and Analysis tasks.The gaps span 18 Perspective-Taking subtasks and eight Comprehensive Reasoning subtasks; five SpatialViz Deformation and Analysis subtasks show gaps ranging from over 30 to more than 50 points.Source: §3 · p. 6
  • GPT-5 scores below random guessing on two Mental Reconstruction subtasks: Attribute (Appearance) in MMSI and 3D Mental Rotation in SpatialViz.Both scores are below 0 on the CAA metric.Source: §3 · p. 7
  • Models perform close to or above humans on non-spatial tasks but struggle on spatial tasks, including 3D Mental Rotation and 3D Transformation.The best model is roughly 46 points lower on 3D than 2D Mental Rotation and about 20 points lower on 3D than 2D Transformation in STARE.Source: §3 · p. 7
  • On the most difficult spatial tasks, proprietary models typically lead open-source models by around or less than 15 points, and open-source models can match or surpass them in some categories.The around-or-less-than-15-point gap applies to subtasks where performance remains more than 60 points below human levels in MMSI; parity or better is reported for Geometric and Hypothetical Reasoning in OmniSpatial's official setting.Source: §3 · p. 7

4 Case Study

The case study finds that GPT-5 excels at some spatial tasks but remains far from human-level spatial intelligence in general. Its qualitative examples show strong performance in metric measurement alongside persistent weaknesses across several other capabilities.

Case-study setup

  • The case study uses website platforms rather than APIs and finds that GPT-5 excels in some tasks but remains short of general human-level spatial intelligence.The authors say these qualitative findings are largely consistent with their previous analysis.Source: §4 · p. 7

Metric Measurement (MM)

  • GPT-5 performs reliably on basic real-world measurement images, using common object sizes to estimate dimensions despite ambiguity without camera intrinsics.The authors consider a relatively large margin of error acceptable for these estimates.Source: §4 · p. 7

Mental Reconstruction (MR)

  • GPT-5 shows stronger multi-view reconstruction and novel-view generation than earlier models, but accurate generation is prompt-sensitive and MR4 defeats all tested state-of-the-art models.The authors report that GPT-5-thinking particularly outperforms o3 in generating correct top-down views.Source: §4 · p. 7

Spatial Relations (SR)

  • GPT-5 generally handles spatial relations well, but complex scenes with perspective effects and visual illusions can confound its judgments of objects’ relative sizes.In the cited example, it shows no substantial improvement over o3.Source: §4 · p. 7

Perspective-taking (PT)

  • GPT-5 struggles to reason across changing camera viewpoints, especially when views overlap minimally, and misinterprets camera rotation.Its reasoning attempts to establish visual correspondences across perspectives.Source: §4 · pp. 7–8

Deformation and Assembly (DA)

  • GPT-5 fails at mental folding and object assembly, including folding a 2D net into a 3D cube, revealing difficulty reasoning beyond rigid shapes.The authors suggest that ineffective training data may contribute to this weakness.Source: §4 · p. 8

Comprehensive Reasoning (CR)

  • GPT-5 recognizes visible blocks but struggles to infer hidden blocks when counting partially occluded objects.The authors interpret this as a gap in reasoning about physical principles beyond what is directly plausible in pixel space.Source: §4 · p. 8

5 Conclusion

The report concludes that spatial intelligence remains a major challenge for multimodal models, despite leading models’ state-of-the-art performance. It presents EASI as a unified framework and releases a toolkit and leaderboard for consistent evaluation.

Conclusion

  • Even leading multimodal models remain substantially below human performance in Mental Reconstruction, Perspective-Taking, Deformation and Assembly, and Comprehensive Reasoning.The report identifies spatial intelligence as a critical challenge for large multimodal models.Source: §5 · p. 8
  • EASI unifies existing spatial-intelligence benchmarks through fundamental capabilities and evaluation protocols for robust model-performance analysis.The report frames closing the remaining gaps as important for advancing robust spatial reasoning toward embodied general intelligence.Source: §5 · p. 8
  • The released EASI toolkit provides standardized interfaces, official protocols and prompts, and reproducible pipelines; its leaderboard offers continually updated performance snapshots.The unified environment is intended to reduce the overhead of configuring and running disparate benchmarks.Source: §5 · p. 8

GPT-o3

Figure 3 compares GPT-5 variants with GPT-o3 and Doubao-Seed-1.6-thinking in qualitative case studies. The caption reports gains for GPT-5-thinking while noting that it has not mastered the full range of spatial intelligence.

GPT-o3

  • Figure 3 compares GPT-5-thinking, standard GPT-5, GPT-o3 [44], and Doubao-Seed-1.6-thinking [51].The comparison uses web-based services.Source: Figure 3 · p. 9
  • GPT-5-thinking shows notable improvement over its predecessors but remains far from mastering the full spectrum of spatial intelligence.Source: Figure 3 · p. 9
  • Doubao-Seed-1.6-thinking is omitted from the MR2 and MR3 visual comparisons because it cannot generate images.Source: Figure 3 · p. 9

A.1 Evaluation Prompt Comparison

The section compares a unified EASI prompt with benchmark-specific official prompts. EASI uses zero-shot chain-of-thought and structured answer formatting, but its performance effect varies across datasets and models.

A.1 Evaluation Prompt Comparison

  • Only results reported under the EASI Protocol use EASI Prompts; Table 4 compares EASI and Official prompt settings across eight benchmarks.For the Official setting, the table follows each benchmark’s original prompt and inference protocol.Source: Figure 4 · p. 15, Table 4 · p. 15
  • EASI combines zero-shot chain-of-thought with instructions to ground answers visually and provide a single letter inside <think>/<answer> tags.Structured answer templates are intended to improve answer matching accuracy.Source: §A.1 · p. 15
  • EASI prompting has no consistent advantage: effects vary by dataset and model, with Official prompts better for open-source models on SITE and MindCube.Explicit reasoning sometimes improves multi-step spatial tasks, but gains are not uniform across datasets or model families.Source: §A.1 · p. 16

A.2 Evaluation Metrics

The section defines the paper’s evaluation metrics and compares circular testing protocols, finding that Non-circular results preserve broadly consistent model rankings while reducing evaluation cost.

A.2 Evaluation Metrics

  • CAA = 1 means all questions are correct, CAA = 0 matches random guessing, and CAA < 0 is worse than random.CAA uses the total question count and the number of options for each question.Source: §A.2 · p. 16
  • For VSI-Bench Numerical Answer questions, the evaluation follows the original paper and reports MRA averaged across 10 confidence thresholds from 0.5 to 0.95.A prediction counts as correct only when its relative error is below 1 minus the threshold.Source: §A.2 · p. 16
  • Detailed benchmark results also report Accuracy and F1, while precision and recall are defined from true-positive, false-positive, and false-negative counts.For Accuracy, the section identifies each question’s ground-truth and predicted labels.Source: §A.2 · p. 16
  • CAA amplifies score differences relative to Accuracy on the same dataset, with stronger amplification when random-guess accuracy is higher.The passage contrasts binary questions, where random-guess accuracy is about 0.5, with many-option questions, where it is lower.Source: §A.2 · p. 16

A.3 Circular Strategies

  • Model rankings are broadly consistent between Non-circular and Hard-circular evaluation, so Non-circular results suffice for fair comparison at lower evaluation time and computational cost.A large drop from Non-circular to circular scores may indicate accuracy from successful random guesses; Hard-circular requires every rotated variant to be answered correctly.Source: §A.3 · p. 17, Table 5 · p. 17

B Cross-benchmark Comparison

The report compares benchmark-native evaluation with a unified EASI protocol, then reports results across VSI-Bench, SITE, MMSI, OmniSpatial, and MindCube. GPT-5 performs strongly on several tasks, but important gaps and benchmark-specific exceptions remain.

B Cross-benchmark Comparison

  • The Official Protocol uses each benchmark’s native prompt and metric, whereas EASI standardizes prompts and metrics to support cross-benchmark comparisons.For EASI, multiple-choice questions use Chance-Adjusted Accuracy and numerical-answer questions use Mean Relative Accuracy.Source: B Cross-benchmark Comparison · p. 17

C.1 VSI-Bench

  • GPT-5 ranks first or near the top on VSI-Bench and approaches or exceeds human performance on some Metric Measurement tasks, but trails humans on Perspective-taking and Comprehensive Reasoning.It exceeds human performance on Object Size and Room Size.Source: §C.1 · p. 17, Table 7 · p. 18, Table 8 · p. 19

C.2 SITE

  • Under the unified EASI setting, GPT-5 is near state of the art across almost all SITE subsets, yet a performance gap above 30 percentage points remains.It approaches or exceeds human performance in 3D Information Understanding, Spatial Relationship Reasoning, and Motion Prediction and Navigation; SITE’s human score is around 60, while other benchmarks are above 75 or 90.Source: §C.2 · pp. 20–21, Table 10 · p. 20

C.3 MMSI

  • On MMSI, proprietary and open-source models differ by less than 15 points even when overall performance is more than 60 points below human level.GPT-5 scores below random guessing on the Approximation task, and viewpoint transformations remain a weakness.Source: §C.3 · pp. 21–22, Table 12 · p. 21

C.4 OmniSpatial

  • On OmniSpatial, proprietary models have no decisive advantage on difficult spatial tasks, and Complex Reasoning and Perspective-taking show the largest human-performance gaps.The Official setting’s best open-source models can outperform or match proprietary state of the art.Source: §C.4 · p. 22, Table 13 · p. 22, Table 14 · p. 23

C.5 MindCube

  • Gemini-2.5-pro is best on MindCube, while proprietary models do not significantly outperform open-source models on Among and Around.Rotation scores around 85–95 points for leading closed-source models, but that task uses fixed-position camera rotation and discrete angular differences; Qwen2.5-VL-7B-Instruct also underperforms its 3B counterpart.Source: Table 15 · p. 23, §C.5 · p. 24, Table 16 · p. 24

C.6 STARE

Results across STARE, CoreCognition, and SpatialViz show that performance depends on task type: models can approach human scores on non-spatial tasks while continuing to struggle with spatial reasoning.

C.6 STARE

  • On STARE, proprietary models outperform open-source models by about 10–15 points on average, and GPT-5 benefits consistently from visual-simulation images under EASI.Under the Official setting, visual simulation helps GPT-5 on Cube Net and Tangram but lowers its scores on 2D and 3D Transformation.Source: §C.6 · p. 24, Table 17 · p. 25, Table 18 · p. 26

C.7 CoreCognition

  • On CoreCognition, GPT-5 substantially outperforms other models on Perspective-taking but remains below human performance, while leading MLLMs outperform humans on several non-spatial tasks.The non-spatial tasks include Boundary, Perceptual Constancy, Conservation, and all Formal Operation subtasks.Source: Table 19 · p. 26, §C.7 · p. 27, Table 20 · p. 27

C.8 SpatialViz

  • On SpatialViz, all models lag humans on spatial tasks, especially Paper Folding, and GPT-5 performs worse than random guessing on some 3D Rotation and Cube Reconstruction tasks.GPT-5’s 3D Rotation score is over 60 percentage points below its 2D Rotation score under the Official setting; Grok-4’s overlong responses were truncated on more than 300 of 1,180 items.Source: Table 21 · p. 27, §C.8 · p. 28, Table 22 · p. 28

D Token Consumption

The paper compares hidden internal reasoning with user-visible output and finds substantial variation in token use across models. GPT-5 is described as efficient, while token limits and long reasoning can pose evaluation risks.

D Token Consumption

  • Grok-4 and Gemini-2.5-pro use substantial hidden reasoning tokens, with Grok-4 often reaching the 16,384-token cap, while GPT-5 and Seed-1.6 use more constrained internal reasoning.Figure 5 plots token usage against the cumulative percentage of questions, separately for internal and external tokens.Source: D Token Consumption · pp. 28–29, Figure 5 · p. 29
  • Gemini-2.5-pro produces the longest user-visible outputs, while Grok-4, Seed-1.6, and InternVL3 each produce approximately 2.5–7.5% extremely short responses.Some Grok-4 responses have no external tokens when the model exhausts its internal token limit.Source: D Token Consumption · p. 29
  • A higher exposure ratio indicates more reasoning in user-visible output, while a lower ratio indicates heavier reliance on hidden internal reasoning.The authors identify p95 token usage as practically relevant when choosing safe token caps.Source: D Token Consumption · p. 29, Table 23 · p. 29
  • The report characterizes GPT-5 as efficient, with minimal internal token use, compact outputs, and strong benchmark performance, while Grok-4 faces truncation risk under fixed token budgets.Open-source models in the study have no internalized reasoning process and therefore no applicable exposure ratio.Source: D Token Consumption · pp. 28, 30

E Thinking Modes of GPT-5

The paper evaluates GPT-5’s reasoning-effort settings on a 118-question SpatialViz-Tiny sample. Higher effort raises reasoning tokens and runtime, but High mode carries timeout and truncation risks.

E Thinking Modes of GPT-5

  • The comparison samples one-tenth of each SpatialViz task for 118 total questions and sets max_completion_tokens to 16,384.Reasoning tokens and runtime increase with thinking-mode level; accuracy improves from Minimal to Medium, with diminishing marginal benefit.Source: E Thinking Modes of GPT-5 · p. 30
  • High mode’s accuracy is 52.54% when 28 timed-out or token-limited questions count as incorrect, and 68.89% when those cases are excluded.The remaining questions are typically less difficult, and Medium is used as a more balanced accuracy–cost trade-off.Source: E Thinking Modes of GPT-5 · p. 30, Table 24 · p. 30

F Elaborated Results in the Case Study

This section presents GPT-5-thinking responses to representative spatial tasks, with correct reasoning marked green and problematic reasoning marked red.

Metric Measurement (MM)

Mental Reconstruction (MR)

Spatial Relations (SR)

Perspective-Taking (PT)

Deformation and Assembly (DA)

Comprehensive Reasoning (CR)

Related papers

Loading 2508.13142v5…