PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Hongcheng Gao, Hailong Qu, Jingyi Tang, Jiahao Wang, Zihao Huang, Hengkang Qiao, Shihong Huang, Junming Yang, Yi Li, Hongyixuan Yuan, Wenjie Li, Bohan Zeng, Wenbo Li, Bo Wang, Jianhui Liu, Olive Huang, Haoyang Huang, Wentao Zhang, Guoqing Huang, Nan Duan, Yinpeng Dong

arXiv:2606.09669v2cs.AIcs.CL

TL;DR

Existing spatial-reasoning benchmarks do not adequately evaluate general interactive understanding because they rely mainly on passive tasks or simulator-specific setups. SpatialWorld introduces a unified benchmark for vision-only interactive tasks and finds that current agents remain unreliable, with GPT-5 achieving only 17.4% average task success.

  • Problem

    Existing spatial-reasoning benchmarks rely mainly on passive evaluation or simulator-specific setups, limiting assessment of general interactive spatial reasoning.

  • Method

    SpatialWorld unifies 760 human-annotated tasks across eight simulation backends through shared observation, action, and verification interfaces.

  • Results

    Across the benchmark, GPT-5 achieves an average task success rate (TSR) of only 17.4%, while Qwen-3.5-397B-A17B reaches 14.1%.

  • Takeaways & Limitations

    Current multimodal agents struggle with robust interactive spatial reasoning, showing low success, execution inefficiency, and substantial domain variance.

  • Takeaways & Limitations

    SPATIALWORLD evaluates simulated rather than physical environments, and its 760 handcrafted tasks remain modest in scale.

Abstract

from arXiv · show

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.

1 Introduction

SpatialWorld addresses the limitations of passive and simulator-specific spatial benchmarks by evaluating multimodal agents on dynamic, interactive task completion across heterogeneous 3D environments. Experiments show that current agents remain far from reliable, with GPT-5 reaching only 17.4% average TSR and Qwen-3.5-397B-A17B reaching 14.1%.

  • Motivation: Spatial reasoning enables MLLMs to perceive, understand, and operate within the physical world.
  • Limitations of Existing Benchmarks: Existing benchmarks mainly use passive evaluation, including static VQA and pre-recorded video understanding, rather than dynamic interaction.
  • Limitations of Existing Benchmarks: Many embodied benchmarks depend on simulator-specific embodiments, sensors, action interfaces, or execution pipelines, obscuring whether success reflects general spatial reasoning.
  • Benchmark: 760 human-annotated tasks span six domains and are instantiated across eight simulation backends in SPATIALWORLD.
  • Results: 17.4% average TSR is achieved by GPT-5, while Qwen-3.5-397B-A17B reaches 14.1% as the best open-source model.

2 SPATIALWORLD Benchmark

SpatialWorld benchmarks interactive spatial reasoning as vision-only, multi-turn decision-making in a unified protocol spanning heterogeneous simulators. Its execution-based evaluation combines terminal-state task success with step efficiency to distinguish effective solutions from exhaustive trial-and-error.

  • Task Formulation: Agents receive only natural-language goals and egocentric RGB observations, then select high-level actions from trajectory history without privileged depth or global-map signals.The interaction continues until EndTask or the step budget is exhausted, matching human-like perceptual conditions.
  • System Architecture: The standardized observation-action bottleneck converts heterogeneous environments into egocentric visual inputs and translates unified high-level decisions into simulator-specific execution commands.The shared action space covers navigation, viewpoint and posture, interaction, and task control or coordination.
  • Environment Suite: 8 backends are integrated across indoor simulation, outdoor navigation, and digital game environments to test object grounding, routines, coordination, and abstract spatial logic.The environment suite is designed to support cross-environment comparisons without interface bias, while lightweight games isolate geometric and topological reasoning.
  • Execution-Based Evaluation: TSR measures the fraction of tasks whose terminal goals are fully satisfied, while SE measures efficiency relative to human-annotated reference lengths.Reporting both metrics distinguishes efficient agents from exhaustive trial-and-error rather than relying on static trajectory matching.
  • Data Construction: 760 tasks are constructed with a taxonomy spanning scenario categories and complexity levels, followed by a unified pipeline intended to ensure quality, consistency, and reproducibility.The construction process includes environment collection, tutorial learning, instruction writing, success-condition definition, automated execution validation, and human cross-validation.

3 Experiment

The benchmark evaluates 15 MLLMs under standardized vision-only interaction and reports low, domain-dependent task success. Analyses show bottlenecks in efficiency, physical complexity, coordination, geometric reasoning, perception, and inference-time configuration.

  • Experimental Setup: 15 state-of-the-art MLLMs are evaluated through official APIs or open-weight checkpoints without task-specific fine-tuning, using egocentric RGB screenshots and natural-language task descriptions at each step.Main experiments use temperature τ = 1.0, retain the latest w = 30 interaction turns, and set each task’s step budget to 2g + 10.
  • Overall Results: 14.4% Physical Overall TSR is achieved by GPT-5, while Qwen-3.5-397B-A17B reaches 12.2%, with successes concentrated in short-horizon fundamental operations.Despite moderately better digital-domain performance, success rates remain universally low across the evaluated environments.
  • Efficiency: 0.745 SE is achieved by GPT-5.4 versus 0.584 for Kimi-K2.5 despite comparable Physical Overall TSRs of 6.6% and 9.2%, respectively.The contrast indicates heavier trial-and-error reliance by Kimi-K2.5; SE is informative mainly when models have comparable TSR and task distributions.
  • Domain Analysis: 14.1% and 13.7% indoor TSRs lead the physical domain for GPT-5 and Qwen-3.5-397B-A17B, while Gemini-3-Flash and GPT-5 lead outdoors at 9.0% and 8.3%.The indoor–outdoor divergence exposes distinct algorithmic biases and domain shifts in scene comprehension.
  • Task Complexity and Coordination: 4.2% mean TSR for Navigation–Interaction tasks contrasts with 50.2% for Interaction tasks, showing that precise manipulation combined with long-term spatial progress is substantially harder.The benchmark also finds varying leaders across complexity modes, while social performance is led by GPT-5 at 34.8% pooled TSR.
  • Game, Perceptual, and Inference Analysis: Top models handle reactive navigation and Snake tasks but struggle with geometric reasoning and multi-step transformations, while higher field of view helps until gains plateau.Resolution effects are comparatively flat and locally non-monotonic; temperature sensitivity is marginal, and optimal history-window and motion settings are model-dependent.

4 Related Work

Related work frames multimodal agents as systems that integrate multimodal perception with state representation, planning, and action, while spatial reasoning requires maintaining evolving beliefs from observations. Existing evaluations emphasize static visual question answering and simulator-specific settings, motivating more interactive assessment.

  • Multimodal Agents: Multimodal agents combine unified multimodal perception and state representations with multi-step planning and decision-making through tools or direct actions.The passage identifies text, images, and videos as example input modalities.
  • Simulation Platforms: Three-dimensional simulation platforms support spatial reasoning, navigation, and autonomous decision-making across diverse domains.Examples include AI2-THOR for interactive indoor manipulation and Habitat for efficient, configurable navigation and instruction-following research.
  • Spatial Reasoning: Spatial reasoning grounds goals in perceptual observations, maintains evolving spatial beliefs, and supports object localization, motion inference, planning, and physical action.These capabilities are described as operating under observability in real-world environments.
  • Existing Evaluation: Existing spatial-reasoning evaluations primarily use visual question answering with fixed 2D observations, with newer benchmarks extending to 3D and video.The extensions test whether models can build and recall spatial structure from sequential observations.
  • Existing Evaluation: Multi-turn interaction is essential because agents must make sequential decisions to gather information and update spatial beliefs over time.The passage contrasts this requirement with existing multi-step benchmarks, which are described as often simulator-grounded.

5 Conclusion

SPATIALWORLD is a unified benchmark for evaluating interactive spatial reasoning in multimodal large language models through active egocentric exploration and decision-making under partial observability. Evaluations of 15 leading models expose a critical gap between strong static scene perception and difficulty with dynamic tasks.

  • Benchmark contribution: SPATIALWORLD evaluates interactive spatial reasoning in MLLMs through a unified benchmark.It abstracts simulator-specific complexities into a shared text-based interface.
  • Evaluation scope: 15 leading MLLMs were evaluated for active egocentric exploration and decision-making under partial observability.The benchmark rigorously assesses these capabilities through its shared text-based interface.
  • Key finding: Current models excel at static scene perception but struggle profoundly with dynamic tasks.The evaluations reveal this as a critical vulnerability in existing multimodal language models.

A Additional Benchmark Details … A.4 Game-Level Performance Breakdown

The appendix extends SpatialWorld’s benchmark construction and evaluation details with cross-benchmark comparisons, task-specific scoring, and fine-grained indoor, outdoor, and digital-game analyses. These breakdowns show environment- and game-dependent differences in model performance, including distinct strengths across spatial tasks.

  • A Additional Benchmark Details: The appendix provides benchmark comparisons, task-specific evaluation criteria, and fine-grained performance breakdowns across indoor, outdoor, and digital-game environments.These materials supplement the benchmark construction and evaluation described in the main protocol.
  • A.1 Detailed Benchmark Comparison: Table 5 compares SpatialWorld with ImageQA, VideoQA, and embodied-agent benchmarks across five critical dimensions, including unified interfaces and dynamic interaction.The comparison also considers whether observations are captured, although the supplied passage is truncated before completing that dimension.
  • A.2 Task-Specific Evaluation Details: Binary TSR measures exact goal satisfaction, but certain environments require task-specific evaluation adaptations.SpatialWorld primarily uses binary TSR, with exceptions for environments where exact completion is insufficiently informative.
  • A.2 Task-Specific Evaluation Details: Snake3D reports a scale-normalized discrete score by dividing achieved snake score by the environment’s spatial edge length.This replaces sparse exact completion with a more granular measure of partial progress.
  • A.3 Indoor vs. Outdoor Performance Breakdown: Table 6 reports per-environment TSR for 15 models across indoor AI2THOR, ProcTHOR, and VirtualHome and outdoor CARLA and EmbodiedCity domains.Multi-agent environments are excluded and analyzed separately; GPT-5 and Qwen-3.5-397B-A17B dominate indoor scenarios, while GPT-5 and Gemini-3-Flash lead outdoor scenarios.
  • A.4 Game-Level Performance Breakdown: Table 7 reports per-game-family TSR across Block3D, Maze, Maze3D, Rubik’s Cube, and Snake, pooling available levels within each game.Snake normalizes scores by spatial edge length and caps each level contribution at 100%.
  • A.4 Game-Level Performance Breakdown: 39.0% is Gemini-3.1-Pro’s highest overall efficacy, with 40.0% on Block3D and 90.0% on Snake; Gemini-3-Flash leads Rubik’s Cube at 50.0%.Qwen3-VL-235B-Thinking leads Maze at 70.0% and Maze3D at 32.0%, whereas GPT-5 is strongest on Snake at 91.2%.

B Ablation Studies … B.3 Continuous versus Discrete Motion

The ablations show that optimal inference-time settings are model-dependent, while the benchmark adopts τ = 1.0, w = 30, and discrete actions for uniform evaluation. Continuous motion has no universal advantage over discrete motion.

  • B Ablation Studies: Inference-time ablations examine temperature, history window size, and action parameterization, finding that optimal settings vary across models.The study analyzes these three factors to complement the main-text analysis.
  • B.1 Temperature: τ = 1.0 is optimal for nearly all models except Gemini-3-Flash.The benchmark therefore follows OSWorld’s τ = 1.0 protocol for all models.
  • B.1 Temperature: τ = 1.0 is adopted universally to preserve protocol uniformity while maintaining moderate sampling diversity during long-horizon interactions.
  • B.2 History Window Size: w = 30 is optimal across most evaluated models, after which performance plateaus or slightly declines.The result suggests that 30 frames provide sufficient temporal information without universal gains from longer visual histories.
  • B.2 History Window Size: w = 30 is used as the default context window for the main benchmark.This setting reflects the observed cross-model performance pattern.
  • B.3 Continuous versus Discrete Motion: ∆TSR = TSRcontinuous −TSRdiscrete has no universal sign across models, so neither continuous nor discrete motion is consistently superior.Positive values favor continuous action parameters, whereas negative values favor the discrete interface.
  • B.3 Continuous versus Discrete Motion: Discrete actions are used in the main benchmark because the optimal parameterization is model-dependent.This choice maintains interface consistency across environments and avoids biasing the evaluation.

C Observation Sensitivity Analysis … F Action Space Definition

SpatialWorld combines resolution-robust spatial perception with a simulator-agnostic suite spanning indoor, outdoor, and controlled game environments. Its 760 human-annotated tasks and unified text action interface support reproducible evaluation across diverse embodied behaviors.

  • C Observation Sensitivity Analysis: Lower visual resolution does not impair spatial reasoning, because physical, projective, and ray relationships remain unchanged.Visualizations confirm that the model maintains accurate spatial awareness across resolutions.
  • D Environment Suite: Eight 3D backends share one agent-side API while preserving differences in scale, dynamics, object affordances, and scene generation.This design makes the environment suite a source of domain diversity rather than a passive collection of scenes.
  • D Environment Suite: Indoor environments test object grounding and manipulation through affordances, physical interactions, procedural layouts, and household activities.AI2-THOR provides hand-designed near-photorealistic scenes, ProcTHOR adds procedurally generated houses, and VirtualHome represents household activities.
  • D Environment Suite: Outdoor environments evaluate road topology, route progress, traffic context, landmark localization, altitude-aware movement, and macroscopic urban planning.CARLA targets urban driving, while EmbodiedCity targets aerial city navigation.
  • D Environment Suite: Lightweight 3D games isolate abstract spatial abilities under closed-loop interaction and broaden coverage beyond realistic household and urban navigation.These controlled probes expose spatial structures that are rare in realistic simulators but central to general 3D reasoning.
  • E Human Annotation: 760 tasks undergo three human-annotation stages covering task design, independent simulator execution, and recording of terminal states and reference trajectories.The process specifies natural-language instructions and initial states before annotators independently solve tasks and record ground truth.
  • F Action Space Definition: The unified action space abstracts backend commands into high-level text primitives grouped into four canonical categories, including explicit no-motion or wait decisions.The interface is designed as an MLLM-native action space, with 0 representing a deliberate hold-position decision.
  • F Action Space Definition: Move spans 0.25 m indoor steps, 10 m driving advances, and waiting, while ChangeState and Manipulate separate persistent state transitions from local interventions.ChangeState covers actions such as opening or cooking; Manipulate covers pushing or grabbing.

G GPT-5 vs. GPT-5.4 Case Study

On 609 shared single-agent physical tasks, GPT-5 outperforms GPT-5.4, with the advantage concentrated in several environments and accompanied by asymmetric task recovery. The models’ error profiles suggest differing termination policies: GPT-5.4 stops prematurely, whereas GPT-5 more often persists without completing.

  • GPT-5 vs. GPT-5.4 Case Study: +28.9 points, +7.7, and +6.2: GPT-5’s largest shared-task advantages occur in VirtualHome, AI2THOR, and CARLA, respectively.EmbodiedCity slightly favors GPT-5.4 (-3.8), while ProcTHOR remains unsolved by both models.
  • GPT-5 vs. GPT-5.4 Case Study: 52 tasks versus 14: GPT-5 solves 52 tasks that GPT-5.4 misses, while GPT-5.4 recovers only 14 in the reverse direction.The disagreement is asymmetric, and most of GPT-5’s advantage comes from Daily Household, Work & Study.
  • GPT-5 vs. GPT-5.4 Case Study: 32.4% and 48.5%: GPT-5.4 failures are primarily premature DONE decisions and explicit FAIL decisions, respectively.This error profile indicates a tendency toward premature termination rather than a difference in raw action speed.
  • GPT-5 vs. GPT-5.4 Case Study: 63.6% and 15.1%: GPT-5 failures more often end at the step limit or after repeated action failures, indicating persistence without completion.GPT-5 typically spends more actions when it succeeds and when it fails, consistent with a slower but more persistent search strategy.

H Qualitative Analysis

Qualitative analysis categorizes 100 failed trajectories into four failure modes: spatial disorientation, object hallucination, premature termination, and action loops. Representative cases across four environments show these failures arising from inaccurate spatial tracking, unsupported object assumptions, and ineffective repeated actions.

  • Failure Mode Breakdown: 100 failed trajectories are categorized into spatial disorientation, object hallucination, premature termination, and action loops.Spatial disorientation involves losing track of position; object hallucination triggers interactions with unseen objects; premature termination ends tasks early; action loops repeat ineffective sequences.
  • Bad cases analysis: Representative bad cases across four environments cover the full spectrum of identified failure modes.The analysis examines GPT-5 and Gemini-3.1-Pro on spatial disorientation and premature termination, alongside object hallucination and action-loop examples.
  • Bad cases analysis: At Step 7, Gemini-3.1-Pro hallucinates having grasped the phone and then continues moving forward after colliding with a wall at Step 9.The example illustrates object hallucination followed by an action loop, attributed to limited complex spatial understanding in real-world scenarios.

I Limitations and Broader Impact

SpatialWorld is limited by its reliance on simulated environments and its modest, handcrafted task scale. It primarily serves as a diagnostic tool that supports reliable spatial-agent development and open, reproducible evaluation.

  • Limitations: SpatialWorld evaluates agents in simulated rather than physical robotic environments, despite near-photorealistic rendering and physically plausible dynamics.Extending evaluation to real-world settings remains a future direction.
  • Limitations: 760 tasks are carefully handcrafted to ensure annotation quality, making the benchmark more modest in scale than automatically generated datasets.The passage contrasts the current task count with automatically generated datasets without specifying their size.
  • Broader Impact: SpatialWorld functions primarily as a diagnostic and scientific tool for characterizing multimodal agents’ spatial-reasoning capabilities and failure modes.This characterization is intended to support more reliable and trustworthy spatial-agent systems.
  • Broader Impact: Its emphasis on open and reproducible evaluation fosters transparency in the research community.The passage presents transparency as a broader-impact benefit of the evaluation framework.

J Compute Resources · K LLM Usage

The evaluation used official APIs for proprietary models and an 8× NVIDIA H200 server for open-source models, consuming approximately 5,000 GPU hours. GPT-5 assisted only with writing and formatting, while 15 multimodal LLMs served as external test agents rather than components of the methodology.

  • J Compute Resources: Approximately 5,000 GPU hours powered the full evaluation campaign across models deployed on the GPU server.The server was equipped with 8× NVIDIA H200 GPUs.
  • J Compute Resources: Proprietary models were accessed exclusively through official APIs.Examples included GPT-5, Gemini-3.1-Pro-Preview, and Claude-Sonnet-4.6.
  • J Compute Resources: Open-source models were deployed on a GPU server equipped with 8× NVIDIA H200 GPUs.Examples included Qwen2.5-VL-72B-Instruct and InternVL3-78B.
  • K LLM Usage: GPT-5 assisted with grammar, phrasing, clarity, figure and table captions, and layout suggestions.The assistance covered surface-level writing and formatting edits.
  • K LLM Usage: The writing assistant did not contribute to research ideation, experimental design, implementation, data analysis, or technical content beyond surface-level edits.The authors reviewed and edited all outputs and retained responsibility for the final text.
  • K LLM Usage: Reported failure cases included Spatial Disorientation, Premature Termination, Object Hallucination, and Action Loop across AI2-THOR, VirtualHome, CARLA, and ProcTHOR.Examples involved GPT-5, Gemini-3.1-Pro, and Qwen-3.5-397B-A17B.
  • K LLM Usage: 15 multimodal LLMs were used as external test agents rather than as core, original, or non-standard components of the methodology.Their role was to evaluate the proposed benchmark, not to define its method design.
Loading 2606.09669v2…