PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

Tianhui Liu, Jie Feng, Zhiheng Zheng, Shengyuan Wang, Yiming Guo, Yanxin Xi, Hangyu Fan, Yong Li, Pan Hui

arXiv:2605.31148v1cs.CVcs.AIcs.CL

TL;DR

Existing spatial evaluations often test passive reasoning or combine action with low-level control, leaving unclear whether VLMs can reason over action-driven 3D changes. SpatialAct evaluates this capability and finds that strong basic-task performance does not consistently transfer to multi-turn repair, while current systems remain far below human reliability.

  • Problem

    Existing evaluations mostly test VLMs as observers or entangle spatial reasoning with low-level control, leaving a gap in evaluating action-conditioned reasoning.

    Source: §1 · p. 2

  • Method

    SpatialAct is a simulator-grounded benchmark evaluating five basic spatial abilities, one-step error repair, and iterative repair through feedback using high-level executable actions.

    Source: §1 · p. 2

  • Results

    Strong basic-task results do not consistently transfer to robust multi-turn repair, and all current systems remain far below human reliability on iterative spatial correction.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    SpatialAct provides an actionable diagnostic framework for developing VLM agents with stronger spatial state tracking, feedback-loop coordination, and context-grounded action generation.

    Source: §5 · p. 11

    All evaluations use simulated scenes, so how performance translates to real-world environments remains unclear.

    Source: §A.1 · p. 15

Abstract

from arXiv · show

Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce \textbf{SpatialAct}, a simulator-grounded benchmark for probing \textit{action-conditioned spatial reasoning} in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear reasoning-to-action gap: current VLMs can perform well on isolated spatial reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.

1 Introduction

SpatialAct targets action-conditioned spatial reasoning: whether VLM agents can use high-level actions to change a 3D scene, track the resulting state, and reason consistently across interactions. Its evaluation exposes a gap between isolated spatial understanding and reliable multi-turn refinement.

Evaluation gap

  • Prior spatial benchmarks largely assess observers, while embodied benchmarks often entangle high-level spatial reasoning with low-level control, leaving multi-round semantic action and state tracking under-evaluated.Existing work spans 2D relations, spatial cognition, 3D reasoning, richer scenes and videos, and active spatial belief construction.Source: §1 · pp. 1–2

Action-conditioned spatial reasoning

  • SpatialAct operationalizes action-conditioned reasoning as interactive 3D layout refinement, using executable high-level actions and verifiable state changes to isolate spatial reasoning from low-level motor control.Layout refinement tests spatial validity, including collision avoidance, boundary consistency, and plausible orientation.Source: §1 · p. 2

Benchmark

  • SpatialAct uses simulator-executed move, rotate, and scale commands, then returns updated multi-view renderings; it covers 333 scenes and 4,355 QA pairs across three scenario types.The scenarios are Abstract Geometric, Urban Architectural, and Indoor Scene; procedural generation and injected spatial errors support controllable evaluation.Source: §1 · p. 2

Results

  • The strongest VLM achieves 0.411 Repair Rate and 0.206 Scene Success Rate, versus human participants’ 0.911 Repair Rate and 0.763 Scene Success Rate, despite around 80% accuracy in several basic categories.The results indicate difficulty maintaining coherent spatial beliefs and producing reliable actions across long-horizon state transitions.Source: §1 · pp. 2–3

Task design

  • Its hierarchical diagnosis progresses from five Basic Spatial Abilities to Single-step Error Detection and Fix, then Multi-turn Interactive Refinement with closed-loop simulator feedback.The basic abilities are object meaning, spatial relations, spatial orientation, mental rotation, and spatial visualization.Source: §1 · pp. 2–3

2 Methods

SpatialAct is a simulator-grounded benchmark for evaluating VLM spatial reasoning and action across geometric and everyday 3D scenes. Its methods combine hierarchical diagnostic tasks with interactive refinement and complementary evaluation measures.

2 Methods

  • SpatialAct contains 4,355 QA pairs across three layout scenarios and three question formats, spanning controlled geometric layouts and daily-life 3D scenes.The scenarios are Abstract Geometric, Urban Architectural, and Indoor Scene; the latter two are constructed from 333 Scenes.Source: §2 · p. 3

2.1 Benchmark Construction

  • Benchmark scenes combine procedurally generated geometric layouts, urban scenes derived from RAISECity [24], and indoor environments built from InternScenes [41].Geometric scenes typically contain 5–7 objects, urban scenes no more than 20 buildings, and indoor scenes 5–15 objects; complex objects are required for Mental Rotation scenes.Source: §2.1 · pp. 3–4
  • Quality control cleans and manually filters urban and indoor scenes for abnormalities and plausible arrangements, with additional semantic-consistency checks for indoor objects.Procedurally generated Abstract Geometric scenes use predefined constraints and require no additional filtering.Source: §2.1 · p. 4
  • The hierarchical task design centers on Multi-turn Interactive Refinement, decomposed into Single-step Error Detection and Fix and five basic spatial ability tasks.The five abilities are Object Meaning, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization; questions use open-ended, multiple-choice, or multi-turn feedback formats.Source: §2.1 · p. 4

2.2 Evaluation Environment Setup

  • In multi-turn refinement, VLMs inspect rendered views, issue move, rotate, or scale commands, and receive updated views plus prior action history until errors are fixed or the turn limit is reached.Urban errors include collision, road conflict, and orientation; indoor errors include collision, wall conflict, and orientation.Source: §2.2 · p. 5
  • Human participants refine scenes through a web interface with direct object manipulation, transform controls, perspective switching, and an operation-history log.The interface supports position, rotation, and scale adjustments through an interactive handle or Transform Properties panel.Source: §2.2 · p. 5
  • Basic spatial abilities and single-step correction use accuracy, while multi-turn refinement is evaluated for repair accuracy and efficiency.Repair metrics include Repair Rate and Scene Success Rate; efficiency metrics cover Effective Repair Turn Ratio, Premature Stop Rate, and Average Completion Tokens per Scene.Source: §2.2 · p. 6

3 Experiments

VLMs perform better on isolated spatial tasks than on multi-turn repair, where they struggle to track scene state and turn diagnoses into reliable actions. The experiments also show that repair performance varies with scene structure, error complexity, context length, and foundational skills.

3.1 Benchmark VLM Agents

  • Seven VLMs are evaluated under unified inference settings, with multi-turn repair capped at 30 iterations and each response capped at 8,096 completion tokens.The evaluated set includes four proprietary models and three open-source models.Source: §3.1 · p. 7

3.2 Main Results

  • 0.411 Repair Rate and 0.206 Scene Success Rate make Gemini-3.1 Pro the leading model; humans reach 0.911 and 0.763, respectively.Closed-source models substantially outperform open-source models and GLM-5V-Turbo on multi-turn repair, but even the strongest model remains far below human performance.Source: §3.2 · pp. 7–8, Figure 5 · p. 8
  • Multi-turn repair failures arise from both misdiagnosing spatial violations and issuing incorrect corrective actions, while models struggle to maintain consistent scene state across feedback turns.Open-source models and GLM-5V-Turbo have Repair Rates close to zero or negative, indicating that interaction can fail to correct errors or introduce new ones.Source: §3.2 · pp. 7–8
  • Gemini-3.1 Pro achieves the best Single-step Detection and Fix accuracy at 0.721, followed by GPT-5.4 at 0.664 and GPT-5.4 mini at 0.595.Open-source models range from 0.343 to 0.442; Gemini-3.1 Pro and GPT-5.4 lead across all five basic spatial tasks, but foundational skills alone do not predict multi-turn success.Source: §3.2 · p. 8

3.3 Spatial Context and Error Sensitivity

  • Gemini-3.1 Pro repairs more effectively in indoor than architectural scenes, yet architectural scenes have higher Scene Success Rate despite weaker per-turn repair effectiveness.Across Gemini-3.1 Pro, GPT-5.4, and Qwen3.6-35B-A3B, orientation errors are most recoverable and road- or wall-related conflict errors are most challenging; repair rates and scene success decrease as initial error counts rise.Source: §3.3 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 10

3.4 Influence of Context and Task Relationship on VLM Behavior

  • Increasing Kimi-K2.5’s context window produces more applicable content and interaction turns, but Repair Rate and Scene Success Rate remain nearly unchanged.This indicates diminishing returns from context scaling: longer contexts expand deliberation without improving correction quality, which instead depends on control quality such as state tracking and action reliability.Source: §3.4 · p. 9, Figure 6 · p. 10
  • Single-step Error Detection and Fix has the strongest association with multi-turn Repair Rate (r = 0.817) and Scene Success Rate (r = 0.690), but foundational competence does not ensure robust repair.The correlation across all six foundational tasks suggests hierarchical skill integration, while multi-turn success additionally requires cross-turn memory, conflict-aware planning, and stable action sequencing.Source: §3.4 · p. 9, Figure 6 · p. 10

4 Related Work

Prior work evaluates spatial abilities across increasingly realistic, interactive settings and explores 3D layout generation and understanding; SpatialAct focuses on whether VLMs can reason and act in interactive 3D environments.

3D Spatial Intelligence Evaluation

  • Spatial-intelligence evaluation has expanded from hierarchical ability suites, including Spatial-DISE [8] and BSA [31], to realistic, multi-view, and action-oriented benchmarks.SpaceVista [21] covers scenes from millimeters to kilometers, CityCube [30] collects images from different viewpoints, and extends object-centric judgment to multi-step interaction; action-oriented evaluations [25] [11] expect models to modify environments.Source: §4 · p. 10

3D Layout Generation and Understanding

  • 3D layout research covers generation and understanding, but leaves open whether VLMs can directly complete layout tasks through reasoning and action.LayoutGPT [5] uses LLMs for structured layouts, while Holodeck [33] and LayoutVLM [20] combine VLM reasoning with constraints or optimization; understanding work includes scene restoration [28] and scene-graph-based editing [40]. SpatialAct evaluates iterative layout repair or modification, including physical validity concerns such as collisions and out-of-bound placements.Source: §4 · p. 10

5 Conclusion

SpatialAct evaluates whether VLMs can turn spatial reasoning into reliable action under dynamic 3D feedback, combining multi-turn refinement, single-step error correction, and basic spatial abilities. Its results show that strong basic-task performance does not reliably transfer to iterative correction, where current systems remain far less reliable than humans.

5 Conclusion

  • Strong basic spatial-task performance does not consistently transfer to robust multi-turn repair; proprietary models lead open-source systems, but all remain far below human reliability.The conclusion identifies cross-turn state maintenance, constraint-aware planning, and stable reasoning-to-action execution—not perception alone—as core bottlenecks.Source: §5 · pp. 10–11
  • SpatialAct jointly evaluates multi-turn interactive refinement, single-step error detection and fix, and basic spatial abilities to expose performance gaps and their underlying capability structure.It is presented as a simulator-grounded hierarchical benchmark for translating spatial reasoning into reliable action under dynamic 3D feedback.Source: §5 · p. 10
  • Performance varies with scene context, structural constraints, and error complexity; models generally handle indoor scenes and orientation-related corrections more effectively.The benchmark is also an actionable diagnostic framework for improving spatial state tracking, coordination across feedback loops, and context-grounded action generation.Source: §5 · p. 11

A Appendix · A.1 Discussion and Future Work

SpatialAct reveals persistent multi-turn reasoning and action-planning challenges, but its simulated-scene evaluations leave real-world transfer unclear; future work should test real scenes and improve spatial reasoning-to-action.

A.1 Discussion and Future Work

  • Models show persistent multi-turn reasoning and action-planning challenges, with systematic biases across scenes and error types.The authors identify this as a central finding of their results.Source: §A.1 · p. 15
  • Because all evaluations use simulated scenes, how model performance translates to real-world environments remains unclear.Extending evaluation to real-world scenes is identified as future work.Source: §A.1 · p. 15
  • The study did not explore ways to enhance model capabilities; future work includes improving multi-turn reasoning and action execution and developing more robust, human-like spatial reasoning-to-action.Source: §A.1 · p. 15

A.2 Task Subcategories and Prompt Design

The section subdivides Basic Spatial Ability and Single-step Error Detection and Fix tasks for finer-grained diagnosis, with representative prompts provided for each subcategory.

A.2 Task Subcategories and Prompt Design

  • The five Basic Spatial Ability tasks cover Object Meaning, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization, each with several subtypes emphasizing different aspects of 3D spatial understanding and reasoning.Table 3 presents representative prompts for the subcategories.Source: §A.2 · p. 15, Table 3 · p. 16
  • Single-step Error Detection and Fix is divided by error pattern to evaluate whether models identify abnormal spatial configurations and select the corresponding correction.Table 3 also provides representative prompts for these subcategories.Source: §A.2 · p. 15, Table 3 · p. 16

Related papers

Loading 2605.31148v1…