Source-linked AI summary
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
Tianhui Liu, Jie Feng, Zhiheng Zheng, Shengyuan Wang, Yiming Guo, Yanxin Xi, Hangyu Fan, Yong Li, Pan Hui
TL;DR
VLMs perform promisingly on observation-conditioned spatial tasks, but it remains unclear whether they can maintain coherent spatial understanding and act reliably as their actions change 3D environments. SpatialAct evaluates this capability through simulator-grounded, hierarchical tasks, finding a clear gap between isolated spatial reasoning and multi-turn refinement, with current systems substantially below humans. The study identifies robust spatial state tracking and reasoning-to-action execution under feedback as central challenges, while noting that all evaluations remain simulated.
Problem
Existing evaluations provide limited evidence about whether VLMs can maintain spatial beliefs, act on them, and adapt reasoning after their actions change an environment.
Method
SpatialAct is a simulator-grounded hierarchical benchmark that evaluates basic spatial abilities, single-step error fixing, and multi-turn interactive refinement using executable high-level actions in 3D scenes.
Results
Strong VLMs reach around 80% accuracy on several basic categories, but the strongest achieves 0.411 Repair Rate and 0.206 Scene Success Rate in multi-turn refinement, versus humans’ 0.911 and 0.763.
Takeaways & Limitations
Strong isolated spatial reasoning does not consistently transfer to robust multi-turn repair, indicating bottlenecks in cross-turn state maintenance, constraint-aware planning, and stable reasoning-to-action execution.
Takeaways & Limitations
All evaluations use simulated scenes, so how performance translates to real-world environments remains unclear.
Abstract
from arXiv · showhide
Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce \textbf{SpatialAct}, a simulator-grounded benchmark for probing \textit{action-conditioned spatial reasoning} in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear reasoning-to-action gap: current VLMs can perform well on isolated spatial reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.
1 Introduction
SpatialAct addresses the gap between passive spatial question answering and full embodied control by evaluating whether VLM agents can reason about action-induced changes in 3D scenes. Its hierarchical benchmark reveals that strong isolated spatial reasoning does not reliably translate into multi-turn spatial repair.
- Motivation: Existing benchmarks mainly evaluate VLMs as observers, while embodied benchmarks often entangle spatial reasoning with low-level control, leaving action-conditioned spatial reasoning underexplored.The missing middle ground concerns high-level actions that alter the environment and require subsequent reasoning about the updated state.
- Problem formulation: Action-conditioned spatial reasoning requires models to reason about the current spatial state, how their actions change it, and how future decisions should adapt.SpatialAct instantiates this capability as interactive 3D layout refinement with executable actions and objectively verifiable state changes.
- Benchmark design: SpatialAct uses a simulator-grounded, three-level hierarchy spanning five basic spatial abilities, single-step error fixing, and multi-turn interactive refinement.The benchmark covers 333 scenes and 4,355 QA pairs across Abstract Geometric, Urban Architectural, and Indoor Scene scenarios, with high-level commands executed in a simulator and updated multi-view renderings returned to the model.
- Results: Around 80% accuracy on several basic categories contrasts with the strongest VLM’s 0.411 Repair Rate and 0.206 Scene Success Rate in multi-turn refinement, versus humans’ 0.911 and 0.763.The results expose a reasoning-to-action gap despite strong performance on isolated spatial tasks.
- Results: Current VLMs may recognize local spatial relations but struggle to maintain coherent spatial beliefs and produce reliable actions across long-horizon state transitions.The benchmark is intended to diagnose this gap between basic spatial understanding and stable multi-turn spatial refinement.
2 Methods
SpatialAct constructs a simulator-grounded benchmark spanning controlled geometric and daily-life 3D scenes, with hierarchical tasks from basic spatial abilities to multi-turn interactive refinement. Its evaluation environment supports executable object manipulation, iterative feedback, human interaction, and metrics for repair accuracy, efficiency, stopping behavior, and reasoning cost.
- Benchmark Construction: SpatialAct contains 4,355 QA pairs across Abstract Geometric, Urban Architectural, and Indoor Scene scenarios, with scenes represented through top-view and isometric-view renderings.The benchmark includes 333 scenes for the latter two scenario types and spans three question formats.
- Benchmark Construction: The benchmark pipeline combines procedural or sourced scene construction with quality control that removes abnormal spatial configurations before QA generation.Abstract Geometric scenes use predefined procedural constraints, while Urban Architectural and Indoor Scene layouts undergo programmatic cleaning.
- Task Design: Multi-turn Interactive Refinement requires models to inspect scenes, identify spatial errors, issue corrective actions, and adapt decisions from updated renderings.Targeted errors include collision, road conflict, and orientation problems in urban scenes, alongside other abnormal configurations in 3D layouts.
- Evaluation Environment Setup: The evaluation interface supports selecting objects and editing position, rotation, and scale through manipulation tools, transform controls, and an action history.The web-based platform is also used for human evaluation of interactive scene refinement.
- Metrics: Basic Spatial Abilities and Single-step Error Detection and Fix use accuracy, while multi-turn refinement is evaluated with repair-accuracy and efficiency metrics.Repair Rate measures error reduction; Scene Success Rate measures complete scene correction; Effective Repair Turn Ratio measures effective actions; Premature Stop Rate and Average Completion Tokens capture stopping and reasoning cost.
- Metrics: Effective Repair Turn Ratio measures the proportion of interaction turns that reduce scene errors, whereas Premature Stop Rate measures scenes stopped with errors remaining.Average Completion Tokens per Scene measures the model’s token cost across interaction turns.
3 Experiments
Experiments show that VLMs remain substantially weaker than humans at multi-turn spatial repair, especially when errors require constraint-aware coordination and consistent state tracking. More context increases reasoning activity but does not reliably improve repair outcomes.
- Seven VLMs were evaluated under a unified setting, with Multi-turn Interactive Refinement capped at 30 iterative turns.
- Gemini-3.1 Pro led multi-turn refinement with a Repair Rate of 0.411 and Scene Success Rate of 0.206, while weaker models often failed to reduce errors.Some open-source models and GLM-5V-Turbo had Repair Rates close to zero or negative, indicating that interaction could introduce additional errors.
- Human Baseline: Humans achieved a Repair Rate of 0.911 and Scene Success Rate of 0.763, substantially exceeding the best-performing model.The repair rate exceeded that of the best-performing model by 50 percentage points.
- Multi-turn interactive spatial correction remains a major bottleneck because models struggle with consistent spatial state tracking and coupled diagnosis-to-action failures.The reported gap persists despite models performing better on many static or single-step spatial tasks.
- Error Sensitivity: Orientation errors were most recoverable, whereas road- and wall-related conflict errors were hardest because they require coordinated, constraint-level reasoning across multiple objects.Incorrect steps during conflict repair can propagate new violations across turns.
- Context Effects: Increasing context length produced more applicable content and interaction turns, but Repair Rate and Scene Success Rate remained nearly unchanged.The findings indicate diminishing returns from context scaling, with the main limitation attributed to cross-turn control quality rather than token budget.
- Influence of Context and Task Relationship: Figure 6 compares performance across initial scene complexities, context-window settings, and correlations between basic spatial abilities and interactive tasks.
4 Related Work
Spatial intelligence evaluation has expanded from classical spatial tasks toward richer 3D, embodied, and active settings. These benchmarks assess how VLMs perceive, reason about, and act upon spatial information from multiple perspectives.
- Recent benchmarks evaluate spatial intelligence through hierarchical suites of classical tasks and fundamental spatial abilities.Examples include Spatial-DISE and BSA.
5 Conclusion
SpatialAct evaluates whether VLMs can translate spatial reasoning into reliable actions under dynamic 3D feedback. Results show persistent gaps in iterative correction, especially in state maintenance, planning, and reasoning-to-action execution.
- SpatialAct jointly evaluates multi-turn refinement, single-step error correction, and basic spatial abilities to diagnose VLM capability structure.
- Strong basic spatial-task performance does not consistently transfer to robust multi-turn repair.
- The main bottlenecks are cross-turn state maintenance, constraint-aware planning, and stable reasoning-to-action execution rather than perception alone.
- Proprietary models outperform open-source models, yet all current systems remain far below human reliability on iterative spatial correction.
- Performance varies systematically with scene context, structural constraints, and error complexity, with indoor and orientation-related corrections generally handled more effectively.
A.1 Discussion and Future Work
The evaluation is limited to simulated scenes, leaving the transfer of results to real-world environments unresolved. Future work also includes improving multi-turn reasoning and action execution.
- All evaluations occur in simulated scenes, so translation of performance to real-world environments remains unclear.
- Models show persistent challenges in multi-turn reasoning and action planning, including systematic biases across scenes and error types.
- The paper does not explore methods for enhancing model capabilities on the task.
A.2 Task Subcategories and Prompt Design
The benchmark uses fine-grained subcategories to diagnose spatial reasoning performance. Basic Spatial Ability tasks span five categories, while Single-step Error Detection and Fix is also subdivided into multiple subcategories.
- Basic Spatial Ability tasks are divided into Object Meaning, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization.
- Each basic-ability category contains subtypes targeting different aspects of 3D spatial understanding and reasoning.
- Single-step Error Detection and Fix is further divided into multiple subcategories for fine-grained performance diagnosis.
- Table 3 presents subcategories and representative prompts for both task groups.