Source-linked AI summary
ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models
Yanpeng Zhao, Wentao Ding, Hongtao Li, Baoxiong Jia, Zilong Zheng
TL;DR
Existing embodied spatial-reasoning evaluations are limited by static VQA paradigms, omitted execution, and costly real-world setups. ESPIRE introduces a physically grounded simulated benchmark that decomposes robotic tasks into generative localization and execution, with systematic task and environment design. Across evaluated VLMs, localization is stronger than execution and orientation reasoning is the main bottleneck, while ESPIRE remains limited to indoor scenes.
Problem
Existing spatial-reasoning evaluations are limited by distractor-based static VQA, omitted execution, and hardware-dependent handcrafted real-world tasks.
Method
ESPIRE uses a physically grounded simulated world, decomposes tasks into generative localization and execution, and systematically varies instructions and environments.
Results
VLMs perform much better in localization than execution, with orientation reasoning the greatest challenge in both stages.
Takeaways & Limitations
ESPIRE provides a scalable, reproducible complement to real-world evaluation for diagnosing spatial reasoning and guiding model improvement.
Takeaways & Limitations
ESPIRE is restricted to indoor scenes and excludes outdoor-only reasoning involving larger scales, larger reference objects, or global reference frames.
Abstract
from arXiv · showhide
A recent trend in vision-language models (VLMs) has been to enhance their spatial cognition for embodied domains. Despite progress, existing evaluations have been limited both in paradigm and in coverage, hindering rapid, iterative model development. To address these limitations, we propose ESPIRE, a diagnostic benchmark for embodied spatial reasoning. ESPIRE offers a simulated world that physically grounds VLMs and evaluates them on spatial-reasoning-centric robotic tasks, thus narrowing the gap between evaluation and real-world deployment. To adapt VLMs to robotic tasks, we decompose each task into localization and execution, and frame both as generative problems, in stark contrast to predominant discriminative evaluations (e.g., via visual-question answering) that rely on distractors and discard execution. This decomposition further enables a fine-grained analysis beyond passive spatial reasoning toward reasoning to act. We systematically design ESPIRE both at the instruction level and at the environment level, ensuring broad coverage of spatial reasoning scenarios. We use ESPIRE to diagnose a range of frontier VLMs and provide in-depth analysis of their spatial reasoning behaviors.
1 INTRODUCTION
ESPIRE addresses limited and poorly scalable evaluations of embodied spatial reasoning by physically grounding VLMs in simulation and unifying localization with execution. Its systematic task design supports fine-grained diagnosis, while experiments reveal stronger localization than execution and orientation as the main challenge.
- Existing evaluations rely mainly on static, distractor-based VQA or costly handcrafted real-world tasks, limiting practical relevance, scalability, and reproducibility.
- ESPIRE is a simulation-based benchmark that evaluates VLMs in physically grounded robotic tasks.
- Each task is decomposed into localization and execution, both framed as generative goal position and goal pose problems.
- The benchmark systematically varies spatial aspects, reference objects, reference frames, and reasoning granularities for fine-grained diagnosis.
- VLMs perform better in localization than execution, while orientation reasoning is the greatest challenge in both stages.
2 RELATED WORK
Related work spans methods for improving VLM spatial intelligence, scalable simulation-based robotics evaluation, and foundation-model manipulation. ESPIRE is positioned within generative 6-DoF approaches that predict and ground goal poses in a simulated physical world.
- Spatial reasoning with vision-language models: Prior VLM research improves spatial intelligence through prompting and by integrating 3D representations such as depth maps and point clouds.
- Simulation-based evaluation through robotic tasks: Simulation-based robotic evaluation offers more scalable and reproducible assessment than human-assisted real-world evaluation.
- Foundation models for robotics manipulation: Foundation-model manipulation research has progressed from task planning with predefined primitives toward trajectory generation for motion planning.
- 6-DoF object rearrangement: 6-DoF object rearrangement predicts an object goal state in SE(3) to support transfer from perception to execution.
- 6-DoF object rearrangement: Generative 6-DoF methods predict goal translation and rotation, whereas discriminative methods filter randomly generated candidates with a critic.
3 SPATIAL-CENTRIC EVALUATION OF EMBODIED VLMS
ESPIRE evaluates embodied spatial reasoning through two sequential generative subtasks: localization identifies the target, and execution predicts a physically achievable 6-DoF goal pose. Shared pointing grounds both understanding and acting in the simulated world.
- ESPIRE evaluates VLM spatial cognition through robotics tasks in a simulated physical world to narrow the gap between evaluation and deployment.
- Localization: Localization identifies an instructed target from a scene and generates its 2D image coordinates as a pointing task.
- Localization: Localization accuracy compares the predicted point directly against the target segmentation mask rather than relying on VQA distractors.
- Execution: Execution follows localization and predicts a pick-or-place goal pose in SE(3), including goal position and orientation.
- Execution: Execution acceptance rate measures the fraction of physically achieved poses assessed by a motion planner.
- Both stages require tool-free 3D position and orientation generation, linking spatial reasoning for understanding with spatial reasoning for acting.
4 THE ESPIRE BENCHMARK
ESPIRE systematically specifies spatial tasks and generates varied tabletop and shelf environments for scalable, fine-grained evaluation. Functional programs control reasoning complexity, while the benchmark covers diverse spatial contexts under a three-hop limit.
- ESPIRE combines systematic instruction and environment design to provide broad spatial coverage, scalable task generation, and targeted analysis across contexts and granularities.
- Task specification: Tasks are characterized by spatial aspect, reference frame, and reference object, whose combination C = (S, F, O) defines a reasoning context.
- Instruction representation: Instructions represent task specification, pick-or-place action, and localization program as T = (C, A, P), with functional programs producing valid scene answers.
- Instruction families: Functional programs enable flexible reasoning complexity by varying the number of reasoning hops, while instruction families provide multiple linguistic templates.
- Instruction families: 148 spatial-reasoning task types are distributed across 65 instruction families, with reasoning limited to at most three hops.
- Simulation environment: Tabletop and shelf scenes use photorealistic objects, varied layouts, lighting, clutter, and multiple granularities to instantiate task specifications.
- Environment representation and generation: Environments begin from randomly sampled 3D scene graphs rendered in Isaac Sim, with annotated object sizes, dimensions, and poses.
- Reducing the real-to-sim visual gaps: Visual diversity in assets, textures, sizes, and backgrounds is used to reduce real-to-sim gaps through a scalable strategy.
5 EXPERIMENTS
ESPIRE evaluates diverse VLMs on physically grounded pick-and-place tasks, measuring localization, execution, and overall success across spatial aspects and difficulty levels. The experiments diagnose execution bottlenecks, especially orientation and 3D rotational reasoning, while examining model behavior and latency.
- Experimental setups: The evaluation includes proprietary, public general-purpose, and spatial-reasoning-enhanced VLMs.Examples include Gemini2.5-Pro, Qwen3-VL, InternVL3, and RoboBrain2.0.
- Experimental setups: ESPIRE evaluates 2,220 tasks comprising 1,095 pick tasks and 1,125 place tasks, with separate attempt limits for localization and execution.The suite allows up to 3 localization attempts and 5 execution attempts, under reflection and non-reflection settings.
- Main results: VLMs generally localize better than they execute, while orientation reasoning is the greatest challenge in both stages.The results identify limited acting-oriented spatial reasoning and weak grounding of 3D rotational geometry as key bottlenecks.
- Main results: Place tasks are harder than pick tasks because placement requires satisfying stricter target-space constraints and predicting a relatively centered pose under occlusion.Pick tasks can use any graspable face and may allow repositioning, whereas place tasks involve partially occupied target spaces and closer-range occlusion.
- Analysis: ESPIRE supports fine-grained behavioral analysis across spatial aspects, task difficulty, successful execution behavior, and qualitative execution categories.The benchmark also examines human-model agreement and reports latency from API calls, inference, and motion planning.
- Analysis: Distance is the weakest localization aspect across pick and place tasks, indicating limited precise distance understanding.Gemini2.5-Pro and RoboBrain2.0-7B show stronger overall localization with smaller performance variation across spatial aspects.
- Analysis: Localization and execution generally decline as scene difficulty increases, although InternVL3-78B, RoboBrain2.0-7B, and Qwen3-VL-30B-A3B show exceptions in some settings.Difficulty is defined by scene complexity, including book count for tabletop tasks and shelf fullness for shelf tasks.
- Ablation of rotation prediction: Rotation ablations indicate that pitch is especially important for pick tasks, roll for constrained place tasks, and execution becomes harder as more axes must be predicted.The analysis connects these axes to grasping-face selection and deviations from upright placement.
6 DISCUSSION AND FUTURE WORK
ESPIRE prioritizes diagnostic isolation of spatial reasoning and supports extensions toward richer, longer-horizon embodied evaluation. Its current scope is indoor scenes, while future directions include multistep reasoning and mobile manipulation.
- Discussion: ESPIRE isolates localization and execution to diagnose where spatial reasoning chains break before integrating reasoning with action.The authors connect this diagnostic design to agentic systems that separate high-level VLM reasoning from controller execution.
- Limitations: ESPIRE is restricted to indoor scenes and omits outdoor reasoning involving larger units, larger reference objects, and global reference frames.The authors note that outdoor extensions could be supported by exposing outdoor reference objects through glass walls.
- Future Work: ESPIRE can be extended to long-horizon multistep tasks and mobile manipulation involving sequential pick and place across different workspaces.These extensions would support analyses of reasoning-step dependencies, memory, and mobile manipulation.
7 CONCLUSION
ESPIRE is a physically grounded simulated evaluation suite for embodied spatial reasoning in VLMs. It unifies passive spatial reasoning and action-oriented reasoning by decomposing robotic tasks into localization and execution.
- Conclusion: ESPIRE evaluates VLMs on robotic tasks in a physically grounded simulated environment to mitigate the gap between evaluation and practical deployment.
- Conclusion: Each task is divided into localization and execution, providing a unified evaluation of passive and action-oriented spatial reasoning.
- Conclusion: ESPIRE is systematically designed to simulate diverse spatial reasoning scenarios.
ABSTRACT
The supplementary material documents ESPIRE’s task definitions, scene setups, sim-to-real relevance, evaluation procedures, and assets. It also provides visualizations and dimensions for the benchmark’s assets.
- Supplementary Material: The supplement details task definitions, including spatial-aspect taxonomy and curated functional programs.
- Supplementary Material: It documents tabletop and shelf scene setups and discusses ESPIRE’s sim-to-real relevance.
- Supplementary Material: It provides evaluation details covering prompting procedures, essential prompts, and evaluation efficiency.
- Supplementary Material: It describes ESPIRE assets with visualizations and dimensions.
A.1 PARTICIPANTS OF A ROBOTICS TASK
ESPIRE defines simple pick and place robotics tasks around manipulable and reference objects, then systematically varies spatial aspects, reference frames, objects, and scene complexity. This design supports automated 6-DoF evaluation while isolating spatial reasoning behaviors.
- Task Definition: ESPIRE uses pick and place as its two primitive actions, with tasks defined by an action and a manipulable object.
- Task Definition: Simple actions help isolate spatial reasoning behaviors for focused analysis.
- Task Specification: Task specifications combine spatial aspects, reference frames, and reference objects to support systematic design and analysis.
- Reference Frames: The benchmark distinguishes relative, intrinsic, and absolute reference frames for describing object positions.Relative frames are viewer-centered, intrinsic frames are object-centered, and absolute frames use fixed global coordinates.
- Objects: Manipulable objects are cuboid-shaped books, while reference objects are oriented or non-oriented and may be near or distant.Regular book geometries facilitate automated verification and support valid grasping or placement poses for 6-DoF tasks.
- Spatial Aspects: Spatial aspects cover attributes, distances, relationships, and orientations, with coarse- and fine-grained expressions such as dimensions, measured distances, positional relations, and clock positions.The task specification is applied consistently across all four aspects, although relationships and orientations depend on reference frames.
A.2 SIMULATED ENVIRONMENT
ESPIRE simulates tabletop and shelf manipulation scenes with varied objects, poses, layouts, and explicit geometric interpretations of spatial language. Its sampling strategy balances task families and difficulty while retaining non-trivial tasks.
- Scene environments: Tabletop scenes vary book sizes and poses, while shelf scenes add above and below relationships through distinct placement layouts.Tabletop books can be small, medium, or large and standing, flat, or tilted; shelf scenes support additional vertical relations.
- Spatial definitions: Spatial language is mapped to unified geometric definitions using reference frames, forward axes, clock-position directions, and tilt angles.The benchmark defines front, left, right, and behind relative to a frame, with orientations represented by clock positions and deviations in degrees.
- Scene construction: Object placement and visibility constraints mitigate ambiguity, although physical rendering can reduce the intended 5cm spacing between tabletop objects.Each tabletop object must initially be spaced at least 5cm apart and have at least 20% of its pixels visible in the global view.
- Task sampling: Balanced task sampling favors underrepresented task families, difficulty levels, and less frequently attempted scenes, retaining only tasks with non-empty answer sets.The procedure updates family, difficulty, and scene-attempt counters during generation and keeps only non-trivial tasks with more than one valid answer.
A.3 SIM-TO-REAL RELEVANCE
ESPIRE’s simulated evaluations are assessed as a proxy for embodied spatial reasoning through benchmark alignment, human solvability, and analysis of evaluation procedures. The evidence also identifies ambiguity around nearby oriented references as an important boundary.
- Performance alignment: ESPIRE model rankings correlate 96.4% with rankings on RefSpatial, supporting its use as a high-fidelity proxy for real-world embodied spatial reasoning.The correlation is Spearman’s rank correlation with p = 0.00498 across six evaluated models.
- Human study: Humans achieve an average success rate of 94.9 ± 3.4% across tasks, indicating that the simulated scenarios are interpretable and solvable by people.The study included five humans and used success across all tasks as the realism and alignment check.
- Human study: Human agreement on the intended reference frame exceeds 97% for distant oriented references and tables but falls to 31.03% for nearby oriented references.The lower agreement for nearby oriented objects is associated with ambiguity in reasoning about their reference frames.
- Evaluation procedures: Localization and execution are evaluated iteratively, optionally adding reflection after failure and obtaining new observations during execution.Localization reflection feeds generated reflection tokens into the next iteration, while execution reflection also updates the scene observation.
- Evaluation procedures: A running example shows a model failing its first goal-position prediction and succeeding on the second attempt after leveraging reflection.The example concerns an end-effector goal position in a pick task.