Source-linked AI summary
RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang
TL;DR
Existing robotic manipulation benchmarks provide limited evidence about VLA reasoning under increasing spatial and procedural complexity. RoboSPA addresses this gap with a large-scale, multi-level benchmark and diagnostic evaluation, finding that current models struggle substantially on complex tasks. Its conclusions are bounded by simulated tabletop settings and underexplored real-world interaction scenarios.
Problem
Existing benchmarks offer limited evaluation of fine-grained spatial reasoning, long-horizon step-level behavior, controlled difficulty scaling, and broad coverage as task complexity increases.
Method
RoboSPA builds a benchmark spanning two reasoning dimensions, five difficulty levels, multiple embodiments, diverse scenes, and diagnostic metrics beyond final success.
Results
All evaluated models average below 25% success on the hardest tasks, with failures in target grounding, low-level manipulation, long-horizon tracking, and memory-based reasoning.
Takeaways & Limitations
RoboSPA provides a challenging diagnostic testbed for evaluating embodied reasoning as spatial and procedural complexity increases.
Takeaways & Limitations
The benchmark is entirely simulated and focuses on tabletop manipulation, limiting direct transfer and coverage of broader real-world settings.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce \textbf{RoboSPA} (\textbf{Robo}t \textbf{S}patial-\textbf{P}rocedural \textbf{A}ssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. \texttt{RoboSPA} focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, \texttt{RoboSPA} introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. These results establish \texttt{RoboSPA} as a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available at https://github.com/fanzhenxuan/RoboSPA.
1 Introduction
VLA models perform well mainly in structured, short-horizon settings, while real-world manipulation requires fine-grained spatial reasoning, extended execution, and complexity-scalable reasoning. RoboSPA addresses these gaps with multi-dimensional, multi-level, step-aware evaluation and large-scale data, revealing substantial failures as complexity increases.
- VLA models show encouraging manipulation performance primarily in structured, short-horizon settings, leaving broader embodied capabilities insufficiently assessed.
- Existing benchmarks rarely test fine-grained spatial reasoning, long-horizon step-level behavior, controlled difficulty scaling, or broad dataset coverage.
- RoboSPA introduces a large-scale benchmark organized around fine-grained spatial reasoning and long-horizon procedural planning.
- Each base task spans five difficulty levels, enabling measurement of performance degradation as spatial and procedural complexity increases.
- 527K trajectories across five embodiments and 280 task variants support step-level evaluation beyond final success rates.
- On the hardest tasks, all evaluated models average below 25% success, with failures involving grounding, manipulation, tracking, and memory-based reasoning.
2 Related Work
Prior work advances VLA capabilities and robotic manipulation coverage, but RoboSPA organizes evaluation around spatial and procedural reasoning under controlled difficulty and varied scenes.
- RoboSPA centers evaluation on two dimensions: fine-grained spatial reasoning and long-horizon procedural planning.
- VLA models map visual observations and language instructions to robot actions through unified perception-to-control frameworks.
- Large-scale manipulation datasets expand real-world and cross-embodiment coverage, while benchmark suites support robotic task evaluation.
- Its task design increases difficulty through object counts or action-sequence length and uses domain-randomized scenes with varied layouts and visual conditions.
3 RoboSPA
RoboSPA organizes embodied reasoning evaluation around fine-grained spatial reasoning and long-horizon procedural planning, using hierarchical tasks, diverse scenes, and diagnostic metrics. Its benchmark design varies spatial and procedural complexity across task variants and supports evaluation beyond final success.
- Capability Taxonomy: RoboSPA evaluates two core dimensions: Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning.The taxonomy covers spatial grounding in complex configurations and extended manipulation involving multiple subgoals, states, actions, and constraints.
- Fine-Grained Spatial Reasoning: Fine-Grained Spatial Reasoning tests geometric attributes, distances, canonical indexing, referential relations, and cross-view spatial instructions.Its categories target object selection under complex spatial configurations rather than category-level recognition alone.
- Long-Horizon Procedural Planning: Long-Horizon Procedural Planning includes repetitive, order-free, order-constrained, composite-action, and memory-intensive execution.These categories assess progress tracking, subgoal ordering, heterogeneous skill coordination, and retention of information needed for later actions.
- Task and Difficulty Design: Each task is instantiated across five difficulty levels, increasing candidate objects for spatial tasks and intermediate steps for procedural tasks.This hierarchical design measures performance degradation under increasing reasoning burden.
- Data Collection: RoboSPA collects trajectories in clean and domain-randomized scenes across five embodiments, with diverse instruction templates and expert-designed task procedures.Domain randomization varies clutter, textures, lighting, and tabletop configurations; task code specifies scenes, procedures, and success conditions.
- Evaluation Metrics: Beyond Success Rate, RoboSPA introduces Object-Normalized Target Accuracy for spatial reasoning and Progress Score for partial procedural completion.ONT A removes object-count-induced chance effects, while Progress Score uses completed subtasks to capture partial completion.
4 Experiments
Experiments evaluate four VLA models across RoboSPA’s difficulty levels and diagnostic metrics. Performance declines substantially with increasing complexity, exposing weaknesses in spatial grounding, execution, long-horizon tracking, and memory.
- Experimental Setup: Four representative VLA models are evaluated across RoboSPA tasks, with separate models tested independently at each difficulty level.Training uses cleanscene data from the Aloha-AgileX embodiment, with data spanning all five difficulty levels.
- Main Results: Model performance declines from L1 to L5, indicating limited reasoning ability under complex task settings.The benchmark includes category-wise and difficulty-wise comparisons beyond final success rate.
- Model Comparison: π0.5 reaches 22.3% success at L5, followed by X-VLA at 19.9%, while GO-1 and RDT reach 8.8% and 6.9%, respectively.π0.5 performs best overall, but all models lose substantial performance as difficulty increases.
- Fine-Grained Spatial Reasoning: X-VLA’s spatial-reasoning success rate falls from 41.9% at L1 to 23.9% at L5, with particular difficulty in CPI and RRR.The results indicate weakening target grounding when selection depends on fine-grained spatial relations.
- Long-Horizon Procedural Planning: π0.5’s procedural-planning performance decreases from 71.2% at L1 to 23.1% at L5, and all models fail on hardest MIP tasks.Models perform best on repetitive procedure following, while other planning categories are substantially weaker.
- Diagnostic Metrics: Object-Normalized Target Accuracy remains low, with several models near the random-selection baseline on fine-grained spatial tasks.RDT and GO-1 obtain negative ONT A on most categories and difficulty levels, while π0.5 and X-VLA perform better but remain close to random selection.
- Diagnostic Metrics: Progress Score generally exceeds final Success Rate, showing that models often complete part of a procedure before accumulated errors, ordering mistakes, or incomplete later execution.π0.5 has the strongest average completion rate, but MIP remains weak at higher difficulty levels.
- Failure Analysis: Failure analysis distinguishes target grounding, target execution, manipulation, memory, temporal ordering, and redundant repetition errors.These categories separate failures in spatial reasoning from failures during extended procedural execution.
5 Conclusion
RoboSPA is a reasoning-focused robotic manipulation dataset and benchmark centered on fine-grained spatial reasoning and long-horizon procedural planning. Its experiments show that existing VLA models remain weak on complex spatial relations, precise execution, and memory-intensive planning, especially at higher difficulty levels.
- Conclusion: RoboSPA covers ten task categories, five difficulty levels, multiple embodiments, diverse scenes, and fine-grained metrics for spatial grounding and subtask completion.The benchmark is designed around fine-grained spatial reasoning and long-horizon procedural planning.
- Conclusion: Existing VLA models struggle with complex spatial relations, precise execution, and memory-intensive planning, particularly at higher difficulty levels.These results motivate RoboSPA as a benchmark for developing more capable and reliable embodied agents.
Limitations
RoboSPA’s evaluation scope is limited by simulation, tabletop manipulation, and a restricted set of real-world task settings.
- Scope: All tasks are constructed in simulation, so the sim-to-real gap may limit direct transfer to physical robots.This constrains how directly benchmark performance can be interpreted for physical deployment.
- Scope: The benchmark focuses on tabletop manipulation and may not capture the diversity and open-endedness of real-world environments.Broader settings such as deformable objects, human-robot interaction, and open-ended instructions remain underexplored.
Ethics Statement
RoboSPA is a simulation-based benchmark rather than a directly deployable real-world robotic system, and its development does not involve human subjects or personal data. The authors acknowledge transferability risks from simulator and task design choices and recommend safety constraints, oversight, and validation for physical deployment.
- Scope and Safety: RoboSPA evaluates VLA models in simulated robotic manipulation environments, not through direct real-world robot deployment.Any transfer to physical environments should include appropriate safety constraints, human oversight, and task-specific validation.
- Data Content and Privacy: The dataset contains no human-subject experiments, personal data, or real-world private data, so no additional anonymization is required.The authors also checked instructions, object categories, scene assets, and demonstrations for identifying or offensive content.
- Supplementary Materials: The appendix provides detailed failure analysis, training and evaluation details, dataset information, task descriptions, experimental results, and visualizations.These materials are organized across Sections A–G.
- Analysis Setup: The benchmark’s failure analysis distinguishes target selection, execution, contact, repetition, memory, manipulation, and temporal-ordering errors.These categories support diagnosis across fine-grained spatial reasoning and long-horizon procedural planning.
A.2 Failure Analysis Results
Failure analysis shows that RoboSPA exposes distinct error patterns across spatial and procedural tasks. Spatial failures are dominated by target grounding, whereas procedural failures are more varied and reflect manipulation, progress tracking, ordering, and memory demands.
- Summary: The π0.5 and X-VLA models show broadly similar error patterns across both capability dimensions, although their exact proportions vary.The task-specific distributions support RoboSPA’s use as a diagnostic benchmark rather than only a binary success test.
- Fine-Grained Spatial Reasoning: Over 70% of fine-grained spatial reasoning errors are Target Grounding Errors, while No-Contact Grasp Errors are rare.Canonical Position Indexing has the highest grounding-error proportion, whereas Cross-View Reasoning has relatively more Target Execution Errors.
- Long-Horizon Procedural Planning: Approximately 50% of long-horizon procedural planning errors are Manipulation Errors, with other error types occurring at roughly similar rates.The distribution is therefore more diverse than in fine-grained spatial reasoning.
- Long-Horizon Procedural Planning: Repetitive Procedure Following is characterized by Redundant Repetition Errors, indicating difficulty tracking task progress from current observations.Order-Free Execution and Composite Action Coordination are dominated by Manipulation Errors, while Order-Constrained Execution also shows many Temporal Ordering Errors.
- Long-Horizon Procedural Planning: Memory-Intensive Planning is primarily affected by Memory Errors, reflecting difficulty retaining task-relevant information across extended action sequences.This pattern directly links the task’s memory demand to a distinct failure mode.
C Dataset Details
RoboSPA broadens dataset coverage through diverse object assets, five robotic embodiments, and domain-randomized scenes. These design choices preserve task semantics while testing robustness across morphology, viewpoints, execution behavior, and environmental variation.
- Object Assets: RoboSPA includes 120 object categories and 581 object variants spanning diverse appearances, geometries, and semantic types.The assets instantiate scenes for both fine-grained spatial reasoning and long-horizon procedural planning.
- Robotic Embodiments: Five embodiments—Aloha-AgileX, ARX-X5, Piper, Franka, and UR5—share task semantics, instructions, object configurations, and success conditions.They differ in morphology, workspace, camera viewpoint, and execution behavior, enabling evaluation beyond one embodiment.
- Domain Randomization: Domain-randomized scenes vary visual and environmental conditions while preserving task semantics and success conditions.The variations include background appearance, tabletop clutter, table height, and lighting conditions.
D Basic Grounding Tasks
Basic grounding variants provide lower-level spatial references, but their relatively high success rates do not expose the reasoning limitations targeted by RoboSPA. The benchmark instead uses more complex spatial layouts and increasing candidate counts to create a stronger diagnostic test.
- Basic Grounding Variants: The three basic grounding variants are Color-Based Grounding, Egocentric Position Grounding, and Simple Ordinal Indexing.They correspond to SDE, CVR, and CPI object categories and candidate counts, respectively.
- Basic Grounding Variants: Simple Ordinal Indexing fixes counting directions to left-to-right and far-to-near, unlike Canonical Position Indexing, whose indexing directions vary.Egocentric Position Grounding also avoids the cross-view transformation used by Cross-View Reasoning.
- Results: Relatively high Success Rates across difficulty levels indicate that basic grounding tasks may be insufficient to expose current VLA reasoning limitations.This conclusion is reported for the variants summarized in Table 7.
- RoboSPA Spatial Tasks: RoboSPA instead evaluates distance estimation, cross-view reasoning, relational grounding, and canonical position indexing in more complex spatial layouts.These settings are intended to provide a more challenging and diagnostic evaluation of spatial reasoning.
- Hierarchical Difficulty Design: Each base task has five difficulty variants, with spatial tasks increasing candidate objects and procedural tasks increasing the number of procedural steps.This task-specific scaling increases the reasoning burden in a controlled way.
F.2 Multi-Embodiment Evaluation
Multi-embodiment and detailed per-task evaluations show that RoboSPA preserves a consistent difficulty trend while exposing failures in spatial grounding and long-horizon planning.
- Multi-Embodiment Evaluation: Across five embodiments, L5 success rates are lower than L1 success rates for every evaluated task and embodiment.This indicates a consistent difficulty trend across robotic embodiments.
- Per-Task Success Rates: Detailed results show better performance with fewer distractors and shorter action sequences, but sharper declines at higher difficulty.The largest drops occur in fine-grained spatial disambiguation, strict ordering, compositional execution, and memory-intensive planning.
- Fine-Grained Spatial Reasoning: ONT A results show poor spatial grounding for RDT and GO-1, while π0.5 and X-VLA perform better but remain near random-guessing on many variants.ONT A is used to distinguish target-disambiguation failures from low-level manipulation errors.
- Long-Horizon Procedural Planning: Progress Scores show consistently low long-horizon performance for RDT and GO-1, with most tasks below 50, while π0.5 and X-VLA complete more intermediate subtasks.Even the stronger models’ scores decline as difficulty increases.
G Task Visualizations
RoboSPA visualizations present the most difficult task variants alongside task metadata, while tables organize task definitions and model results across difficulty levels.
- Task Definitions: Tables 8 and 9 list task descriptions for five fine-grained spatial and five long-horizon procedural categories, respectively.The spatial categories include GAC, SDE, CPI, RRR, and CVR; the procedural categories include RPF, OFE, OCE, CAC, and MIP.
- Fine-Grained Spatial Tasks: Example spatial tasks require ranking or selecting objects by attributes, distance, canonical position, and referential relations.Examples include selecting the third-tallest block, the farthest mug, or an object at a specified row and position.
- Long-Horizon Procedural Tasks: Example procedural tasks require repetition, sorting, strict action order, stacking, and coordinated multi-object operations.Examples include pressing or lifting objects five times, ordering blocks, stacking by color or size, and moving or placing multiple objects in sequence.