RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

Motivation

• Existing benchmarks often omit explicit fine-grained spatial evaluation, step-level diagnosis, or controlled difficulty levels, while most widely used datasets contain fewer than 200K trajectories.The introduction notes short-horizon emphasis in LIBERO (Liu et al., 2023a) and RoboTwin 2.0 (Chen et al., 2026a), and limited step-level diagnosis in RoboCasa and MIKASA-Robo (Cherepanov et al., 2026).Source: §1 · pp. 1–2

Design principles

• RoboSPA’s Fine-Grained Spatial Reasoning dimension tests grounding through geometric attributes, distances, cross-view cues, relations, and canonical indexing.The paper describes RoboSPA as the first VLA dataset to make fine-grained spatial reasoning a core evaluation dimension.Source: §1 · p. 2
• Its Long-Horizon Procedural Planning dimension tests multi-step decisions in repetitive, order-constrained or order-free, composite, and memory-intensive tasks.These procedures target temporal consistency and stepwise progress in real-world manipulation.Source: §1 · p. 2
• Multi-Level Hierarchical Evaluation measures performance changes across five difficulty levels and reports step-level progress to diagnose failures beyond task-level success.The levels enable analysis of degradation as spatial complexity and action horizons grow.Source: §1 · p. 2

Benchmark scope

• RoboSPA spans 56 base tasks across 10 capability categories and five difficulty levels, yielding 280 variants and 527K trajectories across five embodiments.It includes 997 hours of videos in clean and domain-randomized scenes, with step-level evaluation beyond final success rates.Source: §1 · p. 2

Results

• On the hardest tasks, all evaluated VLA models average below 25% success, with some tasks at 0%, and fail in grounding, manipulation, tracking, and memory-based reasoning.These results position RoboSPA as a diagnostic testbed for embodied reasoning.Source: §1 · p. 2

Vision-Language-Action Models

• VLA methods span discrete-token action prediction, diffusion or flow-based continuous policies, hierarchical control, cross-embodiment generalization, and spatial, memory, or future-frame guidance.These approaches build on LLM and MLLM capabilities to map visual observations and language instructions to robot actions.Source: §2 · p. 3

Robotic Manipulation Datasets and Benchmarks

• Existing manipulation datasets and benchmarks cover diverse embodiments, skills, and evaluation settings, but rarely assess fine-grained spatial reasoning and long-horizon procedural planning with step-level diagnostics.RoboSPA addresses this gap with tasks designed around these two capabilities.Source: §2 · p. 3

3.1 Overview

• RoboSPA is built in the SAPIEN simulator using the RoboTwin 2.0 framework, with tasks instantiated across difficulty levels and scene settings.The benchmark is designed to evaluate reasoning-oriented VLA models.Source: §3.1 · p. 4

3.2.1 Capability Taxonomy

• The benchmark evaluates fine-grained spatial reasoning across five categories, requiring target selection from complex spatial configurations.Categories cover geometric attributes, relative distance, row-column indexing, directional relations to reference objects, and translating spatial references across viewpoints.Source: §3.2.1 · p. 4
• Long-horizon procedural planning tests extended tasks with diverse actions, multiple subgoals and states, and procedural constraints.Its categories assess repetition and stopping, flexible or ordered subgoal execution, coordination of different action types, and recall of information for later actions.Source: §3.2.1 · pp. 4–5

3.2.2 Task Suite Construction

• Tasks instantiate specific reasoning requirements through expert-coded scenes, procedures, and success conditions, with five task-specific difficulty levels.Spatial tasks increase candidate objects, while procedural tasks add intermediate steps; 60 instruction templates per task use 50 for training and 10 for testing.Source: §3.2.2 · p. 5

3.2.3 Data Collection

• The benchmark collects simulation trajectories across clean and domain-randomized scenes and five robotic embodiments.Domain randomization varies clutter, textures, lighting, and tabletop configurations to support evaluation of task competence and scene robustness.Source: §3.2.3 · p. 5

3.2.4 Evaluation Metrics

• Beyond Success Rate, Object-Normalized Target Accuracy (ONT A) adjusts spatial target accuracy for object-count-induced chance, while Progress Score (PS) captures partial procedural completion.ONT A is 100 for perfect selection, 0 for random guessing, and negative below chance; PS is computed on a 0–100 scale from completed subtasks.Source: §3.2.4 · p. 5

3.3 Benchmark Statistics

• RoboSPA comprises 56 base tasks across two reasoning dimensions, each instantiated at five difficulty levels for 280 task variants.The dataset contains approximately 527K trajectories, over 997 hours of interaction videos, and 108M timesteps.Source: §3.3 · p. 6

4.1 Experimental Setup

• Four representative VLA models were trained and evaluated separately for each base task using cleanscene data and the Aloha-AgileX embodiment.Each model used data from all five difficulty levels and was evaluated independently at each level, with 100 rollout trials per task variant.Source: §4.1 · p. 6

4.2 Main Results

• At L5, π0.5 achieves 22.3% success, ahead of X-VLA at 19.9%, GO-1 at 8.8%, and RDT at 6.9%.Performance declines from L1 to L5; π0.5 and X-VLA each drop by over 30 percentage points. X-VLA falls from 41.9% at L1 to 23.9% at L5 in fine-grained spatial reasoning, while π0.5 falls from 71.2% to 23.1% in long-horizon planning. Models do best on Repetitive Procedure Following, but all fail on Memory-Intensive Planning at the hardest level.Source: §4.2 · p. 7

4.3 Diagnostic Evaluation

• ONT A scores remain low: RDT and GO-1 are negative on most categories and levels, while π0.5 and X-VLA score better but remain near the random-selection baseline.Progress Score is generally higher than final Success Rate, showing that models can partially execute procedures before accumulated errors, incorrect ordering, or incomplete later-stage execution prevent completion; π0.5 has the strongest average completion rate, followed by X-VLA, while MIP remains weak at higher difficulty.Source: §4.3 · pp. 7–8

4.4 Qualitative Results

• Figure 5 reports Success Rate for four baseline VLA models on RoboSPA across 10 capability categories, with results shown separately for L1 and L5.Source: Figure 5 · p. 8

4.5 Failure Analysis

• Observed failures include target grounding and execution errors, intermediate manipulation errors, memory errors, temporal ordering errors, and redundant repetitions.The analysis points to object-centric grounding, relation awareness, reliable low-level execution, progress tracking, memory, and execution monitoring as needed improvements to maintain task state and reduce ordering errors, repetitions, and cascading failures.Source: §4.5 · pp. 8–9

• RoboSPA presents a reasoning-focused dataset and benchmark for robotic manipulation.Source: §5 · p. 9
• Existing VLA models struggle with complex spatial relations, precise execution, and memory-intensive planning, especially at higher difficulty levels.Source: §5 · p. 9
• RoboSPA centers on Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, with ten task categories, five difficulty levels, multiple embodiments, diverse scenes, and fine-grained metrics for spatial grounding and subtask completion.Source: §5 · p. 9

• All tasks are constructed in simulation, so the sim-to-real gap may limit direct transfer to physical robots.Source: Limitations · p. 9
• The tabletop-manipulation focus may not capture the diversity and open-endedness of real-world environments.Source: Limitations · p. 9
• Deformable-object manipulation, human-robot interaction, and open-ended task instructions remain underexplored despite multiple embodiments and domain-randomized scenes.Source: Limitations · p. 9

Ethics Statement

• RoboSPA is designed for simulated manipulation evaluation, not direct real-world deployment; physical transfer should use safety constraints, human oversight, and task-specific validation.Data collection takes place entirely in simulation, without real-world robot deployment.Source: Ethics Statement · p. 9
• RoboSPA uses no human-subject experiments, personal data, or real-world private data, and its reviewed content requires no additional anonymization.The review covered task instructions, object categories, scene assets, and collected demonstrations for identifying or offensive content.Source: Ethics Statement · p. 9
• RoboSPA uses SAPIEN under Apache 2.0 and RoboTwin 2.0 and RMBench code and assets under MIT, while its introduced artifacts will be released under MIT.Baseline models and pretrained checkpoints follow their original license terms, and third-party copyright and license notices are retained.Source: Ethics Statement · p. 9
• The authors acknowledge possible bias and limited transferability from simulator design, task selection, object categories, and embodiment choices, and encourage efficient training and transparent compute reporting.Mitigations include multiple embodiments, diverse task categories, controlled difficulty levels, and clean and domain-randomized scenes; responsible use also considers environmental impact.Source: Ethics Statement · p. 9

Appendix

• The appendix lists §A failure analysis, §B training and evaluation details, §C dataset details, §D basic grounding tasks, and §E task descriptions.These sections provide supplemental detail about RoboSPA.Source: Appendix · p. 13
• The appendix also includes §F detailed experimental results and §G task visualizations.Source: Appendix · p. 13

A.1 Analysis Setup

• The failure analysis samples about 1,120 evaluation videos, taking up to 20 errors per L5 task, and distinguishes three spatial-reasoning error types.The types are target execution failure, no-contact grasp, and incorrect target grounding.Source: §A.1 · p. 13
• Long-horizon planning analysis defines five error types: target grounding, manipulation, memory, temporal ordering, and redundant repetition.Target grounding is shared with the spatial taxonomy; the remaining types capture execution failure, forgotten prior information, out-of-order subtasks, and repeated completed actions.Source: §A.1 · p. 13

A.2 Failure Analysis Results

• π0.5 and X-VLA show broadly similar error patterns, with Target Grounding Errors causing over 70% of spatial-task errors.CPI has the highest proportion of grounding errors, while CVR has a relatively larger share of Target Execution Errors.Source: §A.2 · p. 13, Figure 7 · p. 14, Figure 8 · p. 14
• In long-horizon tasks, Manipulation Errors account for approximately 50% of failures, while other error types occur at roughly similar rates.This overall distribution contrasts with the more concentrated spatial-task errors.Source: §A.2 · p. 14
• Task-specific failures include redundant repetition in RPF, manipulation errors in OFE and CAC, temporal-ordering errors in OCE, and memory errors in MIP.RPF patterns suggest weak progress tracking and reliance on current visual observations; MIP errors reflect difficulty retaining task-relevant information across extended action sequences.Source: §A.2 · pp. 14–15
• These differentiated failure modes make RoboSPA diagnostic and point to needs for stronger spatial grounding, precise execution, progress tracking, and memory mechanisms.The analysis also identifies low-level execution as a challenge across multiple subgoals.Source: §A.2 · pp. 14–15

B Training and Evaluation Details

• Training uses 250 trajectories per base task across five difficulty levels, with 50 of 60 templates per variant used for training and 10 held out.Task-specific descriptions yield 100 training and 100 non-overlapping evaluation instructions per variant; one training instruction is sampled randomly for each trajectory.Source: B Training and Evaluation Details · p. 15
• All four baselines are fully fine-tuned from official pretrained weights with a unified batch size of 16 and 20,000 steps, totaling approximately 1,568 GPU-hours.Experiments use NVIDIA RTX PRO 6000 GPUs across 56 base tasks, following each model’s official implementation and settings apart from RoboSPA data and evaluation tasks.Source: B Training and Evaluation Details · pp. 15–16
• Each task variant is evaluated over 100 test trials with instructions disjoint from training, and task scripts record individual action or subtask completion for step-level assessment.This captures partial progress on long-horizon rollouts that fail to reach the final goal, alongside binary Success Rate.Source: B Training and Evaluation Details · p. 16

C.1 Object Asset Details

• RoboSPA includes 120 object categories and 581 variants sourced from RoboTwin-OD (Chen et al., 2026a) and RMBench (Chen et al., 2026b).The assets span daily-use objects with different appearances, geometries, and semantic types.Source: §C.1 · p. 16

C.2 Multi-Embodiment Setting

• Five embodiments—Aloha-AgileX, ARX-X5, Piper, Franka, and UR5—share task semantics, instructions, object configurations, and success conditions.They differ in morphology, workspace, camera viewpoint, and execution behavior.Source: §C.2 · p. 16
• The shared task setup across embodiments enables evaluation under diverse robotic configurations and tests generalization beyond a single embodiment.Source: §C.2 · p. 16

C.3 Domain-Randomized Scenes

• Domain-randomized trajectories preserve task semantics and success conditions while varying visual and environmental conditions to evaluate robustness beyond clean environments.Source: §C.3 · p. 16
• Scene randomization varies background appearance and tabletop clutter, exposing models to diverse visual contexts and distractor objects.Source: §C.3 · p. 16
• Scene randomization also perturbs table height and lighting, introducing mild geometric variation and changed illumination conditions.Source: §C.3 · p. 16

D Basic Grounding Tasks

• Basic color-based, egocentric-position, and fixed-direction ordinal grounding variants achieve relatively high Success Rates, suggesting they may not expose current VLA reasoning limitations.Unlike the corresponding RoboSPA tasks, these variants omit cross-view transformation or variable indexing directions; RoboSPA includes more diagnostic fine-grained spatial settings.Source: D Basic Grounding Tasks · pp. 16–17, Table 7 · p. 17

E Task List and Descriptions

• Each base task has five difficulty variants: spatial tasks vary candidate-object counts, while procedural-planning tasks vary the number of steps; higher difficulty also brings sharper performance drops.The task list and objectives are provided in Tables 8 and 9.Source: E Task List and Descriptions · p. 17, §F.3 · p. 18

F.1 Statistical Uncertainty

• Bootstrap 95% confidence intervals for L5 results show that the main performance trends remain unchanged after accounting for finite-sample uncertainty.Evaluation episodes use distinct seeds to determine scene and object layout; simulated rollouts are largely deterministic for the same checkpoint and seed.Source: §F.1 · p. 17

F.2 Multi-Embodiment Evaluation

• Across five embodiments, L5 Success Rate is lower than L1 for every evaluated task and embodiment, despite variation in absolute performance.X-VLA was independently trained and evaluated on one representative task from each of RoboSPA’s ten capability categories.Source: §F.2 · pp. 17–18

F.3 Success Rates

• Models perform better with fewer distractors and shorter action sequences, but success-rate drops sharpen on difficult spatial disambiguation, strict ordering, compositional execution, and memory-intensive planning tasks.The detailed per-task Success Rates cover RDT, GO-1, π0.5, and X-VLA across five difficulty levels.Source: §F.3 · p. 18

F.4 Object-Normalized Target Accuracy

• RDT and GO-1 often score below chance on Object-Normalized Target Accuracy, while π0.5 and X-VLA perform better but remain low or near random guessing on many variants.The metric helps distinguish weak target disambiguation from low-level manipulation errors.Source: §F.4 · p. 18

F.5 Progress Score

• RDT and GO-1 usually score below 50 on Progress Score, while π0.5 and X-VLA score higher but decline as difficulty rises.Progress Score measures partial subtask completion, indicating how much of a procedure a model completes before failing.Source: §F.5 · p. 18

Fine-grained spatial reasoning

• Figures 11–13 show fine-grained spatial reasoning task examples from RoboSPA.The visualizations pair rollout frames with task category, task name, and instruction, illustrating scene layouts, target objects, and execution processes.Source: G Task Visualizations · p. 18, Figure 11 · p. 27, Figure 12 · p. 28

Long-horizon procedural planning

• Figures 14–20 show long-horizon procedural planning task examples from RoboSPA.The visualizations use representative rollout frames to illustrate scene layouts, target objects, and the required execution process for the most difficult L5 variant of each task.Source: G Task Visualizations · p. 18, Figure 14 · p. 30, Figure 15 · p. 31