PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, Masayoshi Tomizuka

arXiv:2607.04434v3cs.ROcs.AIcs.CVcs.GR

TL;DR

Robot policy evaluation remains split between scalable simulation that misses physical factors and costly, hard-to-reproduce real-world testing. RoboDojo combines 42 simulation tasks across five capability dimensions with 18 real-world tasks and standardized physical evaluation. Evaluated policies show incomplete capability coverage in simulation and poor performance on complex real-world tasks.

  • Problem

    Robot policy evaluation separates scalable simulation, which misses physical deployment factors, from real-world testing, which is costly and difficult to reproduce.

    Source: §1 · p. 3

  • Method

    RoboDojo unifies 42 simulation tasks across five capability dimensions and 18 real-world tasks with scalable simulation and standardized physical evaluation.

    Source: §1 · p. 3, §8 · p. 20

  • Results

    Evaluated policies show incomplete capability coverage across five manipulation dimensions in simulation and poor performance on complex real-world tasks.

    Source: §8 · p. 21

  • Takeaways & Limitations

    RoboDojo supports rapid policy iteration and assessment of whether improvements extend to challenging physical deployment conditions.

    Source: §1 · p. 4

    Simulation and real-world tasks are not paired, so RoboDojo does not measure direct sim-to-real transfer on matched tasks.

    Source: §3.2.1 · p. 8

Abstract

from arXiv · show

Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.

5 Dimensions Real-World

RoboDojo combines simulation evaluation across five capability dimensions with reproducible real-world testing.

Five Dimensions

  • The five evaluation dimensions cover generalization, memory, long-horizon execution, precision, and open-vocabulary instruction following.Source: §5 · p. 1

Real-World Testing

  • Real-world testing addresses physical-world challenges, with reproducibility covering layout, lighting, robot setup, and hardware.Source: §5 · p. 1

Unified Evaluation

  • 42 simulation tasks and 18 real-world tasks form RoboDojo’s unified evaluation benchmark for generalist robot manipulation.The framework includes heterogeneous parallel simulation, RoboDojo-RealEval, XPolicyLab, and a continuously updated leaderboard.Source: Figure 1 · p. 1

1. Introduction

RoboDojo unifies scalable simulation and standardized real-world evaluation of generalist robot manipulation policies. Its capability-focused tasks, shared infrastructure, and diagnostic analyses support comprehensive, reproducible assessment across both settings.

1. Introduction

  • Existing benchmarks often vary objects, layouts, or language while retaining similar manipulation patterns, limiting diagnostic coverage beyond simple or short-horizon tasks.Undercovered challenges include generalization, memory, precision, open-semantic grounding, sequential execution, bimanual coordination, and tool use.Source: §1 · p. 3
  • Simulation offers efficient, scalable feedback but misses physical deployment factors, while real-world evaluation is costly, time-consuming, and difficult to reproduce without standardization.This separation motivates combining scalable simulation-based diagnosis with reproducible real-world validation.Source: §1 · p. 3
  • RoboDojo combines simulation-based iteration, standardized physical validation, continuously updated test cases, and a public leaderboard to diagnose policy limitations and track progress.Extensive policy evaluations underpin systematic analysis across simulation and real-world settings; physical testing examines whether improvements transfer to challenging deployment conditions.Source: §1 · p. 4
  • A shared policy interface and evaluation pipeline combine heterogeneous parallel simulation with standardized, reproducible remote physical testing through RoboDojo-RealEval.Isaac Sim runs different tasks and scenes concurrently. RoboDojo-RealEval standardizes hardware, workspace layout, lighting, scene resets, evaluation protocols, and deployment interfaces.Source: §1 · pp. 3–4
  • 42 simulation tasks and 18 real-world tasks cover five simulation capability dimensions and physical deployment across three robot embodiments.Simulation covers Generalization, Memory, Long-Horizon, Precision, and Open. Physical testing includes contact-rich interactions, perception noise, actuation errors, and environmental variations.Source: §1 · p. 4
  • 30 robot manipulation models are integrated into XPolicyLab, enabling policy development, deployment, and evaluation across RoboDojo with minimal policy-side adaptation.Policies can be integrated once for evaluation in both simulation and real-world settings.Source: §1 · p. 4

2. Related Work

RoboDojo responds to policies’ robustness limitations and specialized evaluation benchmarks by combining capability-oriented simulation diagnosis with standardized, reproducible real-world validation.

2.1. Robot Learning in Manipulation

  • Vision-language-action policies, including π0.5 (Intelligence et al., 2025), advance open-vocabulary instruction following, cross-task generalization, and cross-embodiment transfer.This development follows demonstration-based imitation learning. Parallel research targets memory-augmented manipulation (Shi et al., 2025; Torne et al., 2026), dynamic manipulation (Cai et al., 2026b), and world-model-based control (Intelligence et al., 2026a; Bi et al., 2026a; Ye et al., 2026c).Source: §2.1 · p. 4
  • Policies remain vulnerable to temporal reasoning failures, weak spatial precision, limited generalization, and physical-world variation, motivating structured evaluation beyond aggregate task success.RoboDojo addresses these limitations through capability-oriented simulation tasks and standardized real-world tests.Source: §2.1 · p. 4

2.2. Evaluation for Robot Manipulation

  • RoboDojo unifies capability-oriented simulation diagnosis and reproducible real-world validation through a shared policy interface and evaluation pipeline, with policy integration supported by XPolicyLab.Simulation cannot fully capture physical deployment uncertainties, while variation in hardware, scene resets, evaluation protocols, and deployment interfaces complicates real-world reproducibility.Source: §2.2 · p. 5
  • Existing benchmarks provide scalable simulation testbeds and physical deployment evaluations, but often specialize in particular task families, capability axes, or environments.Examples include language-conditioned long-horizon CALVIN (Mees et al., 2022), memory-dependent RMBench (Chen et al., 2026b), and real-world RoboArena (Atreya et al., 2025).Source: §2.2 · pp. 4–5

3. RoboDojo Benchmark

RoboDojo combines 42 simulation tasks for capability-oriented diagnosis with 18 real-world tasks for complementary physical-deployment evaluation. It provides training data, standardized evaluation protocols, and verification requirements for official leaderboard publication.

3.1. Simulation Benchmark

  • 42 simulation tasks evaluate Generalization, Memory, Long-Horizon, Precision, and Open capabilities on the ARX X5 bimanual platform.Generalization comprises 12 tasks with unseen visual variations and up to 25 clutter objects, versus at most 10 in RoboTwin 2.0 (Chen et al., 2025a). Memory comprises six partially observable tasks requiring key-frame retention or reasoning over longer observation sequences.Source: §3.1 · p. 5, §3 · p. 5, §3.1.1 · p. 5

3.1.1. Task Design

  • Eight tasks each test Long-Horizon execution, Precision, and Open instruction following through dependent steps, fine-grained control, and unseen task specifications.Long-Horizon tasks require maintaining progress without early termination or accumulated errors; Precision tasks demand accurate localization, smooth trajectories, and stable contact-rich control. Open tasks test language grounding and recombination of skills encountered during training in different contexts.Source: §3.1.1 · pp. 6–7

3.1.2. Training Data Setting

  • 3,500 simulation-training trajectories cover 34 training tasks and one auxiliary DLC directory; Open tasks are excluded to evaluate transfer rather than task-specific imitation.The dataset contains 1,859,602 frames and 20.66 hours at 25 Hz, with 100 trajectories per training task collected through automated synthesis or VR teleoperation. Generalization demonstrations use standard scenes; 100 evaluation-task-independent DLC trajectories add randomized backgrounds, lighting, and clutter without leaking task-specific solutions.Source: §3.1.2 · p. 7

3.1.3. Evaluation Setting

  • 2,100 simulation episodes evaluate all 42 tasks, reporting binary success rate and average score for partial progress.Each task receives 50 episodes; Generalization splits these into 25 standard and 25 randomized episodes covering backgrounds, clutter, objects, and lighting. Overall performance averages the five capability dimensions equally, preventing larger dimensions from dominating.Source: §3.1.3 · p. 7

3.2. Real-World Benchmark

  • 18 real-world tasks span ARX X5, Piper, and Piper X, testing deployment reliability across differing kinematics, workspaces, cameras, and data distributions.Six tasks per embodiment form a compact suite covering representative capabilities and physical challenges, rather than matched simulation-to-real transfer tests. Open-source RoboDojo-RealEval standardizes hardware, workspace, lighting, resets, evaluation, and deployment interfaces, with cloud access for shared physical testing.Source: §3.2 · p. 7, §3.2.1 · p. 8

3.2.2. Training Data Setting

  • 1,800 real-world demonstration trajectories contain 1,611,841 frames and 17.91 hours of bimanual manipulation recorded at 25 Hz.Each task has 100 demonstrations from four operators using embodiment-matched leader-follower teleoperation to align command spaces and reduce operator-specific bias. Demonstrations include robot states, language, and synchronized head-camera and two wrist-camera RGB observations, with manual checks for completion, quality, and reset-protocol consistency.Source: §3.2.2 · pp. 8–9

3.2.3. Evaluation Setting

  • 10 trials per task evaluate each real-world policy using replayed initial layouts and three independent evaluators under a double-blind protocol.Local or cloud-based evaluation scores final success and intermediate completion, averaging the three evaluator scores for each trial. Leaderboard videos and scores are released, with an appeal mechanism for scoring errors.Source: §3.2.3 · p. 9

3.3. Evaluation Integrity and Anti-Gaming Protocols

  • Verified leaderboard publication requires official evaluation, hidden-layout verification, released reproducibility artifacts through XPolicyLab, and evaluation videos; private remote development does not require disclosure.Public layouts are primary and hidden layouts auxiliary; submissions report simulation mean and standard deviation over three seeds and cover all three real embodiments. Non-profit AI MMLab Club maintains the leaderboard with academic evaluation partners and no commercial participation; results lacking required artifacts are reported separately as unverified.Source: §3.3 · p. 9

4. Technical Implementation

RoboDojo combines a simulation platform, standardized real-world evaluation, and XPolicyLab to support efficient policy iteration. These components allow policies to be integrated once and deployed to physical evaluation with minimal policy-side adaptation.

4.1. Simulation Platform

  • The simulation platform builds on NVIDIA Isaac Sim and Isaac Lab (Mittal et al., 2025) for task construction, scalable evaluation, and demonstration collection.Its three components are a configuration-driven simulation stack, a physically grounded digital asset library, and heterogeneous parallel execution.Source: §4.1 · p. 10

4.1.1. Simulation Platform Setup

  • Modular YAML specifications decouple task definitions from simulator execution, allowing tasks with different assets, layouts, and reset distributions to share one runtime.The stack inherits MagicSim’s modular manager architecture and scene-construction infrastructure (Lu et al., 2026). Deterministically seeded resets vary objects, poses, articulation states, clutter, lighting, and textures for diverse but reproducible scenes.Source: §4.1.1 · p. 10

4.1.2. Physically Grounded Digital Asset Library

  • A physically grounded asset library supplies rigid, articulated, and deformable task objects and distractors with semantic annotations and manipulation affordances.Metadata includes categories, language descriptions, placement regions, and success-checking annotations. Articulated and deformable assets are standardized for kinematic structure, collision geometry, material parameters, and simulation-ready representations.Source: §4.1.2 · p. 10, §4.1 · p. 10

4.1.3. Heterogeneous Parallelism

  • Heterogeneous parallel simulation steps independently configured environments through a shared vectorized interface, improving evaluation speed while preserving scene-level diversity.Parallel environments can differ in object categories, geometries, distractor counts, articulation structures, and layouts, rather than sharing a homogeneous scene template.Source: §4.1.3 · pp. 10–11

4.1.4. Simulation Data Collection

  • Simulation demonstrations come from automated skill composition or VR teleoperation, both grounded in shared annotations of manipulation affordances and task semantics.Automated synthesis executes reusable skills with the cuRobo v2 motion planner (Sundaralingam et al., 2026). VR teleoperation covers tasks that are difficult to synthesize automatically.Source: §4.1.4 · p. 11

4.2. Real-World Platform: RoboDojo-RealEval

  • RoboDojo-RealEval standardizes physical geometry, lighting, scene resets, and evaluation controls to reduce uncontrolled variation across real-world trials.Fixed robot and camera mounts and transparent target-layout overlays support consistent initial conditions; a touchscreen interface handles execution, emergency stops, video collection, and cloud scoring. Local and remote evaluation support ARX X5, Piper, and Piper X collaborative bimanual embodiments.Source: §4.2 · p. 12

4.3. XPolicyLab

  • 30 representative robot policy models share XPolicyLab’s codebase, which standardizes external workflows and observation-action interfaces while preserving each policy’s internal architecture.Common tools cover data conversion, training, deployment, and evaluation; the policy interface supports observation updates, action prediction, resets, and batched queries. The same implementation connects to simulation and remote real-world evaluation with minimal policy-side adaptation.Source: §4 · p. 10, §4.3 · p. 12

5. RoboDojo Leaderboard

RoboDojo’s leaderboards standardize policy comparisons across simulation and real-world settings, combining five-dimensional capability evaluation with physical deployment assessment across multiple robot embodiments.

5.1. Simulation Benchmark Leaderboard

  • 30 representative policies are integrated, trained, and evaluated through XPolicyLab on the simulation leaderboard, with three expert teleoperators providing human references.Each expert has more than 1,000 hours of simulation teleoperation experience and helped construct the benchmark dataset. Their VR-based evaluation uses the same success criteria and execution horizons as policy evaluation.Source: §5.1 · p. 12
  • 150 trials per task are used for most simulation policies, spanning three training seeds with 50 evaluation trials per seed.Generalization splits each seed’s trials into 25 standard and 25 randomized trials. Reports include mean success rate and mean score with corresponding standard deviations per capability dimension; overall metrics average the five dimensions.Source: §5.1 · p. 12

5.2. Real-World Benchmark Leaderboard

  • 10 policies undergo real-world evaluation across 18 tasks and three embodiments, totaling 180 trials per policy.Each embodiment uses one training seed, with 10 trials per task; RoboDojo-RealEval standardizes resets, protocols, and deployment interfaces, reporting task success rate and score aggregated across tasks. Human references have more than 1,500 teleoperation hours, helped collect the dataset, and use the same homogeneous leader-follower setup and task constraints as policies.Source: §5.2 · p. 13

6. Experiments

RoboDojo exposes low task completion, uneven simulation capabilities, and deployment-specific instability in current manipulation policies. Its evaluation infrastructure improves throughput and produces stable aggregate results, although individual real-world tasks can show greater variability.

6.2. Analysis of RoboDojo Real-World Performance

  • 12.8% overall success rate and a 22.9 score for real-world leader π0.5 contrast with human teleoperation’s 100.0% success rate and 100.0 score.Among overlapping policies, simulation and real-world rankings align only partially, with InternVLA-A1 and GalaxeaVLA placing relatively better in reality; however, the benchmarks assess complementary capabilities rather than paired sim-to-real task distributions. Scores exceeding success rates indicate incomplete execution, while jitter, oscillation, contact instability, and occasional safety-critical behaviors expose deployment problems beyond aggregate success.Source: Table 2 · p. 14, §6.2 · pp. 17–18

6.1. Analysis of RoboDojo Simulation Performance

  • 8.80% average success rate and a 13.07 average score mark the simulation leader, versus human experts’ 76.03% success rate and 80.42 score.Hy-Embodied-0.5-VLA leads overall, Long-Horizon, and Memory; its Long-Horizon success rate reaches 14.92% with a 25.74 score, but reliable skill composition, stage decisions, and recovery remain difficult. X-VLA leads Precision with a 12.00% success rate and an 18.32 score, while observed alignment failures and action jitter expose weak state-conditioned correction and motivate smoother, contact-aware control.Source: §6.1 · pp. 14–16
  • 92.9% relative score loss under scene randomization takes Hy-Embodied-0.5-VLA from 21.98 to 1.57, illustrating broad sensitivity to distribution shifts.Spatial Forcing leads Generalization with a 9.33% success rate and a 14.12 score; compared with its π0.5 base, its Random score improves from 5.82 to 6.98 and its relative drop decreases from 72.2% to 67.2%. These gains suggest benefits from spatial grounding, but reliable randomized-scene manipulation still requires stable low-level control, closed-loop correction, and recovery.Source: Table 3 · p. 15, §6.1 · p. 15
  • 1.67% success rate and a 1.98 score for π0.5, the best Open policy, expose fragile semantic-to-action grounding; memory-conditioned execution also remains unreliable.On Memory, Hy-Embodied-0.5-VLA reaches a 12.11% success rate and a 13.37 score, EventVLA reaches 4.78% and 4.92, and X-WAM reaches 4.67% and 6.32, respectively; explicit memory and temporal prediction still struggle to support reliable execution and sparse-evidence recall. The findings motivate combining predictive temporal modeling, explicit memory supervision, and memory-conditioned action learning, alongside open-vocabulary affordance grounding and compositional skill learning.Source: §6.1 · pp. 16–17

6.3.1. Simulation Evaluation Efficiency

  • 1.94× zero-action and 1.63× π0.5-inference speedups demonstrate the throughput benefit of heterogeneous parallel simulation over non-heterogeneous execution.Zero-action throughput increases from 40.0 to 77.4 interactions/s, while heterogeneous execution with π0.5 inference reaches 64.0 interactions/s. Tests use eight RTX 4090 GPUs, each running ten parallel RoboDojo environments, with π0.5 inference on a network-local A800 server.Source: Table 4 · p. 18, §6.3.1 · p. 18

6.3.2. Real-World Evaluation Efficiency

  • 202.0 minutes completes RoboDojo-RealEval’s 18-task evaluation with π0.5, covering 180 physical trials at an average of 11.2 minutes per task.Timing includes scene reset, network-local RTX 4090 policy inference, and robot execution. Each task receives ten full trials without manual interruption or early stopping, making the reported cost conservative.Source: Table 5 · p. 19, §6.3.2 · p. 19

6.4.2. Real-World Evaluation Stability

  • 1.3 percentage points in overall success rate and 1.2 points in overall score bound standard deviations across repeated real-world evaluations.π0.5, InternVLA-A1, and GalaxeaVLA each undergo three independent rounds covering all 18 tasks and three embodiments. Individual contact-rich or multi-stage tasks show greater variance, but these fluctuations largely average out under the standardized evaluation protocol.Source: Table 6 · p. 19, §6.4.2 · pp. 19–20

6.4.1. Simulation Evaluation Stability

  • 0.5 percentage points in overall success rate and 0.49 in overall score bound the cross-GPU standard deviations in simulation.Across individual policies and capability dimensions, maximum standard deviations are 1.1 percentage points for success rate and 1.07 for score. Testing covers π0.5, Xiaomi-Robotics-0, and GalaxeaVLA on three RTX 4090 GPUs, using task layout 0 and three random seeds per GPU.Source: §6.4.1 · p. 19, Table 7 · p. 20

7. Future Extensions

RoboDojo is designed as an extensible benchmark, with planned expansion across manipulation scenarios, robot embodiments, sensing modalities, and evaluation settings.

7. Future Extensions

  • Planned larger-scale benchmarks cover dexterous hand manipulation, humanoid whole-body manipulation, tactile manipulation, and mobile manipulation.These are future extensions, not capabilities reported as already implemented.Source: §7 · p. 20, Figure 9 · p. 20
  • Each planned direction will support multiple robot embodiments to improve accessibility and enable fair comparison across hardware platforms.Source: §7 · p. 20
  • Progressive expansion of tasks, embodiment coverage, and sensing modalities aims to make RoboDojo a general-purpose evaluation platform for embodied manipulation.Source: §7 · p. 20

8. Conclusion · Appendix · A.1. Contributors

RoboDojo unifies simulation and real-world evaluation of generalist robot manipulation policies through shared infrastructure and protocols. Its experiments reveal capability gaps in simulation and poor performance on complex real-world manipulation tasks.

8. Conclusion

  • 42 simulation tasks and 18 real-world tasks cover diverse, long-horizon, precision-demanding, memory-dependent, and open-ended manipulation scenarios.Source: §8 · p. 20
  • RoboDojo-RealEval supports reproducible physical assessment through standardized hardware, scene reset procedures, evaluation protocols, and remote cloud-based access.Source: §8 · p. 20
  • Heterogeneous parallel execution enables efficient large-scale simulation evaluation, rapid feedback, and fine-grained capability diagnosis.Source: §8 · p. 20
  • XPolicyLab supplies unified policy development and deployment infrastructure, enabling integration, evaluation, and comparison under a shared protocol with RoboDojo.Source: §8 · p. 20
  • Current policies remain unreliable for general-purpose manipulation, with incomplete capability coverage in simulation and poor performance on complex real-world tasks.Simulation gaps include generalization, long-horizon execution, precision, memory, and open-ended task understanding; physical evaluations expose insufficient robustness under challenging deployment conditions.Source: §8 · pp. 20–21
  • RoboDojo will update its leaderboard with new policies, tasks, evaluation results, and community submissions to track progress toward more capable omni-manipulation policies.Source: §8 · p. 21

A.1. Contributors

  • The contributor acknowledgments recognize support for developing RoboDojo-RealEval infrastructure and providing policy models, integrating them, and supporting their evaluation.Source: §A.1 · p. 28

A.2. Evaluation Integrity and Leaderboard Publication Details

RoboDojo requires official remote evaluation, repeated simulation runs, multi-embodiment real-world testing, and hidden verification. Publication on its verified leaderboard additionally requires releasing the full evaluation artifact through XPolicyLab.

Official remote evaluation

  • Reported scores come from the official RoboDojo evaluation system, using standardized simulation and real-world evaluation procedures rather than participant self-reporting.Participants submit a deployable policy package or connect a remote policy server. Evaluations use released task layouts, standardized scene resets, the evaluation protocol, and the deployment interface.Source: §A.2 · p. 28

Repeated simulation evaluation and multi-embodiment real-world evaluation

  • Three simulation runs provide mean and standard deviation; real-world evaluation reports per-embodiment and aggregate performance across ARX X5, Piper, and Piper X.Participants provide either three training-seed checkpoints or one checkpoint evaluated under three evaluation seeds, reducing stochastic initialization effects and evaluation noise. Testing every policy on all three embodiments assesses generalization across collaborative bimanual platforms.Source: §A.2 · p. 28

Hidden verification

  • Hidden verification uses randomized layouts to detect overfitting, hand-tuning, or gaming; significant success-rate differences from public evaluation invalidate submissions for the verified leaderboard.Each submitted model undergoes this additional verification during official evaluation.Source: §A.2 · p. 28

Open-source artifact for verified publication

  • Verified leaderboard publication requires releasing the full evaluation artifact through XPolicyLab, enabling community inspection and independent reproduction.Required artifacts include training and deployment code, the evaluated checkpoint, configuration files, and instructions for loading, inference, deployment, and evaluation under the unified interface. This requirement applies at publication, not during private evaluation.Source: §A.2 · p. 28

B. Comparison with Existing Benchmarks

RoboDojo connects rapid simulation-based diagnosis with standardized real-world validation in a unified benchmark suite. Its comparison with existing benchmarks also highlights scale, capability coverage, task diversity, real-world reproducibility, and remote evaluation support.

B. Comparison with Existing Benchmarks

C. Real-World Evaluation Stability · D. Simulation Training and Evaluation Details · D.1. Simulation Training Data Details

RoboDojo assesses real-world evaluation variability through three independent runs of π0.5, GalaxeaVLA, and InternVLA-A1. Its simulation training data comprise 3,500 trajectories, with task-specific demonstrations excluded from the Open dimension.

C. Real-World Evaluation Stability

  • Three independent RoboDojo-RealEval runs evaluate variability for π0.5, GalaxeaVLA, and InternVLA-A1 using success-rate and score standard deviations.Success-rate deviations are measured in percentage points; the Overall row reports the standard deviation of average performance across runs.Source: C. Real-World Evaluation Stability · p. 28

D.1. Simulation Training Data Details

  • 3,500 trajectories across 35 task directories contain 1,859,602 frames, totaling 20.66 hours of bimanual manipulation data recorded at 25 Hz.Table 10 summarizes the simulation training dataset.Source: §D.1 · p. 29
  • Each trajectory provides synchronized 640 × 480 RGB-D observations from a head-mounted camera and two wrist cameras.Depth images are normalized and stored as integer millimeter values; an additional third-view RGB video supports preview and visualization.Source: §D.1 · p. 29
  • The simulation dataset includes robot states and next-frame actions covering end-effector poses, gripper states, and joint positions.Source: §D.1 · p. 29
  • 100 trajectories per task cover Generalization, Memory, Long-Horizon, and Precision, while Open receives no task-specific training demonstrations.Demonstrations use automated trajectory synthesis or VR-based teleoperation according to task structure and difficulty. Open evaluates skill recombination and transfer to unseen task specifications.Source: §D.1 · p. 29
  • 100 domain-randomized trajectories from one auxiliary DLC task broaden visual exposure, while Generalization training trajectories otherwise use normal settings without domain randomization.The auxiliary data vary backgrounds, lighting, and clutter to reduce visual overfitting during supervised fine-tuning without leaking evaluation-task solutions.Source: §D.1 · p. 29

D.2. Simulation Evaluation Details · E. Real-World Training and Evaluation Details

Simulation evaluation horizons are derived from demonstration lengths, with extra allowance for short, automatically synthesized tasks. The simulation training set excludes the Open dimension, reserving it for evaluating skill recombination and transfer to unseen task specifications.

D.2. Simulation Evaluation Details

  • 1.2× the 90th-percentile task-specific demonstration trajectory length sets the standard simulation evaluation horizon.This provides sufficient execution time while keeping the horizon consistent with demonstrated task complexity.Source: §D.2 · p. 29
  • 1.5× the 90th-percentile trajectory length sets the horizon for short, automatically synthesized tasks to avoid premature termination from small trajectory-length variations.Source: §D.2 · p. 29
  • 1,859,602 frames from 3,500 trajectories constitute the simulation training set.These statistics exclude the Open dimension.Source: Table 10 · p. 30
  • The Open dimension is excluded from training and used only to evaluate skill recombination and transfer to unseen task specifications.Source: Table 10 · p. 30
  • 20.66 hours of bimanual manipulation data at 25 Hz are included in the simulation training set.Source: Table 10 · p. 30
  • DLC is an auxiliary domain-randomized task used to broaden visual exposure during simulation training.Source: Table 10 · p. 30

E.1. Real-World Training Data Details … J.3. Piper

This section documents RoboDojo’s standardized real-world data collection and evaluation, reproducible simulation infrastructure, and XPolicyLab’s shared conventions for integrating heterogeneous policies. Its task catalog covers language- and image-guided manipulation, memory, precision, long-horizon execution, and training-only letter arrangement, alongside real-world tasks on three robot platforms with documented instructions, demonstration sources, and training/evaluation splits.

E.1. Real-World Training Data Details

  • 1,800 trajectories contain 1,611,841 frames and 17.91 hours of bimanual demonstrations at 25 Hz across ARX X5, Piper, and Piper X.Homogeneous leader-follower teleoperation collects 100 demonstrations per task from four operators, with 600 trajectories across six tasks per embodiment and embodiment-dependent durations. Demonstrations include joint states, end-effector poses, language, and synchronized 640×480 RGB from head and wrist cameras; multiple reviewers check standardized resets, excessive hesitation, and task completion.Source: §E.1 · p. 30, Table 11 · p. 31

E.2. Real-World Evaluation Details

  • 10 trials per policy and real-world task use replayed initial layouts and double-blind scoring averaged across three independent evaluators.Pink converts end-effector targets into robot motions; horizons equal 1.5 times the demonstration-length 90th percentile, with automatic termination and manual safety stops. Scoring includes outcomes and intermediate steps, videos and scores are released with appeals allowed, and submissions lacking reproducibility materials enter a separate non-official, unverified closed-source track.Source: §E.2 · p. 31

F.1. Simulation Platform Foundation / F.2. Configuration-Driven Simulation Setup / F.3. Digital Asset Processing and Validation

  • RoboDojo builds on MagicSim (Lu et al., 2026), combining deterministic YAML task configurations with annotated, physically validated rigid, articulated, and deformable assets.MagicSim supplies the manager-based runtime and object abstractions, connected to Isaac Lab’s vectorized interface; YAML defines assets, initialization, cameras, lighting, success conditions, and seeds for reproducible scene sampling. Repository assets and Meshy AI reconstructions receive semantic and manipulation annotations, while articulated/deformable checks and simulation rollouts validate physical behavior; deformable assets also include ClothesNet (Zhou et al., 2023).Source: §F.1 · p. 31, §F.2 · p. 32, §F.3 · p. 32

F.4. Heterogeneous Parallelism Implementation

  • RoboDojo combines heterogeneous environments within simulator processes with multi-GPU seed sharding for scalable evaluation and data collection.Disabling strict physics replication lets cloned environments instantiate different objects, geometries, clutter, articulations, and tasks from seeded configurations. Each environment retains independent state, initialization, observations, and success conditions while sharing batched stepping and action dispatch; independent GPU processes aggregate their results.Source: §F.4 · pp. 32–33

F.5. Simulation Demonstration Collection

  • Simulation demonstrations use annotation-grounded automated skill composition or VR teleoperation when automated synthesis is unreliable.Manually ordered skills are grounded in object affordances and executed through cuRobo v2 (Sundaralingam et al., 2026). VR controllers map delta 6D poses to end-effector targets solved online, with approximately 3 ms planning-step latency for a single 6-DoF ARX X5 arm, button-operated grippers, and foot-pedal controls.Source: §F.5 · p. 33

G. RoboDojo-RealEval Platform Details / G.1. Hardware Components of RoboDojo-RealEval

  • RoboDojo-RealEval standardizes sensing, robot placement, lighting, and workspace layout, with image-overlay resets, emergency stopping, and cloud-uploaded evaluation videos.Hardware includes Gemini 305 wrist cameras, a Gemini 335L head camera, replaceable ARX X5/Piper/Piper X embodiments, support structures, curtains, LED lighting, a workstation, and touchscreen controls. Five collectors restored five-object scenes in 14 seconds on average using reference overlays; emergency stops disable policy control and return robots safely, while scoring filters large rater discrepancies before averaging.Source: G. RoboDojo-RealEval Platform Details · p. 33, Figure 10 · p. 34

H. XPolicyLab Design Details

  • XPolicyLab standardizes data formats, policy interfaces, and deployment protocols while preserving each policy’s internal architecture.Policy-specific data adapters, common observation/action conventions, standard and batched Model methods, and offline debugging support integration. Consistent package scripts and experiment parameters, plus WebSocket/MessagePack communication, support the same policy implementation across real-world and batched simulation evaluation.Source: H. XPolicyLab Design Details · p. 34, Table 12 · p. 35

I. Simulation Benchmark Tasks Details / I.1. Generalization / I.2. Open

  • Simulation task specifications document instructions, descriptions, demonstration sources, and training/evaluation splits, including generalization tasks and evaluation-only open tasks.Generalization examples include teleoperated bowl stacking and automatically generated clothes folding, both used for training and evaluation. Open tasks provide no demonstrations and test set-square block alignment, completing equations by selecting missing numbers or operators, and stacking blocks in language-specified color order.Source: §I · p. 35, §I.1 · pp. 35–36, §I.2 · p. 37

I.2. Open

  • Open tasks evaluate language-directed picking, unseen-category sorting, ordered pouring, tool placement, and image-conditioned conveyor picking without training data.Image-conditioned conveyor picking first requires lifting the basket more than 8 cm before collecting the pictured object.Source: §I.2 · pp. 37–38

I.3. Memory

  • Memory tasks require recalling hidden colors, conveyor objects, exchanged poses, button counts, demonstrated sequences, and progress through block-swapping moves.AutoGen supplies demonstrations except for conveyor matching, which uses teleoperation; all these tasks support training and evaluation.Source: §I.3 · pp. 38–39

I.4. Precision

  • Precision tasks test accurate insertion, fastening, pouring, striking, handover-assisted alignment, and stable construction.Examples include color-matched screw tightening, pouring balls into a vase, striking xylophone keys, coin insertion, card handover before slot insertion, and tower construction. These support training and evaluation.Source: §I.4 · pp. 39–40

I.5. Long-Horizon

  • Long-horizon tasks combine repeated manipulation, bimanual coordination, multistep organization, turn-taking, and reactions to an opponent.Examples include bottle disposal requiring handovers, coordinated pen loading, tabletop organization, peg stacking, tic-tac-toe, and Mahjong kong responses. These tasks use teleoperated or AutoGen demonstrations for training and evaluation.Source: §I.5 · pp. 40–41

I.6. DLC

  • DLC is a training-only task requiring robots to identify letters spelling RoboDojo and arrange them in order on a cluttered tabletop with a complex background.The letters must form a straight row.Source: §I.6 · p. 41

J.1. ARX X5

  • ARX X5 tasks combine insertion, food preparation, pouring, hidden-color recall, and packing, using teleoperated demonstrations for training and evaluation.Tasks include tube insertion, bread-and-bowl placement, multistep cooking preparation, fruit collection followed by pouring, covering and uncovering colored blocks, and loading then closing a safe.Source: §J.1 · p. 42

J.2. Piper X

  • Piper X tasks cover LEGO disassembly, pen capping, mug hanging, object classification, broom handover and sweeping, and backpack packing.All use teleoperated demonstrations for training and evaluation. Sweeping requires transferring the broom to the right hand and grasping the dustpan with the left.Source: §J.2 · p. 43

J.3. Piper

  • Piper tasks cover pen loading, basket packing, block stacking and covering, bowl stacking, bottle uprighting, and two-stage charger connection.Teleoperated demonstrations support training and evaluation; charger insertion precedes connecting the cable to the plug.Source: §J.3 · pp. 43–44

K. Policy Training Details

RoboDojo reports policy-specific initialization checkpoints and training budgets for simulation and real-world evaluation, including specialized memory encoding and staged optimization settings.

Hy-Embodied-0.5-VLA

  • 200K simulation steps at batch size 128 fine-tune Hy-Embodied-0.5-VLA, using 6 historical images sampled every 20 steps for memory encoding.Initialization uses the Hy-Embodied-0.5-VLA-UMI foundation checkpoint, following its official RoboTwin 2.0 evaluation setting.Source: K. Policy Training Details · p. 44

Spatial Forcing and π0.5

  • 60K simulation steps at batch size 256 fine-tune Spatial Forcing and π0.5; π0.5 uses 30K real-world steps at the same batch size.Both initialize from the gs://openpi-assets/checkpoints/pi05_base foundation checkpoint.Source: K. Policy Training Details · p. 44

X-VLA and Xiaomi-Robotics-0

  • 100K simulation steps and 50K real-world steps fine-tune both X-VLA and Xiaomi-Robotics-0, each using batch size 256 in both settings.X-VLA initializes from 2toInf/X-VLA-Pt. Xiaomi-Robotics-0 initializes from Xiaomi-Robotics-0-Pretrain.Source: K. Policy Training Details · pp. 44–45

X-WAM

GigaWorld-Policy

  • 300K total simulation optimization steps train GigaWorld-Policy at batch size 32: 50K for video optimization followed by 250K for action optimization.Initialization uses Wan2.2-TI2V-5B; video optimization takes 1 epoch and action optimization takes 2 epochs, totaling 3 epochs.Source: K. Policy Training Details · p. 45

Related papers

Loading 2607.04434v3…