Source-linked AI summary
RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies
Adina Yakefu, Bin Xie, Chongyang Xu, Enwen Zhang, Erjin Zhou, Fan Jia, Haitao Yang, Haoqiang Fan, Haowei Zhang, Hongyang Peng, Jing Tan, Junwen Huang, Kai Liu, Kaixin Liu, Kefan Gu, Qinglun Zhang, Ruitao Zhang, Saike Huang, Shen Cheng, Shuaicheng Liu, Tiancai Wang, Tiezhen Wang, Wei Sun, Wenbin Tang, Yajun Wei, Yang Chen, Youqiang Gui, Yucheng Zhao, Yunchao Ma, Yunfei Wei, Yunhuan Yang, Yutong Guo, Ze Chen, Zhengyuan Du, Ziheng Zhang, Ziming Liu, Ziwei Yan
TL;DR
RoboChallenge targets the need for scalable, reproducible large-scale evaluation of VLA models on real robots. It provides an online fleet, curated tasks, demonstration data, and a controlled evaluation protocol, and its Table30 survey finds substantial differences among models, with π0.5 strongest in almost all respects. The authors identify user-side verification and possible test-case overfitting as limitations.
Problem
Large-scale real-robot benchmarking for VLA models must be fair, scalable, and robust because simulators cannot reproduce all real-world factors.
Method
RoboChallenge offers online-hosted robots, curated tasks, demonstration data, and visual task reproduction to standardize real-robot evaluation.
Results
The Table30 evaluation finds considerable differences among models, with π0.5 remarkably stronger in almost all respects.
Takeaways & Limitations
RoboChallenge frames its benchmark as a necessity test for methods pursuing general robotics and provides rankings and evaluation artifacts for inspection.
Takeaways & Limitations
User-side inference prevents RoboChallenge from verifying that the executed model matches the submitted model name, while fixed test distributions may permit overfitting to reference cases.
Abstract
from arXiv · showhide
Testing on real machines is indispensable for robotic control algorithms. In the context of learning-based algorithms, especially VLA models, demand for large-scale evaluation, i.e. testing a large number of models on a large number of tasks, is becoming increasingly urgent. However, doing this right is highly non-trivial, especially when scalability and reproducibility is taken into account. In this report, we describe our methodology for constructing RoboChallenge, an online evaluation system to test robotic control algorithms, and our survey of recent state-of-the-art VLA models using our initial benchmark Table30.
1. Introduction
RoboChallenge addresses the need for fair, scalable, robust large-scale real-robot benchmarking by providing publicly accessible online machines, curated tasks, demonstration data, and the Table30 benchmark.
- Real-robot benchmarking is needed because digital twins cannot reproduce all real-world factors, creating a large-scale evaluation problem for VLA models.
- RoboChallenge provides a fleet of online-hosted machines and curated tasks for public evaluation of robotic control algorithms.The initial offering includes multiple robot types and demonstration data for the supported tasks.
- The benchmark reports end-to-end task success and partial-progress scores to distinguish model performance across tasks.
- Table30 contains 30 fixed-table tasks designed to stress varied VLA learning capabilities, with trajectories and test videos released publicly.The initial release tested five methods, including π-series implementations and volunteer-submitted methods.
2. RoboChallenge
RoboChallenge is an online evaluation infrastructure whose construction requires deliberate system design for serving robots and supporting reproducible testing.
- Serving robots online is nontrivial, so RoboChallenge documents and justifies the design decisions underlying its evaluation system.
2.1. Online Interface for Serving Robots
RoboChallenge uses a remote-robot interface in which users run models locally while accessing timestamped observations and an asynchronous robot action queue.
- The system rejects model-level, system-level, and model-API submissions because reproducing software and hardware environments across premises is difficult.The authors specifically cite mismatched CUDA, Python, framework, and hardware configurations as obstacles.
- The remote-robot paradigm keeps model execution on the user side and exposes low-level, fully asynchronous camera and machine access.Users handle action formatting and post-processing, enabling temporal alignment and ensembling strategies.
- Users can submit timestamped RGB, depth, and proprioceptive observations while posting timed actions to a FIFO queue whose current length is exposed through the API.Queued actions are irrevocable, and users call the evaluator’s API rather than exposing a publicly accessible endpoint.
- Job-scheduling APIs tell users when models will run, which model to load, and how the evaluation job is progressing.
2.2. The Robot Platforms
RoboChallenge selects durable, popular robot platforms and equips them with timestamped RGB and depth sensing, while tester behavior is treated as a source of evaluation variation.
- The initial release includes UR5, Franka Panda, Cobot Magic Aloha, and ARX-5 platforms selected for durability and research popularity.The platforms span single-arm, dual-arm, and moving-platform configurations.
- The system uses Intel RealSense sensors to provide timestamped RGB and depth streams, with main, wrist, and side camera configurations.
2.3. Evaluation Protocol
RoboChallenge addresses severe run-to-run variability in real-robot testing by controlling task setup through visual reproduction, while distinguishing stability from fairness and acknowledging uncontrolled environmental perturbations.
- 2.3. Evaluation Protocol: Success rates can vary from 0% to 100% across runs with the same props, task, and model, motivating principled control of test factors.
- 2.3.1. Variation of Testers: Adaptive testers exploit favorable object positions, producing seemingly improved performance, while ignorant testers may encounter sweet spots or counter-sweet spots unpredictably.
- 2.3.2. Visual Task Reproduction: The controlled tester matches each rollout’s initial scene to a held-out demonstration frame, largely fixing object and scene states across model evaluations.
- 2.3.2. Visual Task Reproduction: Controlled-tester evaluations are empirically more stable than evaluations by experienced testers and require less familiarity with the demonstration-data process.
- 2.3.3. Background and Environmental Issues: Lighting changes and camera-extrinsic drift remain uncontrolled and are treated as intended perturbations, while a proof-of-concept shows model outputs remain steady under image corruption.
- 2.3.4. Stability v.s. Fairness: Stability concerns repeated-result variation, whereas fairness concerns preserving model ordering; ideal stability implies fairness, but real-world correlations prevent ideal stability.
2.4. Limitation and Known Issues
RoboChallenge identifies two limitations: user-side inference cannot verify submitted models, and fixed test distributions may permit overfitting to reference cases.
- User-side inference cannot verify that the executed model matches the model claimed by the user.This creates opportunities to submit different models, use individually tuned models, or conduct human-in-the-loop runs.
- Fixed test distributions may allow submissions to overfit particular reference test cases, although the authors have not observed this in practice.
3. Table30 Benchmark
Table30 is a 30-task, table-centered benchmark designed to test diverse capabilities and difficulty levels in general robot control. It combines staged progress scoring with task distributions intended to reveal meaningful differences between algorithms.
- 3.1. The Tasks: Table30 contains 30 tasks executed on or around a table, with the full task list provided in Table 1.
- 3.1. The Tasks: Tasks span precise 3D localization, occlusion and multi-view reasoning, temporal dependence, object recognition, bimanual use, and soft-body manipulation.These properties require spatial understanding, multiple-camera information, memory of task progress, object-dependent actions, arm selection, and generalization to deforming objects.
- 3.1. The Tasks: Table30’s diverse tasks expose weaknesses that prevent even the strongest base models from achieving high overall success rates.The benchmark is framed as a “necessity test” for methods pursuing general robotics and provides subset rankings for further inspection.
- 3.2. Grading Protocol: Progress scoring supplements success rate by rewarding completed stages and penalizing retries, while non-critical stages may be omitted without invalidating a task.Each evaluation totals 10 progress points; a retry deducts 0.5 points, and rollouts terminate after negative stage scores or more than 4 failed successive retries.
- 3.3. Designing Tasks that Differentiates: Figure 7 classifies tasks by VLA difficulty, robot type, scenario location, and main target-object properties to demonstrate diversity and coverage.The authors acknowledge that other sets of 30 tasks could satisfy the same criteria but regard this selection as typical for benchmarking.
- 3.3. Designing Tasks that Differentiates: Task selection covers very easy to difficult tasks, diverse algorithmic challenges, and varied real-life scenarios so models retain room for improvement.The authors state that progress should appear as incremental performance gains when methods unlock abilities, working scenarios, or generalization to new object types.
4. Results on Table30
Table30 evaluates VLA models across task-specific and machine-generalist settings, showing clear differences in model strength and task difficulty. The results identify π0.5 as the strongest model and reveal which task properties are hardest or easiest.
- Model results: π0.5 (finetuned) dominates the other models across all percentiles of success rate and progress score.The cumulative distributions show a clear separation between models, while their similar slopes indicate an even distribution of task difficulties.
- Model results: π0.5 maintains fair performance in the machine-generalist setting with about 50 episodes per task, sometimes exceeding its task-specific result.The multi-task model is trained jointly on sampled data from tasks of the same machine type.
- Task analysis: Precise3d tasks achieve 18% success versus 25% without the tag, making precise localization a measurable difficulty.The models operate at 224x224 resolution, which the authors connect to this result.
- Task analysis: Bimanual, multiview, and repeated-task tags do not produce an additional decrease in average performance.The authors attribute this pattern to the global task average already containing at least one type of difficulty.
- Task analysis: Classification and manipulation tasks have slightly higher success rates than the global average, consistent with semantic problems being easier for the models.The benchmark associates these tags with tasks involving object-to-location classification or hinges and racks.
- Task analysis: Simple-pick tasks are the easiest benchmark portion, with about twice the global-average success rate and up to 90% for π0.5.These tasks are deliberately included as simple pick-and-place cases, and the authors describe them as largely solved for strong models.
A. Walkthrough of Submitting a Model for Evaluation
RoboChallenge guides participants from downloading demonstrations and choosing a training setting through API preparation, queued evaluation, and public result inspection. The workflow provides utilities, mock testing, scheduling, and recorded outputs to support submissions.
- Submission workflow: Participants download demonstration videos and robot-state JSON files, then optionally convert them to LeRobot format.The dataset is hosted on Hugging Face, with videos and states stored separately.
- Submission workflow: Participants choose Generalist or Finetuned training, with Generalist models jointly trained across tasks using prompts to distinguish them.Finetuned models have no restriction on how they are trained.
- Submission workflow: Skeleton code and a mock test help participants connect inference code to the asynchronous observe-inference-stop API cycle.The API integration includes polling before evaluation and a pre-submission compatibility check.
- Submission workflow: Submitted requests specify an access key, task set, and claimed model name, with multiple selected tasks treated as a multi-task generalist submission.Models sharing a displayed name can be grouped as one ranked entry when they use the same foundation or essentially the same algorithm.
- Submission workflow: RoboChallenge manually queues evaluations, prepares props and test scenes, and may require hours to days before execution.The resulting numbers, videos, and RRD machine logs become available after evaluation.
- Submission workflow: The website exposes participants’ results and recorded videos to other participants, while grading errors can be submitted for recalculation.Figure 10 presents the intended end-to-end participant workflow.
B. Photos of the Robot Platforms
The benchmark uses multiple robot platforms to support real-robot evaluation. The documented examples include the UR5, Cobot Magic ALOHA, and Franka Research 3.
- Robot platforms: The robot-platform examples shown are the UR5, Cobot Magic ALOHA, and Franka Research 3.These platforms are presented through example photos in the robot-platform section.