PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World

Zhiyuan Zhou, Pranav Atreya, You Liang Tan, Karl Pertsch, Sergey Levine

arXiv:2503.24278v2cs.ROcs.AI

TL;DR

Real-world evaluation of increasingly generalist robot policies is costly because it requires many trials across diverse tasks and scenes. AutoEval automates policy execution, success detection, and scene resets, enabling around-the-clock evaluation with minimal human intervention. Its results closely match human-run evaluations while supporting broader real-world tasks than simulation-based approaches.

  • Problem

    Evaluating generalist robot policies at statistically useful scale requires many real-world rollouts across diverse tasks and scenes, creating substantial human-labor demands.

  • Method

    AutoEval queues policies and uses learned success classifiers, reset policies, and automated evaluation cells to run repeated real-world trials with minimal human intervention.

  • Results

    AutoEval evaluation results closely match oracle human-run evaluations, with average Pearson score 0.942 and MMRV 0.015.

  • Takeaways & Limitations

    Public Bridge-AutoEval cells and setup guidance support more accessible and comparable evaluation of robot policies across institutions.

  • Takeaways & Limitations

    Creating a new AutoEval environment still requires upfront human effort to train reset policies and success classifiers.

Abstract

from arXiv · show

Scalable and reproducible policy evaluation has been a long-standing challenge in robot learning. Evaluations are critical to assess progress and build better policies, but evaluation in the real world, especially at a scale that would provide statistically reliable results, is costly in terms of human time and hard to obtain. Evaluation of increasingly generalist robot policies requires an increasingly diverse repertoire of evaluation environments, making the evaluation bottleneck even more pronounced. To make real-world evaluation of robotic policies more practical, we propose AutoEval, a system to autonomously evaluate generalist robot policies around the clock with minimal human intervention. Users interact with AutoEval by submitting evaluation jobs to the AutoEval queue, much like how software jobs are submitted with a cluster scheduling system, and AutoEval will schedule the policies for evaluation within a framework supplying automatic success detection and automatic scene resets. We show that AutoEval can nearly fully eliminate human involvement in the evaluation process, permitting around the clock evaluations, and the evaluation results correspond closely to ground truth evaluations conducted by hand. To facilitate the evaluation of generalist policies in the robotics community, we provide public access to multiple AutoEval scenes in the popular BridgeData robot setup with WidowX robot arms. In the future, we hope that AutoEval scenes can be set up across institutions to form a diverse and distributed evaluation network.

1. Introduction

Generalist robot policies require evaluation across many tasks and scenes, making manual real-world assessment increasingly costly. AutoEval addresses this bottleneck by automating policy execution, success detection, and scene resets, enabling high-throughput evaluations with results that closely match human evaluations.

  • Motivation: Generalist policies require hundreds of rollouts across diverse tasks and scenes for accurate assessment.A representative OpenVLA evaluation required more than 2,500 rollouts and over 100 hours of human labor.
  • Approach: AutoEval queues user-supplied policies and autonomously runs, scores, and resets evaluations with minimal human intervention.The system is designed to support around-the-clock evaluation and more trials per evaluation.
  • Approach: AutoEval adapts large pre-trained models to each evaluation scene for automatic reset policies and success detection.These learned components are intended to improve reliability while reducing operator involvement.
  • Results: AutoEval instantiates three BridgeData V2 evaluation environments and supports 500 evaluation episodes per 24-hour period.The system automates scene resets and success determination, with results correlating well with ground-truth human evaluations.
  • Results: AutoEval provides more reliable performance estimates than prior simulated evaluation approaches while supporting hard-to-simulate tasks such as cloth manipulation.The claimed advantage combines real-world evaluation with automated operation.
  • Availability: The authors release code, setup guidance, and access to multiple Bridge-AutoEval cells for evaluation across institutions.The intended use is more comparable evaluation on unified setups.

2. Related Work

Existing robot evaluation approaches face scalability and reproducibility limits as policies become more generalist. AutoEval differs by providing a task-agnostic, learned framework intended to automate diverse real-world evaluations and support future cross-institution benchmarks.

  • Generalist robot policies: Generalist policies increase evaluation costs because they cover diverse tasks, scenes, backgrounds, distractors, and robot embodiments.Their broader capabilities require evaluation across more varied conditions.
  • Robot policy evaluation in the real world: The authors demonstrate automation across diverse tasks and provide a guide for setting up new evaluations within hours.They envision reproducing the recipe across institutions to build a comprehensive generalist-policy benchmark.
  • Evaluation in simulation: Simulation offers high-throughput evaluation but remains limited by discrepancies between simulators and the real world.This motivates retaining real-world evaluation despite its human-labor cost.
  • Autonomous robot operations: Prior autonomous robot-learning work addresses related reset and success-detection challenges but generally targets policy improvement rather than evaluation.AutoEval applies related ideas to autonomous policy assessment.
  • Autonomous robot operations: Existing real-robot evaluation automation has largely used task-specific environmental instrumentation, such as spring-driven or scripted reset mechanisms.These approaches are contrasted with AutoEval’s flexible learned components.
  • Autonomous robot operations: AutoEval proposes a task-agnostic and scalable evaluation approach using learned components based on generalizable foundation models.The system is presented as a general framework rather than a single task-specific solution.

3. Autonomous Evaluation of Robot Policies in the Real World

AutoEval reformulates repeated human-run robot evaluation as an autonomous loop that estimates task success through learned success detection and scene resetting. Its design targets statistically meaningful real-world evaluation while reducing the operator time required across many trials.

  • Problem setting: Robot policy evaluation estimates the probability that a policy successfully completes a task under an initial-state distribution.The policy maps observations and language instructions to actions, while the task definition maps states to binary success.
  • Problem setting: Manual evaluations require repeated trials, randomized initial states, scene resets, supervision, and human success scoring.For generalist policies evaluated across many tasks, one checkpoint can require multiple days.
  • System design: AutoEval preserves the conventional multi-trial structure while replacing human evaluation tasks with learned autonomous modules.Its core modules include success classification and reset policies, with safety handling for problematic cases.
  • Success classifier: The success classifier fine-tunes a pre-trained vision-language model using approximately 1000 success and failure images.The images can be collected in less than 10 minutes through teleoperation, and the classifier predicts binary task completion.
  • Reset policy: The reset policy reverses the evaluation rollout to return the scene and robot to the initial-state distribution.This learned reset replaces task-specific hardware mechanisms such as springs or magnets.
  • Platform: The Bridge-AutoEval setup uses a WidowX 250 6-DoF arm and a Logitech C920 HD RGB camera, with scenes reproducing BridgeData evaluation tasks.The platform includes drawer, sink, and cloth scenes.
  • Operational results: Manual interventions occurred three times per 24 hours of autonomous evaluation, while a new cell required 1–3 hours of human effort and less than 5 hours including training.These figures describe the implemented cells and setup process.

4. Bridge-AutoEval: Open-Source Automated Eval Platform

Bridge-AutoEval instantiates AutoEval across three BridgeData-distribution scenes and five manipulation tasks, while exposing public scheduling and monitoring for policy evaluation.

  • Platform basis: The platform uses BridgeData V2 as a testbed for scalable evaluation of generalist policies, with a WidowX robot arm and diverse manipulation data.BridgeData contains 60k+ demonstrations spanning 13 skills and 24 environments; several state-of-the-art generalist policies are trained on it or a superset.
  • Evaluation scenes: Three parallel cells cover drawer, sink, and cloth scenes, representing articulated-object, pick-and-place, and deformable-object manipulation.The drawer cell supports opening and closing; the sink cell supports two eggplant placement tasks; the cloth cell supports folding from top right to bottom left.
  • Automation: Each scene uses trained success classifiers and reset policies, plus safety detectors and Slack notifications for human intervention requests.These components support autonomous operation while retaining an on-call intervention path.
  • Public access: Researchers can access two public Bridge-AutoEval cells through a web UI, choose a task, provide a policy-server IP address, and monitor progress.The public interface is intended to improve reproducibility and comparability across policy evaluations.
  • Evaluation workflow: AutoEval queues submitted jobs, queries hosted policy servers for actions during execution, and returns rollout data and detailed performance reports.Reports include rollout videos, success rates, and episode-level evaluation information.

5. Experimental Results

AutoEval is evaluated against human-run oracle evaluations, simulated evaluation, and offline metrics across diverse robot manipulation tasks. Results show strong agreement with human judgments, substantial reductions in human effort, and reliable long-duration operation, while occasional failures and motor overheating remain practical boundaries.

  • Agreement with Human Evaluation: AutoEval closely matches oracle human-run evaluations, whereas validation MSE is unreliable and SIMPLER performance varies across policies and tasks.SIMPLER can sometimes match real-world performance but is affected by policy-dependent sim-to-real discrepancies.
  • Agreement with Human Evaluation: 0.942 average Pearson score and 0.015 MMRV show strong agreement between AutoEval and human-run evaluations.The plotted 1 − MMRV value is 0.985; low MMRV indicates limited disruption of policy rankings.
  • Human Effort: > 99% reduction in human evaluator time demonstrates that AutoEval substantially lowers the labor required for real-robot evaluation.AutoEval still does not perfectly match human evaluations because success detection and reset failures occur occasionally.
  • Long-Duration Operation: 350 episodes across the first seven evaluation runs stayed within ±10% natural evaluation variance, but performance regressed after about eight hours of continuous operation.The authors attribute this regression to overheating motors on the low-cost WidowX robot.
  • Reliability and Failure Analysis: AutoEval produced consistent results over two months, while automatic reruns handled motor-failure episodes during failure analysis.The authors identify reducing motor failures, potentially through a more compliant controller, as the main efficiency improvement.

6. Conclusion

AutoEval enables high-quality, around-the-clock evaluation of generalist robot policies with minimal human involvement. The authors provide public access to two AutoEval cells for BridgeData V2 tasks to improve availability and comparability.

  • AutoEval performs high-quality evaluations around the clock with minimal human involvement across commonly used robot evaluation tasks.
  • Its results closely match human-run evaluations and are more reliable and applicable to a wider range of tasks than prior simulation-based approaches.
  • The authors provide public access to two AutoEval evaluation cells for BridgeData V2 tasks, with online policy submission and detailed reports.
  • The authors hope additional cells across institutions will form a diverse automated evaluation framework that speeds robot learning research.

7. Limitations

AutoEval still requires up-front human effort to create environments, and richer robustness and performance analyses remain areas for future work.

  • Creating a new automated evaluation environment currently requires a few hours of human effort to train reset policies and success classifiers.The authors state that future work could reduce this setup effort.
  • AutoEval does not yet automatically vary robustness axes such as camera angle, distractors, lighting, or table texture in a controlled way.
  • More granular performance analysis from logged evaluation videos remains time-consuming when performed by re-watching them.
  • Future work could investigate video summarization models for providing more granular automated performance analysis.

B. Visualizations of AutoEval Rollouts

The visualizations present evaluation trajectories across five Bridge-AutoEval tasks and list the language commands given to the evaluated policies.

  • Figure 12 presents evaluation trajectories for five Bridge-AutoEval tasks.
  • The evaluated policies receive natural-language commands specifying each task.
  • The tasks include closing and opening a drawer, placing an eggplant in a yellow basket, placing it in a blue sink, and folding cloth.

C. Detailed Evaluation Results on Bridge-AutoEval

The detailed results compare scalable evaluation approaches with human ground-truth evaluations across five Bridge V2 tasks and six generalist policies.

  • Drawer evaluations run for at most 70 steps, sink evaluations for 100 steps, and cloth evaluations for 80 steps.
  • The tables report AutoEval results across five Bridge-AutoEval tasks and six different generalist policies.
  • The tables also provide ground-truth human evaluation results for the same five tasks and six policies.

D. Evaluation on Bridge-SIMPLER [46]

AutoEval evaluates policies across diverse Bridge-SIMPLER scenes and tasks using automated resets and success detection. Its setup includes randomized trials, learned or scripted reset policies, and VLM-based success classifiers deployed after reliability testing.

  • Evaluation environments: Each evaluation trial randomizes the end-effector pose, drawer pose, and background lighting to add scene variability.
  • Evaluation environments: AutoEval evaluates four tasks across Drawer and Sink scenes, including forward and reverse eggplant movements that make the Sink environment reset-free.The Drawer scene supports opening and closing the drawer, while the Sink setup supports moving the eggplant to the basket or sink.
  • Automatic success detection: Success classifiers are fine-tuned vision-language models trained on roughly 1,000 labeled images per scene and deployed only after exceeding 95% accuracy.Prompts distinguish task-specific states such as eggplant location, drawer openness, and cloth-folding status.
  • Automatic scene resets: Reset policies are trained from 50–100 teleoperated demonstrations per scene, while scripted resets are also used when reset success is insensitive to scene variation.Deployed reset policies are selected using a success-rate threshold above 95%.
  • Deployment: The evaluation infrastructure supports queued policy jobs, safety measures, notifications, and parallel operation of publicly accessible AutoEval cells.The documented setup requires approximately 3 hours of active human effort and 5 hours including model training.

J. Evaluation Results Reproducible Across Months

AutoEval produces consistent evaluation results over more than two months of continuous use. Across two policies and three tasks, results remained similar and reset and success components retained 96% accuracy.

  • Long-term reproducibility: More than two months of continued use produced similarly performing evaluations across two policies and three tasks.The system operated continuously for roughly 200 hours during this interval.
  • Long-term reproducibility: Reset policies and success classifiers each retained 96% accuracy after the two-month evaluation period.
  • Robustness factors: Safety controllers and foundation-model pre-training are identified as contributors to robustness against robot aging and minor scene changes.Observed aging was limited to light drawer scratches that did not affect the policy observations or drawer physics.

K. Initial States in Bridge-AutoEval Cells

AutoEval’s learned reset policy produces a consistent distribution of initial states across repeated runs. The reported distributions overlap and approximately cover the task distribution, while additional human review can correct imperfect evaluations.

  • Initial-state consistency: Three representative AutoEval runs each used 50 trials and showed consistent eggplant initial-position centroids for the Eggplant to Basket task.
  • Initial-state consistency: Reset distributions for other tasks similarly overlap and roughly cover their task distributions.
  • Human verification: AutoEval results highly correlate with human-run evaluations but are not perfect because reset and success-detection errors can occur.Reviewing 50 trials takes roughly 1–2 minutes and allows failed resets to be removed and successes to be relabeled manually.
Loading 2503.24278v2…