PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, Jonathan Tremblay, Kanav Arora, Kirsty Ellis, Luca Macesanu, Marcel Torne Villasevil, Matthew Leonard, Meedeum Cho, Ozgur Aslan, Shivin Dass, Jie Wang, William Reger, Xingfang Yuan, Xuning Yang, Abhishek Gupta, Dinesh Jayaraman, Glen Berseth, Kostas Daniilidis, Roberto Martin-Martin, Youngwoon Lee, Percy Liang, Chelsea Finn, Sergey Levine

arXiv:2506.18123v2cs.ROcs.LG

TL;DR

Generalist robot policies are difficult to compare comprehensively because conventional benchmarks standardize a small set of tasks and environments. RoboArena instead aggregates double-blind pairwise evaluations from a distributed evaluator network, and across 7 policies, 7 universities, and 612 comparisons it produced more accurate rankings than conventional centralized evaluation while retaining sample efficiency. The framework is open-sourced, but the experiments focus on DROID and robustness to adversarial evaluators remains untested.

  • Problem

    Conventional robot evaluations standardize relatively few tasks and environments, limiting comprehensive comparison of generalist policies across diverse real-world conditions.

  • Method

    RoboArena uses decentralized evaluators to perform double-blind pairwise comparisons on chosen tasks and scenes, then aggregates preferences into rankings and qualitative policy reports.

  • Results

    612 pairwise comparisons across 7 generalist policies and 7 universities produced more accurate rankings than conventional centralized evaluation, while matching standard evaluation’s episode efficiency.

  • Takeaways & Limitations

    RoboArena provides a distributed, open-ended framework for more accurate and sample-efficient real-world evaluation of generalist robot policies.

  • Takeaways & Limitations

    Experiments focus on the DROID platform, and robustness to intentionally adversarial evaluators has not been investigated.

Abstract

from arXiv · show

Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies.

1 Introduction

RoboArena addresses the difficulty of evaluating generalist robot policies across diverse real-world tasks and environments. It replaces fixed standardized evaluations with distributed, double-blind pairwise comparisons and produces more accurate rankings than conventional centralized evaluation.

  • Motivation: Conventional benchmarks cover few standardized tasks and scenes, limiting comprehensive evaluation of broadly capable policies.Real-world standardization also faces practical challenges in reproducing layouts, lighting, and hardware conditions at scale.
  • RoboArena: Its decentralized design is open-ended in task and environment coverage, robust through many independent evaluations, and scalable across asynchronous contributors.The framework avoids requiring identical initial conditions beyond the horizon of a pairwise comparison.
  • RoboArena: RoboArena crowdsources double-blind pairwise policy comparisons in evaluator-chosen scenes and tasks, aggregating preferences into global rankings.Evaluators also provide free-form explanations, enabling qualitative policy characteristics alongside rankings.
  • Results: Across 7 generalist policies, 7 universities, and 612 pairwise comparisons, RoboArena produced more accurate rankings than conventional centralized evaluation against an exhaustive-evaluation oracle.The same number of evaluations distributed across RoboArena produced higher-quality rankings, matching standard evaluation’s episode efficiency.

2 Related Work

Prior work includes simulation benchmarks, generalist-policy evaluations, and crowdsourced benchmarks. These approaches provide useful foundations but leave challenges in real-world diversity, reproducibility, or scalable policy comparison.

  • Simulated and real-world robot evaluation: Simulated benchmarks are reproducible and parallelizable, but simulated environments can imperfectly reflect real-world policy performance.They may also support limited task diversity despite efforts to improve visual and physical fidelity.
  • Generalist robot policies: Generalist robot policies increasingly train on diverse experience and perform tasks across many environments, while conventional evaluation struggles to assess them comprehensively.
  • Crowd-sourced benchmarks: Existing crowdsourced benchmarks aggregate evaluations across institutions, but some require retraining policies for each institution and task, limiting tested diversity.RoboArena contrasts with this requirement by proposing distributed comparisons of existing policies.

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

RoboArena replaces repeated standardized evaluations with decentralized, double-blind pairwise comparisons across evaluator-chosen tasks and scenes. Its ranking and analysis pipeline aggregates preferences while modeling task variation and extracting qualitative policy characteristics.

  • Motivation: Standardized real-world evaluations are difficult to reproduce across institutions, especially when matching lighting, camera angles, scenes, and initial conditions.
  • Decentralized evaluation: RoboArena broadens coverage by letting evaluators choose scenes and tasks, while double-blind decentralized comparisons reduce the influence of any single evaluator.Many institutions can contribute asynchronously, making the approach open-ended, robust, and scalable.
  • Evaluation protocol: Each evaluation samples two policies, runs them sequentially on a rearranged scene and task, and returns progress scores, a pairwise preference, and language feedback.Sessions can pause or continue later, and evaluations outside a comparison run asynchronously.
  • Global policy rankings: The ranking model uses pairwise preferences associated with tasks to estimate a global ordering of policies.The Bradley-Terry baseline models preference probability from differences in policy log-abilities.
  • Global policy rankings: Task-aware ranking augments policy abilities with task difficulty, task probabilities, and policy-task offsets to account for changing evaluation conditions.These parameters are learned solely from preference data, with the number of task buckets treated as a hyperparameter.
  • Global policy rankings: An approximate EM algorithm fits the task-aware model by alternating likelihood calculations, derivative updates, maximization, and parameter centering.
  • Qualitative policy characteristics: A VLM categorizes scenes and tasks, while an LLM aggregates rollout videos, instructions, and evaluator feedback into reports of policy strengths and weaknesses.The reports cite concrete evaluation rollout videos as evidence.

4 The DROID-RoboArena Evaluation System

DROID-RoboArena implements the distributed evaluation framework with mobile real-robot setups, remotely hosted policies, evaluation clients, shared result storage, and central orchestration. The system is designed for community participation, safety, and sustainable contribution.

  • DROID platform: The DROID setup uses a Franka Panda 7DoF arm, Robotiq gripper, wrist stereo camera, and external stereo cameras.Its mobile, height-adjustable table supports rapid scene and camera reconfiguration.
  • DROID platform: DROID supports distributed evaluation because its dexterous platform handles varied manipulation tasks and setups are deployed across multiple academic institutions.The platform is also associated with the open-source DROID dataset for training generalizable policies.
  • Scope and limitations: The prototype currently focuses experiments on DROID, while extension to diverse embodiments and policies restricted to specific embodiments remains future work.
  • System design: RoboArena’s system comprises remotely hosted policy servers, distributed evaluator clients, an evaluation database, and a central evaluation server.The central server assigns policies, tracks the policy pool, manages communication, aggregates results, and computes rankings.
  • System design: Remote policy servers let evaluators share inference resources and use lightweight clients without local inference compute.This lowers the barrier for contributing evaluations while allowing users to control their hosted policy servers.
  • System design: The database stores instructions, scores, preferences, natural-language feedback, and rollout videos uploaded by evaluation clients.
  • Community participation: DROID-RoboArena aims to let researchers without physical robots train and evaluate real-world generalist policies using open-source datasets, modeling frameworks, and evaluations.
  • Participation and safety: Safety layers validate policy interfaces and test submitted policies under trained supervision before broader robot evaluation.An evaluation-credit system balances contribution supply and demand by exchanging completed evaluations for requested comparisons.

5 Experiments

RoboArena evaluates generalist policies through diverse pairwise comparisons and produces rankings that better match exhaustive oracle rankings than conventional evaluations. It also reaches high-quality rankings with about 100 comparisons and supports LLM-assisted qualitative analysis.

  • 5 Experiments: RoboArena evaluates seven generalist policies through decentralized comparisons across 4284 rollouts spanning dozens of scenes and hundreds of task instructions.The study also establishes an oracle ranking through exhaustive evaluations of all policies on all tested tasks.
  • 5.2 RoboArena Accurately Ranks Policy Performances: RoboArena pairwise rankings correlate significantly better with oracle rankings than conventional evaluations, with the task-aware TASK method performing best.Standard Elo and Bradley-Terry rankings are also evaluated, while conventional evaluations use tightly standardized tasks and environments.
  • 5.2 RoboArena Accurately Ranks Policy Performances: Progress-based PROG rankings correlate well with the oracle but can miss preferences between policies receiving identical progress scores.Evaluators sometimes preferred a policy because it acted more swiftly or confidently, motivating complementary reporting of preference- and progress-based rankings.
  • 5.3 RoboArena Evaluation is Sample Efficient: RoboArena reaches high-quality rankings within 100 pairwise comparisons and improves further as additional comparisons are collected.At that sample size, it matches conventional evaluations’ convergence speed while producing significantly more accurate rankings.
  • 5.4 Extracting qualitative policy characteristics: VLM task-category predictions are approximately 95% accurate against human expert assignments.The analysis evaluates categorization on 448 samples.
  • 5.4 Extracting qualitative policy characteristics: For most categories, comparative claims about π0-FAST-DROID in generated reports are supported by the evaluation data’s win rates.The reports compare π0-FAST-DROID with other policies across task categories.

6 Discussion

RoboArena aggregates decentralized pairwise evaluations to improve policy rankings over conventional centralized approaches and adds LLM-assisted tools for qualitative analysis. The framework is intended to support more comparable and accessible evaluation of generalist robot policies.

  • 6 Discussion: RoboArena generates more accurate policy rankings than conventional centralized evaluations while retaining high evaluation sample efficiency.The framework aggregates pairwise comparisons collected across many tasks and scenes.
  • 6 Discussion: The authors plan to open-source RoboArena and enable researchers to contribute policies and evaluations.The stated goal is to make generalist-policy evaluations more comparable.

7 Limitations

RoboArena’s scope is bounded by its DROID-focused experiments, difficulty isolating individual evaluation conditions, untested adversarial tampering, and uncertainty about long-term over-optimization.

  • Cross-embodiment: The experiments focus on the DROID platform, leaving extension to diverse robot embodiments and embodiment-specific policies for future work.The authors identify cross-embodiment evaluation as an open direction.
  • Controlled experimentation: RoboArena makes single-condition experiments difficult because evaluators face no restrictions on tasks or scenes.The framework complements smaller targeted evaluations that isolate individual generalization axes.
  • Adversarial evaluators: The evaluation scheme’s robustness to intentionally adversarial evaluators has not been investigated.Potential attacks include random preference ratings or misleading language feedback.
  • Over-optimization and Gotthart’s Law: Whether RoboArena rankings remain correlated with perceived real-world policy performance under future over-optimization remains unresolved.The authors note that current limited policy performance makes over-optimization less likely, but call for further examination.

B.2 Gradient & Hessian Derivation

This section derives the gradients and Hessians used to fit the ranking model, then summarizes baseline ranking methods and the EM implementation.

  • B.2 Gradient & Hessian Derivation: The expected complete-data objective is expanded into gradients and Hessians for each parameter block.The derivation covers policy abilities, policy-by-bucket effects, bucket effects, and related parameters.
  • B.2 Gradient & Hessian Derivation: All gradient and Hessian terms feed the clipped-Newton updates in the M-step.The implementation alternates responsibility computation in the E-step with parameter updates in the M-step.
  • B.4 Baseline Ranking Algorithms: Bradley–Terry MLE fits policy ability parameters by maximizing the likelihood of all win/loss outcomes, typically treating ties as half-wins.A gradient-ascent procedure is iterated to convergence, with L2 regularization used in practice for stability.
  • B.2 Gradient & Hessian Derivation: Algorithm 2 initializes policy and bucket parameters, computes bucket responsibilities, applies clipped-Newton updates, and returns policies sorted by ability.The procedure also updates mixture weights and tie-rate parameters.
  • B.4 Baseline Ranking Algorithms: Elo updates only the two policies involved in each comparison, enabling immediate online rating adjustments without global passes.Ties are represented as half-wins.
  • B.4 Baseline Ranking Algorithms: Partial-Success Averaging ranks policies by mean fractional success across their rollouts but ignores paired-comparison structure.The method sorts policies by descending average partial-success score.

B.6 Simulating Shifts in the Task Distribution and Policy Set

RoboArena is designed to remain useful as policies and evaluation tasks evolve. The shift experiment tests whether its ranking model remains valid as task difficulty and the policy pool change over time.

  • B.6 Simulating Shifts in the Task Distribution and Policy Set: The platform anticipates weaker policies being removed, stronger policies being added, and evaluation tasks becoming more demanding.These changes reflect a long-running benchmark whose policy set and task distribution evolve together.
  • B.6 Simulating Shifts in the Task Distribution and Policy Set: The experiment simulates distribution shift by moving from easier to harder tasks while phasing out weaker policies and adding stronger ones.Task difficulty is judged by average per-task progress, while policy strength is judged by per-policy progress scores.
  • B.6 Simulating Shifts in the Task Distribution and Policy Set: Table 2 reports RoboArena’s behavior under simulated shifts in the task distribution and policy set.

C Evaluation Data Breakdown

RoboArena’s evaluation data spans diverse task commands, object categories, and physical environments. This diversity is enabled by distributed evaluation without standardized tasks or locations.

  • C Evaluation Data Breakdown: RoboArena’s evaluation data includes diverse commands such as uncapping, unplugging, finding, and dusting.Figure 10 summarizes verb frequencies in the collected task commands.
  • C Evaluation Data Breakdown: The distributed procedure permits the task diversity needed to evaluate generalist policies across many queried tasks and object classes.The paper emphasizes that generalist evaluation requires a diverse array of tasks.
  • C Evaluation Data Breakdown: The evaluation covers a wide range of object categories being commanded for interaction.Figure 10 buckets the commanded objects by major object types.
  • C Evaluation Data Breakdown: 32 sample environments illustrate substantial heterogeneity across participating institutions.Differences include lighting, camera viewpoints, tablecloths, and objects, even when the robot’s physical location was unchanged.

D Nuances in Policy Preference Labels

RoboArena collects both partial-success estimates and pairwise preferences for the same task. Preference labels can distinguish policies even when their partial-success scores are identical or disagree.

  • D Nuances in Policy Preference Labels: Evaluators provide a rough partial-success estimate for each rollout and a preference label between policies A and B on the same task.The paired preference records which policy the evaluator liked better.
  • D Nuances in Policy Preference Labels: 11% of A/B evaluations contained preference feedback that disagreed with partial-success feedback.Disagreement included equal partial scores with a non-equal preference or opposing trends between the two signals.
  • D Nuances in Policy Preference Labels: Table 3 gives examples where equal partial-success feedback still produced a clear preference for one policy.
  • D Nuances in Policy Preference Labels: The evaluation analysis pipeline generates structured reports of policy behavior and performance across distributed manipulation episodes.It uses an automated language-model pipeline and provides video references for evaluated policies.

F Strengths and Weaknesses of the Evaluated Generalist Policies

Across diverse real-world tasks, the evaluated generalist policies handle visually grounded object manipulation and varied viewpoints relatively well, but struggle with tool use, deformable objects, precision, and complex semantic demands. Policy families show distinct strengths: autoregressive policies lead on precise language-following tasks, diffusion policies suit fluid motion, and binning policies underperform broadly.

  • Diverse viewpoints, lighting conditions, and background appearances are handled well across the evaluated generalist policies.
  • Direct object manipulation is more reliable than tool use, cloth manipulation, or complex semantic understanding.Policies perform better on straightforward, visually grounded goals but struggle with precision alignment, multi-step reasoning, and specific attribute perception.
  • Tasks involving deformable objects and tools remain key challenges, often producing partial motion or frozen behavior.These weaknesses point to gaps in generalization, robustness, and physical interaction across manipulation tasks.
  • Autoregressive policies consistently achieve higher success rates in pick-and-place, stacking, and classification tasks.The passage attributes this pattern to more precise language following.
  • Diffusion-based policies perform well in fluid or continuous motion tasks, while binning policies underperform in nearly all tasks.Diffusion policies often lag on precise language instructions, whereas binning policies frequently remain inactive and complete few tasks.
  • π0-FAST-DROID: π0-FAST-DROID often outperformed comparison policies in pick-and-place, cloth tasks, sorting, and sliding, but lagged in tool use.It commonly grasped correct objects and reached targets, while competing controllers sometimes froze, mis-grasped, overshot, or stalled; tool-use comparisons favored smoother contact from alternatives.
Loading 2506.18123v2…