PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, Jonathan Tremblay, Kanav Arora, Kirsty Ellis, Luca Macesanu, Marcel Torne Villasevil, Matthew Leonard, Meedeum Cho, Ozgur Aslan, Shivin Dass, Jie Wang, William Reger, Xingfang Yuan, Xuning Yang, Abhishek Gupta, Dinesh Jayaraman, Glen Berseth, Kostas Daniilidis, Roberto Martin-Martin, Youngwoon Lee, Percy Liang, Chelsea Finn, Sergey Levine

arXiv:2506.18123v2cs.ROcs.LG

TL;DR

Generalist robot policies are difficult to compare across varied tasks using benchmarks tied to standardized settings. RoboArena crowdsources double-blind pairwise evaluations across a distributed network and produces more accurate rankings than conventional centralized evaluation.

  • Problem

    Conventional robot benchmarks use tightly standardized tasks and environments, limiting evaluation of generalist policies across a broad range of settings.

    Source: §1 · p. 1

  • Method

    RoboArena aggregates evaluators’ double-blind pairwise policy comparisons, conducted on tasks and scenes they choose, into global policy rankings.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across varied tasks and scenes, RoboArena ranked generalist policies more accurately than conventional centralized evaluations, based on 612 pairwise comparisons against exhaustive-evaluation oracle rankings.

    Source: §1 · p. 3, §5.2 · p. 9

  • Takeaways & Limitations

    The findings support distributed evaluation as a sample-efficient alternative for comparing generalist policies across varied tasks and scenes.

    Source: §5.3 · p. 10, §6 · p. 10

    The experiments focus on the DROID platform, leaving evaluation across diverse robot embodiments for future work.

    Source: §7 · p. 11

Abstract

from arXiv · show

Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies.

1 Introduction

RoboArena replaces fixed, centralized robot evaluation with decentralized, double-blind pairwise comparisons on evaluator-chosen tasks and scenes. The paper reports more accurate policy rankings than conventional centralized evaluation at the same episode efficiency.

1 Introduction

  • Conventional benchmarks standardize a small set of tasks and scenes, limiting coverage of generalist policies and making large-scale real-world comparisons difficult to reproduce and maintain.Challenges include reproducing scene layouts and lighting, maintaining centralized robot fleets, and accounting for differences between robots.Source: §1 · pp. 1–2
  • By embracing physical-world variation and requiring matched conditions only within each policy pair, RoboArena aims to scale evaluation across open-ended tasks and environments without centralized standardization.The paper instantiates the framework on the DROID robot platform [1].Source: §1 · pp. 2–3
  • RoboArena uses decentralized evaluators to perform double-blind pairwise policy comparisons on tasks and scenes they choose, then aggregates preferences into global rankings and qualitative policy characteristics.Evaluators also provide free-form explanations; qualitative characteristics are extracted through LLM-assisted analysis.Source: §1 · pp. 2–3
  • Across 7 universities, RoboArena evaluated 7 policies in 612 pairwise comparisons and 4,284 episodes, producing more accurate rankings than centralized evaluation at the same episode efficiency.The comparison was against an oracle ranking computed by exhaustively evaluating all policies on all tested tasks.Source: §1 · p. 3

2 Related Work

Robot evaluation trades reproducibility against broad real-world task coverage: simulation can misrepresent real-world performance, while centralized evaluation typically restricts tasks and environments. RoboArena proposes decentralized, crowd-sourced pairwise comparisons to evaluate generalist policies across a broader distribution of tasks and environments.

Simulated and real-world robot evaluation

Generalist robot policies

  • Generalist policies trained on diverse robot-experience datasets [37, 38, 2, 39, 40, 3, 41, 42, 43, 44, 45, 5, 6, 46, 47, 48, 49, 50, 2, 1, 51, 52, 53, 54] can perform tasks across many environments out of the box [4] [5], challenging conventional evaluation to scale.This scaling challenge motivates distributed evaluation across broad task and environment distributions.Source: §2 · p. 3

Crowd-sourced benchmarks

  • Crowd-sourced benchmarks span language, generative image and video modeling, and legal model training [15] [16]; unlike Dasari et al.’s institution- and task-specific retraining benchmark [58], RoboArena evaluates generalist policies out of the box across broader task and environment distributions.Dasari et al.’s approach remained limited to a small number of tested environments and tasks because policies had to be retrained for each institution and task.Source: §2 · p. 3

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

RoboArena uses decentralized, double-blind pairwise evaluations on evaluator-chosen tasks and scenes, then aggregates preferences into policy rankings and uses evaluation data to extract qualitative policy characteristics.

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

  • Decentralized, double-blind comparisons across freely chosen tasks and scenes broaden coverage while making evaluation scalable, robust to individual influence, and adaptable to changing policy capabilities.The approach avoids the difficult standardization of tasks, scenes, and initial conditions required by conventional real-world evaluations.Source: §3 · pp. 3–4

3.1 Policy Evaluation Protocol

  • Evaluators receive a randomly sampled policy pair without identities, choose a task and scene, run both policies sequentially under matched conditions, and submit progress, preference, and written feedback.Evaluations outside each pairwise comparison can run asynchronously at any time or place.Source: §3.1 · p. 4

3.2 Computing Global Policy Rankings

  • The ranking model extends Bradley-Terry with task difficulty, task probabilities, and policy-task offsets to account for task-dependent preferences, learning these parameters from preference data without auxiliary task information.Standard Bradley-Terry treats task effects as noise, although task difficulty and policy specialization can alter pairwise preferences.Source: §3.2 · p. 5

3.3 Extracting Qualitative Policy Characteristics

  • A VLM categorizes tasks and scene conditions, and an LLM synthesizes preferences, evaluator feedback, and rollout information into policy reports with video evidence for reported strengths and weaknesses.The tool is intended to help extract qualitative characteristics such as language following and multi-step task performance from distributed evaluation data.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

4 The DROID-RoboArena Evaluation System

DROID-RoboArena uses a distributed network of real-robot evaluators and remote policy servers to support community evaluation. Its design combines open participation with policy-safety checks and incentives for contributing evaluations.

4 The DROID-RoboArena Evaluation System

  • The system uses DROID’s Franka Panda arm, gripper, and stereo cameras; its dexterity, mobile adjustable table, open-source dataset [1], and institutional deployments support diverse, distributed evaluation.The approach can extend to other robot embodiments and potentially cross-embodiment policies [2].Source: §4 · p. 6

4.1 RoboArena System Design

  • DROID-RoboArena links remote policy servers, distributed robot clients, a results database, and a central server that assigns evaluations and computes policy rankings.The central server also tracks policy-pool changes and cancels evaluations after timeouts; clients upload instructions, scores, preferences, feedback, and rollout videos.Source: §4.1 · pp. 6–7, Figure 4 · p. 7
  • Remote policy hosting lets evaluators share inference resources, lowers client hardware requirements, and lets policy owners control execution without sharing proprietary weights.The authors found inference latency negligible for typical static DROID manipulation tasks, while noting dynamic tasks may require partially local inference.Source: §4.1 · p. 7

4.2 Open-Sourcing RoboArena: Interfaces, Safety and Incentives

  • Researchers can submit policies and contribute evaluations; open resources can let researchers without physical robots train and evaluate real-world generalist policies.The website supports browsing evaluations and leaderboards for all policies and for open-source policies, which include public-data training and reproduction details.Source: §4.2 · p. 7
  • New policy servers undergo format checks and supervised test runs across scenes and tasks before policies enter the evaluation pool; evaluation credits reward comparisons and can fund requests for comparisons of one’s own policies.Institutions also sponsor a free evaluation budget for users unable to contribute evaluations themselves.Source: §4.2 · p. 8

5 Experiments

RoboArena’s distributed pairwise evaluations produce more accurate policy rankings than conventional fixed-task evaluations, reach high-quality rankings with 100 comparisons, and support qualitative analysis of policy performance. The experiments cover seven public DROID policies and assess ranking accuracy, sample efficiency, and report quality.

5.1 Experimental Setup

  • The experiments evaluate seven publicly available generalist DROID policies based on PaliGemma or π0 and compare RoboArena with the DROID procedure’s 17 tasks and 44 episodes per policy.The RoboArena pool includes different action representations, and its oracle ranking uses exhaustive policy evaluations on tested tasks; the full evaluation covers 4,284 policy rollouts across dozens of scenes and hundreds of task instructions.Source: §5.1 · p. 8, §5.2 · p. 8

5.2 RoboArena Accurately Ranks Policy Performances

  • RoboArena’s task-aware rankings are more accurate than conventional evaluations, Elo, and Bradley–Terry, while conventional fixed-task evaluations do not reliably estimate generalist-policy performance.The study evaluates rankings against an exhaustive oracle using Pearson correlation and Mean Maximum Rank Violation (MMRV).Source: Figure 6 · p. 9, §5.2 · p. 9
  • The oracle and RoboArena rankings align with prior-work intuitions: expressive action representations outperform simple binning, and discrete action tokenization outperforms diffusion policies in language-conditioned evaluations.The passage identifies FAST and FSQ as discrete tokenization methods and Flow and π0-DROID as diffusion policies.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Progress-score rankings correlate well with the oracle, but preference feedback captures distinctions such as speed or confidence when both policies receive the same progress score.The authors recommend reporting progress-based and preference-based rankings as complementary measures.Source: §5.2 · p. 9, Figure 6 · p. 9

5.3 RoboArena Evaluation is Sample Efficient

  • Within 100 pairwise comparisons, RoboArena converges to high-quality rankings, matching conventional evaluations’ convergence speed while producing significantly more accurate rankings.Ranking quality improves further as comparisons accumulate.Source: Figure 7 · p. 10, §5.3 · p. 10

5.4 Extracting qualitative policy characteristics

  • VLM task-category predictions are approximately 95% accurate, and most comparative claims in π0-FAST-DROID’s generated report are supported by evaluation win rates.The category predictions were compared with human-expert assignments; the report analysis examined π0-FAST-DROID’s performance against other policies across task categories.Source: §5.4 · p. 10, Figure 8 · p. 10

6 Discussion

RoboArena uses decentralized pairwise evaluations across diverse tasks and scenes to produce more accurate policy rankings than conventional centralized approaches while retaining high sample efficiency. The authors also introduce prototype LLM-assisted analysis tools and plan to open-source the framework and enable community contributions.

6 Discussion

  • The prototype LLM-assisted analysis tools make claims about π0-FAST-DROID that are supported by evaluation-data win rates.Figure 9 compares claims that π0-FAST-DROID outperforms, matches, or underperforms other policies with the evaluation win rates.Source: Figure 9 · p. 10, §6 · p. 10
  • RoboArena aggregates pairwise evaluations across diverse tasks and scenes to produce more accurate policy rankings than conventional centralized approaches while retaining high evaluation sample efficiency.Evaluations are conducted by a decentralized network of evaluators.Source: §6 · p. 10
  • The authors plan to open-source RoboArena and give researchers access to contribute policies and evaluations.They aim to make evaluations of generalist robot policies more comparable.Source: §6 · p. 10

7 Limitations … B.1 Model Definition

The paper identifies limitations in RoboArena’s platform coverage, experimental control, resistance to adversarial evaluators, and vulnerability to over-optimization. Its appendices describe the policy-ranking procedure and its latent-task model.

7 Limitations

  • RoboArena’s experiments use the DROID platform, so extending the approach to diverse robot embodiments and policies evaluatable only on specific embodiments remains future work.Source: §7 · p. 11
  • Because RoboArena allows decentralized evaluation without restricting tasks or scenes, it is difficult to vary one experimental condition at a time.The paper describes RoboArena as complementary to smaller-scale evaluations focused on individual generalization axes [72].Source: §7 · p. 11
  • The robustness of RoboArena’s distributed, double-blind evaluations to intentional tampering by evaluators has not been investigated.Examples include random preference ratings and intentionally misleading language feedback.Source: §7 · p. 11
  • The authors call for testing whether RoboArena rankings remain correlated with perceived real-world policy performance as evaluations risk over-optimization.They note that this risk is inherent to measures, has affected crowd-sourced evaluations, and may currently be less likely given robotics policies’ limited performance.Source: §7 · p. 11

A Contributions

  • Contributions include system design and experiment design by Pranav Atreya, Karl Pertsch, and Tony Lee.Source: A Contributions · p. 18

B Policy Ranking EM Procedure

  • Appendix B expands on the policy-ranking algorithm by deriving gradients and Hessians, presenting EM pseudocode, and outlining variants and hyperparameters.It also describes a modification that uses partial-success information when available.Source: B Policy Ranking EM Procedure · p. 18

B.1 Model Definition

  • The ranking model treats each A/B trial as a win, loss, or tie and infers latent task-difficulty buckets rather than observing task identifiers.Source: §B.1 · p. 18
  • The model combines global policy ability, policy-specific bucket offsets, bucket difficulty, and a Davidson tie parameter to estimate outcomes using a sigmoid link.It assumes policies solve tasks independently given the latent bucket and marginalizes over buckets.Source: §B.1 · pp. 18–19

B.2 Gradient & Hessian Derivation … B.5 Hyperparameter Table

These appendices derive the parameter updates used by the clipped-Newton M-step, describe an extension using partial-success information, and summarize baseline ranking algorithms and EM hyperparameters.

B.2 Gradient & Hessian Derivation

  • The full parameter set’s expected complete-data objective is differentiated for the clipped-Newton M-step, with θp contributions limited to trials involving policy p and ψp,t terms specific to bucket t.The Hessian for θp accumulates negative logistic variances; τt enters with a minus sign for both compared policies.Source: §B.2 · pp. 19–20
  • The derived gradient and Hessian terms are used in the clipped-Newton updates of Algorithm 2.Source: §B.2 · p. 20

B.3 Including Partial-Success Information

  • When partial-success scores are available, the method adds a Gaussian likelihood term in the E-step and corresponding gradient and Hessian contributions in the M-step.The modified pseudocode adds two E-step lines and augments each gradient and Hessian.Source: §B.3 · p. 20

B.4 Baseline Ranking Algorithms

  • Offline Bradley–Terry MLE fits ability parameters to win/loss outcomes by gradient ascent until convergence, with L2 regularization used in practice to stabilize training.Ties can be handled with small modifications and are typically treated as half-win/half-loss.Source: §B.4 · p. 20
  • Algorithm 2 initializes parameters, computes E-step responsibilities, applies clipped-Newton M-step updates, and returns policies sorted by descending θp.Source: §B.4 · p. 21
  • Elo-style online updates adjust only the two policies’ ratings after each comparison, using a constant K-factor and no global passes over the dataset.Ties use yAB = 0.5.Source: §B.4 · p. 21
  • Partial-Success Averaging ranks policies by average fractional success across their rollouts, using continuous feedback but ignoring paired-comparison structure.Source: §B.4 · p. 21

B.5 Hyperparameter Table

B.6 Simulating Shifts in the Task Distribution and Policy Set

RoboArena tests whether its rankings remain valid as policies and evaluation tasks evolve over time. Under simulated shifts toward harder tasks and stronger policies, its evaluations retain significantly higher correlation with oracle scores than regular single-laboratory evaluations.

B.6 Simulating Shifts in the Task Distribution and Policy Set

  • As a long-running community benchmark, RoboArena is expected to see weaker policies replaced by stronger ones and evaluators adapt tasks to test new capabilities.The policy set and evaluation tasks may therefore evolve over time.Source: §B.6 · p. 22
  • The ranking test simulates increasingly challenging task distributions while weaker policies leave the pool and stronger policies are added.Task difficulty and policy strength are judged by average per-task and per-policy progress scores, respectively.Source: §B.6 · p. 22
  • Under simulated shifts, RoboArena evaluations retain significantly higher correlation with oracle scores than “Regular” evaluations in a single laboratory.The shifts phase out weaker policies, add stronger ones, and move from easier tasks toward harder ones.Source: Table 2 · p. 22, §B.6 · p. 22

C Evaluation Data Breakdown

RoboArena’s evaluation data spans diverse task commands, object categories, and scenes. Evaluators’ freedom to choose tasks and environments permits this diversity without requiring standardized scenes.

C Evaluation Data Breakdown

  • Figure 10 shows diverse task-command verbs and object categories across RoboArena evaluation episodes.The commands include actions such as uncapping, unplugging, finding, and dusting.Source: C Evaluation Data Breakdown · p. 22, Figure 10 · p. 23
  • Figure 11 depicts 32 sample environments, illustrating heterogeneity across locations and within the same physical robot location.Lighting, camera viewpoints, tablecloths, and objects could vary because exact scenes and tasks did not need to be standardized.Source: C Evaluation Data Breakdown · p. 22, Figure 11 · p. 24

D Nuances in Policy Preference Labels · E Policy Performance Reports

RoboArena preference labels can reveal differences that partial-success scores miss, while automated reports summarize each policy’s behavior across distributed evaluation episodes.

D Nuances in Policy Preference Labels

  • Evaluators provide both rough partial-success estimates and preference labels, which can distinguish policies even when their partial-success feedback is equal.Table 3 illustrates examples of A/B pairs with equal partial-success feedback but differing preferences.Source: D Nuances in Policy Preference Labels · p. 24, Table 3 · p. 25
  • 11% of A/B evaluations had preference feedback that disagreed with partial-success feedback, showing that preferences provide a richer signal for policy evaluation.Disagreement arose when partial-success feedback was equal but preference differed, or when the two signals showed opposing trends.Source: D Nuances in Policy Preference Labels · p. 24

E Policy Performance Reports

  • The automated pipeline generates detailed reports summarizing each policy’s behavior and performance across manipulation tasks using distributed evaluation data.Source: E Policy Performance Reports · p. 25
  • For each policy, the pipeline prompts OpenAI’s o3-2025-04-16 model with a template to generate a structured performance report.Source: E Policy Performance Reports · p. 25
  • The authors present a representative full report for π0-FAST-DROID with video references removed, then summarize full reports using a separate prompt.Reports for all evaluated policies, with references to specific evaluation videos, are available on the RoboArena leaderboard.Source: E Policy Performance Reports · pp. 25–26
  • The full reports for all evaluated policies include references to specific evaluation episodes and are available on the RoboArena leaderboard.Source: E Policy Performance Reports · p. 26

F Strengths and Weaknesses of the Evaluated Generalist Policies

Across diverse real-world tasks, evaluated generalist policies handle varied viewpoints and straightforward, visually grounded manipulation better than tool use, deformable-object tasks, and tasks requiring precision or complex understanding. Policy-family strengths differ by task, while reliable generalization remains an open challenge.

Cross-policy strengths and weaknesses

  • Policies generally solve direct object manipulation more reliably than tool use, cloth manipulation, or tasks requiring complex semantic understanding.They operate across diverse viewpoints, lighting conditions, and background appearances; Figure 12 is cited in the passage for the task comparison.Source: F Strengths and Weaknesses of the… · p. 26
  • Straightforward, visually grounded goals suit current policies, while precision alignment, multi-step reasoning, and specific attribute perception remain difficult.Deformable-object and tool-use tasks often produce partial motion or frozen behavior, exposing gaps in generalization, robustness, and physical interaction.Source: F Strengths and Weaknesses of the… · p. 26

Policy-family comparisons

  • Autoregressive policies excel at pick-and-place, stacking, and classification; diffusion policies suit fluid motion but lag on precise language instructions, while binning policies underperform broadly.The passage attributes autoregressive strengths to more precise language following and notes frequent inactivity and low task completion for binning policies.Source: F Strengths and Weaknesses of the… · p. 26

Related papers

Loading 2506.18123v2…