PaperlayerView exact revision on arXiv

Source-linked AI summary

RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation

Yash Jangir, Yidi Zhang, Pang-Chi Lo, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki

arXiv:2510.23571v3cs.ROcs.AIcs.CVcs.LG

TL;DR

Real-world robot-policy evaluation is labor-intensive, difficult to reproduce, and hard to scale as generalist policies require broader testing. RobotArena ∞ translates real demonstration videos into simulated environments and evaluates policies with VLM scoring and human preferences. Across its evaluations, performance is sensitive to distribution shifts and perturbations, while model rankings remain consistent across conditions.

  • Problem

    Real-world robot-policy evaluation is labor-intensive, slow to scale, and difficult to reproduce, limiting broad evaluation as policies grow more generalist.

    Source: §1 · p. 1

  • Method

    RobotArena ∞ automatically translates real robot videos into simulated environments and evaluates policies with VLM scores and online human preference feedback.

    Source: §1 · p. 2

  • Results

    VLA performance drops on out-of-distribution environments and under perturbations, while model rankings remain consistent across conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations reveal that current robot policies have limited generalization across benchmarks and remain sensitive to perturbations.

    Source: §1 · p. 2, §7 · p. 10

    Current policies do not yet use wrist-camera inputs, limiting fidelity for some manipulations, and simulators struggle to reproduce fine-grained contact dynamics such as charger insertion.

    Source: §6 · p. 10

Abstract

from arXiv · show

The pursuit of robot generalists, agents capable of performing diverse tasks across diverse environments, demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. As policies expand in scope and complexity, these barriers only intensify, since defining "success" in robotics often hinges on nuanced human judgments of execution quality. We introduce RobotArena Infinity, a new benchmarking framework that overcomes these challenges by shifting vision-language-action (VLA) evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision-language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated vision-language-model-guided scoring and scalable human preference judgments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons. To measure robustness, we systematically perturb simulated environments along multiple axes, including textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for real-world-trained robot manipulation policies, addressing a critical missing capability in today's robotics landscape.

1 INTRODUCTION

RobotArena ∞ addresses the difficulty of evaluating increasingly general robot policies at scale by automating simulated evaluation and incorporating VLM scores and human preferences. Its evaluations report sensitivity to distribution shifts and perturbations, alongside consistent model rankings across conditions.

1 INTRODUCTION

  • Real-world policy evaluation is labor-intensive, difficult to reproduce, and constrained by logistics and safety as policies require broader testing.Manual scene resets, trial supervision, and scoring limit evaluation scale and frequency.Source: §1 · p. 1
  • RobotArena ∞ translates real videos into simulated environments, then evaluates policies with VLM scores and online human preference feedback.It tests in-distribution and out-of-distribution environments and introduces perturbations such as lighting, object placement, and background changes.Source: §1 · p. 2, Figure 1 · p. 2
  • The initial benchmark aggregates more than 8500 preference pairs across 100 nominal environments and hundreds of perturbations, comparing six VLAs.The paper describes this as the largest-scale robot evaluation effort to date.Source: §1 · p. 2
  • VLA performance drops on out-of-distribution environments and under perturbations, while model rankings remain consistent across conditions.The authors interpret the OOD drop as evidence that current models specialize to training environments rather than acting as true generalists.Source: §1 · p. 2
  • RobotArena ∞ automates environment construction and evaluation compared with prior efforts involving manual asset creation or hand-designed rewards.The cited comparison names BEHAVIOR and SIMPLER as prior efforts.Source: §1 · pp. 2–3
  • The contributions combine a scalable benchmarking protocol, automated video-to-simulation translation, broad VLA evaluation, and analysis of policy generalization under shifts.Source: §1 · p. 3

2 RELATED WORK

RobotArena ∞ positions simulation as an evaluation environment for policies regardless of training origin, while automating scene generation and task evaluation. Its video-based pipeline is designed to work from a single RGB image, unlike prior approaches that may require multi-view captures, curated assets, or human intervention.

Evaluating Robot Policies in The Real World

  • Real-world evaluation is difficult to compare across labs because hardware, camera placement, lighting, and workspace setups vary, while manual resets constrain scale.Existing real-world systems can also be limited in scope; AutoEval is described as supporting five tasks in three static scenes.Source: §2 · p. 3

Evaluating Robot Policies in Simulation

  • Many simulation benchmarks assume policies are trained and tested in the same environments, whereas RobotArena uses simulation only for evaluation, independent of policy training origin.RobotArena also automates scene generation, task evaluation, and systematic perturbations to assess robustness and generalization.Source: §2 · pp. 3–4

Automated Real-to-Sim Translation

  • RobotArena creates simulated environments from video demonstrations, and its real-to-sim pipeline requires only a single RGB image from a static-camera demonstration.The paper contrasts this with approaches relying on multi-view captures, curated object libraries, fiducial markers, segmentation, or calibration trajectories.Source: Figure 2 · p. 4, §2 · p. 4

3 TRANSLATING VIDEOS TO SIMULATION FOR POLICY EVALUATION

RobotArena ∞ builds physics-consistent simulations from robot demonstration videos by estimating camera pose, reconstructing task objects, completing the scene, and tuning robot control. It also perturbs generated environments along appearance and arrangement dimensions to test policy robustness.

3 TRANSLATING VIDEOS TO SIMULATION FOR POLICY EVALUATION

  • The pipeline extracts camera pose, object meshes and properties, a clean background, and PD gains to construct simulations from demonstrations without manual calibration or curated annotations.The method assumes the robot identity and access to its URDF file; its modular components can be replaced as models improve.Source: §3 · p. 4
  • Camera-to-robot pose is estimated by rendering a joint-angle-conditioned 3D Gaussian robot model and optimizing camera translation and orientation against video frames.The alignment loss combines RGB appearance, optical-flow consistency, and DINOv2 feature matching; available calibration metadata initializes optimization, otherwise a coarse grid search is used.Source: Figure 3 · p. 4, §3 · p. 5
  • Task objects are segmented, converted into textured 3D meshes, and aligned to video crops using feature correspondences lifted into metric-scale 3D.Object poses are inferred by SVD, while Gemini prompts provide physical and material properties for the simulation.Source: §3 · p. 5
  • The scene background is completed by inpainting the robot and object regions, and PD gains are tuned to align simulated end-effector trajectories with demonstrations.Beyond joint-trajectory annotations obtained through teleoperation, the method requires no additional human supervision.Source: §3 · p. 5
  • Controlled perturbations vary background textures, object locations, and RGB channels to stress-test policy generalization.The listed perturbations include object pose changes, background replacement, and color shifts applied at intensities from 0% to 100% in approximately 33% increments.Source: §3 · p. 5

4 EVALUATING ROBOT TRAJECTORIES WITH HUMANS AND VLMS

RobotArena evaluates robot trajectories using VLM task-progress scores and human comparisons of paired executions. It aggregates decisive human preferences into a global policy ranking, while the VLM score summarizes progress near the end of a trajectory.

4 EVALUATING ROBOT TRAJECTORIES WITH HUMANS AND VLMS

  • RobotArena combines automated VLM evaluation with crowdsourced human preferences to evaluate robot execution videos.Source: §4 · p. 5

4.1 EVALUATING ROBOT TASK PROGRESS SCORES WITH VLMS

  • VLMs score each frame using ordered video images and synchronized object and robot states, with the initial state serving as a zero-progress reference.The trajectory-level estimate averages scores over the final 30% of frames, a metric reported to align well with human-annotated progress scores.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 EVALUATING ROBOT PERFORMANCE WITH HUMAN PREFERENCE FEEDBACK

  • Human evaluations use double-blind pairwise comparisons of two policies in the same simulated environment, with identical initial conditions and task instructions.Evaluators select a preference or tie and provide a written rationale; the authors report that requiring justifications led to more accurate annotations.Source: §4.2 · p. 6

4.2.1 GLOBAL RANKING FROM PAIRWISE HUMAN PREFERENCES

  • The global ranking is derived from decisive pairwise preferences, excluding ties from the ranking objective.The Bradley–Terry model estimates policy ability scores from non-tied comparisons, then sorts those scores to produce the ranking.Source: §4.2.1 · p. 6

5 BENCHMARKING ROBOT POLICIES IN ROBOTARENA ∞

RobotArena ∞ evaluates policies in simulated environments derived from robot demonstrations, spanning BridgeSim, DROIDSim, and RH20TSim. Its evaluations reveal weak cross-dataset generalization and sensitivity to perturbations, while model rankings vary across settings.

5 BENCHMARKING ROBOT POLICIES IN ROBOTARENA ∞

  • RobotArena ∞ includes BridgeSim, DROIDSim, and RH20TSim, built from demonstrations in BridgeV2, DROID, and RH20T, respectively.The benchmark includes in- and out-of-distribution environments and tasks selected according to policies’ demonstration training sets.Source: §5 · p. 7

5.1 ROBOT BENCHMARKING RESULTS IN ROBOTARENA ∞

  • Human preferences and VLM task-progress scores produce exactly matching rankings, with π0 and X-VLA preferred more often than the other evaluated policies.The evaluations include BridgeSim, DROIDSim, and RH20TSim, plus perturbed BridgeSim environments.Source: §5.1 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
  • Policies perform substantially worse in environments derived from datasets outside their training data, including DROID and RH20T.The evaluated arenas derive from BridgeV2, DROID, and RH20T demonstrations; only SpatialVLA’s backbone was pretrained on RH20T.Source: §5 · p. 7, §5.1 · p. 8
  • π0 and X-VLA top BridgeSim and its perturbed variants, but RoboVLM scores 19.05% on RH20TSim while X-VLA scores 0.00%.The results distinguish leading architectures in BridgeSim without showing universal superiority across datasets.Source: §5.1 · p. 8
  • The authors suggest π0 and X-VLA may gain implicit 3D structure from wrist-camera pretraining, yielding a more robust spatial prior than SpatialVLA’s explicit 3D inductive biases.They propose cross-view consistency learned from raw multi-view data as a possible explanation.Source: §5.1 · pp. 8–9
  • Policies with stronger VLM backbones are more resilient to color perturbations, while background perturbations and randomized object poses degrade performance across all evaluated policies.The authors suggest stronger backbones rely more on invariant structural cues; they also report that 3D structural modeling provides some resilience to pose changes.Source: §5.1 · p. 9

5.2 POLICY RANKING IN ROBOTARENA ∞VERSUS THE REAL WORLD

  • On the single task “Put the carrot in the plate,” RoboVLM and SpatialVLA succeeded in both simulated and real settings, while Octo failed in both.The authors note that broader real-world correlation estimates would require difficult, bias-free multi-task evaluations recreating precise scene structures, textures, and camera arrangements.Source: §5.2 · p. 9, Figure 8 · p. 9
  • All evaluated VLAs score substantially higher on four reproduced SIMPLER environments than on BridgeSim’s 70 environments, suggesting SIMPLER may overestimate performance because of limited, biased scenario selection.The authors characterize BridgeSim’s broader coverage and diversity as making it a more rigorous and reliable benchmark.Source: §5.2 · p. 9, Figure 9 · p. 9

6 LIMITATIONS AND FUTURE DIRECTIONS

The benchmark uses everyday end-users as evaluators rather than domain experts or robotics researchers. Its current scope is constrained by missing wrist-camera inputs and by simulators’ difficulty reproducing fine-grained contact dynamics.

6 LIMITATIONS AND FUTURE DIRECTIONS

  • Evaluators are everyday end-users, not domain experts or robotics researchers.The authors describe them as the audience robots are ultimately intended to serve.Source: §6 · p. 9
  • Policies do not yet use wrist-camera inputs, restricting the fidelity of certain manipulations; the authors are extending the pipeline to generate complete 3D interactive environments for multi-view observations.Source: §6 · p. 10
  • Current simulators struggle with fine-grained contact dynamics, making tasks such as inserting a charger into a socket difficult to reproduce faithfully.The authors identify this as a direction for future research and expect the benchmark to benefit from advances in simulation, physics engines, and environment generation.Source: §6 · p. 10

7 CONCLUSION

RobotArena ∞ scales evaluation through automated reality-to-simulation translation and online human preference feedback. Evaluations across hundreds of environments and more than 8,500 human preferences reveal generalization gaps, perturbation sensitivity, and architecture differences.

7 CONCLUSION

  • RobotArena ∞ automatically constructs simulated environments from real-world videos using vision-language models, 2D-to-3D generative models, and differentiable rendering.The framework combines reality-to-simulation translation with online human preference feedback.Source: §7 · p. 10
  • Evaluations across hundreds of environments and over 8,500 human preference judgments reveal limited cross-benchmark generalization, perturbation sensitivity, and systematic differences across architectures.The authors describe this as the most extensive study of policy robustness and generalization to date.Source: §7 · p. 10
  • The authors plan to expand tasks and real-world data sources, leverage improvements in physics engines and real-to-sim translation, and release benchmark environments and evaluation code.They aim to provide an open, extensible platform for rigorous policy evaluation.Source: §7 · p. 10

B INFERRING CAMERA-ROBOT POSE TRANSFORMATION VIA ANALYSIS-BY-SYNTHESIS

The pipeline aligns rendered robot motion with demonstration video, reconstructs task-relevant objects and their poses, and prepares simulated scenes with matched backgrounds and robot dynamics. It also creates controlled environment variants, including backgrounds that change while foreground objects remain fixed.

B INFERRING CAMERA-ROBOT POSE TRANSFORMATION VIA ANALYSIS-BY-SYNTHESIS

  • The camera-to-robot transform is estimated by minimizing RGB, optical-flow, and DINOv2 feature losses between rendered images and video frames.Dataset joint angles drive rendering; the transform aligns the robot coordinate system with the camera.Source: B INFERRING CAMERA-ROBOT POSE… · p. 15

C RELEVANT OBJECT SEGMENTATION, 3D RECONSTRUCTION, AND MATERIAL PROPERTY ESTIMATION

  • Gemini segments the robot and foreground objects; super-resolved object crops are converted into textured 3D meshes, with inferred physical properties and scene-aligned scale and placement.Task-conditioned prompts can identify target and destination objects.Source: C RELEVANT OBJECT SEGMENTATION, 3D… · p. 15, §C.1 · p. 16

C.2 OBJECT POSE ESTIMATION

  • Object poses are recovered by matching rendered mesh views to masked image crops, lifting keypoint correspondences into 3D, and estimating a rigid transform with SVD.Metric depth is obtained by scaling MoGE relative-depth estimates against the simulated robot arm’s ground-truth depth; rendered views vary camera elevation and azimuth.Source: §C.2 · pp. 16–18

D BACKGROUND INPAINTING

  • A static background is generated by inpainting the robot and object regions in the first video frame with LaMa.The appendix visualizes inpainting results for four Bridge scenes.Source: D BACKGROUND INPAINTING · p. 18

E SYSTEM IDENTIFICATION

  • System identification tunes PD controller gains to reduce simulated-versus-recorded end-effector trajectory error; Figure 11 shows closer XY-plane alignment after identification.The process excludes the gripper because its dataset state is binary open/close rather than continuous.Source: E SYSTEM IDENTIFICATION · pp. 18–19, Figure 11 · p. 19

F BridgeSim ENVIRONMENT VISUALIZATIONS

G ENVIRONMENT PERTURBATIONS

G.1 BACKGROUND CHANGE (∆BG)

  • Each scene receives five inpainted-background variants while all foreground objects remain unchanged.Figure 14 illustrates the background-flipped variants.Source: §G.1 · p. 19

G.2 COLOR SHIFT (∆COLOR)

The perturbations vary background color or object arrangement while holding other scene elements fixed. Color variants blend an RGB-to-BGR remapping at specified intensities; pose variants reassign assets among existing positions.

G.2 COLOR SHIFT (∆COLOR)

  • Background color is perturbed by blending each pixel’s RGB vector with its BGR-swapped version at four intensities: 0%, 33%, 66%, and 100%.Foreground objects, lighting, and dynamics remain unchanged; Figure 15 illustrates the variants.Source: §G.2 · pp. 20–21, Figure 15 · p. 21

G.3 OBJECT POSE CHANGE (∆OBJPOSE)

  • Object-pose perturbation generates N scene variants by randomly permuting N assets among their original world-space positions, changing only their arrangement.Figure 16 illustrates the object-position perturbation.Source: §G.3 · p. 21, Figure 16 · p. 22

H AUTOMATED TASK PROGRESS SCORING WITH VLMS

The automated evaluator uses Gemini to assign frame-level task-progress scores from task instructions, visual observations, robot and object state, and action history. Its prompt defines how scores track completed subgoals, reversals, and severe object damage.

H AUTOMATED TASK PROGRESS SCORING WITH VLMS

  • Gemini receives the task instruction, structured robot and object states, and visual frames with action history to estimate task progress.The evaluator compares initial and current states using both visual evidence and structured state transitions.Source: H AUTOMATED TASK PROGRESS SCORING WITH… · p. 22
  • The prompt sets initial progress to 0 and completion of all necessary steps to 100, with scores rising for meaningful subgoals and falling when progress is undone or severe object damage occurs.Source: H AUTOMATED TASK PROGRESS SCORING WITH… · p. 23
  • Figure 17 shows example VLM-generated task-evaluation curves on the base environment.Its right panels plot completion score against frame ID; left panels show representative low- and high-progress frames.Source: H AUTOMATED TASK PROGRESS SCORING WITH… · p. 23

I VLM EVALUATIONS

In RH20TSim, Octo’s significantly higher VLM score is attributed to a minimal motion that succeeds by coincidence because the target lies in the arm’s path.

I VLM EVALUATIONS

  • Octo scores significantly higher in RH20TSim because it slowly lowers its arm onto the workbench and coincidentally reaches the target, outperforming more exploratory but unsuccessful baseline policies.Source: I VLM EVALUATIONS · p. 23

J HUMAN EVALUATIONS

The study validates automated VLM task-success scoring against human judgments and derives global policy rankings from pairwise preferences. VLM scores align more strongly with human evaluations when structured state information is provided.

J HUMAN EVALUATIONS

  • Annotators described each attempt and chose the better of two same-task policy videos or a tie, after passing a quiz requiring at least 8 of 10 correct judgments.The videos were presented side-by-side.Source: J HUMAN EVALUATIONS · p. 25, Figure 20 · p. 25
  • The validation study collected 8,749 pairwise comparisons across 70 environments from participants on Amazon Mechanical Turk.Participants were assigned random subsets of comparisons.Source: J HUMAN EVALUATIONS · pp. 23, 25
  • Robust sandwich variance estimates quantify uncertainty in policy abilities, and non-overlapping confidence intervals support a confident preference between policies.The ranking procedure can use the estimated abilities or centered log-scores.Source: GLOBAL RANKING FROM PAIRWISE PREFERENCES · p. 26

K MEASURING CORRELATION BETWEEN VLM AND HUMAN SCORES

  • VLM scores correlate positively with human evaluations across four input settings, with strongest alignment when both robot and object states are provided.Figure 21 compares images only, robot state only, object state only, and combined robot and object states.Source: Figure 21 · p. 26, K MEASURING CORRELATION BETWEEN VLM… · p. 26
Loading 2510.23571v3…