RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation

I. INTRODUCTION

• ROBOEVAL combines efficiency, safety/stability, and coordination metrics with outcome measures of task progression and binary success.The framework is intended to assess both how a task is completed and whether it is completed.Source: §I · p. 1
• Behavioral metrics are proposed to distinguish policies with similar success but different execution quality, and to remain informative across task and scene variation.The motivating criteria are skill relevance, discrimination, and consistency.Source: §I · p. 1
• The benchmark includes eight bimanual tasks, controlled variations, over three thousand expert demonstrations, and a modular simulation platform.Tasks are instrumented with behavioral and outcome metrics, and the framework is tested with state-of-the-art visuomotor policies.Source: §I · p. 1, Figure 1 · p. 2
• Experiments report that behavioral metrics distinguish execution quality beyond success rates, while outcome metrics expose structured failure modes.The study examines metric stability under variation, discrimination between policies with similar success, and relationships to task success.Source: §I · p. 1

II. RELATED WORKS

• Existing benchmarks provide limited insight into when and why policies fail, especially in coordinated bimanual settings; real-world evaluations are often restricted to single tasks.ROBOEVAL complements prior benchmarks with structured, fine-grained evaluation across diverse tasks.Source: §II · p. 2
• Existing evaluation approaches are described as fragmented, with isolated measures of success or robustness rather than a unified account of progression, execution quality, and failure modes.Prior work includes metrics surveys and benchmarks for generalization under environmental variation.Source: §II · p. 2

III. ROBOEVAL BENCHMARK

• ROBOEVAL's first release spans service, warehouse, and industrial domains with eight tasks, systematic spatial variations, 3,000 demonstrations, and behavioral and outcome metrics.A modular simulation framework supports reproducible evaluation and extension.Source: §III · p. 2

Design Philosophy

• The benchmark's design principles are diversity, interpretability, and extensibility, with standardized interfaces for adding tasks, variations, and metrics.Its task range spans different temporal complexity and coordination demands, while its metrics support analysis of failure modes and sensitivity to variation.Source: Design Philosophy · pp. 2–3

III. ROBOEVAL BENCHMARK

• ROBOEVAL's base task suite contains eight bimanual tasks with structured variations and behavior logging and task-stage definitions.The benchmark comparison covers task design, evaluation, and data; the base-task table summarizes variations and demonstration statistics.Source: Figure 2 · p. 3, Table I · p. 3, Table II · p. 3

Tasks

• Tasks span short-horizon objectives and long-horizon, multi-step activities, with stagewise progress tracking.The suite is intended as a systematic testbed across settings and skill requirements.Source: Tasks · p. 3
• Each task is a goal-conditioned episode defined by state, action, transition dynamics, goal, initial-state distribution, and binary success set.Parameterized variants alter scene layout or semantic content; expert demonstrations are collected by human teleoperation.Source: Tasks · p. 3
• The task suite ranges from single-arm actions and long-horizon interactions to tightly coordinated dual-arm motions, probing coordination, precision, and smooth execution.The cited taxonomy is attributed to.Source: Tasks · p. 3
• Structured spatial variations are designed to probe robustness and generalization while preserving task semantics; future extensions may add visual, lighting, or physical-property changes.The initial release focuses on spatial variation.Source: Tasks · p. 3
• A modular task-generation pipeline supports integration of new or external tasks with minimal code overhead and built-in behavioral and outcome metrics.The benchmark comparison highlights tiered variations, fine-grained metrics, and expert demonstrations.Source: Tasks · pp. 3–4

Task and Dataset Statistics

• ROBOEVAL provides 3,000 VR-collected expert demonstrations across eight core bimanual skills, spanning prehensile and non-prehensile actions.The dataset captures proprioception, visual observations, and scene-annotated interaction states; natural demonstration variability includes different strategies, trajectories, and coordination styles.Source: Task and Dataset Statistics · p. 4

• The metric suite combines behavioral measures across efficiency, safety/stability, and coordination with task progression and binary success.Source: Evaluation Scoring · p. 4
• Efficiency is measured through trajectory length and completion time, plus cumulative path length in joint, Cartesian, and orientation spaces.Source: Evaluation Scoring · p. 4
• Joint and Cartesian jerk measure motion smoothness, while self-collisions, environment collisions, and object slips track contact stability and control reliability.Source: Evaluation Scoring · p. 4
• Coordination measures address spatial alignment and temporal synchronization between arms; lower ∆z and ∆v indicate better spatial and temporal coordination.Source: Evaluation Scoring · p. 4
• Task progression logs binary indicators for discrete task phases, while overall success is the proportion of evaluation episodes completed.Source: Evaluation Scoring · p. 4

• The study is organized around whether metrics add insight beyond binary success, when they are most informative, and how patterns vary by domain and complexity.Source: §IV · p. 4
• The evaluation covers representative policies across diverse bimanual tasks and systematically varied configurations.Source: §IV · p. 4
• The analysis considers both overall performance and the diagnostic value of different metrics.Source: §IV · p. 4

• Five policy architectures are evaluated: ACT, Diffusion Policy, GR00T N1.6, X-VLA, and π0.5, with human teleoperation demonstrations as an upper-bound reference.Source: Implementation Details · p. 5
• ACT and Diffusion Policy use official LeRobot implementations with ResNet-18 encoders and 16-step action horizons, trained as single-task specialists.Source: Implementation Details · p. 5
• The main experiments cover eight tasks, each with three to four variations: static layouts, position perturbations, and orientation perturbations.Source: Implementation Details · p. 5

RQ1

• Across eight tasks, the top-success model leads in only 34 of 104 other metric comparisons (33%), showing success is a poor proxy for overall quality.In Rotate Valve, ACT records 1.0 vs. 3.0 m/s3 Cartesian and joint jerk, despite leading in success on only one task.Source: RQ1: To what extent do behavioral and… · p. 5
• In Pack Box, Pi-0.5 and ACT have comparable success (0.54 vs. 0.48), yet ACT achieves 4× lower jerk and 2.7× shorter paths.Source: RQ1: To what extent do behavioral and… · p. 5
• Path-length, jerk, and temporal metrics form tight clusters, but the full metric suite spans distinct branches; completion time clusters with Success and TP despite a negative correlation.The path-length metrics have ρ ≥ 0.95, jerk metrics ρ = 0.96, and temporal metrics ρ = 0.94; completion time clusters with Success and TP at distance 0.42 despite ρ ≈−0.6.Source: RQ1: To what extent do behavioral and… · p. 5
• Policies are slower and use longer trajectories than human demonstrations, yet often show lower Cartesian jerk, which reflects smoothing from learned action generation rather than better control.The fastest policies are 3–10× slower and trajectories 2–9× longer; lower jerk occurs in 26 of 48 Cartesian comparisons and 21 of 48 joint comparisons.Source: RQ1: To what extent do behavioral and… · p. 5

RQ2: Under what conditions are the metrics most informative?

• The highest-success model leads on only 33% of the remaining metrics across all eight tasks.Safety metrics also vary: three to four different models claim top rank across tasks for slip count and environment collisions.Source: RQ2: Under what conditions are the… · p. 5
• In Pack Box, ACT has 4× lower Cartesian jerk (6.8 vs. 27.6 m/s3) and 2.7× shorter Cartesian paths (1.7 vs. 4.6 m) than Pi-0.5 despite overlapping success.Success rates are 0.54 ± 0.14 for Pi-0.5 and 0.48 ± 0.14 for ACT.Source: RQ2: Under what conditions are the… · p. 5
• In Lift Tray, ACT has 5× lower Cartesian jerk (6.1 vs. 31.3 m/s3) and 3.5× shorter paths (1.4 vs. 4.8 m) than Pi-0.5 despite overlapping success.Success rates are 0.32 ± 0.14 for Pi-0.5 and 0.26 ± 0.14 for ACT.Source: RQ2: Under what conditions are the… · p. 5
• Table III reports mean ± 95% confidence intervals for methods on combined task variation, with best-performing metrics marked within each task.Its final row gives successful human teleoperation demonstrations as an execution reference.Source: Table III · p. 6
• Stage-wise failures distinguish low-success policies: GR00T and xVLA make no measurable progress in 66% and 58% of episodes, while ACT failures spread across intermediate stages.ACT failures occur at intermediate gripping and closing stages at 10–16% per stage; GR00T and xVLA stall at progression values of 0.07 and 0.12.Source: RQ2: Under what conditions are the… · pp. 5–6

RQ3: How do the patterns and discriminative power of metrics vary across task domains and levels of complexity?

• Success rate is the most discriminating metric, with CV ≥1.0 in all conditions; task progress and self-collision count follow, while trajectory length and slip count have low CV.Higher CV indicates greater separation between policies.Source: RQ3: How do the patterns and… · p. 6
• CV shows no systematic decline as variation complexity rises from static to combined position-and-orientation perturbations.The metrics remain informative under challenging conditions rather than collapsing as policies degrade uniformly.Source: RQ3: How do the patterns and… · p. 6
• Across tasks and models, correlation clustering broadly matches predefined metric groups without collapsing the suite into one latent factor.Fig. 3 uses Spearman correlations and average-linkage clustering with distance 1 −|ρSpearman|.Source: Figure 3 · p. 7
• In Pack Box, ACT and Pi-0.5 have overlapping success rates, yet ACT shows the lowest jerk and shortest path length while Pi-0.5 has the highest success rate.The figure also breaks down each policy's farthest subtask stage reached.Source: Figure 4 · p. 7
• Discriminative strength varies by task: SCC stands out on RotateValve and StackTwoBlocks, ECC on LiftPot, and coordination metrics on RotateValve and StackTwoBlocks.CPL and CT show moderate, relatively uniform discrimination across tasks; a single metric would under-represent policy differences on some tasks.Source: RQ3: How do the patterns and… · p. 7

V. DISCUSSION

• RoboEval combines behavioral measures of fluency, precision, and coordination with outcome measures for stagewise progress and failure modes.The paper reports that these metrics predict success and expose distinctions hidden by success rates.Source: §V · p. 7
• The authors identify behavioral and outcome metrics as both a diagnostic lens and a potential training signal through auxiliary objectives or reward shaping.They also point to expanding variation modalities as a possible extension.Source: §V · pp. 7–8
• Metric discrimination is largely stable across variation complexity, but task-level patterns differ, including SCC on RotateValve and StackTwoBlocks and ECC on LiftPot.Success rate remains the most discriminating metric across tasks.Source: Figure 5 · p. 8