PaperlayerView exact revision on arXiv

Source-linked AI summary

EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

Ning Gao, Jinliang Zheng, Xing Gao, Haoxiang Ma, Hanqing Wang, Yukai Wang, Jiantong Chen, Zanxin Chen, Shujie Zhang, Mingda Jia, Xuekun Jiang, Zihou Zhu, Xinyu Li, Shuai Wang, Hao Li, Wenzhe Cai, Yuqiang Yang, Xudong Xu, Zhaoyang Lyu, Yao Mu, Tai Wang, Jiangmiao Pang, Jia Zeng, Weinan Zhang, Chunhua Shen

arXiv:2606.18239v2cs.RO

TL;DR

Aggregate success rates do not show where generalist manipulation policies excel, fail, or generalize under distribution shifts. EBench diagnoses these patterns across diverse tasks and dimensions, revealing sharply different capability profiles among policies with similar overall scores.

  • Problem

    Aggregate success rates cannot show where manipulation policies are strong, where they fail, or how performance changes under deployment distribution shifts.

  • Method

    EBench evaluates 26 tasks spanning mobile, long-horizon, and dexterous manipulation, annotating capabilities and controlled generalization dimensions.

  • Results

    Policies with aggregate test success rates of 24.4–29.5% exhibit divergent capability profiles; π0.5 leads with 29.5% test success and 0.92 train–test retention.

  • Takeaways & Limitations

    EBench shows that no single policy covers the capability space, with models dominating disjoint subsets of manipulation skills.

  • Takeaways & Limitations

    EBench is entirely simulated, and its scores are not claimed to predict real-robot performance; scene-level rankings are preliminary because the 26-task suite sparsely covers nine scene categories.

Abstract

from arXiv · show

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.

1. Introduction

EBench addresses the limits of aggregate success rates by diagnosing generalist manipulation policies across capabilities and distribution shifts. It combines broad task coverage with interpretable capability profiles and reveals divergent strengths among models with similar overall performance.

  • Motivation: Aggregate success rates cannot show where a manipulation policy is strong, where it fails, or how performance changes under deployment distribution shifts.
  • Motivation: Existing benchmarks either cover a narrow slice of physical interaction or focus on a single regime, limiting diagnosis of generalist manipulation.
  • EBench: EBench covers long-horizon, dexterous-and-precise, and mobile manipulation across 9 scene categories, with 26 tasks tagged along 5 capability axes and 4 generalization dimensions.
  • EBench: Its design decomposes a single scalar success rate into an interpretable capability profile.
  • Results: EBench evaluates four recent VLAs whose aggregate test success rates span 24.4–29.5%, while their five-dimensional capability profiles diverge by tens of points.
  • Results: π0.5 achieves the highest test SR of 29.5% and a 0.92 train–test retention ratio, while InternVLA-A1 dominates mobile manipulation but has a 29-point mobile–dexterous gap.

2. Related Work

Prior manipulation benchmarks either standardize short-horizon, fixed-base tabletop tasks or broaden scene diversity through mobile and multiscene environments. Recent generalist policies differ in architecture, including flow-matching, modular-decoder, and hierarchical-planner designs.

  • Simulation benchmarks for manipulation: RLBench, CALVIN, and LIBERO standardized tabletop evaluation but cover a narrow regime of short-horizon, fixed-base pick-and-place.
  • Simulation benchmarks for manipulation: Habitat, SAPIEN, ManiSkill, and RoboCasa broaden benchmark scene diversity through mobile and multiscene suites.
  • Vision–language–action models: π0 and π0.5 use flow-matching action heads, XVLA uses a modular decoder, and InternVLA-A1 uses a hierarchical planner.
  • Vision–language–action models: Adjacent generalist policies include OpenVLA, GR00T-N1, and RDT.

3. The EBench Benchmark

EBench is a 26-task benchmark spanning mobile, long-horizon, and dexterous-and-precise manipulation, with annotations and protocols designed to diagnose capabilities and generalization. It unifies action outputs across regimes and combines teleoperated and motion-planned demonstrations for evaluation.

  • Task families: EBench comprises 26 tasks across Mobile Pick-and-Place, Mobile Long-Horizon, and Table-Top Dexterous-and-Precise families.The families contain 10 mobile tasks, 9 multi-stage mobile sequences, and 7 fixed-base dexterous tasks.
  • Task taxonomy: Each task is annotated by scene, atomic skill, temporal horizon, precision, and operating mode, enabling interpretable capability queries.The taxonomy is intended to prevent strong performance on easy majority categories from masking weaknesses elsewhere.
  • Action space: A unified dual-arm, mobile-base action space lets one model checkpoint operate across tasks without architectural modification.Actions include per-arm joint positions or end-effector poses, gripper widths, and base planar velocity commands.
  • Demonstration collection: EBench uses teleoperation for its 7 dexterous tasks and key-frame pose generation with cuRobo for the remaining 19 mobile and long-horizon tasks.The two streams address contact-rich dexterous control and the practical difficulty of teleoperating mobile or long-horizon behaviors.
  • Demonstration collection: The post-training dataset contains 91.4 hours of demonstrations and 6,600 episodes organized in LeRobot format.Each dexterous task contributes 400 teleoperated episodes, while each mobile and long-horizon task contributes 200 motion-planned episodes.
  • Evaluation protocol: Evaluation probes four generalization dimensions—Background, Object, Instruction, and Mix—and requires one checkpoint to solve all 26 tasks.The protocol includes Val-Train, Val-Unseen, and a Test split spanning all four generalization dimensions, using SR as primary and Score for partial progress.

4. Experimental Setup

The experiments evaluate four recent VLA models under a consistent fine-tuning recipe on EBench demonstrations. Policies receive multi-view RGB observations, proprioceptive state, and natural-language instructions at the simulation rate.

  • Evaluated Models: Four VLA models—𝝅0, 𝝅0.5, XVLA, and InternVLA-A1—are evaluated, spanning distinct architectures and pre-training mixtures.All models are fine-tuned from pretrained checkpoints on the same EBench training data.
  • Evaluated Models: 200K gradient steps, batch size 128, AdamW, cosine learning-rate scheduling with warm-up, and peak lr 1e−5 define the shared fine-tuning recipe.The recipe is applied consistently across all evaluated models.
  • Post-Training and Evaluation Protocol: 6,600 demonstration episodes are used for post-training, with teleoperated and motion-planned trajectories roughly balanced at the frame level.The demonstrations are those described in §3.2.
  • Post-Training and Evaluation Protocol: Each policy receives 224×224 RGB images from left, right, and topdown views, together with proprioceptive state and a natural-language instruction.Frames are recorded at the 60 Hz physics-step simulation rate, and policies are queried at the same rate.

5. Capability Profiling

Although the four models achieve similar aggregate test success rates, EBench reveals sharply different capability profiles across operating mode, precision, horizon, atomic skills, and scenes. π0.5 leads overall test performance and retention, while other models show specialized strengths and weaknesses across task dimensions.

  • Overall capability and retention: 29.5% test SR is π0.5’s highest result, alongside the strongest retention ratios: 0.92 SR and 0.95 Score.The models’ test SRs remain within a narrow 24.4–29.5% range.
  • Overall capability and retention: 33.1% Val-Train SR makes InternVLA-A1 strongest in-distribution, but its Val-Unseen SR falls to 20.8% and test SR retention to 0.83.π0 reaches 24.4% test SR with the lowest retention ratio, 0.80.
  • Capability axes: Capability profiles differ markedly across operating mode, temporal horizon, precision, atomic skill, and scene categories despite close aggregate success rates.Figure 3 reports mean test SR with standard deviation across seeds for these breakdowns.
  • Operating mode: InternVLA-A1 matches π0.5 at around 34.7% mobile test SR but drops to 5.8% on dexterous tasks, producing a 29-point gap.π0 has the most balanced profile, with an 11-point mobile–dexterous gap between 29.2% and 18.1%.
  • Precision and horizon: 13.8% SR makes π0 best on sub-centimetre high-precision tasks, whereas π0.5 leads low-precision tasks at 44.2%; InternVLA-A1 leads long-horizon tasks at 29.1%.Short-horizon tasks are easier across models, with SRs ranging from 24–32%.
  • Atomic skills and scenes: No model dominates all eleven atomic skills: π0 leads Pull and Press, XVLA leads Push at 73.8%, and InternVLA-A1 wins Move and Sweep.XVLA scores 5.8% on Handover, InternVLA-A1 scores 0% on Press and Flip, and π0.5 has no catastrophic-zero categories; scene rankings are heterogeneous.

6. Generalization Diagnosis

The section diagnoses generalization beyond a single checkpoint by tracking Validation-Train and Test SR across post-training. It shows that post-training improves Test SR overall, but transfer from training gains to held-out rollouts is model-dependent, with π0.5 exhibiting the most stable dynamics and highest final Test SR.

  • Generalization Diagnosis: Models are evaluated at 25k, 50k, 100k, and 200k post-training steps using Validation-Train and Test SR.This temporal evaluation examines how generalization evolves beyond a static aggregate success rate.
  • Fit–generalization dynamics: The fit–generalization gap is the vertical distance between Validation-Train and Test curves, with smaller gaps indicating better held-out transfer.Dashed and solid curves represent Train and Test results, respectively.
  • Fit–generalization dynamics: π0.5 shows the most stable dynamics, with Validation-Train and Test curves rising largely together and achieving the highest final Test SR.Additional post-training improves Test SR for all models by 200k steps, but transfer from Validation-Train to Test is model-dependent.
  • Generalization dimensions: Test SR is also examined across four representative generalization dimensions.Figure 5 summarizes Test SR across these dimensions.

7. Pretraining Sensitivity Across Benchmarks

The section evaluates whether large-scale pretraining effects are measurable by comparing five architectures across EBench, LIBERO, and RoboTwin 2.0 under pretrained and from-scratch regimes. Pretraining substantially improves EBench performance, but LIBERO and RoboTwin 2.0 show little or reversed benefit, limiting their diagnostic value.

  • Experimental setup: The study compares 𝜋0, 𝜋0.5, XVLA, Fast-WAM, and StarVLA-OFT under pretrained and from-scratch regimes on EBench, LIBERO, and RoboTwin 2.LIBERO uses average success across Spatial, Object, Goal, and Long; RoboTwin 2.0 uses the hard-task split.
  • Results: On EBench, pretraining raises 𝜋0 from 11.2 to 24.4% SR, 𝜋0.5 from 8.5 to 29.5%, and XVLA from 15.7 to 24.7%.The passage characterizes these gains as large-margin improvements for every evaluated architecture with reported EBench values.
  • Results: On LIBERO, all five entries score between 94 and 98%, with from-scratch 𝜋0 at 95.7 versus pretrained 𝜋0 at 94.1.The near-overlap indicates essentially no difference between the compared training regimes on this benchmark.
  • Results: On RoboTwin 2.0 Hard, Fast-WAM reaches 91.8 and 𝜋0 reaches 88.8 without pretraining, exceeding pretrained baselines spanning 58.4 to 76.8.Both reported from-scratch entries score above every pretrained baseline.
  • Interpretation: The results suggest that disentangling pretraining requires benchmarks where pretrained and from-scratch baselines do not coincide, unlike LIBERO and RoboTwin 2.0.The passage explicitly states that LIBERO and RoboTwin 2.0 are not designed for this distinction.

8. Limitations

EBench is limited by its exclusive reliance on simulation and sparse scene coverage. It is intended as a reproducible screening substrate before physical evaluation, not a replacement for real-robot testing.

  • Limitations: EBench operates entirely in simulation, and its simulation scores are not claimed to predict real-robot performance.The authors plan to study sim-to-real correlation using EBench tasks in future work.
  • Limitations: EBench is intended to precede physical evaluation as a reproducible screening substrate rather than replace it.This positions the benchmark as an intermediate evaluation step.
  • Limitations: The 26-task suite sparsely covers 9 scene categories, making scene-level rankings preliminary.The authors propose expanding the suite toward 50 or more tasks.

9. Conclusion … C. Capability Profiling Summary and Task-Level Analysis

EBench combines capability profiling with controlled camera-perspective analysis to expose differences among generalist manipulation policies that scalar success rates obscure. Its results show architecture- and task-dependent performance patterns, while standardized evaluation and complementary metrics support fair diagnosis.

  • 9. Conclusion: EBench evaluates long-horizon, dexterous-and-precise, and mobile manipulation through 26 capability-tagged tasks and four controlled generalization dimensions.The benchmark unifies settings that current public benchmarks typically approximate separately.
  • C. Capability Profiling Summary and Task-Level Analysis: 𝜋0.5 leads aggregate test performance, while InternVLA-A1, 𝜋0, and XVLA dominate disjoint capability subsets.Models with similar scalar success rates differ substantially across operating mode, precision, horizon, and atomic skill.
  • 9. Conclusion: Large-scale pretraining lifts every architecture by 9–21 SR points, and EBench is the only benchmark among EBench, LIBERO, and RoboTwin 2.0 whose from-scratch and pretrained baselines do not coincide.This controlled pretraining comparison appears alongside the benchmark’s capability and generalization analyses.
  • A. Implementation Details: Every baseline uses a common training budget and action-chunk schedule, including global batch size 128, a 50-timestep prediction horizon, and a 30-step open-loop execution horizon.Policies predict a 50-step chunk at each replanning step but execute only the first 30 actions before replanning.
  • A. Implementation Details: Each episode reports binary success, SR, and a continuous task score that rewards partial progress and captures near misses discarded by binary success.The continuous score is particularly informative on long-horizon tasks with partially completed sub-goals.
  • B.1. Overall Perspective Effect: 𝜋0 gains +2.48% Test SR with Headview, whereas 𝜋0.5 loses −4.21% under the same camera switch.For 𝜋0, Test SR changes from 24.44 to 26.92; for 𝜋0.5, it changes from 29.53 to 25.32.
  • B.3. Discussion: The largest stratified camera-perspective difference is 8.38%, far below the approximately 30% pretraining gap, making deployment camera choice architecture-dependent.𝜋0 benefits from tight Headview framing in small workspaces, whereas 𝜋0.5 benefits from Overview coverage of large mobile workspaces.

C.1. Task-Level Complementarity and Hard Tasks · D. Training Loss

Per-task results reveal strong complementarity among generalist policies and identify five universally hard tasks that remain beyond current capability. Training-loss curves are compared across baselines using each model’s official loss and aligned reporting procedures through 200k optimizer steps.

  • C.1. Task-Level Complementarity and Hard Tasks: Figure 6 shows per-task success rates for 26 tasks across four baselines and Validation-Train, Validation-Unseen, and Test splits.Rows are sorted by total Test SR, with color encoding per-task success rate.
  • C.1. Task-Level Complementarity and Hard Tasks: The π-family outperforms XVLA on detergent by +71%, perfume_to_cosmetics_rack by +35%, and remote_to_holder by +19% Test-SR margin.The π-family comprises π0 and π0.5 taken together.
  • C.1. Task-Level Complementarity and Hard Tasks: Shop, bottle, peg_in_hole, collect_coffee_beans, and flip_cup_collect_cookies remain near-white across every baseline and split.These five tasks form the proposed hard suite for future generalist-policy evaluations.
  • C.1. Task-Level Complementarity and Hard Tasks: ≤5% SR is achieved by all four baselines on the hard tasks across multiple evaluation snapshots.Peg_in_hole requires high-precision insertion, while flip_cup_collect_cookies requires coordinated flipping and collection; both demand force-aware feedback loops.
  • D. Training Loss: Figure 7 plots each baseline’s full training-loss trajectory from initialization through the 200k checkpoint on a shared logarithmic vertical axis.Losses are taken directly from each model’s official open-source repository.
  • D. Training Loss: π0 and π0.5 use flow-matching action loss, XVLA uses diffusion-policy denoising loss, and InternVLA-A1’s loss is plotted from its trainer log.The plotted curves preserve each model’s official loss definition.
  • D. Training Loss: π0, π0.5, and XVLA use 100-step loss averaging, whereas InternVLA-A1 is logged once every 200 steps and plotted as-is.All four runs target 200k optimizer steps.

E. Controlled Breakdown Analysis

The controlled breakdown analysis uses task-level permutation tests to estimate tag effects while addressing multi-factor confounding in EBench’s 26-task design. It reports observed Test SR differences and two-tailed p-values as reference statistics for interpreting cluster-level patterns, highlighting several model-specific effects without binary significance claims.

  • Method: Task-level permutation tests with 10,000 iterations estimate each tag’s net effect without relying on distributional assumptions.Multiple linear regression is under-powered because more than 20 tag categories approach the 26-task sample size.
  • Method: Labels are shuffled at the task level, preserving within-task model correlations; atomic-skill labels are additionally shuffled within shared scene categories.The two-tailed p-value is the fraction of 10,000 permutations whose absolute mean difference is at least as large as the observed difference.
  • Reporting: Each contrast reports Δ, the observed category-minus-reference difference in Test SR, alongside a two-tailed p-value from task-level label shuffles.Positive Δ means the model scores higher on the category subset than on the reference subset.
  • Implications: Permutation p-values are reference statistics for gauging consistency with task-level chance, not binary accept/reject claims.The analysis recommends including such statistics alongside cluster-level tables because correlated categories can attribute effects to the wrong tags.
  • Findings: InternVLA-A1’s Mobile advantage is Δ= + 30.9%, 𝑝=0.008, while π0.5’s Low-precision advantage is Δ= + 37.1%, 𝑝=0.030.The analysis omits Move because Move-tagged tasks nearly coincide with Mobile tasks, so the contrast would largely repeat the Operating Mode signal.
Loading 2606.18239v2…