PaperlayerView exact revision on arXiv

Source-linked AI summary

EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

Ning Gao, Jinliang Zheng, Xing Gao, Haoxiang Ma, Hanqing Wang, Yukai Wang, Jiantong Chen, Zanxin Chen, Shujie Zhang, Mingda Jia, Xuekun Jiang, Zihou Zhu, Xinyu Li, Shuai Wang, Hao Li, Wenzhe Cai, Yuqiang Yang, Xudong Xu, Zhaoyang Lyu, Yao Mu, Tai Wang, Jiangmiao Pang, Jia Zeng, Weinan Zhang, Chunhua Shen

arXiv:2606.18239v2cs.RO

TL;DR

Aggregate success rates leave unanswered where generalist manipulation policies are strong, where they break, and how their performance shifts under distribution drift. EBench evaluates policies across a broad, multi-axis simulation benchmark, revealing distinct capability profiles despite similar aggregate success rates.

  • Problem

    Aggregate success rates do not show where generalist manipulation policies are strong, where they break, or how their performance shifts as deployment distributions drift from training.

    Source: §1 · p. 1

  • Method

    EBench unifies 26 capability-tagged tasks spanning mobile, long-horizon, and dexterous manipulation with four controlled generalization dimensions in one evaluation protocol.

    Source: §9 · p. 10

  • Results

    Models diverge across capability dimensions despite near aggregate success rates; π0.5 achieves the highest test SR of 29.5%, while policies lead disjoint capability subsets.

    Source: §1 · p. 3, §9 · p. 11

  • Takeaways & Limitations

    The evaluated policies show distinct capability strengths, and no single policy covers the full capability space.

    Source: §1 · p. 3

    EBench is simulation-only and is intended as a screening substrate before physical evaluation, not as evidence that simulation scores predict real-robot performance.

    Source: §8 · p. 10

Abstract

from arXiv · show

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.

1. Introduction

EBench is a simulation benchmark designed to profile generalist manipulation policies across broad task families and controlled distribution shifts, addressing questions that aggregate success rates cannot answer. Evaluations show that models with similar overall success rates have divergent capability profiles and generalization behavior.

Introduction

  • Existing benchmarks either cover narrow task or embodiment regimes, isolate a single capability, or report overall scores, leaving broad, fine-grained diagnosis unmet.Examples include RLBench (James et al., 2020), CALVIN (Mees et al., 2022), LIBERO (Liu et al., 2023), RMBench (Chen et al., 2026), and BEHAVIOR-1K.Source: §1 · p. 1
  • EBench covers 26 tasks across mobile, long-horizon, and dexterous manipulation, tagging capabilities and testing four distribution shifts to decompose success into interpretable profiles.The tasks include 10 mobile pick-and-place, 9 long-horizon multi-stage, and 7 dexterous-and-precise tasks; Figure 1 specifies 9 scene categories, 5 capability axes, and 4 generalization dimensions.Source: Figure 1 · p. 2, §1 · p. 2
  • EBench’s codebase combines task-data synthesis, composable success and partial-progress scoring, and distributed evaluation for mobile manipulation.It pairs teleoperation for dexterous tasks with key-frame poses and cuRobo for mobile and long-horizon tasks; the full validation split runs on 8 consumer GPUs in roughly 30 minutes.Source: §1 · p. 2
  • Across four evaluated VLAs, π0.5 leads test success at 29.5% and train–test retention at 0.92, while InternVLA-A1’s mobile-to-dexterous gap is 29 points and skill rankings are disjoint.The models’ aggregate test success rates fall within 24.4–29.5%.Source: §1 · pp. 2–3

2. Related Work

Prior manipulation benchmarks either cover narrower task regimes or report scalar success, while VLA comparisons often use hardware-specific runs or narrow simulation subsets. EBench evaluates mobile, dexterous, and long-horizon manipulation under one protocol and provides interpretable capability and generalization coordinates.

Simulation benchmarks for manipulation

  • Tabletop suites target short-horizon fixed-base tasks, while mobile and multiscene suites seldom test sub-centimetre dexterity; EBench spans these regimes with structured task annotations.Procedural suites scale task counts but retain per-task scalar success; real-to-sim benchmarks prioritize fidelity to one embodiment, and RMBench probes one capability axis.Source: §2 · p. 3

Vision–language–action models

3. The EBench Benchmark

EBench is a 26-task benchmark spanning mobile and dexterous manipulation, with a shared action space and diagnostic task annotations. It combines complementary demonstration-collection methods with controlled generalization shifts and a common evaluation protocol.

3.1. Task Design and Diversity

  • EBench covers 26 tasks across mobile pick-and-place, mobile long-horizon, and table-top dexterous-and-precise families, annotated along five capability dimensions.The families contain 10, 9, and 7 tasks, respectively; all use a unified dual-arm-plus-mobile-base action space, allowing one checkpoint to be evaluated across regimes without architectural modification.Source: §3.1 · pp. 3–4

3.2. Data Synthesis: Teleoperation and Motion Planning

  • Seven dexterous tasks use actor-follower teleoperation for reactive contact corrections, while the other 19 use key-frame poses and cuRobo-planned trajectories.This division addresses the difficulty of programming dexterous interactions with motion planning and the difficulty of teleoperating mobile and long-horizon tasks; the resulting dataset contains 91.4 hours of demonstrations.Source: §3.2 · pp. 4–5

3.3. Generalization Dimensions

  • Evaluation varies background, object, instruction, and combined Mix conditions, with training and test assets drawn from disjoint pools.Background and instruction shifts probe perceptual and linguistic robustness without changing underlying physics; object swaps require physical generalization, while Mix compounds perturbations.Source: §3.3 · p. 5

3.4. Evaluation Protocol and Metric Library

  • The primary protocol requires one checkpoint to solve all 26 tasks and tests performance on 510 episodes spanning all four generalization dimensions.Validation includes 130 Val-Train episodes and 154 Val-Unseen episodes with object swaps; the test assets are disjoint from training assets.Source: §3.4 · p. 5
  • EBench reports binary success rate and stage-based task progress, using shared simulator-state evaluation primitives composed into ordered task stages.Success rate registers only at the final stage, while task Score advances as successive stage conditions are satisfied, replacing per-task hand-coded evaluators.Source: §3.4 · pp. 5–6

4. Experimental Setup

Section 4 evaluates four fine-tuned VLA models using a shared training recipe and a repeated evaluation protocol that accounts for renderer nondeterminism.

Evaluated Models

Post-Training and Evaluation Protocol

  • Evaluation used 6,600 demonstrations and three runs per model to account for IsaacSim renderer nondeterminism.Teleoperated and motion-planned trajectories were roughly balanced at the frame level; results report the mean and standard deviation across runs. Observations included 224×224 RGB from left, right, and topdown views, proprioceptive state, and a natural-language instruction; frames were recorded and policies queried at 60 Hz.Source: §4 · p. 6

5. Capability Profiling

Although the four models achieve similar test success rates, their capability profiles diverge across operating mode, task horizon, precision, atomic skills, and scenes. π0.5 has the highest test success rate and strongest train–test retention, while the other models show distinct strengths and weaknesses.

5.1. Overall Performance

  • π0.5 achieves the highest test SR (29.5%) and strongest retention (SR: 0.92, Score: 0.95), while π0 has the lowest retention ratio (0.80).The four models’ test SRs fall within a five-point range (24.4–29.5%); InternVLA-A1 has the highest Val-Train SR (33.1%) but drops to 20.8% SR on Val-Unseen and has 0.83 SR retention, while π0’s test SR is 24.4%.Source: §5.1 · pp. 6–7

5.2. Five-Dimensional Capability Breakdown

  • Figure 3 breaks down test SR across overall performance, operating mode, temporal horizon, precision, atomic skills, and scene categories, revealing markedly different profiles despite similar aggregate SRs.Bars show mean test SR and error bars show standard deviation across seeds.Source: §5.2 · p. 7, Figure 3 · p. 7
  • InternVLA-A1 matches π0.5 at around 34.7% SR on mobile manipulation but falls to 5.8% SR on fixed-base dexterous tasks, the cohort’s largest gap at 29 points.π0 has a more balanced profile, with an 11-point gap between mobile (29.2%) and dexterous (18.1%) settings, but lower absolute performance than π0.5.Source: §5.2 · pp. 7–8
  • π0 leads on sub-centimetre high-precision tasks at 13.8% SR, while π0.5 leads low-precision tasks at 44.2% SR.Short-horizon tasks yield SRs of 24–32% for all models; on long-horizon tasks, InternVLA-A1 leads at 29.1% SR, while XVLA drops from 28.9% on short tasks to 13.5% on long tasks.Source: §5.2 · p. 8
  • No model dominates all eleven atomic skills: π0 leads Pull (47%) and Press (50%), XVLA leads Push (73.8%), and InternVLA-A1 leads Move and Sweep.XVLA scores 5.8% on Handover, InternVLA-A1 scores 0% on Press and Flip, and π0.5 is the only model without any catastrophic-zero categories. Scene leaders also vary: π0.5 leads Bedroom, Bathroom, and Living Room; InternVLA-A1 leads Kitchen and Dining; XVLA leads Supermarket.Source: §5.2 · p. 8

6. Generalization Diagnosis

EBench tracks Validation-Train and Test success rates across post-training checkpoints to show how generalization changes over time. By 200k steps, all models improve on Test SR, but their transfer dynamics differ: π0.5 has the highest final Test SR and the most stable fit–generalization dynamics.

Fit–generalization dynamics

  • Models are evaluated at 25k, 50k, 100k, and 200k post-training steps using Validation-Train and Test SR.Figure 4 plots the two success-rate curves across checkpoints.Source: §6 · p. 8, Figure 4 · p. 8
  • π0 improves steadily but develops a more visible late-stage gap, while XVLA’s Test SR trajectory is non-monotonic before recovering at 200k steps.The curve gap measures the empirical fit–generalization gap; smaller gaps indicate better transfer of in-distribution gains to held-out rollouts.Source: §6 · p. 8, Figure 4 · p. 8
  • By 200k steps, all models improve Test SR, while π0.5 shows the most stable fit–generalization dynamics and the highest final Test SR.Its Validation-Train and Test curves rise largely together, indicating a smaller empirical fit–generalization gap.Source: Figure 4 · p. 8, §6 · p. 8
  • InternVLA-A1 has the strongest final Validation-Train SR but retains a larger Test gap, suggesting its added fitting benefits training-distribution rollouts more than OOD rollouts.Source: §6 · pp. 8–9

7. Pretraining Sensitivity Across Benchmarks

EBench reveals sizable pretraining gains where LIBERO and RoboTwin 2.0 are largely saturated. The comparison evaluates released pretrained checkpoints and models trained from scratch on each benchmark’s training split.

Pretraining Sensitivity Across Benchmarks

  • The study compares five architectures across EBench, LIBERO, and RoboTwin 2.0 using released-checkpoint fine-tuning or random initialization and benchmark-only training.Fast-WAM and StarVLA-OFT appear only in the from-scratch regime because they have no released pretrained checkpoint.Source: §7 · p. 9
  • Published from-scratch π0.5 and XVLA results are unavailable on LIBERO and RoboTwin 2.0, so those entries are unreported and Fast-WAM supplies the without-pretrain comparison.LIBERO scores average its Spatial, Object, Goal, and Long suites; RoboTwin 2.0 scores average the hard-task split.Source: §7 · p. 9
  • Pretraining raises EBench SR from 11.2 to 24.4% for π0, 8.5 to 29.5% for π0.5, and 15.7 to 24.7% for XVLA.By contrast, LIBERO shows essentially no difference: its five entries score 94–98%, and from-scratch π0 scores 95.7 versus pretrained π0 at 94.1.Source: §7 · pp. 9–10
  • EBench exposes a 9–21 SR-point pretraining gap, while LIBERO is saturated at 94–98% and scratch models match or exceed pretrained RoboTwin 2.0 Hard baselines.On RoboTwin 2.0 Hard, from-scratch Fast-WAM scores 91.8 and π0 scores 88.8, above the pretrained baselines’ 58.4–76.8 range.Source: Table 3 · p. 10, §7 · p. 10

8. Limitations

EBench is a simulation-only screening substrate, not a predictor or replacement for real-robot evaluation; its sparse scene coverage makes scene-level rankings preliminary.

Limitations

  • EBench scores are not claimed to predict real-robot performance, and the benchmark is intended to precede rather than replace physical evaluation.The authors plan to study sim-to-real evaluation correlation using EBench tasks.Source: §8 · p. 10
  • With 26 tasks spanning 9 scene categories sparsely, EBench scene-level rankings should be treated as preliminary.The roadmap is to expand the suite toward 50 or more tasks, enabling multi-way regression instead of permutation tests.Source: §8 · p. 10
  • The authors plan to study the correlation between simulation and real-robot evaluation on EBench tasks in future work.Source: §8 · p. 10

9. Conclusion … C. Capability Profiling Summary and Task-Level Analysis

EBench reveals capability-specific differences among generalist manipulation policies that aggregate success rates alone obscure, with π0.5 leading test performance and other models excelling on distinct capability subsets. Camera-perspective effects are smaller and architecture-dependent, while EBench also uniquely surfaces benefits from large-scale pretraining.

9. Conclusion

  • EBench combines 26 capability-tagged tasks, four controlled generalization dimensions, a disjoint test asset pool, and 91.4 hours of data from two synthesis tracks.Teleoperation supplies dexterous-and-precise tasks, while key-frame poses with cuRobo supply mobile and long-horizon tasks; the tasks, data, evaluation code, and splits are publicly released.Source: §9 · pp. 10–11
  • π0.5 leads aggregate test performance, while InternVLA-A1, π0, and XVLA dominate disjoint capability subsets and differ across interpretable axes.The axes include operating mode, precision, horizon, and atomic skill; the models also follow distinct fit–generalise trajectories as training proceeds.Source: §9 · pp. 10–11
  • Large-scale pretraining raises every architecture’s EBench SR by 9–21 points, including π0.5 from 8.5 to 29.5% and π0 from 11.2 to 24.4%.On LIBERO and RoboTwin 2.0, from-scratch models match or surpass pretrained counterparts; EBench is the only one of these benchmarks where the baselines do not coincide.Source: §9 · p. 11

A. Implementation Details

  • Baselines share a global batch size of 128 and a 50-timestep prediction chunk, executing 30 actions before replanning; episodes report binary SR and a continuous task score.The 20-step lookahead buffer absorbs inference latency without introducing closed-loop instability; the continuous score rewards partial progress that binary success misses.Source: A. Implementation Details · p. 13

B.1. Overall Perspective Effect

  • π0 gains +2.48% Test SR with Headview (24.44 → 26.92), while π0.5 loses −4.21% (29.53 → 25.32); both perspectives have three 200k seeds per policy.Table 4 reports the point estimates averaged over three seeds. The comparison switches the primary stream between Overview and Headview while keeping auxiliary cameras and the rest of the protocol fixed.Source: Table 4 · p. 14, §B.1 · p. 14, B. Camera-Perspective Sensitivity · p. 14

B.2. Decomposition by Operating Mode and Horizon

  • π0’s Headview gain is concentrated in dexterous fixed-base tasks (+8.38%), while π0.5’s Overview preference is concentrated in mobile (−6.21%) and long-horizon tasks (−5.90%).The pattern is stratum-specific: π0 favors a tight view of small workspaces, whereas π0.5 favors Overview for the mobile platform’s larger reach. The largest stratified |Δ| is 8.38%, much smaller than the approximately 30% pretraining gap; the proposed workspace–field-of-view rule needs testing with additional perspectives and camera-fusion ablations.Source: §B.2 · pp. 14–15, Table 5 · p. 15, §B.3 · p. 15

C.1. Task-Level Complementarity and Hard Tasks · D. Training Loss

Per-task results reveal complementary strengths among the policy families and five tasks that remain uniformly difficult. The training-loss curves use model-specific loss definitions and reporting cadences, so their absolute values are not directly comparable.

C.1. Task-Level Complementarity and Hard Tasks

  • On Test SR, the 𝜋-family leads XVLA by +71% on detergent, +35% on perfume_to_cosmetics_rack, and +19% on remote_to_holder.Figure 6 shows these task-level differences across baselines and evaluation splits.Source: §C.1 · p. 15, Figure 6 · p. 16
  • XVLA leads the 𝜋-family on Test SR by +51% on apple_to_fruit_bowl, +44% on soap_to_dish, and +38% on utensils_to_holder.These task-level differences are visible in Figure 6’s per-task success-rate heatmap.Source: §C.1 · p. 15, Figure 6 · p. 16
  • InternVLA-A1 falls between the two families on most tasks, resolves 𝜋-family weaknesses on apple_to_fruit_bowl and soap_to_dish, but is weaker on the dexterous tabletop tail.Source: §C.1 · p. 15
  • Shop, bottle, peg_in_hole, collect_coffee_beans, and flip_cup_collect_cookies remain near-white across baselines and splits, with all four baselines at ≤5% SR across multiple snapshots.The paper proposes these five tasks as a hard suite; crossing 10% SR on the full set is described as moving the frontier, while full-task aggregates are dominated by easier tasks. Peg insertion and coordinated flipping and collection demand force-aware feedback loops lacking in current open-loop action models.Source: §C.1 · pp. 15–16

D. Training Loss

  • Training losses are model-specific: π0 and π0.5 use openpi flow-matching action loss, XVLA uses X-VLA diffusion-policy denoising loss, and InternVLA-A1 uses OT-Train combined action plus auxiliary generative loss.Different units, normalisations, action chunk sizes, and auxiliary-term weights make absolute loss magnitudes incomparable across models.Source: D. Training Loss · p. 16
  • Figure 7 plots each baseline’s official training loss from initialization through the 200k checkpoint on a shared logarithmic axis to show convergence shape, not comparable absolute loss magnitudes.The plotted runs target 200k optimizer steps.Source: D. Training Loss · p. 16, Figure 7 · p. 17
  • π0 and π0.5 losses are averaged over 100-step windows, XVLA is block-averaged to the same cadence, and InternVLA-A1 is plotted as logged every 200 steps.Figure 7 plots InternVLA-A1’s once-per-200-step values as-is.Source: D. Training Loss · p. 17, Figure 7 · p. 17

E. Controlled Breakdown Analysis

EBench uses controlled task-level permutation tests to estimate tag effects while accounting for correlated task tags, and presents the resulting statistics as references for interpreting cluster-level breakdowns rather than binary significance claims.

E. Controlled Breakdown Analysis

  • Because correlated task tags can make cluster-level differences reflect other factors, EBench recommends reporting permutation reference statistics alongside cluster-level tables in future fine-grained benchmarks.The tests gauge how consistent an observed difference is with task-level chance; they are not used for accept/reject claims.Source: E. Controlled Breakdown Analysis · pp. 17, 20
  • With 26 tasks and more than 20 tag categories, EBench uses 10,000 task-level permutation iterations instead of under-powered linear regression.Labels are shuffled at the task level to preserve within-task model correlations; multi-hot atomic-skill labels are shuffled only within shared scene categories to control scene–skill confounding.Source: E. Controlled Breakdown Analysis · pp. 17–18
  • Each contrast reports observed Test-SR difference Δ and a two-tailed p-value, which should be read together with sample size rather than as a binary significance decision.The p-value is the fraction of task-level shuffles with an absolute difference at least |Δ|; Δ is a difference between task subsets, not a percentage of an absolute score.Source: E. Controlled Breakdown Analysis · pp. 18–19
  • +30.9% was InternVLA-A1’s Mobile advantage (p=0.008), while π0.5 showed +37.1% for Low-precision and −32.1% for Insert; Horizon and Scene contrasts had wider nulls.The analysis omits Move because Move-tagged tasks nearly coincide with Mobile tasks; apparent scene rankings may instead reflect each scene’s operating-mode and precision composition.Source: Table 6 · p. 18, E. Controlled Breakdown Analysis · p. 19, Figure 9 · p. 20

Related papers

Loading 2606.18239v2…