PaperlayerView exact revision on arXiv

Source-linked AI summary

Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands

Zhenjie Yang, Yideng Zhang, Dongjie Zhang, Chenyu Jiang, Xianshuai Liu, Yufeng Li, Zuhao Ge, Xingyu Jiao, Zheng Zhang, Kaiyu He, He Wang, Yuwen Zhong, Yi Deng, Muyun Jiang, Xianliang Huang, Haisheng Su, Donghang Zhang, Jian Zhang, Xue Yang, Hongyang Li, Zuxuan Wu, Yu-Gang Jiang, Xiaosong Jia, Junchi Yan

arXiv:2609.15726v1cs.ROcs.AIcs.CV

TL;DR

Existing benchmarks and datasets do not jointly support diverse bimanual dexterous embodiments, visuo-tactile data collection, and controlled evaluation. Bench2Dex provides a simulation benchmark with a shared tactile interface, demonstrations, and separate robustness axes. Evaluations report lower aggregate success under combined scene perturbations than under matched scenes, while the benchmark remains distinct from validation against physical tactile hardware.

  • Problem

    Existing benchmarks lack a shared setting combining diverse bimanual dexterous embodiments, visuo-tactile data collection, and evaluation protocols suited to comparing hand designs.

    Source: §1 · p. 2

  • Method

    Bench2Dex combines teleoperated bimanual tasks, a shared tactile interface, synchronized multimodal data, executable metrics, and separate invariance and equivariance perturbation axes.

    Source: §1 · pp. 2–3

  • Results

    Evaluations using RGB and proprioceptive observations show lower aggregate success under combined scene perturbations than under matched scenes, with outcomes varying across tasks and policies.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Bench2Dex provides reusable components and a common setting for studying bimanual dexterous manipulation across embodiments.

    Source: §5 · p. 10

    The hand and tactile models have not been calibrated against matching physical hardware, and the benchmark does not reproduce a specific physical hand–sensor system.

    Source: §5 · p. 10

Abstract

from arXiv · show

Tactile sensing provides contact information that can be difficult to infer from vision alone, but tactile hardware for dexterous hands has not converged to a common design. Dexterous hands differ in finger structure, contact surfaces, and sensor layouts, while simulated tactile signals still differ from measurements produced by physical sensors. These factors make it difficult to study visuo-tactile manipulation across diverse dexterous hands within a consistent experimental setting. We present Bench2Dex, a simulation benchmark for visuo-tactile bimanual manipulation across 12 dexterous hands. We adapt existing robot models with a shared simulated tactile interface that converts local contact geometry into image-like tactile observations. The interface provides a consistent observation format across different hand morphologies without attempting to reproduce the output of a specific physical tactile sensor. Bench2Dex includes 26 bimanual manipulation tasks that involve tool use, articulated-object interaction, and multi-stage manipulation, together with about 1.3K human-teleoperated demonstrations. The benchmark provides synchronized visual, tactile, proprioceptive, action, and object-state observations, together with executable task metrics. For robustness, we group seven perturbation types into invariance axis, where the correct action does not change, and equivariance axis, where the correct action changes together with the perturbation. We evaluate ACT, Diffusion Policy, pi0.5, and GR00T N1.5 on Bench2Dex and report their performance and failure modes. Bench2Dex is meant as a platform for studying visuo-tactile learning across dexterous hands. It does not assume that simulated tactile observations can replace real tactile sensing; it offers a shared setting for algorithm development while tactile hardware and simulation models are still evolving.

1 Introduction

Bench2Dex addresses the lack of a shared benchmark for visuo-tactile manipulation across diverse bimanual dexterous hands. It combines cross-embodiment data collection and evaluation with a common tactile interface and separates robustness tests by whether perturbations should change the correct action.

1 Introduction

  • Existing benchmarks and datasets often use one platform or narrow sensing, while cross-embodiment efforts lack protocols for comparing dexterous hand designs and contact surfaces.The authors identify no existing benchmark for visuo-tactile data collection across multiple dexterous hands under one setting.Source: §1 · p. 2
  • Different hand kinematics, finger designs, and contact patterns affect grasping, tool use, and long-horizon execution, complicating interpretation of cross-hand policy performance.A shared visuo-tactile setting can help assess whether policies use contact information rather than fitting one hand’s tactile layout.Source: §1 · p. 2
  • Bench2Dex brings together 12 robot embodiments, 26 long-horizon tasks, 1.3K teleoperated demonstrations, synchronized multimodal data, and executable evaluation.Its evaluation includes task completion, stage-level progress, and seven perturbation types grouped into invariance and equivariance axes.Source: §1 · pp. 2–3
  • The benchmark groups texture, lighting, background, camera pose, and distractors as invariance perturbations, while object pose and table height are equivariance perturbations.Separating the groups distinguishes sensitivity to nuisance factors from adaptation to changed task geometry.Source: §1 · p. 3
  • Bench2Dex’s simulated tactile interface does not reproduce any specific physical sensor, and dexterous-hand hardware has not converged on a common design.The shared setting is intended for study while tactile hardware and simulation models evolve.Source: §1 · p. 3

2 Related Work

Prior work offers manipulation benchmarks, dexterous data collection, and tactile evaluation, but these efforts cover complementary slices of the problem. Bench2Dex combines diverse bimanual dexterous embodiments, teleoperation, tactile data, and controlled-shift evaluation in one simulation pipeline.

2.1 Manipulation Benchmarks and Datasets

  • Manipulation benchmarks and datasets support standardized evaluation or scalable data, but the cited robustness-benchmark line relies on parallel-jaw grippers rather than multi-finger tactile dexterity.The cited benchmarks include Meta-World [10], RLBench [11], CALVIN [12], and ManiSkill3 [46], while large datasets include BridgeData V2 and Open X-Embodiment [17].Source: §2.1 · pp. 3–4

2.2 Dexterous and Bimanual Manipulation Benchmarks

  • Existing dexterous and bimanual benchmarks each omit part of the combination: multiple embodiments, tool use, tactile sensing, or visual observations.For example, DexMimicGen [33] spans 3 embodiments and 9 tasks without tool use or tactile sensing, while DexVerse [36] includes teleoperated tool-use demonstrations on one embodiment without tactile sensing.Source: §2.2 · p. 4

2.3 Dexterous Data Collection and Teleoperation

  • Dexterous demonstration generation and teleoperation have advanced, but cited systems are evaluated on custom setups rather than a shared multi-embodiment benchmark.Bimanual teleoperation systems Mobile ALOHA [9] and ALOHA 2 [62] use parallel-jaw grippers; AnyTeleop [29] and From One Hand to Multiple [30] address cross-hand retargeting.Source: §2.3 · p. 4

2.4 Tactile Sensing and Benchmarking for Manipulation

  • Tactile benchmarks address varied learning and evaluation goals, but the cited efforts do not jointly provide a common tactile representation for heterogeneous bimanual dexterous hands and long-horizon tasks.Bench2Dex instead reconstructs contact surfaces from different hand meshes and encodes contact depth as a common surface-aligned tactile map, within a pipeline for teleoperation, synchronized data, and controlled shifts.Source: §2.4 · pp. 4–5

3 Bench2Dex Benchmark

Bench2Dex is a simulation pipeline connecting task construction, teleoperation, multimodal data collection, and executable evaluation. Its components support synchronized observations and a shared tactile representation across diverse hands, with evaluation channels that distinguish nuisance variation from changed task geometry.

3 Bench2Dex Benchmark

  • Bench2Dex connects task construction, human teleoperation, multimodal data collection, and executable evaluation in one simulation framework.The benchmark is designed as a pipeline rather than a task collection alone.Source: §3 · p. 5

3.1 Online Teleoperation Setup

  • Human teleoperation runs inside the Isaac Lab simulation loop and records actions, robot and object states, camera observations, task metadata, and metric signals.The Manus glove and ARKit wrist tracking are retargeted to twelve hand embodiments, with arm targets computed by a closed-loop inverse-kinematics controller.Source: §3.1 · p. 5

3.2 Offline Multimodal Data Acquisition

  • Offline replay separates lightweight motion recording from rendering the remaining modalities, including RGB-D views, object labels, occupancy grids, bounding boxes, and tactile observations.Episodes are replayed with the same random seed; the replay can be parallelized, rerun with updated sensor configurations, or applied selectively.Source: §3.2 · pp. 5–6

3.3 Unified Visuo-Tactile Data Acquisition

  • A shared surface-aligned tactile layer maps heterogeneous hand contact geometry into image-like tactile observations.It uses reconstructed contact surfaces and inward-facing surface normals for ray casting; metric depth and a contact-validity mask are also retained alongside the encoded tactile image.Source: §3.3 · pp. 6–7, Figure 2 · p. 6

3.4 Datasets and Policy

  • The dataset contains about 1.3K demonstrations across 26 tasks and 12 embodiments, while policy evaluation includes ACT, Diffusion Policy, pi0.5, and GR00T N1.5.Each trajectory is replayed into an HDF5 record with synchronized RGB-D, states, actions, object states, tactile maps, bounding boxes, and occupancy grids.Source: §3.4 · p. 7

3.5 Evaluation Metrics

  • The evaluation suite measures stable success, latched stage progress, efficiency, safety diagnostics, and success across generalization channels.Stable success requires the executable terminal predicate to remain true for its configured dwell time; stage latching preserves completed progress when later actions reverse an earlier predicate.Source: §3.5 · p. 7

3.6 Generalization Strategy

  • Generalization tests keep the semantic goal and success conditions fixed while varying nuisance factors, task geometry, or both across four channels.None restores a matched anchor episode; Equi. resamples geometry, Inv. resamples nuisance factors, and Full resamples both groups without an anchor, with resolved scene parameters saved for replay and audit.Source: §3.6 · pp. 7–8

4 Experiments

Across 26 task–embodiment settings, GR00T leads under matched conditions and in the four-channel mean, while the policy ordering varies under distribution shifts. Results also show task-level heterogeneity and distinguish stable success from stage-level progress.

4.1 Experimental Setup

  • The evaluation covers all 26 task–embodiment settings with 50 rollouts per policy and channel, using task-macro success rates across the settings.None, Equi., Inv., and Full use matched anchors for the first three channels, while Full is sampled independently.Source: §4.1 · p. 8

4.2 Main Results

  • 631/1,300 successes (48.5%) makes GR00T the matched-None leader, with the highest task-level success on 23 of 26 settings.ACT records 383 (29.5%), 𝜋0.5 355 (27.3%), and DP 168 (12.9%).Source: §4.2 · p. 8, Table 2 · p. 9
  • Under Full, GR00T and 𝜋0.5 are nearly tied at 19.8% and 19.7%, while GR00T leads the equal-weight four-channel mean at 28.3%.𝜋0.5 has the most strict task-level leads under Full: 12, versus eight for GR00T and two for ACT, with four ties.Source: §4.2 · p. 8, Table 2 · p. 9
  • LSCR ranks the policies like mean SR but captures dependency-valid stage progress that binary stable success can miss.On Jigsaw Puzzle Assembly, DP and 𝜋0.5 each have 0.0% mean SR but LSCR of 6.3% and 7.4%, respectively.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Generalization Patterns

  • None has the highest observed success rate for every policy, but channel differences are marginal comparisons and do not establish per-instance difficulty ordering or interactions.The realized channel ordering varies by policy: ACT and DP decline through Full, whereas 𝜋0.5 and GR00T have higher success under Full than Equi. but lower than None.Source: §4.3 · p. 9
  • Full-to-None success-rate ratios range from 29.8% for DP to 72.1% for 𝜋0.5, and Full success is lower in 89 of 104 task–policy comparisons.These are observed marginal comparisons; the sampling design does not pair Full episodes with None episodes.Source: §4.3 · pp. 9–10
  • Equi. exceeds Inv. for ACT and DP, while Inv. exceeds Equi. for 𝜋0.5 and GR00T, exposing policy-dependent sensitivity across the isolated axes.The corresponding rates are 15.4% versus 14.0% for ACT, 6.3% versus 5.2% for DP, 13.2% versus 21.0% for 𝜋0.5, and 18.8% versus 26.0% for GR00T.Source: §4.3 · p. 10

4.4 Task-Level Heterogeneity

  • Full outcomes span 12 settings with nonzero success for every policy, three with zero success for every policy, and 11 with mixed outcomes; these stable-success results do not estimate between-training variability or isolate embodiment effects.Tasks and embodiments are not factorially crossed, and stable SR alone does not distinguish partial progress, safety, or execution time.Source: §4.4 · p. 10

5 Conclusion

Bench2Dex brings bimanual dexterous manipulation, multimodal data collection, and executable evaluation into a simulation benchmark spanning 12 embodiments and 26 tasks. Evaluations show lower aggregate success under combined scene perturbations than matched scenes, while the benchmark remains a shared simulation interface rather than a hardware-calibrated tactile model.

5 Conclusion

  • Bench2Dex combines 12 embodiments, 26 long-horizon tasks, approximately 1.3K teleoperated demonstrations, synchronized multimodal data, and executable task evaluation.The data include visual, geometric, proprioceptive, action, and visuo-tactile signals.Source: §5 · p. 10
  • ACT, Diffusion Policy, π0.5, and GR00T N1.5 show lower aggregate success under combined scene perturbations than under matched scenes, with outcomes varying across tasks and policies.The reported results use RGB and proprioceptive observations.Source: §5 · p. 10
  • The hand and tactile models are not calibrated against matching physical hardware, and the interface does not reproduce a specific physical hand–sensor system.Future extensions could add hardware-specific calibration and physical validation to assess which findings carry over to real-world manipulation.Source: §5 · p. 10

A Full Task Catalog

The supplement catalogs Bench2Dex tasks and supported bimanual embodiments, and describes how tactile contact surfaces are reconstructed and grouped across hands.

A Full Task Catalog

  • The task catalog lists descriptions, evaluator success conditions, embodiments, and scene snapshots for tasks including interhand wine pouring, trash disposal, and fridge sorting.Other listed tasks include placing food into a microwave and loading a breadbasket.Source: Table S1 · p. 20, A Full Task Catalog · pp. 20–21

B Supported Bimanual Robot Embodiments

  • Each of the 12 embodiments pairs a robot arm with a dexterous hand, spanning different arm kinematics, hand morphologies, palm geometries, and finger configurations.The supported embodiments are listed in Table S2.Source: Table S2 · pp. 22–23

C Unified Tactile Surface Reconstruction

  • Tactile contact surfaces are reconstructed from intended contact regions and represented as surface-aligned point and inward-facing normal maps, with grouping adapted to each hand’s mesh sharing.The current setup uses four-finger-shared grouping for nine hands and independent per-finger assets for three; each finger has a tactile site, with palm sites excluded.Source: C Unified Tactile Surface Reconstruction · pp. 23–24, Table S3 · p. 24

C.1 Signal Semantics and Depth Quantization

Tactile observations encode contact penetration using inward-facing surface-normal rays, a contact-validity check, and depth quantization. The supplement also specifies how episode metrics are aggregated and identifies the recorded data schema.

C.1 Signal Semantics and Depth Quantization

  • Inward-facing rays measure penetration past the nominal contact surface; a two-pass reverse-ray check validates hits on dynamic task objects and articulated-object links within 15 mm.Static scene geometry is excluded; missed or invalid rays receive zero distance and a separate binary validity-mask entry.Source: §C.1 · p. 28
  • Valid ray distances are converted to millimeters and quantized into 8-bit values, with finer 0.005 mm levels below 0.5 mm and coarser 0.03 mm levels above that threshold.Values of 5.15 mm and above saturate at 255; the map is spatially smoothed before storage.Source: §C.1 · pp. 28–29

D HDF5 Schema Details

  • The full-collection HDF5 schema records eight modalities and specifies frame, joint, camera, and tactile-map dimensions.The released tactile resolution is 240 × 240.Source: Table S8 · p. 29, Table S7 · p. 29
  • Benchmark-level episode metrics are equal-weight means across tasks, and the default evaluation uses the reach-and-stop protocol unless fixed-horizon evaluation is explicitly identified.Because each reported cell has 50 episodes, the task-macro mean equals the pooled episode mean for metrics averaged over episodes.Source: D HDF5 Schema Details · p. 30

E.1 Primary Metrics

The benchmark centers completion on stable success and supplements it with progress, efficiency, safety, and robustness measures. Success requires a task’s terminal predicate to remain true for a configured dwell time.

E.1 Primary Metrics

  • Stable success rate is the primary completion outcome, while LSCR and SafeSR add complementary views of progress and safety.Stable success rate can include a central 95% confidence interval computed from 10,000 percentile-bootstrap resamples.Source: §E.1 · p. 30
  • Under reach-and-stop, an episode succeeds only after its terminal predicate remains true for the configured dwell time, defaulting to 0.5 s.Otherwise, it ends at the evaluation horizon or when evaluation cannot proceed; transient contacts or unstable placements do not count as success.Source: §E.1 · p. 30
  • LSCR averages dependency-valid milestones reached at any time, with each stage latched once its predicate and dependencies are satisfied.Terminal-state stage status is retained separately as a diagnostic.Source: §E.1 · p. 30
  • Completion time is a success-conditional mean and is undefined when no episode succeeds, so it should be read with SR and successful-episode count.Expert-normalized speed and step-based execution counts are auxiliary diagnostics, not primary ranking fields.Source: §E.1 · p. 31
  • Hard violations combine task-defined drops and tracked-object high-speed events by default; high-speed events are severe-motion proxies, not contact-force or collision measurements.Joint-limit rules enter the core safety criterion only when a task explicitly promotes them; violation-event rates need not sum to HVR.Source: §E.1 · p. 31
  • Robustness uses None, Equi., Inv., and Full channels, reporting channel-wise SRs and RR as a supporting descriptor rather than a paired causal estimate.The aggregate excludes None and weights shifted channels by their episode counts.Source: §E.1 · p. 31

E.2 Diagnostic Metrics

Diagnostic metrics expose task progress, tool use, efficiency, safety, and grasp behavior beyond primary rankings. Evaluation summaries retain the protocol and seed information needed to interpret and reproduce scores.

E.2 Diagnostic Metrics

  • Diagnostic metrics are not primary ranking fields because their failure-mode and execution measures are not uniformly applicable or comparable across tasks and embodiments.A quantity undefined for a task is marked unavailable, not zero.Source: §E.2 · p. 31
  • Completion diagnostics distinguish ever-satisfied and terminal-state predicates, while stage diagnostics separate terminal completion from latched progress reached earlier.LSCR remains the primary stage-completion measure.Source: §E.2 · p. 32
  • Task efficiency is an expert-reference speed ratio available only for successful episodes with a configured reference, and is comparable only under matched evaluation conditions.Execution summaries also report rollout length and episodes ending because evaluation cannot proceed.Source: §E.2 · pp. 32–33
  • Safety diagnostics track violations and kinematic constraints, while GSI proxies grasp stability using lift, holding, object motion, hold duration, and slip.GSI is used because reliable hand–object contact geometry and hand forward kinematics are not uniformly available across embodiments.Source: §E.2 · p. 32
  • Grasp diagnostics also distinguish peak grasp quality from positive-score object–step observations.These are separate diagnostic summaries, not primary ranking fields.Source: §E.2 · pp. 31–32
  • Tool diagnostics measure allowed-tool selection and stable tool switching; motion summaries cover joint velocity, acceleration, jerk, and effort under matched embodiment and timing.Tool-switch success is unavailable when no switch is attempted.Source: §E.2 · p. 33

E.3 Reproducibility Metadata

  • Each evaluation summary records policy identity, seeds, episode and failure counts, termination rule, horizon, and dwell time; randomized rollouts retain resolved seeds and sampled parameters.This metadata supports reconstruction under matched simulator, backend, task, and perturbation settings.Source: §E.3 · p. 33

F Policy Training Configurations

ACT and Diffusion Policy are trained from scratch, while pi0.5 and GR00T N1.5 are fine-tuned from public checkpoints. The configurations adapt each policy’s inputs and action outputs to task–embodiment settings.

F Policy Training Configurations

  • ACT uses four camera views and proprioception to predict 30-step action chunks, replanning each step and temporally ensembling overlapping predictions.It is trained from scratch with a cVAE formulation and an L1 reconstruction loss plus KL regularization.Source: F Policy Training Configurations · p. 33
  • Diffusion Policy conditions on three recent observations, predicts an 8-step trajectory, and executes six actions before replanning.It is trained from scratch with an image-conditioned 1D U-Net and a DDPM epsilon-prediction objective.Source: F Policy Training Configurations · p. 34
  • pi0.5 is fully fine-tuned from its public base checkpoint, predicts 20-step chunks from four RGB views, proprioception, and task instruction, and executes each chunk before replanning.Compatible pretrained parameters are retained; mismatched state/action projection layers remain randomly initialized.Source: F Policy Training Configurations · p. 34
  • GR00T N1.5 is fine-tuned per task with four synchronized RGB views, proprioception, and instruction, using a padded 64-dimensional state and extended action interface.It predicts and executes 16-step chunks generated with four flow-matching denoising steps.Source: F Policy Training Configurations · p. 34

G Generalization Configuration

The generalization configuration perturbs visual conditions, object and camera poses, distractors, and table height, with task constraints retained where specified. Tactile observations visualize per-site quantized penetration depth across the supported embodiments.

G Generalization Configuration

  • Scene-background randomization samples from 60 indoor scenes, including seen and unseen partitions, while background geometry affects rendering but not collision or contact simulation.Tabletop texture randomization samples from 11,824 materials while preserving surface geometry and contact behavior.Source: G Generalization Configuration · pp. 34–35
  • Lighting randomization varies distant and dome-light parameters to probe sensitivity to illumination, shadows, contrast, and exposure.The stated ranges are shared across seen and unseen splits.Source: G Generalization Configuration · p. 35
  • Object-pose perturbations adjust positions and yaw within valid bounds, while task-specific constraints can fix poses or narrow perturbation ranges.Invalid samples are reduced progressively, with nominal pose retained only if no valid perturbed pose is found.Source: G Generalization Configuration · p. 35
  • Camera randomization perturbs world and wrist viewpoints; stereo cameras share a sample, and seen/unseen profiles use the same numerical ranges.The configuration therefore probes calibration and viewpoint perturbations rather than a separate camera-hardware split.Source: G Generalization Configuration · p. 35
  • Distractor randomization adds one to three dynamic objects from seen and unseen assets, excluding duplicates or semantically confusable objects and respecting placement constraints.It evaluates grounding under occlusion, ambiguity, crowding, and incidental contact without deliberately blocking the primary workspace.Source: G Generalization Configuration · p. 35
  • Table-height randomization shifts the surface by up to 0.05 m while keeping the robot mount fixed and updating dependent placements and collision checks.This changes robot-to-table geometry for reaching, grasping, and contact heights.Source: G Generalization Configuration · p. 35
  • Table S10 lists the composition and strengths of four evaluation channels, with some parameters replayed from matched anchor episodes and task-level pose overrides allowed.The caption notes that object overrides may narrow or disable pose perturbations.Source: Table S10 · p. 36

H Collected Tactile Observations

  • Tactile figures show one replayed frame per embodiment, with a heatmap per tactile site where brighter pixels encode larger quantized penetration depth.The values range from 0 to 255 across representative observations from 12 robot–hand embodiments.Source: H Collected Tactile Observations · p. 37

Related papers

Loading 2609.15726v1…