PaperlayerView exact revision on arXiv

Source-linked AI summary

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks

Yong Wang, Kecheng Zheng, Xing Zhu, Chao Xu, Yujun Shen, Shengze Cai

arXiv:2610.25459v1cs.ROcs.CVcs.LG

TL;DR

Existing benchmarks leave bimanual dexterous manipulation capabilities insufficiently characterized. BiDex-100 evaluates 106 tasks across five capability axes, with controlled out-of-domain variants, and seven VLA models reveal capability-specific performance and robustness patterns.

  • Problem

    Few benchmarks specifically target dexterous manipulation, leaving existing models’ capabilities insufficiently characterized.

    Source: §I · p. 2

  • Method

    BiDex-100 provides 106 tasks across five capability axes, controlled out-of-domain variants, two embodiments, and 8,080 teleoperated demonstrations, and evaluates seven VLA models under one protocol.

    Source: §I · p. 2, §VI · p. 7

  • Results

    On ID Long-Horizon tasks, the highest success rate is 38.20%, while the highest progress score is 66.78, exposing near-complete attempts hidden by binary success.

    Source: C. Main Results · p. 7

  • Takeaways & Limitations

    Per-axis profiles, ID/OOD gaps, and low-data regimes provide a diagnostic framework for capability-specific and reproducible evaluation.

    Source: §XIV · p. 23

    Robustness evidence measures aggregate performance under combined scripted shifts, including appearance, layout, distractor, scale, and mirroring changes.

    Source: §XIV · p. 23

Abstract

from arXiv · show

Bimanual manipulation with dexterous hands has attracted growing attention in research on robotic action models. However, few studies have developed benchmarks specifically tailored for dexterous manipulation, leaving the dexterous manipulation capabilities of existing models insufficiently characterized. In this paper, we present BiDex-100, a simulation benchmark for bimanual dexterous manipulation that combines broad task coverage, multi-dimensional evaluation, and high-DoF hand embodiments. Concretely, the benchmark comprises 106 tasks organized along five capability axes: generalization, instruction following, limited demonstrations, long horizon, and memory. Each task is paired with an out-of-domain variant featuring controlled visual and spatial changes, allowing systematic assessment of robustness. In addition, the benchmark supports two embodiments and provides 8,080 success-validated teleoperated demonstrations, totaling 6.25 million frames. We evaluate seven recent Vision-Language-Action models under a unified protocol, establishing baselines for capability-specific performance, robustness, and data efficiency. The website is available at https://BiDex-100.github.io/BiDex-100/.

I. INTRODUCTION

BiDex-100 addresses the lack of benchmarks that combine bimanual dexterous hands with capability-focused, reproducible evaluation. It organizes 106 tasks across five capability axes and provides matched out-of-domain variants, demonstrations, and a unified protocol for evaluating VLA models.

I. INTRODUCTION

  • Existing benchmarks often use grippers, narrowly scoped dexterous tasks, or binary scoring, leaving bimanual dexterity and capability-specific diagnosis uncombined.The paper identifies gaps in task coverage, language grounding, demonstrations, compositional scoring, and matched in-domain/out-of-domain evaluation.Source: §I · p. 2
  • BiDex-100 introduces 106 bimanual dexterous manipulation tasks organized around generalization, instruction following, limited demonstrations, long horizon, and memory.The resulting evaluations produce capability profiles rather than a single aggregate score.Source: §I · p. 2
  • Each task has a reproducible out-of-domain counterpart with controlled changes to appearance, layout, distractors, object scale, or left-right configuration.This design supports direct comparison of in-domain performance and robustness.Source: §I · p. 2
  • The benchmark releases 8,080 success-validated teleoperation episodes spanning 6.25M frames and evaluates seven recent VLA models under a unified protocol.The protocol uses a fixed 58-DoF interface, shared demonstration corpus, and stratified automated scoring.Source: §I · p. 2, §II · p. 3

II. RELATED WORK

  • Prior suites center on grippers, while dexterous benchmarks often have small task sets; newer bimanual gripper benchmarks remain largely binary, unlike BiDex-100’s progress scoring.DexJoCo is described as a bimanual dexterous benchmark with demonstrations and matched ID/OOD variants, but it has 11 tasks and binary scoring.Source: §II · p. 2
  • BiDex-100 also builds on teleoperation approaches by unifying immersive video and force-aware exoskeleton input through an optimization-based retargeting stack.The cited modalities are Open-TeleVision [25] and FACTR [26].Source: §II · p. 2

III. THE BIDEX-100 BENCHMARK

BiDex-100 organizes dexterous bimanual evaluation around interpretable capability profiles and reproducible scoring. It pairs two embodiments with a declarative task formalism that tracks subtask progress while preserving strict success criteria.

A. Design Principles

  • BiDex-100 prioritizes dexterity, capability diagnosis, and reproducible, comparable evaluation over conventional gripper skills, task categories, or human-rated scores.Its tasks stress capabilities such as visual and geometric generalization, instruction fidelity, data efficiency, horizon depth, and latent-state memory.Source: A. Design Principles · p. 3

B. Embodiments and Interfaces

  • The dual Franka embodiment uses two 7-DoF arms with 22-DoF hands and executes 96 tasks; the remaining 10 Generalization tasks use the Astribot S1.Observations combine synchronized RGB-D views and proprioception; actions are 58-dimensional joint targets, with end-effector control also available through inverse kinematics.Source: B. Embodiments and Interfaces · p. 3

C. Declarative Task Formalism and Scoring

  • Each task specifies placement regions, movable objects, articulation initial states, a compositional goal, and language, with goals formed from evaluation subtasks and typed predicates.The language may include an instruction schedule that changes mid-episode; predicates cover geometry, events, memory, and forbidden outcomes.Source: C. Declarative Task Formalism and… · pp. 3–4

B. Embodiments and Interfaces

  • Across 8,080 demonstrations, the hands substantially share workspace, and eight principal components capture 80% of bimanual postural variance.The hands coordinate at close range through grasp changes, while all five fingers engage and thumb CMC and finger MCP joints show the largest excursions.Source: Figure 2 · p. 3, Figure 4 · p. 4, Figure 3 · p. 4

C. Declarative Task Formalism and Scoring

  • Subtask completion latches once achieved, while must-hold conditions, latch violations, and ordering constraints prevent credit for transient or invalid outcomes.Must-hold predicates must remain true at episode end; a forbidden state can invalidate a subtask, and later subtasks wait for earlier completion.Source: C. Declarative Task Formalism and… · p. 4
  • Subtasks receive nonnegative integer weights totaling 100, supporting a progress score while strict success remains independent of that score.Across 106 specifications, tasks contain 1–10 subtasks, averaging 3.9, and compose 470 predicate instances.Source: C. Declarative Task Formalism and… · p. 4

D. Five Capability Axes

BiDex-100 organizes 106 tasks around five capability axes, profiling distinct dimensions of bimanual dexterous manipulation rather than reducing performance to one question.

Generalization

  • Generalization spans 40 single- and few-stage skills over varied objects, with randomized object poses testing transfer across geometry and embodiment.Tasks include lifting, placement, insertion, threading, and cooking; ten kitchen tasks run on Astribot S1 and thirty on dual Franka.Source: D. Five Capability Axes · p. 4

Instruction-Following

  • Instruction-Following tests command fidelity through object-count variants, combined instructions, and mid-episode switches that require following the current command.Count variants share a scene but change the requested number of objects or actions.Source: D. Five Capability Axes · p. 4

Limited-Demonstrations

  • Limited-Demonstrations fixes a low-data regime at 30 demonstrations per task, including articulated mechanisms requiring force-aware actuation and bimanual hand coordination.The 26-task axis includes microwaves, cash registers, and rice cookers.Source: D. Five Capability Axes · p. 4

Long-Horizon

  • Long-Horizon covers multi-stage chores with 4–9 subtasks and 6.5 objects per scene, requiring plans that persist beyond a policy’s action chunk.Sampled demonstrations average 48.3 seconds.Source: D. Five Capability Axes · p. 4

Memory

  • Memory tasks require temporal state tracking under perceptual aliasing; they cannot be solved from a single frame and test more than reactive control.Examples include recalling a hidden color, preserving insertion order, and remembering revealed contents after they are concealed.Source: D. Five Capability Axes · pp. 4–5

E. Matched In-Domain / Out-of-Domain Evaluation

BiDex-100 pairs each task with a reproducible, evaluation-only OOD variant to compare in-domain performance and robustness. Demonstrations can be collected through two teleoperation modalities that feed a shared retargeting stack.

E. Matched In-Domain / Out-of-Domain Evaluation

  • Each task has an evaluation-only OOD counterpart with reproducible scene, camera, layout, and compatible optional shifts, while each evaluation key is held identical across models.Optional changes include distractors, articulated-object rescaling, and left-right mirroring; no OOD episode enters training.Source: E. Matched In-Domain / Out-of-Domain… · p. 5, Figure 6 · p. 5

A. Demonstration Collection

  • An exoskeleton glove tracks finger joints, while a head-mounted device streams wrist poses and hand keypoints; both modalities feed one retargeting stack and homogeneous data stream.The shared mapping is parameterized by embodiment-specific kinematics, joint bounds, and scale factors.Source: A. Demonstration Collection · p. 5

B. Dataset Contents and Statistics

The dataset combines success-validated teleoperation episodes across 106 tasks with synchronized observations and actions, varied assets, and differing task-duration profiles.

B. Dataset Contents and Statistics

  • 8,080 success-validated episodes cover 106 tasks, totaling 6.25 million frames of 25 Hz teleoperation data and 94 GB.Most listed axes and core instruction tasks have 100 episodes per task; Limited-Demonstrations and shelf-transfer instruction variants have 30.Source: B. Dataset Contents and Statistics · p. 5
  • The 106 tasks and 8,080 demonstrations are distributed across five capability axes, with statistics also covering episode durations, compositional depth, and goal-predicate use.The figure identifies the 14 most frequent predicates among 42 types.Source: Figure 8 · p. 6
  • The dataset records head and wrist camera data, 58-DoF actions, 130-dimensional observations, subtask progress, and a serialized initial state to enable exact reset replay.LeRobot v2.1 and v3.0 layouts [34] are provided as parquet and H.264 mp4.Source: B. Dataset Contents and Statistics · p. 6
  • The benchmark uses 266 visible assets across nine semantic categories; Long-Horizon and Memory demonstrations average 48.3 and 50.9 seconds, respectively.Every visible asset referenced by task specifications is represented and simulated with mesh, mass, and friction properties.Source: B. Dataset Contents and Statistics · p. 6, Figure 9 · p. 6

V. EXPERIMENTS

Seven recent VLA models are trained and evaluated under a unified benchmark protocol, with results reported by capability and evaluation domain. The study establishes diagnostic baselines while identifying the tabletop simulation setting as a boundary of its scope.

A. Baselines and Training Protocol

  • Seven recent VLA models use official checkpoints co-trained on one mixture of demonstrations from all 106 tasks, with a shared 58-DoF interface.Optimizer, schedule, freezing strategy, and training steps follow each model’s official configuration.Source: A. Baselines and Training Protocol · p. 6

C. Main Results

  • Fig. 10 reports overall success and progress averaged across 106 tasks and both domains; Tables III–IV show capability-stratified results and ID-to-OOD degradation.Evaluation uses 50 rollouts per domain for each task and 10,600 rollouts per model.Source: Figure 10 · p. 7, C. Main Results · p. 7

B. Evaluation Protocol

  • Each model receives 50 ID and 50 OOD rollouts per task, with matched task, seed, episode index, reset, and OOD randomization conditions.Task-specific horizons use an explicit max_steps value when available, or 1.2× the P90 demonstration length otherwise.Source: B. Evaluation Protocol · p. 7

C. Main Results

  • On ID Long-Horizon tasks, the highest success rate is 38.20%, while the highest progress score is 66.78, revealing near-complete attempts hidden by binary success.Capability-stratified success and progress vectors distinguish short-horizon performance from memory and long-horizon competence.Source: C. Main Results · p. 7

VI. CONCLUSION

  • The conclusion presents BiDex-100 as a five-axis, two-embodiment benchmark with progress-based scoring, matched ID/OOD evaluation, and 8,080 validated demonstrations.Its rigid-body tabletop setting leaves sim-to-real transfer and richer physical phenomena for future work.Source: §VI · p. 7

BiDex-100: Supplementary Material

  • The supplement documents the goal-predicate catalog, task and scoring semantics, OOD rules, dataset quality control, model configurations, evaluation protocol, task catalog, and limitations.Source: BiDex-100: Supplementary Material · p. 9

VII. GOAL-PREDICATE CATALOG

BiDex-100 specifies tasks through typed goals and scored subtasks, then pairs each in-domain task with a reproducible out-of-domain variant that preserves its goal. Its evaluator supports instruction schedules, episode horizons, and distinct rules for latching and completion.

VII. GOAL-PREDICATE CATALOG

  • Table V catalogs 42 canonical predicate types used across the 106 released base task specifications.The evaluator also accepts backward-compatible aliases for several types.Source: Table V · p. 9, §VII · p. 9

VIII. TASK SPECIFICATION AND SCORING SEMANTICS

  • Each task defines an embodiment, scene, placed objects and spawn regions, a natural-language instruction, a predicate-based goal, and scored subtasks.Object poses are sampled uniformly from their assigned spawn regions at reset.Source: §VIII · p. 10
  • Subtasks latch when all predicates hold simultaneously; strict success requires every subtask to complete, while progress scores weight subtask completion and ordered tasks gate later credit.Must-hold and latched-violation conditions also affect effective completion.Source: §VIII · p. 10
  • Instruction-following switch tasks expose the active segment of an ordered language schedule while retaining the declared predicate set and subtask sequence for scoring.Nonfinal segments specify a control-step boundary; the final instruction remains active for the rest of the episode.Source: §VIII · p. 10
  • Episode horizons use a task-specific max_steps when present, otherwise 1.2× the 90th-percentile demonstration length or a 30-second fallback; timeouts retain latched partial scores.The fallback applies when demonstration lengths are unavailable.Source: §VIII · p. 10

IX. OUT-OF-DOMAIN TRANSFORMATION RULES

  • Each OOD variant keeps the goal unchanged while applying reproducible scene changes, including appearance, camera, layout, distractors, scale, or left-right mirroring.The task YAML fixes applicable transformations, and episode count, horizon, and scoring rules are shared across domains.Source: §IX · p. 10

X. DATASET STRUCTURE AND QUALITY CONTROL

This section describes the dataset’s construction and replayable initial states, the shared training and evaluation setup for seven baselines, and the task catalog’s structure, scope, and scoring. It includes a Generalization catalog spanning 40 skills across two robot platforms and transfer across object geometry and embodiment, with tasks scored using weighted completion criteria from single-subtask to multi-step.

X. DATASET STRUCTURE AND QUALITY CONTROL

  • Episodes are reviewed by operators and automatically replayed to recheck goal predicates, making the released corpus success-only; saved initial states enable exact replay and deterministic resets.Initial-state dimensions range from 120 to 222, depending on the object set.Source: §X · p. 10

XI. PER-MODEL TRAINING CONFIGURATIONS

  • All seven baselines are fine-tuned on the same all-category mixture with 58-DoF absolute joint targets and head and wrist color streams, using selected checkpoints at model-specific training steps.The section documents optimizer settings and individual model configurations.Source: §XI · p. 11

XII. EVALUATION PROTOCOL

  • Each model-task-domain evaluation uses two fixed seeds and 25 episodes, with deterministic object placement and domain-specific robot resets for reproducible rollouts.Episodes end on full success, timeout, or—in OOD scenes with distractors—grasping a distractor. Metrics include strict success, progress, subtask latch times, and time-to-first-failure; overall performance equally weights 212 task-domain means.Source: §XII · p. 12

XIII. TASK CATALOG

  • The catalog specifies all 106 tasks with scene images, identifiers, scored subtasks, demonstration counts, language instructions, and criteria; it also defines must-hold and latch-violation markers.For Instruction-Following shelf-transfer switch and combination tasks, ten specifications have 30 demonstrations each and ten have 100 each.Source: §XIII · p. 12

A. Generalization (40 specifications, 40 with released demonstrations)

  • The 40 Generalization tasks span single- and few-stage skills, with ten kitchen tasks on Astribot S1 and 30 on dual FR3v2; randomized object poses mean the axis tests transfer across geometry and embodiment.Skills include lifting, placement, insertion, threading, and cooking across food, tableware, containers, and tools.Source: A. Generalization (40 specifications,… · p. 12
  • Scoring examples include one-subtask apple placement, two-subtask spoon placement, and weighted microwave steps.The apple task has 100 demos and a 100-point container-placement criterion; the spoon task has 100 demos and two 50-point drop criteria; the microwave example assigns 40, 30, and 30 points across its steps.Source: A. Generalization (40 specifications,… · p. 16

B. Instruction-Following (20 specifications, 20 with released demonstrations)

The instruction-following specifications test whether policies act on the current verbal command across variations in counts, combined instructions, and mid-episode switches. Their examples include paired-arm manipulation and changing object-placement commands.

B. Instruction-Following

  • Count variants, combined instructions, and mid-episode switches test command fidelity through different verbal-command structures.Switch tasks provide a language schedule, and grasping a wrong object terminates the episode.Source: B. Instruction-Following (20… · p. 16
  • The specifications also include complementary bimanual tasks such as hammering nails, watering plants, and threading bagels and donuts onto rods.The supplied examples vary which arm handles each object and, for threading, which rod receives it.Source: B. Instruction-Following (20… · pp. 16, 18
  • A bottle task pairs left-arm cap opening with right-arm sugar placement, while related variants require one or two cubes and penalize extra cubes entering the bottle.The scoring also penalizes picking up extra cubes.Source: B. Instruction-Following (20… · p. 16
  • Mango-packing tasks contrast commands for placing four mangoes across two baskets with commands selecting only the two outermost mangoes.The latter scores against extra mangoes entering either basket or being picked up.Source: B. Instruction-Following (20… · pp. 16–17
  • Switch examples change the requested object or destination during an episode, including moving a lemon to a shelf and then changing the command to jam.Other examples reverse the placement of the same orange or jam across the two instruction phases.Source: B. Instruction-Following (20… · p. 17

C. Limited-Demonstrations (26 specifications, 26 with released demonstrations)

The limited-demonstrations specifications provide 30 demonstrations per task and span articulated mechanisms as well as object manipulation. Examples range from single-step actuation to multi-stage tasks requiring coordinated handling and sequencing.

C. Limited-Demonstrations

  • Every task in this data-scarce skill-acquisition axis has exactly 30 demonstrations, including ten articulated-mechanism tasks.Those tasks include hinged, sliding, and rotary joints, and the benchmark describes 44-DoF hand coordination as decisive.Source: C. Limited-Demonstrations (26… · p. 18
  • The task set includes single-step mechanism actions such as pulling a slot-machine lever, flipping open a bottle cap, and opening household appliance lids.Examples include a microwave, rice cooker, bread maker, and scanner.Source: C. Limited-Demonstrations (26… · pp. 18–19
  • Other specifications require multi-stage object handling, including assembling an arch bridge and storing food in a freezer through opening, loading, and closing.The bridge task scores placement of two piers, a deck, and a capstone; the freezer task includes sequential placement and closure.Source: C. Limited-Demonstrations (26… · pp. 19–20

D. Long-Horizon (10 specifications, 10 with released demonstrations)

The Long-Horizon tasks test multi-stage chores that require plans extending beyond a policy’s action chunk. Their progress-based subtasks cover coordinated handling of multiple objects and arms across varied kitchen and tabletop activities.

D. Long-Horizon (10 specifications, 10 with released demonstrations)

  • Multi-stage chores span barbecue grilling, fridge restocking, dessert plating, and recycling sorting, with 4–9 scored subtasks and 6.5 objects per scene on average.Their sampled demonstrations average 48.3 seconds, and success requires sustaining a plan beyond the policy’s action chunk.Source: D. Long-Horizon (10 specifications, 10… · p. 21
  • The task set also scores repeated placements and stacking, from inserting three plates into a dish rack to stacking sugar cubes with both arms.The plate task scores picking up and inserting each plate, while the cube task scores two stacking actions per arm.Source: D. Long-Horizon (10 specifications, 10… · pp. 21–22
  • Grilling and fridge-restocking tasks score sequential actions such as opening and closing lids or doors around placing several objects.The grill task has seven scored subtasks, while fridge restocking includes keeping a water bottle upright.Source: D. Long-Horizon (10 specifications, 10… · p. 21
  • Other chores require dividing objects between containers or completing nested placement sequences, including filling a pot and bottle, sorting food, and filling a jar with snacks.The pot-and-bottle task assigns actions across both arms and includes an articulated cap; the snack task requires returning the jar to a shelf after filling it.Source: D. Long-Horizon (10 specifications, 10… · p. 21

E. Memory (10 specifications, 10 with released demonstrations)

Memory tasks require tracking information across time, including when visual cues are removed or objects are moved. Their scoring covers order recall, color recall, and delayed actions after button sequences.

E. Memory (10 specifications, 10 with released demonstrations)

  • Memory tasks use perceptual aliasing or hidden contents to require temporal state tracking that cannot be solved from a single frame.Examples include recalling a cube’s color after a button sequence, matching removal order to insertion order, and remembering contents revealed before being hidden.Source: E. Memory (10 specifications, 10 with… · p. 22
  • Basket tasks score whether objects are removed in the same order they were placed, including tasks with three fruits or four picnic items.The scoring assigns half the points to placing the items and half to removing them in the original order.Source: E. Memory (10 specifications, 10 with… · p. 22
  • The color-swap task hides the cube’s color before requiring it to be placed in the matching-color cup after the cups swap colors five times.The matching-cup placement is worth 45 points in the task’s scoring.Source: E. Memory (10 specifications, 10 with… · p. 22
  • Other tasks test ordered uncovering and delayed actions, including uncovering cubes in sequence or pressing one button a specified number of times before another action.The button-then-grasp task requires four presses before the other arm picks up and places a mug; another task requires ten red-button presses before one blue-button press.Source: E. Memory (10 specifications, 10 with… · pp. 22–23

XIV. DISCUSSION AND LIMITATIONS

The benchmark uses five capability axes and progress-based predicate scoring to provide capability profiles, expose partial progress, and compare robustness under scripted ID/OOD shifts. The authors describe its physical and distributional scope limits and release evaluation resources for reproducible use.

XIV. DISCUSSION AND LIMITATIONS

  • Progress-based predicate scoring makes heterogeneous tasks commensurable and exposes partial progress when strict success is not reached.Per-axis vectors show whether short-horizon competence extends to memory and long-horizon tasks; ID/OOD gaps measure robustness under combined scripted shifts.Source: §XIV · p. 23
  • BiDex-100 is a rigid-body benchmark centered on tabletop food and kitchen scenes, and sim-to-real transfer of policies trained on its demonstrations remains unvalidated.MuJoCo does not model fluids, deformable objects, or rich frictional contact at hardware fidelity; the OOD shifts are scripted and finite.Source: §XIV · p. 23
  • Evaluating seven contemporary VLA models under one protocol establishes reference baselines and a diagnostic framework for capability profiles, robustness gaps, and low-data regimes.The code, tasks, data, and evaluation tooling are released for reproducible community use.Source: §XIV · p. 23

Related papers

Loading 2610.25459v1…