PaperlayerView exact revision on arXiv

Source-linked AI summary

DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu

arXiv:2610.29997v1cs.ROcs.CV

TL;DR

Existing dexterous trackers largely focus on single-hand tasks, while contact-rich bimanual data and coordinated hand-role learning remain challenging. DONIC trains one generalist bimanual tracker using BiHOI references and verified executions. Without fine-tuning, its single checkpoint reaches 86.79% success on unseen trajectories and 57.32% on held-out objects.

  • Problem

    Dexterous trackers remain largely limited to single-hand tasks, while fully tracked sequences with sustained two-hand contact are scarce and shared policies must coordinate changing hand roles and contact timing.

    Source: §1 · p. 2

  • Method

    DONIC combines PPO and behavioral cloning with verified-execution reuse, object-centered per-hand rewards, and Clip-Centered Advantages to train one bimanual tracker across references.

    Source: §1 · pp. 2–3

  • Results

    86.79% and 57.32% success on unseen trajectories and objects, respectively, with DONIC exceeding the stronger bimanual baseline by 77.36 and 45.73 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A single checkpoint tracks new references without further training, jointly across operation categories and human-motion datasets.

    Source: §1 · pp. 2–3

    The quantitative evaluation covers reference-conditioned, single-rigid-object tracking in simulation; unconstrained goal-conditioned manipulation and held-out operation-category generalization are outside its scope.

    Source: §D.6 · p. 25

Abstract

from arXiv · show

Despite recent progress in dexterous reference tracking, existing models focus primarily on single-hand manipulation and a limited range of interactions, leaving the bimanual dexterous behaviors common in real life unaddressed. The scarcity of contact-rich bimanual manipulation data and the increased complexity of coordinating contacts make developing a generalist bimanual tracker particularly challenging. Therefore, we present 1) DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, together with 2) BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects and emphasizing sustained bimanual interaction, where reconstructed interactions are retargeted into robot references. DONIC combines proximal policy optimization with behavioral cloning in a feedback loop that recycles verified successful executions into training demonstrations. An object-centered per-hand reward maintains object-motion feedback while supervising each hand according to its reference contact role. Clip-Centered Advantages remove reference-dependent advantage offsets while preserving within-trajectory differences. Together, these designs enable one single policy to learn coordinated manipulation across heterogeneous references. Without finetuning, a single checkpoint achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing the state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.

1. Introduction

The paper targets generalist bimanual tracking across diverse human references, addressing limited contact-rich data and the challenge of coordinating changing hand roles. DONIC combines object-centered per-hand rewards, Clip-Centered Advantages, and verified-execution reuse to train one policy.

1. Introduction

  • Fully tracked sequences with sustained two-hand contact remain scarce, while hand-pose accuracy alone does not ensure successful object manipulation.Shared training must also address changing hand roles, contact timing, and unequal clip difficulty.Source: §1 · p. 2
  • BiHOI adds 1,575 multi-view sequences spanning 229 objects, with symmetric and asymmetric interactions reconstructed and retargeted into robot references.Combined with public references, it broadens training objects and coordination patterns.Source: §1 · pp. 2–3
  • The object-centered per-hand reward maintains object-motion feedback while supervising each hand’s reference contacts, including asynchronous engagement.Clip-Centered Advantages subtract each clip’s mean estimated advantage before PPO updates, removing clip-dependent offsets while preserving within-clip differences.Source: §1 · p. 3
  • DONIC jointly trains one bimanual tracker across operation categories and human-motion datasets, evaluating a single checkpoint on unseen trajectories and objects without fine-tuning.Successful shared-policy executions are verified and reused as training demonstrations.Source: §1 · pp. 2–3
  • 86.79% and 57.32% success on unseen trajectories and objects, respectively, with DONIC exceeding the stronger bimanual baseline by 77.36 and 45.73 percentage points.The evaluation uses two public-data splits and bimanual adaptations of DexTrack and Dexplore.Source: §1 · p. 3

2. Related Work

Prior work includes interaction datasets, reference-conditioned trackers, and generalist manipulation policies, but much of the tracking literature learns per reference or focuses on single-hand settings. DONIC instead jointly trains one bimanual tracker across operation categories and human-motion datasets.

From human references to executable manipulation

  • Sequence-specific trackers use per-reference learning or task-specific residuals, whereas DONIC tracks diverse references without per-reference training or test-time fine-tuning.Examples include ManipTrans [15], DexMachina [23], CHORD [44], and ConTrack [17].Source: §2 · p. 3

Generalist manipulation policies

  • Generalist manipulation policies predominantly target single-hand settings; bimanual work includes category-conditioned policies and transfers to new tool or object geometries.BiDexHD distills a separate trajectory-conditioned policy for each operation category, while PhysGraph [16] transfers task-trained policies.Source: §2 · p. 3

Bimanual hand–object interaction datasets

  • Existing interaction datasets differ in capture and annotation conventions, and human kinematics alone do not ensure executable robot actions.BiHOI's pipeline harmonizes, retargets, and screens references for zero-residual feasibility.Source: §2 · pp. 3–4

3. Bimanual Hand–Object Dataset

BiHOI expands bimanual training data with multi-view recordings and a pipeline that converts reconstructed interactions into robot references. Its collection emphasizes sustained two-hand contact across varied objects and coordination patterns.

3. Bimanual Hand–Object Dataset

  • Table 1 compares source-dataset statistics and simultaneous-contact rates on analyzed recordings.The caption identifies the table’s scope; its footnote defines the pooled contact rate and unavailable entries.Source: Table 1 · p. 4, §3 · p. 4

3.1. Multi-View Capture and Dataset

  • 32 calibrated RGB cameras capture the tabletop workspace at 1920 × 1232 pixels and 30 fps, with complementary views supporting reconstruction under occlusion.Calibration supplies a common metric coordinate system.Source: §3.1 · p. 4
  • BiHOI contains 1,575 sequences across 229 objects and 14.77 hours, covering symmetric, asymmetric, and coordinated multi-object interactions.Among frames with at least one hand in contact, 65.91% have both hands in contact, versus 38.57% for H2O’s highest analyzed public subset.Source: §3.1 · p. 4

3.2. Annotation and Reference Construction

  • The annotation pipeline reconstructs metric hand–object trajectories from synchronized RGB recordings using multi-view object and hand reconstruction.It produces bimanual MANO parameters, a metric object mesh, and a 6-DoF object trajectory in a common world frame.Source: §3.2 · pp. 4–5
  • Reconstructed human motion is fitted to Wuji robot hands under joint limits, then paired with the object trajectory as wrist and finger references.Framewise fitting is initialized and regularized against the preceding solution.Source: §3.2 · p. 5

4. Methodology

DONIC trains one policy to reproduce object motion from references for both hands, using an object-centered per-hand reward and PPO with verified behavioral-cloning demonstrations. Clip-Centered Advantages address reference-specific advantage offsets while preserving within-reference differences.

4.1. Task Definition

  • One policy controls both hands from a reference of the hands and object, aiming to reproduce object motion through physical interaction.The policy is evaluated on unseen trajectories and objects without fine-tuning.Source: §4.1 · p. 5

4.2. Reference-Conditioned Bimanual Control

  • Each policy action corrects the next reference rather than accumulating corrections, and a PD controller executes the resulting target.The observation includes per-hand proprioception, object state, upcoming reference targets, errors, and fingertip-to-object distances.Source: §4.2 · p. 6

4.3. Object-Centered Per-Hand Reward

  • The reward tracks object position, orientation, and velocity throughout the clip while supervising each hand’s contact only when required by its reference.Engaged-hand signals are averaged so one hand’s lost contact does not erase the other hand’s feedback; inactive hands are excluded.Source: §4.3 · pp. 6–7
  • Contact is estimated from fingertip force and distance, while proximity guides contact acquisition and the hold term rewards required-contact coverage.The force–distance proxy cannot identify the collision partner, and instantaneous coverage permits sliding or rolling.Source: §4.3 · p. 7

4.4. Learning a Generalist from Verified Demonstrations

  • Verified demonstrations are added to a library and recycled into BC updates alongside PPO; CCA subtracts each reference clip’s mean advantage while preserving within-clip differences.CCA is applied after GAE, leaving the critic and temporal advantage estimation unchanged.Source: §4.4 · pp. 7–8

5. Experiments

The experiments assemble public and BiHOI bimanual references, reserve unseen-trajectory and held-out-object splits, and compare methods using shared evaluation conditions.

5.1. Setup

  • The Ours+BiHOI training pool combines 1,845 public-data clips with 1,575 BiHOI sequences, totaling 3,420 clips.Public clips require tracked hands and object, and each hand must contact the object at some point; simultaneous contact is not required.Source: §5.1 · pp. 8–9
  • Evaluation reserves 53 unseen trajectories of training objects and 164 clips involving held-out objects, with neither split used for training or demonstration collection.All methods use the same splits and are evaluated without fine-tuning.Source: §5.1 · p. 9
  • The bimanual protocol reports object, wrist, and finger-joint errors, plus SR and Comp based on joint criteria for both hands.SR requires both hands to meet object/hand translation, rotation, and finger-error thresholds; Comp is the trajectory-averaged fraction of frames satisfying all three conditions for both hands.Source: §5.1 · p. 9
  • DexTrack-bi and Dexplore-bi use adapted interfaces and share DONIC’s public-data pool, simulator, robot model, and held-out splits.DexTrack-bi follows the released specialist–distillation pipeline, while Dexplore-bi retains its original reward and reference-scoped exploration.Source: §5.1 · p. 9
  • Table 2 reports reference tracking without fine-tuning using one checkpoint per method across both splits.Black and blue bold indicate the best and second-best results within each split, with ties sharing highlighting.Source: Table 2 · p. 10

Appendix D.3.

DONIC improves bimanual tracking on unseen trajectories and held-out objects, while ablations support its learning signals and added BiHOI data. The paper limits its generalization claims to the evaluated splits and notes that its quantitative evaluation uses simulated single-object interactions.

5.2. Main Results

  • 86.79% SR on UNSEEN-TRAJ exceeds Dexplore-bi by 77.36 percentage points; UNSEEN-OBJ SR is 57.32% versus 11.59%.UNSEEN-TRAJ Comp is 67.62%, exceeding Dexplore-bi by 39.75 percentage points; UNSEEN-OBJ Comp is 54.69% versus 26.37%.Source: §5.2 · p. 10
  • On two held-out object trajectories, DONIC coordinates both hands and more closely reproduces reference object motion than the bimanual baselines.Dexplore-bi moves its hands but produces limited object motion, while DexTrack-bi loses hand–object alignment.Source: §5.2 · p. 10

5.3. Ablation Studies

  • Removing CCA or the contact-ownership gate lowers success rate and completion, supporting both learning signals for joint bimanual performance.Removing CCA also increases all five tracking errors; the contact-ownership variant has lower completion on both splits despite lower individual errors in some cases.Source: §5.3 · p. 10
  • Adding BiHOI expands training from 1,845 to 3,420 clips and improves all seven metrics on both evaluation splits.Ours+BiHOI reaches 88.68% and 59.76% SR on unseen trajectories and objects, respectively.Source: §5.3 · pp. 10–11

6. Conclusion, Limitations and Future Work

  • The quantitative evaluation is limited to simulated single-object interactions using reconstructed, retargeted references; future work targets longer-horizon, multi-object, and physical-system evaluation.The authors also aim to evaluate on an independent benchmark.Source: §6 · p. 11

A. Additional Related Work

  • The evaluation tests new trajectories of training objects and trajectories with held-out objects, but does not establish generalization to held-out operation categories or arbitrary goals.Source: A. Additional Related Work · p. 15
  • The public bimanual pool requires tracked hands and object plus contact by each hand at some point, allowing sequential as well as simultaneous participation.BiHOI is described as complementing existing sources with a higher measured proportion of simultaneous bimanual contact in the analyzed recordings.Source: A. Additional Related Work · p. 15

B.1. Capture, Composition, and Training Inventory

BiHOI records natural bimanual interactions, and the appendix details how multi-view recordings become robot references. It also specifies which clips enter the training pool and how references are screened.

B.1. Capture, Composition, and Training Inventory

  • BiHOI contains 1,575 sequences across 229 objects and 14.77 hours, captured with 32 calibrated RGB cameras; recordings include symmetric, asymmetric, and coordinated multi-object interactions.Source: §B.1 · p. 15
  • 65.91% of BiHOI contact-active time has simultaneous bimanual contact, compared with 38.57% for the highest analyzed public subset, H2O.The measure pools frames across analyzed recordings, using frames with at least one hand in contact as the denominator; these subset measurements do not represent every recording released by each source.Source: §B.1 · p. 16, Table S1 · p. 16
  • Public-pool clips require both hands and the object to be tracked, with each hand contacting the object during the clip; the controller retains one selected object as a rigid mesh and pose.Continuous or simultaneous contact is not required for admission, and articulation or deformation is not explicitly modeled.Source: §B.1 · p. 16

B.2. Multi-View Reconstruction and Annotation

  • The reconstruction pipeline extracts masks and metric depth, reconstructs and tracks the object, reconstructs calibrated hand motion, then jointly refines hand and object estimates using shared depth.Hand motion combines triangulated joints, multi-view keypoints, and calibrated projections; joint refinement optimizes per-frame hand and object poses plus a sequence-shared object scale.Source: Figure S1 · p. 17, §B.2 · pp. 17–18
  • Joint refinement aligns geometry to depth and regularizes initialization and temporal evolution, but includes no explicit contact, penetration, collision, or snapping losses.Translation and rotation corrections are bounded to 3 cm and 20° of the input trajectory.Source: §B.2 · pp. 18–19

B.3. Kinematic Retargeting and Replay Screening

  • Retargeting fits hand keypoints to robot links, pairs robot configurations with the object trajectory, and screens references through zero-policy-correction replay.Replay excludes immediate object loss and ejection, while retaining contact-without-lifting through complete-execution categories for closed-loop training.Source: §B.3 · p. 19

C. Method and Training Details

DONIC uses a shared two-hand controller with bounded corrections around reference poses, and a reward that combines hand, object, and contact supervision. PPO with clip-centered advantages is paired with behavioral cloning from verified successful executions.

C.1. Controller Interface

  • The shared policy outputs 52 actions for both hands from a 3,016-dimensional observation, with bounded corrections applied around the next reference at every step.The interface uses coordinate-specific correction bounds, and corrections do not accumulate.Source: §C.1 · pp. 19–20

C.2. Reward Definition and Coefficients

  • The reward combines hand tracking, object motion, per-hand contact, and contact maintenance, while inactive hands contribute no contact reward.Contact uses a force–distance proxy, and maintenance measures instantaneous coverage rather than enforcing no-slip contact.Source: §C.2 · pp. 20–21

C.3. Clip-Centered Advantages and PPO+BC

  • CCA subtracts each reference clip’s mean advantage after GAE, removing common offsets while preserving within-clip differences for PPO updates.The adjustment retains the critic and temporal advantage estimation; it is a mean shift, not standard-deviation scaling.Source: §C.3 · p. 21

C.4. Demonstration Collection, Sampling, and Compatibility

  • PPO is combined with behavioral cloning from a library populated by successful rollouts that are replay-verified before admission.Held-out references are excluded from training and demonstration collection.Source: §C.3 · p. 22, §C.4 · p. 22

C.5. Optimization and Simulation Settings

  • The generalist training setup uses Isaac Gym with 8,192 environments and separate actor and critic networks.Demonstrations use a separate minibatch budget, with λBC = 50; adaptive recovery sampling and object-position perturbation curricula are disabled.Source: §C.5 · p. 22

D.1. Evaluation Splits

Evaluation uses frozen checkpoints on unseen trajectories and held-out objects, with no evaluation clips used for training or demonstration collection. Bimanual tracking metrics require both hands to meet the stated criteria, and baselines use the same simulation and split identities.

D.1. Evaluation Splits

  • 53 unseen trajectories of training objects and 164 clips with held-out objects are evaluated using one checkpoint without fine-tuning.Neither split contributes transitions or demonstrations to training.Source: §D.1 · pp. 22–23

D.2. Bimanual Tracking Metrics

  • Bimanual success requires both hands to independently pass their trajectory-mean criteria, while completion tests the criteria at each frame.Hand-error metrics use the worse of the two per-hand trajectory means before averaging across clips.Source: §D.2 · pp. 23–24

D.3. Baseline Implementations and Comparison Scope

  • All methods use Isaac Gym, two Wuji hands, the same public training clips, and the same held-out split identities.The baselines adapt DexTrack’s specialist–distillation pipeline and Dexplore’s two-hand interfaces for comparison.Source: §D.3 · p. 24

D.4. Ablation Definitions and Additional Data

The ablations test removing CCA or the contact gate, and adding the BiHOI sequences to training. A separate qualitative comparison describes DONIC’s tracking relative to two bimanual baselines.

D.4. Ablation Definitions and Additional Data

  • The ablations change only the indicated learning component, while Ours+BiHOI expands the reference pool with all 1,575 captured sequences.The variants retain the full model’s training data and optimization settings except for the expanded pool in Ours+BiHOI.Source: §D.4 · p. 25
  • Removing either learning signal lowers success rate and completion on both splits; adding BiHOI improves all seven metrics on both splits.With BiHOI, success rate reaches 88.68% on unseen trajectories and 59.76% on held-out objects; unseen-object translation error falls from 15.74 to 13.96 cm.Source: §D.4 · p. 25

D.5. Qualitative Comparisons

  • Across five qualitative sequences, DONIC more closely reproduces reference hand–object coordination and object motion than the baselines.The baselines show limited object motion or loss of hand–object alignment.Source: §D.5 · p. 25

D.6. Limitations

The evaluation is limited to reference-conditioned, single-rigid-object tracking in simulation and does not test unconstrained goal-conditioned manipulation or held-out operation-category generalization.

D.6. Limitations

  • Quantitative evaluation covers reference-conditioned, single-rigid-object tracking in simulation, not unconstrained goal-conditioned manipulation or held-out operation-category generalization.The capture inventory includes broader object types than the rigid-object representation used in the evaluation.Source: §D.6 · p. 25
  • Contact supervision uses a net-force/proximity proxy, and reconstruction has no explicit contact or collision loss.These constraints apply to the stated contact supervision and reconstruction setup.Source: §D.6 · p. 25

Related papers

Loading 2610.29997v1…