PaperlayerView exact revision on arXiv

Source-linked AI summary

DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation

Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song

arXiv:2505.24853v1cs.ROcs.AIcs.LG

TL;DR

DexMachina studies how to learn dexterous policies that track object states from human hand-object demonstrations in challenging long-horizon bimanual tasks. It uses decaying virtual object controllers with motion and contact guidance, and its benchmark results show stronger performance than baseline methods while supporting functional hand comparisons.

  • Problem

    Functional retargeting must learn feasible dexterous manipulation from human demonstrations despite high-dimensional exploration, complex contact sequences, and the embodiment gap between human and robot hands.

  • Method

    DexMachina is a curriculum-based reinforcement-learning algorithm that uses virtual object controllers and motion and contact guidance before gradually transferring control to the policy.

  • Results

    DexMachina significantly outperforms baseline methods across a benchmark spanning diverse dexterous hands, articulated objects, tasks, and long-horizon demonstrations.

  • Takeaways & Limitations

    The algorithm and benchmark enable functional comparisons across dexterous hand designs and provide a platform for evaluating desirable hand capabilities.

  • Takeaways & Limitations

    The approach relies on privileged simulator state information and high-quality human hand-object demonstrations that can be difficult or expensive to obtain in the real world.

Abstract

from arXiv · show

We study the problem of functional retargeting: learning dexterous manipulation policies to track object states from human hand-object demonstrations. We focus on long-horizon, bimanual tasks with articulated objects, which is challenging due to large action space, spatiotemporal discontinuities, and embodiment gap between human and robot hands. We propose DexMachina, a novel curriculum-based algorithm: the key idea is to use virtual object controllers with decaying strength: an object is first driven automatically towards its target states, such that the policy can gradually learn to take over under motion and contact guidance. We release a simulation benchmark with a diverse set of tasks and dexterous hands, and show that DexMachina significantly outperforms baseline methods. Our algorithm and benchmark enable a functional comparison for hardware designs, and we present key findings informed by quantitative and qualitative results. With the recent surge in dexterous hand development, we hope this work will provide a useful platform for identifying desirable hardware capabilities and lower the barrier for contributing to future research. Videos and more at https://project-dexmachina.github.io/

1 Introduction

The paper frames functional retargeting as learning feasible dexterous manipulation from human hand-object demonstrations, focusing on challenging long-horizon bimanual tasks with articulated objects. DexMachina addresses these challenges with curriculum-based learning and a benchmark for comparing dexterous hands.

  • Prior learning-based methods often target simple, short-horizon tasks and face manual reward engineering, costly data collection, and the human–robot embodiment gap.
  • Functional retargeting learns dexterous hand policies that manipulate an object along a demonstrated object-state trajectory, unlike kinematic retargeting, which does not ensure feasibility.
  • Long-horizon bimanual manipulation with articulated objects combines high-dimensional exploration, intricate contact sequences, and infeasible direct transfer from human motions to robot actions.
  • DexMachina uses virtual object controllers plus motion and contact guidance, gradually shifting from human-motion imitation to policy-controlled manipulation as controller strength decays.
  • The benchmark contains 6 dexterous hands and 5 articulated objects, while DexMachina outperforms baselines across varied hands, objects, and long-horizon demonstrations.
  • The benchmark and algorithm support functional comparisons of hardware using policy-learning performance as a measure of hand functionality and readiness to learn from human guidance.

2 Related Work

Prior work has used reinforcement learning, imitation learning, and curriculum learning for dexterous manipulation, but long-horizon bimanual tasks remain difficult because exploration, task-specific engineering, and robot-specific data are costly.

  • Reinforcement learning has addressed in-hand orientation and single-hand grasping, while longer-horizon manipulation remains challenging because reward design burdens exploration.
  • Model-based methods have handled tasks such as ball dribbling and Rubik’s cube turning but require careful engineering for each object and task.
  • The benchmark evaluates policies across several dexterous hands and diverse tasks defined by human demonstrations, complementing existing dexterous-manipulation RL benchmarks.
  • Imitation learning can reduce exploration burdens, but dexterous manipulation often requires accurate on-robot action data or teleoperation systems customized to each hand embodiment.
  • Human hand data has supported grasp affordances, retargeting, and co-training, but prior approaches were mainly limited to short-horizon manipulation such as grasping.
  • Curriculum learning moves from easier to more difficult problems, extending optimization warm starts and relaxed physical constraints to reinforcement-learning methods.

3 Functional Retargeting Formulation

Functional retargeting is formulated as learning a robot policy that tracks object states in a human hand-object demonstration, including pose and articulation for articulated objects.

  • Given an object, one human hand-object demonstration, and a pair of dexterous robot hands, the goal is to learn a policy that tracks the demonstrated object states.
  • A demonstration contains T timesteps of densely tracked object states and human hand poses, while articulated-object states include part pose and revolute joint angles.
  • The distance function evaluates rotation, position, and articulation-joint error, and the learned policy minimizes accumulated tracking error across timesteps.

4 Method

DexMachina combines demonstration-derived motion and contact guidance with an adaptive curriculum of virtual object controllers for long-horizon functional retargeting. The controllers initially drive object states toward demonstration targets, then decay as the policy learns to take over.

  • Demonstration Processing and Rewards: DexMachina combines task rewards with motion and contact guidance extracted from densely tracked human hand-object demonstrations.The guidance includes retargeted joints, reference keypoints, and approximated hand-object contacts.
  • Action Formulation: The hybrid action formulation uses retargeted wrist joints as base actions and policy residuals, while finger joints use normalized absolute actions.This constrains the policy’s action space toward the human demonstrator’s motion and improves learning efficiency.
  • Reward Design: The final reward is a weighted sum of task, motion-imitation, behavior-cloning, and contact rewards.The task reward tracks object position, rotation, and articulation, while auxiliary terms guide hand motion and interaction strategy.
  • Motivation: The curriculum addresses catastrophic early failures in long-horizon tasks by allowing the policy to experience the full sequence while object motion is stabilized.This is intended to discourage myopic strategies during complex contact and hand-repositioning sequences.
  • Virtual Object Controllers: Virtual object controllers initially move the object along demonstration targets, then decay their influence so the policy assumes greater control.Spring-damper control strength is parameterized by kp and kv and reduced through a structured hand-off.
  • Curriculum Scheduling: Curriculum gains begin high with critical damping and decay exponentially according to learning progress measured from reward history.Reward thresholds and deques track task and auxiliary learning progress before controller gains are reduced.

5 Experiments

Experiments evaluate DexMachina across diverse hands, demonstrations, baselines, ablations, and long-horizon tasks. The results show that curriculum-based guidance improves functional retargeting and enables hardware-adaptive strategies.

  • Experiment Setup: The evaluation uses Genesis simulation, PPO, seven demonstrations, five articulated objects, and six open-source dexterous hand models.Baselines include kinematic replay, ObjDex-style task reward, task plus auxiliary rewards, and ManipTrans.
  • Main Results: DexMachina consistently improves performance across four hands and seven tasks, especially on long-horizon demonstrations with complex motion sequences.The Waffleiron-300 example requires repeated mid-air opening, closing, lifting, and flipping actions.
  • Main Results: Auxiliary rewards improve some tasks inconsistently, whereas DexMachina significantly outperforms the no-curriculum setting using the same rewards.This isolates the curriculum as an important contributor to the observed gains.
  • Main Results: Task rewards and hybrid actions achieve reasonable short-horizon performance, while direct kinematic retargeting cannot complete the tasks beyond slightly lifting objects.The ObjDex re-implementation also performs better than its original reporting on the same demonstrations.
  • Hand-Specific Strategies: Policies learn different strategies for different hands, such as two-hand stabilization for Inspire and extra wrist movements for Schunk.These strategies reflect hardware constraints rather than strict imitation of the human motion and contact guidance.
  • Ablations: More restrictive wrist-motion bounds produce the best overall performance in the action ablation, while merely decaying physics parameters fails to improve stability.ManipTrans initially reaches high task reward but degrades as its curriculum progresses.
  • Hand Embodiment Analysis: Larger, fully actuated hands achieve higher final performance and better learning efficiency, although the conclusions are limited by the tested objects and tasks.The authors note that larger hands may perform poorly on smaller objects such as tweezers.

6 Conclusion

The conclusion presents DexMachina as a curriculum-based functional-retargeting algorithm and Figure 5 as a full evaluation across six hands on long-horizon tasks.

  • Conclusion: DexMachina uses virtual object controllers so policies can explore task strategies under motion and contact guidance.The curriculum lets the policy gradually take over object control.
  • Conclusion: The algorithm and benchmark are intended to identify desirable dexterous hand capabilities and lower barriers to future research.The conclusion frames the platform as useful for functional hardware comparison.
  • Conclusion: The benchmark evaluates all six hands on long-horizon tasks for comparing dexterous hand functionality.The figure summarizes the full hand evaluation enabled by the benchmark.

7 Limitations

The paper identifies limitations involving simulator-state observations, costly high-quality demonstrations, and estimated physical properties of open-source hand assets.

  • Limitations: The policy uses privileged simulator state, which can be difficult to acquire in the real world.The authors suggest vision-based RL or distillation into visuomotor policies as possible remedies.
  • Limitations: The formulation assumes high-quality human hand-object demonstrations requiring reconstruction, accurate pose tracking, and careful curation.Such data can be expensive to collect, as illustrated by ARCTIC’s motion capture and annotation requirements.
  • Limitations: Simulated hands may fail to capture some real physical properties because the evaluation uses open-source assets with estimated mass, inertia, and collision shapes.This limits how directly simulation results represent hardware behavior.

A Demonstration Data Processing Details

Demonstration processing converts human hand-object sequences into collision-aware joint targets, reference keypoints, and approximated contacts for policy learning. The pipeline also defines clipped joint actions, wrist scaling, and contact representations for articulated objects.

  • ARCTIC Demonstration Selection and Curation: ARCTIC clips provide tracked MANO hand poses and articulated object states, trimmed to fixed-length sequences.Each clip is identified by an object, subject tag, and start/end tuple.
  • Object-aware Retargeting Post-processing: Collision-aware retargeting replays joint targets in simulation with the object fixed, recording achieved joints and dexterous-hand keypoints.This reduces object penetration and produces Q ∈ R^T×J and X ∈ R^T×K×3.
  • Object-aware Retargeting Post-processing: Figure 6 contrasts the improved object-aware retargeting scheme with pure kinematic retargeting.The simulation replay is parallelizable across dexterous hands and timesteps.
  • Hybrid Action Outputs: Policy joint actions are clipped to [−1, 1], with separate scaling factors for wrist translation and rotation actions.Finger, wrist-translation, and wrist-rotation indices determine how actions contribute to controller targets.
  • Contact Approximation: The final retargeting outputs include controller joint targets and contact-derived information for each bimanual environment.Each environment contains two copies of contact information with identical shapes, one for each dexterous hand.
  • Contact Approximation: Contact approximation identifies nearby object vertices, subsamples them when necessary, and assigns contacts to dexterous-hand links.The resulting contact positions and validity information are defined over timesteps, object parts, and hand links.

B.1 RL Training and Evaluation Details

Training uses state-based observations and large-scale parallel physics simulation with PPO, while baseline reimplementations align methods within the same framework. Several implementation choices affect reported baseline performance and curriculum reproduction.

  • RL Training: PPO training uses 12,000 parallel environments, except Dex Hand, which uses 10,000 because of memory constraints.Each run uses one NVIDIA L40S or H100 GPU and five random seeds per demonstration and hand pair.
  • RL Training: Policy observations include object states, joint targets, finger-to-object distances, and normalized hand-object contact forces.These state-based inputs expose both task state and interaction guidance to the policy.
  • Baseline Reimplementation: Baseline methods are faithfully reimplemented in the shared Genesis training framework and RL environments for fair comparison.Genesis provides more stable contact modeling and greater memory efficiency than the original setup.
  • Baseline Reimplementation: > 90% success rate is achieved by the ObjDex reimplementation for all hands on Ketchup-100, versus 41.2% reported originally.On Mixer-170, the reimplementation exceeds 70% for three of four hands, versus 57.6% in the original paper.
  • Baseline Reimplementation: The ObjDex reimplementation directly uses kinematic wrist base actions instead of its original learned high-level wrist planner.The original method combines a high-level wrist planner with a low-level residual-action RL policy.
  • Baseline Reimplementation: ManipTrans curriculum parameters are decayed with an exponential scheduler because the original paper does not disclose exact schedules or parameter ranges.The reproduced gravity range is g_gravity ∈ [0, −9.81], and friction uses μ ∈ [4.0, 1.0].

B.4 Policy Evaluation Setup

Evaluation compares achieved object trajectories with demonstrations across repeated seeds and episodes. The ADD-AUC metric summarizes articulated-object pose and joint-angle tracking while avoiding threshold sensitivity and multiple task-specific errors.

  • Evaluation Protocol: Each method and task uses five random seeds, with the best checkpoint from each seed evaluated over 20 episodes.Best checkpoints are selected by cumulative task reward.
  • Evaluation Protocol: Evaluation records object pose and revolute joint angles, then compares achieved object states against the demonstration trajectory.This directly measures the functional retargeting objective over the full sequence.
  • Performance Metrics: ADD-AUC computes average distance separately for each articulated object part before averaging and integrating over the curve.The metric balances pose and joint-angle tracking over long time sequences.
  • Performance Metrics: ADD-AUC avoids threshold sensitivity from per-step success rates and the unwieldy presentation of three separate tracking errors.Per-step success requires thresholds for position, rotation, and joint-angle errors, which depend on object size and geometry.

C Additional Experiment Results

Additional experiments visualize long-horizon policy rollouts and compare action-output formulations. Hybrid actions with tighter wrist bounds perform best in the reported no-curriculum ablation.

  • Long-Horizon Task Visualizations: Figure 7 visualizes long-horizon manipulation tasks achieved by the trained RL policy.The brown platform matches the ARCTIC data-collection setup of objects on a cardboard box atop a table.
  • Hand Action Ablation: Figure 8 compares absolute, full residual, and hybrid action formulations on a subset of hands and objects without curriculum.Results average three random seeds for each method.
  • Hand Action Ablation: Hybrid actions with more restrictive wrist bounds achieve the best learning performance under task-only and task-plus-auxiliary rewards.All formulations benefit from auxiliary rewards, but tighter wrist constraints remain strongest overall.
Loading 2505.24853v1…