Source-linked AI summary
FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting
Kyungmin Lee, Sibeen Kim, Dongyoon Hwang, Yoonsang Oh, Donghu Kim, Youngdo Lee, I Made Aswin Nahrendra, Jaegul Choo, Hojoon Lee
TL;DR
Human hand-object demonstrations are a reusable source of manipulation data, but transferring them to robot hands requires physically feasible retargeting, and existing approaches have limits in success or motion-specific training efficiency. FlashDexRetarget jointly trains a multi-reference dexterous tracking policy using geometry- and interaction-aware observations and rewards, separate hand networks, and off-policy learning. On the 50-motion benchmark, it retargets 90% of motions with about 30 GPU-hours, compared with about 46% success at about 3,000 GPU-hours for CHORD.
Problem
Existing physics-based retargeting methods have limitations in success or motion-specific training efficiency, despite human hand-object demonstrations being a potential source of robot data.
Source: §I · p. 1Method
FlashDexRetarget jointly trains a reference-conditioned policy across demonstrations using interaction-aware observations and rewards, separate hand actor-critic networks, and off-policy learning.
Source: §I · p. 2Results
90% of the 50-motion benchmark was retargeted with about 30 GPU-hours, versus about 46% success at about 3,000 GPU-hours for CHORD.
Source: §I · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Human hand-object demonstrations offer a reusable source of dexterous robot manipulation data, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based approaches face limitations in retargeting success, motion-specific training efficiency, or both. To address these limitations, we introduce FlashDexRetarget, an RL-based framework for high-success, efficient dexterous motion retargeting. To make the demonstrated interaction easier to learn, we combine object point-cloud observations, hand-object distance features, and future trajectory encodings with complementary rewards that supervise object motion and reference hand-object relationships. To further accelerate learning, we employ separate left- and right-hand actor critic networks and adapt the off-policy algorithm, FlashSAC to dexterous motion tracking. On a benchmark of 50 motions spanning single-object and two-object interactions, FlashDexRetarget achieves a 90% success rate, approximately 2.5x that of the evaluated sampling-based baselines, while requiring up to 100x less training compute than the evaluated RL-based baselines. Evaluations on both XHand and Sharpa Wave Hand show consistent gains, and component-wise ablations examine the contributions of our design choices. Beyond the 50-motion benchmark, experiments with 200, 500, and 1,000 motions demonstrate that our method remains stable at larger scales and produces successful retargeted motions more efficiently as the training set grows. Qualitative replay results using real-world-captured demonstrations further illustrate the applicability of our framework to recorded human manipulation. Videos and code are available at https://davian-robotics.github.io/FlashDexRetarget/
I. INTRODUCTION
Physics-based retargeting could turn human demonstrations into robot data, but existing methods retarget demonstrations independently, limiting dataset-scale efficiency. FlashDexRetarget jointly tracks multiple references with one policy and achieves high success with substantially lower compute than evaluated baselines.
I. INTRODUCTION
- Existing sampling-based and RL-based methods retarget each demonstration independently, so computational cost grows linearly with the number of demonstrations.Sampling-based methods search action sequences through simulation, while RL-based methods train a separate closed-loop policy for each reference.Source: §I · p. 1
- Multi-reference tracking trains one reference-conditioned policy jointly across demonstrations, potentially amortizing training cost while retaining accurate tracking.Dexterous manipulation adds varying interaction geometries, imperfect reconstructed contacts, a broader training distribution, and asymmetric bimanual signals.Source: §I · p. 2
- FlashDexRetarget combines geometry and future-reference inputs, distance-based interaction rewards, FlashSAC, and separate left- and right-hand actor-critic networks.These choices address varied interactions, imperfect contact reconstruction, experience reuse, and different hand roles and learning signals.Source: §I · p. 2
- 90% of the 50-motion benchmark was retargeted with about 30 GPU-hours, versus about 46% success at about 3,000 GPU-hours for CHORD.The comparison corresponds to approximately 100× lower training compute and a 44-percentage-point improvement in retargeting success.Source: §I · p. 2
- The evaluation covers 50 motions and two robot embodiments, with ablations, scaling experiments, and real-world replay also reported.The benchmark draws motions from TACO, OakInk2, and HOT3D and uses XHand and Sharpa Wave Hand.Source: §I · p. 2
II. RELATED WORK
Human hand-object datasets capture diverse manipulation behaviors, but their motions cannot be executed directly by robot hands. Prior retargeting approaches use geometric adaptation, trajectory search, or per-demonstration policy learning, whereas FlashDexRetarget shares a policy across references.
A. Human Hand-Object Interaction Data
- Human hand-object datasets cover diverse objects and behaviors, but robot hands cannot execute the recorded human motions directly.Kinematic retargeting is commonly used to bridge this embodiment gap.Source: A. Human Hand-Object Interaction Data · p. 2
- Geometric pipelines such as UniDex and VideoManip adapt human trajectories, but impose contact geometrically rather than through robot-object interaction, so demonstrated object motion is not guaranteed.The related-work discussion therefore calls for physics-based retargeting with verified contacts.Source: A. Human Hand-Object Interaction Data · p. 2
B. Dexterous Retargeting from Human Demonstrations
- Sampling-based physics methods search simulated control rollouts; SPIDER and Do as I Do refine demonstrations, but require a separate trajectory search for each reference.Their effectiveness depends on initialization and reference quality.Source: B. Dexterous Retargeting from Human… · p. 2
- RL-based methods learn policies from human references and task objectives, but still train or adapt separately for each demonstration; FlashDexRetarget instead jointly learns tracking across demonstrations.Examples include residual correction in ManipTrans, reducing object assistance in DexMachina, and contact-wrench guidance in CHORD.Source: B. Dexterous Retargeting from Human… · p. 2
- FlashDexRetarget’s overview combines object geometry and future motion inputs, shared left- and right-hand actor-critic pairs, and geometric interaction rewards.The caption identifies object surface alignment, reference-conditioned fingertip proximity, hand tracking, and regularization as learning signals.Source: Figure 2 · p. 3
III. METHOD
The method trains a shared reference-conditioned policy to reproduce human hand-object interactions on a target robot. Sharing learning across the reference collection is intended to make large-scale trajectory generation more efficient.
III. METHOD
- The method aims to generate many physically grounded robot trajectories from a collection of human hand-object demonstrations.Each reference contains recorded human hand states and object poses over its trajectory length.Source: §III · p. 3
- Unlike single-reference retargeting, FlashDexRetarget jointly trains one reference-conditioned policy over the collection, sampling references into parallel simulation environments.At each step, the policy receives the current simulated state together with the next reference state.Source: §III · p. 3
- The policy predicts robot actions to track reference object motion and hand-object interaction, with tracking rewards defined from resulting robot and object states.The formulation shares policy learning across demonstrations to support efficient conversion of large reference collections.Source: §III · p. 3
- The system combines interaction-aware observations and future-reference conditioning with object-surface and hand-object-distance rewards, per-hand actor-critic networks, and off-policy learning.These are the method components listed in the overview of the framework.Source: §III · p. 3
A. Observation
FlashDexRetarget represents object geometry and hand-object relationships alongside future reference motion. These inputs provide interaction-specific and temporal context to the shared policy.
A. Observation
- The observation includes the current simulated state and the next reference state, with future reference hand and object states encoded relative to the current wrist frame.The future window provides temporal context beyond the next reference frame.Source: A. Observation · p. 3
- Object point clouds and hand-object distance features represent interaction geometry in a wrist-local frame, giving a hand-centered representation across poses and object motions.The object surface is represented by 128 uniformly sampled points; fingertip and wrist distances and corresponding surface normals describe local relationships.Source: A. Observation · p. 3
- K = 10 future reference steps are mapped by a temporal encoder to a 128-dimensional latent representation.The encoding supplies the policy with temporal context beyond the next reference state.Source: A. Observation · p. 4
B. Reward
The reward combines object tracking, hand-object interaction, hand tracking, and regularization. Object and interaction terms provide supervision for matching the demonstrated motion and hand-object relationship.
B. Reward
- Object tracking uses the three largest rescaled surface-point errors to capture object translation and rotation in one metric.The surface points are rescaled to a common radius, and the mean of the three largest pointwise errors emphasizes significant misalignments.Source: B. Reward · p. 4
- Hand-object interaction uses continuous fingertip-to-object distances rather than binary contact labels because reconstructed contacts can be imprecise.The robot is penalized only when fingertip distance exceeds the corresponding reference distance.Source: B. Reward · p. 4
- An auxiliary hand-tracking reward combines wrist position, wrist orientation, mean fingertip errors, and a behavior-cloning term toward kinematic retargets.Source: B. Reward · p. 4
- Regularization penalizes large actions and failures from the hand or object exceeding predefined distance thresholds, while excluding timeouts and successful clip completion.For bimanual clips, each hand is evaluated relative to its associated object, which may be shared.Source: B. Reward · p. 4
C. Bimanual Architecture
The bimanual architecture assigns a dedicated actor and critic to each hand, giving each a hand-specific learning signal while executing both actions together.
C. Bimanual Architecture
- Each hand has a dedicated actor-critic pair trained with its hand-specific reward, rather than aggregating both hands' rewards into one scalar signal.The authors report that this decomposition improves retargeting success, particularly when the hands manipulate separate objects with distinct interaction objectives.Source: C. Bimanual Architecture · p. 4
- Each actor observes the full bimanual hand-object state but controls only its own hand.Source: C. Bimanual Architecture · p. 4
- The two hand actions execute jointly in simulation, so the controllers remain physically coupled despite receiving separate learning signals.Source: C. Bimanual Architecture · p. 4
D. Off-policy Learning
FlashDexRetarget adapts off-policy FlashSAC for multi-reference training by enlarging replay and critic capacity. Qualitative simulation snapshots show retargeted motions across varied object types and configurations.
D. Off-policy Learning
- FlashSAC reuses transitions across references through a replay buffer, and its capacity is increased to accommodate the broader multi-reference state distribution.The replay buffer grows from 10M to 50M transitions, while critic hidden dimension grows from 256 to 1024.Source: D. Off-policy Learning · p. 4
- The off-policy learner is presented as a way to improve data efficiency in multi-reference training.Source: D. Off-policy Learning · p. 4
- The qualitative simulation examples span tool use, container handling, and everyday-object manipulation.They illustrate varied object geometries and hand-object configurations.Source: Figure 3 · p. 5
IV. EXPERIMENTS
The experiments compare retargeting success and compute, examine design components, test scaling to larger motion collections, and assess real-world execution. The setup evaluates a 50-motion benchmark across two target embodiments and specifies distinct metrics and compute accounting for single- and multi-reference methods.
A. Experimental Setup
- The 50-motion benchmark contains 25 single-object and 25 two-object interactions and evaluates both XHand and Sharpa Wave Hand.The demonstrations come from TACO, OakInk2, and HOT3D; inactive clips with stationary objects and no hand manipulation are excluded.Source: A. Experimental Setup · p. 5
- The protocol compares FlashDexRetarget with Do as I Do, DexMachina, and CHORD, using released implementations for Do as I Do and CHORD.DexMachina uses random state initialization along the reference trajectory rather than always starting at the motion beginning.Source: A. Experimental Setup · p. 5
- The study reports SRMT-Obj, SRMT, and SRSPIDER, using SRSPIDER for the main compute-success comparison and SRMT-Obj for component ablations.The authors caution that SRSPIDER's averaging across objects can overestimate success when one reference object is nearly stationary.Source: A. Experimental Setup · p. 5
- Single-reference compute sums GPU-hours across attempted motions, including unsuccessful runs, while multi-reference training uses a fixed 300-million-step budget.Multi-reference success is counted cumulatively when a reference has a stored rollout satisfying SRMT-Obj.Source: A. Experimental Setup · p. 5
- The ablation figure tracks cumulative successful references over 300 million environment steps and varies future encoding, observation-reward components, actor-critic structure, and learner settings.Source: Figure 4 · p. 6
B. Retargeting Performance and Computational Efficiency
FlashDexRetarget outperforms prior methods on the 50-motion benchmark across two embodiments. Ablations favor future conditioning, geometry-aware interaction supervision, separate hand networks, and expanded FlashSAC capacity.
B. Retargeting Performance and Computational Efficiency
- 86% SRMT success on XHand versus 12% for DexMachina and 0% for Do as I Do and CHORD; Sharpa reaches 70% versus 10% for Do as I Do and 0% for CHORD.Under SRSPIDER on XHand, success is at least 40 percentage points higher than the baselines, with approximately 2× less compute than Do as I Do and nearly 100× less than CHORD.Source: B. Retargeting Performance and… · p. 6
C. Key Components for Multi-Reference Retargeting
- Future reference conditioning accumulates successful trajectories faster and converts more references, improving hand-pose and object-rotation tracking.The comparison removes the future trajectory encoder while retaining the next-step reference input.Source: C. Key Components for Multi-Reference… · p. 6
- Retaining either point-cloud tracking or interaction supervision improves performance; using both yields the fastest success accumulation and most converted references.Removing both components causes the largest degradation in learning speed and cumulative retargeting success.Source: C. Key Components for Multi-Reference… · pp. 6–7
- The robot successfully executes retargeted wiping, pouring, and lid-closing tasks in the real world.Source: Figure 5 · p. 7
- Separate left- and right-hand actor-critic networks improve learning speed and cumulative retargeting success over a unified network.Source: C. Key Components for Multi-Reference… · p. 7
- At 300 million environment steps, default FlashSAC converts approximately 34% of references while PPO achieves near-zero cumulative success.Increasing replay-buffer capacity or critic width improves success, and combining both yields the best performance among evaluated configurations.Source: C. Key Components for Multi-Reference… · pp. 6–7
- Larger reference collections yield more successfully converted motions within the same environment-interaction budget.The figure covers collections of 200, 500, and 1,000 reference motions; a conversion counts once a rollout satisfying SRMT-Obj is collected and stored.Source: Figure 6 · p. 7
D. Scaling to Larger Reference Collections
Scaling experiments test shared-policy retargeting on collections of 200, 500, and 1,000 motions. Larger collections yield more successful conversions within a fixed training-step budget.
D. Scaling to Larger Reference Collections
- Larger collections yield more successfully converted references under the same training-step budget.Training budgets increase from 300 million to 600 million environment steps for these experiments; within each collection size, success is counted under SRMT-Obj.Source: D. Scaling to Larger Reference… · p. 7
- Beyond the 50-motion setting, tests use collections of 200, 500, and 1,000 motions, increase training from 300 million to 600 million environment steps, and find larger collections yield more successful conversions under SRMT-Obj.Source: D. Scaling to Larger Reference… · p. 7
- The authors report that multi-reference tracking generates data more efficiently as the number of reference motions grows, whereas single-reference cost grows linearly with demonstrations.Source: D. Scaling to Larger Reference… · pp. 7–8
E. Real-World Validation
Physical-robot replays show successful execution of retargeted manipulation trajectories. The authors note that clean trajectories, accurate object geometry, and privileged simulator information may constrain robustness and deployment.
E. Real-World Validation
- The robot completes wiping, pouring, and lid-closing tasks, spanning sustained surface contact, controlled object tilting, and contact-guided closure.The authors treat these executions as evidence that retargeted motions preserve task-relevant hand-object interactions beyond simulation.Source: E. Real-World Validation · p. 8
V. CONCLUSION
- The framework combines geometry- and interaction-aware observations, future conditioning, hand-object rewards, per-hand actor-critic networks, and off-policy learning with increased capacity.The conclusion reports higher success at substantially lower computational cost on XHand and Sharpa Wave Hand, with scaling experiments yielding more successful trajectories under a fixed interaction budget.Source: §V · p. 8
- Reliance on relatively clean human trajectories, accurate object geometry, and privileged simulator information may limit robustness and real-world deployment.Source: §V · p. 8