PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation

Jielin Wu, Shenzhe Yao, Guanqi He, Xiaohan Liu, Zhaoqing Zeng, Xiangrui Jiang, Han Yang, Wentao Zhang, Hang Zhao

arXiv:2606.16272v2cs.RO

TL;DR

Human hand-object demonstrations offer dense references for dexterous-manipulation RL, but retargeting must preserve task-relevant contacts as well as hand pose. TopoRetarget generates interaction-preserving robot references with a fixed parameter setting. It improves contact fidelity and downstream tracking, with reported gains on pen spinning and zero-shot hardware transfer.

  • Problem

    Retargeting human demonstrations for robot policy learning must bridge an embodiment gap while preserving local hand-object interaction, which hand-pose matching alone does not ensure.

    Source: §1 · p. 2

  • Method

    TopoRetarget generates robot references by preserving local hand-object relationships while enforcing hand kinematics and penetration constraints, using one fixed parameter setting across experiments.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    TopoRetarget improves contact precision, contact alignment, and penetration metrics over representative baselines, while reaching 87.5% success on the MoCap Pen-Spin Dataset, over 40 percentage points above all baselines.

    Source: §5.1 · p. 6, §6 · p. 8, §5.2 · p. 8

  • Takeaways & Limitations

    The retargeted references support stronger downstream RL tracking and zero-shot Wuji Hand transfer on cube reorientation and pen spinning.

    Source: §1 · p. 2, §6 · p. 8, §5.2 · p. 8

    TopoRetarget depends on upstream reference quality and is less effective for intended virtual contacts where source fingers do not touch the object surface.

    Source: §7 · p. 8

Abstract

from arXiv · show

Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.

1 Introduction

Human demonstrations can guide dexterous-manipulation RL, but retargeting must preserve task-relevant hand-object interaction, not just hand pose. TopoRetarget addresses this with interaction-preserving references and a tracking pipeline and dataset for dexterous manipulation.

  • TopoRetarget preserves local hand-object interaction while enforcing dexterous-hand kinematic and penetration constraints.It preserves object-relative positional and directional relationships between hand links and object regions, as well as relationships among hand links.Source: §1 · p. 2
  • Its lightweight reference-based RL tracking pipeline supports zero-shot sim-to-real transfer for pen spinning and cube reorientation.Source: §1 · p. 2
  • The paper contributes a dataset of retargeted trajectories, task references, and trained policies for reproducible reference-based dexterous manipulation.Source: §1 · p. 2

2 Related Work

Prior work spans hand-centric retargeting, reference-guided RL, and interaction-preserving data generation. TopoRetarget targets their shared challenge: generating dexterous-hand references that preserve local hand-object interaction while respecting robot constraints.

2.1 Dexterous Hand Motion Retargeting

  • Hand-centric retargeting objectives align poses or fingertips but do not directly specify object contact; contact-transfer methods still face frequent contact-mode transitions.TopoRetarget frames retargeting as interaction-preserving reference generation with robot-kinematic and penetration constraints.Source: §2.1 · pp. 2–3

2.2 Reinforcement Learning for Dexterous Manipulation

  • Reference-guided dexterous-hand RL methods include DexTrack [4], Dexplore [5], DexMachina [6], and ManipTrans [7], but policies depend on being able to follow their references.TopoRetarget targets reference quality by preserving task-relevant local interaction before RL training.Source: §2.2 · p. 3

2.3 Interaction-Preserving Data Generation

  • Human-motion datasets and contact-aware trajectory methods address data generation, but embodiment gaps can still yield contact misalignment, penetration, or infeasible poses.TopoRetarget provides real-time retargeting with local contact fidelity using fixed parameters across dexterous in-hand manipulation settings.Source: §2.3 · p. 3

3 TopoRetarget: Interaction-Preserving Hand-Object Retargeting

TopoRetarget converts human hand-object demonstrations into robot references through bone-direction initialization, a shared interaction mesh, and topology-aware Laplacian refinement. Its optimization preserves local interaction relationships while incorporating smoothness, bone-direction priors, and penetration constraints.

3 TopoRetarget: Interaction-Preserving Hand-Object Retargeting

  • TopoRetarget converts demonstrations into robot references through bone-direction initialization, interaction-mesh construction, and topology-aware Laplacian optimization.The overview identifies the output as a robot motion reference that preserves hand-object interaction.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Notation Definition

  • The retargeting problem takes a human-hand trajectory, object-pose trajectory, and object mesh as inputs and computes robot trajectories.The human-hand keypoints are MediaPipe [28] points, and the formulation includes robot base-pose and configuration variables.Source: §3.1 · p. 3

3.2 Relative Bone-Direction Initialization

  • Initialization estimates robot-hand configurations by matching local finger articulation and maintaining temporal consistency with the previous estimate.The initialization objective balances source-relative bone directions against temporal consistency.Source: §3.2 · p. 4

3.3 Interaction Mesh Construction

  • A shared interaction graph is built from source hand keypoints and object-surface samples, then reused for the robot vertices to compare local hand-object neighborhoods.The graph can be rebuilt after object scaling or replacement, supporting the same Laplacian refinement without new contact targets.Source: §3.3 · p. 4

3.4 Topology-Aware Laplacian Optimization

  • The refinement matches topology-aware Laplacian coordinates on a shared interaction graph, using source-frame distance-aware weights and interaction, bone-direction, smoothness, and signed-distance constraints.The weights are computed on the source configuration and reused for the robot; one fixed parameter setting is used across experiments.Source: §3.4 · p. 5

4 Minimal RL Tracking Controller

The controller trains a policy to track retargeted trajectories in a finite-horizon MDP using PPO. Its reward tracks object pose, hand-link positions, finger joints, and smoothness, with domain randomization during training.

4 Minimal RL Tracking Controller

  • The retargeted-trajectory tracking task is formulated as a finite-horizon MDP and optimized with PPO [29].References are expressed in the robot base frame so the controller focuses on relative hand-object interaction rather than global motion.Source: §4 · p. 5
  • At reset, each environment samples a reference start frame uniformly and initializes the hand and object to the corresponding states.Source: §4 · p. 5
  • The reward combines tracking terms for object pose, hand-link positions, finger joints, and motion smoothness.The term weights are listed in Appendix A.5.Source: §4 · p. 5
  • Training randomizes object mass and center of mass, contact parameters, actuator gains, joint damping, and hand-link inertial properties, and applies external object perturbations.Source: §4 · p. 6

5 Experiments

The experiments evaluate the proposed retargeting pipeline through three questions.

5 Experiments

  • The experiments are organized around three questions evaluating the proposed retargeting pipeline.Source: §5 · p. 6

1. Does TopoRetarget preserve local hand-object interactions across diverse human motions?

The evaluation tests interaction preservation, downstream policy performance, and generalization across hand embodiments and object scales without per-case retuning. TopoRetarget reports lower interaction errors, stronger tracking results, and transfer across these settings.

5.1 Interaction-Preserving Retargeting Quality

  • TopoRetarget achieves the lowest contact precision and alignment errors on ContactPose, at 7.71 mm and 15.67° respectively, with maximum penetration of 1.07 mm.Average solve time is below 5 ms per frame.Source: §5.1 · p. 6, Table 1 · p. 6
  • Figure 3 summarizes contact drift, invalid grasps or penetration, misplaced non-tip contacts, and unnatural joint configurations among retargeting artifacts.TopoRetarget preserves source hand-object interaction while reducing artifacts.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Downstream Tracking Policy Performance

  • 87.5% success on the MoCap Pen-Spin Dataset is over 40 percentage points above all baselines, alongside the lowest position and rotation errors.On Ho-cap, TopoRetarget reaches 84.4% success and 0.87 cm position error, with rotation errors comparable across methods.Source: Table 2 · p. 7, §5.2 · p. 8
  • Figure 4 shows zero-shot Wuji Hand executions for cube reorientation and pen spinning alongside their retargeted reference trajectories.The results demonstrate execution of skills with repeated in-hand contact changes.Source: Figure 4 · p. 7, §5.2 · p. 8

5.3 Generalization and Augmentation without Per-Case Retuning

  • With parameters fixed, TopoRetarget transfers hand motion and hand-object interaction across dexterous hand embodiments and object scales.Figure 5 depicts augmentation across these settings without per-case retuning.Source: §5.3 · p. 8, Figure 5 · p. 8

6 Conclusion

TopoRetarget converts human hand-object demonstrations into dexterous robot references while maintaining local interaction and enforcing robot constraints. The paper reports improved retargeting and downstream tracking, and augmentation across scales and embodiments without task-specific tuning.

  • TopoRetarget maintains object-frame hand-object interaction through distance-weighted Laplacian optimization, directional consistency, kinematic constraints, and penetration handling.Source: §6 · p. 8
  • The framework achieves better contact precision, contact alignment, and penetration metrics than representative baselines on ContactPose.Source: §6 · p. 8
  • Its references support higher downstream RL success rates and lower object-tracking errors, especially on agile pen-spinning demonstrations with frequent non-tip contact transitions.Source: §6 · p. 8
  • The formulation supports augmentation across object scales and hand embodiments without task-specific tuning.Source: §6 · p. 8

7 Limitations and Future Work

TopoRetarget depends on upstream human reference quality and is less effective when intended finger-object contacts do not reach the object surface. The authors identify source-motion preprocessing as future work for these missing-contact cases.

  • TopoRetarget depends on the quality of upstream human reference motion.Source: §7 · p. 8
  • The method handles contact distortion from source-motion penetration but is less effective for virtual contacts where the source finger does not touch the object surface.Source: §7 · p. 8
  • Future work can preprocess source motion to correct missing-contact cases before retargeting.Source: §7 · p. 8

A.1 Retargeting Details

The appendix describes the final-refinement regularization term and selected parameter settings. Its regularization includes temporal smoothness and a lightweight base-pose prior for floating-base retargeting.

  • The final-refinement regularization term is expanded in the implementation details.Source: §A.1 · p. 12
  • The regularization includes temporal smoothness in optimizer coordinates and a lightweight base-pose prior for floating-base retargeting.Source: §A.1 · p. 12
  • The base pose notation distinguishes its translational and rotational components.Source: §A.1 · p. 12

A.2 OmniRetarget Baseline

The appendix adapts OmniRetarget for dexterous hands and defines the reference metrics, evaluation setup, and RL training pipeline. The policy tracks retargeted references using residual actions, structured observations, reference-based resets, and randomized training conditions.

A.2 OmniRetarget Baseline

  • OmniRetarget’s interaction-mesh objective is retained, while wrist motion is optimized jointly and collision handling covers the full hand geometry.This accommodates contacts from fingertips, intermediate phalanges, finger sides, or the palm.Source: §A.2 · p. 12

A.3 Retargeting Metrics

  • ContactPose evaluation marks links as in contact using annotated vertices, assigns them to the nearest of 20 bones, and requires at least 10 vertices per link.Penetration metrics report maximum depth and the frame fraction exceeding 2 mm; contact alignment measures angular deviation between source and retargeted in-contact bone segments.Source: §A.3 · pp. 12–13

A.4 Experimental Setup

  • Evaluation uses 25 ContactPose grasps, 32 Ho-cap references, and 32 MoCap pen-spinning demonstrations, with trained policies also deployed on Wuji Hand hardware.Three ContactPose grasps with deeply concave geometry are excluded; the pen-spinning trajectories average 12.4 s and include frequent non-tip contacts and contact-mode transitions.Source: §A.4 · p. 13
  • The comparison includes OmniRetarget, Mink, DexPilot, and GeoRT, with reference-level interaction and penetration metrics plus policy success and object-pose tracking error.Methods use the same human references and object geometry when applicable.Source: §A.4 · p. 13

A.5 RL Training

  • The RL task is a finite-horizon MDP that tracks a manipulation clip, with reference quantities expressed in the base frame to emphasize relative hand-object interaction.The reference includes finger joints, object pose, and tracked hand-link positions.Source: §A.5 · p. 13

A.5.1 State and action

  • The policy receives proprioception, object pose encoded by attached axis points, and current plus lookahead reference features; residual joint-position actions let it correct tracking errors around the reference pose.Lookahead offsets are 1, 3, and 5 control steps, and training adds small noise to proprioception and observed object pose.Source: §A.5.1 · pp. 13–14, §A.5.2 · p. 14

A.5.3 Reference initialization and sampling

  • At each reset, a reference frame is sampled uniformly and the simulator resets to its hand and object poses, exposing the policy to all trajectory phases.Small object-pose reset noise is optional.Source: §A.5.3 · p. 14

A.5.4 Reward

  • The reward combines object, hand-link, joint, and smoothness tracking, while training randomizes physical and actuator properties and applies external object disturbances.Object tracking receives the largest reward weight; PPO and network settings are summarized in Table 6.Source: §A.5.4 · p. 14
Loading 2606.16272v2…