PaperlayerView exact revision on arXiv

Source-linked AI summary

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Tara Sadjadpour, Siming He, C. K. Wolfe, Haozhi Qi, Lea Wilken, S. Shankar Sastry, Claire Tomlin, Jitendra Malik

arXiv:2609.28660v1cs.RO

TL;DR

Human hand-object demonstrations offer useful data for dexterous manipulation, but transferring them requires addressing hand-morphology mismatch, dynamical feasibility, and sim-to-real deployment. Morphometric Imitation combines contact-aware kinematic retargeting, residual RL, and visuomotor policy distillation; it reports gains across retargeting stages and 89.3% zero-shot success in real-world trials.

  • Problem

    Transferring human hand-object interactions to robots must address morphology-related contact discrepancies, physically feasible and safe motion, and the sim-to-real gap in visuomotor deployment.

    Source: §I · p. 2

  • Method

    Morphometric Imitation uses MMO for morphology- and contact-aware kinematic retargeting, residual RL with object pose and contact information for dynamic retargeting, and policy distillation for deployment.

    Source: §VIII · p. 11

  • Results

    Across three robot hands and ten trajectories, MMO improves contact F1 by at least 8 points over the strongest baseline and downstream task success by as much as 35 points; policies achieve 89.3% zero-shot success in 300 real-world trials.

    Source: §I · p. 2

  • Takeaways & Limitations

    Object pose guides trajectory tracking and task completion, contact information promotes preservation of demonstrated interactions, and combining them yields the strongest dynamic-retargeting performance.

    Source: §VIII · p. 11

    Separate residual RL and visuomotor policies are trained for each object category, and hardware deployment is evaluated on a single arm-hand system.

    Source: §VIII · p. 11

Abstract

from arXiv · show

Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand, while also improving the success rate of downstream dynamic retargeting by as much as 35 points. Ablations on the residual RL show complementary benefits from using object pose and contact information. Finally, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects. Project page: $\href{https://morphometricimitation.github.io}{\text{this https URL}}$

I. INTRODUCTION

Morphometric Imitation addresses morphology mismatch, physical feasibility, and sim-to-real transfer by turning reconstructed human hand-object interactions into robot visuomotor policies. The reported results span retargeting and hardware deployment.

I. INTRODUCTION

  • MMO aligns hand morphology and recovers demonstrated contacts; residual RL refines its reference using object pose and contact information before trajectories train visuomotor policies.The residual policy adapts the kinematic reference to produce dynamically feasible trajectories with stable contact dynamics and reliable collision avoidance.Source: §I · p. 2
  • The framework addresses morphology- and size-related contact discrepancies, the need for physically feasible and safe robot motion, and the sim-to-real gap in visuomotor deployment.Some prior approaches address these challenges using constrained demonstrations or real teleoperated data.Source: §I · p. 2
  • The authors propose that morphology- and contact-aware retargeting improves both kinematic and dynamic retargeting, yielding physically feasible and safe demonstrations.The framework also explicitly addresses robot-table collisions during transfer of natural tabletop motions.Source: §I · p. 2
  • MMO improves contact F1 by at least 8 points over the strongest baseline, downstream task success by as much as 35 points over the strongest baseline, and achieves 89.3% zero-shot success in 300 real-world trials.The trials cover 30 objects.Source: §I · p. 2

A. Kinematic Retargeting

Prior kinematic retargeting approaches use annotations, paired data, wearables, or hand constraints; newer methods incorporate contact or morphology. This work compares retargeting directly and also evaluates downstream performance.

A. Kinematic Retargeting

  • Some prior methods rely on artist-annotated contact correspondences or paired human-robot data –, while others use wearables; this work focuses on unsupervised methods without wearables [22]–.Source: A. Kinematic Retargeting · pp. 2–3
  • Dex-Retargeting [4] aligns scaled human fingertip vectors with robot fingertips, while DexPilot adds inter-finger constraints; the Dex-Retargeting codebase also provides an offline position-based formulation.PyRoki adds fingertip-object proximity and link-scale optimization, while OmniRetarget uses tetrahedral mesh matching for contact-aware locomanipulation.Source: A. Kinematic Retargeting · p. 3
  • Unlike those formulations, this method optimizes the human hand model to match robot morphology, then recovers hand-object contacts altered by that transformation.Source: A. Kinematic Retargeting · p. 3
  • The evaluation combines kinematic metrics with downstream performance from the same RL formulation trained on each method’s trajectories, following OmniRetarget’s direct-retargeting evaluation.Source: A. Kinematic Retargeting · p. 3
  • Concurrent methods build on OmniRetarget: REGRIND [8] reports no improvement over it, while TopoRetarget [7] reports improvement but had not released code.Source: A. Kinematic Retargeting · p. 3

B. Dynamic Retargeting

Dynamic retargeting has used physics simulation with RL, optimization, and hybrid approaches, but methods vary in their demonstration sources and how they use kinematic references.

B. Dynamic Retargeting

  • Dynamic retargeting methods use physics simulation with RL [8], [10], [12]–[15], optimization [9], or both [11], and vary in whether they rely on natural motion or robot data.Some use constrained human demonstrations intended to facilitate robot transfer [12], [16], whereas this method uses unconstrained natural human motion.Source: B. Dynamic Retargeting · p. 3
  • Methods use kinematic references differently: Chen et al. [10] refine retargeted wrist motion with object-tracking reward, while Human2Sim2Robot [12] initializes a pre-grasp and trains with object-tracking reward.Other examples include MPC warm-starting [9], tracking hand keypoints or joints [13],, and sim-in-the-loop optimization followed by teleoperated-data fine-tuning [11].Source: B. Dynamic Retargeting · p. 3
  • REGRIND [8] encourages object tracking, while DemoMimic [15] encourages object tracking and contact alignment; DemoMimic targets articulated box trajectories rather than rigid object categories.Its code was unavailable at the time of writing.Source: B. Dynamic Retargeting · p. 3

III. MODELING HUMAN HANDS WITH MANO

The paper builds on MANO, a hand model whose template mesh and shape components are used to form rest-pose geometry and joints, then pose the hand.

III. MODELING HUMAN HANDS WITH MANO

  • The skinning function poses the rest-pose mesh about rest-pose joints using per-vertex blend weights; its outputs include posed vertices and joint locations, while translation places the hand in world coordinates.Source: §III · p. 4
  • MANO derives rest-pose mesh and joints from a template hand mesh, shape blend shapes, and a joint regressor.Source: §III · p. 4
  • Pose-corrective offsets depend on deviations of joint rotations from rest pose, avoiding overly smooth deformations and joint collapse associated with standard linear blend skinning.Source: §III · p. 4
  • MANO has 15 finger joints plus the wrist at the kinematic-chain root, with local finger-joint rotations and wrist global orientation in the pose.Source: §III · p. 4

IV. KINEMATIC RETARGETING: MORPHOMETRIC OPTIMIZATION

Morphometric Optimization (MMO) converts human hand motion into robot hand motion through morphology matching, contact recovery, and robot-pose recovery.

IV. KINEMATIC RETARGETING: MORPHOMETRIC OPTIMIZATION

  • MMO first matches the MANO model to the robot hand’s morphology, then recovers the original MANO hand’s contacts after morphology changes.It maps the aligned MANO skeleton to the robot skeleton with linear blend retargeting and solves inverse kinematics for the robot joint configuration.Source: §IV · p. 4
  • Contact matching reoptimizes the morphology-aligned hand at each frame to recover contacts from the original MANO hand.Source: §IV · p. 4
  • Robot-pose recovery transfers the aligned MANO skeleton to the robot skeleton with linear blend retargeting, then solves for robot joint configurations with inverse kinematics.Source: §IV · p. 4

A. Morphology Matching

Morphology matching scales MANO to the robot’s palm and finger proportions, then aligns corresponding joints and fingertips to the robot hand.

A. Morphology Matching

  • Scaled MANO independently scales the palm and each finger to represent hands with different proportions.Palm scale is set from wrist-to-MCP distances, while finger scales use MCP-to-fingertip distances.Source: A. Morphology Matching · p. 4
  • Because palm-vertex blend weights are distributed across the wrist and multiple MCP joints, the method scales the hand in two stages.It first scales the whole hand about the wrist, then adjusts each finger’s length about its MCP joint.Source: A. Morphology Matching · p. 4
  • A semantic correspondence mapping pairs MANO and robot joints, using interpolated phantom positions when the robot lacks a corresponding joint.The method then obtains corresponding joint positions and fingertip positions from the scaled MANO model.Source: A. Morphology Matching · p. 4

Contact

Morphometric Imitation recovers human hand-object contacts after morphology changes, transfers the aligned motion to robot joints, and refines it with residual RL. The resulting trajectories serve as demonstrations for visuomotor students evaluated in simulation and on hardware.

B. Contact Matching

  • Contact matching re-optimizes the morphology-aligned hand to recover the reference contacts, including during non-contact approach frames.For those frames, the method tracks the reference hand region that will form the grasp, maintaining a continuous target into contact.Source: B. Contact Matching · p. 5

C. Robot Pose Recovery

  • Linear blend retargeting maps the aligned MANO skeleton to robot joint and fingertip targets, from which inverse kinematics recovers robot joint configurations.It supplies Cartesian targets for every robot joint and fingertip, including the wrist, rather than only sparse fingertip targets.Source: C. Robot Pose Recovery · p. 6

D. Implementation

  • MMO runs at 180 frames per second for single-hand and 150 for bimanual trajectories on an NVIDIA RTX 4090, excluding inverse kinematics.The same hyperparameters are used across hands except for the finger correspondence mapping.Source: D. Implementation · p. 6

V. DYNAMIC RETARGETING: RESIDUAL REINFORCEMENT LEARNING

  • Residual RL adds a PPO-predicted joint residual to the kinematic reference and uses object-pose and contact information to refine the motion.Its reward jointly encourages object-trajectory tracking and contact matching; rollouts also terminate on tracking or contact errors and safety violations.Source: §V · pp. 6–7

VI. ZERO-SHOT SIM-TO-REAL VISUOMOTOR POLICY

  • Residual RL trajectories provide demonstrations for distilling a privileged teacher into a visuomotor student conditioned on point clouds and robot proprioception.Demonstrations use rendered depth-camera point clouds, while student training additionally randomizes simulated point clouds.Source: §V · p. 6, §VI · p. 7

V. DYNAMIC RETARGETING: RESIDUAL REINFORCEMENT LEARNING

  • Domain randomization varies robot and object properties, initial object poses, and the reference starting timestep; random external wrenches are also applied to the object.For complex geometries, sampled center-of-mass locations preserve stability of the object’s initial resting configuration.Source: §V · p. 7

A. Experimental Setup

  • The simulation evaluation uses ten GRAB trajectories across Dex3, Allegro, and Sharpa, whose demonstrations include reaching, grasping, lifting, and some task-specific object motions.Source: A. Experimental Setup · p. 8
  • Hardware evaluation tests one policy per object category on a KUKA iiwa14 with a Sharpa Wave hand, using three physical instances and randomized initial poses.The setup totals 300 trials; the transparent wineglass is made partially observable with ping pong balls in its bowl.Source: A. Experimental Setup · p. 8

B. Evaluation Metrics

The evaluation measures location-aware contact preservation, compares five kinematic baselines, and reports MMO’s contact-preservation results across three robot hands.

B. Evaluation Metrics

  • Location-aware F1 requires agreement with the demonstrated contact in frame, object location, and hand part; failure-adjusted scores assign zero to sequences a method cannot retarget.Contact patch distance measures spatial error continuously, including demonstrated contact locations the robot never reaches.Source: B. Evaluation Metrics · p. 8

D. Kinematic Retargeting Evaluation

  • MMO exceeds the strongest baseline in contact F1 by 8.3, 27.8, and 10.5 percentage points for Allegro, Dex3, and Sharpa, respectively, and has the lowest patch distance for all three hands.Mean patch distances are 9.1, 7.5, and 9.7 mm for MMO, versus 18.7, 18.9, and 10.7 mm for the respective strongest baselines.Source: Table II · p. 9, D. Kinematic Retargeting Evaluation · p. 9

C. Baselines

  • The five kinematic baselines are DexPilot, AnyTeleop, Position Dex-Retargeting, Contact-Aware PyRoki, and OmniRetarget.For downstream dynamic retargeting, three baselines without arm joints are augmented with arm IK to place the wrist at the demonstrated position.Source: C. Baselines · p. 9

D. Kinematic Retargeting Evaluation

  • At the wineglass and bowl end frames, MMO better recovers the reference hand placement and finger arrangement than the baselines, with F1 scores of 38.5% and 43.0%, respectively.The reconstructed human thumb itself penetrates the bowl by 5.8 mm; the paper addresses such residual geometric errors in dynamic retargeting.Source: D. Kinematic Retargeting Evaluation · p. 9, Figure 5 · p. 9

E. Dynamic Retargeting Evaluation

MMO kinematic references improve residual-RL dynamic retargeting across all three hands, while object pose and contact information contribute complementary benefits. Outcomes also vary with robot morphology, and retargeting reduces but does not eliminate those constraints.

Object and Contact Ablation

Cross-Embodiment Analysis

  • Sharpa's five fingers and 22 DoF most closely resemble the human hand; Dex3's thinner fingers aid clearance, while Allegro's larger, thicker fingers hinder human-scale grasps.Dex3 achieves higher SR than Allegro despite having fewer DoF; improved retargeting reduces, but does not eliminate, morphology constraints.Source: E. Dynamic Retargeting Evaluation · p. 10

F. Sim-to-Real Visuomotor Policy Evaluation

The distilled student closely matches its teacher in simulation and transfers to hardware without real-world training; the conclusion links performance to morphology- and contact-aware retargeting.

F. Sim-to-Real Visuomotor Policy Evaluation

  • 93.8% macro-average success for the student versus 91.8% for the teacher on unseen simulation rollouts, with the student matching or exceeding the teacher in seven of ten categories.The largest student drops were for the cube (92.1% →76.6%) and wineglass (92.2% →85.9%).Source: F. Sim-to-Real Visuomotor Policy… · p. 10
  • 89.3% zero-shot success across 300 hardware trials on 30 objects, with every category at least 80% and the lightbulb succeeding in all 30 trials.The partially observed wineglass achieved 93.3% success; ping pong balls were placed in its bowl because the transparent glass is invisible to LiDAR.Source: F. Sim-to-Real Visuomotor Policy… · pp. 10–11, Figure 6 · p. 11, Table V · p. 11
  • Hardware failures came mainly from position-dependent localization errors and inadequate pre-grasps, while tape-like torus instances were out of distribution for the scale randomization.No failures were due to hard table collisions; the thickest red alarm-clock instance triggered the inadequate-pre-grasp failure.Source: F. Sim-to-Real Visuomotor Policy… · p. 11

VIII. CONCLUSION

  • Across three-, four-, and five-fingered hands, MMO achieved the highest contact F1 and lowest patch distance, while combining object pose and contact cues yielded the strongest dynamic-retargeting performance.The conclusion also reports that the hand ranking follows the embodiment gap to the human hand, while MMO improves over the strongest baseline within each hand.Source: §VIII · p. 11

A. Trajectory alignment and geometry

  • The geometry evaluator aligns patches in object coordinates and uses discretized surface queries, not exact continuous collision certificates.Robot hand surfaces are sampled at nominal 2 mm spacing; signed-query interpretation depends on mesh orientation and closure, including for the benchmark's open bowl mesh.Source: A. Trajectory alignment and geometry · p. 13

B. Location-unaware contact; C. Location-aware contact

  • Location-aware contact requires agreement in frame, mapped hand part, and object vertex, so robot-only patches count as false positives even when location-unaware contact matches.The primary location-aware tables use matched 1, 5, and 10 mm thresholds; these are operational geometric definitions, not estimates of annotation accuracy or sensor noise.Source: C. Location-aware contact · p. 14, E. Choice of contact tolerance · p. 15

D. Precision, recall, F1, and failure handling

  • Failure-adjusted contact scores assign zero to confirmed failed retargetings, and reported F1 averages per-trajectory F1 rather than the harmonic mean of displayed macro precision and recall.The scores include failure zeros, and their standard deviations reflect variation between demonstrations, not training-seed uncertainty or confidence intervals.Source: D. Precision, recall, F1, and failure… · p. 14

E. Choice of contact tolerance; F. Continuous error and coverage; G. Penetration

A. Additional quantitative results

Additional analyses report MMO’s consistent contact-F1 ranking, quantify dynamic-retargeting geometry and penetration, and describe qualitative and paired-difference checks.

A. Additional quantitative results

  • Alternative aggregations leave the F1 comparison unchanged; successful-run-only 5 mm location-aware F1 rises from 12.8% to 14.3% on Allegro and from 7.6% to 9.6% on Dex3 for OmniRetarget.MMO also has the highest pooled 5 mm location-aware F1 on all three hands: 28.8% for Allegro, 42.3% for Dex3, and 39.4% for Sharpa.Source: A. Additional quantitative results · pp. 15–16
  • MMO has the highest macro location-aware F1 at every tolerance on every hand.Table VI reports failure-adjusted macro percentages over ten attempts per hand and method.Source: A. Additional quantitative results · p. 16, Table VI · p. 16

B. Qualitative comparison with all baselines

C. Threshold and paired-difference diagnostics

  • Paired F1 differences at 5 mm use 20,000 paired demonstration resamples and pointwise 95% percentile intervals, not policy-seed variation.The intervals characterize the selected benchmark and are not population-wide guarantees.Source: C. Threshold and paired-difference… · p. 16

A. Contact metrics of the dynamically retargeted grasps

  • Table III scores each rollout’s achieved final state, not its tracking target, using location-aware area-weighted mapped-finger contact and distance to the human contact patch.Every eighth rollout is scored for contact metrics; success rates use all rollouts.Source: A. Contact metrics of the dynamically… · p. 16
  • For every method, rollouts penetrating deeper than 5 mm average at most 0.4% over categories.Table IX reports contact metrics for achieved final grasps, with successful-rollout-only metrics also shown.Source: A. Contact metrics of the dynamically… · p. 17
  • On 107 cells across seven categories, changing from the visual mesh to the convex collision surface shifts successful-rollout F1 by -3.1 to +4.6 points, while patch-distance rankings remain identical.F1 rankings change only between methods less than 0.4 points apart; Table VII reports geometry on matched sequences with valid outputs from every method.Source: A. Contact metrics of the dynamically… · p. 17, Table VII · p. 17

APPENDIX D REAL-WORLD EXPERIMENTS

The hardware evaluation reports 89.33% success across 300 trials on 30 physical objects, while detailing partial-depth observations and location-dependent failures.

APPENDIX D REAL-WORLD EXPERIMENTS

  • 89.33% (268/300) is the hardware macro mean across 30 trials per category on three distinct physical objects per category.Each object was tested at ten poses randomized within a 10 cm × 10 cm region and ±15° in yaw.Source: Appendix D · p. 18
  • Despite incomplete point clouds, the policy succeeds on the wineglass in 28 of 30 trials and the small flashlight in 7 of 10.The wineglass glass returns no depth; ping-pong balls in its bowl are the only observed part.Source: Appendix D · p. 18, Figure 9 · p. 19
  • Of 270 on-grid trials, 241 succeed; 22 of the 29 on-grid failures occur in the near row, versus five in the far row and two in the mid row.The on-grid poses span the 10 × 10 cm region; the tenth pose per instance is off-grid and omitted from Figure 10.Source: Appendix D · p. 18, Figure 10 · p. 20
  • Tables X and XI list each physical instance’s photo, mass, dimensions, and pick-up success rate alongside its category’s simulation-mesh dimensions.The tables cover the first five and remaining five object categories, respectively.Source: Table X · p. 23, Table XI · p. 24
Loading 2609.28660v1…