Source-linked AI summary
OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport
Guillaume Besset, Erwann Carn, Timothée Carecchio, Valentin Tordjman-Levavasseur, Fabian Schramm, Yann de Mont-Marin, Justin Carpentier, Ajay Suresha Sathya
TL;DR
Contact-rich motion retargeting must adapt robot and object trajectories while preserving interactions that skeletal motion alone does not fully describe. OTRETARGET transfers surface interaction targets with optimal transport and jointly optimizes robot and object poses. It improves motion-style tracking and interaction fidelity across two datasets, and policies trained on its references transfer to a physical G1.
Problem
Contact-rich retargeting must preserve ground and manipulated-object interactions across embodiments, while skeletal motion alone incompletely describes surface contact and fixed object trajectories constrain adaptation.
Source: §I · p. 1Method
OTRETARGET transfers surface interaction targets with optimal transport and balances them against motion-style targets while jointly optimizing robot and object poses frame by frame.
Source: §I · p. 2, B. Surface-level proximity and optimal… · p. 3, C. Interaction residuals · p. 4Results
Across two datasets comprising 6 448 clips, OTRETARGET improves motion-style tracking and interaction fidelity over existing methods and supports transfer to unseen object shapes and sizes.
Source: §V · p. 8Takeaways & Limitations
Whole-body policies trained on retargeted references transfer to a physical G1, including for two-handed box pick-and-place.
Source: §V · p. 8, E. Policy training and hardware transfer · p. 8The evaluation is limited to one robot, flat ground, and a single two-object capture.
Source: §V · p. 8
Abstract
from arXiv · showhide
Transferring human motion to humanoid robots requires adapting the demonstrated motion to the robot morphology while preserving interactions with the environment. This is particularly challenging for loco-manipulation tasks, where contacts with the ground and manipulated objects must remain consistent despite differences in body proportions. Yet, skeletal motion alone does not fully describe these interactions, and fixing object trajectories limits the adaptation to a new embodiment. In this paper, we introduce OTR ETARGET, a unified approach to jointly retarget robot and multi-object motion from human demonstrations. Our approach represents surface interactions through signed distances, closest surface points, and relative directions, and uses entropic optimal transport to transfer these quantities across human, robot, and object geometries. We incorporate the resulting interaction targets into a constrained inverse kinematics formulation that balances contact preservation with motion style and jointly optimizes robot and object poses at each frame. This formulation accommodates robot-object and object-object interactions without rescaling the scene or the demonstration. We validate the proposed approach on OMOMO, where it achieves a robot- object interaction Jaccard score of 87% and a depth error of 8.7 mm, compared with 28% and 29.3 mm for OmniRetarget. Finally, we demonstrate transfer to a physical G1 humanoid using whole-body policies trained with reinforcement learning on the retargeted references, across motions including two-handed box pick-and-place onto a table.
I. INTRODUCTION
Contact-rich motion retargeting must preserve interactions while adapting robot and object trajectories to a different embodiment. OTRETARGET addresses gaps in skeletal and relational retargeting by transferring surface-level interaction targets and jointly adapting robot and object motion.
I. INTRODUCTION
- Retargeting errors can propagate into learned tracking policies unless reward engineering compensates for them.The pipeline converts human motion into a kinematic reference, which a whole-body policy learns to follow before physical deployment.Source: §I · p. 1
- Contact-rich motions require preserving ground and object interactions while adapting robot and object trajectories to a new embodiment.Trajectory rescaling can misalign floor or table contacts, and fixed object trajectories can be unreachable or incompatible with a changed grasp.Source: §I · p. 1
- Existing methods relying on skeletal motion provide only a partial description of the surface interactions that govern contact.OTRETARGET transfers interaction targets from scene geometry and jointly adapts robot and object motion.Source: §I · pp. 1–2
- OTRETARGET uses signed distances, closest surface points, and relative directions to express interactions, then combines interaction and motion-style targets in constrained inverse kinematics.Joint robot and multi-object optimization preserves robot–object, object–object, and ground interactions without rescaling; transport can also map targets to substitute object geometries.Source: §I · p. 2
II. RELATED WORK
- Point-stream methods optimize pose tracking without modeling ground or manipulated objects, while relational methods generally keep object poses fixed.OmniRetarget solves for the robot and globally rescales the demonstration; fixed object references leave scaling inconsistency and unadapted object trajectories unresolved.Source: §II · pp. 2–3
III. METHOD
The method converts demonstrated geometry and motion trajectories into interaction and style targets, then solves a constrained inverse-kinematics problem frame by frame. Robot configuration and all manipulated-object poses are optimized together.
III. METHOD
- Dense samples on human, object, robot, and target-object surfaces provide geometry for defining desired interactions in the target scene.The method uses deviations from both skeletal and object pose trajectories and target interactions as inverse-kinematics costs.Source: A. Problem setting and formulation · p. 3
- At each frame, constrained inverse kinematics balances interaction deviation against robot and object motion-style deviation while solving for robot and object poses.The formulation treats the robot configuration and every manipulated-object pose as optimization variables.Source: A. Problem setting and formulation · p. 3
- The constraints encode joint and joint-velocity limits, per-object velocity limits, and collision avoidance, including self-collision.Source: A. Problem setting and formulation · p. 3
B. Surface-level proximity and optimal transport
Surface proximity is encoded with a distance, a closest-point witness, and a direction, then optimal transport transfers interaction targets between human and robot geometry. The transport plan is computed once for each human-part and robot-link pair.
B. Surface-level proximity and optimal transport
- Each probe point is expressed in the channel object's frame, and its proximity triple records signed distance, closest surface witness, and the direction from witness to probe.The triple remains meaningful as the channel moves; surface clouds are densely sampled and proximity grids are precomputed offline.Source: B. Surface-level proximity and optimal… · p. 3
- Optimal transport matches human-part and robot-link surface point clouds using uniform marginals and squared Euclidean distance as the cost.The transport plan uses entropic regularization and is solved with Sinkhorn; transferred targets are selected through the resulting correspondence.Source: B. Surface-level proximity and optimal… · p. 3
- 42% of extremal samples remain unmatched with nearest-neighbor assignment, versus 14% with optimal transport.The comparison considers the top distance decile from the point-cloud centroid; the transport plan is precomputed with both bodies in a T-pose.Source: B. Surface-level proximity and optimal… · p. 3
C. Interaction residuals
Interaction residuals compare demonstrated and retargeted proximity to preserve contact geometry and direction. Their weighted cost is conditioned on contact activity and distance, with probe weights normalized by link.
C. Interaction residuals
- Proximity-based residuals compare demonstrated and target interactions using signed distance, surface geodesic distance, and direction.Clamping the demonstrated distance prevents noisy demonstrations from imposing penetration targets; direction residuals resolve ambiguities that distance alone cannot distinguish.Source: C. Interaction residuals · pp. 3–4
- The interaction cost sums weighted quadratic residual losses, with per-link normalization preventing sampled surface area from increasing a link’s authority.Probe weights are (256, 625, 2500) for robot probes and four times those values for object probes; links use per-link weight 1.Source: C. Interaction residuals · p. 4
- Quadratic kernels of width 13 cm attenuate pair contributions beyond that distance; direction and tangential residuals are active only for demonstrated contact.The distance kernel uses demonstrated and current distance to control pairs with an active contact in either scene; penetration can raise it above 1, capped at 5.Source: C. Interaction residuals · p. 4
D. Joint kinematic retargeting solver
The solver combines skeleton-based style costs with interaction-aware native-scale initialization and hard kinematic constraints. A sequential quadratic programming procedure jointly updates robot configuration and manipulated-object poses.
D. Joint kinematic retargeting solver
- Desired robot link orientations are aligned to corresponding human links, and forward kinematics computes target positions using the robot’s own link lengths.Using orientations to derive positions avoids conflicts introduced by scaling the human skeleton; link-length ratios range from 0.19 to 1.18.Source: D. Joint kinematic retargeting solver · p. 4
- Style costs track robot links and object frames, but the object term centers each manipulated object on its demonstrated trajectory and contains no robot-object interaction term.The interaction cost is separate from style tracking.Source: D. Joint kinematic retargeting solver · p. 4
- At native metric scale, a small position weight lets interaction targets determine link positions while retaining posture priors for links not involved in interaction.The robot root and scene remain at captured scale; demonstrated table height is retained despite differences in body scale.Source: D. Joint kinematic retargeting solver · p. 4
- Hard constraints enforce sampled-point non-penetration and robot joint limits, while object feasible sets bound frame-to-frame displacement and deviation from reference motion.Fixed objects can be kept out of the optimization as free pose variables; the sequential SQP solver uses robot and object pose increments as decision variables.Source: D. Joint kinematic retargeting solver · pp. 4–5
E. Generalization to new object shapes
The method transfers interaction targets to substitute object shapes using optimal-transport correspondences. The substitute object’s pose is jointly optimized so relative motions can adapt while demonstrated interactions are preserved.
E. Generalization to new object shapes
- Optimal transport maps interaction targets from the demonstrated object to a substitute shape, extending the approach to new object shapes and sizes from a single demonstration.The transfer is hypothesized to work for objects with similar surface topology when approach motion and contact relationships are preserved.Source: E. Generalization to new object shapes · p. 5
- Surface point clouds are normalized to unit radius only to establish correspondences; mapped witness locations retain metric interaction targets without rescaling.Correspondences are precomputed between point clouds expressed in each object’s local frame.Source: E. Generalization to new object shapes · p. 5
- The substitute object pose is a decision variable in joint retargeting, allowing robot-object and object-object relative motions to adapt to the new geometry.The optimization preserves the demonstrated interactions while adapting those relative motions.Source: E. Generalization to new object shapes · p. 5
IV. EXPERIMENTS
The experiments assess motion and interaction fidelity, runtime, generalization to new objects, and hardware transfer using learned tracking policies.
IV. EXPERIMENTS
- The experiments evaluate motion and interaction fidelity alongside runtime.These are two stated evaluation dimensions.Source: §IV · p. 5
- Object generalization is an evaluation dimension.The passage identifies generalization to objects as part of the evaluation scope.Source: §IV · p. 5
- Hardware transfer is evaluated through learned tracking policies.The passage specifies learned tracking policies as the route for assessing hardware transfer.Source: §IV · p. 5
A. Experimental setup and evaluation protocol
The evaluation covers locomotion, single-object manipulation, and a two-object capture, with each method retaining its scaling convention. Interaction, style, collision, foot-contact, and object-drift metrics are aggregated by dataset median.
A. Experimental setup and evaluation protocol
- Evaluation spans 2,027 body-only AMASS motions, 4,421 OMOMO single-object sequences, and one two-object RGB-D capture.The two-object sequence uses estimated human motion and object poses with primitive meshes at measured dimensions.Source: A. Experimental setup and evaluation… · p. 5
- Methods retain distinct scale conventions, including trajectory scaling for OmniRetarget and the scaled ablation without resizing objects.The G1 is 1.32 m tall; PHC and native-scale variants use sc = 1.Source: A. Experimental setup and evaluation… · p. 5
- Interaction fidelity is measured by precision, recall, Jaccard agreement, depth error, and surface-placement error, with ground and object contacts reported separately.Interactions use a 20 mm threshold; penetrations count as interactions and their magnitude contributes to depth error.Source: A. Experimental setup and evaluation… · p. 5
- Style, self-collision, foot sliding, stance clearance, and object drift provide additional measures; dataset results are medians over returned sequences.OTRETARGET uses shared weights without per-sequence tuning, and its objective does not directly optimize self-collision or foot-contact metrics.Source: A. Experimental setup and evaluation… · p. 5
B. Comparison with existing retargeting methods
OTRETARGET improves robot–object interaction agreement over OmniRetarget and performs best on ground interaction and style measures across both datasets. The comparisons also expose differences in contact behavior and runtime.
B. Comparison with existing retargeting methods
- 87% robot–object Jaccard agreement versus 28% for OmniRetarget, with one third of its depth and placement errors.The scaled, fixed-object ablation retains most of the gain, implicating interaction residuals rather than native scene scale or variable objects.Source: B. Comparison with existing… · p. 6
- OTRETARGET achieves the best style error, floor depth, and ground-interaction agreement on AMASS and OMOMO.The table reports ground fidelity for pooled AMASS locomotion and OMOMO loco-manipulation; on OMOMO, interaction-aware methods retain ground contact while style error increases as the robot adapts its grip.Source: B. Comparison with existing… · p. 6, Table I · p. 6
- On the floor-lamp stem, OmniRetarget loses the interaction, while OTRETARGET's result is unchanged.On the large convex box, OmniRetarget matches demonstrated depth more closely but predicts contact in non-contact frames, lowering precision.Source: B. Comparison with existing… · p. 6
- OTRETARGET runs at 21.5 ms per frame for locomotion and 63.8 ms for object carrying, 12–22× faster than OmniRetarget on those sequences.GMR is faster still, at 3.2 and 2.5 ms, but does not model environment interactions.Source: B. Comparison with existing… · p. 6
C. Retargeting the native scene
In the captured native-scale scene, freeing the object lets the robot adapt object height and recover posture, stance, and grip. The approach also generalizes to multiple objects and variations in object size and shape.
C. Retargeting the native scene
- A variable object moves 12 cm on average to a reachable height in the overhead-lift case, recovering posture, stance, and grip.With the object fixed at native scale, style error doubles and the feet lift; the table compares the captured scene with fixed- and variable-object ablations.Source: C. Retargeting the native scene · pp. 6–7, Table II · p. 6
- In the two-object capture, box-table and box-arm interaction recall and precision are 95% and 98%, with 2.7 cm mean drift and 8.8° style error.The box moves up to 27 cm in flight to remain reachable, then settles at the demonstrated place during demonstrated frames.Source: C. Retargeting the native scene · p. 7
D. Retargeting generalization to object variations
- Across resized boxes and eight unseen shapes, grip recall remains nearly constant and swapped-object depth error stays within 3 mm.Drift increases for shapes less similar to a box; the table evaluates multiple unseen objects while leaving the capture unchanged.Source: D. Retargeting generalization to… · p. 7, Table IV · p. 7
E. Policy training and hardware transfer
The section reports reinforcement-learning transfer of retargeted references, including physical G1 pick-and-place, and closes with broader benchmark claims and evaluation boundaries.
E. Policy training and hardware transfer
- Fig. 5 plots grip recall and precision against box scale for one demonstration replayed across a distribution of object sizes.The figure includes largebox and OMOMO object scales.Source: Figure 5 · p. 7
- Fig. 6 plots MPJPE against PPO iterations for ten AMASS clips trained with identical pipelines, marking OmniRetarget’s final accuracy with a dotted line.The figure compares tracking performance over training.Source: Figure 6 · p. 8
- The physical G1 completed 6 of 8 pick-and-place trials without retraining or object perception; training reached 48% success under a 25 cm object termination.The policy’s box was 11 cm from its reference.Source: E. Policy training and hardware transfer · p. 8
V. CONCLUSION
- Across 6,448 clips in two datasets, OTRETARGET improves motion-style tracking and interaction fidelity over existing methods and transfers to unseen object shapes and sizes.The evaluation remains limited to one robot, flat ground, and a single two-object capture.Source: §V · p. 8
- The formulation is kinematic and frame-by-frame, without explicit force modeling or optimization over a time horizon.The paper identifies accounting for these modalities as a direction for handling more complex terrain and scenarios.Source: §V · p. 8