Source-linked AI summary
Unified Motion Retargeting for Humanoids with Learned Point Cloud Correspondence
Hanyang Cao, Yuetong Fang, Taesoo Kwon, Runyi Yu, Ji Ma, Jing Tan, Yangchen Zhou, Baoze Du, Yi Gu, Yukang Gao, Ruoli Dai, Lei Han, Renjing Xu
TL;DR
Humanoid retargeting must convert abundant human motion into robot trajectories despite morphological and kinematic differences, but existing methods depend on sparse manual correspondences. UMR learns dense source-robot point-cloud mappings and reuses them for constrained surface alignment and contact transfer, with reported gains in downstream tracking and interaction quality. The framework is currently bounded by its need for mesh-based source geometry and a canonical source template.
Problem
Existing retargeting relies on manually designed sparse skeletal correspondences, limiting fine-grained pose and contact guidance across motion sources and robot morphologies.
Method
UMR learns dense correspondence between canonical source and robot surface point clouds, then uses paired geometric anchors for constrained motion optimization and contact transfer.
Results
UMR improves downstream tracking over GMR, remains competitive with curated Unitree references, and reduces interaction-policy joint error by approximately 40% to 56% versus OmniRetarget.
Takeaways & Limitations
UMR provides a common surface-based interface across heterogeneous motion representations, robot embodiments, and downstream tasks spanning locomotion and interaction.
Takeaways & Limitations
UMR assumes access to mesh-based source geometry and a canonical source template.
Abstract
from arXiv · showhide
Humanoid learning increasingly relies on transforming vast and diverse human motion data into high-quality robot reference trajectories. However, retargeting human motion to humanoid robots is challenging due to substantial differences in morphology, degrees of freedom, joint ranges, and kinematic constraints between humans and robots. Existing retargeting methods typically address these differences by defining human-robot correspondence through hand-crafted sparse keypoints or body-part pairs. As a result, retargeting quality depends heavily on manual semantic design, limiting scalability across motion sources and robot morphologies and providing only sparse guidance for reproducing detailed poses and interactions. In this paper, we present Unified Motion Retargeting (UMR), a framework that learns dense point cloud correspondence without requiring manually designed human-robot mappings. By treating exterior point clouds as a unified interface between human motion and humanoid robots, UMR decouples retargeting from source-specific skeletal semantics and robot-specific topology. The learned dense correspondence provides fine-grained geometric anchors for constrained point cloud matching optimization, enabling surface-level pose alignment and direct transfer of interaction contacts. Experiments demonstrate that UMR unifies retargeting across heterogeneous motion sources, robot embodiments, and downstream scenarios ranging from locomotion to interaction, while achieving higher motion fidelity and plausibility than state-of-the-art methods. UMR therefore provides a scalable foundation for transforming large-scale human motion references into robot-ready training data.
I. INTRODUCTION
UMR addresses the scalability limits of skeleton-centric retargeting by learning dense surface correspondences and reusing them for pose and contact transfer across embodiments.
- Motivation: Robot motion data are costly to collect, while human motion is abundant but difficult to use directly because human and robot bodies differ substantially.The differences include proportions, topology, degrees of freedom, and kinematic constraints.
- Limitations of Existing Methods: Existing methods rely on manually selected joint or body-part mappings that are embodiment-specific and provide sparse geometric guidance.Changing the target robot commonly requires redefining correspondences and retuning fitting parameters.
- Surface Representation: Meshes provide dense spatial and orientation information, including regions between joints and contact-relevant surface geometry.This allows correspondence to be inferred from shared spatial extent and local geometry rather than fixed joint semantics.
- UMR Contributions: UMR learns ordered source-robot point-cloud correspondence in a canonical pose, binds pairs to meshes, and uses them for constrained position, orientation, and contact matching.The same correspondence is reused as the meshes move, avoiding additional hand-designed mappings.
- UMR Contributions: UMR provides a unified surface-centric framework that supports heterogeneous motion representations and robot embodiments while enabling surface pose matching and direct contact-map transfer.The evaluation covers whole-body tracking, policy learning, and contact-rich interaction tasks.
II. RELATED WORK
Prior retargeting remains largely skeleton-centric, while surface geometry and contact representations have developed separately; UMR combines them into whole-body cross-embodiment mapping.
- Prior Retargeting: Earlier retargeting methods transferred selected motion features through spacetime optimization, online tracking, and robot-specific kinematic or dynamic constraints.Later work also addressed balance, morphology adaptation, scaling, staged optimization, and interaction contacts.
- Skeleton-Centric Correspondence: Cross-embodiment correspondence in prior humanoid methods is predominantly defined over selected joints, links, keypoints, or rigid-body mappings.Supporting a new embodiment commonly requires redefining the skeletal correspondence and retuning fitting objectives.
- Point-Cloud Correspondence: Point-cloud correspondence methods register observed human geometry to canonical models, using template deformation or implicit fields to establish associations.These associations serve as constraints for estimating model pose and shape.
- Surface-Based Retargeting: Prior surface-based work separately addressed geometry-aware motion transfer or surface contact representation rather than whole-body mapping across embodiments.UMR fills this gap by learning correspondence without a predefined skeletal map and using it for constrained retargeting and contact transfer.
- UMR Overview: UMR organizes retargeting into correspondence learning and correspondence-guided optimization using reusable indexed point pairs.The overview describes matching surface positions, orientations, and contact maps for locomotion and interaction.
A. Overview
UMR represents motion with moving exterior surfaces and separates retargeting into correspondence learning followed by feasible robot-pose computation.
- Input Representation: UMR accepts motion sources that provide a canonical T-pose mesh and posed surfaces over time, including parametric models, rigged characters, and scanned meshes.Surface sampling avoids requiring compatible skeletons or mesh topology between source and robot.
- Point Cloud Correspondence Learning: UMR first learns an ordered mapping between aligned source and robot point clouds in canonical T-poses and binds each pair to the corresponding meshes.The binding allows the correspondence to persist as both bodies move.
- Correspondence-Guided Retargeting: The learned point pairs then provide geometric constraints for computing feasible robot poses from moving surface motion.This stage addresses the need to determine corresponding locations and satisfy robot kinematic constraints.
B. Point Cloud Correspondence Learning
UMR learns reusable, ordered human–robot surface correspondences from canonical point clouds, then uses them to preserve geometric structure across motion sequences and robot embodiments.
- Correspondence Learning: UMR learns indexed human–robot surface pairs once in aligned canonical T-poses for reuse throughout a motion sequence.The paired points are attached to their respective meshes and transported as both bodies move.
- Correspondence Learning: A canonical human template provides fixed point indices across frames, while each target robot contributes an unordered exterior point cloud.This interface accommodates different source formats through canonical templates and posed point-cloud sequences.
- Correspondence Learning: A PointNet-style encoder summarizes the robot point cloud, and an MLP decoder predicts one deformation vector for every indexed human point.The reconstructed robot-side point cloud preserves the human-derived ordering needed to define correspondence.
- Training Objective: The correspondence loss combines Chamfer, repulsion, and edge-smoothness terms to cover the robot surface while preventing collapse and preserving local coherence.Edge smoothness enforces similar deformation vectors between human-template points connected by a fixed geodesic graph.
- Correspondence Learning: Shared point ordering lets robot points inherit human body-segment labels, so segment-specific weights and parameters can transfer without additional body mappings.This ordering forms the unified interface for the subsequent optimizer across different robots.
C. Correspondence-Guided Retargeting
UMR converts dense surface correspondence into point-level pose and contact targets, then optimizes robot motion under kinematic and environmental constraints.
- Correspondence-guided targets: Canonical point pairs become motion targets as source points follow the posed mesh and robot points follow link-local bindings through forward kinematics.This turns retargeting into a surface-matching problem.
- Optimization objective: At each frame, UMR estimates robot generalized coordinates by balancing whole-body pose matching with additional emphasis on active contacts.The objective minimizes stacked point-level residuals.
- Pose matching: Pose residuals match both point positions and local surface orientations, with segment-dependent weights balancing contributions from different human body regions.The selected correspondence set defines the points included in the residual.
- Contact matching: Contact residuals use identical human and robot point indexing to match paired contact vectors without an additional body map.The contact map is represented using paired human and robot contact vectors.
- Contact selection: Active contacts include correspondence points within threshold τc, and the same construction applies to ground, objects, scene geometry, and self-contact.Self-contact replaces the environment point with a point on a distinct, nonadjacent body segment.
- Constrained update: A constrained Gauss-Newton quadratic program handles nonlinear forward-kinematics residuals while enforcing damping, joint limits, trust-region bounds, and floor clearance.The floor-clearance constraint is expressed as At∆q ≤ bt.
IV. EXPERIMENTS
UMR is evaluated across unified retargeting, motion tracking, policy learning, and contact-rich interaction, with quantitative tracking tests conducted on Unitree G1.
- Evaluation scope: The experiments assess unified retargeting across heterogeneous motion representations and embodiments, reference quality for tracking and policy learning, and contact-rich interactions.All quantitative evaluations are conducted on the Unitree G1.
- Implementation: UMR uses MuJoCo and constrained Gauss-Newton subproblems solved with Clarabel, with downstream reinforcement-learning policies trained on NVIDIA RTX 4090 GPUs.Runtime statistics are reported separately in Table I.
- Tracking evaluation: Motion-tracking success rates are evaluated on LAFAN1 using 4,096 trials per entry, with success defined as completing the fixed reference window without termination.The table reports best and second-best values using bold and underlining.
- Tracking errors: Motion-tracking errors are reported over 4,096 trials as Sim (w/o DR) / Sim (w/ DR), enabling comparison across the two simulation settings.Table III reports the statistic in that paired format for each cell.
- Qualitative comparison: The BONES-SEED comparison contrasts UMR on SOMA Proportional reference motion with GMR on SOMA Uniform reference motion.The qualitative comparison is shown with UMR on top and GMR on the bottom.
A. Unified Retargeting Across Sources and Embodiments
UMR provides a shared surface-based interface for diverse motion sources and humanoid embodiments, using region definitions appropriate to each source representation.
- Unified retargeting: Figure 5 demonstrates retargeting across four source representations and five humanoid embodiments through a shared surface-based interface.The sources include MimicKit, BONES-SEED SOMA, LaFAN1 SMPL-X, and an in-house scanned human mesh.
- Region definitions: The source representations use different region definitions: shared body regions for SMPL-X and SOMA, simplified-geometry partitions for the humanoid character, and one whole-body region for the unsegmented mesh.These choices support source-specific geometry while retaining the unified interface.
B. Whole-Body Tracking Performance Comparison
UMR improves whole-body tracking across per-motion evaluation and large-scale policy learning, while preserving actor-specific posture and contact relationships. Its advantage is especially evident under domain randomization, Sim2Sim, and training without the SMPL encoder.
- Evaluation Setup: UMR is evaluated against GMR and Unitree references for per-motion tracking and against large-scale SONIC training settings.The evaluations span 40 LAFAN1 sequences and BONES-SEED policy learning.
- Per-Motion Tracking Quality: UMR achieves higher tracking success rates than GMR across challenging LAFAN1 motions, with broader gains under domain randomization and Sim2Sim.It remains competitive with released Unitree references in those settings.
- Per-Motion Tracking Quality: UMR consistently improves tracking accuracy over GMR and achieves the lowest mean joint-angle errors in both evaluated settings.Body-part position errors remain comparable to Unitree retargeted references.
- Large-Scale Policy Learning: UMR better preserves actor-specific target posture and contact relationships than GMR, which introduces lower-body pose distortion and ground-contact artifacts.The comparison uses actor-specific SOMA-Proportional motions for UMR and released SOMA-Uniform references for Unitree.
- Large-Scale Policy Learning: Without the SMPL encoder, UMR consistently outperforms released Unitree references across total reward, anchor-position error, and mean joint-angle error.The reported relative improvement is roughly 10% toward the end of training.
C. Contact-Rich Interaction Retargeting
UMR improves contact-rich retargeting by preserving geometric relationships between humanoids and interaction targets. It outperforms OmniRetarget across robot-object tasks and remains effective on challenging human-scene interactions, with one task favoring OmniRetarget.
- Evaluation Setup: The interaction evaluation compares UMR and OmniRetarget under the same downstream training protocols, changing only the retargeted references.Tasks cover robot-object and robot-scene interactions from OmniContact and GRAIL motions.
- Human-Object Interaction: UMR preserves intended hand-object contact geometry, whereas OmniRetarget can shift contact to an incorrect surface region despite similar hand positions.This difference is illustrated in a representative human-object interaction example.
- Human-Object Interaction: UMR outperforms OmniRetarget across all reported metrics on Carry, Kick, and Push.Policies trained with UMR references reduce joint error by approximately 40% to 56% across all three tasks.
- Human-Scene Interaction: On noisier GRAIL-derived scene motions, UMR substantially outperforms OmniRetarget on Stair and performs better on both Slope metrics, while OmniRetarget retains a slight advantage on Chair.The authors suggest heuristic stance detection and foot-sticking constraints may mismatch across motion distributions.
V. CONCLUSION
UMR presents dense surface correspondence as a common interface for retargeting heterogeneous human motion to humanoid robots. Experiments support its generality and downstream usefulness, while its current scope assumes mesh-based source geometry and a canonical source template.
- Conclusion: UMR learns dense correspondence between source and robot surface point clouds without manually specified skeletal correspondences.The paired points serve as geometric anchors for constrained motion optimization and contact transfer.
- Conclusion: Across diverse sources and embodiments, UMR improves downstream tracking over GMR, remains competitive with curated Unitree references, and substantially outperforms OmniRetarget on most contact-rich tasks.Resulting policies also transfer successfully to real robots in high-dynamic motions and diverse interaction scenarios.
- Limitations and Future Work: UMR's current limitation is its assumption of mesh-based source geometry and a canonical source template.Future work targets less structured observations, dexterous hands, and multi-agent interactions.