Source-linked AI summary
Make Tracking Easy: Neural Motion Retargeting for Humanoid Whole-body Control
Qingrui Zhao, Kaiyue Yang, Xiyu Wang, Shiqi Zhao, Yi Lu, Xinfang Zhang, Qiu Shen, Xiao-Xiao Long, Xun Cao
TL;DR
Human-to-humanoid retargeting must handle non-convex optimization and noisy demonstrations that can produce physically infeasible motion. NMR instead learns a distribution mapping, using CEPR-refined data to train a transformer-based retargeter; on Unitree G1, it improves retargeting quality and accelerates downstream whole-body policy convergence.
Problem
Optimization-based retargeting is non-convex and existing data-driven methods can inherit local-optimum artifacts while lacking physical reasoning for source-motion noise.
Method
NMR learns distribution mappings from human motion to the robot manifold, while CEPR uses VAE clustering, parallel RL experts, and physics refinement to generate physically consistent supervision.
Results
NMR achieves zero joint jumps, reduces self-collision frames by 54%, cuts joint-limit violations by 61%, and accelerates downstream whole-body control policy convergence on Unitree G1.
Takeaways & Limitations
NMR provides an efficient learned retargeting process that produces physically feasible references without requiring a physics simulator during inference.
Takeaways & Limitations
CEPR is morphology-specific, so extending it to other platforms requires regenerating the data pipeline.
Abstract
from arXiv · showhide
Humanoid robots require diverse motor skills to integrate into complex environments, but bridging the kinematic and dynamic embodiment gap from human data remains a major bottleneck. We demonstrate through Hessian analysis that traditional optimization-based retargeting is inherently non-convex and prone to local optima, leading to physical artifacts like joint jumps and self-penetration. To address this, we reformulate the targeting problem as learning data distribution rather than optimizing optimal solutions, where we propose NMR, a Neural Motion Retargeting framework that transforms static geometric mapping into a dynamics-aware learned process. We first propose Clustered-Expert Physics Refinement (CEPR), a hierarchical data pipeline that leverages VAE-based motion clustering to group heterogeneous movements into latent motifs. This strategy significantly reduces the computational overhead of massively parallel reinforcement learning experts, which project and repair noisy human demonstrations onto the robot's feasible motion manifold. The resulting high-fidelity data supervises a non-autoregressive CNN-Transformer architecture that reasons over global temporal context to suppress reconstruction noise and bypass geometric traps. Experiments on the Unitree G1 humanoid across diverse dynamic tasks (e.g., martial arts, dancing) show that NMR eliminates joint jumps and significantly reduces self-collisions compared to state-of-the-art baselines. Furthermore, NMR-generated references accelerate the convergence of downstream whole-body control policies, establishing a scalable path for bridging the human-robot embodiment gap.
I. INTRODUCTION
NMR addresses the physical-feasibility gap in human-to-humanoid motion retargeting by replacing non-convex geometric optimization with learned distribution mapping. Its CEPR pipeline generates physically consistent training data, while the retargeting network improves motion quality and downstream policy training on Unitree G1.
- Motivation: Optimization-based retargeting is non-convex, initialization-sensitive, and prone to artifacts including joint jerks, self-interpenetration, and foot sliding.These failures can force downstream controllers to learn compensatory behaviors or reduce stability.
- NMR framework: NMR reformulates retargeting as distribution mapping from human motion space to the robot’s feasible motion manifold.This learned formulation targets local-minimum, joint-discontinuity, and self-collision issues associated with optimization-based methods.
- CEPR pipeline: CEPR clusters heterogeneous human motions with a VAE, then uses parallel RL experts and physics simulation to generate high-fidelity, physically consistent paired data.The refined motions provide supervision for training the neural retargeting model.
- Retargeting network: The proposed transformer-based retargeting network and two-stage training strategy support broad motion coverage while incorporating physical feasibility.The architecture is designed to replace static geometric mapping with learned motion retargeting.
- Evaluation: On the Unitree G1, NMR reduces joint discontinuities, self-collisions, and joint-limit violations while improving downstream whole-body policy training efficiency and tracking performance.The cited introduction result reports these improvements across diverse motion experiments.
B. Data-driven Retargeting Methods
Data-driven retargeting replaces fragile frame-wise geometric optimization with learned mappings between human motion and feasible robot motion. The paper motivates this shift through Hessian-based non-convexity analysis and addresses supervision quality with physics-based data generation.
- Latent-space methods enable cross-skeleton motion transfer, but humanoid-robot approaches remain limited by optimization artifacts and insufficient physical reasoning over source-motion noise.
- NMR learns a direct mapping from human SMPL motion sequences to feasible humanoid robot motion, bypassing local-optima failures in conventional optimization-based retargeting.
- Non-convexity analysis: Hessian analysis attributes retargeting non-convexity to forward-kinematics curvature and logarithmic-map nonlinearity, permitting strictly negative curvature.
- Non-convexity analysis: The resulting landscape makes gradient-based retargeting sensitive to initialization and capable of stalling in poor local minima, even for simple target motions.
- Non-convexity analysis: Differential IK provides only a locally valid convex approximation, with no assurance of global convergence and sensitivity to initialization and weighting parameters.
- Data-driven retargeting: Physics-simulation-based data generation addresses the supervision bottleneck, while the proposed retargeting network is trained to cover diverse motions while preserving physical feasibility.
B. Clustered-Expert Physics Refinement.
CEPR hierarchically filters human motions, clusters semantically related behaviors, and uses physics-based RL experts to generate physically consistent robot-motion pairs. Kinematic and geometric checks remove abrupt jumps, self-intersections, and floating motions before refinement.
- Pipeline overview: A hierarchical pipeline filters semantically incompatible or physically inconsistent human motions before robot-motion refinement.It removes excessive jerk, unsupported center-of-mass positions, and inadequate foot-ground contact, then applies kinematic retargeting and quality checks.
- Quality filtering: Hard-threshold filtering rejects kinematic outputs with excessive joint velocity, self-intersection above 0.05, or mean lowest-foot elevation above 0.10 m.These checks target IK-induced discontinuities, geometric collisions, and floating motions.
- Motion clustering: Motion clustering assigns behaviorally related sequences to specialized RL experts, avoiding both distributional conflict and the cost of one policy per sequence.Latent motion features are clustered by semantic type, with K-Means using cosine similarity.
- Physics refinement: Each clustered expert tracks reference motions in massively parallel physics simulation, producing robot trajectories paired with the corresponding SMPL sequences.The policy observes reference and proprioceptive states to estimate tracking error and improve whole-body tracking.
- Physics refinement: Adaptive reward tightening lets experts learn coarse motion patterns first and progressively refine tracking precision as training proceeds.The reward standard deviation decreases from σstart to σend over training iterations.
C. Motion Retargeting Network
The motion retargeting network maps human and humanoid motions using distinct representations and a temporal encoder–Transformer architecture. Full self-attention enables parallel predictions conditioned on the entire aligned input sequence.
- Motion representations: Human and humanoid motions use distinct representations to accommodate differences in joint structure and robot actuation.The humanoid representation additionally includes robot joint degrees of freedom.
- Network architecture: A 1D ResNet encoder extracts human-motion features before a Transformer maps them to corresponding humanoid sequences.The architecture follows the design of LLaMA and directly processes motion sequences.
- Temporal modeling: Full self-attention replaces causal attention because human and humanoid sequences are strictly one-to-one temporally aligned.This supports parallel timestep-wise prediction using the entire input sequence.
D. Two-Stage Training Scheme
The network is trained in two stages: broad kinematic pre-training followed by physics-guided fine-tuning. Together, the stages combine motion coverage with improved physical feasibility while avoiding the weaknesses of either dataset alone.
- Kinematic alignment: Kinematic pre-training establishes a broad embodiment mapping across locomotion, manipulation, and martial-arts motions.It uses a large-scale dataset despite residual artifacts such as foot skating and ground penetration.
- Physical grounding: Physics-guided fine-tuning uses approximately 30,000 simulation-validated motion pairs to shift outputs toward the dynamically feasible motion manifold.The physics dataset is smaller by roughly an order of magnitude but provides strong feasibility signals.
- Two-stage rationale: Using physics data without pre-training overfits limited motion patterns, whereas pre-training without fine-tuning leaves outputs physically unconstrained.The combined strategy provides broad motion coverage and improved physical feasibility.
IV. EXPERIMENT
The experiment uses a two-stage optimization schedule and reports quantitative comparisons of retargeting methods on joint jumps, self-collisions, and joint-limit violations.
- Training setup: Kinematic alignment trains with AdamW for 500 epochs, using batch size 128 and an initial learning rate of 2 × 10^-4 with cosine annealing.The physical-grounding stage follows with a reduced learning rate and additional training.
- Training setup: Physical grounding uses learning rate 1 × 10^-5 for an additional 50 epochs.The optimizer and batch size remain unchanged from kinematic alignment.
- Evaluation metrics: Table III compares methods using joint jump, self collision, and reaching joint limit.These metrics target continuity, physical interaction, and actuation-boundary behavior.
B. Datasets and Baselines
The evaluation uses 82 held-out AMASS sequences spanning upper-limb, whole-body primitive, and whole-body complex motions, with short, medium, and long sequence lengths. Against GMR and PHUMA, NMR produces more physically plausible retargeting and improves downstream tracking.
- Datasets: The test suite contains 82 held-out AMASS motion sequences totaling 119K frames at 120Hz.The data are excluded from training for unbiased assessment.
- Datasets: Sequences cover upper-limb-only, whole-body primitive, and whole-body complex motions, testing workspace mapping, multijoint coordination, stability, joint limits, and singularities.Whole-body complex motions include acrobatics and martial arts.
- Datasets: The benchmark partitions sequences into short, medium, and long durations to evaluate initialization, smoothness, cyclic consistency, and cumulative-error suppression.The thresholds are fewer than 250, 250–1000, and more than 1000 frames.
- Retargeting quality: NMR achieves zero joint jumps, 54% fewer self-collisions than GMR, and 16.80% joint-limit violations, roughly half of PHUMA.The evaluation defines joint jumps using a 0.5 rad single-step change and joint-limit violations within 0.05 rad of hardware boundaries.
- Retargeting quality: GMR undergoes a 1.5 rad shoulder-roll change within 0.2s, whereas NMR and PHUMA produce smooth continuous trajectories.GMR’s corresponding angular velocity reaches 7.5 rad/s after becoming trapped near a joint limit.
- Policy tracking: Policies trained on NMR references reach longer episodes and higher rewards, while NMR achieves the highest success rate and lowest MPJPE and W-MPJPE across all sequence lengths.For short sequences, NMR’s W-MPJPE is 0.237 m versus 0.660 m for PHUMA.
D. Correcting Upstream Errors
NMR corrects upstream SMPL pose-estimation errors by mapping motion distributions with global temporal context rather than solving frames independently. This produces smoother, physically feasible trajectories and avoids propagating local artifacts into robot control.
- Error correction: When SMPL sequences contain abnormal jitters, NMR filters the artifacts and generates smooth, physically feasible robot trajectories.The comparison concerns abnormal poses appearing between t+2 and t+5.
- Mechanism: Physical-consistency filtering makes estimation jitters out-of-distribution, encouraging the neural model to generalize toward continuous plausible motion.This differs from frame-wise optimization, which lacks learned distributional smoothing.
- Mechanism: Bidirectional self-attention uses motion before and after an anomalous frame to interpolate and correct it, suppressing local-noise propagation.The model accesses full temporal context at each frame.
- Downstream impact: Frame-wise optimization methods can pass upstream errors directly to the output, making downstream tracking harder to compensate for and causing unstable real-robot motion.The contrast is explicitly drawn with GMR.
- Design rationale: NMR’s learned distribution mapping is designed to avoid the local optima and physical artifacts associated with frame-wise geometric optimization.The framework combines this formulation with CEPR-refined data and a Transformer-based network.
A. Non-convexity Analysis of Retargeting Optimization
The surrogate retargeting objective is generally non-convex because its Hessian combines positive semi-definite Gauss–Newton curvature with potentially negative curvature from forward kinematics and the rotational logarithmic map.
- Geometric formulation: The surrogate uses a coupled SE(3) logarithmic pose representation with rotational and translational coordinates rather than treating translation as ordinary Euclidean position error.The translational log-coordinate is generally not identical to Euclidean position error.
- Hessian decomposition: The Hessian decomposes into a positive semi-definite term and a curvature correction arising from the second-order geometry of the pose-error map.The analyzed map is θ 7→ Log((T⋆)^−1T(θ)).
- Forward-kinematics curvature: Nonlinear forward kinematics can induce negative second-order curvature around an extended configuration with at least two revolute joints.A perturbation bending an interior joint can produce positional variation opposite the extension direction.
- Logarithmic-map curvature: The rotational logarithmic map provides a distinct source of negative curvature even when the kinematic map is locally regular.Its differential varies increasingly sharply as the rotation error approaches π.
- Proposition 1: Together, forward-kinematics curvature and logarithmic-chart nonlinearity establish strictly negative directional curvature for feasible targets and configurations.The result applies to the surrogate objective with n ≥2 and positive rotational and positional weights.
B. Test Motion Names Used in Evaluation
The evaluation uses test motion files from the AMASS dataset, listed across two tables.
- Table V lists the first portion of the AMASS test motion files used for evaluation.
- Table VI continues the AMASS test motion-file list.