Source-linked AI summary
Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking
Joao Pedro Araujo, Yanjie Ze, Pei Xu, Jiajun Wu, C. Karen Liu
TL;DR
Humanoid motion tracking depends on retargeting human demonstrations across different embodiments, but artifacts can leave policies with physically infeasible references. This paper compares retargeting methods under limited reward tuning, proposes GMR, and finds that cleaner retargets improve tracking and source-motion faithfulness.
Problem
Retargeting artifacts can leave humanoid RL policies imitating physically infeasible reference motions across the human–humanoid embodiment gap.
Method
The paper compares PHC, ProtoMotions, GMR, and Unitree retargets using BeyondMimic to isolate retargeting effects without reward tuning.
Results
GMR consistently outperforms the open-source retargeters in tracking performance and source-motion faithfulness, approaching the closed-source baseline.
Takeaways & Limitations
Ground penetrations, self-intersections, and sudden joint-value jumps are critical artifacts that increase motion-learning difficulty, while GMR generally yields faithful training references.
Takeaways & Limitations
The study uses motions from only LAFAN1 and considers only the Unitree G1.
Abstract
from arXiv · showhide
Humanoid motion tracking policies are central to building teleoperation pipelines and hierarchical controllers, yet they face a fundamental challenge: the embodiment gap between humans and humanoid robots. Current approaches address this gap by retargeting human motion data to humanoid embodiments and then training reinforcement learning (RL) policies to imitate these reference trajectories. However, artifacts introduced during retargeting, such as foot sliding, self-penetration, and physically infeasible motion are often left in the reference trajectories for the RL policy to correct. While prior work has demonstrated motion tracking abilities, they often require extensive reward engineering and domain randomization to succeed. In this paper, we systematically evaluate how retargeting quality affects policy performance when excessive reward tuning is suppressed. To address issues that we identify with existing retargeting methods, we propose a new retargeting method, General Motion Retargeting (GMR). We evaluate GMR alongside two open-source retargeters, PHC and ProtoMotions, as well as with a high-quality closed-source dataset from Unitree. Using BeyondMimic for policy training, we isolate retargeting effects without reward tuning. Our experiments on a diverse subset of the LAFAN1 dataset reveal that while most motions can be tracked, artifacts in retargeted data significantly reduce policy robustness, particularly for dynamic or long sequences. GMR consistently outperforms existing open-source methods in both tracking performance and faithfulness to the source motion, achieving perceptual fidelity and policy success rates close to the closed-source baseline. Website: https://jaraujo98.github.io/retargeting_matters. Code: https://github.com/YanjieZe/GMR.
I. INTRODUCTION
Humanoid motion tracking uses retargeted human demonstrations to bridge embodiment differences, but retargeting artifacts can leave policies imitating physically infeasible references. The paper isolates retargeting quality and introduces GMR to improve policy performance and motion fidelity.
- The embodiment gap is a central obstacle to using 3D human motion data for humanoid learning.
- Retargeting artifacts such as foot sliding, ground penetration, self-penetration, and physically impossible motion can burden RL policies with correcting infeasible references.
- The study compares PHC, ProtoMotions, GMR, and a closed-source Unitree dataset while suppressing reward tuning through BeyondMimic.
- Evaluation combines strict full-trajectory success under noise, parameter errors, and latency with a user study of perceptual faithfulness.
- Retargeting choice critically affects policy performance: artifacts can make some motions difficult or impossible to learn despite broad overall tracking ability.
- GMR is proposed to address retargeting issues and produce high-quality motion across a wide range of human movements.
II. RELATED WORK
Prior retargeting work spans optimization-based, data-driven, and inverse-kinematics approaches, but humanoid morphology and limited paired data create persistent practical challenges. Existing methods can produce artifacts through inadequate scaling or contact handling.
- Motion retargeting maps articulated character motion using heuristic kinematic constraints or data-driven approaches requiring paired data, semantic labels, or visual evaluation.
- Humanoid retargeting is constrained by the difficulty of acquiring paired or semantically labeled motion data on real robots.
- Directly copying joint rotations can cause floating, foot penetration, foot sliding, and end-effector drift when human and humanoid morphologies differ.
- Whole-body geometric retargeting uses inverse kinematics across mismatched joint spaces, but vanilla methods may ignore Cartesian size differences.
- PHC fits SMPL to the robot and solves inverse kinematics, but ignores contact state and may produce floating, foot sliding, and floor penetrations.
- ProtoMotions scales Cartesian joint positions with global axis-aligned factors before minimizing position and orientation errors.
III. EVALUATION METHOD
The evaluation asks whether retargeting affects tracking, which artifacts hinder learning, and how well methods preserve source-motion appearance. It compares multiple retargeters using BeyondMimic and describes PHC and ProtoMotions processing choices.
- The evaluation tests retargeting effects on policy performance, artifact-related learning failures, and preservation of the source motion’s look.
- BeyondMimic trains motion-tracking policies without reward tuning or extensive domain randomization, enabling retargeting methods to be compared more directly.
- PHC: PHC converts SMPL parameters into robot joint targets, then optimizes root translation, orientation, and joint angles using forward-kinematics position errors.
- PHC: PHC solves its optimization with gradient descent on frame-averaged L2 error and enforces joint limits by clamping.
- ProtoMotions: ProtoMotions uses world-axis scaling and Mink to minimize joint position and orientation errors, then sets retargeted height to the source motion’s lowest height.
- GMR: GMR was developed to address deviation, foot sliding, ground penetration, and self-intersection artifacts associated with prior retargeters.
- Baselines: Unitree’s proprietary retargeted dataset serves as a baseline for comparing open retargeting methods with closed-source retargeting.
B. Data Processing
The study samples varied LAFAN1 motions and retargets each sequence to a Unitree G1, with format conversion for PHC and ProtoMotions. Additional height correction is required for some PHC sequences.
- The selected LAFAN1 sample spans walking and turning through martial arts, kicks, and dancing, while mostly excluding complex environmental interactions.
- Every motion sequence is retargeted to the Unitree G1 robot.
- GMR accepts LAFAN1 BVH files directly, whereas PHC and ProtoMotions require conversion to SMPL or SMPL-X formats.
- The conversion fits body shape, copies matching joint rotations, and estimates root translation by minimizing posed-skeleton position error.
- SMPL-X fits the LAFAN1 skeleton better than SMPL and is therefore used as ProtoMotions’ source representation.
- PHC post-processing can create severe floating of 30cm or higher, so sequences are corrected using mean minimum body height offsets.
C. Motion Tracking Evaluation
Policies are evaluated for robustness and tracking under simulation, randomized conditions, and a ROS-based sim2sim setup, with an accompanying perceptual comparison of retargeted motions. Success requires completing the rollout without exceeding anchor-body deviations.
- Evaluation setups: 100 rollout trials are used for sim2sim evaluation in a ROS environment that models timing, synchronization, and state-estimation noise without privileged simulator information.The robot starts from its default pose, and each policy is evaluated until failure or reference completion.
- Success criterion: A rollout counts as successful only if the policy reaches the episode end without excessive anchor-body height or orientation deviation from the reference.Tracking errors are computed over frames for which the policy remains alive.
- Perceptual evaluation: The user study presents a reference motion and retargeted videos, asking participants which retargeted video more closely resembles the reference.Figure 1 depicts the reference video and retargeted alternatives used for this comparison.
E. User Study Evaluation
GMR retargets human motion to humanoid robots through mapped key bodies, rest-pose alignment, scaling, and constrained inverse-kinematics optimization. The pipeline applies the pose retargeting procedure sequentially across motion frames.
- GMR pipeline: GMR begins by mapping human skeleton bodies to corresponding humanoid key bodies such as the torso, head, legs, feet, arms, and hands.Users may also provide weights for position and orientation tracking errors.
- GMR pipeline: Rest-pose alignment offsets human body orientations, and sometimes positions, to match the robot’s rest pose and mitigate artifacts such as toe-in.This alignment supplies the reference configuration for subsequent optimization.
- GMR pipeline: Body positions are scaled using the human skeleton height, a reference height, body positions, and body-specific scaling factors.The root case simplifies the scaling equation, and uniform root-translation scaling is reported as crucial for avoiding foot sliding.
- GMR pipeline: Robot generalized coordinates are optimized to reduce body position and orientation errors, using a two-stage process with end-effector constraints in the first stage.The optimization uses robot forward kinematics, joint limits, orientation differences, and weighted errors; differential IK computes velocities rather than directly minimizing coordinates.
- GMR pipeline: For motion sequences, GMR retargets frames sequentially and initializes each optimization with the result from the previous frame.The solver terminates on convergence or after 10 iterations, while the differential-IK time parameter need not equal the reference frame interval.
V. EVALUATION RESULTS
The evaluation reports success rates across three setups and tracking-error statistics over the 21 evaluated motions.
- Reported metrics: Success rates are reported for sim, sim-dr, and sim2sim, while tracking-error statistics are reported across 21 motions.These results are presented in Tables I and II.
A. Impact of retargeting method in policy performance
Across the 21 tested motions, most policies succeeded, but retargeting method strongly affected difficult cases, tracking errors, and robustness. Unitree performed near perfectly, while GMR and ProtoMotions generally followed it and PHC performed worst.
- 11 of 21 motions exceeded 98% success across all retargeting methods, while 3 achieved perfect performance.
- Unitree policies achieved near-perfect performance on all motions, followed by GMR and ProtoMotions, while PHC had the lowest performance.
- PHC- and ProtoMotions-based policies could have considerable global and local tracking errors despite completing episodes successfully.
- Successful tracking across varied motions can come at the expense of higher tracking errors and reduced robustness for some retargeted data.
B. Impact of retargeting artifacts
Low policy success is directly associated with artifacts in retargeted motion. The reported problematic artifacts include ground penetration, self-intersections, and sudden joint-value jumps, while GMR’s jumps were rare and optimization-dependent.
- Low sim2sim scores can be directly connected to artifacts in the retargeted motion.
- PHC retargets of “Dance 1” and “Dance 2” contained noticeable ground penetration, reaching 60 cm in one case.
- ProtoMotions’ retarget of “Run (stop & go)” contained intersecting robot legs.
- GMR’s “Dance 5” retarget had many sudden waist-roll jumps, although these artifacts occurred on less than 2% of the full motion dataset.
- The three artifact types to avoid are physically inconsistent height, self-intersections, and sudden joint-value jumps.
C. User study
The user study found that GMR retargets were judged more faithful to the reference motion than PHC and ProtoMotions, while Unitree remained the most faithful overall.
- GMR was judged more faithful to the reference motion than retargets produced by PHC or ProtoMotions.The study included 20 users.
- Unitree retargets were considered more faithful than GMR, but users had greater difficulty distinguishing the two.
- GMR’s high policy success rates and close faithfulness score indicate it is a viable alternative to Unitree-retargeted data.
D. First reference frame
Reference-motion initialization materially affects policy performance, while the broader evaluation finds that retargeting artifacts and dataset or robot scope constrain the analysis.
- First reference frame: The starting frame of a reference motion can have a large impact on policy performance.The same policy was evaluated from different start frames in sim2sim MuJoCo experiments.
- First reference frame: Reference motions should start from poses the robot can safely reach when inference begins and end in stable poses for safe deactivation.
- Ground penetrations, self-intersections, and sudden joint-value jumps greatly increase the difficulty of learning motion-tracking policies.
- The study uses motions from only LAFAN1 and evaluates only the Unitree G1, limiting its empirical scope.The authors recommend extending the analysis to additional datasets and humanoid robots.
- The evaluated retargeting methods are general, so extending the analysis to other humanoid robots remains a future-work direction.