Source-linked AI summary
Dense Temporal Motion Retargeting for Legged Robots
Jaeryeong Kim, Taerim Yoon, Jin Cheng, Sungjoon Choi, Stelian Coros
TL;DR
Dynamic motions challenge legged-robot retargeting because timing and control must adapt together to robot capabilities. DTMR jointly optimizes both at every control step, deforming only the motion parts that need timing changes. It reports more precise retargeting than a temporal baseline under the same deformation budget, about 19× faster, and transfer of trained policies to a real humanoid.
Problem
Dynamic motions require careful timing and control adjustment because robot capabilities constrain motion execution and the two are interdependent.
Source: §I · p. 1Method
DTMR jointly optimizes timing and control at every frame in one optimization, deforming only motion parts that need timing changes.
Source: §I · p. 1, §VII · p. 7Results
Under the same deformation budget, DTMR retargets more precisely than the baseline and is ∼19× faster; policies trained on its references transferred to a real humanoid.
Source: §I · p. 2, §VII · p. 8Takeaways & Limitations
Allowing more temporal deformation gives a knob between timing preservation and retargeting precision, while dense adjustment particularly helps dynamic motions.
Source: §I · p. 1, §VII · p. 8DTMR does not preserve motion meaning through timing constraints such as dance-music synchronization, and it does not handle object interaction requiring joint timing adjustment.
Source: §VII · p. 8
Abstract
from arXiv · showhide
Legged robots can learn expressive whole-body skills from the motions of humans and animals. Due to the morphology gap between the source and the robot, however, the motion must be tailored to the dynamic properties of the robot. In particular, dynamic motions such as a jump require careful adjustment, since their timing and control are interdependent. We propose dense temporal motion retargeting (DTMR), which jointly optimizes timing and control within a single optimization, where dense means that the timing is adjusted for every control step. This dense formulation enables DTMR to deform only the parts of the motion that need a change in timing. The problem is solved with sampling-based model predictive control (MPC) in parallel on a GPU. We evaluate DTMR against baselines on two hours of human motion with four humanoid robots, where the results show that DTMR outperforms baseline methods, particularly on dynamic motions. We also show that allowing more temporal deformation yields more precise retargeting. We further compare DTMR with a baseline that optimizes the temporal dimension, where the result shows that DTMR retargets more precisely under the same deformation budget while being ~19x faster. Lastly, policies trained on our references transfer to a real humanoid robot.
I. INTRODUCTION
Human and animal motions offer expressive skills for legged robots, but morphology gaps and dynamic feasibility require adapting both motion timing and control. DTMR jointly optimizes them densely, changing timing only where needed; evaluations report more precise retargeting and real-robot transfer.
I. INTRODUCTION
- Dynamic retargeting must coordinate timing and control because a robot’s capabilities constrain motion speed, while control must adapt to the changed motion.A weaker actuator may require slowing the motion down.Source: §I · p. 1
- Existing temporal retargeting scales whole motion segments, changing frames that need no adjustment, and uses a slow nested optimization that repeatedly solves control.DTMR instead optimizes timing and control together.Source: §I · p. 1
- DTMR adjusts timing at every frame alongside control in one optimization, so only motion parts needing timing changes are deformed.Sampling-based MPC with GPU parallelization solves the resulting problem.Source: §I · pp. 1–2
- The evaluation uses two hours of human motion and reports better downstream imitation than baselines, especially for dynamic motions, with policies transferring to a real humanoid.The paper also reports that allowing more temporal deformation yields more precise retargeting.Source: §I · pp. 1–2
- Under a similar deformation budget, DTMR retargets more precisely than baseline temporal optimization and runs ∼19× faster.The evaluation also reports stronger retargeting on dynamic motions, greater precision with more temporal deformation, and transfer of trained policies to a real humanoid.Source: §I · pp. 1–2
II. RELATED WORK
Prior retargeting work addresses spatial plausibility or robot dynamics, but these approaches either omit robot dynamics or preserve source timing. Temporal retargeting relaxes that timing constraint, though earlier segment-wise nested optimization is slow and deforms frames unnecessarily.
II. RELATED WORK
- Kinematic retargeting methods produce spatially plausible poses but do not account for target-robot dynamics.Approaches include learned pose mappings, inverse kinematics, shared skeletons, and physical-plausibility constraints.Source: §II · p. 2
- Dynamics-aware methods make motions feasible but retain source timing, which a robot may be unable to follow.These methods include balance constraints, whole-body optimization with contact forces, and differentiation through optimal control.Source: §II · p. 2
- Earlier temporal retargeting scaled coarse motion segments through nested optimization, making it slow and deforming frames that needed no change.DTMR instead optimizes every frame’s timing together with control in one MPPI problem, locally reparameterizing infeasible portions.Source: §II · p. 2
III. PRELIMINARIES
MPPI updates a control plan using sampled rollouts, while the problem formulation makes source-motion progress a decision variable alongside motor control. Together, these let the robot adjust how quickly it follows the motion while realizing it under its dynamics.
III. PRELIMINARIES
- MPPI updates a horizon plan from the current state, applies the first input, then shifts the plan forward to warm-start the next control step.The update uses noisy plan samples, rollout costs, and cost-weighted sampled noise sequences.Source: §III · p. 2
- The phase-augmented problem fits MPPI, which the paper uses to optimize motor control and source-motion progress as a single control signal.Independent rollouts can be parallelized on a GPU.Source: §III · p. 2, §IV · p. 2
IV. PROBLEM FORMULATION
- Phase ϕ maps the source motion from beginning to end, and its increment sets the robot’s progress rate through that motion.A smaller increment slows a portion the robot’s actuators cannot follow at source speed.Source: §IV · p. 2
- The phase trajectory selects which source-motion part is imitated at each control step, while motor controls must realize that timing under robot dynamics.Eq. (2) searches timing and motor control together.Source: §IV · p. 2
V. METHOD
DTMR jointly optimizes motor control and motion timing by treating the phase increment as a control input. Its MPC cost balances reference tracking, temporal deformation, and control regularization.
V. METHOD
- DTMR makes motion phase part of the control problem, jointly optimizing timing and motor control through the phase increment at each step.The rollout determines its duration by continuing until the phase reaches the end of the motion.Source: §V · p. 3
- The running cost compares the simulated robot with the reference frame selected by the phase increment, while also penalizing torque energy and action rate.Tracking includes root position, link positions, orientations, and velocities adjusted for the reference playback speed.Source: §V · p. 3
- A phase penalty trades temporal deformation against retargeting precision, penalizing slowdown while leaving speedup unpenalized in the stated setup.The phase penalty is quadratic in deformation rate and is zero at source speed.Source: §V · p. 3
- MPC plans over a horizon using interpolated control nodes, initialized from the source-speed reference joint trajectory and shifted forward after each control step.Repeated sampling updates use annealed noise and only the lowest-cost elite fraction of rollouts.Source: §V · pp. 3–4
VI. RESULTS
The evaluation covers human and quadruped motions retargeted across humanoid and quadruped robots. DTMR supports downstream imitation learning better than other retargeting datasets, and allows precise retargeting under matched deformation budgets; policies trained on its references are deployed on a real robot.
VI. RESULTS
- DTMR retargets a two-hour human-motion dataset to four humanoids and quadruped clips to the Unitree Go1, then uses the resulting references for downstream imitation learning.The humanoids are the Unitree G1, R1, H1-2, and Booster T1.Source: §VI · p. 4
- Allowing more temporal deformation yields more precise retargeting, and DTMR achieves higher precision than the baseline under the same deformation budget.The dataset produced with DTMR serves downstream imitation learning better than datasets from other retargeting methods.Source: §VI · p. 4
- Policies trained on DTMR references are deployed on a real robot.The paper reports deployment on a real robot without specifying further details in these passages.Source: §VI · p. 4
A. Dataset Construction with DTMR
DTMR retargets 999 human-motion clips, totaling 120 minutes, to each of four humanoids. It redistributes timing by motion class while keeping total clip lengths close to their sources.
A. Dataset Construction with DTMR
- The corpus contains 999 clips (120 minutes) retargeted to each of four humanoids, with 496 locomotion clips and the remainder spanning dance, gesture, object interaction, idle, or seated motion.Clips were randomly sampled from BONES-SEED.Source: A. Dataset Construction with DTMR · pp. 4–5
- Refining one clip takes roughly 80–100× its duration on an RTX 4090, totaling about 200 GPU-hours for the two-hour corpus.The solver rolls out samples in parallel on one GPU.Source: A. Dataset Construction with DTMR · p. 5
- Jogging and running are stretched 1.13–1.42× at the median, while idle and seated clips are compressed to 0.90–0.95×; jumps remain near source length at 0.96–1.03×.The slowed take-off is repaid by a hurried recovery.Source: A. Dataset Construction with DTMR · p. 5
- Total corpus length changes by at most 9.5% on H1-2 and stays within 1% of the source on G1, so DTMR mostly redistributes time within clips.Source: A. Dataset Construction with DTMR · p. 5
B. Evaluating Improvements over Baseline Motion Retargeters
Across four humanoids, dense temporal optimization improves both dynamic feasibility and semantic fidelity over baseline retargeters. The gains are substantially larger on dynamic, harder-to-track clips than on static clips.
B. Evaluating Improvements over Baseline Motion Retargeters
- Against the average of four baselines, DTMR improves dynamic-feasibility error by 15.1% at Q25 and 43.4% at Q75.Against SoMa-RT alone, median error falls 13.4% on average across the four robots, with Q75 improving 14.9% versus 1.8% at Q25.Source: B. Evaluating Improvements over… · p. 5
- DTMR has the lowest dynamic-feasibility error in every reported column except Q25 on R1 and H1-2, where SoMa-RT is marginally ahead.The evaluation uses policy tracking error as the dynamic-feasibility measure and reports Q25, Q50, and Q75 across clips.Source: B. Evaluating Improvements over… · p. 5
- Against the average of four baselines, DTMR reduces semantic-fidelity error by 15.8% at Q25 and 39.8% at Q75.Against SoMa-RT, median DTW error falls 12.9% on average across the four robots; gains are 11.4% at Q75 and 2.2% at Q25.Source: B. Evaluating Improvements over… · p. 5
- DTMR has the lowest semantic-fidelity error in every reported column except Q25 on H1-2.Semantic fidelity is measured by DTW retargeting error.Source: B. Evaluating Improvements over… · p. 5
C. Evaluating Improvements over the Temporal Retargeting Baseline
Compared with STMR, DTMR combines dense timing and control optimization rather than coarse-segment timing with nested optimization. It achieves lower retargeting error on the tested motions and substantially shorter runtimes.
C. Evaluating Improvements over the Temporal Retargeting Baseline
- STMR scales coarse motion segments and uses nested timing and control optimization, whereas DTMR optimizes timing densely at every frame with control in one optimization.The comparison uses ten G1 clips and two Go1 motions because STMR’s nested optimization is slow.Source: C. Evaluating Improvements over the… · pp. 5–6
- On the illustrated flight-phase clips, STMR skips the jump or fails the landing, whereas DTMR completes both.Source: Figure 3 · p. 6
D. Evaluating the Deformation–Precision Trade-off
DTMR exposes a trade-off between preserving source timing and retargeting precision: allowing more temporal deformation improves precision. It can deform only motion parts that need timing changes and is more precise than STMR at comparable deformation.
D. Evaluating the Deformation–Precision Trade-off
- At wdϕ = 0.5, retargeting error is 170 mm versus 202 mm at wdϕ = 10; settings with wdϕ ≤0.5 score 164–170 mm.Lower wdϕ permits more temporal deformation.Source: D. Evaluating the… · p. 6
- Reducing wdϕ from 5 to 0.5 raises mean absolute deformation rate from 0.20 to 0.23 and lowers retargeting error by 30 mm.Source: D. Evaluating the… · p. 6
- On G1, error decreases as DTMR allows more temporal deformation, and it is more precise than STMR at the same absolute deformation rate; on Go1, it reaches similar error with a smaller rate.Source: Figure 5 · p. 7
- DTMR adjusts timing at every frame, unlike the baseline’s coarse-segment scaling, allowing only the parts needing a timing change to be deformed.Source: Figure 6 · p. 7, Figure 5 · p. 7
- DTMR is 22% more precise than STMR at the same deformation with wdϕ = 0.1, and 5% more precise with 22% less deformation at wdϕ = 5.Source: D. Evaluating the… · p. 7
E. Evaluating the Timing Adjustment
DTMR concentrates timing changes where a motion needs them, unlike segment-wise timing adjustment, which can deform motion that needs no change. In the G1 jump example, DTMR preserves source timing outside the jump while adjusting its phases.
E. Evaluating the Timing Adjustment
- For the G1 high jump, DTMR delays jump onset and shortens flight while the rest of the motion retains source timing.These localized changes follow from adjusting deformation rate at every step.Source: E. Evaluating the Timing Adjustment · p. 7
- STMR lengthens the static and dynamic clips by 32% and 12%, respectively, applying one deformation rate per segment.The comparison uses a static clip and a jump retargeted to the G1.Source: E. Evaluating the Timing Adjustment · p. 7
- DTMR keeps the static clip at source tempo and changes timing around the jump, slowing the crouch and speeding up the landing.Unlike STMR, DTMR adjusts the deformation rate at every step rather than once per segment.Source: E. Evaluating the Timing Adjustment · p. 7
F. Real-World Deployment
DTMR references supported retargeting and real-robot imitation, while dense timing changes preserve source timing where changes are unnecessary. The conclusion reports stronger dynamic-motion learning and transfer, alongside limits in preserving dance meaning and handling object interaction.
F. Real-World Deployment
- On the deployed G1 dances, final mean tracking reward rises from 40.8 to 42.5 on the foot-shuffle dance and from 38.9 to 41.7 on the weight-shift dance.The policies were trained on DTMR references; the compared training used kinematic retargeting alone.Source: F. Real-World Deployment · p. 7
VII. CONCLUSION
- DTMR jointly optimizes timing and control for every frame in one optimization, deforming only motion parts that need timing changes.Unaffected parts retain source timing.Source: §VII · p. 7
- Under a similar deformation budget, DTMR retargets more precisely than baseline temporal optimization and is ∼19× faster; the study covers two hours of motion retargeted to four humanoids.The conclusion also reports that policies trained on the references learned dynamic motions better and transferred to a real robot.Source: §VII · p. 8
- DTMR does not identify which dance parts must stay in time with music, and it does not handle object interaction requiring joint timing adjustment.A phase-cost weight bounds timing changes but does not identify which motion parts must preserve timing.Source: §VII · p. 8