Source-linked AI summary
Motion Retargeting with Spatiotemporal Preferences for Humanoid Loco-Manipulation
Xiongbo Shi, Shihang Du, Zongyou Yu, Ao Zhou, Xudong Zhang, Ziyu Lin, Haodong Zhang, Junqiao Zhao, Guang Chen
TL;DR
Object-interaction retargeting must preserve body–object relations at critical stages, but uniform per-frame iteration budgets do not reflect those localized accuracy needs. STR assigns additional solver effort to keyframes and selected body parts, and STRL builds on the references for policy learning. STR reports the highest PECR and ECC among compared methods and approximately 76% fewer SQP iterations than OmniRetarget.
Problem
Object-interaction retargeting needs to preserve body–object relations at critical stages, but uniform per-frame iteration budgets do not reflect where accuracy matters most.
Source: §I · p. 1Method
STR uses demonstration-derived keyframes and selected body parts to allocate retargeting effort, while MCA refines object-surface contact positions and STRL uses the resulting references for policy learning.
Source: §I · pp. 1–2, D. Spatiotemporal Reinforcement… · p. 6Results
STR achieves the highest PECR and ECC among compared methods and uses approximately 76% fewer SQP iterations than OmniRetarget.
Source: B. Retargeting Results · pp. 7–8Takeaways & Limitations
The reported references improve geometric contact coverage and continuity while remaining within the same per-frame solver budgets.
Source: §I · p. 2
Abstract
from arXiv · showhide
Human motion retargeting bridges the embodiment gap between humans and humanoid robots by converting human demonstrations into reference trajectories for whole-body control and loco-manipulation. For object-interaction tasks, successful execution depends on accurately preserving body--object relations at critical stages. However, uniform per-frame iteration budgets do not account for these localized accuracy requirements, limiting the computational efficiency of retargeting. To address this limitation, we present Spatiotemporal Retargeting (STR), which uses keyframes and key body parts to allocate optimization effort. Specifically, a vision-language model selects relevant motion signals and body parts, and keyframes are identified from changes in the selected signals. Based on these selections, STR assigns additional solver iterations to keyframes and higher interaction weights to the selected parts. To further improve contact fidelity, we introduce Mechanics-Aware Contact Anchoring (MCA) to refine object-surface contact positions using an approximate object-wrench model while retaining the selected surface directions. Building on the resulting references, we develop Spatiotemporal Reinforcement Learning (STRL), which combines adaptive key-stage sampling with a contact reward based on target proximity and object contact forces for downstream control. Simulation experiments on a Unitree G1 and real-world experiments on a Unitree R1 demonstrate that STR improves geometric contact coverage and continuity at lower computational cost, while the resulting references improve downstream task completion across diverse object-interaction motions.
I. INTRODUCTION
Object-interaction retargeting must preserve body–object relations at critical stages, where errors can disrupt the motion even if the rest of the reference is faithful. STR and its contact anchoring allocate retargeting effort to those priorities, while STRL uses the resulting references for downstream policy learning.
A. Motion Retargeting
- Prior retargeting methods improve reference fidelity, but do not explicitly identify when extra accuracy matters or which body parts need emphasis.The cited methods include interaction-mesh objectives and kinematic or non-penetration constraints.Source: §I · p. 1
I. INTRODUCTION
- MCA refines object-surface contact positions using an approximate object-wrench model while retaining selected surface directions to improve contact coverage and continuity.The resulting tracking targets improve coverage and continuity within the same per-frame solver budgets.Source: §I · pp. 1–2
- STRL combines adaptive key-stage sampling with a contact reward that encourages selected parts to approach their object-surface targets and establish contact.The paper evaluates reference quality, computational cost, and downstream control.Source: §I · p. 2
- STR uses demonstration-derived keyframes and body-part priorities to allocate solver iterations and interaction weights, reducing computation while retaining dense reference motions.The method gives ordinary frames fewer iterations and keyframes more, while increasing weights for selected body parts.Source: §I · pp. 1–2
A. Motion Retargeting
- Retargeting research spans spacetime constraints, inverse kinematics, learned motion mappings, vision-language supervision, and interaction-aware objectives,,,,,,,,.The cited work includes methods for preserving local relations and contact during motion transfer.Source: A. Motion Retargeting · p. 2
B. Humanoid Whole-Body Control
- Humanoid control work ranges from motion-imitation and whole-body tracking to object-aware control and loco-manipulation skills,,,,,,,,,,,,,, [38], [39].These works address policy learning, object interaction, and downstream task control.Source: B. Humanoid Whole-Body Control · pp. 2–3
C. Preference-Aware Motion Control
The paper places spatiotemporal preferences in a broader line of work on spatial targets and temporal guidance, then describes a pipeline that extracts task-relevant stages and body parts from demonstrations. STR uses those selections to construct dense robot references for policy learning.
C. Preference-Aware Motion Control
- STR couples keyframe-focused iteration budgets with higher objective weights for associated key body parts, and uses the resulting dense references for policy learning.Prior work addresses spatial targets and temporal guidance through distinct mechanisms, including keyframe objectives and body targets,,,,,,, [39],,,,, [46],.Source: C. Preference-Aware Motion Control · p. 3
III. METHOD
- The method combines spatiotemporal priorities with object-geometry and motion-based contact targets to optimize a dense robot reference, which STRL then uses for whole-body policy learning.The overall pipeline proceeds from demonstration analysis through retargeting to policy training.Source: §III · p. 3
A. Problem Formulation
- Each detected stage is represented by its frame interval, onset keyframe, and key body parts; the object trajectory remains fixed while the robot trajectory is optimized under kinematic constraints.The keyframe determines where extra iterations are allocated, and the selected parts receive higher relative weights within the stage.Source: A. Problem Formulation · p. 3
B. Keyframe and Body-Part Selection
- A vision-language model selects task stages, motion signals, and participating body parts, while trajectory analysis detects stage onsets from changes in the selected signals.The first frame of each detected stage is its keyframe, supporting sequential retargeting; selected parts are mapped to interaction-mesh vertices for weighting.Source: B. Keyframe and Body-Part Selection · pp. 3–4, Figure 2 · p. 4
C. Spatiotemporal Retargeting (STR)
STR allocates retargeting effort to keyframes and key body parts, then refines contact targets using an approximate object-wrench model. These preferences shape a constrained, frame-by-frame optimization that produces dense robot references.
C. Spatiotemporal Retargeting (STR)
- Optimization allocates 50 SQP iterations initially, 4 at subsequent keyframes, and 2 at ordinary frames; selected body parts and adjacent object vertices retain higher interaction weights through each key stage, peaking at keyframes and decaying to a positive floor.Source: C. Spatiotemporal Retargeting (STR) · pp. 4–5
- MCA initializes contact anchors from demonstrated grasp locations, estimates forces for the required object wrench, and adjusts anchors within selected surface regions.It keeps selected surface directions fixed and projects updated anchors onto the object surface before defining position and direction targets.Source: C. Spatiotemporal Retargeting (STR) · p. 5
- The constrained per-frame optimization combines weighted interaction preservation with contact-position and contact-direction tracking.Forward kinematics supplies support-point and outward-direction targets; anchor tracking follows MCA-refined locations, while direction tracking aligns the contact surfaces.Source: C. Spatiotemporal Retargeting (STR) · pp. 5–6
D. Spatiotemporal Reinforcement Learning (STRL)
STRL combines adaptive sampling of key and difficult stages with a contact reward for selected body parts. It adds these mechanisms to whole-body tracking rewards for downstream policy learning.
D. Spatiotemporal Reinforcement Learning (STRL)
- STRL prioritizes key stages initially and increases sampling priority for stages with higher estimated failure rates, while a global uniform branch maintains timeline coverage.Stage lengths are included in the priority so short stages do not receive excessive sampling mass solely because the timeline was partitioned.Source: D. Spatiotemporal Reinforcement… · p. 6
- The contact reward encourages selected parts to approach their object-surface targets and establish contact using simulated support points and object poses.The target remains attached to the object during execution, and the contact objective is zero when no part is active.Source: D. Spatiotemporal Reinforcement… · pp. 6–7
- STRL retains tracking rewards and penalties while augmenting them with contact-position tracking in a PPO whole-body policy.The actor receives proprioception, reference-motion commands, and the previous action; the critic also receives privileged body and object states during training.Source: D. Spatiotemporal Reinforcement… · pp. 6–7
IV. EXPERIMENTS
The experiments evaluate retargeting and downstream control across diverse object-interaction tasks using simulation and hardware platforms. The setup compares retargeting methods and reports contact, motion-quality, and policy metrics.
IV. EXPERIMENTS
- Retargeting is evaluated on ten OMOMO human–object interaction tasks using a 29-DoF Unitree G1, with hardware experiments on a Unitree R1.The retargeting comparisons include PHC, GMR, VideoMimic, and OmniRetarget; downstream comparisons vary the retargeter and RL formulation.Source: A. Experimental Setup · p. 7
- Retargeting metrics cover penetration, foot skating, contact preservation, effective contact coverage and continuity, and executed SQP iterations.Simulation policy metrics include success rate and object- and body-tracking RMSE; hardware reports successful trials and body-position and body-rotation RMSE.Source: A. Experimental Setup · p. 7
- Fig. 3 qualitatively compares body–object configurations supporting interaction, with STR combining keyframe budgets, body-part weights, and explicit contact-position and direction targets.The caption identifies the comparison's focus but does not state a quantitative outcome.Source: Figure 3 · p. 7
B. Retargeting Results
STR achieves the highest reported PECR and ECC among compared retargeting methods, with fewer SQP iterations than OmniRetarget. STR references also improve G1 downstream performance, and the paper reports qualitative box-carrying results on the R1.
B. Retargeting Results
- STR achieves the highest PECR and ECC among compared methods while using approximately 76% fewer SQP iterations than OmniRetarget.The iteration reduction is maintained with MCA enabled; the supplied passages do not include Table I's cells.Source: B. Retargeting Results · pp. 7–8
- Fig. 3 shows STR aligning hands with the object surface, whereas OmniRetarget leaves visible hand–object gaps in the examples.The method prioritizes key body parts and explicitly optimizes their contact positions and directions relative to the object.Source: B. Retargeting Results · pp. 7–8
- Without spatial preference, PECR and ECC decrease substantially because the remaining interaction-mesh objective does not explicitly align key body parts with the object surface.Removing temporal preference also increases iteration count and reduces PECR and ECC, particularly ECC.Source: B. Retargeting Results · p. 8
- Removing both spatial and temporal preferences yields the lowest PECR and ECC, while the full method achieves the highest by combining both preferences.The combined preferences prioritize key body parts and keyframes or key stages.Source: B. Retargeting Results · p. 8
C. Downstream RL Performance
- STR improves SR and tracking over OmniRetarget under the original RL formulation; adding STRL further improves SR and Object-Orientation RMSE, with mixed changes in other RMSE metrics.Body-Position RMSE decreases slightly, while Object-Position RMSE and Body-Rotation RMSE increase; Fig. 4 shows STR references complete a lift while OmniRetarget references do not.Source: C. Downstream RL Performance · p. 8, Table III · p. 8, Figure 4 · p. 8
D. Real-World Experiments
- In the reported R1 box-carrying experiment, the robot maintains intended key-part contacts and follows retargeted references; Table IV summarizes success and body-tracking performance.The passage characterizes these results as qualitative evidence that STR-established contact relationships can be realized during physical execution.Source: D. Real-World Experiments · pp. 8–9, Table IV · p. 9
APPENDIX A EVALUATION METRICS
The appendix defines geometric contact and motion-quality metrics, then specifies how simulation and hardware policy outcomes are evaluated.
A. Retargeting Metrics
- Contact Preservation reports frames with no lost source contact label using a 0.28 m object-origin proximity test, unlike PECR and ECC, which assess surface-contact coverage and continuity.SQP iterations are averaged over sequences, and the reduction is computed from unrounded means.Source: A. Retargeting Metrics · pp. 10–11
- Geometric contact validity uses collision-geometry gap and distance to the original object surface, with 10 mm tolerances allowing shallow penetration while requiring contact and surface proximity.The tolerances are fixed across methods after inspecting the contact geometry.Source: A. Retargeting Metrics · p. 10
- PECR measures valid-contact coverage during expected-contact frames, while ECC captures uninterrupted valid-contact runs within each expected-contact interval.Fragmentation reduces ECC at a fixed valid-contact count, and parts with no expected-contact frames are excluded.Source: A. Retargeting Metrics · p. 11
B. Simulation Policy Evaluation
- Simulation SR and four RMSE metrics are averaged across six sequences using 1,920 rollouts per group, with checkpoints selected by highest observed SR.Checkpoint selection does not use RMSE; errors include valid executed frames from successful and failed rollouts, so early failures shorten evaluated trajectories.Source: B. Simulation Policy Evaluation · p. 11
C. Hardware Evaluation
- Hardware success is reported as successful trials out of attempts, while tracking RMSE compares time-aligned robot poses with the R1 reference in centimeters and degrees.Squared errors are averaged over tracked bodies and evaluated frames before taking the square root.Source: C. Hardware Evaluation · p. 11
APPENDIX B TRAINING CONFIGURATION
The appendix records the G1 policy preset, tracking rewards, domain randomization, stage-sampling rules, and contact-target transfer caveats.
A. Training Settings
- The current G1 PPO preset uses 200 Hz simulation and 50 Hz control, with separate actor and critic MLPs and a 154-input actor producing 29 joint commands.Archived checkpoints may retain different observation configurations and training budgets.Source: A. Training Settings · p. 11
B. Tracking Rewards and Regularization
- The baseline tracking rewards weight reference-anchor position and orientation at 0.5 each, while relative-body, velocity, and object terms each receive weight 1.0.Action-rate, joint-limit, and undesired-contact penalties have weights −0.1, −10, and −0.1, respectively.Source: B. Tracking Rewards and Regularization · p. 11
C. Domain Randomization
- The object-interaction preset randomizes object mass from 0.1–2.0 kg and inertia from 50–150% of nominal, but does not implement object-shape randomization.It also randomizes object center-of-mass offsets within ±0.08 m per axis.Source: C. Domain Randomization · p. 12
D. Stage-Sampling Implementation
- Adaptive stage failure estimates use newly resolved assignments only; early termination before a stage end counts as failure, while reaching the end counts as success.Timeouts and external resets before an assigned target are excluded, and evaluation rollouts do not update the sampler.Source: D. Stage-Sampling Implementation · p. 12
E. Contact-Target Transfer and Reward
- The contact-position reward transfers object-local targets and activation curves to the RL reference clock, but altered robot-side contact points are not followed by reference reoptimization.Consequently, exact contact alignment under the transferred reference is not guaranteed.Source: E. Contact-Target Transfer and Reward · p. 12
A. Parameter Settings
The appendix specifies STR settings, ablation variants, iteration and weighting rules, contact activation, and robot-side support-point construction.
A. Parameter Settings
- The reported comparison uses STR settings in Table VI, with contact design preceding per-frame SQP solves and contact activation leaving iteration caps unchanged.The supplied passage gives no numerical parameter values.Source: A. Parameter Settings · p. 12, Table VI · p. 12
B. Ablation Protocol
- Ablations retain a 50-iteration initialization cap; temporal preference assigns four iterations to trigger frames and two to ordinary frames, while disabling it assigns two to all later frames.Disabling spatial preference sets interaction-mesh weights to one and removes both contact objectives; disabling both uses the native uniform-budget solver.Source: B. Ablation Protocol · p. 12
C. Spatiotemporal Weighting and Iteration Budgets
- At initialization the cap overrides a keyframe cap; selected hand vertices receive multiplier four, other selected body and adjacent-object vertices receive two, and all others receive one.Overlapping selections take the largest multiplier, while stage weights and contact objectives remain active at ordinary frames independently of their iteration cap.Source: C. Spatiotemporal Weighting and… · p. 12
D. Contact Interval and Activation
- The contact interval starts at the earliest stage involving both hands and ends after wrist-distance release is confirmed by three consecutive frames beyond a threshold.Wrist-to-object distances are median-filtered over three frames; activation ramps over the interval as specified by its ramp length and normalized progress.Source: D. Contact Interval and Activation · pp. 12–13
E. Body-Part Weights and Robot Contact Geometry
- Hand support points are constructed from directional mesh extent and need not lie on an anatomical palm patch; configured body-part multipliers apply only when a selected part maps to a graph vertex.The resulting point must lie within 2 mm of the physical mesh.Source: E. Body-Part Weights and Robot Contact… · p. 13
F. Source Coordinates and Required Wrench
The method derives object-motion quantities in source/world coordinates and uses them to specify a required contact wrench. It then allocates contact forces jointly over the contact interval, while noting that soft wrench matching does not establish dynamic feasibility.
F. Source Coordinates and Required Wrench
- Opposing object-face regions are selected by minimizing summed wrist-to-region distance, while their outward directions are retained during refinement.Source translation and angular velocity are smoothed before differentiation to obtain translational and angular accelerations used with mass, inertia, and gravity to form the required contact wrench.Source: F. Source Coordinates and Required… · p. 13
G. Force Allocation
- Force allocation jointly matches the required wrench across the contact interval using fixed anchors and inward contact directions.Compression is positive into the object; the formulation penalizes force magnitude and variation and constrains compression and friction.Source: G. Force Allocation · p. 13
- Soft wrench matching does not establish dynamic feasibility.The paper explicitly distinguishes the allocation’s wrench-matching objective from dynamic feasibility.Source: G. Force Allocation · p. 13
H. Anchor Adjustment
MCA adjusts contact anchors within selected object-surface regions while keeping their directions fixed. The robot’s contact positions and directions are then linearized in the target world frame for configuration updates.
H. Anchor Adjustment
- Anchor adjustment parameterizes each contact within a shared tangent plane and bounds displacement according to the region’s extent.One anchor per part is shared across sampled frames; updated anchors are projected back onto their regions with surface directions fixed, over three allocation–adjustment–projection rounds.Source: H. Anchor Adjustment · p. 13
I. Contact Coordinates and Linearization
- Contact positions and directions are expressed in object coordinates using the adjusted anchors and an inward offset.The contact position uses the anchor shifted along the retained surface normal; the inward offset is nonnegative.Source: §I · pp. 13–14
- Contact Jacobians remove translation from the relative motion and support SQP updates of robot configuration toward fixed offline-refined targets.Contact activation is independent of γe(t).Source: §I · p. 14
J. Interaction Linearization and Regularization
The interaction formulation linearizes changes in source and robot vertices over a shared graph, with prescribed object points held fixed. The R1 deployment uses four task-specific PPO actors trained in simulation and exported for onboard use.
J. Interaction Linearization and Regularization
- Interaction linearization stacks vertex-position Jacobians through the shared source graph’s fixed connectivity and uniform Laplacian, with prescribed object points assigned zero rows.The resulting Jacobian maps vertex displacements through the graph Laplacian.Source: J. Interaction Linearization and… · p. 14
- The solver retains linearized non-penetration, stance-foot XY, configuration-limit, and increment-bound constraints, retrying failed initialization without the increment bound.Accepted updates normalize the base quaternion and relinearize; solving ends at convergence or the frame’s iteration budget.Source: J. Interaction Linearization and… · p. 14
APPENDIX D REAL-WORLD DEPLOYMENT CONFIGURATION
- The R1 deployment runs four PPO actors for pickup, rising, walking, and placement, trained in IsaacSim and exported to ONNX with learned observation normalization.The actors share a reference trajectory; Table VII lists the verified deployment settings.Source: Appendix D · p. 14, Table VII · p. 14