PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction

Lujie Yang, Xiaoyu Huang, Zhen Wu, Angjoo Kanazawa, Pieter Abbeel, Carmelo Sferrazza, C. Karen Liu, Rocky Duan, Guanya Shi

arXiv:2509.26633v3cs.ROcs.AIcs.LGeess.SY

TL;DR

Existing humanoid motion-retargeting methods struggle with embodiment differences, physical artifacts, and neglected robot-object-terrain interactions. OmniRetarget uses an interaction mesh and hard constraints to generate diverse references, enabling long-horizon skills and zero-shot sim-to-real transfer.

  • Problem

    Existing retargeting methods produce artifacts such as foot skating and penetration while failing to explicitly preserve spatial and contact relationships in scene interactions.

  • Method

    OmniRetarget minimizes interaction-mesh deformation while enforcing collision avoidance, joint and velocity limits, and foot-contact constraints to augment single demonstrations across configurations.

  • Results

    30-second complex, multi-stage tasks were enabled on a physical humanoid through zero-shot sim-to-real transfer using OMNIRETARGET-retargeted data.

  • Takeaways & Limitations

    The framework provides diverse, high-quality interaction-preserving trajectories that support proprioceptive RL for dynamic whole-body scene-interaction skills.

Abstract

from arXiv · show

A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning (RL) policies. However, existing retargeting pipelines often struggle with the significant embodiment gap between humans and robots, producing physically implausible artifacts like foot-skating and penetration. More importantly, common retargeting methods neglect the rich human-object and human-environment interactions essential for expressive locomotion and loco-manipulation. To address this, we introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh that explicitly models and preserves the crucial spatial and contact relationships between an agent, the terrain, and manipulated objects. By minimizing the Laplacian deformation between the human and robot meshes while enforcing kinematic constraints, OmniRetarget generates kinematically feasible trajectories. Moreover, preserving task-relevant interactions enables efficient data augmentation, from a single demonstration to different robot embodiments, terrains, and object configurations. We comprehensively evaluate OmniRetarget by retargeting motions from OMOMO, LAFAN1, and our in-house MoCap datasets, generating over 8-hour trajectories that achieve better kinematic constraint satisfaction and contact preservation than widely used baselines. Such high-quality data enables proprioceptive RL policies to successfully execute long-horizon (up to 30 seconds) parkour and loco-manipulation skills on a Unitree G1 humanoid, trained with only 5 reward terms and simple domain randomization shared by all tasks, without any learning curriculum.

I. INTRODUCTION

Humanoid whole-body control is limited by reward-sensitive exploration and the embodiment gap, while existing retargeting often fails to preserve physical feasibility and scene interactions. OmniRetarget addresses this with interaction-preserving constrained retargeting and single-demonstration augmentation for diverse robot-object-terrain behaviors.

  • Motivation: Humanoid RL faces a data bottleneck because high-dimensional actions and complex dynamics make natural behavior difficult to learn from scratch.Reward engineering strongly affects efficient exploration.
  • Motivation: Human demonstrations capture coordinated whole-body behaviors, but morphology and degrees-of-freedom differences make direct motion adaptation insufficient.Usable references must also adapt scene interactions to the robot’s form.
  • Limitations of prior work: Existing retargeting produces foot skating and penetration while omitting explicit spatial and contact preservation, complicating downstream RL training.Many methods use unconstrained or softly penalized optimization with simple keypoint matching.
  • OMNIRETARGET: OMNIRETARGET models robot-object-terrain relationships with an interaction mesh and generates kinematically feasible variations from a single demonstration.Augmentation spans object configurations, shapes, robot embodiments, and environments.
  • OMNIRETARGET: Constrained optimization enforces collision avoidance, joint limits, and foot-contact stability while minimizing interaction-mesh deformation.The resulting references provide dense learning signals for RL with minimal reward engineering.
  • Contributions: The framework combines interaction-preserving retargeting, scalable trajectory augmentation, an open-source dataset, and zero-shot sim-to-real demonstrations on a physical humanoid.Reported skills include object carrying, platform climbing, jumping, rolling, and wall-flipping.

C. Data Generation for Humanoid Loco-Manipulation

The data-generation pipeline constructs an interaction mesh over body joints, objects, and terrain, then retargets motion by minimizing mesh deformation under hard kinematic constraints. Its formulation supports multiple embodiments and interaction types while preserving spatial and contact relationships.

  • Data generation: Teleoperation provides online adaptation but remains labor-intensive, operator-fatiguing, and difficult to scale for large-scale data generation.Embodiment mismatch, limited haptic feedback, and extreme-motion stabilization further constrain its applicability.
  • Interaction mesh construction: The interaction mesh is a volumetric structure whose vertices combine robot or human joints with sampled object and environment points.Shrinking or stretching the mesh warps human motion onto the robot while preserving relative configurations and contacts.
  • Interaction mesh construction: Delaunay tetrahedralization connects user-defined joint positions with sampled object and environment points, using denser surface sampling to maintain contacts.The construction explicitly includes both body and scene geometry.
  • Optimization objective: The objective minimizes Laplacian deformation between corresponding source and target meshes built from anatomical, object, and environmental points.Correspondences need only remain semantically consistent, such as hand-to-hand matching.
  • Optimization objective: Laplacian coordinates compare each keypoint with a weighted average of its neighbors, and deformation energy measures changes between source and retargeted meshes.Experiments use uniform neighbor weights.
  • Hard constraints: At each timestep, the solver finds a robot configuration minimizing deformation subject to collision, joint, velocity, and stance-foot constraints.Sequential SQP-style optimization is warm-started from the previous frame for temporal consistency and faster convergence.
  • Generalization: The pipeline transfers across Unitree G1, H1, and Booster T1 embodiments and supports robot-object, robot-terrain, and robot-only motions.Cross-embodiment adaptation changes keypoint correspondences and the robot collision model.

B. Terrain, Object Shape and Spatial Augmentation

OMNIRETARGET augments a single demonstration across object configurations, shapes, and terrain features while preserving essential spatial and contact relationships. Added anchoring prevents trivial rigid transformations and yields diverse, kinematically valid motions.

  • Augmentation mechanisms: A single demonstration can generate variations in object configuration, shape, and terrain features by re-solving the interaction-mesh optimization.Object poses and dimensions are modified, while terrain height and depth are varied.
  • Interaction preservation: Minimizing interaction-mesh deformation produces a new robot motion that preserves the original interaction’s essential spatial and contact relationships.The optimization is solved for each augmented scenario with the source and augmented object or terrain states fixed.
  • Object augmentation: Local-frame interaction meshes make the robot’s interacting body parts follow object transformations during pose and shape augmentation.Translations, rotations, and dimension scaling modify the object while retaining the relevant interaction geometry.
  • Demonstration: Fig. 4 illustrates terrain-height, object-pose, and object-shape variations whose optimized motions transfer from simulation to hardware.The figure presents the three variation axes and sim-to-real transfer of the resulting motions.
  • Diversity constraints: Anchoring selected robot body parts to the nominal trajectory prevents augmentation from collapsing into a rigid transformation of the entire robot.In pick-up tasks, lower-body deviations are penalized to encourage new upper-body coordination while retaining nominal foot poses.
  • Terrain augmentation: Terrain augmentation varies platform height and depth and can add sampled ground-surface points to the interaction mesh for stable elevated-terrain contact.The added terrain constraints encode environmental contact during optimization.

IV. RL TRAINING WITH MINIMAL FORMULATION

OMNIRETARGET uses high-quality reference motions with a minimal proprioceptive RL formulation to bridge kinematics and dynamics. The approach relies on five reward terms, simple observations, and limited randomization rather than extensive reward engineering.

  • Policy role: RL converts OMNIRETARGET’s kinematic references into physically realizable actions and enables zero-shot transfer from simulation to hardware.The low-level policy bridges the gap between reference-motion generation and robot dynamics.
  • Reward design: Clean, interaction-preserving references reduce the need for ad-hoc reward regularizers that compensate for artifacts in noisy retargeted motions.The paper contrasts minimal rewards for clean references with tedious and fragile reward tuning for artifact-laden data.
  • Observations: The policy uses a minimal proprioceptive observation space containing reference motion, pelvis state, joint state, and the previous action.The agent is blind to explicit scene and object information and follows the reference trajectory through these observations.
  • Observations: For agile motions, pelvis linear position error and velocity are masked when state estimation is unreliable.This modification addresses unreliable state estimates while retaining the rest of the observation design.
  • Rewards: Only five reward terms are used, including body and object tracking, action-rate, soft joint-limit, and self-collision penalties.The stated reward design includes body tracking, object tracking where applicable, action rate, soft joint limits, and self-collision.
  • Training setup: Training uses the same weights and hyperparameters from prior work without tuning, while object-tracking termination occurs beyond 1.0m and 45° deviation.The object termination criterion is applied only after reasonable body tracking is achieved.
  • Randomization: Object physical parameters are randomized across mass, center of mass, inertia, and shape, while robot randomization uses four specified terms.The supplied passage lists object ranges and introduces the robot-side randomization scheme.
  • Behaviors: The training demonstrations include crawling down a slope among the evaluated whole-body behaviors.This behavior is identified in the supplied section material.

V. EXPERIMENTAL RESULTS

Experiments show that OMNIRETARGET supports diverse whole-body behaviors, long-horizon parkour, high-dynamic wall flips, and systematic sim-to-real augmentation. Augmented data remains effective without reward tuning, while full augmentation achieves 79.1% success versus 82.2% on nominal motions.

  • Agile Loco-Manipulation: OMNIRETARGET policies achieve zero-shot sim-to-real transfer for agile whole-body loco-manipulation on a Unitree G1 humanoid.The demonstrated behaviors include box carrying, platform climbing, and slope crawling with clean contact sequences.
  • Long-Horizon Parkour: A 30-second sequence carries a 4.6 kg chair, uses it as a stepstone, climbs, leaps off, and performs a parkour-style landing roll.The multi-stage task demonstrates long-horizon interaction involving an object, platform, and landing dynamics.
  • High-Dynamic Motion: The wall-flip completes in approximately 0.5 second and reaches a peak angular velocity of 15 rad/s.The rigid robot foot requires closer wall alignment for sufficient contact area and friction.
  • Sim-to-real with Augmented Data: A single nominal motion can be augmented for box-picking across shapes and positions and platform climbing across heights, with hardware transfer without reward tuning.The interaction-mesh formulation expands the repertoire of scenes and behaviors available in hardware.
  • Sim-to-real with Augmented Data: Domain randomization alone performs poorly because policies struggle to explore far beyond the nominal reference.The comparison concerns perturbing object shapes and poses only during training under the paper’s RL formulation.
  • Sim-to-real with Augmented Data: 79.1% success on the full augmented dataset compares with 82.2% when evaluating on nominal motions only.The authors report this as showing expanded coverage without significant performance degradation.

B. Benchmark Against Prior Retargeting Pipelines

Against PHC, GMR, VideoMimic, and a public locomotion dataset, OMNIRETARGET reports stronger kinematic quality and downstream RL performance across tasks. Its hard-constraint, interaction-preserving formulation produces diverse trajectories with fewer artifacts and supports long-horizon hardware skills.

  • Benchmark Setup: The benchmark compares OMNIRETARGET with PHC, GMR, and VideoMimic using datasets totaling 2.78 hours of OMOMO, 1 hour of in-house MoCap, and 4.6 hours of LAFAN1.The retargeted dataset is intended for open-source release.
  • Kinematic Quality: Kinematic quality is evaluated through penetration, foot skating, and contact preservation criteria.Penetration measures intersection duration and depth; foot skating measures stance-foot duration and velocity; contact preservation measures desired contact duration.
  • Kinematic Quality: OMNIRETARGET significantly outperforms baselines across most kinematic metrics, with minimal penetration violations that can be efficiently fixed by RL.The paper attributes occasional minor penetration to constraint linearization in the sequential SOCP solver.
  • Locomotion Benchmark: For pure locomotion, OMNIRETARGET achieves fewer physical artifacts and better hard-constraint satisfaction than the publicly available Unitree LAFAN1 retargeted dataset.LAFAN1 is treated as a strong baseline for RL-based locomotion training.
  • Downstream RL: Using identical hyperparameters without manual tuning, OMNIRETARGET policies exceed baselines by over 10% with lower variance across challenging motions.The study selects 39 challenging motions and evaluates success in simulation using training termination criteria.
  • Downstream RL: Terrain contact preservation is directly proportional to success rate, while GMR’s terrain contact preservation drops by nearly 50%.The results indicate that contact preservation and penetration reduction matter for generalization across diverse tasks.
  • Baseline Analysis: VideoMimic has the weakest interaction preservation among baselines, with poor object-manipulation performance linked to its collision model and soft-cost formulation.The paper notes that tuning soft penalties may partly account for this result.
  • Conclusion: OMNIRETARGET’s unified interaction mesh and hard constraints generate artifact-free trajectories from single demonstrations and enable long-horizon dynamic skills on hardware.The conclusion connects improved kinematic quality with proprioceptive RL and zero-shot sim-to-real transfer.

APPENDIX

The appendix standardizes heterogeneous human-motion datasets into robot-compatible keypoint trajectories while accounting for morphology differences. It contrasts model fitting with direct scaling for SMPL-based data.

  • Data Standardization: Human motion inputs differ in format and physical attributes, so preprocessing converts them into consistent global 3D keypoint time series.The representation must account for differences between human demonstrators and the target robot.
  • Input Formats: OMOMO uses the parametric SMPL model, whereas the in-house MoCap and LAFAN1 datasets use BVH skeleton hierarchies.SMPL represents shape and pose with β and q parameters; BVH defines a skeleton hierarchy.
  • SMPL Preprocessing: SMPL preprocessing either fits the model to robot morphology or directly scales human keypoints.These strategies provide alternative ways to adapt source motion to the target robot.
  • Model Fitting: PHC optimizes a global scale and SMPL shape parameters to match robot link lengths in a canonical T-pose before generating source keypoints.VideoMimic instead optimizes per-link scale factors jointly with robot motion over the trajectory.
  • Direct Scaling: GMR and OMNIRETARGET generate keypoints from the original SMPL parameters and use a global scale equal to the robot-to-human height ratio.Both methods also support per-bone scaling based on corresponding human–robot link lengths, although this work uses one global factor.

2) Skeleton Hierarchy Data:

Skeleton-hierarchy inputs are converted through forward kinematics and then retargeted using method-specific optimization objectives. Compatibility differs across pipelines, with GMR and OMNIRETARGET supporting both SMPL and BVH data.

  • Preprocessing: BVH keypoint positions are derived from the skeleton forward-kinematics function and scaled using the robot-to-human height ratio.This provides robot-sized source keypoints for skeleton-hierarchy data.
  • Data Compatibility: GMR and OMNIRETARGET process both parametric-model and raw-skeleton data, while PHC and VideoMimic primarily target SMPL inputs.Converting other formats to SMPL adds a separate, tedious preprocessing step.
  • Method Comparison: After preprocessing, the methods differ in optimization strategy and objective, including trajectory-wise matching, per-frame IK, and interaction-mesh deformation.The comparison is summarized in the paper’s methodology table.
  • PHC: PHC minimizes trajectory-wide source-to-robot keypoint position error using gradient descent.Its optimization operates over the entire trajectory rather than independently at each frame.
  • GMR: GMR solves an inverse-kinematics problem at each frame to match source keypoint positions and orientations.It uses Sequential Quadratic Programming through the mink library.

3) VideoMimic:

VideoMimic jointly optimizes robot motion and SMPL scaling to preserve pairwise geometric relationships, while adding soft penalties for contacts, collisions, limits, and smoothness. In the comparison, OMNIRETARGET instead uses per-frame interaction-mesh optimization.

  • VideoMimic: VideoMimic jointly optimizes robot motion and SMPL per-link scaling factors across the full trajectory.Its primary objective preserves scaled pairwise distances and orientations between keypoint pairs.
  • VideoMimic: VideoMimic adds soft penalties for foot-contact matching, foot skating, collisions, joint limits, and temporal smoothness.These penalties supplement its pairwise distance-and-orientation objective.
  • Comparison: The methodology comparison lists VideoMimic as trajectory-wise optimization with pairwise distance-and-orientation preservation and model fitting for SMPL data.OMNIRETARGET is listed separately as per-frame interaction-mesh deformation with direct scaling for SMPL and BVH.
  • IMMA: IMMA first warps the interaction mesh by optimizing intermediate robot keypoints, then solves a separate inverse-kinematics problem for joint angles.Later stages impose hard foot and waist constraints, but the sequential approach omits joint and velocity limits.

5) OMNIRETARGET:

OMNIRETARGET retargets motion frame by frame by minimizing interaction-mesh deformation, while supporting task-specific costs and object-trajectory augmentation. Its solver enforces small, constrained configuration updates for feasible motion.

  • OMNIRETARGET: OMNIRETARGET minimizes Laplacian deformation of interaction meshes for each frame and can incorporate task-specific costs such as orientation matching.This makes the retargeting objective flexible and extensible.
  • Augmented Object Trajectory: Object augmentation applies transient positional and rotational offsets that decay exponentially after object motion begins.Separate time constants control translational and rotational decay, and orientation composition uses quaternion multiplication.
  • Object Frame: Constructing the interaction mesh in the object’s local frame makes Laplacian coordinates invariant to the object’s global rotation and translation.This preserves relative spatial relationships when the object changes pose.
  • Sequential SOCP: The interaction-mesh optimization is solved sequentially with an SOCP for the configuration increment, iterating up to 10 times per timestep.Each frame starts from the previous frame’s solution, supporting temporal consistency.
  • Interaction Mesh: The target and source interaction meshes provide the corresponding geometric structures used for optimization.The paper illustrates these meshes side by side in Fig. 9.
  • Sequential SOCP: The solver constrains joint positions and velocities while using a trust-region radius ε = 0.2 to keep linearized approximations valid.The configuration update is linearized around the current iterate and bounded by physical limits.

E. Downstream RL Evaluation Breakdown

The downstream evaluation examines failure distributions across robot–object and robot–terrain interactions. OmniRetarget is most robust under randomized object properties and provides the most accurate references for demanding climbing terrains, with one remaining failure attributed to the simple RL formulation.

  • Evaluation breakdown: Fig. 10 histograms show failure rates and variance across OmniRetarget and baselines for robot–object and robot–terrain interaction tasks.Failures are broken down by individual motions, while augmented motions are excluded because baselines do not support augmentation.
  • Robot–object interaction: OmniRetarget achieves the most robust robot–object interaction performance, with results concentrated in the high-success region.This task heavily randomizes object properties and therefore measures robustness more than reference accuracy.
  • Overall finding: The evaluation indicates that OmniRetarget provides both superior robustness under variation and higher reference accuracy.The distinction is visible across the randomized-object and demanding-terrain settings.
  • Robot–terrain interaction: OmniRetarget fails on only one climbing motion, compared with four for GMR and nearly half the motions for PHC and VideoMimic.Climbing terrains require precise references because low-quality motions can cause outright failure with no successes.
  • Limitation: The one remaining failure is attributed to the simple RL formulation used by the authors.The authors suggest curriculum learning as a possible future extension for extremely difficult motions.
Loading 2509.26633v3…