PaperlayerView exact revision on arXiv

Source-linked AI summary

ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills

Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi "Jim" Fan, Yuke Zhu, Changliu Liu, Guanya Shi

arXiv:2502.01143v3cs.ROcs.AIcs.LGeess.SY

TL;DR

Agile humanoid whole-body control is hindered by simulation–real-world dynamics mismatch, while existing SysID and domain-randomization approaches have practical or agility limitations. ASAP pre-trains motion-tracking policies on retargeted human motion, learns a residual action model from real-world rollouts, and fine-tunes policies in an aligned simulator. It reduces tracking errors across simulator and real-world transfers and supports diverse agile Unitree G1 skills, while real-world hardware stress and full-body delta-action training remain important limitations.

  • Problem

    Simulation–real-world dynamics mismatch makes agile, coordinated humanoid whole-body skills difficult, while SysID and domain randomization have parameter-tuning, hardware, or conservatism limitations.

  • Method

    ASAP pre-trains simulation motion-tracking policies on retargeted human motion, learns a delta action model from real-world rollouts, and fine-tunes policies with that model integrated into simulation.

  • Results

    ASAP reduces motion-tracking errors across IsaacGym-to-IsaacSim, IsaacGym-to-Genesis, and IsaacGym-to-real transfers, with reductions up to 52.7% in sim-to-real tasks.

  • Takeaways & Limitations

    ASAP supports deployment of diverse agile whole-body skills, including agile jumps and kicks, on the Unitree G1 humanoid.

  • Takeaways & Limitations

    Hardware failures constrain safe real-world data collection, and training a full 23-DoF delta action model remains impractical because of its data demand.

Abstract

from arXiv · show

Humanoid robots hold the potential for unparalleled versatility in performing human-like, whole-body skills. However, achieving agile and coordinated whole-body motions remains a significant challenge due to the dynamics mismatch between simulation and the real world. Existing approaches, such as system identification (SysID) and domain randomization (DR) methods, often rely on labor-intensive parameter tuning or result in overly conservative policies that sacrifice agility. In this paper, we present ASAP (Aligning Simulation and Real-World Physics), a two-stage framework designed to tackle the dynamics mismatch and enable agile humanoid whole-body skills. In the first stage, we pre-train motion tracking policies in simulation using retargeted human motion data. In the second stage, we deploy the policies in the real world and collect real-world data to train a delta (residual) action model that compensates for the dynamics mismatch. Then, ASAP fine-tunes pre-trained policies with the delta action model integrated into the simulator to align effectively with real-world dynamics. We evaluate ASAP across three transfer scenarios: IsaacGym to IsaacSim, IsaacGym to Genesis, and IsaacGym to the real-world Unitree G1 humanoid robot. Our approach significantly improves agility and whole-body coordination across various dynamic motions, reducing tracking error compared to SysID, DR, and delta dynamics learning baselines. ASAP enables highly agile motions that were previously difficult to achieve, demonstrating the potential of delta action learning in bridging simulation and real-world dynamics. These results suggest a promising sim-to-real direction for developing more expressive and agile humanoids.

I. INTRODUCTION

ASAP addresses the simulation–real-world dynamics mismatch limiting agile humanoid whole-body skills through a two-stage alignment framework. It combines simulation pre-training on retargeted human motions with real-world residual correction and policy fine-tuning.

  • Agile whole-body humanoid skills remain difficult because simulation and real-world physics mismatch, beyond existing locomotion- and upper-body-focused work.
  • SysID can require predefined parameter spaces and torque measurements, while domain randomization may produce overly conservative policies.
  • ASAP pre-trains phase-conditioned motion-tracking policies in simulation using human videos retargeted to humanoid robots.
  • The retargeting pipeline reconstructs 3D human motion in SMPL format, filters it through simulation, retargets it to the Unitree G1, and deploys the trained policy on the real robot.
  • Real-world deployment and simulator transfer evaluations show reduced motion-tracking error against SysID, domain-randomization, and delta-dynamics baselines across sim-to-sim and sim-to-real scenarios.

B. Phase-based Motion Tracking Policy Training

The phase-based tracking policy is trained as a goal-conditioned PPO task using robot proprioception and a motion phase, with asymmetric critics and curricula supporting agile motion learning.

  • The policy tracks retargeted robot trajectories as a goal-conditioned reinforcement-learning task using PPO to maximize cumulative discounted reward.
  • The state combines five-step proprioceptive history with a phase variable ϕ ∈ [0, 1], where ϕ marks progress from motion start to end.
  • Asymmetric actor-critic training gives the critic privileged reference-motion and root-velocity information while the actor uses real-world-observable inputs.
  • A termination curriculum progressively tightens tracking-error tolerance to help policies learn difficult agile motions such as jumping.
  • Reference State Initialization samples motion phases so policies learn different phases in parallel rather than strictly sequentially.
  • The pretraining reward sums penalty, regularization, and task-reward terms, alongside basic domain randomization for robustness.

III. POST-TRAINING: TRAINING DELTA ACTION MODEL AND FINE-TUNING MOTION TRACKING POLICY

In post-training, ASAP deploys the pretrained policy on the real robot, records whole-body trajectories and sensor states, and uses these data for subsequent delta-action learning.

  • The pretrained policy can track real-world reference motions but does not initially achieve high motion quality, motivating policy refinement.
  • Real-world rollouts produce trajectories containing recorded states and actions from whole-body motion-tracking tasks.
  • At each timestep, motion capture and onboard sensors record base orientation, joint positions, and joint velocities.

B. Training Delta Action Model

ASAP learns corrective actions from discrepancies between replayed real trajectories and simulated trajectories, integrates those corrections into simulator dynamics, and fine-tunes the tracking policy.

  • Replaying real-world trajectories in simulation exposes tracking discrepancies that provide a learning signal for the simulation–real-world dynamics mismatch.
  • The delta action model outputs corrective actions from the current state and action, which are added to recorded real-world actions.
  • The modified simulator transition is st+1 = f_sim(st, ar_t + ∆at), combining simulator dynamics with reference actions and learned corrections.
  • PPO trains the delta action policy by minimizing simulated-versus-recorded next-state discrepancy with action-magnitude regularization.
  • By reducing lower-body action intensity when simulated motors are stronger than real motors, the model can reproduce real-world failures for policy fine-tuning.

C. Fine-tuning Motion Tracking Policy under New Dynamics

ASAP fine-tunes a pretrained motion-tracking policy by incorporating a frozen delta action model into the simulator under new dynamics, then deploys the refined policy without that model. The evaluation covers simulator transfers and real-world Unitree G1 deployment using motion-tracking metrics.

  • C. Fine-tuning Motion Tracking Policy under New Dynamics: ASAP reconstructs the simulation environment with the learned delta action model while keeping its parameters frozen during policy fine-tuning.The delta model is integrated into the simulator to compensate for dynamics mismatch, while the pretrained policy is optimized with the same reward.
  • C. Fine-tuning Motion Tracking Policy under New Dynamics: The fine-tuned policy is deployed directly in the real world without the delta action model.This deployment produces enhanced real-world motion tracking compared with the pretrained policy.
  • C. Fine-tuning Motion Tracking Policy under New Dynamics: Experiments evaluate ASAP across IsaacGym-to-IsaacSim, IsaacGym-to-Genesis, and IsaacGym-to-real-world Unitree G1 transfers.The study includes simulation and real-world motion-tracking tasks across diverse motion sequences and difficulty levels.
  • C. Fine-tuning Motion Tracking Policy under New Dynamics: Performance is assessed using success rate, global-body-position error, root-relative MPJPE, acceleration error, and root-velocity error.Metric means are computed across all motion sequences used.

A. Comparison of Dynamics Matching Capability

ASAP is compared with Vanilla, SysID, and DeltaDynamics for dynamics matching and closed-loop imitation across simulator transfers and motion difficulties. The reported results show lower tracking errors and stronger performance for ASAP across the evaluated settings.

  • A. Comparison of Dynamics Matching Capability: Open-loop evaluation replays testing-environment trajectories in the training environment and assesses discrepancies using metrics such as MPJPE.Table III is identified as the open-loop performance comparison across simulators and motion lengths.
  • A. Comparison of Dynamics Matching Capability: ASAP consistently outperforms Vanilla, SysID, and DeltaDynamics across Easy, Medium, and Hard closed-loop imitation tasks in IsaacSim and Genesis.The comparison uses variants trained with identical rewards.
  • A. Comparison of Dynamics Matching Capability: ASAP achieves 100%±0.000% success, 106 ±0.498 Eg-mpjpe, and 44.3 ±0.103 Empjpe in the Easy results row.These values are reported for the first simulator transfer shown in the Easy comparison.
  • A. Comparison of Dynamics Matching Capability: ASAP achieves 100%±0.000% success, 112 ±1.648 Eg-mpjpe, and 49.3 ±0.574 Empjpe in the Medium results row.The same row reports acceleration and root-velocity errors of 2.53 ±0.019 and 4.45 ±0.026.
  • A. Comparison of Dynamics Matching Capability: ASAP achieves 100%±0.000% success, 129 ±1.57 Eg-mpjpe, and 56.5 ±1.15 Empjpe in the Hard results row.The same row reports acceleration and root-velocity errors of 3.72 ±0.036 and 6.52 ±0.042.

B. Comparison of Policy Fine-Tuning Performance

ASAP improves policy fine-tuning across simulator-transfer and real-world humanoid evaluations, while supporting dynamic motion execution on the Unitree G1.

  • Comparison across simulators: ASAP consistently outperforms Vanilla, SysID, and DeltaDynamics across Easy, Medium, and Hard levels in IsaacSim and Genesis.It maintains a 100% success rate across both simulators.
  • Policy behavior: ASAP adapts to new dynamics and maintains stable tracking, whereas unfine-tuned baseline policies accumulate errors over time.The comparison concerns closed-loop deployment performance.
  • Experimental setting: The real-world evaluation uses five motion-tracking tasks and additional locomotion data on the Unitree G1.The tasks include kicking, forward jumping, stepping, single-foot balance, and single-foot jumping.
  • Experimental setting: A locomotion policy transitions between motion-tracking tasks because the robot cannot be reset as easily in the real world as in simulation.It keeps the robot balanced until the next tracking task begins.
  • Real-world evaluation: The forward-jump deployment challenges the 1.35m-tall Unitree G1 to leap over 1m.
  • Real-world evaluation: ASAP reduces tracking errors across key metrics for both in-distribution and out-of-distribution real-world motions.The evaluated metrics are Eg-mpjpe, Empjpe, Eacc, and Evel.

V. EXTENSIVE STUDIES AND ANALYSES

The extensive studies examine how to train and use the delta action model, identifying dataset size, training horizon, and action norm weight as important factors.

  • Study scope: The analysis studies dataset size, training horizon, and action norm weight for their effects on open-loop and closed-loop delta-model performance.
  • Training Horizon: A 1.5s training horizon achieves the lowest open-loop errors at 0.25s, 0.5s, and 1.0s evaluation points.The best closed-loop results occur at a 1.0s training horizon instead.
  • Action Norm Weight: Both open-loop and closed-loop errors reach their lowest values at an action norm weight of 0.1.Further increases raise open-loop errors because the minimal-action reward can dominate delta-action reinforcement learning.

B. Different Usage of Delta Action Model

The delta action model supports reinforcement-learning fine-tuning and captures non-uniform dynamics discrepancies across joints, outperforming random action-noise fine-tuning.

  • Delta-action usage: ASAP fine-tunes a nominal simulation policy after integrating the frozen delta action model into the simulator.The delta model is used as a residual correction for the dynamics gap.
  • Delta-action usage: RL fine-tuning achieves the lowest deployment tracking error, outperforming fixed-point iteration and gradient-based optimization.The RL-free methods are described as myopic and affected by out-of-distribution issues.
  • Random action noise: Random action noise improves MPJPE over no fine-tuning for β ∈[0.025, 0.2], but ASAP achieves 126 MPJPE versus 150 for action noise.The broader figure caption reports 173 MPJPE for the best action-noise setting; the detailed experiment reports 150.
  • Joint-wise discrepancies: The learned delta action model produces non-uniform joint corrections, with larger dynamics discrepancies in lower-body motors.The largest gap is observed at the G1 ankle pitch joint.

VI. RELATED WORKS

Prior humanoid work has achieved locomotion and some expressive motions, but agile whole-body skills remain difficult because simulation and real-world dynamics differ. ASAP builds on residual-action modeling to reduce this mismatch and reports improved transfer across simulation and real-world settings.

  • Prior Work: Learning-based humanoid control has produced locomotion, jumping, parkour, dancing, loco-manipulation, and backflipping skills.Physics-based animation has also demonstrated expressive motions such as cartwheels, backflips, and sports movements.
  • Dynamics Mismatch: Simulation-to-reality mismatch arises from inaccurate robot models and complex real-world dynamics that simulators cannot fully capture.Traditional SysID calibrates robot models or simulators using real-world performance.
  • Residual Methods: Residual components have been used to refine controller actions, correct dynamics models, and model residual trajectories.ASAP extends this direction with reinforcement-learning-based residual actions for humanoid sim-to-real transfer.
  • ASAP: ASAP reports reduced motion-tracking errors by up to 52.7% in sim-to-real tasks and deploys agile jumps and kicks on the Unitree G1.The framework is presented as a two-stage approach for adapting simulation-trained policies to real-world physics.
  • Limitations: Agile whole-body experiments are constrained by hardware stress, motion-capture dependence, and the large data demand of full delta-action training.The authors identify motor overheating and failures, unavailable MoCap setups, and impractical 23-DoF training requirements as limitations.

APPENDIX

The appendix describes parameter randomization and SysID settings, then details how real-world trajectories are replayed in simulation to learn and integrate a delta dynamics model. The corrected simulator is used to fine-tune the pretrained policy.

  • Policy Robustness: Domain randomization is used to improve the robustness and generalization of the pretrained motion-tracking policy.The appendix refers to the domain-randomization techniques listed in Table VI.
  • SysID Parameters: The simulated model identifies base center-of-mass shifts, base-link mass offset, and low-level PD gain ratios as representative robot parameters.These parameters are listed in the SysID configuration summarized by Table VII.
  • Trajectory Replay: Collected real-world actions are replayed in simulation to record the corresponding simulated state trajectory.The replayed action sequence and resulting trajectory provide paired data for delta dynamics learning.
  • Delta Dynamics Learning: The neural delta dynamics model is trained to predict the difference between real-world and simulated states.Training uses an autoregressive mean-squared-error objective with a schedule that gradually increases the prediction horizon K.
  • Policy Fine-tuning: The learned delta model is integrated into the simulator so each robot state update includes the predicted correction before policy fine-tuning.Fine-tuning adapts the pretrained policy to the corrected dynamics for improved real-world alignment.

D. Derivation of Training-free Methods of Using Delta Action

The training-free alternatives derive action updates from a one-step matching assumption, using fixed-point iteration or gradient-based optimization. These methods approximate the desired policy but face out-of-distribution issues and one-step limitations, whereas RL fine-tuning performs multi-step matching and achieves superior performance.

  • One-Step Matching: The derivation assumes one-step consistency between real and simulated dynamics.This assumption forms the basis for the subsequent one-step matching condition.
  • Fixed-Point Iteration: Fixed-point iteration initializes y0 = ˆπ(s) and repeatedly updates the estimate until it converges after K iterations.The method is one of the two reinforcement-learning-free approaches considered.
  • Gradient-Based Optimization: Gradient-based optimization minimizes a defined loss with gradient descent to solve for the action estimate y.Together with fixed-point iteration, it approximates π(s) without reinforcement learning.
  • RL Fine-tuning: RL fine-tuning directly optimizes π(s) for real-world deployment and achieves superior performance relative to the training-free approximations.The alternatives can suffer from out-of-distribution issues when trained on limited trajectories.
  • Method Limitation: Fixed-point iteration and gradient-based optimization assume one-step matching because differentiating through f_sim for multi-step matching is generally intractable.RL-based fine-tuning instead performs a gradient-free multi-step matching procedure.
Loading 2502.01143v3…