PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanoidNDM: Closing the Training-Deployment Gap of Motion Tracking via Humanoid Neural Dynamics Model

Chao Chen, Zhenrong Wang, Jilong Wang, Yusen Feng, Jiangran Lyu, Li Yi, He Wang

arXiv:2610.34126v1cs.ROcs.AIcs.LG

TL;DR

Humanoid tracking policies trained in simulation face real-world gaps from disturbances and inaccurate modeling, making real-world fine-tuning important. HumanoidNDM combines adaptive reference perturbations, MoCap-free neural dynamics learning, and model-gradient policy updates; experiments report closing the gap and adapting to diverse disturbances.

  • Problem

    Simulation-trained humanoid tracking policies can degrade under real-world disturbances or inaccurate simulation modeling, while simulation retuning relies on inefficient black-box tuning.

    Source: §1 · p. 2

  • Method

    HumanoidNDM combines adaptive low-frequency reference perturbations, proprioceptive neural dynamics modeling with differentiable forward kinematics, and policy fine-tuning through model gradients.

    Source: §1 · p. 2

  • Results

    Experimental results show HumanoidNDM closes the training-deployment gap and enables real-world adaptation to diverse disturbances.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The framework is presented as the first online fine-tuning approach for humanoid tracking policies, validated on a physical robot.

    Source: §1 · p. 2, §6 · p. 8

    The current method is limited to time-invariant disturbances.

    Source: §6 · p. 8

Abstract

from arXiv · show

Humanoid robots have recently achieved remarkable success in motion tracking. However, during real-world deployment, robots inevitably encounter dynamics gaps arising from either extra physical disturbances or inaccurate simulation modeling, resulting in a significant training-deployment gap. Therefore, effectively leveraging real-world data to fine-tune policies represents the crucial last mile for robust deployment. To bridge this gap, we propose HumanoidNDM, an online fine-tuning framework for humanoid motion tracking driven by a real-world-trained neural dynamics model. First, in the data collection stage, we introduce a low-frequency reference perturbation method with an adaptive intensity mechanism, balancing deployment feasibility and exploration efficiency. Second, for the model training stage, we train a robust history-based, proprioceptive neural dynamics model in Euclidean space by incorporating differentiable forward kinematics, eliminating the reliance on costly motion capture systems. Finally, in the policy fine-tuning stage, we directly fine-tune the policy via backpropagated model gradients, achieving high data efficiency. Through physical validation on the Unitree G1 robot, we demonstrate that HumanoidNDM enables rapid real-world adaptation to various perturbations, representing the first online, MoCap-free fine-tuning framework for humanoid tracking policies.

1 Introduction

HumanoidNDM targets the training-deployment gap by fine-tuning humanoid tracking policies online through data collection, model training, and policy updates. Its components address deployment-feasible exploration, MoCap-free dynamics learning, and gradient-guided fine-tuning.

1 Introduction

  • Simulation-trained policies can degrade under real-world disturbances or inaccurate modeling, while black-box simulation retuning is inefficient.The paper identifies real-world data fine-tuning as a crucial last mile for robust deployment.Source: §1 · p. 2
  • HumanoidNDM structures online real-world adaptation into data collection, model training, and policy fine-tuning, and is presented as the first online fine-tuning framework for humanoid tracking policies.The framework is validated for adaptation to challenging disturbances.Source: §1 · p. 2
  • FENCE perturbs reference motions with low-frequency noise and adaptively scales intensity using policy-neighborhood coverage to balance deployment feasibility and exploration efficiency.The method leverages policy robustness for smoother, more stable exploration.Source: §1 · p. 2
  • The dynamics model uses historical observations and differentiable forward kinematics to optimize in Euclidean space from proprioceptive data without MoCap.The paper says this approach improves model reliability and robustness.Source: §1 · p. 2
  • Policy fine-tuning uses supervised updates through model gradients, with longer rollout horizons supporting optimization across the trajectory.The paper contrasts this with prior methods limited by privileged information and short rollouts that cause greedy updates.Source: §1 · p. 2
  • Simulation experiments show significant improvement across training-deployment gaps, while real-world experiments demonstrate efficient adaptation to challenging disturbances.The reported simulation results surpass baselines across various metrics.Source: §1 · p. 2

2 Related Work

Prior real-world policy-learning approaches face data-efficiency or stability constraints, while existing exploration strategies trade hardware feasibility against state-action coverage. HumanoidNDM adopts model-based supervised optimization and proposes FENCE for deployment-viable exploration.

2.1 Learning Robotics Control Policies with Real-World Data

  • Simulator-based adaptation uses system identification, transition compensation, or trajectory alignment but carries OOD risks and slow training; real-world model-free RL is unstable with limited data and complex humanoid dynamics.Real-world model-based methods can support value-based updates or supervised policy optimization through analytical model gradients; HumanoidNDM adopts the latter for data efficiency.Source: §2.1 · p. 3

2.2 Real-world Exploration in Legged Robots

  • Action noise can cause unsafe torques, passive exploration limits coverage, and other smooth or guided perturbations remain tuning-sensitive or difficult to deploy; FENCE uses structured low-frequency reference perturbations with adaptive intensity.Table 1 compares FENCE with representative exploration strategies; its caption marks partial or conditional capability with “∼”.Source: §2.2 · p. 3, Table 1 · p. 3

3 Preliminaries

The preliminaries define humanoid motion tracking as following a reference trajectory and describe a proprioceptive policy whose actions are converted into hardware-executable torques.

3 Preliminaries

  • The tracking task asks the robot to replicate a reference trajectory of target joint positions and velocities.The trajectory is denoted T.Source: §3 · p. 3
  • At each step, the policy receives proprioceptive state, the previous action, and future reference states, then outputs an action in [-1, 1].The proprioceptive state includes joint positions and velocities, gravity vector, and angular velocity.Source: §3 · p. 3
  • The policy action is scaled and added to the reference position; a PD controller computes torques that are clipped to hardware limits before execution.The base policy is trained with PPO in the highly parallelized MuJoCo-JAX environment.Source: §3 · p. 3

4 Method

HumanoidNDM organizes online real-world adaptation into data collection, neural dynamics model training, and policy fine-tuning. Its three components are FENCE exploration, MoCap-free dual-space model learning, and model-gradient-guided policy updates.

4 Method

  • Each adaptation iteration collects real-world data, trains a model, then fine-tunes the policy.The pipeline starts from a simulation-trained policy and repeats these three stages online.Source: §4 · p. 4, Figure 2 · p. 4

4.1 Deployment-viable Real-world Exploration with Frequency-Domain Perturbations

  • FENCE perturbs reference trajectories with low-frequency noise for stable exploration and adaptively scales intensity using policy-neighborhood coverage.It selects among candidate intensity levels using a multi-armed bandit.Source: §4.1 · pp. 4–5

4.2 MoCap-Free Neural Dynamics Model Training with Dual-Space Objective

  • The history-based dynamics model retains a proprioceptive-only interface while differentiable forward kinematics enables Euclidean-space supervision without MoCap.Autoregressive rollouts support longer-term predictive accuracy, and the dual-space objective penalizes both proprioceptive-state and link-pose errors.Source: §4.2 · p. 5

4.3 Efficient and Stable Policy Fine-Tuning via Supervised Gradients

  • Policy fine-tuning backpropagates analytic gradients through the learned dynamics model using autoregressive rollouts and a dual-space tracking loss.The method uses supervised optimization to avoid greedy updates and support longer rollout horizons.Source: §4.3 · pp. 5–6

5 Experiments

HumanoidNDM improves real-world tracking across six tested tasks and performs strongly across simulated disturbance settings. Simulation results and ablations also examine baseline comparisons, exploration stability, dual-space objectives, and history-based modeling.

5.1 Evaluation in Simulation

  • The simulation evaluation uses five motions and introduces inertial, contact, and actuation variations to represent training-deployment gaps.Methods are assessed with four tracking metrics, and exploration stability is assessed with joint-jitter and roll-pitch smoothness metrics.Source: §5.1 · p. 6
  • The simulation baselines include policies trained with exact deployment disturbances, direct transfer, ASAP, and RWM.HumanoidNDM, ASAP, and RWM use 200 trajectories collected from the deployment environment.Source: §5.1 · p. 6

5.1.1 Comparison with Baselines

  • HumanoidNDM achieves the best performance in 40 of 60 simulated metrics, with clear advantages in walk, squat, and swing and comparable results in balance and stride.Across 15 settings, it reduces Elink pos by 30% and Elink rot by 47% versus Direct Transfer; ASAP reductions are 23% and 27%, and RWM reductions are 9% and 35%, respectively.Source: §5.1.1 · p. 6, Table 2 · p. 7

5.1.2 Ablation Study

  • Compared with the dual-space objective, configuration-space-only optimization performs significantly worse on both Elink pos and Eqpos.The history-based model also improves training stability and overall tracking performance over a single-frame variant.Source: §5.1.2 · pp. 7–8, Figure 4 · p. 8

5.2 Evaluation in Real-World

  • HumanoidNDM improves performance across all 6 real-world tasks, with especially significant gains in task-specific motion-completion metrics.For the three weight-disturbance settings, its policy performs comparably to or better than one trained directly in simulation with explicitly modeled disturbances.Source: §5.2 · p. 8, Table 4 · p. 8
  • FENCE remains nearly as stable as No Noise, while the other three tested exploration methods cause obvious joint jitter and roll-pitch shaking on the robot.The real-world comparison deploys the methods for lateral raise with parameters unchanged from simulation.Source: §5.2 · p. 8

6 Conclusion and Limitations

HumanoidNDM combines adaptive exploration, constrained dynamics prediction, and model-gradient policy optimization for online humanoid tracking adaptation. Its current scope is time-invariant disturbances, with more directed exploration identified as future work.

6 Conclusion and Limitations

  • Table 5 is identified as a real-world comparison of exploration methods reporting mean and standard deviation, but the supplied caption gives no results.Source: Table 5 · p. 8
  • The method is limited to time-invariant disturbances, and future work could incorporate prior knowledge for more directed and efficient exploration.Source: §6 · p. 8

A.1.1 Weighted Policy Neighborhood Coverage Metric

  • The weighted Policy Neighborhood Coverage metric uses tracking error to weight coverage across timesteps, while motion-specific neighborhood radii are larger for stable dynamics and smaller for sensitive motions.Source: §A.1.1 · p. 13

A.1.2 Adaptive Intensity Adjustment via Multi-Armed Bandit

  • FENCE uses a UCB multi-armed bandit to adjust perturbation intensity among adjacent levels, with the exploration bonus controlled by β.Source: §A.1.2 · p. 13

A.2.1 Kinematically Constrained Configuration and Gravity Prediction

  • The model integrates predicted joint-velocity residuals to update joint positions, enforcing position-velocity consistency, and represents gravity with roll-pitch coordinates before reconstructing its 3D prediction.Source: §A.2.1 · p. 14

A.2.2 Shared-Backbone Multi-Head Ensemble

  • The dynamics model uses a shared history encoder and decoder with bootstrapped prediction heads, reducing overhead relative to separate full models.Source: §A.2.2 · p. 14

A.2.3 Training Objectives of Neural Dynamics Model

  • The model training objectives include link rotation, root roll-pitch, joint position, root height, and link position prediction losses.Source: §A.2.3 · p. 15

A.3.1 Initialization of Autoregressive Rollout; A.3.2 Training Objectives of Policy

  • Policy rollouts start from buffered real-world histories, anchoring model-gradient updates in visited states; policy losses include link position and rotation objectives.Source: §A.3.1 · p. 15, §A.3.2 · p. 16

A.5 Architecture of the Neural Dynamics Model

The neural dynamics model uses a history-based Transformer ensemble to predict future states, with shared encoding and decoding and separate prediction heads. The appendix also describes porting and tuning the ASAP and RWM baselines.

A.5 Architecture of the Neural Dynamics Model

  • A history-based Transformer ensemble predicts future states from a history of 10 steps, using five ensemble members and separate prediction heads.Source: §A.5 · p. 16
  • The history encoder processes past state-action pairs, and its feature is combined with the current state and action for decoding.Source: §A.5 · pp. 16–17
  • Independent prediction heads estimate next-state residuals, and joint positions are integrated from predicted joint velocities with a 0.02 s time step.Source: §A.5 · p. 17

B Implementation of Baselines

  • ASAP and RWM were ported to MuJoCo-Jax, grid-searched over selected hyperparameters for each setting, and reported as averages over five trials.Source: B Implementation of Baselines · p. 18

C Simulation Experiment Settings

Simulation experiments pair five motions with three disturbances: extra weight, altered joint proportional gains, and reduced foot-ground friction. The motion descriptions span locomotion, balance, and arm-focused actions.

C Simulation Experiment Settings

D Real-World Experiment Settings

The real-world evaluation covers six tasks spanning added weights and hard-to-model disturbances, with task-specific metrics. Experiments average three trials and report no failed runs; 200 adaptation iterations can be completed within an hour.

D Real-World Experiment Settings; D.2 Hard-to-Model Disturbances

D Real-World Experiment Settings

  • Real-world results average three independent trials with no failed runs, and 200 adaptation iterations can be completed within one hour.Each adaptation iteration takes about 3 seconds for model and policy optimization plus 10 seconds for data collection and robot return.Source: D Real-World Experiment Settings · p. 19

D.1 Added Weight Disturbances; Swallow Balance; Arm Swing; Pen Capping

Swallow Balance; Arm Swing; Pen Capping

D.2 Hard-to-Model Disturbances; Lateral Raise; Walking; Bowing

Lateral Raise; Walking; Bowing

E Simulation Training Curves

Figures 5–9 show HumanoidNDM and baseline training curves for walk, balance, squat, swing, and stride motions.

E Simulation Training Curves

F Analysis of Perturbation Intensity Adaptation

Adaptive perturbation intensity improves real-world fine-tuning for stride and swing; HumanoidNDM also improves jump tracking, while simulation pretraining offers no real-world improvement.

F Analysis of Perturbation Intensity Adaptation

  • In simulation, adaptive intensity nearly matches well-chosen fixed σ values of 2, 3, and 4; σ = 1 and σ = 6 perform worse.The authors attribute the similarity to initial policies responding nearly indistinguishably to reasonable perturbation intensities.Source: F Analysis of Perturbation Intensity… · p. 22, Figure 10 · p. 22
  • Adaptive exploration outperforms all fixed perturbation scales for real-world stride and swing motions.The authors hypothesize that larger early perturbations aid exploration, while smaller perturbations near convergence avoid unstable data that may hinder fine-tuning.Source: F Analysis of Perturbation Intensity… · pp. 22–23, Figure 11 · p. 23
  • The best perturbation intensity depends on initial policy responsiveness, which FENCE adapts to with one hyperparameter set.Larger σ helps weakly responsive policies; smaller σ suffices for highly responsive policies, while moderate σ works best for intermediate responsiveness.Source: F Analysis of Perturbation Intensity… · p. 23, Figure 12 · p. 23

G Applicability Beyond Ground-Contact Motions

  • HumanoidNDM consistently improves jump tracking under extra weight, scaled KP, and small friction disturbances.Jump includes a flight phase with both feet off the ground; the method removes root height from NDM inputs, outputs, and prediction loss, using reference root height for global link positions.Source: G Applicability Beyond Ground-Contact… · p. 24, Figure 13 · p. 24

H Effectiveness of NDM Pretraining

I Visualization of Model Ablations

Model ablations show that both history information and the dual-space objective are crucial for rollout accuracy and policy fine-tuning performance.

I Visualization of Model Ablations

Related papers

Loading 2610.34126v1…