PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control

Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang

arXiv:2601.21363v3cs.RO

TL;DR

Humanoid control still lacks a reliable bridge between large-scale pretraining and sample-efficient, safe adaptation to new environments. LIFT uses SAC for parallel-simulation pretraining and model-based finetuning with deterministic real-world actions and stochastic world-model exploration, enabling zero-shot transfer and data-efficient adaptation.

  • Problem

    Humanoid control lacks a reliable bridge between large-scale pretraining and sample-efficient, safe adaptation when real-world or cross-task data are limited.

  • Method

    LIFT uses large-scale SAC pretraining and model-based finetuning that executes deterministic actions in new environments while confining stochastic exploration to a physics-informed world model.

  • Results

    LIFT enables fast, robust humanoid pretraining, zero-shot sim-to-real transfer, and stable learning under limited data during model-based finetuning.

  • Takeaways & Limitations

    The framework provides a practical path toward continuous, efficient humanoid learning by coupling massively parallel simulation with data-efficient adaptation.

  • Takeaways & Limitations

    Real-world finetuning remains time-consuming, requiring sequential iterations, multi-hour wall-clock time, frequent battery swaps, and safety monitoring against actor-model errors.

Abstract

from arXiv · show

Reinforcement learning (RL) is widely used for humanoid control, with on-policy methods such as Proximal Policy Optimization (PPO) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off-policy RL and model-based RL have shown improved sample efficiency, the gap between large-scale pretraining and efficient finetuning on humanoids still exists. In this paper, we find that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots. For adaptation, we demonstrate that these SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods. Data collection in the new environment executes a deterministic policy while stochastic exploration is instead confined to a physics-informed world model. This separation mitigates the risks of random exploration during adaptation while preserving exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning. For code and videos, see https://lift-humanoid.github.io

1 INTRODUCTION

The paper addresses the gap between scalable humanoid-control pretraining and safe, sample-efficient finetuning by combining SAC pretraining with model-based adaptation. Its approach uses deterministic data collection in new environments and confines stochastic exploration to a physics-informed world model.

  • Motivation: PPO is a mainstream humanoid-control baseline because it converges quickly and robustly in massively parallel GPU simulation, but discards off-policy experience.This limits adaptation when real-world or cross-task data are scarce.
  • Motivation: Off-policy algorithms can replay experience for sample-efficient finetuning, but replay overfitting and stochastic exploration can bias adaptation and produce unsafe actions.The introduction identifies limited attention to large-scale parallel simulation as an additional challenge for off-policy methods.
  • Approach: A physics-informed world model improves synthetic-rollout fidelity, reduces model bias, and enables more reliable finetuning with limited data.The model constrains exploration while incorporating structural priors from physics.
  • Contributions: The scalable JAX SAC implementation converges robustly in massively parallel simulation and enables zero-shot physical humanoid deployment within one hour on a single NVIDIA RTX 4090 GPU.The resulting SAC policy also serves as the policy module for model-based finetuning.
  • Contributions: The finetuning strategy executes deterministic actions in new environments while confining stochastic exploration to a physics-informed world model, improving sample efficiency and stabilizing convergence.The design supports data-efficient in-distribution adaptation and stronger out-of-distribution generalization, alongside an open-source pipeline spanning pretraining, deployment, and finetuning.

2 RELATED WORKS

Related work spans large-scale GPU-based RL pretraining, off-policy methods that improve sample efficiency through experience reuse, and model-based techniques that use synthetic rollouts or learned world models. These approaches motivate efficient and safer adaptation for humanoid control.

  • Large-Scale RL Pretraining in Simulation: GPU-based simulators enable fast parallel simulation of thousands of robot environments, making PPO a dominant baseline for large-scale RL pretraining.The cited simulators include IsaacGym, Mujoco Playground, and Brax; PPO is favored for ease of implementation and robust convergence.
  • Off-Policy RL: Off-policy algorithms such as TD3 and SAC improve sample efficiency by reusing past experiences and leveraging critic gradients for policy updates.SAC has also been used to train quadruped robots from scratch with limited real-world data.
  • Model-Based Techniques: Model-based methods improve sample efficiency through synthetic rollouts or learned world models, but often still require stochastic exploration in the environment.MBPO uses synthetic rollouts, while Dreamer has shown strong data efficiency in real-world tasks; related world-model work reconstructs environment state for locomotion robustness.

3 PRELIMINARIES

The paper formulates humanoid reinforcement learning as a Markov Decision Process and distinguishes stochastic exploration during training from deterministic data collection during deployment and finetuning. Its model-based finetuning uses a physics-informed world model that combines known rigid-body dynamics with learned residuals to generate trajectories for policy optimization.

  • Problem Formulation: RL is formulated as an MDP with state and action spaces, transition and reward functions, an initial-state distribution, and a discount factor.A stochastic policy πθ specifies action probabilities conditioned on state.
  • Problem Formulation: Domain randomization perturbs environment transitions during pretraining, while the stochastic policy promotes diverse exploration.For deployment and finetuning, data collection and evaluation use the deterministic policy induced by the mean action µθ(s).
  • Model-based RL Problem: Model-based finetuning seeks a policy πθ ∈ Πθ that maximizes return predicted by a learned world model.The world model generates subsequent states after the initial state is sampled from the true initial distribution.
  • Physics-Informed World Model for Robots: The physics-informed world model uses Lagrangian robot dynamics and learns residual contributions for contact and dissipative effects.Known rigid-body terms depend on robot shape and inertial parameters, while a residual network models the uncertain contributions and next-state prediction uncertainty.
  • Physics-Informed World Model for Robots: The hybrid model combines known rigid-body dynamics with learned residuals and can be rolled out to generate more precise trajectories for policy optimization.Its neural network outputs residual effects and predicted next-state variance through separate heads.

4 LARGE-SCALE PRETRAINING AND EFFICIENT FINE-TUNING

The framework uses SAC for large-scale humanoid pretraining and model-based fine-tuning, unifying both stages while separating deterministic real-environment data collection from stochastic world-model exploration. Large-batch, high-UTD training and a physics-informed differentiable world model support efficient, stable adaptation.

  • Large-Scale Pretraining: SAC is used across pretraining and fine-tuning to align objectives, enable continuous policy updates, and provide a unified implementation.The actor uses proprioceptive observations, while critics can use privileged states; the implementation normalizes features and runs thousands of vectorized environments on one GPU.
  • Large-Batch Updates and High UTD: Increasing UTD from 1 to 10 improves sample efficiency in T1LowDimJoystick without auxiliary stabilizers or architectural modifications.The setup uses 1,024 parallel environments, batch size 1,024, and a 106-transition replay buffer; gains diminish beyond this range.
  • Offline Pretraining from Logged Transitions: The method decouples SAC pretraining from world-model training by logging all transitions during massive parallel simulation and training the model after policy convergence.This avoids slowing pretraining by fitting the world model online while retaining the logged replay data for later adaptation.
  • World Model and Loss: The physics-informed world model differentiably maps privileged states and actions through Brax dynamics, a PD controller, semi-implicit Euler integration, and state reconstruction.Residual torque and predictive uncertainty are learned end-to-end, with rollouts sampling next states from the predicted Gaussian distribution.
  • World Model and Loss: Including base height in the privileged state improves humanoid world-model stability, whereas omitting it causes unstable rollouts and non-convergent fine-tuning.The framework also corrects state mappings and aligns observation dimensions between MuJoCo Playground and Brax.
  • Deployment & Data Collection: Data collection uses deterministic policy actions, while stochastic SAC exploration occurs inside the world model with safety resets and bounded rollouts.Collected transitions fine-tune both model and policy; model rollouts use replay-buffer states as initial conditions and horizon H_wm = 20.

5 EXPERIMENTS

Experiments across simulated and physical humanoids show that LIFT matches or exceeds baseline pretraining performance while enabling stable, data-efficient fine-tuning through deterministic environment execution and a physics-informed world model. Ablations identify world-model pretraining, physics-informed modeling, and large-batch high-UTD training as important to convergence and adaptation efficiency.

  • Pretraining results: LIFT achieves comparable or higher returns than PPO and FastTD3, with faster peak-return stabilization on rough terrain and similar performance and runtime on flat terrain.Pretraining spans six MuJoCo Playground tasks across Booster T1 and Unitree G1 configurations.
  • Sim-to-real deployment: A pretrained low-dimensional Booster T1 policy transfers zero-shot to unseen physical surfaces, including grass, uphill, downhill, and mud.The deployment provides practical evidence that large-scale parallel SAC pretraining can produce deployable humanoid controllers.
  • Sim-to-sim finetuning: During Booster T1 fine-tuning, LIFT consistently converges across in-distribution, long-tail, and out-of-distribution velocity targets, whereas SAC quickly diverges and fails to recover.LIFT uses deterministic action-mean execution in the environment while evaluating convergence, target-speed tracking, body oscillations, and directional deviation.
  • Ablations: With SAC and world-model pretraining, LIFT converges within 4 × 10^4 environment steps and tracks the target speed; removing world-model pretraining slows convergence, while removing both reduces performance to near-static standing.The ablation compares large-scale SAC pretraining and world-model pretraining as separate contributors to fine-tuning efficiency.
  • Ablations: MBPO’s ensemble world model fails to converge with near-zero episode return, while LIFT’s physics-informed model enables stable, learnable rollouts under limited data.The comparison uses an ensemble size of 5, elite size of 3, the same dataset, and identical fine-tuning hyperparameters.
  • Ablations: Increasing UTD from 1 to 5 speeds pretraining convergence, and larger replay buffers and batches accelerate learning until additional increases provide little benefit.Excessively large buffers increase GPU memory usage, and the experiments also vary fine-tuning entropy coefficient α and autoregressive loss horizon.

6 CONCLUSION AND DISCUSSION · A ADDITIONAL EXPERIMENTS · A.1 PRELIMINARY STUDY

LIFT bridges large-scale SAC pretraining with data-efficient model-based finetuning for humanoid control, while preliminary results compare PPO and SAC for learning from scratch under deterministic data collection. The discussion also identifies safety risks during real-world finetuning and motivates future work.

  • 6 CONCLUSION AND DISCUSSION: LIFT combines massively parallel SAC pretraining, zero-shot sim-to-real transfer, and model-based finetuning for data-efficient humanoid adaptation.Pretraining data also bootstrap a physics-informed world model used during finetuning.
  • 6 CONCLUSION AND DISCUSSION: Real-world finetuning remains risky because deterministic data collection can be affected by actor-network model errors.The method uses strict termination conditions and remote human intervention to halt unsafe episodes.
  • A.1 PRELIMINARY STUDY: The preliminary study tests whether finetuning can learn a new policy from scratch when the pretrained policy entirely fails at deployment.The BoosterT1 robot has 12 lower-body degrees of freedom, and data collection remains constrained to a deterministic policy.
  • A.1 PRELIMINARY STUDY: The BoosterT1 reward combines swing-leg reference, body-height tracking, and linear velocity tracking terms commonly used in humanoid locomotion.The experiment targets a 12-degree-of-freedom lower body.
  • A.1 PRELIMINARY STUDY: PPO and SAC are compared within SSRL while all other components remain fixed and only hyperparameters are tuned.SAC uses a state-dependent stochastic actor to explore in the world model and improve policy performance.
  • A.1 PRELIMINARY STUDY: PPO performance is highly sensitive to the initial action standard deviation, making this parameter critical in the preliminary comparison.The comparison used PPO implemented in RSL-RL and SAC as the default setting.
  • A.1 PRELIMINARY STUDY: Figure 6 reports evaluation return and x-axis body linear velocity averaged over 8 random seeds in 128 parallel environments.The preliminary finetuning experiment ran for 48 hours on a single Brax simulation.

A.2 PRETRAINING EXPERIMENTS

Figure 7 compares pretraining performance for LIFT, PPO, and FastTD3 across six humanoid locomotion tasks spanning rough and flat terrain configurations.

  • Pretraining performance comparison: Figure 7 compares LIFT, PPO, and FastTD3 across six humanoid tasks.Results are shown for three robot configurations on rough terrain and three on flat terrain.
  • Pretraining performance comparison: Results report the mean over 8 random seeds.

A.3 FINETUNING EXPERIMENTS IN THE REAL-WORLD … A.6 REAL WORLD EXPERIMENTS

The real-world experiments show that LIFT improves locomotion during finetuning, while hyperparameter studies identify larger UTD and batch sizes, smaller entropy coefficients, and longer horizons as favorable settings. Additional figures document LIFT’s diverse simulated gaits and sim-to-real locomotion across indoor and outdoor environments.

  • A.3 FINETUNING EXPERIMENTS IN THE REAL-WORLD: Across 3 random seeds, LIFT consistently improves real-world episode return, forward-velocity tracking, and angular-velocity tracking while reducing the action-rate penalty.Velocity tracking error at the 0.6 m/s target may arise from noisy onboard-IMU base-velocity estimation.
  • A.4 THE EFFECT OF HYPERPARAMETER: Larger UTD and batch sizes accelerate convergence, while larger buffers improve learning speed but substantially increase GPU memory usage.The compared UTD values are 1, 5, 10, 15, and 20; buffer sizes are 10,000, 100,000, and 1,000,000; batch sizes are 256, 512, 1,024, 4,096, and 8,192.
  • A.4 THE EFFECT OF HYPERPARAMETER: Larger entropy coefficients α degrade finetuning performance, causing less stable convergence and lower final forward velocity.Using the pretraining α gives reasonable results, whereas smaller α yields more stable learning.
  • A.4 THE EFFECT OF HYPERPARAMETER: With target speed = 1.5 m/s, autoregressive loss horizon = 1 sometimes fails to reach the target velocity, while horizons = 2 and = 4 ensure more reliable learning.The figure evaluates episode return and body forward velocity on Booster T1.
  • A.5 PRETRAINING EXPERIMENTS: Figure 11 presents LIFT gaits across different tasks in the MuJoCo Playground.The figure documents simulated gait behavior across multiple tasks.
  • A.6 REAL WORLD EXPERIMENTS: Figure 12 documents sim-to-real reinforcement learning with LIFT.The figure concerns LIFT’s transfer from simulation to real-world reinforcement learning.
  • A.6 REAL WORLD EXPERIMENTS: Figure 13 compares LIFT’s sim-to-real gaits in indoor and outdoor environments.The comparison covers both indoor and outdoor real-world settings.

A.7 FINETUNING: G1 EXPERIMENTS

LIFT fine-tunes Unitree G1 policies in Brax, improving unstable or oscillatory behavior into stable walking across target velocities of 0.6 m/s and 1.5 m/s. At 1.5 m/s, fine-tuning produces a stable, human-like gait with reduced torso pitch.

  • A.7 FINETUNING: G1 EXPERIMENTS: At 1.5 m/s, fine-tuning transforms an unstable, shuffling gait into stable, human-like walking with reduced torso pitch.This sim-to-sim transfer is evaluated for the Unitree G1 in Brax.
  • A.7 FINETUNING: G1 EXPERIMENTS: At 0.6 m/s, LIFT significantly reduces body oscillations and consistently improves policy behavior and reward performance.The result is reported for Unitree G1 finetuning in Brax.
  • A.7 FINETUNING: G1 EXPERIMENTS: At 1.5 m/s, a policy initially unstable in sim2sim successfully stands and walks after finetuning.The finetuning result is reported for the Unitree G1 in Brax.

A.8 NON-LOCOMOTION TASKS … B.3 TRAINING DETAILS OF WORLD MODEL IN FINETUNING STAGE

The framework supports proprioceptive-state rewards through world-model rollouts, while object-centric tasks require extended dynamics modeling. Training combines SAC objectives, differentiable physics, and autoregressive world-model learning for finetuning.

  • A.8 NON-LOCOMOTION TASKS: Rewards depending only on robot proprioceptive state can be optimized through world-model rollouts, including locomotion gaits, whole-body tracking, and disturbance balance.This framework is designed for tasks whose rewards are computable from the robot state.
  • A.8 NON-LOCOMOTION TASKS: The authors extended pretraining from velocity-tracking locomotion to BeyondMimic-style whole-body tracking using Unitree G1 and the LAFAN1 motion dataset.The observation and reward structure was reimplemented in JAX within MuJoCo Playground.
  • A.8 NON-LOCOMOTION TASKS: Object-centric tasks such as kicking require modeling object dynamics, while obstacle avoidance and complex navigation are left for future work or higher-level handling.Suggested extensions include object pose in the state and physical priors such as conservation of momentum.
  • B TRAINING DETAILS: Finetuning collects environment transitions, updates the world model, generates model-based rollouts, and updates the SAC actor and critic from replayed transitions.The procedure uses pretrained policy and world model components alongside separate world-model and SAC replay buffers.
  • B.1 PRETRAINING DETAILS OF SAC: SAC pretraining minimizes entropy-regularized Bellman residuals, maximizes entropy-augmented Q-values, tunes α toward target entropy, and Polyak-averages target critics.Hyperparameter tuning uses Optuna with CmaEsSampler and PatientPruner-based early termination.
  • B.2 TRAINING DETAILS OF WORLD MODEL: The world model converts privileged states to Brax generalized coordinates, maps actions to torques, and advances differentiable rigid-body dynamics through simulator substeps.Brax provides differentiable mass-matrix, Coriolis/centrifugal, and gravity primitives, while control intervals contain N simulator substeps.
  • B.2 TRAINING DETAILS OF WORLD MODEL: Each world-model rollout uses semi-implicit Euler integration, quaternion updates, and normalization to produce the next state after N substeps.The integration pipeline updates velocities, joint positions, base position, and orientation while avoiding numerical quaternion drift.
  • B.3 TRAINING DETAILS OF WORLD MODEL IN FINETUNING STAGE: World-model finetuning minimizes an autoregressive diagonal-Gaussian negative log-likelihood over an H-step unroll, with gradients backpropagated through the full rollout.The loss uses mini-batch size B, horizon H, model predictions, and elementwise log-variance.

B.4 TRAINING DETAILS OF POLICY IN FINETUNING STAGE … C.2 PRETRAINING ENVIRONMENTS

The finetuning design uses physics-informed signals for exact simulator-consistent rewards, while the environment setup specifies a 12-DoF humanoid control task with continuous actions and PD-based motor targets. The preliminary booster environment further defines asymmetric leg-joint gains, phase-based gait generation, and safety termination conditions.

  • B.4 TRAINING DETAILS OF POLICY IN FINETUNING STAGE: Physics-informed signals enable exact simulator-consistent per-step reward computation without training a reward model.Each reward component is computed analytically from kinematic and dynamic signals.
  • C ENVIRONMENT SETUP: The agent controls a 12-DoF T1 humanoid using a 12D continuous action that perturbs motor targets relative to a default pose.The action is applied to simplified lower-body kinematics.
  • C.1 PRELIMINARY STUDY - BOOSTER ENVIRONMENT: Policy outputs are constrained to [−1, 1], and motor targets are computed with proportional-derivative control.The controller uses the stated action vector with PD-based motor-target computation.
  • C.1 PRELIMINARY STUDY - BOOSTER ENVIRONMENT: Hip joints use gains of 200 for Kp and 5 for Kd, while knee joints use 50 for Kp and 1 for Kd.The gains are symmetric across both legs.
  • 1. High-level Policy: Outputs joint position increments, control frequency 100 Hz: Gait generation uses phase ϕ = mod(t · fg, 1.0), with alternating left- and right-leg swing phases and Tswing = 0.2s.The left and right swing phases are centered at 0.25 and 0.75, respectively.
  • C.2 PRETRAINING ENVIRONMENTS: Episodes terminate when base velocities exceed 10.0 m/s or 10.0 rad/s, torso roll or pitch exceeds π/4, or joint limits are violated.These conditions cover excessive motion, torso instability, and mechanical-limit violations.
  • C.1 PRELIMINARY STUDY - BOOSTER ENVIRONMENT: The preliminary study targets a forward velocity of 0.2 m/s and emphasizes swing-leg reference, base-height tracking, and linear-velocity tracking rewards.These are the main reward terms reported for the experimental setup.

C.2.1 T1LOWDIMJOYSTICK · C.3 FINETUNING ENVIRONMENTS · C.3.1 T1LOWDIMJOYSTICK (FINETUNE)

The paper specifies a 12-DoF T1 humanoid joystick-control environment with 12D continuous PD-controlled actions, noisy observations, and fall-based termination. Its finetuning counterpart simplifies and normalizes the environment for MuJoCo Playground-to-Brax transfer while retaining essential behaviors and compatible reward scales.

  • C.2.1 T1LOWDIMJOYSTICK: The pretraining task controls a 12-DoF T1 humanoid with a 12D continuous action that perturbs motor targets relative to a default pose.The environment returns noisy states and a scalar reward, terminating on falls or numerical errors.
  • C.2.1 T1LOWDIMJOYSTICK: The policy observes a 47D state containing body-frame gravity, gyroscope readings, joystick commands, gait phase, joint angles, scaled velocities, and the previous action.A 110D privileged state additionally includes raw sensors, root height, actuator forces, contacts, and foot dynamics.
  • C.2.1 T1LOWDIMJOYSTICK: The 12-D action lies in [−1, 1], and motor targets are converted to torques through a PD control law.The torque uses proportional and derivative gains together with nominal and current joint positions and velocity.
  • C.2.1 T1LOWDIMJOYSTICK: The pretraining setup uses ctrl dt = 0.02 s and sim dt = 0.002 s, external pushes every 5.0-10.0s with magnitude 0.1-1.0N, and enhanced locomotion tracking and foot-kinematics rewards.T1LowDimJoystick is presented as an example of the pretraining environment setup, with T1Joystick and G1Joystick omitted.
  • C.3.1 T1LOWDIMJOYSTICK (FINETUNE): The finetuning environment is a simplified T1LowDimJoystick version for sim-to-sim transfer from MuJoCo Playground to Brax.Its reward settings are adopted from Booster Gym and are designed to preserve similar reward scales and performance characteristics.
  • C.3.1 T1LOWDIMJOYSTICK (FINETUNE): Finetuning normalizes observations with predefined limits, simplifies rewards by zeroing many terms, and represents gait phase continuously using gait process and frequency.The normalized state remains 47D, while the normalized privileged state is 87D.
  • C.3.1 T1LOWDIMJOYSTICK (FINETUNE): Finetuning disables penalties for torques, power, several velocity and acceleration terms, action rate, joint-position limits, collisions, and vertical foot velocity.External pushes remain enabled every 5.0-10.0s with magnitude 0.1-1.0N, and T1LowDimJoystick illustrates the finetuning setup with G1LowDimJoystick omitted.
Loading 2601.21363v3…