PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning

Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, Jianyu Chen

arXiv:2408.14472v1cs.ROcs.AIcs.SYeess.SY

TL;DR

Humanoid robots are suited to human-centric environments, but real-world RL locomotion has been demonstrated on relatively simple terrain. This paper introduces DWL, an end-to-end RL framework for humanoid locomotion; its experiments report challenging-terrain navigation with zero-shot sim-to-real transfer using the same policy across scenarios.

  • Problem

    Real-world RL control of humanoid robots had been limited to relatively simple terrains despite their suitability for human-centric environments and the additional challenges of their structure.

    Source: §I · p. 2

  • Method

    DWL combines state denoising and policy learning, masking unavailable information and adding simulation noise before an encoder-decoder reconstructs the robot's true state.

    Source: §I · p. 2, §VII · p. 9

  • Results

    DWL achieved real-world navigation across snowy land, stairs, deformable ground, and irregular surfaces with end-to-end RL and zero-shot sim-to-real transfer, including 100% success across various terrains in irregular-terrain experiments.

    Source: B. Indoor Experimental Validation · p. 7, §VII · p. 9

  • Takeaways & Limitations

    The paper's in-depth analysis indicates the effectiveness of state estimation and the importance of active 2-DoF ankle control in the reported experiments.

    Source: §VII · p. 9

Abstract

from arXiv · show

Humanoid robots, with their human-like skeletal structure, are especially suited for tasks in human-centric environments. However, this structure is accompanied by additional challenges in locomotion controller design, especially in complex real-world environments. As a result, existing humanoid robots are limited to relatively simple terrains, either with model-based control or model-free reinforcement learning. In this work, we introduce Denoising World Model Learning (DWL), an end-to-end reinforcement learning framework for humanoid locomotion control, which demonstrates the world's first humanoid robot to master real-world challenging terrains such as snowy and inclined land in the wild, up and down stairs, and extremely uneven terrains. All scenarios run the same learned neural network with zero-shot sim-to-real transfer, indicating the superior robustness and generalization capability of the proposed method.

I. INTRODUCTION

Humanoid robots suit human-centric environments, but their locomotion presents added robustness challenges, and real-world RL demonstrations have remained limited to simpler terrain. DWL is introduced as an end-to-end RL approach for complex terrain, with a world model and active ankle control among its contributions.

I. INTRODUCTION

  • Humanoid mobility matters because human-designed environments suit their skeletal structure and require walking across complex real-world terrain.The introduction identifies mobility as crucial for completing diverse real-world tasks.Source: §I · pp. 1–2
  • Higher center of gravity, swinging-leg instability, greater leg inertia, torso and arm weight, and larger size make humanoid locomotion robustness challenging; real-world RL had been limited to simpler terrains [26].The paper contrasts this with RL work on quadrupedal [28] and bipedal robots [27].Source: §I · p. 2
  • DWL is presented as enabling a humanoid robot to master real-world challenging terrains through end-to-end RL and zero-shot sim-to-real transfer, with the same policy used across scenarios.The reported environments include snowy inclined land, stairs, and irregular surfaces.Source: §I · p. 2
  • The paper also identifies active 2-DoF ankle control with a closed kinematic chain as a contribution, contrasting it with one-DoF or passive ankle control [32], [26].The ankle mechanism is described as reducing leg inertia while providing two degrees of freedom in each ankle.Source: §I · p. 2, Figure 2 · p. 2

II. RELATED WORKS

  • DWL places a world model in an encoder-decoder framework and uses masking loss to predict state from observations, unlike conventional multi-stage training, detailed reward design, or behavior cloning approaches,, [24].The related-work discussion also notes model-based methods' reliance on accurate environment dynamics [2, 38, 5] and RL's lesser reliance on exact modeling.Source: §II · p. 2, §I · p. 2

III. PROBLEM SETTING

The problem setting treats locomotion as decision-making from partial, noisy observations and describes the robot hardware used. DWL addresses the sim-to-real gap by encoding noisy observation histories into a latent state and reconstructing the full state for learning and adaptation.

A. Reinforcement Learning Background

  • In simulation the agent observes the complete state, whereas real-world sensors provide partial observations, so the policy maps observation history to actions in a POMDP.The stated RL objective is to maximize expected total return.Source: A. Reinforcement Learning Background · p. 3

B. Humanoid Robot Hardware

  • Experiments use XBot-S and XBot-L, with arm motors held stationary and six motors powering each leg, including two ankle motors in a closed kinematic chain.The robots weigh 38 kg and 57 kg, respectively; the ankle mechanism is intended to reduce leg inertia while retaining degrees of freedom.Source: B. Humanoid Robot Hardware · p. 3

A. Denoising World Model Learning

  • DWL targets the sim-to-real gap, attributed mainly to inaccurate hardware simulation and limited onboard sensor information, through representation learning for online adaptation and state estimation.Its two stated components are an encoder-decoder world model and a policy-gradient method for iterative controller improvement.Source: A. Denoising World Model Learning · p. 3

1) Encoder-Decoder Architecture of DWL:

  • DWL treats environmental, dynamics, sensor, and masking problems as noise, simulates noisy observations, and trains an encoder-decoder to recover the true state and dynamics.Domain randomization perturbs state and dynamics, while masking emulates information unavailable from real-robot sensors.Source: 1) Encoder-Decoder Architecture of DWL · p. 3
  • A recurrent encoder maps historical noisy sensor observations to a latent state, and a decoder reconstructs the robot's true state from that representation.The latent representation is described as a condensed summary of situational awareness; an L1 regularization term promotes sparsity, with λr as its coefficient.Source: 1) Encoder-Decoder Architecture of DWL · pp. 3–4

2) Policy Learning in DWL:

DWL uses an asymmetric actor-critic arrangement: the actor acts from encoded observation history, while the critic can use state information for value estimation. Privileged state information is used during training, with a unified state definition for estimation and value assessment.

2) Policy Learning in DWL:

  • The asymmetric actor-critic uses PPO for the actor, while the critic can use state information to calculate the value function.The actor policy conditions on the encoder's latent representation given the observation history.Source: 2) Policy Learning in DWL · p. 4
  • The actor's PPO objective compares target-policy and behavior-policy probabilities and applies a clipping range c1, c2.The behavior policy supplies the data used for sampling.Source: 2) Policy Learning in DWL · p. 4
  • Privileged information supports training, and the framework uses a unified state definition for state estimation and critic value assessment.The paper states that this avoids designing separate privileged information.Source: 2) Policy Learning in DWL · p. 4

3) Formulating the DWL Loss Function:

The DWL objective combines denoising, policy, and value losses, while its reward design targets commanded velocity, stable gait, and gentle foot contact. The reward uses foot-velocity tracking to provide a signal when contact-force feedback is sparse.

3) Formulating the DWL Loss Function:

  • The total DWL loss is a weighted sum of denoising, policy, and value losses, combining state reconstruction with policy optimization.The policy and value components have weighting factors λπ and λv, respectively.Source: 3) Formulating the DWL Loss Function · p. 4
  • DWL reconstructs state from masking loss and domain-randomization noise, using the learned representation for real-world robustness and adaptability.The conclusion describes masking noise and domain-randomization noise as cultivating a robust latent space.Source: 3) Formulating the DWL Loss Function · p. 4

B. Reward Formulation

  • The reward guides velocity-command following, stable gait, and gentle contact to support locomotion over challenging terrain and above-ground obstacles.Source: B. Reward Formulation · p. 4

1) Composition of Rewards:

  • The reward has four components: velocity tracking, periodic reward, foot-trajectory tracking, and regularization terms.The periodic reward is inspired by prior work [26] [32].Source: 1) Composition of Rewards · p. 4
  • Because contact-force feedback is sparse, the method tracks foot velocity and prescribes foot-trajectory velocities at ground contact to provide a consistent reward signal.The paper says this promotes gentle contacts, reduces impact forces, and enhances sim-to-real transfer.Source: 1) Composition of Rewards · p. 4

2) Quintic Polynomial Foot Trajectory Interpolation:

Quintic interpolation shapes the foot’s swing trajectory using gait-cycle constraints, while the DWL configuration combines sensor inputs and privileged training state to produce a policy for direct robot deployment.

2) Quintic Polynomial Foot Trajectory Interpolation:

  • Quintic polynomial interpolation shapes swing trajectories for smoother motion and precise foot placement, supporting stability and efficient walking.The coefficients control swing height, velocity, and acceleration profiles, with the stated aim of soft landings and reduced impact at footfall.Source: 2) Quintic Polynomial Foot Trajectory… · pp. 4–5
  • The swing trajectory is constrained at key gait-cycle moments, including initial height, velocity, and acceleration, midpoint maximum height, and final height and velocity.A swing phase in one leg coincides with stance in the other, and together they complete a gait cycle.Source: 2) Quintic Polynomial Foot Trajectory… · p. 5

C. Configuration of DWL training process

  • DWL uses a GRU encoder and two-layer MLP decoder and actor networks to encode and decode the world-model representation and produce actions.The framework diagram describes noisy observations being encoded to a latent state and decoded to reconstruct the true state.Source: C. Configuration of DWL training process · p. 5, Figure 3 · p. 5
  • The policy uses proprioceptive data, a periodic clock, and velocity commands, while height scans are privileged critic inputs rather than policy observations.The policy does not use LiDAR or depth-camera information.Source: C. Configuration of DWL training process · p. 6
  • The policy runs at 100 Hz and is optimized in Isaac Gym with virtual motors to model the closed-chain ankle; the resulting policy transfers directly to the physical robot without adjustment.The internal PD controller runs at 500 Hz.Source: C. Configuration of DWL training process · p. 6

D. Domain Randomization

The paper randomizes environment, dynamics, and sensory conditions to address real-world variability, then evaluates the learned approach on challenging terrain using real humanoid robots and baseline variants.

D. Domain Randomization

  • Domain randomization varies environmental conditions, robot dynamics, motor parameters, and system latencies to address real-world variability and disturbances.The stated variability sources include environment noise, dynamics noise, and sensory noise.Source: D. Domain Randomization · p. 6

V. EXPERIMENTS

  • The evaluation applies learned policies in zero-shot real-world transfer on XBot-S and XBot-L across challenging terrains.The benchmark comparisons use the 1.2-meter XBot-S; the study also deploys on the 1.65-meter XBot-L.Source: A. Benchmark Comparison · p. 6, §V · p. 6

A. Benchmark Comparison

  • The authors report the world’s first humanoid robot to robustly navigate complex environments using end-to-end reinforcement learning.Their benchmark comparison uses the smaller XBot-S robot for consistency.Source: A. Benchmark Comparison · p. 7
  • The benchmark compares DWL with active ankle control against PPO with ankle control and DWL with passive ankle control.The scenarios include snowy ground, stairs, and disturbance rejection, with the robot’s arms held stationary to isolate locomotion.Source: A. Benchmark Comparison · p. 7

B. Indoor Experimental Validation

Indoor validation covers slopes, stairs, and irregular terrain; the reported comparisons show DWL’s strongest advantage on irregular ground and stair ascent.

B. Indoor Experimental Validation

  • The terrain tasks included a slope with gradient 0.25, stairs 20cm wide and 10cm high, and irregular terrain with variable elevations up to 10cm.The staircase tasks included both ascent and descent.Source: B. Indoor Experimental Validation · p. 7
  • DWL achieves 100% success across the tested terrains, and distinctly outperforms PPO on irregular terrain and stair climbing.PPO and passive-ankle DWL also show competent success on slopes and stair descent.Source: B. Indoor Experimental Validation · pp. 7–8

C. Outdoor Experiments

Outdoor and robustness tests show stable locomotion across varied terrain and added loads, while the result analysis describes state predictions during slope-and-stair traversal.

C. Outdoor Experiments

  • DWL maintained stable walking on cement, brick, soil, and snowy terrain, including prolonged walks on snow that can sink and cause slipping.The authors note that snow is difficult to simulate.Source: C. Outdoor Experiments · p. 8

D. Robustness Testing

  • Domain randomization may be limited by the sim-to-real gap and the impracticality of accounting for every possible real-world scenario.Source: D. Robustness Testing · p. 8
  • The framework predicts the true state so the controller can rapidly adapt to situations such as a tendency to fall.Source: D. Robustness Testing · p. 8

1) Mass Displacement:

  • The robot sustained stable locomotion while carrying up to 15kg of added weight, over a third of its own weight.Source: 1) Mass Displacement · p. 8

2) Heavy load:

  • With additional loads up to 20kg, the robot walked with slightly lowered height and pushed a cart loaded with 60kg.Source: 2) Heavy load · p. 8

3) Push Recovery:

  • Under external forces, the robot maintained its stance on both flat and sloped terrain during continuous standing commands.Source: 3) Push Recovery · p. 8

VI. RESULT ANALYSIS

  • The Fig. 5 sequence depicts predicted foot contact, base velocity, and heightmap as the robot traverses a slope and stairs.Source: Figure 5 · p. 8

1) Terrain Height Scan Prediction and Gait Adaptation:

Real-world experiments covered snowy land, stairs, deformable ground, and irregular surfaces. The analysis describes terrain, contact, and velocity estimation, and emphasizes active ankle control.

1) Terrain Height Scan Prediction and Gait Adaptation:

  • Using only proprioceptive inputs, DWL estimates rough terrain profiles and distinguishes uphill slopes from descending stairs, prompting gait adjustments despite difficult precise shape prediction.Source: 1) Terrain Height Scan Prediction and… · p. 8

2) Foot Contact Detection and Its Implications:

3) Velocity Estimation:

4) Benefits and Importance of Ankle Control:

  • Active ankle control supports balance in single-leg support and on irregular or deformable terrain; without it, the robot failed to recover from foot deformation on uneven ground.Source: 4) Benefits and Importance of Ankle… · p. 9

VII. CONCLUSION

  • DWL masks privileged information, adds noise to simulated true states, and uses an autoencoder to denoise observations and reconstruct the true state.Source: §VII · p. 9
  • Real-world experiments succeeded on snowy land, stairs, deformable ground, and irregular surfaces; analysis highlights state estimation and active 2-DoF ankle control.Source: §VII · p. 9
  • The authors identify adding visual information for more efficient navigation in challenging terrain while maintaining robustness as future work.Source: §VII · p. 9

APPENDIX

The appendix describes a smooth quintic-polynomial foot trajectory and a periodic reward that distinguishes swing from stance. The reward combines phase-specific foot signals with other weighted components.

A. Quintic Polynomial Foot Trajectory

B. Reward Function

  • The periodic reward promotes higher foot velocities during swing and reasonable foot forces during stance using a binary periodic stance mask.The mask alternates between planned contact and swing phases for each leg across a locomotion cycle.Source: B. Reward Function · p. 12, Figure 8 · p. 12
  • Foot forces and velocities are scaled and clipped to account for their differing magnitudes; forces are divided by 400 and clipped to [0, 1].The equation summary omits scaling and clipping factors typically applied to forces and velocities.Source: B. Reward Function · p. 12
  • The total reward is a weighted sum of individual reward components, while CMDz,γ,β are held at zero because they are not controlled.Each component ri is weighted by µi.Source: B. Reward Function · p. 12

C. Training Details

Training details identify the DWL and PPO network architectures and DWL hyperparameters. Additional experiments report IMU-drift mitigation and prediction errors for yaw and forward velocity.

C. Training Details

D. Additional Experimental Results

  • Figure 9 presents additional experiment setups and results, without specifying their outcomes in the accompanying text.The appendix introduces these demonstrations as further experimental results.Source: D. Additional Experimental Results · p. 12, Figure 9 · p. 13
  • Approximately 87% of IMU drift is mitigated by DWL’s prediction of real IMU data, even when the robot is stationary.Fig. 10 contrasts actual IMU drift with the algorithm’s predicted readings.Source: Figure 10 · p. 13
  • DWL’s predicted Euler yaw angle has MSE 0.074, and its predicted forward velocity has MSE 0.046 in MuJoCo.The yaw evaluation varies command input every 1,000 steps (10 seconds).Source: Figure 11 · p. 14, Figure 12 · p. 14
Loading 2408.14472v1…