PaperlayerView exact revision on arXiv

Source-linked AI summary

Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer

Xinyang Gu, Yen-Jen Wang, Jianyu Chen

arXiv:2404.05695v2cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Humanoid locomotion faces a difficult sim-to-real transition, with few open-source resources. Humanoid-Gym trains policies in Isaac Gym and validates them across simulations and on two robot sizes; the reported MuJoCo calibration closely matches real-world dynamics and performance.

  • Problem

    Humanoid locomotion has a challenging sim-to-real gap, while open-source robot-learning resources for this transition remain scarce.

    Source: §I · p. 1, A. Robot Learning on Locomotion Tasks · p. 2

  • Method

    Humanoid-Gym combines Isaac Gym RL training with specialized rewards, domain randomization, and sim-to-sim policy validation in MuJoCo.

    Source: §I · p. 1

  • Results

    Humanoid-Gym achieved zero-shot sim-to-real transfer on two differently sized humanoid robots, and adjusted MuJoCo closely mirrored real-world dynamics and performance.

    Source: §I · p. 1, §V · p. 3

  • Takeaways & Limitations

    Researchers without physical robots can use the adjusted MuJoCo simulation to validate training policies before real-world transfer.

    Source: §V · p. 3

Abstract

from arXiv · show

Humanoid-Gym is an easy-to-use reinforcement learning (RL) framework based on Nvidia Isaac Gym, designed to train locomotion skills for humanoid robots, emphasizing zero-shot transfer from simulation to the real-world environment. Humanoid-Gym also integrates a sim-to-sim framework from Isaac Gym to Mujoco that allows users to verify the trained policies in different physical simulations to ensure the robustness and generalization of the policies. This framework is verified by RobotEra's XBot-S (1.2-meter tall humanoid robot) and XBot-L (1.65-meter tall humanoid robot) in a real-world environment with zero-shot sim-to-real transfer. The project website and source code can be found at: https://sites.google.com/view/humanoid-gym/.

I. INTRODUCTION

Humanoid-Gym is an open-source RL framework for humanoid locomotion that targets zero-shot simulation-to-real transfer and offers sim-to-sim policy validation. It has been tested on two differently sized humanoid robots.

I. INTRODUCTION

  • Zero-shot transfer was tested on RobotEra’s XBot-S (1.2 m) and XBot-L (1.65 m) in real-world environments.The framework uses specialized humanoid rewards and domain randomization to simplify sim-to-real transfer.Source: §I · p. 1
  • The open-source framework trains humanoid locomotion policies in simulation with meticulous system design.Source: §I · p. 1
  • Its sim-to-sim tool lets users test trained policies across physical simulations and environmental dynamics.The framework transfers policies from Isaac Gym to MuJoCo for validation.Source: §I · p. 1

II. RELATED WORKS

Prior work has applied RL to locomotion and humanoid walking, but humanoid control presents a more challenging scenario, and sim-to-real transfer remains difficult. The section identifies a lack of open-source humanoid robot-learning resources.

A. Robot Learning on Locomotion Tasks

  • Prior RL work covers quadrupeds, Cassie, humanoid walking on flat surfaces with transformers, and upper-body humanoid skills.The passage contrasts this prior work with the more challenging control scenario addressed by humanoid robots.Source: A. Robot Learning on Locomotion Tasks · p. 1
  • Humanoid-Gym’s pipeline combines massively parallel Isaac Gym training with terrain diversity, dynamics randomization, and subsequent sim-to-sim testing.The caption reports close performance alignment in MuJoCo and real-world settings after calibration.Source: Figure 2 · p. 2
  • Humanoid sim-to-real transfer remains a significant challenge, and open-source resources for robot learning are scarce.Source: A. Robot Learning on Locomotion Tasks · p. 2

III. METHOD

Humanoid-Gym formulates robot control as a partially observable RL problem and trains policies with PPO, an asymmetric actor-critic, and privileged information. The method uses GAE for advantage estimation.

III. METHOD

  • The method section introduces Humanoid-Gym’s problem setting, system design, and reward design.Source: §III · p. 2

A. Reinforcement Learning For Robot Control

B. System design

The system represents gait timing and foot-contact patterns explicitly, while the policy uses proprioceptive input, a periodic clock signal, and velocity commands to set joint-position targets. Training and validation use different simulators and control frequencies.

B. System design

  • The gait representation divides each cycle into two double-support and two single-support phases, with a periodic stance mask encoding foot contacts.Reference motion is sinusoidal and includes pitch, knee, and ankle movements.Source: B. System design · p. 2
  • The policy receives proprioceptive data, a periodic clock signal, and velocity commands, and outputs target joint positions for the PD controller.Source: B. System design · p. 2
  • The observation-space table categorizes components as observation or state and lists their dimensions.The state frame also includes foot-contact detection and other privileged observations.Source: Table I · p. 2, B. System design · p. 2
  • The policy runs at 100 Hz and the internal PD controller at 1000 Hz; Isaac Gym is used for training and MuJoCo for sim-to-sim validation.The passage contrasts fast GPU parallel simulation with more accurate but slower CPU simulation.Source: B. System design · p. 2
  • Multi-frame observations and privileged observations are used for locomotion on uneven terrain.Source: B. System design · p. 3

C. Reward Design

The reward design targets commanded-velocity tracking, stable gait, and smooth contact. It combines velocity tracking, gait reward, and regularization terms, with a weighted sum of components.

  • The reward function targets velocity-command adherence, stable gait, and smooth contact.Source: C. Reward Design · p. 3
  • Velocity-mismatch commands CMDz,γ,β are set to zero because they are uncontrolled and should remain zero for stable, smooth walking.Source: C. Reward Design · p. 3
  • The total reward at each time step is a weighted sum of individual reward components.The passage defines each component weight as µi and each component reward as ri.Source: C. Reward Design · p. 3

IV. EXPERIMENTS

The experiments evaluate zero-shot transfer in sim-to-sim and sim-to-real settings and visualize MuJoCo calibration. Real-world validation uses Robot Era’s XBot-S and XBot-L humanoids.

  • The experiments examine zero-shot transfer in both sim-to-sim and sim-to-real scenarios.Source: §IV · p. 3
  • Real-world validation uses Robot Era’s XBot-S and XBot-L humanoids.Source: §IV · p. 3
  • The experiments also visualize MuJoCo calibration to assess sim-to-sim validation.Source: §IV · p. 3

A. Zero-shot Transfer

The trained policies transfer from simulation to real-world settings and are evaluated on flat and uneven terrain. The policies successfully traverse both terrain types, including uneven terrain that differs substantially from training.

B. Calibration for MuJoCo

After calibration, MuJoCo trajectories closely match real-world trajectories, and its dynamics are closer to the real environment than Isaac Gym’s. The comparisons use sine-wave trajectories and phase portraits.

V. CONCLUSIONS

Humanoid-Gym supports zero-shot sim-to-sim and sim-to-real transfer for two humanoid sizes using a specialized reward function. The adjusted MuJoCo simulation closely mirrors real-world dynamics and performance, enabling policy validation without a physical robot.

V. CONCLUSIONS

  • Humanoid-Gym supports zero-shot sim-to-sim and sim-to-real transfer for humanoid robots of two distinct sizes through a specialized reward function.The reward function is tailored for humanoid robotics.Source: §V · p. 3
  • The adjusted MuJoCo simulation closely mirrors the dynamics and performance of the real-world environment.Source: §V · p. 3
  • Researchers without physical robots can use sim-to-sim validation of policies, which the authors say significantly enhances the potential for successful sim-to-real transfers.Source: §V · p. 3

APPENDIX

The appendix presents the robot platforms, stance-mask and terrain figures, and tables covering hyperparameters, domain randomization, and reward-function details.

APPENDIX

  • Fig. 5 shows Humanoid-Gym tested on Robot Era’s XBot-S and XBot-L, two distinct humanoid robot sizes.Source: Figure 5 · p. 5
  • Fig. 6 encodes expected foot contact: 0 marks swing and 1 marks stance for the left and right feet.Source: Figure 6 · p. 5
  • Fig. 7 presents the two MuJoCo validation terrain types: flat planes and uneven terrains.Source: Figure 7 · p. 5
  • Table II is identified as the hyperparameter table.Source: Table II · p. 5
  • Table III describes domain-randomization terms and parameter ranges, distinguishing additive increments from multiplicative scaling.Source: Table III · p. 5
  • Table IV defines the tracking-error metric as ϕ(e, w) := exp(−w · ∥e∥2) and sets the target base height to 0.7 m.Source: Table IV · p. 6
Loading 2404.05695v2…