PaperlayerView exact revision on arXiv

Source-linked AI summary

BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds

Huayi Wang, Zirui Wang, Junli Ren, Qingwei Ben, Tao Huang, Weinan Zhang, Jiangmiao Pang

arXiv:2502.10363v3cs.ROcs.AIcs.LG

TL;DR

Humanoid robots must traverse risky sparse-foothold terrains despite difficult polygonal-foot placement, sparse rewards, and inefficient exploration. BEAMDOJO combines a polygonal-foot sampling reward, double critic, two-stage RL, and LiDAR mapping; experiments report agile real-world locomotion and an 80% zero-shot sim-to-real success rate.

  • Problem

    Humanoid locomotion on risky sparse-foothold terrains requires precise placement and stability, but polygonal feet, sparse foothold rewards, and early termination make learning difficult.

  • Method

    BEAMDOJO combines a sampling-based reward for polygonal feet, separate critics for dense and sparse rewards, two-stage terrain-aware RL, and LiDAR-based elevation mapping.

  • Results

    80% zero-shot sim-to-real transfer success rate was reported, with agile and robust humanoid locomotion on sparse footholds in simulation and real-world experiments.

  • Takeaways & Limitations

    BEAMDOJO enables precise foothold placement, agile locomotion, stable walking under substantial disturbances, and backward walking using LiDAR-based mapping.

  • Takeaways & Limitations

    Performance is constrained by LiDAR odometry inaccuracies, jitter, map drift, and difficulty handling sudden terrain changes and significant foothold height variations.

Abstract

from arXiv · show

Traversing risky terrains with sparse footholds poses a significant challenge for humanoid robots, requiring precise foot placements and stable locomotion. Existing learning-based approaches often struggle on such complex terrains due to sparse foothold rewards and inefficient learning processes. To address these challenges, we introduce BeamDojo, a reinforcement learning (RL) framework designed for enabling agile humanoid locomotion on sparse footholds. BeamDojo begins by introducing a sampling-based foothold reward tailored for polygonal feet, along with a double critic to balancing the learning process between dense locomotion rewards and sparse foothold rewards. To encourage sufficient trial-and-error exploration, BeamDojo incorporates a two-stage RL approach: the first stage relaxes the terrain dynamics by training the humanoid on flat terrain while providing it with task-terrain perceptive observations, and the second stage fine-tunes the policy on the actual task terrain. Moreover, we implement a onboard LiDAR-based elevation map to enable real-world deployment. Extensive simulation and real-world experiments demonstrate that BeamDojo achieves efficient learning in simulation and enables agile locomotion with precise foot placement on sparse footholds in the real world, maintaining a high success rate even under significant external disturbances.

I. INTRODUCTION

BEAMDOJO addresses humanoid locomotion on risky terrains with sparse footholds, where polygonal feet, sparse rewards, and inefficient exploration complicate learning. It combines a polygonal-foot foothold reward, double critic, two-stage RL, and LiDAR-based mapping, achieving agile real-world traversal with an 80% zero-shot sim-to-real success rate.

  • Sparse foothold terrains require humanoid robots to process perception, place feet precisely within safe areas, and maintain base stability.
  • Polygonal humanoid feet make point-foot foothold rewards unsuitable and impose additional planning challenges compared with point-modeled feet.
  • Sparse foot-placement rewards and early termination after missteps make credit assignment and trial-and-error exploration difficult.
  • BEAMDOJO combines a sampling-based polygonal-foot foothold reward, double critic, and two-stage RL to train terrain-aware humanoid locomotion.The first stage trains on flat terrain with task-terrain perceptive observations, while the second fine-tunes on the true task terrain.
  • 80% zero-shot sim-to-real transfer success rate was reported in extensive simulation and Unitree G1 experiments.The experiments demonstrate agile and robust locomotion on sparse footholds.
  • A LiDAR-based elevation map supports agile humanoid locomotion over risky terrains and smooth movement in both forward and backward directions.

B. Reinforcement Learning in Locomotion Control

BeamDojo formulates terrain-aware humanoid locomotion as partially observable RL and addresses sparse foothold feedback through polygon-aware rewards and a double critic. The framework separates dense locomotion and sparse foothold learning before synthesizing their normalized advantages for PPO policy updates.

  • Problem formulation: BeamDojo models terrain-aware humanoid locomotion as a partially observable reinforcement-learning problem because sensory limitations and environmental noise obscure the true state.The policy observes incomplete information rather than the full underlying state.
  • Foothold reward: The sampling-based foothold reward evaluates polygonal-foot placement by sampling sole points and penalizing samples whose terrain height falls below a depth tolerance.Contact indicators identify feet in contact, while the tolerance distinguishes low terrain height associated with placement outside safe areas.
  • Double critic: A double critic separately estimates value functions for dense locomotion rewards and the sparse task-specific foothold reward.This decoupling addresses the disparity between dense gait-regularization feedback and sparse foothold feedback.
  • Double critic: Generalized Advantage Estimation computes separate advantages, which are individually normalized and combined using component weights into the overall advantage for policy optimization.The combined advantage is then used in the PPO update, whose probability ratio is clipped by a hyperparameter.
  • Training organization: The two-stage simulation design decouples observations and reward groups initially, with flat-terrain locomotion information and task-terrain foothold feedback, before using the full set on task terrain.The overview describes the double critic as separately learning the two reward groups and stage two as fine-tuning the policy on the task terrain.

C. Learning Terrain-Aware Locomotion via Two-Stage RL

BeamDojo uses two-stage RL to promote exploration and terrain-aware locomotion on complex footholds. It first trains on flat terrain with task-terrain height-map observations and nonterminating missteps, then fine-tunes on real task terrains where missteps terminate episodes.

  • Soft terrain dynamics constraints: The first stage trains the humanoid on flat terrain while providing the true task terrain’s height map as perceptive input.Missteps are penalized without terminating episodes, encouraging broad trial-and-error exploration.
  • Hard terrain dynamics constraints: The second stage trains on the real task terrains in simulation, where missteps cause termination and the policy is fine-tuned for accurate foothold stepping.This stage follows the exploratory first stage with hard terrain dynamics constraints.

1) Stage 1: Soft Terrain Dynamics Constraints Learning:

Stage 1 trains the humanoid on safer flat terrain while exposing it to the true terrain’s perception and foothold rewards, enabling exploration without immediate termination from missteps.

  • Stage 1: Soft Terrain Dynamics Constraints Learning: The robot traverses flat terrain F while receiving proprioceptive observations and the elevation map of corresponding task terrain T.This lets the robot imagine the true terrain while physically avoiding termination from missteps.
  • Stage 1: Soft Terrain Dynamics Constraints Learning: Flat terrain supplies locomotion rewards, whereas task terrain supplies perceptive information and the foothold reward.The two reward components are trained separately with a double critic.
  • Stage 1: Soft Terrain Dynamics Constraints Learning: Missteps are penalized instead of terminating episodes, allowing continuous foothold-placement attempts and improving sampling efficiency.Conventional early termination disrupts trajectories and makes successful foothold samples difficult to obtain.
  • Stage 1: Soft Terrain Dynamics Constraints Learning: Stage 2 fine-tunes the policy directly on task terrain T, where missteps immediately terminate episodes and enforce precise, safe locomotion.The double critic continues optimizing locomotion and foothold rewards.
  • Stage 1: Soft Terrain Dynamics Constraints Learning: The policy uses commands, proprioceptive observations, perceptive elevation maps, and previous actions to produce lower-body joint-position targets.The actor outputs 12 joint targets, while a PD controller converts them into torques.

2) Terrain and Curriculum Design:

BeamDojo trains across progressively harder sparse-foothold terrains, beginning with broad stone layouts and fine-tuning on stepping stones and balancing beams before evaluation on additional terrains.

  • Terrain and Curriculum Design: Five sparse-foothold terrain types are designed for training and evaluation: Stones Everywhere, Stepping Stones, Balancing Beams, Stepping Beams, and Gaps.The terrain set spans scattered footholds, aligned stones, beams, randomly distributed stepping beams, and gaps requiring large steps.
  • Terrain and Curriculum Design: Figure 4 orders training terrains from simple to difficult, using (a) for Stage 1, (b) and (c) for Stage 2, and (b)-(e) for evaluation.The caption maps terrain panels to training stages and evaluation.
  • Terrain and Curriculum Design: Stage 1 trains on Stones Everywhere with soft constraints, while Stage 2 fine-tunes on Stepping Stones and Balancing Beams with hard constraints.Evaluation additionally uses Stepping Beams and Gaps.
  • Terrain and Curriculum Design: Zero-shot transfer succeeds on Stepping Beams and Gaps despite training exclusively on the first three terrains.These terrains test generalization beyond the training distribution.
  • Terrain and Curriculum Design: The curriculum advances after three consecutive successful traversals of a terrain level and does not regress before all levels are passed.This preserves exposure to difficult terrain levels once reached.

3) Sim-to-Real Transfer:

BeamDojo supports real-world deployment by combining domain randomization with onboard LiDAR-based elevation mapping and separate update rates for perception, policy, and actuation.

  • Sim-to-Real Transfer: Domain randomization perturbs observations, humanoid properties, terrain dynamics, and elevation-map measurements to improve robustness and sim-to-real transfer.Map noise models vertical measurement errors, odometry-related rotations, foothold extension, and repeated maps.
  • Sim-to-Real Transfer: The Unitree G1 uses a Jetson Orin NX and Livox Mid-360 LiDAR supplying IMU data and feature points for perception.The robot weighs 35 kg, stands 1.32 m tall, and has 23 actuated degrees of freedom.
  • Sim-to-Real Transfer: FAST-LIO fuses LiDAR feature points with IMU data to generate odometry for a robot-centric elevation map.The map addresses occlusion and noise in raw LiDAR point clouds.
  • Sim-to-Real Transfer: The elevation map updates at 10 Hz, the learned policy runs at 50 Hz, and the PD controller runs at 500 Hz.This hierarchy supports smooth and precise actuation during deployment.

A. Experimental Setup

The simulation study compares BeamDojo with baseline and ablation methods across sparse-foothold terrains using success, traversal, and foothold-error metrics, finding stronger performance and learning efficiency for BeamDojo.

  • A. Experimental Setup: BeamDojo is compared with PIM, a naive foothold-reward method, and ablations removing soft dynamics or the double critic.All methods are adapted to two-stage evaluation settings, with differing Stage 1 terrain constraints.
  • A. Experimental Setup: Evaluation covers Stepping Stones, Balancing Beams, Stepping Beams, and Gaps at medium and hard difficulty levels.Three differently seeded policies are evaluated over 100 random episodes per setting.
  • A. Experimental Setup: The study measures success rate, traverse rate, and foothold error, defined respectively as full-terrain success, traveled-distance fraction, and outside-foothold sample proportion.Traverse rate uses the total terrain length of 8 m.
  • Quantitative results: BeamDojo consistently outperforms single-stage approaches and ablations, achieving high success rates and low foothold errors across challenging terrains.The naive implementation is almost incapable of traversing stepping stones and balancing beams at hard difficulty.
  • Quantitative results: BeamDojo reaches higher terrain levels faster, with both two-stage training and the double critic improving learning efficiency.The two-stage setup contributes the most, while the naive method struggles to reach higher levels.
  • Quantitative results: BeamDojo demonstrates zero-shot generalization on Stepping Beams and Gaps despite not being explicitly trained on them.Existing humanoid controllers struggle to adapt to risky terrains with fine-grained footholds.

2) Detailed Ablation Analysis:

The ablation studies show that the double critic and two-stage training improve learning efficiency, gait regularization, foot placement planning, and foothold accuracy. The double critic separates sparse foothold learning from dense locomotion signals, while two-stage training supports broader trial-and-error exploration.

  • Foot Placement Accuracy: BEAMDOJO achieves low foothold errors, while the naive implementation more often places feet outside safe foothold areas.The comparison attributes much of the accuracy improvement to the double critic.
  • Learning Efficiency: BEAMDOJO converges significantly faster than the compared designs, with two-stage training contributing more to learning efficiency than the double critic.All designs were trained for 10,000 iterations in both stages to ensure convergence.
  • Learning Efficiency: Two-stage training enables continued foot-placement attempts after missteps, while the double critic keeps foothold updates separate from noisy unstable-locomotion signals.This separation is especially relevant during early training, when locomotion signals are unstable.
  • Gait Regularization: The double critic improves motion smoothness and feet clearance compared with naive and no-double-critic designs by independently normalizing dense and sparse reward advantages.The compared designs show poorer smoothness and feet air time when gait regularization is combined with sparse foothold rewards without this separation.
  • Foot Placement Planning: BEAMDOJO produces smoother foot-placement planning, whereas the no-double-critic baseline makes most adjustments only near the target foothold.The visualization covers the full foot-lifting and landing subprocess.

C. Real-world Experiments

Real-world experiments evaluate BEAMDOJO across sparse foothold terrains, speed commands, payloads, external pushes, and missteps. The robot transfers zero-shot to deployment, generalizes to unseen terrain types, and maintains locomotion under substantial disturbances.

  • Terrain Transfer: BEAMDOJO achieves a high real-world success rate and generalizes to Stepping Beams and Gaps, although neither terrain was included in training.Five trials were conducted on each terrain for the deployment comparison.
  • Terrain Transfer: Removing height-map domain randomization significantly lowers success rate, highlighting its importance for sim-to-real transfer.The comparison is between BEAMDOJO and the “ours w/o HR” ablation.
  • Agility: At the highest training command velocity of 1.0 m/s, the robot reaches an average speed of 0.88 m/s with minimal tracking error.Performance degrades significantly beyond 1.25 m/s on the challenging terrain.
  • Experimental Terrains: The real-world tests cover stepping stones, balancing beams, stepping beams, and gaps with specified stone or beam sizes, distances, and sparsity levels.The stepping-stone terrain uses 20 cm stones and a maximum 45 cm separation; gaps reach 50 cm.
  • Robustness: The robot maintains agile locomotion and precise foot placements with a 10 kg payload, recovers from external pushes, and recovers from occlusion-induced missteps.The external-force test includes transition to single-leg support and recovery to stable two-leg support.

D. Extensive Studies and Analysis

Additional studies examine foothold reward design, terrain curriculum, and heading commands. Continuous foothold rewards improve placement accuracy, curriculum supports performance across difficulties, and heading-command policies perform poorly under real-world deployment conditions.

  • Foothold Reward Design: The continuous foothold reward enables more accurate foot placements than coarse designs by gradually encouraging greater overlap with safe footholds.The comparison evaluates success rate and foothold error on medium-difficulty stepping stones.
  • Foothold Reward Design: Foothold-50% outperforms foothold-30% and foothold-70%, because the 30% threshold is too strict and 70% is overly loose.These variants apply coarse penalties based on the proportion of sampled points outside the safe area.
  • Terrain Curriculum: The curriculum ablations compare success and traverse rates on medium- and hard-difficulty stepping-stone terrains without curriculum learning.The reported comparison includes “w/o curriculum-medium” and “w/o curriculum-hard” designs.
  • Heading Command: The no-heading-command deployment achieves 4/5 success, compared with 1/5 for the heading-command design on real-world stepping stones.The heading-command design is affected by simulation overfitting to commanded angular velocity and noisy real-world odometry.

4) Generalization to Non-Flat Terrains:

BEAMDOJO generalizes beyond sparse foothold terrains to stairs and slopes, but performance declines on extremely small footholds and large step distances. The paper also identifies perception and elevation-variation limitations that constrain broader deployment.

  • Non-Flat Terrain Generalization: BEAMDOJO traverses stairs and slopes with success rates of 8/10 and 10/10, respectively.For these non-flat terrains, the base-height reward is computed relative to foot height, and Stage 1 pre-training is unnecessary because footholds are no longer sparse.
  • Failure Cases: Performance drops sharply on 10 cm stones and 55 cm steps, even with the most challenging training settings.The 10 cm stones are approximately half the foot length, while 55 cm steps are roughly equal to leg length.
  • Failure Cases: Very small footholds and large strides shift the difficulty toward balance and stride execution, which the current reward function does not adequately address.The evaluation varies stone sizes and step distances across policies trained with different terrain parameters.
  • Limitations: LiDAR odometry inaccuracies, jitter, map drift, and difficult-to-simulate stone motion constrain rapid adaptation to unexpected terrain changes.The perception module must trade off confidence in noisy measurements against dynamic terrain changes.
  • Limitations: The method does not yet fully exploit elevation-map information or adequately handle terrains with significant foothold height variations.The authors identify broader terrain generalization and elevation changes as future directions.
  • Conclusion: The conclusion reports agile, robust traversal on sparse footholds, supported by the polygonal-foot reward, two-stage learning, double critic, and LiDAR-based mapping.The reported scope includes stepping stones, balance beams, gaps, and stable backward walking in real-world scenarios.

APPENDIX

The appendix documents reward-function references, symbols, training-terrain curricula, domain-randomization settings, and hyperparameters used by BeamDojo.

  • Reward and symbol references: Training uses reward functions drawn mainly from prior work, with corresponding symbols and descriptions provided separately.
  • Terrain curriculum: The curriculum spans Stones Everywhere, Stepping Stones, and Balancing Beams across difficulty levels 0 through 8.
  • Terrain curriculum: Training terrains include 1.0m-deep gaps, ±0.05m height variations, and a depth tolerance threshold of −0.1m.
  • Terrain curriculum: Stones Everywhere reduces stone size with difficulty, while stone distance increases according to the curriculum level.
  • Terrain curriculum: Stepping Stones use progressively smaller stones and a maximum stone distance of 0.1 + 0.05l.
  • Terrain curriculum: Balancing Beams vary stone size and spacing by difficulty, reaching a single continuous beam at the highest level.
  • Configuration: Domain-randomization settings and training hyperparameters are provided in Tables IX and X.
Loading 2502.10363v3…