PaperlayerView exact revision on arXiv

Source-linked AI summary

Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds

Min Dai, William D. Compton, Junheng Li, Lizhi Yang, Aaron D. Ames

arXiv:2601.06286v1cs.RO

TL;DR

Constrained foothold locomotion requires precise placement and timing, but model-based methods are sensitive to perception while end-to-end RL struggles to discover such motions. The paper combines a reduced-order stepping planner with CLF-guided teacher-student RL, achieving hardware-validated stepping-stone locomotion with improved reliability over a model-free baseline.

  • Problem

    Constrained foothold locomotion requires precise contact timing and placement, while model-based control is brittle to uncertainty and RL struggles to discover precise constrained motions.

  • Method

    A reduced-order, terrain-aware stepping planner generates dynamically consistent foot-placement, CoM, and timing references that guide teacher-student RL through CLF-based rewards.

  • Results

    100% success was achieved on challenging flat stepping stones versus 56% for the end-to-end RL baseline, with additional simulation and hardware validation.

  • Takeaways & Limitations

    Model-guided references support precise stepping strategies while retaining transfer across simulators and physical hardware.

Abstract

from arXiv · show

Bipedal humanoid robots must precisely coordinate balance, timing, and contact decisions when locomoting on constrained footholds such as stepping stones, beams, and planks -- even minor errors can lead to catastrophic failure. Classical optimization and control pipelines handle these constraints well but depend on highly accurate mathematical representations of terrain geometry, making them prone to error when perception is noisy or incomplete. Meanwhile, reinforcement learning has shown strong resilience to disturbances and modeling errors, yet end-to-end policies rarely discover the precise foothold placement and step sequencing required for discontinuous terrain. These contrasting limitations motivate approaches that guide learning with physics-based structure rather than relying purely on reward shaping. In this work, we introduce a locomotion framework in which a reduced-order stepping planner supplies dynamically consistent motion targets that steer the RL training process via Control Lyapunov Function (CLF) rewards. This combination of structured footstep planning and data-driven adaptation produces accurate, agile, and hardware-validated stepping-stone locomotion on a humanoid robot, substantially improving reliability compared to conventional model-free reinforcement-learning baselines.

I. INTRODUCTION

Constrained foothold locomotion demands precise contact timing and placement, exposing a trade-off between model-based precision and RL robustness. The proposed physics-guided framework combines terrain-aware reduced-order planning with CLF-based rewards and is validated in simulation and on hardware.

  • Motivation: Stepping stones impose strict contact-location and timing constraints, so a single misstep or localized slip can cause unrecoverable failure.These conditions are relevant to humanoids operating in disaster zones and cluttered industrial environments.
  • Limitations of Existing Methods: Pure model-based pipelines depend on accurate terrain representations and state feedback, while perception errors can impose invalid constraints and hinder deployment in unstructured environments.Noisy point clouds, segmentation errors, and odometry errors add latency, uncertainty, complexity, and conservatism.
  • Motivation: Model-based control plans feasible motion under strict constraints but is brittle to uncertainty, whereas RL handles uncertainty but struggles to discover precise constrained motions.This contrast motivates guiding RL with structured feasible references rather than relying solely on reward engineering.
  • Approach: The framework integrates a reduced-order, terrain-aware stepping planner with CLF-based rewards to generate terrain-consistent foot placement, CoM, and timing references during RL training.The references are generated with minimal computation overhead for discrete aperiodic terrains.
  • Approach: The approach removes dependence on human motion datasets and retargeting pipelines while providing a physically grounded, controllable, and transferable prior.This design targets multi-environment generalist locomotion in severely constrained settings.
  • Validation: The method is validated on the Unitree G1 across randomized stepping-stone terrains with gap and height variations, showing improved stability, foot placement, generalization, and gains over model-free baselines.The experiments include both numerical and experimental validation.

II. PRELIMINARIES

The preliminaries model bipedal locomotion as hybrid dynamics and use a passive-ankle Linear Inverted Pendulum to represent underactuated center-of-mass motion. Orbital energy, closed-form trajectories, and impact dynamics provide the basis for viable stepping and timing references.

  • A. Bipedal Robot Hybrid Dynamics: A bipedal robot is represented as a hybrid dynamical system with continuous domains, control inputs, impact guards, discrete resets, and continuous dynamics.The state contains generalized coordinates and velocities, while impacts trigger a reset map.
  • B. Reduced Order Models for Underactuated Dynamics: Bipedal walking is underactuated because the horizontal CoM cannot be arbitrarily regulated, motivating a passive-ankle LIP model for dynamic gait behavior.The model represents varying-height stepping stones as virtual slopes connecting consecutive stone centers.
  • B. Reduced Order Models for Underactuated Dynamics: The LIP state uses CoM position relative to the stance foot and mass-normalized angular momentum, with momentum generated by gravity-induced torques during single support.The CoM height is assumed constant relative to the virtual slope.
  • B. Reduced Order Models for Underactuated Dynamics: Orbital energy partitions motion into viable trajectories: E > 0 supports forward progression past midstance, while E < 0 indicates bounded oscillations.A target E* is selected to ensure forward progression across stepping stones.
  • B. Reduced Order Models for Underactuated Dynamics: The closed-form CoM trajectory is used to compute step duration T for reaching the next stone’s target switching state, enabling timing regulation over non-uniform footholds.At impact, the swing foot becomes the stance foot and angular momentum is reset under instantaneous, inelastic, no-slip assumptions.

C. Nonlinear Output Dynamics and Control

The framework uses controllable outputs and a Control Lyapunov Function to promote exponential tracking, embedding the resulting CLF condition directly into RL rewards while respecting robot constraints.

  • The controllable outputs include center-of-mass position, torso orientation, and swing-foot pose.
  • A Control Lyapunov Function is defined to ensure exponential convergence of these outputs.
  • The CLF decreasing condition is typically enforced through a quadratic program that computes torque inputs under dynamics, torque, and contact constraints.
  • In this work, the CLF is embedded directly into reinforcement-learning training as a reward.

D. Reinforcement Learning

Sparse foothold traversal is formulated as a partially observable decision problem and optimized with PPO using a teacher-student pipeline that transfers privileged-information behavior to an deployable student policy.

  • Sparse foothold traversal is modeled as a POMDP with states, actions, observations, transitions, observation likelihoods, rewards, and a discount factor.
  • PPO optimizes both teacher and student policies while constraining policy updates through a clipped surrogate objective.
  • The teacher accesses privileged information such as ground-truth terrain and contact phase, whereas the student acts from non-privileged observations.
  • The student first matches teacher actions through supervised behavior cloning, then is fine-tuned with PPO for partial observability and noise robustness.

III. MODEL-BASED REFERENCE GENERATION

A reduced-order planner generates dynamically and environmentally consistent references for stepping-stone locomotion, including timing, momentum targets, smooth center-of-mass motion, posture, and contact constraints.

  • The planner generates per-step references for foot placement, center-of-mass trajectory, and step timing using a reduced-order walking model.
  • Step timing is dynamically updated, and the planner regulates post-impact sagittal orbital energy toward E*=0.6 through the desired pre-impact vertical center-of-mass velocity.
  • The desired vertical velocity is computed from momentum transfer and the required angular-momentum change associated with the stone configuration.
  • Closed-form cubic splines provide smooth, time-normalized center-of-mass references across different step durations.
  • Terrain heading sets pelvis and swing-foot orientation references, while synchronized arm motion supports angular-momentum cancellation and natural gait dynamics.
  • The generated trajectories include desired and actual outputs plus stance-foot holonomic constraints used for CLF rewards and constraint terms during RL training.

IV. LEARNING ENVIRONMENT SETUP

The training framework is hierarchical: a reduced-order planner supplies timing and dynamic targets, while an RL controller learns full-body residual dynamics for robust target tracking.

  • The reduced-order model acts as a high-level stepping-stone planner that dictates step timing and dynamic targets.
  • The RL policy operates as a low-level, high-frequency controller learning residual full-body dynamics on the high-degree-of-freedom humanoid.
  • The hybrid design combines constraint-consistent references with RL handling of unmodeled friction and compliance while minimizing tracking errors.
  • The approach is described as improving sample efficiency compared with model-free baselines that struggle to discover precise stepping strategies from scratch.

A. Terrain Generation

The framework procedurally generates diverse stepping-stone and stair terrains, then increases geometric difficulty through curriculum-based sampling. This exposes policies to widening gaps, changing elevations, and randomized support platforms.

  • Four terrain types—upstairs, downstairs, flat stepping stones, and height-varying stepping stones—vary in sagittal spacing, vertical offsets, and sequence periodicity.
  • Terrain difficulty expands gap widths and positive or negative stone-height offsets while retaining a unified progression across terrain families.
  • Gap distance ranges from 0.3 to 0.3 + 0.4d meters, where d denotes terrain difficulty.
  • Stone dimensions are randomized with x values from 0.13 to 0.3 m and z values from 0.75 to 1.25 m.
  • Stair depth spans 0.2–0.3 m, while absolute stair height reaches 0.2d m across 10 discrete terrain levels.
  • The curriculum advances terrain level after three consecutive successful traversals and never regresses after progression.

B. Training Framework

Training combines privileged CLF-guided teacher learning, behavior-cloning distillation, and noisy student fine-tuning. The student uses deployable sensing while learning to track model-guided foothold and center-of-mass targets.

  • The CLF reward is phase-dependent, but phase information is unavailable at deployment and a standard asymmetric actor-critic failed to converge on stepping-stone locomotion.
  • Teacher training first uses privileged phase and CLF information, then behavior cloning transfers stable walking competence to the student before curriculum fine-tuning.
  • The student observes commands, proprioception, and a local 1m×1m heightmap sampled at 0.1m grid resolution.
  • The proposed method achieves better CLF tracking than Fixed ˙zcom and Fixed T policies, while a naive policy cannot handle harder terrains.
  • Actions specify desired angles for 21 joints, tracked by a low-level PD controller with zero desired joint velocities.
  • After noise-free teacher training and distillation, reinforcement-learning fine-tuning injects observation noise and removes teacher supervision and privileged inputs.

V. RESULTS

The model-guided policies learn more reliably than the end-to-end baseline and transfer across challenging simulated terrains. Ablations reveal that adaptive timing and vertical center-of-mass regulation support stable, transferable gait behavior.

  • Quantitative Results: The proposed policy and its ablations learn consistently and rapidly, whereas the end-to-end baseline improves only on flat stepping stones and saturates at a lower curriculum level.
  • Quantitative Results: 100% success on challenging flat stepping stones compares with 56% for the end-to-end RL baseline across 4096 environments.
  • Quantitative Results: The full method achieves the highest CLF reward; removing vertical COM regulation lowers performance, while fixed step timing causes slower improvement and lower steady-state reward.
  • Quantitative Results: The proposed method outperforms both ablations on the most challenging height-varying terrain, while all model-guided approaches perform highly in simpler settings.
  • Simulation Evaluation: In MuJoCo, the proposed policy traverses mixed courses containing stairs, varied-height stones, sparse narrow footholds, and an out-of-distribution sequence.
  • Simulation Evaluation: Ablated policies show reduced sim-to-sim reliability, including inaccurate foot placement and undesirable flight phases for fixed T and y-direction drift for fixed ˙zcom.
  • Simulation Evaluation: The proposed policy withstands unknown body pushes of ±100Nm applied for 0.2s during stepping-stone traversal.

C. Real-World Transfer

The controller transfers to hardware with stable, repeatable stepping and generalizes zero-shot to an unseen mixed stepping-stone course. The framework combines reduced-order planning with model-guided RL to support constrained-terrain locomotion.

  • Real-World Transfer: Hardware rollouts produced a stable and repeatable stepping pattern with consistent swing trajectories and smooth motion.The deployment used motion capture and elevation maps built from a terrain mesh matching the physical testbed.
  • Real-World Transfer: The policy successfully generalized to irregular stone depths in a mixed course absent from its training curriculum.This distribution shift tested performance beyond the standard evaluation configurations.
  • Real-World Transfer: The model-guided references helped prevent overfitting to fixed geometric templates, enabling zero-shot generalization to unstructured terrain topologies.The evidence comes from successful performance on the unseen mixed stepping-stone configuration.
  • Framework: The framework integrates a reduced-order stepping planner into RL training to bridge model-based precision and data-driven robustness on constrained terrains.The planner generates dynamically consistent references that guide the policy toward feasible motion manifolds without over-constraining learning.
  • Real-World Transfer: Successful zero-shot transfer to a disparate physics simulator and physical hardware highlights the framework’s effectiveness and robustness.The authors identify deployment in complex, unstructured environments as a promising direction.

APPENDIX

The appendix summarizes training configurations for PPO training, PPO fine-tuning, and student–teacher distillation.

  • APPENDIX: The appendix table summarizes configurations used for PPO training, PPO fine-tuning, and student–teacher distillation.
Loading 2601.06286v1…