PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Jonah Siekmann, Kevin Green, John Warila, Alan Fern, Jonathan Hurst

arXiv:2105.08328v1cs.RO

TL;DR

The paper asks whether a human-scale biped can traverse unknown stair-like terrain without fragile external terrain estimation. It adapts sim-to-real reinforcement learning through stair-like terrain randomization, and demonstrates blind stair traversal using proprioceptive feedback, with memory appearing important for robustness.

  • Problem

    The paper addresses the challenge of robustly traversing stairs and stair-like terrain without external perception or accurate terrain models.

  • Method

    The authors modify an existing sim-to-real RL framework by randomizing stair-like terrain during training without adding stair-specific reward terms.

  • Results

    The learned policies are demonstrated in simulation and varied real-world settings, including blind ascent and descent of unknown stairs and other stair-like terrain.

  • Takeaways & Limitations

    Robust proprioceptive bipedal control for complex stair-like terrain can be learned with surprisingly little modification to an existing RL training pipeline.

Abstract

from arXiv · show

Accurate and precise terrain estimation is a difficult problem for robot locomotion in real-world environments. Thus, it is useful to have systems that do not depend on accurate estimation to the point of fragility. In this paper, we explore the limits of such an approach by investigating the problem of traversing stair-like terrain without any external perception or terrain models on a bipedal robot. For such blind bipedal platforms, the problem appears difficult (even for humans) due to the surprise elevation changes. Our main contribution is to show that sim-to-real reinforcement learning (RL) can achieve robust locomotion over stair-like terrain on the bipedal robot Cassie using only proprioceptive feedback. Importantly, this only requires modifying an existing flat-terrain training RL framework to include stair-like terrain randomization, without any changes in reward function. To our knowledge, this is the first controller for a bipedal, human-scale robot capable of reliably traversing a variety of real-world stairs and other stair-like disturbances using only proprioception.

I. INTRODUCTION

The paper investigates whether a bipedal robot can traverse stairs and stair-like terrain without external perception or terrain models. It uses sim-to-real RL with proprioceptive feedback and stair-like terrain randomization to learn robust blind locomotion.

  • Motivation: Bipedal robots need robust stair traversal for operation in human-centric environments, but unlike quadrupeds they rely on dynamic stability.Stair-like terrain includes raised platforms and sudden vertical drops.
  • Motivation: Vision-based stair traversal can be fragile under occlusion, fog, and changing lighting, while integrating vision into high-speed control is technically difficult.The paper therefore asks how robust a blind bipedal robot can be.
  • Motivation: RL can learn proprioceptive reflexes for unexpected contact and rough terrain, but its limits for human-scale bipedal stair traversal were unclear.Unlike optimization or heuristic controllers, RL policies can acquire these responses through varied training disturbances.
  • Approach: The proposed approach modifies an existing RL framework primarily by randomizing upward and downward stairs over contact-plane height, width, and slope.No stair-specific reward terms are required, and the resulting controller handles unknown stairs and stair-like terrain such as logs, curbs, and dropoffs.
  • Approach: Training is performed entirely in simulation with dynamics randomization before deploying the resulting policy on the real robot.This sim-to-real setup avoids the impracticality of learning biped locomotion from scratch through real-world falls and crashes.

A. State Space

The policy state combines proprioceptive physical measurements, randomized human commands, and cyclic leg-clock inputs. The clock inputs encode opposing leg phases for gait coordination.

  • State components: The state includes pelvis orientation, pelvis angular velocity, joint positions, and joint velocities as instantaneous physical information.These measurements form the proprioceptive physical-state component of the policy input.
  • State components: Human command inputs are randomized during training to expose policies to varied traversal speeds and approach angles.Each command is altered with probability 1/300 at each timestep and resampled uniformly when altered.
  • State components: Two cyclic clock inputs correspond to the robot’s legs and track gait phase.The phase variable increments from 0 to 1 and then rolls over to 0.
  • State components: Phase offsets of 0.0 and 0.5 keep the left and right legs diametrically opposite in phase during locomotion.This supplies the alternating timing structure used by the bipedal gait.

B. Action Space

The controller outputs joint-level PD targets and a clock delta that can regulate gait frequency. The framework was adapted from prior work with minimal changes, while the reward retains its general foot-force and foot-velocity structure.

  • Action representation: At 40Hz, the policy outputs an 11-dimensional action vector, with the first 10 entries serving as joint PD targets.Each target is sent to a joint PD controller operating at 2KHz.
  • Training changes: The stair-capable controller was obtained after ablation showed that seemingly necessary extra reward functions and state features were unnecessary.The key adaptation was terrain randomization rather than a redesigned action or reward formulation.
  • Action representation: The final action dimension is a clock delta that lets the controller regulate stepping frequency for the command, gait, and terrain.The policy can choose a suitable stepping frequency rather than directly issuing higher-rate actuation commands.
  • Action representation: The clock delta is bounded to regulate the gait cycle between 0.5x and 1.5x of the nominal stepping frequency.The nominal gait is approximately one cycle every 0.7 seconds, and the authors report that the learned policy varies the delta little during disturbances.
  • Reward structure: The reward framework penalizes foot forces and foot velocities at selected gait intervals while permitting them to be large at other times.This structure encourages lifting the foot when force is penalized and appropriate contact when velocity is penalized.

D. Dynamics Randomization

The sim-to-real setup randomizes dynamics to reduce simulation overfitting and uses recurrent or feedforward policy representations, with a caveat about an untested hypothesis.

  • The policy is represented with an LSTM recurrent network, while feedforward networks provide a non-memory ablation with approximately matched parameter count.
  • The authors leave the proposed role of memory as a hypothesis because it was not tested on the real robot at submission time.
  • Dynamics randomization samples damping, mass, friction, and encoder offset per rollout, while execution rate varies each timestep to mimic system delay.
  • Stair generation randomizes terrain parameters at each episode start, including stair count, height, length, landing length, and adjacent ground slopes.
  • PPO training includes KL-threshold termination and a mirror loss to discourage asymmetric gaits.

III. TERRAIN RANDOMIZATION

Training randomizes stair-like terrain across geometry and starting conditions, then compares memory, terrain exposure, and proximity information in policy groups. Stair LSTM policies achieve the strongest reported success, while speed, memory, and terrain randomization affect robustness.

  • III. TERRAIN RANDOMIZATION: Stair terrain is randomized over incline, friction, starting position, rise, run, step count, and per-step geometric noise.The randomized stairs contain 1–8 steps, with rises of 10–21 cm and runs of 24–30 cm.
  • III. TERRAIN RANDOMIZATION: Four policy groups isolate terrain randomization, memory, and proximity information: Stair LSTM, Stair FF, Flat Ground LSTM, and Proximity LSTM.Each group contains five policies initialized with different random seeds.
  • 1) Probability of Successfully Ascending and Descending Stairs: The evaluation uses 150 ascent and 150 descent trials on five-step stairs with 17 cm treads and 30 cm depths, counting arrival without falling as success.
  • 1) Probability of Successfully Ascending and Descending Stairs: Stair LSTM has the highest overall success probability, but failures increase at both low and high approach speeds.Low speeds may lack momentum after poor foot placement, whereas high-speed gaits are more dynamic.
  • 1) Probability of Successfully Ascending and Descending Stairs: Flat Ground LSTM has high failure rates for ascent and descent, while Stair FF fails to learn an effective stair strategy despite stair training.The results imply that terrain randomization and memory may both contribute to robustness.
  • 1) Probability of Successfully Ascending and Descending Stairs: Success probability is evaluated against commanded speeds from 0.25 m/s to 1.5 m/s over 150 trials, with separate optimal speeds suggested for Stair LSTM ascent and descent.Stair FF does not attain high performance, and Flat Ground LSTM is virtually unable to climb stairs.

2) Energy Efficiency Comparison:

The comparison evaluates locomotion efficiency using cost of transport (CoT), while showing that stair robustness trades off against flat-ground energy efficiency. A binary stair-presence input can recover some of that efficiency.

  • Energy metric: Cost of transport measures energy used per distance, normalized by robot weight to produce a unitless efficiency metric.The calculation uses motor energy, total mass, gravitational acceleration, and distance traveled.
  • Comparison: Table III compares flat-ground CoT at 1 m/s across Flat Ground LSTM, Stair LSTM, and Proximity LSTM policies over five random seeds.The calculations exclude computation and control-electronics power overhead.
  • Results: Flat Ground LSTM policies learn the most energy-efficient gaits for flat-ground walking.
  • Results: Stair LSTM policies use less efficient flat-ground gaits because they are trained for robustness to stairs.
  • Results: The Proximity LSTM’s binary stair input helps recover some lost efficiency by switching between stair-ready and flat-ground gaits.

B. Behavior Analysis

The behavior analysis examines how the policy handles the first upward or downward step after flat-ground walking, focusing on swing-leg placement and stance-force modulation.

  • B. Behavior Analysis: The analysis studies robot behavior when contacting the first step up or down after walking along flat ground.
  • B. Behavior Analysis: Swing-leg motion is analyzed to understand how the policy places its foot on step ups and step downs.
  • B. Behavior Analysis: After swing-foot contact, the foot’s stance-phase ground force can be modulated to prepare the robot for the terrain.

1) Swing Foot Motion:

Training on randomized stairs substantially changes swing-foot behavior, producing higher clearance, steeper descent, and faster retraction than flat-ground training. The Stair LSTM also limits forward motion near step height and modulates stance forces across step changes.

  • Swing-foot trajectory: The Stair LSTM takes a much higher swing-foot path than the Flat Ground LSTM, increasing clearance for stepping onto large steps.The comparison is performed during locomotion at 1.0 m/s.
  • Swing-foot trajectory: The Stair LSTM swing foot moves forward only 14 cm while near a step-up front face, potentially reducing hard toe stubbing.The authors present this as a hypothesis about why the strategy may prevent forward trips.
  • Leg retraction: The Stair LSTM exhibits faster leg-angle retraction than the Flat Ground LSTM, despite no explicit training incentive for this behavior.The authors observe the behavior emerging after stair training but do not establish that it causes improved stair performance.
  • Ground-reaction-force response: Across 10 cm step-ups and step-downs, nominal peak leg force remains relatively constant while the second ground-reaction-force hump increases on step-downs and decreases on step-ups.Horizontal forces oscillate at the policy-evaluation frequency, which the authors hypothesize reflects body-attitude control.

C. Hardware

The recurrent policies transferred to Cassie and supported robust outdoor locomotion across stairs and unmodeled stair-like terrain. In ten outdoor trials per direction, the selected Stair LSTM achieved 80% ascent success and 100% descent success.

  • Hardware transfer: The recurrent policies transferred to hardware without notable difficulties and enabled repeatable stair ascents and descents across a university campus.The authors also observed error correction during outdoor walking.
  • Unmodeled terrain: The policy showed robustness to uneven terrain, logs, and curbs, although these terrain types were not modeled during training.These observations came from one-off campus experiments.
  • Outdoor staircase trials: 80% ascent success and 100% descent success were recorded across ten outdoor trials in each direction.The policy remained recoverable after missteps but could fall after a particularly egregious error.
  • Conclusion: The study concludes that blind bipedal control can climb a wide variety of real-world stairs using proprioceptive feedback.The conclusion frames this capability as a demonstrated property of the controller.
  • Training implications: Stair-capable policies required only terrain randomization changes to an existing training pipeline and no stair-specific reward terms.The conclusion states that adding stairs without further information was sufficient for learning stair-capable control.

Reward Function

The reward function uses probabilistic, phase-dependent penalties to alternate between encouraging foot lifting and foot placement, while additional terms regulate motion, orientation, energy, and commands.

  • Phase-dependent penalties: Binary random indicator functions identify gait-cycle intervals when selected quantities should be penalized, and their expectations are used for more stable learning.The indicators are defined through a Von Mises distribution rather than applied as raw random variables.
  • Reward construction: The full reward is a bias minus the expectation of a probabilistic penalty term, with detailed quantities and weights listed in Table IV.The expected penalty combines gait-phase terms with other locomotion costs.
  • Orientation regulation: Orientation error compares foot and pelvis quaternion orientations against a desired forward-facing orientation.The desired orientation is fixed to facing straight ahead for these experiments.
  • Additional costs: Additional costs track commanded-versus-actual speed, pelvis rotation and acceleration, action changes, and joint torques to shape stable, smooth, and energy-efficient gaits.The torque term is intended to encourage energy-efficient locomotion.
  • Foot-control terms: Penalizing foot forces during stance encourages foot lifting, while penalizing foot velocities during swing encourages foot placement.These complementary terms form the foundation for learning simple walking behavior.
Loading 2105.08328v1…