PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, Koushil Sreenath

arXiv:2401.16889v2cs.ROcs.AIeess.SY

TL;DR

The paper asks how deep RL can provide general, robust control across diverse dynamic bipedal skills despite nonlinear dynamics and changing contacts. It introduces an end-to-end dual-history controller using long- and short-term I/O histories, and demonstrates versatile Cassie locomotion with real-world running, jumping, walking, and standing.

  • Problem

    A general framework for robust control across diverse bipedal locomotion skills remains difficult because high-dimensional nonlinear dynamics and changing contacts complicate control.

  • Method

    The paper trains RL controllers end-to-end with a non-recurrent architecture integrating long-term and short-term robot I/O histories.

  • Results

    The controllers consistently perform across diverse skills in simulation and reality, with Cassie completing a 400-meter dash in 2 minutes 34 seconds.

  • Takeaways & Limitations

    I/O history supports adaptation to changing dynamics and contacts, while task randomization improves task generalization and disturbance compliance.

Abstract

from arXiv · show

This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal robots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot's I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps.

I. INTRODUCTION

The paper addresses the open problem of controlling diverse, agile, and robust locomotion skills on high-dimensional human-sized bipedal robots. It introduces an RL framework centered on dual I/O histories, adaptivity, task randomization, and extensive Cassie experiments.

  • Developing one robust framework for walking, running, jumping, and standing remains difficult because bipedal robots have underactuated, nonlinear dynamics and changing contact plans.
  • The proposed framework uses a non-recurrent policy integrating explicit long and short robot I/O histories with end-to-end training.
  • The RL controllers adapt to time-invariant dynamics shifts and time-variant changes such as contact events.
  • Task randomization enhances robustness by promoting task generalization and compliance with disturbances beyond dynamics randomization.
  • Cassie demonstrates robust standing, walking, running, and jumping in extensive real-world experiments, including diverse periodic and aperiodic skills.

1) Choice of Models:

The paper contrasts model-based and model-free approaches to bipedal locomotion and motivates RL as a scalable alternative for diverse skills. Its design emphasizes direct motor commands, I/O history, dual-history structure, and end-to-end training.

  • 1) Choice of Models:: Fixed contact sequences simplify model-based optimization but constrain controllers across walking, running, and jumping behaviors.
  • 3) Scalability to Different Locomotion Skills:: Model-based control scales poorly across locomotion skills because robot models and control frameworks are task-specific.
  • 1) Choice of Models:: RL-based bipedal policies increasingly use longer histories, but prior work varies between state-only and short or long I/O histories.
  • 2) Sim-to-real Transfer:: Policy distillation commonly trains a privileged-information teacher and proprioceptive student, whereas this work argues end-to-end training is more effective for diverse dynamic skills.
  • 1) Choice of Models:: The proposed architecture combines long-term and short-term I/O histories, with the long history supporting adaptivity and the short history complementing it.

A. Cassie Robot Model

Cassie is modeled as a floating-base, torque-controlled biped whose partial observations and changing contacts motivate history-based RL control. Locomotion skills are represented as command-conditioned POMDP policies.

  • A. Cassie Robot Model: Cassie is a human-sized biped standing 1.1 m tall and weighing 31 kg, with 14 leg joints and a 20-dimensional full generalized coordinate vector.
  • A. Cassie Robot Model: Cassie has 10 actuated motor joints, while the shin and tarsus joints are passive and connected by leaf springs.
  • A. Cassie Robot Model: Its dynamics include mass, Coriolis and centrifugal, gravity, motor torque, spring torque, and external-force terms, including variable foot contacts.
  • Past I/O sequences can identify system parameters and change the control law over time, supporting adaptation to uncertain robot dynamics.
  • Because the robot observes only part of its full state, locomotion control is formulated as a POMDP whose optimal policy maps observation-action history to actions.

V. BIPEDAL LOCOMOTION CONTROLLER WITH I/O HISTORY

The controller uses a shared architecture combining short- and long-term robot I/O histories with skill-specific commands and reference motions. This design supports multiple locomotion skills while retaining the same policy structure.

  • Policy architecture: A 1D CNN encodes the long history before merging it with a base MLP, while the short history enters the base MLP directly.The policy also receives skill-specific reference motion and commands.
  • Control pipeline: The policy is queried at 33 Hz, and its desired motor positions are smoothed before 2 kHz joint-level PD controllers calculate motor torques.This separates the neural policy rate from the higher-frequency torque-control loop.
  • Dual-history architecture: The policy combines a 4-timestep short I/O history with a 2-second long I/O history for control.The short history supplies recent feedback, while the long history supports system identification and state estimation, including during ballistic flight.
  • General policy structure: The same policy architecture applies to standing, walking, running, and jumping by changing reference motions and commands.The underlying architecture remains unchanged across the reported experiments.
  • Training interface: The training framework is designed to learn policies in simulation and transfer them to hardware without further fine-tuning.The multi-stage framework includes single-task learning, task randomization, and dynamics randomization for zero-shot transfer.

VI. MULTI-STAGE TRAINING FOR VERSATILE LOCOMOTION CONTROLLERS

The training framework uses a curriculum that progresses from single-skill imitation to diverse task learning and dynamics randomization. It accommodates multiple reference-motion sources and supports transitions between locomotion and standing.

  • Multi-stage curriculum: The three-stage curriculum begins with single-task reference-motion training, adds diverse commands, and then randomizes dynamics for robust sim-to-real transfer.The stages progressively target skill acquisition, task generalization, and robustness.
  • Standing transitions: A single policy can transition between standing and other locomotion skills, enabling rapid transitions and improved standing robustness.Standing is added after the versatile policy is learned for periodic skills such as walking and running.
  • Reference motions: The framework accepts reference motions from trajectory optimization, motion capture, and keyframe animation.Different reference sources are used for different skills, including walking, running, and jumping.
  • Walking references: The walking reference library contains 1331 motions spanning command ranges from [−1.0, −0.3, 0.65] to [1.0, 0.3, 1.0].Each walking reference uses a fixed 0.8-second gait period, and turning yaw is not included.
  • Running and jumping references: The running and jumping skills each use one reference motion, with running averaging 3 m/s and jumping reaching 0.5 m foot height over 1.66 seconds.The running reference lacks lateral or turning movements, while the jumping reference ends in a stance pose.
  • Reference-motion scope: Motion-capture and animation references are not trajectory-optimized to become dynamically feasible for the robot.This is an explicit boundary on the reference-motion construction process.

1) Reward Components:

The reward combines motion tracking, task completion, and smoothing objectives, with largely unified weights across skills and training stages. Weight changes mainly reflect task diversity and whether the skill includes flight.

  • Motion tracking: Motion tracking rewards follow the reference motor positions, pelvis height, and foot height.Vertical reference displacements include terrain or target-elevation offsets through δz.
  • Weight sharing: 24 of 72 reward terms vary across three locomotion skills and multi-stage training, while the remaining terms retain nominal values.Differences mainly reflect task diversity and the presence of a flight phase.
  • Task completion: Task completion rewards alignment with desired velocity, turning rate, global position, and orientation.Walking and running integrate velocity commands into position targets, whereas jumping specifies landing targets first.
  • Smoothing: Smoothing rewards reduce impact forces, energy consumption, motor velocities, joint acceleration, and action changes.These components discourage jerky behavior and regulate aggressive movements.
  • Stage- and skill-specific weighting: Stage 1 emphasizes motion tracking, while later randomized-task stages increase the importance of task completion.Running and jumping also use higher foot-height and task-completion weights than walking because of their flight phases.

1) Unified Approach:

The framework uses unified episode and termination design together with staged randomization to discourage conservative behaviors and improve robustness. Dynamics, measurements, perturbations, and terrain are randomized with skill-dependent scope.

  • Episode design: Episodes generally last 2500 timesteps, or 76 seconds, with Stage 2 commands randomized every 1 to 15 seconds.Stage 1 jumping instead uses a 22-second, 750-timestep episode to include post-landing standing.
  • Early termination conditions: Additional termination conditions address foot-height tracking and task-completion errors beyond standard falling or ground-contact termination.These conditions prevent the robot from collecting reward through undesirable stationary or poorly tracking behaviors.
  • Early termination conditions: Foot-height termination is especially effective for flight-phase skills because removing it lets early training policies remain mostly on the ground.The tolerance can be tightened initially and relaxed as training progresses.
  • Terrain and perturbation randomization: External perturbations and variable terrain are optional additions, and perturbations may hinder meaningful learning for highly dynamic jumping.The table recommends these additions only after general dynamics randomization has been learned.
  • Dynamics randomization: Stage 3 randomizes dynamics parameters to improve robustness to modeling and measurement uncertainty for zero-shot sim-to-real transfer.Randomized parameters include friction, damping, link mass, inertia, center of mass, and leaf-spring stiffness.
  • Dynamics randomization: Joint-level PD gains are randomized by ±30% to mimic variation in motor responses associated with aging and degradation.Cassie’s leaf-spring stiffness is also randomized by ±20%, which is reported as important for running and jumping transfer.
  • Terrain and perturbation randomization: Terrain randomization covers waves, slopes, monotonic stairs, and random steps, but the policy must infer terrain changes without vision.Terrain randomization is recommended only after proficiency with general dynamics randomization.
  • Training implementation: All policies are trained in MuJoCo with PPO, using an actor based on the proposed architecture and a value function with ground-truth observations.The training setup is applied across the different skills and stages.

A. Baselines

The study compares policy architectures that vary action representation, observation history, temporal context, and training strategy. The proposed dual-history, end-to-end design consistently performs best across dynamic locomotion skills.

  • The proposed policy jointly trains a CNN long-history encoder and base MLP, directly specifying desired motor positions from short and long I/O histories.
  • Residual actions consistently deteriorate learning performance across locomotion skills, despite potentially accelerating initial skill learning.Adding reference motion can introduce extra movements that the policy must correct.
  • Omitting robot input history reduces learning performance, showing that state-feedback history alone is insufficient even with the same architecture and training approach.
  • Long I/O history alone does not surpass short-history-only performance, whereas direct short-history access complements the long-history encoder.The encoder can obscure very recent trajectory information, making explicit short history useful to the base MLP.
  • Recurrent LSTM policies do not gain significantly from dual history and tend to converge to more suboptimal policies than non-recurrent policies.
  • End-to-end training outperforms policy distillation, while A-RMA improves RMA but remains slightly below the proposed method despite more training samples.RMA degradation is especially prominent for running, where its encoder struggles to estimate environment parameters.

C. Case Study: In-place Walking Experiments

Real-world in-place walking tests examine whether policies adapt from simulation to Cassie without tuning or global position feedback. The proposed controller achieves lower tracking errors, minimal drift, and latent representations that reflect activities, contacts, and disturbances.

  • In-place walking experiments: The proposed policy maintains Cassie’s in-place walking with notably lower tracking errors and minimal sagittal and lateral drift in the real world.The comparison uses policies trained with the same number of samples.
  • In-place walking experiments: Simulation and real-world tests evaluate speed and base-orientation MAE over 10 seconds across three random-seed policies.
  • In-place walking experiments: The proposed design excels at bridging the sim-to-real gap and controlling the robot in the real-world environment.
  • Adaptive latent embedding: Long-history latent embeddings adapt to external disturbances and locomotion tasks, implicitly estimating contact events or external forces across walking, running, and jumping.
  • Adaptive latent embedding: During running, latent values become periodic after gait stabilization and vary during a 40 N backward perturbation.
  • Adaptive latent embedding: During jumping, latent representations distinguish jumping from standing, different target tasks, and separate take-off and landing signals.The separated contact cues may provide more informative control signals because flight and landing have different control complexity.

B. Adaptive Embedding for Changes in Dynamics

The controller’s long-history embedding adapts to time-varying events, altered dynamics, and corrupted measurements, while robustness also comes from task randomization. Versatile policies can generalize across tasks and use alternative gaits to respond to disturbances.

  • Changes in dynamics: The ablation study varies link centers of mass, masses, damping, PD gains, and ground friction to test adaptation to dynamics changes.
  • Changes in dynamics: A 0.025 s control and observation latency changes the latent embedding, whereas noise twice the training upper bound has little effect across running, jumping, and walking.
  • Changes in dynamics: Control performance remains minimally degraded under time-invariant parameter changes because the history encoder captures informative I/O patterns.
  • Sources of robustness: Versatile walking policies can stabilize under perturbations by using learned side- or backward-walking gaits, although single-task perturbation-trained policies complete the assigned task with less tracking error.
  • Sources of robustness: During running, single-task policies fail to maintain stable gaits under a constant 30 N forward perturbation, whereas versatile policies trained for faster running adapt successfully.
  • Sources of robustness: Task randomization provides robustness and compliance by generalizing across locomotion tasks, while dynamics and perturbation randomization expand the scenario range but limit single-task policies to trained tasks.

C. Case Study: Robust Standing Experiments

The standing case study examines how versatile policies recover from perturbations when standing, including transitions into walking and back to standing. These recovery behaviors were not specifically trained as standing responses.

  • Single-Task Comparison: Single-task standing policies lose balance when forced to lean beyond their support region, including when trained with external perturbations.This contrasts with the versatile walking policy’s recovery behavior.
  • Recovery from Perturbations: Versatile walking recovers from forward perturbations by transitioning into walking, taking several steps, and returning smoothly to standing without commands.The only reference motion provided was standing, and external perturbations were not simulated during standing-skill training.
  • Cross-Skill Recovery: Versatile policies use walking, running, and jumping maneuvers to recover from lateral perturbations, guard contact, and unstable landings.The reported recoveries include lowering the center of mass, side-stepping, and corrective hopping before stabilizing.
  • Training Distributions: The training distribution can expand through dynamics randomization, while task randomization provides an additional robustness route.Dynamics randomization cannot be increased arbitrarily because excessive ranges can prevent meaningful skill learning.

A. Walking Experiments

A single versatile walking controller tracks varied commands on Cassie, remains consistent over more than a year, and handles terrain changes and external perturbations without additional terrain sensing.

  • Long-Term Consistency: More than one year of hardware use produced consistent walking control with only minimal tracking-error degradation.The motivation is adaptation to changing hardware dynamics caused by wear and tear.
  • Tracking Performance: The walking policy tracks sagittal velocity, lateral velocity, walking height, and turning commands using one controller.The reported MAE values for ( ˙qx, ˙qy, qz) are (0.10 m/s, 0.10 m/s, 0.06 m), respectively.
  • Fast Walking: The policy reaches an average forward walking velocity of 1.14 m/s while tracking a 1.4 m/s command and can transition between walking and standing.This includes fast forward and backward walking maneuvers.
  • Uneven Terrains: Without terrain training or elevation sensors, Cassie walks backward over small stairs and declined slopes by tolerating contact-timing and wrench changes.The controller does not require explicit contact estimation or control.
  • Perturbation Robustness: After a lateral perturbation producing a 0.5 m/s velocity peak, the robot moves oppositely to recover, while persistent dragging does not cause loss of balance.The tests include impulse perturbations and persistent forces in changing directions and at different walking heights.
  • Source of Robustness: Task randomization across lateral, forward, backward, and turning skills supports generalization to disturbances encountered during deployment.The paper contrasts this versatile policy with a model-based controller that crashes under lateral perturbation.

3) Summary of Results:

The real-world evaluations show that Cassie’s versatile policies support sustained running, command tracking, and dynamic gait transitions. The 400-meter demonstration reached 2 minutes and 34 seconds, although high-speed sagittal-velocity estimates contain nontrivial error.

  • Walking Results: The versatile walking policy controls diverse tasks in real time, remains consistent for more than a year, and withstands terrain changes and external perturbations.It tracks velocities and heights, turns in both directions, walks quickly, and transitions to and from standing.
  • Measurement Caveat: High-speed sagittal-velocity logs have nontrivial state-estimation error: the 154-second run implies 2.6 m/s average speed versus an estimated 2.15 m/s.The paper emphasizes that the logged values are estimates rather than true values.
  • Running Results: 5.95 degrees was the average turning-angle MAE during the 400-meter dash.The recorded data include sagittal velocity, lateral velocity, and turning yaw angle.
  • Running Results: 154 seconds of running sustained substantial flight phases while maintaining the dynamic gait across changing speeds and turning angles.The running gait is distinguished from fast walking by its flight phases.
  • Running Results: Cassie completed the 400-meter dash in 2 minutes and 34 seconds using a running policy, then transitioned to standing.The run used a single policy and demonstrated a full 400-meter lap by a human-sized bipedal robot.

2) Tracking Varying Commands while Running:

Cassie’s running policy tracks changing velocity and yaw commands, including an untrained sharp turn, while maintaining running over uneven terrain and recovering from disturbances.

  • Command tracking: The policy tracks sagittal and lateral velocities while running fast, with command changes in one dimension not affecting constant commands in others.The authors interpret this as decoupling among sagittal velocity, lateral velocity, and yaw command dimensions.
  • Command tracking: A direct 90-degree yaw-command step is completed within 5 steps over 2 seconds, despite training using only smoothly changing yaw commands.Training bounded yaw-command changes to 30 deg/s, so the sharp-turn test was outside that training scenario.
  • Uneven terrain: The robot traverses sequential 7° sagittal, 3° lateral, and 10° sagittal slopes without explicit terrain-height estimation or external sensors.It maintains a stable running gait and retains a flight phase across the varied elevations.
  • Disturbance recovery: During high-speed running, task-randomized policies remain robust to safety-cord perturbations that reduce speed and induce lateral leaning.The perturbation scenario is presented as evidence of robustness to unexpected real-world disturbances.
  • Fast running: The running policy completes the 100-meter dash in 27.06 seconds at a peak speed of 4.2 m/s.The robot transitioned from standing to fast running within 1.8 seconds during the dash.

3) Robust Jumping Maneuvers:

Two versatile jumping policies support precise target-directed maneuvers and recovery from unexpected perturbations. Their learned task repertoire enables alternative contact strategies, although stationary recovery after large jumps can oscillate.

  • Robust recovery: The jumping policies were never explicitly trained for perturbations, yet the robot uses learned maneuvers to recover after an unexpected airborne impulse.During recovery, it adjusts its pose in the air and can execute a small hop after landing to improve its configuration.
  • Limitation: Jumping policies can occasionally oscillate after large jumps because one RL policy must support both dynamic jumping and stationary standing.The oscillation eventually damps out, but the recovery to stationary behavior takes noticeably longer.
  • Experimental scope: The real-world experiments demonstrate 19 distinct jumps spanning landing locations, turns, and elevations using only two versatile policies.The jumping experiments illustrate both adaptivity to hardware dynamics and robustness to unexpected perturbations.
  • Versatile jumping: The flat-ground policy performs in-place jumps with 60° turns, backward jumps, and forward jumps to commanded targets.The robot changes its take-off pose to follow different commands and lands precisely on marked targets.

B. Versatility Improves Robustness

The paper distinguishes robustness gained from dynamics randomization from robustness gained through task randomization and broader learned task repertoires. It also identifies limits in unified-policy training and precision control.

  • Sources of robustness: Task randomization adds robustness by training policies across diverse tasks, whereas dynamics randomization mainly broadens trajectories within a specific task.A randomized standing policy does not thereby acquire walking or jumping recovery skills.
  • Motion generalization: Task randomization enables robots to vary running and jumping maneuvers beyond single reference motions through goal-conditioned policies.The paper notes that a forward-only reference motion makes learning backward jumps challenging.
  • Online recovery: Versatile policies can develop contact strategies online and use varied motion sequences for recovery without an online motion or contact scheduler.Examples include changing walking gaits after standing perturbations and hopping after jump landings.
  • Unified-policy limits: Adding new skills to a unified policy can cause catastrophic forgetting, while AMP may struggle with aggressive brief motions and the bipedal sim-to-real gap.The paper presents skill-specific policies as a practical alternative under these constraints.
  • Generalization versus precision: A single policy can generalize across tasks and dynamics, but perfect precision remains open; fast-running sagittal velocity can still be difficult to track with minor errors.The paper frames this as a trade-off between broad generalization and task-specific precision.

APPENDIX

The appendix examines action filtering, standing integration, training settings, history length, and temporal encoders for dynamic locomotion policies. It finds that dual-history non-recurrent policies support learning across skills, while recurrent policies are more sensitive to tuning.

  • Advantages of Using an Action Filter: A 4 Hz Butterworth low-pass filter smooths policy actions and improves jumping learning performance by reducing jittering motion.Without the filter, jumping remains possible but converges to a worse return; the filter also reduces reward-tuning burden.
  • Learning to Combine a Standing Skill: Standing is added after randomized walking episodes by changing the reference motion and reward, while increased smoothing encourages stationary behavior.The walking policy also learns the transition back from standing to walking.
  • Skill Transitions: Running uses an additional human-motion reference for the transition from fast running to standing, whereas jumping needs no standing-training sub-stage after landing.The additional reference facilitates training during the more challenging running-to-standing transition.
  • Comparison of Use of Different History Lengths: Increasing running-history length from 1 to 2 to 3 seconds improves learning, but extending it to 4 seconds causes a performance drop.The authors recommend starting with a 2-second history because it offers relatively stable training and good performance.
  • Comparison of Use of Different Temporal Encoders: Adding a short 4-timestep I/O history alongside a TCN significantly improves learning over TCN alone, whereas adding short history does not improve the LSTM policy.The LSTM also tends to converge to a lower return plateau and failed to learn jumping under the tested settings.
  • Comparison of Use of Different Temporal Encoders: Dual-history designs consistently benefit explicit-history non-recurrent policies, while recurrent LSTM policies are more sensitive to hyperparameter tuning and may converge to suboptimal policies.The appendix therefore identifies dual-history non-recurrent policies as more favorable for bipedal locomotion control.

G. Latent Visualization of Walking Policy

The walking-policy analysis shows that latent embeddings and saliency patterns change with perturbations, while short and long I/O histories contribute different information to action selection. It also documents estimator error as a boundary on interpreting high-speed running results.

  • Saliency Map: The policy focuses especially on the short I/O history and most recent observation during both perturbed running and walking.This saliency pattern highlights the role of recent explicit history in producing actions.
  • Saliency Map: Perturbations change which components of the encoded long history receive saliency, suggesting that the policy uses long-history information to adjust actions across scenarios.The long-history embedding is examined for both running and walking tests.
  • History Ablation: Removing either the short or long I/O history degrades learning performance and sim-to-real transfer in the ablation study.This supports using both history scales rather than relying on only the most recent observation or one history branch.
  • Estimator Errors in High-Speed Running: Above 3 m/s, the onboard EKF velocity estimate shows large error, so actual high-speed running may be faster than recorded and tracking error may be smaller.The remaining high-speed tracking error is attributed to the sim-to-real gap and robot hardware limitations.
Loading 2401.16889v2…