Source-linked AI summary
Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, Koushil Sreenath
TL;DR
Bipedal locomotion controllers must handle complex dynamics and modeling uncertainty without sacrificing adaptable behavior. This paper trains model-free reinforcement-learning policies from diverse HZD-based gaits with domain-randomized simulation, then transfers them to Cassie. The resulting policies perform diverse parameterized walking behaviors and show robustness across simulators, the real robot, perturbations, environmental changes, and hardware variation.
Problem
Bipedal locomotion involves high-dimensional hybrid dynamics and hard-to-model impacts, while traditional model-based controllers require careful modeling and may lack environmental adaptability.
Method
The method combines model-free reinforcement learning, a diverse parameterized HZD-based gait library, randomized simulation training, and sim-to-real transfer for Cassie control.
Results
The learned policies control Cassie across walking velocities, heights, turning, and challenging real-world conditions, with the feasible command set more than 4 times larger than an HZD-based controller.
Takeaways & Limitations
Parameterized reinforcement learning provides Cassie with diverse walking control and robustness to modeling errors, perturbations, environmental changes, and partially damaged motors.
Abstract
from arXiv · showhide
Developing robust walking controllers for bipedal robots is a challenging endeavor. Traditional model-based locomotion controllers require simplifying assumptions and careful modelling; any small errors can result in unstable control. To address these challenges for bipedal locomotion, we present a model-free reinforcement learning framework for training robust locomotion policies in simulation, which can then be transferred to a real bipedal Cassie robot. To facilitate sim-to-real transfer, domain randomization is used to encourage the policies to learn behaviors that are robust across variations in system dynamics. The learned policies enable Cassie to perform a set of diverse and dynamic behaviors, while also being more robust than traditional controllers and prior learning-based methods that use residual control. We demonstrate this on versatile walking behaviors such as tracking a target walking velocity, walking height, and turning yaw.
I. INTRODUCTION
The paper uses model-free reinforcement learning with gait-library references and domain randomization to learn diverse, robust locomotion policies for Cassie. The resulting controller supports parameterized walking and transfers from simulation to the real robot under varied conditions.
- Bipedal locomotion is difficult because robots have many degrees of freedom, hybrid nonlinear dynamics, and persistent hard-to-model ground impacts.
- Residual-control policies can stabilize Cassie but depend on pre-existing controllers and remain close to their reference motions.
- The proposed controller combines reinforcement learning with a diverse HZD-based gait library instead of relying on residual corrections.
- The policy follows frontal and lateral walking speeds, walking height, and turning yaw rate while expanding feasible commands and improving gait-transition stability over an HZD baseline.
- Domain randomization supports transfer from simulation to other simulators and Cassie, with robustness to modeling errors, perturbations, friction changes, unknown loads, and motor malfunctions.
II. PARAMETERIZED CONTROL OF CASSIE
Cassie is a person-sized, dynamic, underactuated bipedal robot with 20 degrees of freedom. Its observable state excludes pelvis translational position because that quantity cannot be reliably measured on the real robot without external instrumentation.
- Cassie is a person-sized, dynamic, underactuated bipedal robot with 20 degrees of freedom.
- The robot has 10 actuated rotational joints and four passive joints corresponding to shin and tarsus joints.
- Cassie’s floating-base pelvis has three translational and three rotational degrees of freedom and defines the robot’s local reference frame.
- The observable state is 17-dimensional and excludes pelvis translational position, which is not reliably measurable without external instrumentation.
B. Gait Library and Parameterized Control
The controller parameterizes Cassie’s locomotion through a gait library of periodic joint trajectories and reinforcement learning. The gait parameters specify forward velocity, lateral velocity, and walking height, enabling online control across gait behaviors.
- A gait parameter determines the desired gait, while each gait is a set of periodic joint trajectories encoding a locomotion behavior.
- Fifth-order Bézier curves represent smooth profiles for Cassie’s 10 actuated joints over a normalized step period.
- The gait parameters are forward velocity, lateral velocity, and walking height, represented as a three-dimensional vector.
- The HZD-based gait library indexes gaits by their parameters and is combined with an online regulator for parameterized walking control.
- Reinforcement learning combines the pre-computed gait library with feedback so Cassie’s policy maps observed states and goals to action distributions.
A. Cassie Simulation Environment
The policy is trained in a MuJoCo-based Cassie simulation and receives commands, reference gaits, recent observations, and past actions. It outputs filtered target motor positions that a joint-level PD controller converts into motor torques.
- The reinforcement-learning environment is based on the open-source MuJoCo simulator for Cassie.
- The action specifies target positions for Cassie’s 10 motors, which are low-pass filtered before a joint-level PD controller generates torque.
- The policy state contains observable robot states from the current and previous four time steps plus actions from the previous four time steps.
- The policy goal includes current user commands, an interpolated reference gait, and reference motor positions at current and sampled future time steps.
B. Reward Function
The reward combines command tracking with reference-motion reproduction and terms that regulate posture, energy use, and contact forces. Desired velocities stabilize pelvis roll and pitch, while user-specified yaw encourages behaviors beyond the gait library.
- The reward encourages satisfying the command while reproducing the corresponding gait-library reference motion on the dynamic robot.
- Motor-position tracking minimizes discrepancies between actual motor positions ˆqm and reference positions qr_m.
- Pelvis terms track reference translational position, translational velocity, and rotational velocity in the robot’s local frame.
- The torque reward encourages reduced energy consumption, while the ground-reaction-force reward helps minimize vertical contact forces ˆqf.
- Desired roll and pitch velocities are fixed at 0 for pelvis stabilization, whereas desired yaw velocity follows the user command; integrated velocities provide desired positions.
- Because gait-library references omit turning yaw, non-zero desired yaw in the reward can encourage walking behaviors absent from those references.
C. Domain Randomization
Dynamics are randomized during simulation training to improve policy robustness and reduce the simulation-to-real-world gap. The randomization targets modeling error, sensor noise, and communication delay through parameterized environment properties.
- Simulation dynamics are randomized during training to improve policy robustness and bridge the simulation-to-real-world gap.
- The randomization addresses robot and environment modeling error, sensor noise, and communication delay between the policy and joint-level controller.
- These randomized dynamics properties are parameterized as µ, with values varied over the ranges specified in Table II.
D. Learning Model
The learning model maximizes expected discounted reward with PPO using separate policy and value networks. The policy observes noisy, delayed state and goals, while the value function additionally receives privileged information.
- The reinforcement-learning objective is to maximize total expected reward over trajectories τ sampled from the policy-induced distribution.
- PPO trains policy and value networks, each using two hidden layers of 512 tanh units.
- The policy receives observed robot state and goal inputs, with observation noise and delay included in the observed state.
- The policy outputs a 10-dimensional Gaussian action distribution, and sampled actions represent desired motor positions.
- The value network outputs expected return and receives ground-truth state plus randomized dynamics parameters µ as additional inputs.
E. Training Setup
Training runs a 30 Hz policy with a 2000 Hz joint-level controller across long episodes with periodically resampled commands. Command ranges extend beyond gait-library parameters, while curriculum randomization limits conservative standing behaviors.
- The policy operates at 30 Hz, while the joint-level PD controller runs at 2000 Hz; each episode lasts up to 2500 steps, approximately 83 s.
- Commands are uniformly resampled every 8 s and remain fixed during each window, covering specified ranges of velocity, height, and yaw.
- Training commands exceed the gait-library parameter range, allowing the agent to learn commands and behaviors beyond the library.
- Dynamics randomization is introduced gradually through a curriculum to prevent excessively conservative sub-optimal behaviors.
- With full randomization from the start, the policy may avoid falling by standing in place instead of learning dynamic locomotion.
IV. EXPERIMENTS
Experiments evaluate the learned policy across simulation domains, training strategies, command-space robustness, and comparisons with residual and HZD-based controllers. The results show faster curriculum training, broader command coverage, and improved tracking performance for the RL-based controller.
- Evaluation setup: The policy is trained in MuJoCo, tested in high-fidelity SimMechanics, and finally validated on the physical Cassie robot.SimMechanics is used primarily for testing because it runs slower than real time.
- Learning performance: Curriculum randomization learns substantially faster and achieves higher return than starting with the full randomization range.Full randomization initially encourages excessively conservative and sub-optimal behavior.
- Control comparison: Non-residual and residual policies perform similarly without external perturbations, with the non-residual policy marginally better.The non-residual policy is evaluated against residual control used in prior work.
- Command-space robustness: The RL-based controller’s feasible command set is more than 4 times larger than the HZD-based controller’s.The comparison uses identical command-set and safe-set generation procedures.
- Command-space robustness: The RL-based controller covers a broader safe set and tracks commands more closely than the HZD-based controller.It supports forward and backward walking from −1.2 m/s to 1.2 m/s, while the HZD-based controller remains below 1 m/s.
C. Robustness in the Real World
Real-world experiments show that Cassie performs diverse agile behaviors and remains robust under perturbations, environmental changes, unknown loads, and partially damaged motors. The learned policy therefore operates across challenging physical conditions without additional tuning after deployment.
- Agile behaviors: Cassie performs fast forward and backward walking, sideways walking, walking-height changes, and turning in real-world experiments.These behaviors are demonstrated across the reported experiment scenarios.
- Environmental robustness: The policy recovers from random perturbations and adapts to changed ground friction and unknown loads.The experiments include pushes, foot disturbances, altered friction, and an applied unknown load.
- Hardware variation: After two Cassie motors were partially damaged, model-based controllers could not reliably produce walking, whereas the learned controller remained operational.The damaged motors produced less torque than corresponding motors on the opposite side or in simulation.
- Hardware variation: The learned controller successfully controlled Cassie on its first deployment after motor damage, without additional tuning.Dynamics randomization, especially joint damping-ratio randomization, was used during training.
2) Perturbation:
The proposed policy is evaluated for robustness under simulated random perturbations and diverse real-world disturbances. It outperforms comparison policies at larger simulated perturbation intensities and shows improved robustness across real-world scenarios.
- Simulated perturbations: The evaluation compares a gait-library policy, a single-reference non-residual policy, and a single-gait residual policy under random 6 DoF pelvis perturbations.All policies in this comparison are trained without domain randomization.
- Simulated perturbations: At β ∈ {0.8, 1}, the proposed method shows significant advantages in achieved return over other models.The comparison includes non-residual policies trained with a gait library or a single reference motion, and a residual policy trained with the same single gait.
- Real-world disturbances: In real-world tests, Cassie is pushed from multiple directions and experiences foot perturbations, including stepping on the gantry.The tests include pushes from the front, back, left, and right of the pelvis.
- Real-world disturbances: The learned policy remains robust when Cassie carries an unknown load, encounters changed ground friction, or undergoes random pushes during walking.The paper reports improved robustness over previous work across all described real-world scenarios.