Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control

• Developing one robust framework for walking, running, jumping, and standing remains difficult because bipedal robots have underactuated, nonlinear dynamics and changing contact plans.Source: §I · p. 1, C. Contributions · p. 2
• The proposed framework uses a non-recurrent policy integrating explicit long and short robot I/O histories with end-to-end training.Source: C. Contributions · p. 2
• The RL controllers adapt to time-invariant dynamics shifts and time-variant changes such as contact events.Source: C. Contributions · p. 2, §I · p. 2
• Task randomization enhances robustness by promoting task generalization and compliance with disturbances beyond dynamics randomization.Source: C. Contributions · p. 2
• Cassie demonstrates robust standing, walking, running, and jumping in extensive real-world experiments, including diverse periodic and aperiodic skills.Source: C. Contributions · p. 2

1) Choice of Models:

• Fixed contact sequences simplify model-based optimization but constrain controllers across walking, running, and jumping behaviors.Source: 2) Contact Planning · p. 3, 3) Scalability to Different Locomotion… · p. 3

3) Scalability to Different Locomotion Skills:

• Model-based control scales poorly across locomotion skills because robot models and control frameworks are task-specific.Source: 3) Scalability to Different Locomotion… · p. 3

1) Choice of Models:

• RL-based bipedal policies increasingly use longer histories, but prior work varies between state-only and short or long I/O histories.Source: 1) Control Policy Structure · p. 4

2) Sim-to-real Transfer:

• Policy distillation commonly trains a privileged-information teacher and proprioceptive student, whereas this work argues end-to-end training is more effective for diverse dynamic skills.Source: 2) Sim-to-real Transfer · p. 5

1) Choice of Models:

• The proposed architecture combines long-term and short-term I/O histories, with the long history supporting adaptivity and the short history complementing it.Source: §III · p. 5

A. Cassie Robot Model

• Cassie is a human-sized biped standing 1.1 m tall and weighing 31 kg, with 14 leg joints and a 20-dimensional full generalized coordinate vector.Source: 1) Floating-base Coordinates · pp. 5–6
• Cassie has 10 actuated motor joints, while the shin and tarsus joints are passive and connected by leaf springs.Source: 1) Floating-base Coordinates · p. 6
• Its dynamics include mass, Coriolis and centrifugal, gravity, motor torque, spring torque, and external-force terms, including variable foot contacts.Source: 2) Full-order Dynamics Model · p. 6

• Past I/O sequences can identify system parameters and change the control law over time, supporting adaptation to uncertain robot dynamics.Source: 3) System Identification and Adaptive… · p. 6, 2) Solving POMDP with I/O History · p. 7
• Because the robot observes only part of its full state, locomotion control is formulated as a POMDP whose optimal policy maps observation-action history to actions.Source: 1) Legged Locomotion Control as a POMDP · pp. 6–7, 2) Solving POMDP with I/O History · p. 7

Policy architecture

• A 1D CNN encodes the long history before merging it with a base MLP, while the short history enters the base MLP directly.The policy also receives skill-specific reference motion and commands.Source: Figure 3 · p. 7, §V · pp. 7–8

Control pipeline

• The policy is queried at 33 Hz, and its desired motor positions are smoothed before 2 kHz joint-level PD controllers calculate motor torques.This separates the neural policy rate from the higher-frequency torque-control loop.Source: §V · p. 7

Dual-history architecture

• The policy combines a 4-timestep short I/O history with a 2-second long I/O history for control.The short history supplies recent feedback, while the long history supports system identification and state estimation, including during ballistic flight.Source: §V · p. 8

General policy structure

• The same policy architecture applies to standing, walking, running, and jumping by changing reference motions and commands.The underlying architecture remains unchanged across the reported experiments.Source: §V · p. 8

Training interface

• The training framework is designed to learn policies in simulation and transfer them to hardware without further fine-tuning.The multi-stage framework includes single-task learning, task randomization, and dynamics randomization for zero-shot transfer.Source: Figure 4 · p. 8

Multi-stage curriculum

• The three-stage curriculum begins with single-task reference-motion training, adds diverse commands, and then randomizes dynamics for robust sim-to-real transfer.The stages progressively target skill acquisition, task generalization, and robustness.Source: A. Overview · pp. 8–9

Standing transitions

• A single policy can transition between standing and other locomotion skills, enabling rapid transitions and improved standing robustness.Standing is added after the versatile policy is learned for periodic skills such as walking and running.Source: A. Overview · p. 9

Reference motions

• The framework accepts reference motions from trajectory optimization, motion capture, and keyframe animation.Different reference sources are used for different skills, including walking, running, and jumping.Source: B. Reference Motion · p. 9

Walking references

• The walking reference library contains 1331 motions spanning command ranges from [−1.0, −0.3, 0.65] to [1.0, 0.3, 1.0].Each walking reference uses a fixed 0.8-second gait period, and turning yaw is not included.Source: B. Reference Motion · p. 9

Running and jumping references

• The running and jumping skills each use one reference motion, with running averaging 3 m/s and jumping reaching 0.5 m foot height over 1.66 seconds.The running reference lacks lateral or turning movements, while the jumping reference ends in a stance pose.Source: B. Reference Motion · p. 9

Reference-motion scope

• Motion-capture and animation references are not trajectory-optimized to become dynamically feasible for the robot.This is an explicit boundary on the reference-motion construction process.Source: B. Reference Motion · p. 9

Motion tracking

• Motion tracking rewards follow the reference motor positions, pelvis height, and foot height.Vertical reference displacements include terrain or target-elevation offsets through δz.Source: 1) Reward Components · p. 9

Weight sharing

• 24 of 72 reward terms vary across three locomotion skills and multi-stage training, while the remaining terms retain nominal values.Differences mainly reflect task diversity and the presence of a flight phase.Source: Table III · p. 10, 2) Reward Weights · p. 10

Task completion

• Task completion rewards alignment with desired velocity, turning rate, global position, and orientation.Walking and running integrate velocity commands into position targets, whereas jumping specifies landing targets first.Source: 1) Reward Components · p. 10

Smoothing

• Smoothing rewards reduce impact forces, energy consumption, motor velocities, joint acceleration, and action changes.These components discourage jerky behavior and regulate aggressive movements.Source: 1) Reward Components · p. 10

Stage- and skill-specific weighting

• Stage 1 emphasizes motion tracking, while later randomized-task stages increase the importance of task completion.Running and jumping also use higher foot-height and task-completion weights than walking because of their flight phases.Source: 2) Reward Weights · pp. 10–11

Episode design

• Episodes generally last 2500 timesteps, or 76 seconds, with Stage 2 commands randomized every 1 to 15 seconds.Stage 1 jumping instead uses a 22-second, 750-timestep episode to include post-landing standing.Source: 1) Unified Approach · p. 11

Early termination conditions

• Additional termination conditions address foot-height tracking and task-completion errors beyond standard falling or ground-contact termination.These conditions prevent the robot from collecting reward through undesirable stationary or poorly tracking behaviors.Source: 2) Early Termination Conditions · p. 11
• Foot-height termination is especially effective for flight-phase skills because removing it lets early training policies remain mostly on the ground.The tolerance can be tightened initially and relaxed as training progresses.Source: 2) Early Termination Conditions · p. 11

Terrain and perturbation randomization

• External perturbations and variable terrain are optional additions, and perturbations may hinder meaningful learning for highly dynamic jumping.The table recommends these additions only after general dynamics randomization has been learned.Source: Table IV · p. 11

Dynamics randomization

• Stage 3 randomizes dynamics parameters to improve robustness to modeling and measurement uncertainty for zero-shot sim-to-real transfer.Randomized parameters include friction, damping, link mass, inertia, center of mass, and leaf-spring stiffness.Source: E. Dynamics Randomization · p. 11
• Joint-level PD gains are randomized by ±30% to mimic variation in motor responses associated with aging and degradation.Cassie’s leaf-spring stiffness is also randomized by ±20%, which is reported as important for running and jumping transfer.Source: E. Dynamics Randomization · pp. 11–12

Terrain and perturbation randomization

• Terrain randomization covers waves, slopes, monotonic stairs, and random steps, but the policy must infer terrain changes without vision.Terrain randomization is recommended only after proficiency with general dynamics randomization.Source: E. Dynamics Randomization · p. 12

Training implementation

• All policies are trained in MuJoCo with PPO, using an actor based on the proposed architecture and a value function with ground-truth observations.The training setup is applied across the different skills and stages.Source: F. Training Details · p. 12

• The proposed policy jointly trains a CNN long-history encoder and base MLP, directly specifying desired motor positions from short and long I/O histories.Source: A. Baselines · p. 12
• Residual actions consistently deteriorate learning performance across locomotion skills, despite potentially accelerating initial skill learning.Adding reference motion can introduce extra movements that the policy must correct.Source: A. Baselines · pp. 12–13, 1) Benchmark Analysis · p. 13
• Omitting robot input history reduces learning performance, showing that state-feedback history alone is insufficient even with the same architecture and training approach.Source: A. Baselines · p. 13, 1) Benchmark Analysis · p. 14
• Long I/O history alone does not surpass short-history-only performance, whereas direct short-history access complements the long-history encoder.The encoder can obscure very recent trajectory information, making explicit short history useful to the base MLP.Source: A. Baselines · p. 13, 1) Benchmark Analysis · p. 14
• Recurrent LSTM policies do not gain significantly from dual history and tend to converge to more suboptimal policies than non-recurrent policies.Source: 1) Benchmark Analysis · p. 14
• End-to-end training outperforms policy distillation, while A-RMA improves RMA but remains slightly below the proposed method despite more training samples.RMA degradation is especially prominent for running, where its encoder struggles to estimate environment parameters.Source: A. Baselines · p. 13, 1) Benchmark Analysis · p. 15

In-place walking experiments

• The proposed policy maintains Cassie’s in-place walking with notably lower tracking errors and minimal sagittal and lateral drift in the real world.The comparison uses policies trained with the same number of samples.Source: C. Case Study: In-place Walking… · p. 15
• Simulation and real-world tests evaluate speed and base-orientation MAE over 10 seconds across three random-seed policies.Source: C. Case Study: In-place Walking… · p. 16
• The proposed design excels at bridging the sim-to-real gap and controlling the robot in the real-world environment.Source: C. Case Study: In-place Walking… · p. 16, D. Summary of Results · p. 16

Adaptive latent embedding

• Long-history latent embeddings adapt to external disturbances and locomotion tasks, implicitly estimating contact events or external forces across walking, running, and jumping.Source: §VIII · p. 16, A. Time Varying Embedding · p. 16, Figure 9 · p. 17
• During running, latent values become periodic after gait stabilization and vary during a 40 N backward perturbation.Source: A. Time Varying Embedding · p. 16
• During jumping, latent representations distinguish jumping from standing, different target tasks, and separate take-off and landing signals.The separated contact cues may provide more informative control signals because flight and landing have different control complexity.Source: A. Time Varying Embedding · pp. 17–18

Changes in dynamics

• The ablation study varies link centers of mass, masses, damping, PD gains, and ground friction to test adaptation to dynamics changes.Source: B. Adaptive Embedding for Changes in… · p. 18
• A 0.025 s control and observation latency changes the latent embedding, whereas noise twice the training upper bound has little effect across running, jumping, and walking.Source: B. Adaptive Embedding for Changes in… · p. 18
• Control performance remains minimally degraded under time-invariant parameter changes because the history encoder captures informative I/O patterns.Source: C. Summary of Results · p. 18, B. Adaptive Embedding for Changes in… · p. 18

Sources of robustness

• Versatile walking policies can stabilize under perturbations by using learned side- or backward-walking gaits, although single-task perturbation-trained policies complete the assigned task with less tracking error.Source: B. Source of Robustness · pp. 19–20
• During running, single-task policies fail to maintain stable gaits under a constant 30 N forward perturbation, whereas versatile policies trained for faster running adapt successfully.Source: B. Source of Robustness · p. 20
• Task randomization provides robustness and compliance by generalizing across locomotion tasks, while dynamics and perturbation randomization expand the scenario range but limit single-task policies to trained tasks.Source: §IX · p. 18, A. Baselines · p. 18, B. Source of Robustness · pp. 19–20

Single-Task Comparison

• Single-task standing policies lose balance when forced to lean beyond their support region, including when trained with external perturbations.This contrasts with the versatile walking policy’s recovery behavior.Source: C. Case Study: Robust Standing… · p. 20

Recovery from Perturbations

• Versatile walking recovers from forward perturbations by transitioning into walking, taking several steps, and returning smoothly to standing without commands.The only reference motion provided was standing, and external perturbations were not simulated during standing-skill training.Source: C. Case Study: Robust Standing… · p. 21

Cross-Skill Recovery

• Versatile policies use walking, running, and jumping maneuvers to recover from lateral perturbations, guard contact, and unstable landings.The reported recoveries include lowering the center of mass, side-stepping, and corrective hopping before stabilizing.Source: C. Case Study: Robust Standing… · p. 21

Training Distributions

• The training distribution can expand through dynamics randomization, while task randomization provides an additional robustness route.Dynamics randomization cannot be increased arbitrarily because excessive ranges can prevent meaningful skill learning.Source: D. Understand Robustness from Training… · p. 21

Long-Term Consistency

• More than one year of hardware use produced consistent walking control with only minimal tracking-error degradation.The motivation is adaptation to changing hardware dynamics caused by wear and tear.Source: 1) Tracking Performance · pp. 22–23

Tracking Performance

• The walking policy tracks sagittal velocity, lateral velocity, walking height, and turning commands using one controller.The reported MAE values for ( ˙qx, ˙qy, qz) are (0.10 m/s, 0.10 m/s, 0.06 m), respectively.Source: 1) Tracking Performance · pp. 22–23

Fast Walking

• The policy reaches an average forward walking velocity of 1.14 m/s while tracking a 1.4 m/s command and can transition between walking and standing.This includes fast forward and backward walking maneuvers.Source: 1) Tracking Performance · p. 23, Figure 16 · p. 23

Uneven Terrains

• Without terrain training or elevation sensors, Cassie walks backward over small stairs and declined slopes by tolerating contact-timing and wrench changes.The controller does not require explicit contact estimation or control.Source: 2) Robust Walking Maneuvers · pp. 23–24, Figure 17 · p. 24

Perturbation Robustness

• After a lateral perturbation producing a 0.5 m/s velocity peak, the robot moves oppositely to recover, while persistent dragging does not cause loss of balance.The tests include impulse perturbations and persistent forces in changing directions and at different walking heights.Source: 2) Robust Walking Maneuvers · p. 24, Figure 19 · p. 25

Source of Robustness

• Task randomization across lateral, forward, backward, and turning skills supports generalization to disturbances encountered during deployment.The paper contrasts this versatile policy with a model-based controller that crashes under lateral perturbation.Source: 2) Robust Walking Maneuvers · pp. 24–25

Walking Results

• The versatile walking policy controls diverse tasks in real time, remains consistent for more than a year, and withstands terrain changes and external perturbations.It tracks velocities and heights, turns in both directions, walks quickly, and transitions to and from standing.Source: 3) Summary of Results · p. 25

Measurement Caveat

• High-speed sagittal-velocity logs have nontrivial state-estimation error: the 154-second run implies 2.6 m/s average speed versus an estimated 2.15 m/s.The paper emphasizes that the logged values are estimates rather than true values.Source: 1) Running a 400-meter Dash · p. 25

Running Results

• 5.95 degrees was the average turning-angle MAE during the 400-meter dash.The recorded data include sagittal velocity, lateral velocity, and turning yaw angle.Source: 1) Running a 400-meter Dash · p. 26
• 154 seconds of running sustained substantial flight phases while maintaining the dynamic gait across changing speeds and turning angles.The running gait is distinguished from fast walking by its flight phases.Source: 1) Running a 400-meter Dash · p. 27
• Cassie completed the 400-meter dash in 2 minutes and 34 seconds using a running policy, then transitioned to standing.The run used a single policy and demonstrated a full 400-meter lap by a human-sized bipedal robot.Source: Figure 20 · p. 26, 1) Running a 400-meter Dash · p. 27

Command tracking

• The policy tracks sagittal and lateral velocities while running fast, with command changes in one dimension not affecting constant commands in others.The authors interpret this as decoupling among sagittal velocity, lateral velocity, and yaw command dimensions.Source: 2) Tracking Varying Commands while… · p. 27
• A direct 90-degree yaw-command step is completed within 5 steps over 2 seconds, despite training using only smoothly changing yaw commands.Training bounded yaw-command changes to 30 deg/s, so the sharp-turn test was outside that training scenario.Source: 2) Tracking Varying Commands while… · p. 27

Uneven terrain

• The robot traverses sequential 7° sagittal, 3° lateral, and 10° sagittal slopes without explicit terrain-height estimation or external sensors.It maintains a stable running gait and retains a flight phase across the varied elevations.Source: 4) Running on Uneven Terrains (Trained) · p. 28, Figure 23 · p. 29

Disturbance recovery

• During high-speed running, task-randomized policies remain robust to safety-cord perturbations that reduce speed and induce lateral leaning.The perturbation scenario is presented as evidence of robustness to unexpected real-world disturbances.Source: 5) Robust Running Maneuvers · p. 28

Fast running

• The running policy completes the 100-meter dash in 27.06 seconds at a peak speed of 4.2 m/s.The robot transitioned from standing to fast running within 1.8 seconds during the dash.Source: 6) Summary of Results · p. 28, 3) Running a 100-meter Dash · p. 28

Robust recovery

• The jumping policies were never explicitly trained for perturbations, yet the robot uses learned maneuvers to recover after an unexpected airborne impulse.During recovery, it adjusts its pose in the air and can execute a small hop after landing to improve its configuration.Source: 3) Robust Jumping Maneuvers · p. 29

Limitation

• Jumping policies can occasionally oscillate after large jumps because one RL policy must support both dynamic jumping and stationary standing.The oscillation eventually damps out, but the recovery to stationary behavior takes noticeably longer.Source: 3) Robust Jumping Maneuvers · pp. 29–30

Experimental scope

• The real-world experiments demonstrate 19 distinct jumps spanning landing locations, turns, and elevations using only two versatile policies.The jumping experiments illustrate both adaptivity to hardware dynamics and robustness to unexpected perturbations.Source: 4) Summary of Results · p. 30

Versatile jumping

• The flat-ground policy performs in-place jumps with 60° turns, backward jumps, and forward jumps to commanded targets.The robot changes its take-off pose to follow different commands and lands precisely on marked targets.Source: Figure 25 · p. 31

Sources of robustness

• Task randomization adds robustness by training policies across diverse tasks, whereas dynamics randomization mainly broadens trajectories within a specific task.A randomized standing policy does not thereby acquire walking or jumping recovery skills.Source: B. Versatility Improves Robustness · p. 33

Motion generalization

• Task randomization enables robots to vary running and jumping maneuvers beyond single reference motions through goal-conditioned policies.The paper notes that a forward-only reference motion makes learning backward jumps challenging.Source: C. Exploring What RL Can Enable for… · p. 33

Online recovery

• Versatile policies can develop contact strategies online and use varied motion sequences for recovery without an online motion or contact scheduler.Examples include changing walking gaits after standing perturbations and hopping after jump landings.Source: C. Exploring What RL Can Enable for… · p. 33

Unified-policy limits

• Adding new skills to a unified policy can cause catastrophic forgetting, while AMP may struggle with aggressive brief motions and the bipedal sim-to-real gap.The paper presents skill-specific policies as a practical alternative under these constraints.Source: C. Exploring What RL Can Enable for… · pp. 33–34

Generalization versus precision

• A single policy can generalize across tasks and dynamics, but perfect precision remains open; fast-running sagittal velocity can still be difficult to track with minor errors.The paper frames this as a trade-off between broad generalization and task-specific precision.Source: C. Exploring What RL Can Enable for… · p. 34

Advantages of Using an Action Filter

• A 4 Hz Butterworth low-pass filter smooths policy actions and improves jumping learning performance by reducing jittering motion.Without the filter, jumping remains possible but converges to a worse return; the filter also reduces reward-tuning burden.Source: A. Advantages of Using an Action Filter · p. 39

Learning to Combine a Standing Skill

• Standing is added after randomized walking episodes by changing the reference motion and reward, while increased smoothing encourages stationary behavior.The walking policy also learns the transition back from standing to walking.Source: B. Learning to Combine a Standing Skill · p. 39

Skill Transitions

• Running uses an additional human-motion reference for the transition from fast running to standing, whereas jumping needs no standing-training sub-stage after landing.The additional reference facilitates training during the more challenging running-to-standing transition.Source: B. Learning to Combine a Standing Skill · p. 39

Comparison of Use of Different History Lengths

• Increasing running-history length from 1 to 2 to 3 seconds improves learning, but extending it to 4 seconds causes a performance drop.The authors recommend starting with a 2-second history because it offers relatively stable training and good performance.Source: E. Comparison of Use of Different… · pp. 40–41

Comparison of Use of Different Temporal Encoders

• Adding a short 4-timestep I/O history alongside a TCN significantly improves learning over TCN alone, whereas adding short history does not improve the LSTM policy.The LSTM also tends to converge to a lower return plateau and failed to learn jumping under the tested settings.Source: F. Comparison of Use of Different… · p. 41
• Dual-history designs consistently benefit explicit-history non-recurrent policies, while recurrent LSTM policies are more sensitive to hyperparameter tuning and may converge to suboptimal policies.The appendix therefore identifies dual-history non-recurrent policies as more favorable for bipedal locomotion control.Source: F. Comparison of Use of Different… · pp. 41–42

Saliency Map

• The policy focuses especially on the short I/O history and most recent observation during both perturbed running and walking.This saliency pattern highlights the role of recent explicit history in producing actions.Source: H. Saliency Map · p. 42
• Perturbations change which components of the encoded long history receive saliency, suggesting that the policy uses long-history information to adjust actions across scenarios.The long-history embedding is examined for both running and walking tests.Source: H. Saliency Map · pp. 42–43

History Ablation

• Removing either the short or long I/O history degrades learning performance and sim-to-real transfer in the ablation study.This supports using both history scales rather than relying on only the most recent observation or one history branch.Source: H. Saliency Map · p. 43

Estimator Errors in High-Speed Running

• Above 3 m/s, the onboard EKF velocity estimate shows large error, so actual high-speed running may be faster than recorded and tracking error may be smaller.The remaining high-speed tracking error is attributed to the sim-to-real gap and robot hardware limitations.Source: §I · p. 43