PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust and Versatile Bipedal Jumping Control through Reinforcement Learning

Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, Koushil Sreenath

arXiv:2302.09450v2cs.ROcs.AIcs.SYeess.SY

TL;DR

Bipedal jumping requires control across rapid contact changes and underactuated flight, with precise target landings. The paper trains a goal-conditioned policy using staged simulation training and a history-based structure, then demonstrates versatile real-world jumps and recovery behaviors on Cassie.

Abstract

from arXiv · show

This work aims to push the limits of agility for bipedal robots by enabling a torque-controlled bipedal robot to perform robust and versatile dynamic jumps in the real world. We present a reinforcement learning framework for training a robot to accomplish a large variety of jumping tasks, such as jumping to different locations and directions. To improve performance on these challenging tasks, we develop a new policy structure that encodes the robot's long-term input/output (I/O) history while also providing direct access to a short-term I/O history. In order to train a versatile jumping policy, we utilize a multi-stage training scheme that includes different training stages for different objectives. After multi-stage training, the policy can be directly transferred to a real bipedal Cassie robot. Training on different tasks and exploring more diverse scenarios lead to highly robust policies that can exploit the diverse set of learned maneuvers to recover from perturbations or poor landings during real-world deployment. Such robustness in the proposed policy enables Cassie to succeed in completing a variety of challenging jump tasks in the real world, such as standing long jumps, jumping onto elevated platforms, and multi-axes jumps.

I. INTRODUCTION

Bipedal jumping combines rapid contact changes, underactuated flight, and precise landing demands, while prior RL work had not established dynamic jumps on Cassie beyond periodic hopping. This work targets versatile, goal-conditioned jumping and reports real-world multi-axis jumps and recovery behaviors.

I. INTRODUCTION

  • Jumping requires managing takeoff, underactuated flight, and landing impacts within typically less than 2 seconds, while producing momentum for a target landing location.Source: §I · p. 1
  • Prior Cassie optimal-control methods use carefully designed models and task-specific structures, while the most relevant prior RL system performed periodic hopping rather than standing long jumps.Source: §I · pp. 1–2

A. Objective of this Paper

  • The paper aims to test whether learning varied jumping tasks can improve robustness, including recovery by switching to a learned hop after a large landing impulse.Source: A. Objective of this Paper · p. 2

B. Contributions

  • The system enables real-world Cassie to perform controlled long, elevated, and multi-axis jumps, with learned maneuvers used to recover from landing impacts or perturbations.Reported examples include a 1.4 m long jump and a jump onto a 0.44 m platform.Source: B. Contributions · p. 2

II. RELATED WORK

  • Related work on Cassie jumping spans optimal control and reinforcement learning, the two categories compared in the paper’s benchmark.Source: §II · p. 2, Table I · p. 2

1) Model-based optimal control for legged jumping:

Prior work addresses legged jumping through model-based optimal control and model-free reinforcement learning, while sim-to-real methods commonly rely on simulation training and transfer. The paper positions its approach against limitations in task flexibility and real-world dynamic bipedal control.

1) Model-based optimal control for legged jumping:

  • Model-based jumping methods either prescribe contact sequences, limiting scalability across distances and directions, or use contact-implicit optimization; both remain limited to offline computation for legged robots.Source: 1) Model-based optimal control for… · p. 2
  • Many prior bipedal jump controllers use a separate landing controller, while most prior bipedal jumping work focuses on vertical jumps without controlled landing locations.The proposed controller is described as controlling landing pose without position feedback or explicit contact estimation.Source: 1) Model-based optimal control for… · pp. 2–3

2) Model-free RL for legged locomotion control:

  • Prior model-free RL work has learned versatile locomotion, but often relies on single-goal policies, reference-motion parameterization, or prescribed contact sequences that limit flexibility under perturbations.Source: 2) Model-free RL for legged locomotion… · p. 3

3) Sim-to-real transfer for legged robots:

  • For sim-to-real transfer, common approaches use proprioceptive history or teacher-student training; this work reports better learning performance from end-to-end training with its new policy structure than from teacher-student training.Teacher-student training separates the process and increases training time and data.Source: 3) Sim-to-real transfer for legged… · p. 3

A. Floating-base Model of Cassie

The paper describes Cassie, goal-conditioned reinforcement learning, and a three-stage training scheme for versatile jumps. The scheme progresses from a single jump to randomized targets and then dynamics randomization for transfer robustness.

A. Floating-base Model of Cassie

  • Cassie is a life-sized, dynamic, underactuated biped with 20 degrees of freedom, including a six-degree-of-freedom floating base and observable joint positions and velocities.It has five actuated motors and two passive joints per leg; the observable state has 26 components.Source: A. Floating-base Model of Cassie · p. 3

B. RL Background and Goal-Conditioned Policy

  • The goal-conditioned policy receives a target location, elevation, and landing turn direction, and selects actions based on state and the supplied goal.Elevation is defined as the change in floor height, not the change in robot base height.Source: B. RL Background and Goal-Conditioned… · p. 3

A. Overview of the Multi-Stage Training Schematic

  • Training proceeds from a single in-place goal to episode-randomized targets, then adds extensive dynamics randomization while retaining randomized goals to improve sim-to-real robustness.Stage 1 imitates a reference animation; training is performed in simulation and the resulting models can be deployed directly on real Cassie.Source: §IV · p. 3, A. Overview of the Multi-Stage… · p. 4, Figure 2 · p. 4

B. Reference Motion

Training starts from a single human-authored in-place jump reference, while a phase- and stage-weighted reward balances motion tracking, task completion, and smoothing. During multi-goal training, task completion is emphasized so the robot can learn varied maneuvers.

B. Reference Motion

  • A human-authored in-place jump animation initializes training; it is kinematically feasible but not optimized for dynamic feasibility.The animation has a 0.5 m apex foot height, 1.1 m apex pelvis height, and 1.66-second duration, then switches to a fixed standing pose.Source: B. Reference Motion · p. 4

C. Reward

  • Reward weights vary by jumping phase and training stage, combining reference-motion tracking, task completion, and smoothing.The reward switches at the reference-motion duration TJ between aggressive jumping and stationary standing objectives.Source: Table II · p. 4, C. Reward · p. 4
  • During multi-goal training, reference-motion tracking has relatively low weight so the robot can infer diverse maneuvers for different targets.The in-place reference may no longer be reasonable for jumps to different locations.Source: C. Reward · p. 4
  • Task-completion rewards encourage reaching the target location and orientation, while pelvis velocity and angular-rate tracking shape the sparse task reward.The reward also encourages zero pelvis roll and pitch to help stabilize the pelvis.Source: C. Reward · pp. 4–5
  • Smoothing rewards penalize impact force, post-landing oscillation, energy use, and action changes, with standing torque and motor-velocity penalties weighted more after landing.The post-landing penalties address oscillations that dynamics randomization can cause during standing.Source: C. Reward · p. 5
  • Despite a fixed reward-switch time, the robot learns different flight times for different targets.The larger foot-tracking tolerance also permits small post-landing hops within the task error bound, which the authors associate with better robustness.Source: C. Reward · p. 5

D. Episode Design

Multi-goal episodes alternate standing and jumps to randomly sampled targets, while early termination and motion imitation favor jumping over conservative alternatives. Stage 3 randomizes dynamics and sensing to address sim-to-real differences.

D. Episode Design

  • Repeated jumps to random targets after random standing intervals expose the policy to varied initial states and improve robustness.Targets are sampled over horizontal displacements, elevation changes, and turning directions; the standing interval is sampled from 1 to 15 seconds.Source: D. Episode Design · p. 5
  • A larger foot-height error tolerance allows small post-landing hops, while the task-completion bound still requires landing near the target.The authors state that hopping within the task bound can improve robustness, although they do not specifically train or encourage task deviation for robustness.Source: D. Episode Design · p. 5
  • A single-jump first stage and later early-termination conditions plus motion imitation bias the robot toward jumping to targets rather than walking.The episode can terminate for falls, excessive foot-height tracking error, or failing to reach the target after landing.Source: D. Episode Design · p. 5

E. Dynamics Randomization

  • Stage 3 randomizes dynamics parameters to address modeling errors, sensor noise, and communication delays that contribute to the sim-to-real gap.Randomized properties include friction, joint damping, link mass and inertia, center-of-mass position, and PD gains.Source: E. Dynamics Randomization · p. 5
  • A 20% uncertainty is introduced for Cassie’s leaf-spring stiffness, and the authors report that motor-dynamics and spring-stiffness randomization materially affects sim-to-real success.Joint damping is randomized up to four times its default value to approximate motor aging; PD gains vary by ±30%.Source: E. Dynamics Randomization · pp. 5–6

V. TRAINING SETUP

The jumping policy combines a base MLP with a CNN encoder for long-term I/O history. Its inputs include jump goals, reference-trajectory previews, and short- and long-term histories to support control of dynamic maneuvers.

A. Policy Architecture

  • The policy receives the goal, a reference-trajectory preview, short-term I/O history, and the previous 2 seconds of I/O history.The short history covers four timesteps, while the longer history contains 66 timesteps of observations and actions.Source: A. Policy Architecture · p. 6
  • The reference preview includes current pelvis height and reference motor positions sampled 1, 4, and 7 timesteps ahead.The authors say the preview gives the policy information about future joint positions, velocities, and higher-order terms.Source: A. Policy Architecture · p. 6
  • A 1D CNN compresses the long-term history into a latent representation that is concatenated with inputs to the base MLP.The MLP has two hidden layers with 512 tanh units and outputs the mean of a Gaussian action distribution with fixed standard deviation 0.1I.Source: A. Policy Architecture · p. 6
  • The 2-second history is intended to span a jump, helping the policy infer dynamics, trajectory, and contacts beyond the reach of short history alone.The authors note that landing can depend on angular momentum gained before take-off, with the interval between those events potentially much longer than 0.12 seconds.Source: A. Policy Architecture · p. 6

B. Training Details

The simulation benchmark compares policy structures under multi-goal training, while the training setup separates policies for flat-ground jumps and elevated-platform jumps. The benchmark evaluates learning performance and generalization to dynamics outside training distributions.

B. Training Details

  • The authors train separate policies because Cassie finds simultaneous turning and jumping to different elevations very difficult without a torso.The flat-ground policy has cz = 0; the discrete-terrain policy jumps to platforms of different elevations without turning, cϕ = 0.Source: B. Training Details · p. 6
  • PPO trains the policies in simulation for 6k, 12k, and 20k iterations across the three stages, collecting 65,536 samples per iteration.The value function is a two-layer MLP with access to ground-truth observations.Source: B. Training Details · p. 6

VI. SIMULATION VALIDATION

  • The simulation validation asks whether the proposed architecture improves on prior policy structures and whether multi-task training improves robustness to unstable states or unknown perturbations.Baseline policies are trained on multiple goals with three random seeds.Source: §VI · pp. 6–7, A. Baselines · p. 7

A. Baselines

  • The study compares multi-goal policies with single-goal training and single-goal training augmented by randomized perturbation wrenches.The two comparison criteria are learning performance and generalization to dynamics parameters outside the training distributions.Source: A. Baselines · p. 7
  • In Stage 3, the proposed method performs similarly to the privileged expert; A-RMA is second-best but uses significantly more samples, and residual policy performs worst.The curves show average normalized returns across three random seeds, with min–max ranges; the residual reference motion limits exploration of diverse maneuvers.Source: Figure 5 · p. 7

B. Policy Structure Choice

The policy performs best when it combines long-term I/O history encoding with direct short-term history access. Desired-position actions and end-to-end training also contribute to the proposed method’s learning performance.

B. Policy Structure Choice

  • A residual structure using a dynamically infeasible reference animation had the worst learning performance across training stages.The animation may consume learning effort correcting default motions and restrict exploration beyond its range.Source: B. Policy Structure Choice · p. 7
  • Combining long-term history encoding with direct short-term I/O access substantially enhances learning over using either history alone.The authors note that immediate feedback may be hidden from the long-history encoder.Source: B. Policy Structure Choice · p. 7
  • The proposed method degrades only a little relative to the privileged expert during randomized goal-conditioned training, while converged A-RMA returns remain a bit worse and require additional training and samples.The expert requires privileged information and cannot be zero-shot transferred to the real world.Source: B. Policy Structure Choice · p. 8
  • The ablation summary identifies desired motion positions as the action space, access to both short- and long-term I/O history, and end-to-end training as factors improving learning.The comparison with RMA and A-RMA used an added short-term I/O history and the same long-term encoder as the proposed method.Source: B. Policy Structure Choice · p. 8

C. Advantages of the Verstiale Policy

In unseen simulation tests, the multi-goal policy stabilizes Cassie where single-goal policies fail, including a policy trained with perturbations. On hardware, the transferred policies complete varied jumps and adjust their maneuvers to reach targets despite sim-to-real differences.

C. Advantages of the Verstiale Policy

  • Both single-goal policies fail in the two unseen tests, while the multi-goal policy stabilizes Cassie and selects learned lateral or forward jumps despite an in-place command.The single-goal policy trained with extensive perturbations also fails; tests apply lateral force or shift link CoM positions by +8 cm in all dimensions.Source: C. Advantages of the Verstiale Policy · p. 8

A. Task Completion in the Real World

  • Without finetuning or global position feedback, the flat-ground policy completes turning, backward, and 1 m forward jumps on hardware.On the forward jump, Cassie lands first at 0.5 m and then hops forward to the 1 m target, a maneuver not seen in simulation with nominal dynamics.Source: A. Task Completion in the Real World · p. 9, §VII · p. 9, Figure 7 · p. 9
  • The discrete-terrain policy accurately reaches 1 m and 1.4 m forward targets and a target 0.88 m ahead and 0.44 m above ground.It adjusts take-off and contact plans for distance and elevation changes and stabilizes across different landing events.Source: A. Task Completion in the Real World · pp. 9–10

B. Sim-to-Real Gap

  • Despite a large sim-to-real joint-profile deviation, including a significant difference in passive, leaf-spring-driven tarsus joints and delayed real-world flight phase, the policy robustly controls the robot to its target.Source: Figure 8 · p. 10

B. Sim-to-Real Gap

The policy succeeds on hardware despite substantial joint-profile differences between simulation and reality. Cassie also uses learned maneuvers and additional contacts to recover from perturbations and unstable landings while performing diverse jumps.

B. Sim-to-Real Gap

  • The turning task succeeds despite large sim-to-real joint-position deviations, including over 0.35 rad maximum error at the passive tarsus joint.The tarsus is driven by a leaf spring with nominal stiffness of 1250 Nm/rad.Source: B. Sim-to-Real Gap · p. 10

C. Diverse and Robust Maneuvers by the Goal-Conditioned Policy

  • Cassie performs flat-ground multi-axis jumps and uses small post-landing hops to adjust its pose from unstable states.The tested maneuvers include combining forward, lateral, and turning motions.Source: C. Diverse and Robust Maneuvers by the… · p. 10
  • After a backward perturbation at jump apex, Cassie deviates from an in-place command and uses a learned backward hop to land stably.The hop adjusts body pose during flight after landing leaves the robot underactuated relative to its contact points.Source: C. Diverse and Robust Maneuvers by the… · p. 10
  • The policy can break contact after landing and make contact again when needed to use impacts for stabilization, without a predefined contact plan.The authors contrast this online capability with contact-implicit trajectory optimization schemes that still require offline computation for legged robots.Source: C. Diverse and Robust Maneuvers by the… · p. 10
  • The discrete-terrain policy adjusts take-off maneuvers to land accurately on targets with different locations and elevations.The reported changes in commanded distance and elevation are relatively small.Source: C. Diverse and Robust Maneuvers by the… · p. 11

VIII. DISCUSSION OF DESIGN CHOICES FOR RL-BASED LEGGED LOCOMOTION CONTROL

The discussion attributes stronger learning and recovery to combining short- and long-term I/O histories and training across multiple goals. The conclusion reports direct simulation-to-hardware transfer for varied Cassie jumps, while noting occasional post-jump oscillations.

VIII. DISCUSSION OF DESIGN CHOICES FOR RL-BASED LEGGED LOCOMOTION CONTROL

  • Combining direct short-term I/O access with an encoded long-term history outperforms policies using only short- or long-term history in the reported ablation.Recent feedback can be obscured in a compressed long-term representation.Source: §VIII · pp. 11–12
  • Unlike RMA/Teacher-Student methods that estimate selected environment parameters, the jointly trained encoder uses past I/O directly and can capture impact events and contact wrenches.The discussion notes that RMA/TS may have advantages when combined with external sensors such as vision.Source: §VIII · p. 12
  • Multi-goal training supports recovery from perturbations and landing deviations that single-goal training with dynamics randomization does not readily provide.The conclusion also reports that this robustness facilitates sim-to-real transfer.Source: §VIII · p. 12, §IX · p. 12

IX. CONCLUSION

  • The framework trains goal-conditioned policies in simulation through three stages and transfers them directly to real Cassie for varied locations, elevations, and multi-axis jumps.The conclusion says the robot occasionally oscillates after jumping, possibly because one policy handles both dynamic jumps and stationary standing.Source: §IX · p. 12

APPENDIX

The appendix details staged reward and episode choices, dynamics randomization, and the policy structures used as baselines. Its comparison reports the proposed policy performing best in early training across single-goal and multi-goal stages.

1) Reward:

  • Stage 1 rewards emphasize tracking motor position and foot height to imitate a jump in place, while a small smoothing reward is included and action-change reward is omitted.The task-completion term is included with a fixed goal, c = 0.Source: 1) Reward · p. 16

2) Episode Design:

  • Stage 1 uses 750 timesteps (23 seconds), requiring a jump at episode start and standing afterward to encourage learning both behaviors.The stage uses a smaller foot-height tracking error bound (0.22 m) and a larger task-completion bound ([1.0, 45°]).Source: 2) Episode Design · p. 16

B. Details of Dynamics Randomization

  • Cassie’s reliable onboard sensors motivate relatively small training-noise bounds, including 0.1° for joint position and 0.5° for joint velocity.The text recommends larger training noise bounds for robots with greater sensor noise.Source: B. Details of Dynamics Randomization · p. 16

C. Details of Baseline Models

  • The proposed baseline jointly trains a CNN long-history encoder and MLP policy, while comparison policies use residual outputs, only one history duration, or teacher-student extrinsics prediction.RMA and A-RMA also receive short I/O history for a fair comparison; A-RMA additionally fine-tunes its base MLP with the history encoder fixed.Source: C. Details of Baseline Models · p. 16
  • The proposed policy shows the best early-stage learning performance for learning one goal from scratch and for multi-goal Stage 2.Figure 10 summarizes means across three random seeds, with min–max boundaries.Source: Figure 10 · p. 16

D. Learning Performance in Early Stages

In early training, the proposed policy matches or slightly exceeds a privileged-information expert, while history-only baselines show similar performance in multi-goal learning. Additional appendix material records further learning curves and hardware jump experiments.

D. Learning Performance in Early Stages

  • Without dynamics randomization, the proposed method performs similarly to or slightly better than the privileged-information expert in Stages 1 and 2; short-history-only learning is faster initially, but history-only methods are similar in Stage 2.The comparison reports learning curves rather than a numerical performance difference.Source: D. Learning Performance in Early Stages · pp. 16–17

E. Additional Learning Curves

  • Additional learning curves record single-goal policies trained with dynamics randomization and RMA variants using different long-history encoders.Figure 11 also compares the original RMA encoder structure with the encoder used in this work.Source: E. Additional Learning Curves · p. 17, Figure 11 · p. 17

F. Additional Hardware Experiments

  • Additional hardware experiments show the flat-ground policy performing more challenging jumps on Cassie, including an elevated forward target and varied targets.Figure 12 shows joint-position profiles in simulation and on hardware for a 0.44 m elevation 0.88 m forward; Figure 13 shows further target jumps.Source: F. Additional Hardware Experiments · p. 17, Figure 12 · p. 17, Figure 13 · p. 17
Loading 2302.09450v2…