PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedback Control For Cassie With Deep Reinforcement Learning

Zhaoming Xie, Glen Berseth, Patrick Clary, Jonathan Hurst, Michiel van de Panne

arXiv:1803.05580v2cs.RO

TL;DR

Bipedal locomotion is difficult to control, and prior DRL studies often use simplified models that do not clearly represent realizable robots. This paper formulates feedback tracking as an MDP and trains DRL controllers on a realistic Cassie model, demonstrating robust walking across disturbances and terrain, speed-policy interpolation, and improved robustness with interpolation.

  • Problem

    Prior model-based methods simplify robot dynamics, while many DRL studies use models unrelated to corresponding hardware, leaving performance on realizable bipeds unclear.

  • Method

    The paper formulates reference-motion feedback control as an MDP and uses DRL to learn policies that augment reference active-joint angles for low-level PD control.

  • Results

    The learned controller handles terrain up to h = 0.15 without falling versus h = 0.07 for the reference controller, and interpolated policies handle h = 0.22.

  • Takeaways & Limitations

    DRL can produce robust walking controllers on a realistic Cassie model, including speed variation through time-scaled references and policy interpolation.

Abstract

from arXiv · show

Bipedal locomotion skills are challenging to develop. Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions. In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity. Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics. However, current results in the machine learning literature are often based on ad-hoc simulation models that are not based on corresponding hardware. Thus it remains unclear how well DRL will succeed on realizable bipedal robots. In this paper, we demonstrate the effectiveness of DRL using a realistic model of Cassie, a bipedal robot. By formulating a feedback control problem as finding the optimal policy for a Markov Decision Process, we are able to learn robust walking controllers that imitate a reference motion with DRL. Controllers for different walking speeds are learned by imitating simple time-scaled versions of the original reference motion. Controller robustness is demonstrated through several challenging tests, including sensory delay, walking blindly on irregular terrain and unexpected pushes at the pelvis. We also show we can interpolate between individual policies and that robustness can be improved with an interpolated policy.

I. INTRODUCTION

The paper addresses whether deep reinforcement learning can produce realizable, robust walking controllers for an underactuated biped using a realistic Cassie model. It formulates feedback control as reinforcement-learning policy optimization and tests robustness across disturbances, terrain, speeds, and policy interpolation.

  • Model-based controllers use simplified dynamics and local feedback, which can limit robustness and prevent full exploitation of the robot’s capabilities.
  • The paper trains feedback controllers with DRL on a realistic model of Cassie by casting reference-motion tracking as an optimal policy problem for a Markov Decision Process.
  • Robustness is evaluated using sensory delays, blind walking over varied terrain, and unexpected pushes applied to the robot’s body.
  • Time-scaled reference motions produce controllers for different walking speeds, while interpolation enables speed changes between policies and can improve robustness.
  • DRL results have often relied on simplified or non-hardware-based models, leaving performance on realizable bipedal robots unclear.

III. BACKGROUND

The background frames reinforcement learning as optimization over a Markov Decision Process and describes policy-gradient methods for improving neural-network policies from sampled rollouts.

  • A Markov Decision Process is defined by state space S, action space A, transition function p, discount factor γ, and reward function r.
  • The transition function specifies system dynamics, while the reward function assigns a scalar value to each state transition.
  • A policy represents the action distribution conditioned on the current state, and reinforcement learning seeks the policy maximizing expected return.
  • Policy-gradient methods estimate the gradient of the objective from rollouts and iteratively update policy parameters by gradient ascent.

B. Feedback Control

Bipedal feedback control tracks an offline-generated nominal trajectory by computing online control inputs from the current and reference states, commonly through locally linearized dynamics.

  • Trajectory optimization produces nominal states and controls satisfying the robot’s dynamics before online tracking begins.
  • The feedback law computes control inputs from the current state and nominal reference state while minimizing distances in state and control spaces.
  • TVLQR formulates this tracking problem as a quadratic program based on linearization along the nominal trajectory.

C. Feedback Control interpreted as Reinforcement Learning Problem

The paper converts reference-trajectory feedback control into an MDP whose policy observes robot and reference states, then learns nonlinear corrections to reference joint targets for low-level PD control.

  • The MDP state combines the robot state with the reference state, while actions are drawn from the robot’s control-input space.
  • The transition function follows the robot dynamics and advances the reference, while the reward is defined as the negative tracking distance.
  • With unit discounting, linearized transitions, and quadratic metrics, the formulation reduces to TVLQR; the learned formulation instead supports nonlinear policies.
  • Cassie’s combined representation has 82 dimensions before omitting x position, yielding an 80D state space for the studied motion.
  • The policy outputs corrections δa added to reference active-joint angles, and a low-level PD controller tracks the resulting target angles.

B. Reference Motion and Simulation

The framework extends a two-step Cassie reference motion into continuous forward walking and trains an actor-critic policy with PPO to produce PD-control targets. Episodes begin from randomly selected reference poses, while trajectory samples support critic and actor updates.

  • Reference Motion: The reference motion contains two Cassie steps in the +x direction and is repeated with smoothly increasing pelvis position for continuous forward movement.
  • Episode Initialization: Episodes initialize Cassie at a randomly selected pose from the reference motion and terminate early after unstable or low-reward states.
  • Network: The actor and critic use two-layer neural networks with 256-unit hidden layers, ReLU activations, and a tanh-limited actor output.
  • Learning Algorithm: PPO optimizes the policy using on-policy, model-free reinforcement learning.
  • Learning Procedure: Before updates, trajectories are sampled from the current policy, and collected transitions and rewards estimate values and advantages for critic and actor updates.
  • Control Interface: The policy output δa is added to active-joint reference angles, producing target joint angles for PD control.

V. RESULTS

Experiments use a realistic simulated Cassie model with normalized observations, Gaussian exploration during training, and a reward combining reference tracking with spring stabilization. Training uses bounded episodes and scheduled optimization updates.

  • Simulation: Cassie is simulated in MuJoCo with 20 degrees of freedom, 10 actuators, and unactuated floating-base and spring joints, using a 0.5-meter two-step reference trajectory.
  • Observation Processing: Observation normalization uses means and standard deviations computed from 50,000 states sampled from random-policy trajectories.
  • Action Selection: Training samples actions from a Gaussian policy with diagonal covariance 0.018, while testing directly uses the actor output.
  • Reward: The reward combines active-joint tracking, pelvis position and orientation tracking, and shin-spring stabilization, with weights 0.5, 0.3, 0.1, and 0.1.
  • Termination: Episodes last at most 300 control steps, approximately 10 seconds, and stop when pelvis height or reward crosses the stated termination thresholds.
  • Speed Targets: Learning curves compare target movement speeds defined by the reference distance traveled per two-step stride, with the reference trajectory covering 0.5 meter per stride.
  • Optimization: Adam step sizes begin at 1e−3 for the actor and 1e−2 for the critic, then decrease by 1% after each iteration.

A. 3D Walking

The learned controller is trained on a 3D simulated Cassie model using a reference motion and achieves high reward during undisturbed walking.

  • Training: The policy reaches peak performance within the first 200 iterations, and training stops after 300 iterations in approximately 2.5 hours.
  • Undisturbed Walking: Without external disturbances, the learned controller accumulates about 250 reward out of a maximum possible 300.

B. Sensory Delay

The controller tolerates a small simulated sensory delay but fails quickly under a larger delay that also affects the PD-control loops.

  • Delay Robustness: 5 ms sensory delay yields an average reward of 247 over 10 runs.
  • Delay Robustness: With a 10 ms delay, the robot quickly falls under the assumption that the delay also affects the PD-control loops.

C. Terrain and Perturbation Test

The learned Cassie controller remains stable while walking blindly over sinusoidal terrain and recovering from substantial pushes, outperforming the manually tuned reference controller on both tests.

  • The learned controller walks blindly across uneven sinusoidal terrain, which is generated in MuJoCo.The terrain uses a sinusoidal function.
  • h = 0.15 is the learned controller’s maximum tested terrain ratio without falling, versus h = 0.07 for the reference controller.The height map is z = h sin x, where h is the center-to-peak terrain height ratio.
  • 140 N forward, 90 N backward, and 50 N lateral pushes lasting 0.2 s are recoverable for the learned controller.The reference controller copes with 50 N forward, 70 N backward, and 15 N lateral disturbances.

D. Different Speed

The framework learns controllers for time-scaled reference motions at multiple speeds and interpolates between policies to change gait speed and improve terrain robustness.

  • D. Different Speed: Controllers are trained for reference motions scaled to 0, 2, 3, and 4× the original speed.Each speed takes 3 to 5 hours of training, and the controllers successfully track the desired speed.
  • E. Interpolation: The interpolation schedule uses λ = 1 − 0.625t_sim until λ reaches 0.Here t_sim is the simulation clock.
  • E. Interpolation: The method successfully speeds up and slows down gaits between 0.5 m per stride and 1.0 m per stride, and between 1.0 m per stride and 1.5 m per stride.These transitions are produced by interpolating between policies trained for different target speeds.
  • E. Interpolation: h = 0.22 is handled with an adaptive interpolated policy on sinusoidal terrain, a significant increase over the non-interpolated case.The policy interpolates between 0.5 m-per-stride and 1.0 m-per-stride policies based on pelvis speed.

VI. CONCLUSION

The paper presents robust reference-motion imitation controllers for Cassie, including adaptive interpolation across walking speeds. The experiments remain limited to simulation and full-state feedback, with vision identified as a possible robustness improvement.

  • The framework learns feedback controllers that imitate reference trajectories and remain robust to large disturbances.The evaluation includes sensory delay, blind walking across varying terrain, and large pelvis pushes.
  • The framework can produce reasonable controllers from physically infeasible reference trajectories created by simple retiming.The paper also demonstrates interpolation between policies representing different walking speeds.
  • Adaptive interpolation between walking-speed policies yields more robust gaits.The paper demonstrates interpolation between policies representing different walking speeds.
  • The controllers rely on full state information, whereas real-world deployment requires estimating state from noisy sensor measurements.The authors are extending the framework to use output, or sensory, feedback.
  • The authors believe robustness could be improved by incorporating terrain knowledge through vision.
  • All experiments are simulated, and the authors plan real-robot evaluation with possible on-robot fine-tuning to address the reality gap.
Loading 1803.05580v2…