PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning

Joonho Lee, Jemin Hwangbo, Marco Hutter

arXiv:1901.07517v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Recovering from falls remains difficult for legged robots, while existing approaches often rely on predefined trajectories or simplified models. The paper uses a hierarchical, model-free RL controller with separate behaviors trained in simulation and deployed on ANYmal. The robot recovered from random falls within 5 seconds, with success higher than 97% across more than 100 tests.

  • Problem

    Fall recovery is challenging because uncertain contacts complicate model-based control, while existing methods often use handcrafted sequences or simplified models and require substantial engineering.

    Source: §I · p. 2

  • Method

    A hierarchical neural controller coordinates separately trained self-righting, standing-up, and locomotion policies, with a learned behavior selector and height estimator.

    Source: §I · p. 3, §IV · p. 8

  • Results

    ANYmal recovered from multiple random fall configurations within 5 seconds, and the recovery maneuver succeeded in more than 97% of more than 100 physical-system trials.

    Source: A. Robustness · p. 7, §IV · p. 8

  • Takeaways & Limitations

    The controller produced dynamic recovery maneuvers and switched seamlessly among three behaviors in response to disturbances on ANYmal.

    Source: §IV · p. 8

    The method was trained and tested only on flat ground and will fail on large inclinations or rough ground.

    Source: §IV · p. 8

Abstract

from arXiv · show

The ability to recover from a fall is an essential feature for a legged robot to navigate in challenging environments robustly. Until today, there has been very little progress on this topic. Current solutions mostly build upon (heuristically) predefined trajectories, resulting in unnatural behaviors and requiring considerable effort in engineering system-specific components. In this paper, we present an approach based on model-free Deep Reinforcement Learning (RL) to control recovery maneuvers of quadrupedal robots using a hierarchical behavior-based controller. The controller consists of four neural network policies including three behaviors and one behavior selector to coordinate them. Each of them is trained individually in simulation and deployed directly on a real system. We experimentally validate our approach on the quadrupedal robot ANYmal, which is a dog-sized quadrupedal system with 12 degrees of freedom. With our method, ANYmal manifests dynamic and reactive recovery behaviors to recover from an arbitrary fall configuration within less than 5 seconds. We tested the recovery maneuver more than 100 times, and the success rate was higher than 97 %.

I. INTRODUCTION

The paper addresses robust recovery from falls, where uncertain contacts make model-based or handcrafted approaches difficult to generalize. It presents a hierarchical neural controller whose policies are trained in simulation and deployed on ANYmal, with over 97% recovery success in more than 100 trials.

I. INTRODUCTION

  • Recovery is challenging because falls involve uncertain contact points, while optimization-based methods often require analytic dynamics and predefined contact sequences.Existing approaches use handcrafted sequences, simplified models, or added mechanisms, and require considerable engineering effort.Source: §I · p. 2

A. Overview

  • A learned height estimator addresses base-height estimation during deployment, including degenerated contact conditions when the robot is fallen.The behavior selector and three behaviors form the hierarchical controller; each behavior is a control policy.Source: §I · p. 3, A. Overview · p. 3

I. INTRODUCTION

  • The recovery controller was tested on quadrupedal robot ANYmal in more than 100 trials, with a success rate higher than 97%; it also coped with corner cases where previous solutions failed.Source: §I · p. 3

A. Overview

  • A behavior selector coordinates separate self-righting, standing-up, and locomotion policies rather than relying on one policy for multiple skills.The selector uses recent observations, commands, the previously selected behavior, and the previous action.Source: A. Overview · p. 3
  • The policies are trained individually in simulation, then the behavior selector and height estimator are trained using the pretrained behaviors.The simulated training uses a data-driven actuator model and contact solver; deployment is on the real robot.Source: A. Overview · p. 3

B. Feature Selection for the State Spaces

The state spaces are selected to avoid relying on estimates that become unreliable during falls. The self-righting controller excludes drifting linear position and velocity, while orientation is represented by gravity direction in the base frame.

B. Feature Selection for the State Spaces

  • When feet slip or lose contact, TSIF base position and linear velocity can drift, so both are excluded from the self-righting state space.TSIF recursively fuses kinematics and IMU measurements and uses foot contacts as kinematic constraints.Source: B. Feature Selection for the State… · p. 4
  • Base orientation is represented by gravity direction in the base frame because the IMU observes only a two-dimensional orientation subspace and yaw is unobservable.Base x- and y-position are also excluded because they are unobservable with the IMU.Source: B. Feature Selection for the State… · p. 4
  • Base height is retained in the locomotion policy because it can be accurately estimated using kinematics while walking and is critical for fast learning.A new estimation network is used to estimate height.Source: B. Feature Selection for the State… · p. 4

C. Cost terms

The method defines separate bounded-cost RL tasks for self-righting, standing up, and locomotion. Their initial states, targets, and cost terms reflect each behavior’s purpose, while the action representation encourages task-appropriate exploration.

C. Cost terms

  • Each behavior has its own cost function, initial-state distribution, and termination conditions, while bounded cost functions help prevent termination from being more rewarding than exploration.The logistic kernel allows relative cost-term importance and sensitivity to error to be adjusted through its parameters.Source: 1) Tasks · p. 4, C. Cost terms · p. 4

1) Tasks:

  • Self-righting regains an upright base pose from arbitrary configurations and repositions the joints into a sitting configuration with all feet grounded for a safe stand-up.Its orientation cost receives the highest weight; contact impulses and joint accelerations are penalized to avoid violent motions and encourage smooth motions.Source: 1) Tasks · p. 4, Figure 3 · p. 5
  • Standing up starts from arbitrary sitting configurations and targets the standing joint configuration, adding a height cost to the self-righting-style cost terms.Initial states are sampled using the same strategy as self-righting but from a near-upright pose; termination is by time limit.Source: 1) Tasks · p. 5
  • Locomotion tracks sampled forward, lateral, and turning-rate commands while penalizing velocity errors, foot motions, and constraint violations.The command ranges are U(−1, 1) m/s, U(−0.4, 0.4) m/s, and U(−1.2, 1.2) rad/s, respectively.Source: 1) Tasks · p. 5
  • The controller outputs a 12-dimensional real-valued vector mapped to position targets for low-impedance PD joint controllers.Position-target actions are cited as achieving better final performance, faster learning, and higher robustness than torque or velocity targets.Source: 1) Tasks · p. 5

2) Action Representation:

For self-righting and standing up, the policy output is scaled and added to current joint positions, encouraging joint-space exploration and faster self-righting learning.

2) Action Representation:

  • For self-righting and standing up, desired joint positions equal a scaled policy output plus the current joint positions.This mapping promotes exploration in joint spaces and results in faster self-righting learning.Source: 2) Action Representation · p. 5
  • Adding the scaled output to current joint positions promotes exploration in joint spaces.Source: 2) Action Representation · p. 5
  • The action mapping is reported to result in faster learning of self-righting.Source: 2) Action Representation · p. 5

3) Architecture:

The controller combines separately trained behavior policies with a selector that chooses which behavior to execute. Its selector is trained to return ANYmal to a locomotable state, with costs encouraging power efficiency and smooth transitions.

3) Architecture:

  • Each policy uses a two-layer feed-forward network; the self-righting and standing-up policies have 128 units per hidden layer, while locomotion uses 128 and 256.Source: 3) Architecture · p. 5

4) Training:

  • Policies are trained separately with TRPO+GAE, while torque, velocity, acceleration, and action-difference penalties increase during curriculum learning.Without the gradually increased terms, the agent converged to a local minimum where it stopped moving.Source: 4) Training · p. 5

E. Behavior Selector

  • The behavior selector chooses which behavior to execute based on the current situation.It is trained to return ANYmal to a nominal operating state, defined as a state where it can locomote.Source: E. Behavior Selector · p. 5, 1) Task · p. 5

1) Task:

  • The selector's cost function includes power-efficiency and smooth-transition terms alongside costs similar to those used for locomotion.The power term encourages energy-efficient behavior; action-difference and torque costs encourage smooth transitions.Source: 1) Task · p. 5
  • Selector training samples initial states from the initial-state distributions of randomly selected behaviors.Source: 1) Task · p. 5

2) State and Action Spaces:

The selector combines behavior state spaces and outputs a probability distribution over behaviors. It is trained with pre-trained behaviors, while a learned height estimator addresses unreliable base-height estimates during falls.

2) State and Action Spaces:

4) Training:

  • The selector is trained with TRPO+GAE using pre-trained behaviors as part of the environment, while the height estimator is trained concurrently to match state-visitation frequency.Source: 4) Training · p. 5

F. Height Estimation

  • Base-height estimates become unreliable when ANYmal falls, so a neural network estimates height from body orientation and joint positions instead.Forward kinematics supplies the training target under a flat-ground assumption and known link geometry.Source: F. Height Estimation · pp. 5–6
  • The height estimator is trained from stored state–true-height pairs sampled from replay memory.After a warm-up period, the selector uses estimated height; the algorithm collects true height and updates the estimator from sampled pairs.Source: F. Height Estimation · p. 6
  • Linear-velocity estimates had the same issue, but their errors were not significant.Source: F. Height Estimation · p. 6

G. Handcrafted Behavior Selector

The handcrafted alternative uses a finite-state machine in which behaviors are states and transitions are designed by a domain expert. It waits for TSIF to converge after self-righting.

G. Handcrafted Behavior Selector

  • The handcrafted selector treats behaviors as FSM states and uses expert-designed transitions, which can also be set through trial and error.After self-righting, it waits 0.5 seconds for TSIF to converge.Source: G. Handcrafted Behavior Selector · p. 6

H. Simulating ANYmal

The ANYmal simulator uses RaiSim with actuator models and stochastic components to account for modeling errors. Randomized physical properties and a data-driven actuator model are used to support simulation-to-real policy transfer.

H. Simulating ANYmal

  • The simulator uses RaiSim and includes actuator models and stochastic components intended to account for modeling errors and robustify policies.Source: H. Simulating ANYmal · p. 6
  • Collision geometry and friction are randomized because the simulator cannot accurately reproduce material properties.Friction is sampled uniformly from 0.8 to 2.0 each time step.Source: H. Simulating ANYmal · p. 6

1) Randomized Physical Properties:

  • Link masses are randomized by up to 10% and the base center of mass by up to 3 cm in each axis per episode.The approach uses CAD-derived physical properties while randomizing the simulation to address modeling errors.Source: 1) Randomized Physical Properties · p. 6

2) Actuator Model:

  • A two-layer neural network trained on real data predicts joint torque from position commands and histories of joint-position errors and velocities.The model uses 32 softsign units and represents the dynamics of ANYmal's Series Elastic Actuators.Source: 2) Actuator Model · p. 6

3) Additive Noise to the Observation:

The observation-noise setup adds sensor-like noise during simulation and increases base-state noise when all legs lose contact. Implementation details include asynchronous behavior policies, a history buffer, and a freeze period before control begins.

3) Additive Noise to the Observation:

  • Training added up to 0.2 m/s linear-velocity, 0.25 rad/s angular-velocity, 0.5 rad/s joint-velocity, and 0.05 rad joint-position noise.Base linear-velocity and position noise increases when all four legs lose contact to replicate TSIF behavior.Source: 3) Additive Noise to the Observation · p. 6
  • ANYmal is shown recovering from arbitrary configurations and reacting to a kick.The passages identify these as Fig. 6 and Fig. 7, respectively.Source: Figure 6 · p. 7, Figure 7 · p. 7

I. Implementation Details

  • The self-righting, standing-up, and locomotion policies run at 20, 100, and 200 Hz, while the behavior selector runs at 50 Hz.The height estimator runs synchronously with the 400 Hz TSIF; a selected behavior's output is computed immediately, so self-righting does not preserve its lowest-frequency time step after a switch.Source: §I · p. 7
  • ANYmal waits in freeze mode while a 400 Hz history buffer fills with 0.05 seconds of joint-measurement history.The policies require this history as input.Source: §I · p. 7

III. RESULT AND DISCUSSION

  • The behaviors for standing up and locomotion are not assessed in detail here; the paper refers readers to [13] for locomotion analysis.Source: §III · p. 7

A. Robustness

ANYmal recovered from more than 100 falls with a success rate above 97%, including tests from varied ground configurations and responses to disturbances. The section also compares simulated and real behavior and reports that the learned height estimator tracks simulated height more accurately than TSIF.

A. Robustness

  • All 50 tested ground configurations recovered to an upright base pose within 5 seconds, including cases with trapped legs or an almost upside-down base.With trapped legs, ANYmal flips to its side to free them before regaining an upright pose.Source: A. Robustness · p. 7
  • During walking or standing disturbances, ANYmal switched smoothly to self-righting or standing-up behavior without noticeable delay.Source: A. Robustness · p. 7
  • More than 100 falls produced a recovery success rate higher than 97%; self-righting failed when a joint position was ≥2π.The reported failure was attributed to that joint position being rarely experienced during training; the authors note modulo 2π as a possible fix.Source: A. Robustness · p. 7

B. Comparison to Simulation

  • Behavior-switch timing in simulation and reality was nearly the same, and the motions were visually identical in the comparison.The comparison used the same initial state and velocity command; the figure caption says motions were not hand-tuned to match.Source: B. Comparison to Simulation · p. 7, Figure 8 · p. 8
  • The height estimator's RMS error was less than 1 cm against simulated height, while TSIF showed large initial error and fluctuated during self-righting.The experiment lacked motion-capture measurements, so experimental height could not be measured; without the estimator, height error sometimes caused an undesirable behavior switch.Source: B. Comparison to Simulation · pp. 7–8, Figure 9 · p. 8, Figure 10 · p. 8

C. FSM behavior selector

The discussion identifies limitations of the FSM selector, including missed corner cases and unsmooth transitions, while the conclusion reports dynamic recovery and seamless behavior switching. The proposed method is bounded to flat-ground training and testing.

C. FSM behavior selector

  • The FSM missed corner cases despite tuning, produced unsmooth transitions, and had no assessed success rate.The illustrated corner case switches to standing-up at a bad time and falls; the authors say additional states or transitions could improve performance.Source: C. FSM behavior selector · p. 8, Figure 11 · p. 8
  • The FSM still requires design iterations and experiments on the real robot.Source: C. FSM behavior selector · p. 8

IV. CONCLUSION

  • ANYmal recovered from multiple random fall configurations within 5 seconds and switched seamlessly between three behaviors in response to disturbances.The controller's dynamic recoveries involved multiple ground contacts and motions consistent with simulation; robustness was tested more than 100 times on the physical system.Source: §IV · p. 8
  • The proposed method was trained and tested only on flat ground and will fail on large inclinations or rough ground.Source: §IV · p. 8

Related papers

Loading 1901.07517v1…