Source-linked AI summary
Learning agile and dynamic motor skills for legged robots
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter
TL;DR
Controlling legged robots is difficult because their high-dimensional, contact-rich dynamics challenge conventional and specialized methods. This paper trains neural-network policies in simulation and transfers them to ANYmal, achieving more precise, energy-efficient locomotion, a 25% speed-record improvement, and dynamic recovery from falls.
Problem
Legged-robot control remains challenging because changing contacts, inaccurate dynamics, sensor delays, and physical constraints make conventional control insufficient and specialized methods laborious to design and tune.
Method
The authors train neural-network control policies in simulation and transfer them to the physical ANYmal quadruped to learn agile, dynamic motor skills.
Results
25%: the learned controller exceeded ANYmal’s previous speed record while also producing more precise, energy-efficient motion and dynamically restoring the robot after falls.
Takeaways & Limitations
Simulation-trained policies can deliver agile locomotion on a complex quadruped while reducing reliance on prolonged physical-robot controller tuning.
Takeaways & Limitations
A single neural network trained in one session produces single-faceted behaviors that do not generalize across multiple tasks.
Abstract
from arXiv · showhide
Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is complicated and expensive. In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes. The approach is applied to the ANYmal robot, a sophisticated medium-dog-sized quadrupedal system. Using policies trained in simulation, the quadrupedal machine achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.
INTRODUCTION
The paper addresses the difficulty of controlling complex legged robots by combining simulation-based learning with simulation-to-reality transfer. Applied to ANYmal, the method produces accurate, efficient locomotion, record speed, and dynamic fall recovery.
- Problem: Legged robots are difficult to control because their contacts vary with each maneuver, analytical dynamics are uncertain, and their sensing and software systems are complex.These systems are high-dimensional, non-smooth, and subject to many physical constraints.
- Motivation: Reinforcement learning can optimize controllers end-to-end, but real-robot training is impractical because it requires prolonged interaction and may produce chaotic, unsafe behavior.Consequently, advanced legged-locomotion reinforcement learning has largely remained in simulation, while deployment is hindered by the reality gap between simulated and real dynamics and perception.
- Method: The methodology learns and transfers agile, dynamic motor skills to ANYmal, a large and difficult-to-control quadruped with a relatively long leg length and 12 series-elastic actuators.These characteristics make ANYmal more dynamic, less statically stable, and harder to control than the robots used in earlier work.
- Results: ANYmal follows base velocity commands more accurately and energy-efficiently than the best prior controller, runs 25% faster than its previous speed record, and performs dynamic fall recovery.The recovery maneuver coordinates all limbs, uses momentum to flip the robot, and had not previously been achieved on a quadruped of comparable complexity.
RESULTS
On the real ANYmal robot, simulation-trained policies enabled accurate, robust command following, higher-speed locomotion, autonomous recovery from difficult falls, and stable control that surpassed model-based alternatives. The learned controller also exploited the hardware’s torque and velocity limits while discovering distinct agile gaits.
- Command-conditioned locomotion: The policy followed random velocity commands robustly for about 40 seconds under repeated external pushes and for five minutes without failure.The policy adapted its gait to runtime commands without prior knowledge of command timing or sequence.
- Command-conditioned locomotion: The learned controller was more accurate than the prior controller across all commanded velocities, with Fig. 2 comparing velocity error, power efficiency, and torque magnitude.The comparison used flying trot and dynamic lateral walk gaits from the prior method.
- Actuator-model ablation: Both ideal-actuator and analytical-actuator alternatives failed to complete a single step without falling, with violent limb shaking observed.The reported failure was attributed probably to unmodeled actuator delays and limited bandwidth effects.
- High-speed locomotion: 1.5 m/s was reached on the physical system under a 1.6 m/s command, while the controller used the maximum 40 Nm torque and 12 rad/s joint velocity.The corresponding simulation speed was 1.58 m/s, and the experiment showed that the policy exploited the hardware’s full capacity.
- Autonomous recovery: ANYmal successfully flipped itself upright from all nine tested random configurations, including nearly upside-down and complex self-contact poses.The recovery policy was tested on the real robot.
DISCUSSION
The simulation-trained learning approach achieved precise, energy-efficient locomotion and dynamic recovery without tedious physical-robot tuning. Its advantages include robustness, low onboard computation, and broad applicability, while task design, robot modeling, and multi-task generalization remain limitations.
- The approach achieved more precise and energy-efficient motions, exceeded the prior speed record by 25%, and consistently restored the robot by dynamically rolling over its body.
- The framework applies to any rigid body system and requires a task description comprising a cost function and initial-state distribution for new tasks.
- Learned actuator dynamics reduced the reality gap, stochastic modeling encouraged conservative policies, and recovery success reached 100% after relaxing joint-velocity constraints.
- The policies remained robust for more than three months on the real robot without modification despite hardware wear, configuration changes, and heavy use.
- Inference required less than 25 µs on one CPU thread, while prior controllers often required two orders of magnitude more onboard computation.
- Limitations include task-specific cost-function and initial-state design, robot modeling requirements, and single-session policies that do not generalize across multiple tasks.
METHOD
The method trains a history-dependent neural control policy in a fast, accurate rigid-body simulation and transfers it to ANYmal. It combines actuator-system identification, observable state histories, TRPO, and curriculum shaping to support robust locomotion learning.
- METHOD: The training loop simulates robot dynamics, buffers joint histories, and uses a multilayer perceptron policy to select actions from recent observations.The policy operates in discretized time, with observations consisting of robot-state measurements and actions consisting of actuator position commands.
- METHOD: A hard-contact rigid-body solver provides accurate Coulomb-friction dynamics while generating about 900,000 simulated time steps per second for efficient large-scale training.The solver was needed because training the complex policy required hundreds of millions of samples.
- METHOD: The method models actuator hardware and software dynamics with supervised actuator networks that predict joint torque from histories of position errors and velocities.Each actuator is modeled separately under the assumption that actuator dynamics are independent; the network uses three hidden layers of 32 units and softsign activations.
- METHOD: TRPO optimizes a stochastic policy using recent observations, with default algorithm parameters and rewards designed to induce the desired locomotion behavior.The observation history includes measurable joint angles, velocities, and body twist, while joint-state history supports contact detection without adding force sensors.
- METHOD: A curriculum initially shapes the cost landscape toward locomotion and progressively adjusts cost coefficients and disturbances to refine the motion.The curriculum progression is controlled by a multiplicative curriculum factor.
CONCLUSION
Controllers trained in simulation within a few hours outperformed the best existing model-based controller on the same robot, while achieving faster, more precise, and more efficient locomotion. The approach also enabled dynamic roll-over recovery and is presented as a step toward resilient, versatile locomotion controllers for legged robots.
- CONCLUSION: Simulation-trained controllers outperformed the best existing model-based controller on the same robot, running faster and more precisely while using less energy, torque, and computation.The controllers were trained in a few hours, whereas the compared model-based controllers had been designed and tuned over many years.
- CONCLUSION: The recovery controller achieved dynamic roll-over involving multiple unspecified contacts with the environment, a behavior not previously achieved on a comparable real robot.
- CONCLUSION: The approach is not fundamentally limited to known and simple environments and represents a step toward comprehensive locomotion controllers for resilient and versatile legged robots.
SUPPLEMENTARY MATERIALS
The supplementary materials define terminology, evaluation command sampling, training cost terms, and joystick-based velocity-tracking performance for the learned controller.
- Supplementary materials: The supplement provides nomenclature, random command sampling for controller evaluation, and cost terms for command-conditioned locomotion, high-speed locomotion, and fall recovery.These topics are organized in Sections S1–S4.
- Supplementary materials: Fig. S1 reports the learned controller’s base-velocity tracking performance while following random joystick commands.
SUPPLEMENTARY MATERIALS
The supplementary materials define notation for curriculum and cost terms, velocities, angular quantities, foot states, contacts, impulses, and gap functions.
- The notation includes a curriculum factor kc and coefficient c for a cost term.
- It defines linear and angular velocities, desired quantities, joint torque, and foot velocity and position variables.
- It specifies contact-related notation for contact-point velocities, impulses, gap functions, contact index sets, and set cardinality.
S2. Random command sampling method employed for evaluating the learned command-conditioned controller.
To accommodate the limited experimental area, the evaluation samples velocity commands while simulating the resulting ideal position trajectory to avoid driving the robot beyond the available physical space.
- S2. Random command sampling method employed for evaluating the learned command-conditioned controller.: The method samples a velocity command from table S1’s distribution, then simulates the body’s ideal position trajectory before proceeding.This scheme addresses the risk that a sufficiently long unconstrained random command sequence would move the robot outside the physical experimental area.
S3. Cost terms for training command-conditioned locomotion and high-speed locomotion
The training design uses bounded logistic-kernel costs to make tracking errors less likely to encourage termination, with task-specific command and randomized initial-state distributions. Learned controllers track randomly varying base-velocity commands with reasonable accuracy.
- Cost terms: The logistic kernel bounds tracking-error rewards and avoids the high initial costs of Euclidean norms that can make falling more rewarding than continuing training.The authors found the logistic kernel more useful because it keeps costs lower-bounded by zero and makes termination less favorable.
- Cost terms: The training cost design includes time-step-scaled terms for integrated objectives, including base linear velocity, torque, joint-position, and foot-related costs.Joint-position costs use minimum angular differences, while the listed coefficients specify terms such as torque cost cτ = 0.0005∆t and HFE cost cHFE = 7.0∆t.
- Training distributions: Random command distributions are used for both command-conditioned and high-speed locomotion, with forward velocity varied most widely in the high-speed task.The command ranges were selected to match existing-controller capabilities, while the high-speed task emphasizes forward velocity.
- Training distributions: Initial states are randomized during training to make the command-conditioned and high-speed locomotion policies more robust.The initial-state distribution is presented separately for the two locomotion controllers.
- Tracking performance: The learned controller follows randomly issued forward, lateral, and yaw-rate commands with reasonable accuracy.Figure S1 compares commanded velocities, shown as dotted lines, with measured velocities, shown as solid lines.