Source-linked AI summary
Rapid Locomotion via Reinforcement Learning
Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, Pulkit Agrawal
TL;DR
Real-time control for fast running on natural terrain is difficult when full-model trajectory optimization is too slow and reduced-order models may be unavailable. The paper learns an end-to-end policy with reinforcement learning and an automatic curriculum, achieving high-speed locomotion across terrains and disturbances while remaining limited to ground-plane body-velocity control without vision.
Problem
Real-time control in complex environments is challenging when efficient reduced-order models are unavailable and full-model trajectory optimization is too slow for fast running on natural terrain.
Method
The approach learns a direct sensory-observation-to-action policy with reinforcement learning and automatically expands velocity-command tasks through a curriculum.
Results
The policy sustained a top speed of 3.9 m/s on flat ground, averaged 3.4 m/s over a 10 m grass dash, and spun at 5.7 rad/s on flat ground while also spinning on ice.
Takeaways & Limitations
The system demonstrates rapid linear and angular locomotion across indoor and outdoor terrains using minimal sensing, including recovery from tripping and compensation for a malfunctioning motor.
Takeaways & Limitations
The system controls only ground-plane body velocity and lacks vision, limiting tasks requiring behaviors such as jumping, stair ascent, or avoiding pitfalls.
Abstract
from arXiv · showhide
Agile maneuvers such as sprinting and high-speed turning in the wild are challenging for legged robots. We present an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah, sustaining speeds up to 3.9 m/s. This system runs and turns fast on natural terrains like grass, ice, and gravel and responds robustly to disturbances. Our controller is a neural network trained in simulation via reinforcement learning and transferred to the real world. The two key components are (i) an adaptive curriculum on velocity commands and (ii) an online system identification strategy for sim-to-real transfer leveraged from prior work. Videos of the robot's behaviors are available at: https://agility.csail.mit.edu/
I. INTRODUCTION
High-speed locomotion across natural terrains is difficult because terrain variation and speed-dependent dynamics challenge real-time control. The paper uses reinforcement learning and an automatic velocity curriculum to learn a single controller that transfers from simulation to the real world.
- Motivation: Terrain variation and high-speed effects make fast running difficult for legged robots.Relevant effects include changing friction, softness, uneven geometry, actuator limits, large contact forces, and flight-phase body control.
- Motivation: Reinforcement learning learns a direct observation-to-action policy when full-model trajectory optimization is not possible in real time.The policy is trained across human-designed environments and reward functions using trajectories sampled from a full physics model.
- Results: A single neural network trained in simulation enabled sprinting, rough-terrain running, and spinning after zero-shot real-world deployment.The controller used the MIT Mini Cheetah platform and was implemented as a neural-network controller.
- Challenge: Expanding commanded velocities causes training failure because rapid linear and angular velocities are difficult and physically constrained.Uniformly sampling many challenging tasks yields little reward, while centrifugal force limits realizable velocity combinations.
- Contribution: An automatic curriculum expands velocity-command tasks while respecting locomotion constraints, improving learning of omnidirectional high-speed locomotion.It replaces manual ranking of velocity combinations whose difficulty depends on both system dynamics and the optimization algorithm.
- Results: 3.9 m/s was the highest sustained flat-ground speed reported for the MIT Mini Cheetah.The policy also averaged 3.4 m/s over a 10-meter grassy dash and spun on flat and icy terrain.
A. Control Architecture
The controller maps sensory histories and velocity commands to joint-position actions, which a PD controller converts into torques. Simulation randomization and teacher-student adaptation support transfer across uncertain robot and terrain dynamics.
- A. Control Architecture: The policy uses joint angles, joint velocities, gravity-vector orientation, previous actions, and velocity commands as inputs.The gravity-vector orientation is measured by the IMU, while joint states are measured by motor encoders.
- A. Control Architecture: The action is a 12-dimensional joint-position command for a PD controller with proportional gain 20 and derivative gain 0.5.The low gains were selected to promote smooth motions and were not tuned during the experiments.
- A. Control Architecture: The reward combines linear and angular velocity tracking with stability, smoothness, and safety terms.Penalties include body roll, pitch, height, torque, acceleration, action changes, self-collision, and joint-limit violations.
- A. Control Architecture: Table I organizes randomized domain-parameter ranges and velocity-command ranges generated by the curriculum strategy.The command curriculum is described as expanding tasks while respecting physical locomotion constraints.
- A. Control Architecture: Domain randomization varies body mass, center of mass, motor strength, ground friction, and ground restitution during simulation training.The policy is transferred to the real world without fine-tuning.
- B. Teacher-Student Training: The teacher receives domain parameters through an encoder, while the student infers an adaptation latent from recent observations and actions.The teacher and policy body are optimized with PPO; the student adaptation module is trained by supervised learning and reuses the teacher’s policy body.
C. Policy Optimization
The policy uses teacher–student neural networks for adaptive dynamics handling and learns velocity-conditioned locomotion with reward-based curriculum strategies. The curriculum begins with easier commands and expands sampling based on success rather than a fixed schedule.
- Policy architecture: The teacher encodes domain parameters into a latent vector, while the policy body predicts actions from that latent and the current observation.Both modules are neural networks optimized jointly with PPO.
- Policy architecture: The student estimates the teacher’s latent representation from a history of observations, enabling deployment without direct access to domain parameters.The identification module is trained so its latent predictions match the teacher encoder output.
- Policy architecture: The adaptation module uses a history of h = 15 observations and is trained simultaneously with the teacher using on-policy data.The authors report that high-speed running was not sensitive to these design choices.
- Curriculum strategy: Uniform sampling over a large command range fails because high-velocity commands are difficult and provide insufficient reward for learning.Small-range commands can be learned, whereas sampling over [−5.0, 5.0] leads to failure.
- Curriculum strategy: The adaptive curriculum updates command sampling from reward success, avoiding the manual retuning required by fixed schedules.The Box Adaptive curriculum maintains independent distributions over linear and angular velocity.
- Curriculum strategy: The Grid Adaptive curriculum models the joint distribution of linear and angular velocities because simultaneous running and turning is more demanding than either alone.It is designed to account for the interaction between command dimensions.
1) Box Adaptive Curriculum Update Rule:
The Box Adaptive curriculum samples linear and angular velocity commands independently and expands each marginal distribution when the corresponding tracking reward meets a threshold. Its updates add neighboring commands around successful regions.
- Box Adaptive Curriculum Update Rule: When a command’s tracking reward meets the threshold, probability density increases on neighboring commands in the relevant marginal distribution.Neighbors are adjacent elements in the discretized domain of each marginal distribution.
- Box Adaptive Curriculum Update Rule: If a command is among the most challenging commands in one distribution and meets the reward threshold, that distribution expands.The update therefore grows the command range locally along that dimension.
- Box Adaptive Curriculum Update Rule: The Box Adaptive curriculum tracks a smaller velocity range than the Grid Adaptive curriculum for all error thresholds.The comparison is shown in Fig. 3(a).
- Box Adaptive Curriculum Update Rule: The Box Adaptive curriculum samples linear and angular velocity commands independently from separate distributions.The resulting probability density in the velocity plane has a box-like shape.
2) Grid Adaptive Curriculum Update Rule:
The Grid Adaptive curriculum samples commands from a joint linear–angular velocity distribution and expands it through successful neighboring cells. Its 4-connected grid captures interactions between running and turning commands.
- Grid Adaptive Curriculum Update Rule: The Grid Adaptive curriculum samples linear and angular velocity commands from a joint distribution rather than independent marginals.The joint distribution is not constrained to factor into separate linear- and angular-velocity distributions.
- Grid Adaptive Curriculum Update Rule: After an episode, neighboring commands receive added probability density when the relevant velocity-tracking rewards meet the threshold.The update applies to neighboring commands that have not already been added.
- Grid Adaptive Curriculum Update Rule: Neighbors are defined by a 4-connected grid with resolution [0.5 m/s, 0.5 rad/s].The grid represents adjacent cells in the joint command domain.
- Grid Adaptive Curriculum Update Rule: When a challenging joint command meets the reward threshold, the distribution expands locally around that command.This update grows the feasible command region without assuming independent command dimensions.
E. Evaluation Metrics
The evaluation measures velocity tracking across a discretized forward-velocity and yaw-rate command plane. It summarizes the range of jointly trackable commands with a thresholded command-area metric.
- Evaluation Metrics: The controller tracks body velocity commands represented as points in the forward-velocity and yaw-rate plane.The plane is discretized into grid cells indexed by i and j.
- Evaluation Metrics: Each grid cell’s tracking error ϵij is the root mean square deviation averaged over trials.Experiments compute tracking error from 5 trials per grid cell.
- Evaluation Metrics: The command area is the region of the command plane whose tracking errors remain below a specified maximum error threshold ϵ0.It captures combinations of longitudinal and yaw velocity rather than either dimension in isolation.
- Evaluation Metrics: A larger command area indicates a greater range of speeds achievable below the same error threshold ϵ0.Command area has units of m/s · rad/s and is reported across five random seeds with standard-deviation error bars.
A. Curriculum Learning Enables High-Speed Locomotion
Curriculum learning enables the policy to acquire high-speed locomotion, while grid-based command sampling improves tracking by modeling feasible linear-angular velocity combinations. The resulting controller reaches high speeds indoors and outdoors, turns on ice, and transfers better with online system identification, though simulation still incompletely explains real-world performance.
- Curriculum Learning: The policy trained without curriculum fails because random exploration rarely produces fast motion and therefore provides little reward signal.
- Curriculum Learning: The Box Curriculum first trains on a small command distribution, then expands commands as tracking capability improves.
- Curriculum Learning: The Grid Curriculum achieves a larger command area by modeling joint feasibility between linear and angular velocities, including their inverse relationship at the turning limit.
- Real-world Testing: 3.9 m/s was the highest sustained indoor speed among three seeds, exceeding the previous 3.7 m/s record on the same robot.The policies with system identification and grid curriculum averaged 3.8 m/s across seeds.
- Real-world Testing: Online system identification reduced the sim-to-real gap, with average real speeds of 3.81 m/s versus 2.49 m/s without it.Simulation speed at a 6.0 m/s command was 5.46 m/s with system identification and 5.07 m/s without it.
- Real-world Testing: 5.7 rad/s was achieved for yaw control, while a single policy also maintained stability while spinning on icy terrain.
C. Ablation Studies
The ablations show that privileged information benefits high-speed tracking, while online system identification lets the student nearly match the teacher. Training only on flat ground can still support outdoor locomotion, but speed and uneven-terrain robustness trade off.
- Terrain-Robustness Ablation: Increasing terrain roughness during training shrinks the command area on flat ground, revealing a trade-off between rough-terrain robustness and flat-ground top speed.
- Impact of Online System Identification: Privileged information improves performance across all speeds, with the greatest benefit at high speeds.
- Impact of Online System Identification: The student policy using online system identification nearly matches the teacher policy’s command-area performance across speeds.
- Terrain-Robustness Ablation: Despite training only on flat ground, the policy remains sufficiently robust for deployment on various outdoor terrains.
V. RELATED WORK
Model-based controllers use simplified or expanded models for dynamic locomotion across selected terrains, while reinforcement learning learns locomotion policies without requiring manually designed reduced-order models. This work demonstrates end-to-end learned high-speed, omnidirectional locomotion with minimal sensing, while recognizing limits in outdoor evaluation and task scope.
- Model-based approaches progressed from hand-specified gaits and simplified models to layered controllers for subsets of rough, soft, and slippery terrains.
- 3.7 m/s: a whole-body ground-reaction-force controller enabled Mini Cheetah running, while Regularized Predictive Control supported high-speed spinning and tight cornering.
- Reinforcement-learning locomotion studies learned fixed-speed or velocity-tracking controllers on Minitaur, ANYmal, Cassie, and other platforms, extending robustness through sim-to-real methods.
- Unlike a concurrent fixed-schedule curriculum on forward velocity alone, this work uses reinforcement learning with online system identification to adapt implicitly across environments.
- Minimal sensing makes the controller suitable for typical quadrupeds and relatively inexpensive commercial robots, while enabling rapid locomotion across indoor and outdoor terrains and disturbances.
- The demonstrated system controls only ground-plane body velocity, lacks vision for advance planning, and was quantitatively characterized outdoors only to a limited extent.
APPENDIX
The appendix specifies training and reward references and introduces the Froude number for comparing legged-robot agility across hardware with different leg lengths.
- The PPO training parameters used in all experiments are provided in Table V.
- Froude number: Fr = v^2/(gl), using body velocity, gravity, and nominal leg length to characterize legged agility across systems.
- The reward terms for task, stability, and smoothness are provided in Table VI, adapted from prior work with minor changes.