Source-linked AI summary
Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
Nikita Rudin, David Hoeller, Philipp Reist, Marco Hutter
TL;DR
The paper asks how real-world robotic policies can be trained much faster than with conventional simulation pipelines. It combines GPU-based massive parallelism, PPO adaptations, and a game-inspired curriculum, achieving minute-scale ANYmal locomotion training and transferring policies to the real robot.
Problem
Deep reinforcement learning for complex robotic tasks can require days or weeks of training, making repeated policy development and deployment costly.
Method
The authors use an end-to-end GPU pipeline with thousands of simulated robots, modified on-policy PPO settings, and a game-inspired curriculum that adapts task difficulty.
Results
Under 20 minutes trains a perceptive ANYmal locomotion policy on challenging terrain, with the approach reducing training time by multiple orders of magnitude compared with previous work.
Takeaways & Limitations
Minute-scale training enables many training runs, setup simplification, and validation through transfer of the learned policies to the real robot.
Takeaways & Limitations
Imperfect terrain height maps reduce sim-to-real robustness, particularly at high velocities, so deployed policies use maximum linear velocity commands of 0.6 m/s.
Abstract
from arXiv · showhide
In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion.
1 Introduction
The paper addresses long training times in deep reinforcement learning by studying massive parallelism and a game-inspired curriculum for thousands of simulated robots. It applies this setup to fast, challenging legged-locomotion training and real-robot transfer.
- Motivation: 14 days for OpenAI’s block reorientation task and several months for its Rubik’s cube policy illustrate the long training times of current simulators.Training data requirements increase with task complexity, motivating simulation-based acceleration.
- Motivation: Massive parallelism can improve policy quality and time-to-deployment by allowing more training-setup iterations within the same time frame.This is especially relevant because hyper-parameter tuning requires sequentially rerunning training.
- Approach: The study examines how on-policy DRL formulations and hyper-parameters should be adapted for efficient learning in highly parallel regimes.The experiments use an Isaac Gym pipeline capable of simulating thousands of robots and training on the GPU.
- Approach: The proposed game-inspired curriculum automatically adapts task difficulty to policy performance without requiring tuning.Its architecture is designed for training with thousands of simulated robots in parallel.
- Results: Under 20 minutes trains a perceptive quadrupedal locomotion policy on a single GPU, including the demands of sim-to-real transfer.The approach targets challenging, unstructured terrain and validates the result on hardware.
2 Massively Parallel Reinforcement Learning
The training pipeline moves simulation, data collection, and policy updates toward GPU-parallel execution, while modifying PPO settings for thousands of robots. It also addresses throughput, batch sizing, mini-batches, resets, and terrain handling.
- Simulation Throughput: GPU-based simulation and training reduce data copying and improve simulation throughput by keeping the end-to-end pipeline on the GPU.CPU-based simulation and reward or observation calculation create communication bottlenecks for GPU policy inference.
- Simulation Throughput: Thousands of robots are simulated together, with terrain types and difficulty levels tiled on one mesh and robot levels changed by repositioning them.The number of parallel robots is identified as the main factor affecting total simulation throughput.
- PPO Adaptation: The custom PPO implementation stores data and performs operations on the GPU, with algorithmic and hyper-parameter modifications for massively parallel learning.The implementation is designed specifically for learning from thousands of robots in parallel.
- PPO Adaptation: Batch size B = n_robots n_steps must remain reasonable as robot count grows, requiring small but not arbitrarily low rollout lengths.Very short trajectories fail to provide the coherent temporal information needed for effective learning.
- PPO Adaptation: Tens-of-thousands-sample mini-batches stabilize learning without increasing total training time.These mini-batches are substantially larger than usual best-practice sizes in the reported massively parallel setting.
- Reset Handling: Time-out resets require bootstrapping the critic target with its own prediction because resets otherwise violate the critic’s infinite-horizon assumption.Failure-based and goal-based resets are treated differently because the critic can predict those terminations.
- Terrain Setup: Terrain training uses procedurally generated flat, sloped, rough, obstacle, and stair terrains, with randomized commands encouraging traversal across tiled terrain squares.The terrain set is intended to represent real-world environments.
3 Task Description
The task trains ANYmal and related quadrupeds to follow velocity and heading commands across procedurally generated terrains using an automatic curriculum that progressively increases difficulty. The policy combines proprioceptive and terrain observations with rewards promoting commanded motion, smoothness, collision avoidance, and longer steps.
- Task and terrains: ANYmal learns to walk across flat, sloped, rough, obstacle-filled, and stair terrains while following base-heading and linear-velocity commands.Simulation also covers ANYmal B, ANYmal C with an attached arm, and Unitree A1.
- Automatic curriculum: The automatic curriculum assigns robots terrain types and difficulty levels, then progressively increases terrain complexity during training.Stair and obstacle difficulty is increased by raising step height from 5 cm toward 20 cm.
- Automatic curriculum: After 1000 policy updates, robots reached the most challenging level across all terrain types; training continued to 1500 updates for convergence.At 500 updates, climbing stairs and traversing obstacles still required more training than crossing slopes or descending stairs.
- Observations, actions, and rewards: The policy observes proprioceptive state, previous actions, and 108 terrain-height measurements sampled around the robot’s base.Each terrain measurement is the distance from the surface to the robot’s base height.
- Observations, actions, and rewards: A nine-term weighted reward encourages commanded velocities and longer steps while penalizing unwanted base motion, torques, accelerations, target changes, collisions, and crashes.The same reward is used across all terrains, and actions are desired joint positions sent to motors through a PD controller.
- Sim-to-real additions: Sim-to-real training randomizes ground friction, adds measured observation noise, and applies random pushes to promote a more stable stance.Friction is sampled uniformly from [0.5, 1.25], and pushes accelerate the base up to ±1 m/s in x and y.
4 Results
The experiments identify a trade-off between parallelism, policy performance, and training time, then evaluate the resulting policy across challenging terrains, robot platforms, and hardware deployment.
- 4.1 Effects of Massive Parallelism: Too many robots sharply reduced performance, while larger batch sizes delayed the time-horizon-related performance drop.Fewer robots also reduced performance slowly, likely because their samples were less diverse and less independent.
- 4.1 Effects of Massive Parallelism: 2048 to 4096 robots with batch sizes near 100k or 200k provided the best trade-off between policy performance and training time for this setup.Training time scaled nearly linearly up to 4000 robots before simulation throughput gains slowed.
- 4.2 Simulation: Under 20 minutes of training with 4096 robots and a batch size of 98304 produced a policy that nearly always climbed stairs up to 0.2 m.Randomized obstacles were more demanding, and climbing success was not achieved beyond 25° slopes, although descending remained moderately successful.
- 4.2 Simulation: The policy converged to a trotting gait and, after reward-weight tuning, satisfied constraints sufficiently for physical transfer.Before tuning, behaviors could include a dragging leg or unusually high or low base heights.
- 4.2 Simulation: The approach retrained similar policies for ANYmal variants, adapted to the smaller Unitree A1, and trained Cassie on the same terrains with an added single-foot-standing reward.The simulations included ANYmal C with a fixed arm, ANYmal B, A1, and Cassie.
- 4.3 Sim-to-real Transfer: On hardware, imperfect terrain height mapping reduced robustness relative to simulation, especially at high velocities, so maximum linear commands were reduced to 0.6 m/s.The deployed robot nevertheless walked up and down stairs and handled obstacles dynamically.
5 Conclusion
The work demonstrates that complex real-world robotic tasks can be trained in minutes using massively parallel on-policy DRL, while explicitly prioritizing training speed over maximum robustness. The resulting setup uses simple spaces and rewards and remains usable on real hardware.
- Massively parallel on-policy DRL trains a complex real-world robotics task in minutes.
- The setup reduces training time by multiple orders of magnitude compared to previous work.
- The task is solved with simple observation and action spaces, straightforward rewards, and no encouraged gaits or motion primitives.
- The work prioritizes record-time training and usability on real hardware rather than the absolute best-performing, most robust policy.
A Appendix
Figure 8 measures environment-step computation and total learning-iteration time for a batch of B = 98304 samples.
- Figure 8(a) reports the computational time of an environment step.
- Figure 8(b) reports total learning-iteration time for a batch size of B = 98304 samples.
A.1 Simulation Throughput Analysis
The throughput analysis breaks down environment-step and learning-iteration time across robot counts, while additional techniques optimize simulation throughput. Increasing parallelism reduces most fixed-sample iteration costs, but learning time remains independent of robot count.
- Simulation is the most time-consuming environment-step component and slowly increases with the number of robots.
- Observation and reward computation is the second-slowest component and also slowly increases with the number of robots.
- Policy and actuator-network inference remain nearly constant as the number of robots increases.
- Increasing parallel robots decreases the time of all fixed-sample iteration subparts except learning, which is independent of robot count.
- Additional techniques are used to optimize simulation throughput.
A.1.1 Time Step
The appendix analyzes GPU memory requirements across robot counts and simulation timing constraints. The reported VRAM needs vary with rendering and terrain, while the timestep is constrained by actuator-network stability.
- The policy runs at 50 Hz and uses multiple actuator and simulator steps per policy step for stable simulation.
- The timestep cannot be smaller than 0.005 s, corresponding to four simulation steps per policy step.
A.1.2 Contact Handling
Contact-handling optimizations reduce simulation cost by simplifying collision geometry, improving terrain representation, and managing robot placement. These choices target the computational load of contact detection and handling.
- A.1.2 Contact Handling: Keeping only feet, shanks, knees, and the base as collision bodies reduces potential contact pairs.The optimization removes unnecessary collision bodies from the robot model.
- A.1.2 Contact Handling: Converting low-resolution height fields to triangle meshes enables steep or vertical terrain surfaces without requiring high terrain resolution.High-resolution height fields degrade simulation performance, whereas the mesh conversion corrects vertical surfaces.
- A.1.2 Contact Handling: Robot placement influences computational load because PhysX still detects inter-robot contacts even when those contacts are ignored.Spreading robots apart is beneficial, although the curriculum initially places them close together before they disperse.
A.2 Effect of Time-out Bootstrapping
The authors evaluate reward bootstrapping at time-outs using total reward and critic loss on flat and rough terrain. Bootstrapping lowers critic loss and raises total reward in both tasks.
- A.3 Effect of Time-out Bootstrapping: 10% to 20%: reward bootstrapping improves total reward for both flat and rough terrain tasks.Learning can succeed without bootstrapping, but the reported improvement occurs for both evaluated tasks.
- A.3 Effect of Time-out Bootstrapping: Critic loss is higher without reward bootstrapping at time-outs.The comparison covers training with and without the specified time-out handling.
- A.3 Effect of Time-out Bootstrapping: Total reward is lower without reward bootstrapping at time-outs.Figure 10 compares total reward and critic loss for the two handling conditions.
A.3 Reward Terms
This section provides the definitions and training specifications used by the experiments, including symbols, reward terms, PPO settings, adaptive learning rate computation, and observation noise scales.
- A.3 Reward Terms: Table 1 defines the symbols used in the paper.The supplied table caption identifies it as a symbol-definition table.
- A.3 Reward Terms: Table 2 defines the reward terms and introduces φ(x) := exp(−||x||2.The supplied caption contains the displayed definition fragment.
- A.3 Reward Terms: The z axis is aligned with gravity.This supplied passage states the coordinate convention used in the section.
- A.3 Reward Terms: Table 3 lists the PPO hyper-parameters used to train the tested policy.The table also references an adaptive learning rate based on KL-divergence and Algorithm 1.
- A.3 Reward Terms: Table 4 specifies observation-noise scales sampled from uniform distributions and added to observations.The caption describes the noise procedure for each observation element.