Source-linked AI summary
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, Dieter Fox
TL;DR
Complex Deep RL tasks require large numbers of samples, while distributed systems have typically left environment simulation on CPUs. This paper uses GPU-accelerated physics simulation to train many locomotion agents in parallel, reporting humanoid training in under 20 minutes on one GPU and CPU core and scaling to harder multi-GPU settings.
Problem
Deep RL’s high sample complexity motivates distributed computation, but continuous-control environment simulation has largely remained CPU-based.
Method
The paper builds a GPU-accelerated RL framework using a GPU physics engine to simulate hundreds to thousands of robot agents in parallel.
Results
A single GPU and CPU core trained the Humanoid running task in less than 20 minutes, while the framework also scaled to multi-GPU and multi-node settings.
Takeaways & Limitations
GPU-accelerated RL simulation is presented as a viable alternative to CPU-based simulation, with greater learning-speed improvements on more complex locomotion tasks in multi-GPU settings.
Takeaways & Limitations
The paper focuses on applying the GPU physics engine to RL simulation rather than comparing or benchmarking the physics engine itself.
Abstract
from arXiv · showhide
Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks.
1 Introduction
Deep RL’s high sample complexity has driven distributed approaches that scale training and simulation, but environment simulation has largely remained CPU-based. This work proposes GPU-accelerated simulation and reports faster training with substantially fewer CPU resources.
- Deep RL for complex, high-dimensional tasks requires enormous compute because exploration has high sample complexity.
- Distributed RL approaches commonly train policies on GPUs while scaling environment simulation across many CPUs.
- Experiments benchmark ant and humanoid running tasks, including changing targets, fall recovery, and complex uneven terrains.
- The proposed framework uses a GPU-accelerated physics simulator to concurrently simulate and train hundreds to thousands of agents.
- The authors report faster training than prior CPU-cluster methods and describe GPU-accelerated simulation as a viable alternative to CPU simulation.
- The paper focuses on applying the GPU physics engine to RL simulation rather than benchmarking the physics engine itself.
2 Related Works
Prior distributed RL work has scaled learning with CPU clusters, GPU policy computation, and distributed frameworks, while continuous-control simulation has remained CPU-based. The paper positions GPU simulation as an alternative for scaling locomotion training.
- Gorila DQN and A3C parallelized RL across CPU resources, with Gorila DQN using hundreds of cores and A3C using 16 cores for Atari tasks.
- GA3C moved policy inference and training to a GPU, achieving more than 4× speed-up over CPU-only A3C on one machine.
- Evolutionary strategies trained a humanoid to walk in 10 minutes with 1440 CPU cores, while genetic algorithms trained humanoid locomotion much slower.
- Ray-based distributed methods learned humanoid walking in 3.7 minutes with more than 8000 CPU cores for ES and 21 minutes with 48 CPU cores for ARS.
- IMPALA scaled multi-task video-game learning to 500 CPU actors and 8 GPU learners, illustrating distributed learning beyond locomotion simulation.
- Previous continuous-control studies used CPU-based simulation, motivating GPU-accelerated simulation for locomotion tasks that otherwise required hundreds or thousands of CPU cores.
3 GPU-Accelerated Robotics Simulation
The framework places many robot environments in one GPU simulation and performs their physics computations in parallel. Its measured throughput scales favorably, but the agent count that maximizes learning speed depends on the task.
- The physics engine uses a non-smooth Newton rigid-body solver, maximal coordinates, torque control, unilateral contact constraints, and smooth isotropic Coulomb friction.
- Time-stepping uses implicit discretization, with each Newton iteration solved by a sparse iterative Krylov linear solver.
- Multiple robots and task objects are loaded into one simulation, with GPU physics computations providing parallelism and allowing agent interactions.
- GPU simulation frame-time measurements increase the number of concurrently simulated humanoids on an NVIDIA Tesla V100, excluding reward, action, and Python data-transfer overheads.
- Peak simulation throughput does not determine optimal learning speed because agent count affects exploration and the best threshold depends on the task.
4 Experiments
The experiments evaluate GPU-accelerated simulation for continuous-control locomotion across single- and multi-GPU settings, varying parallel agents and task complexity. Training uses PPO-based methods, fixed experience per update, and comparisons of learning speed and resource use.
- Experimental setup: The GPU simulator concurrently runs hundreds to thousands of agents, with experiments measuring learning as parallel-agent counts vary on one GPU and across multiple GPUs and nodes.Multi-GPU experiments run a simulation and training instance on each GPU and average gradients with Horovod.
- Tasks: The evaluation covers Ant, Humanoid, Humanoid Flagrun Harder, and Humanoid Flagrun Harder on Complex Terrain locomotion tasks.The tasks range from forward running to changing targets, fall recovery, and uneven terrain with obstacles.
- Experimental setup: PPO experiments keep 32768 frames per optimization iteration constant while changing agent counts, isolating learning-speed differences from experience-volume changes.For example, 512 agents use 64 frames per iteration, whereas 2048 agents use 16.
- Results: With one GPU and one CPU core, Humanoid training finishes in less than 20 minutes, using 10× to 1000× fewer CPU cores than prior CPU-based works.The single-GPU Humanoid result is compared with prior works using CPU-based physics simulation.
5 Conclusion and Future Work
The paper presents GPU-accelerated simulation as a viable alternative to CPU-based simulation for continuous-control locomotion and reports scaling to multi-GPU and multi-node settings. Future work targets more complex environments, additional robot-manipulation tasks, and GPU-resident vision-based training.
- Conclusion: The framework simulates hundreds to thousands of robots concurrently and trains a humanoid to run in less than 20 minutes using one GPU and one CPU core.The paper contrasts this with prior CPU-cluster approaches using hundreds to thousands of CPU cores.
- Conclusion: Multi-GPU simulation and training produces greater learning-speed improvements for more complex locomotion tasks.The framework is also described as scalable to multi-node settings.
- Future Work: Future work includes dynamic-obstacle navigation, manipulation with Fetch, Baxter, and YuMi, and zero-copy training for vision-based tasks.Zero-copy training would feed GPU-generated simulation data directly into the deep-learning framework without transferring it to the CPU.
A Rewards vs Frames
The reward-versus-frames analysis reports learning progress for single- and multi-GPU experiments, including zoomed-in views. Different total-frame counts arise because training stops after a fixed time.
- Interpretation: Different agent counts produce different total-frame counts because training stops based on a fixed amount of time.The plots therefore compare reward progress under a time-based stopping condition.
- Single-GPU experiments: Figure 5 plots reward against frames for the single-GPU experiments.A zoomed-in version is shown in the second row.
- Multi-GPU experiments: Figure 6 plots reward against frames for the multi-GPU experiments.A zoomed-in version is shown in the second row.
B Comparison of Observations
The observation comparison table contrasts the Humanoid running task across MuJoCo, Roboschool, Control Suite, and the paper’s environments. It specifies different representations for heading direction and root-body rotation.
- Observation comparison: Table 2 compares Humanoid running observations and their dimensionalities across MuJoCo, Roboschool, Control Suite, and the paper’s environments.The comparison concerns the observation design used for the same running task.
- Observation representations: Roboschool and the paper’s agents represent root-body heading direction with sine and cosine of the angular difference to the target direction.For Humanoid running, the target direction is forward from the starting location.
- Observation representations: Root-body rotation uses quaternions in MuJoCo, roll and pitch angles in Roboschool and the paper’s environments, and a z-projection of the rotation matrix in Control Suite.These are alternative representations within the observation comparison.
C Rewards
The paper defines a shared reward function for four locomotion tasks, combining survival, speed, heading, standing, torque, joint-limit, and foot-collision terms.
- The same reward function is used for all four tasks.
- The reward combines alive, speed, heading, and standing terms with additional penalties or bonuses.
- θtarget measures the angle between the robot’s current heading and the direction toward the target location.
- Ralive is set to 0.5 for Ant and 2 for humanoids.
- The formulation includes action-related torque terms and penalties based on joints at limits and feet colliding with the ground.
D Comparison of Reward Functions
The paper compares its Humanoid Running reward coefficients with MuJoCo and Roboschool, while noting that DeepMind Control Suite uses a structurally different reward.
- Table 3 compares Humanoid Running reward-function coefficients across MuJoCo, Roboschool, and the paper’s environments.
- DeepMind Control Suite is omitted because its humanoid walking reward has a different structure.
- DeepMind Control Suite’s reward is the product of running speed and coefficients depending on control magnitude and how upright the humanoid is.
E Hyperparameters
Training uses task-specific hyperparameters and feed-forward value and policy networks, with batch timing scaled according to the number of parallel agents.
- Table 4 reports training hyperparameters for Ant, Humanoid, and Humanoid Flagrun Harder.
- Timesteps per batch are specified relative to 1024 parallel agents and scaled across experiments.
- For 512 agents, a 32-timestep batch becomes 64, while for 2048 agents it becomes 16.
- The value and policy networks in PPO use the same feed-forward architecture, specified as a list of layer sizes.
- For Humanoid Flagrun Harder, the height map and other observations use separate fully connected layers before concatenation and control output.
F MuJoCo Simulation Times
The MuJoCo comparison evaluates simulation throughput and per-agent frame time as concurrent humanoids increase, using measured MuJoCo 1.5 data and projected MuJoCo 2.0 curves.
- The comparison measures total frames per second and frame time per agent as the number of concurrent humanoids increases.
- MuJoCo 1.5 is measured with random actions applied to 28-DoF humanoids lying on the floor, using one Intel Core i9-7960X CPU core.
- MuJoCo 2.0 curves are projected by reducing MuJoCo 1.5 simulation time by 40%.
- MuJoCo performs well with one humanoid on one core, but its simulation time increases as more humanoids are added.
- GPU simulation reduces per-agent simulation time as concurrent simulations and contacts increase.