Source-linked AI summary
Accelerated Methods for Deep Reinforcement Learning
Adam Stooke, Pieter Abbeel
TL;DR
Deep RL experiments are slowed by long turnaround times, and this paper adapts existing algorithms to use parallel simulators and CPU-GPU hardware. The resulting framework reports faster Atari learning across policy-gradient and Q-value methods without loss of sample complexity or final game score.
Problem
Deep RL research relies heavily on empirical evaluation, making experiment turnaround time a key bottleneck.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The results indicate that parallelized deep RL can boost experiment scale and achieve unprecedented wall-clock learning times across a range of algorithms.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.
1 INTRODUCTION
The paper develops a unified way to parallelize deep RL across CPUs and GPUs, targeting experiment turnaround time. It reports faster learning in Atari across policy-gradient and Q-value methods, without harming sample complexity or final game score.
1 INTRODUCTION
- The framework unifies policy-gradient and Q-value learning methods, which traditionally used distinct implementations and different hardware resources.A3C and DQN are given as representative examples of the two families.Source: §1 · p. 1
- The framework parallelizes deep RL without changing algorithms’ underlying formulations and uses GPUs to accelerate both inference and training.The authors demonstrate multi-GPU versions of A3C, PPO, DQN, Categorical DQN, and Rainbow.Source: §1 · p. 1
- Highly parallel sampling with batched inference accelerates experiment turnaround without hindering training, and considerably larger batches need not harm sample complexity or final game score.The tests use the Atari-2600 domain through the Arcade Learning Environment.Source: §1 · p. 1
- Policy-gradient runs learned successful Atari strategies in under 10 minutes, while some standard Q-value runs fell from 10 days to under 2 hours.The reported timings contrast with runs taking hours and days, respectively.Source: §1 · p. 1
2 RELATED WORK
Prior work parallelized deep RL through distributed Q-learning and GPU-based actor-critic systems, but used different approaches and faced trade-offs in sample complexity, policy lag, or scale. This paper positions its contribution against those methods and related batched actor-critic work.
2 RELATED WORK
- Gorila parallelized DQN across hundreds of computing units with central parameter servers, achieving significant but sub-linear speedups and worse sample complexity than single-threaded DQN.A later distributed prioritized replay system used hundreds of CPU cores for simulation and one GPU for training, with larger batches and a brief learning-rate study.Source: §2 · pp. 1–2
- GA3C batched GPU inference and training asynchronously, but its policy lag worsened sample complexity; IMPALA used V-trace to mitigate lag and did not use GPU inference.IMPALA is described as independent work simultaneous with this paper.Source: §2 · p. 2
- PAAC studied many simulators and increased batch sizes and learning rates for single-GPU batched A2C; this paper claims improved sampling organization, multi-GPU scale and speed, and asynchronous optimization.The text identifies these as the chief ways its actor-critic contributions exceed PAAC.Source: §2 · p. 2
3 RL ALGORITHM BACKGROUND
The background contrasts policy-gradient methods, which optimize a parameterized action distribution, with Q-value methods, which learn action values from replayed experience. It also introduces discounted return and state-value concepts used in these algorithms.
3 RL ALGORITHM BACKGROUND
- Discounted return sums future rewards weighted by powers of γ, and a state’s value is its expected return under a policy.The discount factor satisfies γ ≤ 1.Source: §3 · p. 2
- Policy-gradient methods parameterize a distribution over actions; A3C estimates state values and updates the policy from fresh experience using advantage-weighted gradients.PPO instead optimizes a likelihood-ratio surrogate with clipping and performs multiple parameter updates on each experience set.Source: §3 · p. 2
- Q-value methods parameterize Q(s, a), train against data-estimated targets, and reuse randomly selected recent experiences from a replay buffer.Categorical DQN learns distributions over discretized Q-values; Rainbow combines it with five additional enhancements, and experiments use ϵ-Rainbow without parameter noise.Source: §3 · p. 2
4 PARALLEL, ACCELERATED RL FRAMEWORK
The framework combines CPU-based parallel simulation with GPU inference and learning, organizing both synchronized sampling and synchronous or asynchronous multi-GPU optimization. Its sampling design addresses idle time and synchronization delays, while its optimization procedures differ in how GPUs share updates.
4.1 Synchronized Sampling
- Multiple CPU simulators submit observations together for batched GPU inference, while multiple GPUs repeat the sampling-learning template with CPU cores allocated evenly.Simulator processes can be pinned to cores, with one core reserved for each GPU.Source: §4.1 · pp. 2–3
- Alternating two simulator-process groups keeps utilization high when simulation and inference loads are balanced, hiding the quicker computation’s execution time.Synchronized sampling can be slowed by stragglers; stacking independent simulators per process mitigates this effect and lets inference batches exceed the number of processes.Source: §4.1 · pp. 2–3
4.2 Synchronous Multi-GPU Optimization
- Synchronous multi-GPU optimization keeps parameters identical by computing local gradients, all-reducing them across GPUs, and updating each GPU’s parameters.The framework uses NVIDIA’s Collective Communication Library for inter-GPU communication.Source: §4.2 · p. 3
4.3 Asynchronous Multi-GPU Optimization
- Asynchronous optimization makes each GPU a sampler-learner that updates a central CPU parameter store after applying the update rule on the GPU.Updates overwrite central values under locks; dividing parameters into disjoint chunks with separate locks balances update efficiency against lock contention.Source: §4.3 · p. 3
5 EXPERIMENTS
The Atari experiments test synchronized sampling, adaptation to parallel simulators, and whether larger batches or asynchronous methods can accelerate learning without worsening performance. Training intensity was held at reference levels, and learning curves averaged at least two seeds.
5.1 Sampling
- Synchronized sampling scaled to over 35,000 samples per second at 256 simulators or more, while 8 environments per core let one GPU support 16 CPU cores at roughly 80% of inference-free speed.Multiple simulators per core mitigated synchronization losses and hid neural-network inference time.Source: §5.1 · pp. 3–4, Figure 1 · p. 4
5.2 Learning with Many Simulator Instances
- Policy-gradient methods largely retained learning performance with more simulators: A2C scores were largely unchanged, A3C curves were nearly indistinguishable in most cases, PPO learned through 512 simulators, and APPO remained consistent in several games.A2C showed gradual sample-efficiency decay at large scales, and some A3C results degraded at the largest scales; APPO tested reduced communication frequency.Source: §5.2 · pp. 4–5, Figure 2 · p. 6
- DQN performance was largely independent of simulator count up to over 200 when its replay buffer was organized by simulator and update steps per optimization cycle were not too high.The total replay buffer remained at 1 million transitions, shortening each simulator’s history.Source: §5.2 · p. 5
5.3 Q-Value Learning with Large Training Batches
- DQN learned consistently through batch size 512, Categorical DQN retained maximum scores through 2,048, while ε-Rainbow lost performance above 512 in some games.For DQN, finding one scaled learning rate that worked across test games became difficult above batch size 512.Source: §5.3 · p. 5, Figure 3 · p. 7
5.4 Learning Speed
- DQN and ε-Rainbow completed 50 million steps in 8 and 14 hours, respectively, versus 10-day reference times; Categorical-DQN finished in under 2 hours on the full server.Categorical-DQN achieved over 6× speedup relative to 1 GPU.Source: Table 2 · p. 5, §5.4 · p. 5
5.5 Effects of Batch Size on Optimization
- A 2,048-batch secondary DQN learner failed when paired with the ordinary sampler, whereas a 64-batch secondary learner tracked it; the authors attribute the mismatch to slower optimization and off-policy data.The 2,048-batch learner needed more samples than the 64-batch learner to reach the same BREAKOUT score when each acted as sampler-learner.Source: §5.5 · p. 6
- Adam kept Categorical-DQN parameter norms relatively close across batch sizes without changing learning rate, while A2C gradient-estimate saturation was connected to worse sample efficiency.The authors report Adam was superior to RMSProp for large-batch learners’ ability to traverse parameter space.Source: §5.5 · pp. 6–7
5.4 Learning Speed
- On an 8-GPU, 40-core server, policy-gradient methods completed their steep initial learning phase in under 10 minutes; PPO mastered Pong in 4 minutes.A2C with 256 environments processed more than 25,000 samples per second.Source: §5.4 · p. 5, Figure 4 · p. 7
6 CONCLUSIONS AND DISCUSSION
The paper presents a unified hardware-accelerated framework for parallelizing policy-gradient and Q-value deep RL, with Atari experiments showing fast learning without loss of sample complexity. It identifies unresolved limits to scaling and proposes broader domains and distributed algorithms as directions for extension.
6 CONCLUSIONS AND DISCUSSION
- The framework applies hardware accelerators to parallelize policy-gradient and Q-value methods, which learned Atari games in parallel without loss of sample complexity and at unprecedented wall-clock times.The authors say the results point toward scaling experiment size and that they will release the code-base.Source: §6 · p. 7
- The authors propose applying the framework beyond Atari, especially to domains involving perception.They suggest the current single-node implementation may serve as a building block for distributed algorithms.Source: §6 · p. 7
- The extent and limiting factors of parallelization remain unresolved, and the authors did not determine the best algorithm configurations.They also leave limits to asynchronous scaling unexplored.Source: §6 · p. 7
A EXPERIMENT DETAILS
The Atari preprocessing downsizes cropped frames to 104 × 80, while using standard algorithm-specific network architectures for Q-learning and policy gradients.
A EXPERIMENT DETAILS
- Frames are cropped by two rows and downsized by a factor of two to 104 × 80; Q-learning uses a three-convolution-layer network and policy gradients a two-convolution-layer feed-forward network.The authors describe the preprocessing as closely resembling the original DQN publication, except for square-frame sizing.Source: §A.1 · p. 9
A.2 DGX-1 Sampling Speed
The appendix reports DGX-1 sampling-speed measurements and algorithm-specific hyperparameter settings. It also specifies a multi-step asynchronous Adam rule that combines local gradient steps with central parameter updates.
A.2 DGX-1 Sampling Speed
- DGX-1 sampling speed was measured with A3C-Net across total simulator counts, with synchronized GPU processes separated by a barrier every five steps or GPUs and associated cores running independently.Table 3 reports sampling speeds in thousands of samples per second.Source: §A.2 · p. 9, Table 3 · p. 9
A.3 Hyperparameters for Scaling
- For policy-gradient scaling, A2C increased learning rate with simulator count, A3C and PPO kept their settings, and APPO changed clipping, learning rate, and its schedule.APPO used gradient-norm clipping, a fourfold lower learning rate, and no learning-rate schedule; each change benefited learning.Source: §A.3 · p. 9
- DQN adjusted sampling horizon and update steps to maintain training intensity, while Categorical DQN and ϵ-Rainbow used batch-size-dependent Adam epsilon settings.Categorical DQN used learning rate 4.2 × 10−4 at batch sizes 256 and above and epsilon 0.01/L; ϵ-Rainbow used epsilon 0.005/L.Source: §A.3 · p. 9
B UPDATE RULE FOR MULTI-STEP ASYNCHRONOUS ADAM
- The asynchronous Adam update allows multiple local gradient steps between synchronizations before updating the central parameters; with one learner thread, it reduces to usual Adam.Local accumulation variables are initialized at zero, and n denotes the number of local gradient steps between synchronizations.Source: B UPDATE RULE FOR MULTI-STEP… · p. 10
C FIGURES FOR SECONDARY-LEARNER EXPERIMENT (DQN)
The BREAKOUT secondary-learner experiments show that the effect of batch size depends on the sampler-learner’s batch size. Parameter norms also distinguish a large-batch learner that fails to learn from one that tracks the game score.
C FIGURES FOR SECONDARY-LEARNER EXPERIMENT (DQN)
- A 64-batch secondary learner kept pace with a 2048-batch sampler, but a 2048-batch secondary learner failed to track a 64-batch sampler or learn.Curves averaged over two random trials.Source: Figure 5 · p. 11
- When the secondary learner used a 2048 batch against a 64-batch sampler, its parameter norm continued growing while its game score remained nil.The figure reports that the large-batch learner lagged behind the small-batch learner in both cases.Source: Figure 6 · p. 11
- With the same batch size as the sampler, the secondary learner tracked the BREAKOUT score imperfectly but learned successfully.Curves averaged over two random trials.Source: Figure 7 · p. 11
D OBSERVATIONS ON UPDATE RULES AND BATCH SIZE SCALING
The Q*BERT analysis shows that batch size and update rule alter layer-wise parameter, gradient, and step behavior even when learning curves roughly match. Additional learning curves compare baseline and scaled policy-gradient and Q-value configurations, with a few game-specific exceptions and an unexplained ϵ-Rainbow anomaly.
D OBSERVATIONS ON UPDATE RULES AND BATCH SIZE SCALING
- The Q*BERT experiments adjusted learning rates to align performance curves, tracked parameter, gradient, and step norms, and fixed DQN training intensity at 8.Figure 8 shows L-2 norms for the first convolutional and fully connected layers across batch sizes under Adam and RMSProp; two random seeds were run per setting.Source: D OBSERVATIONS ON UPDATE RULES AND… · p. 12, Figure 8 · p. 13
- Although game scores roughly matched during training, the solutions differed by parameter norms, and whole-network norm trends resembled FC-0.No regularization was used.Source: D OBSERVATIONS ON UPDATE RULES AND… · p. 12
- Larger batches reduced FC-0 gradient magnitudes but increased Adam step sizes; RMSProp needed a 20x learning-rate increase from batch 64 to 1,024 for similar FC-0 norms.Under both update rules, larger-batch step increases did not fully offset fewer update steps, indicating straighter parameter-space trajectories; RMSProp produced larger steps overall but ended at smaller weights.Source: D OBSERVATIONS ON UPDATE RULES AND… · p. 12
- As batch size increased, Adam shifted learning emphasis away from Conv-0, while RMSProp's increased learning rate enlarged that layer's weights.Adam's Conv-0 gradient norm showed a crossover, and its step norm changed little with batch size; RMSProp's gradient and step patterns resembled FC-0.Source: D OBSERVATIONS ON UPDATE RULES AND… · p. 12
E ADDITIONAL LEARNING CURVES
- Across PPO and A2C learning curves, the baseline stood above both scaled versions only in Asteroids and Bowling, and in Atlantis, Gopher, and possibly Krull.Source: Figure 9 · p. 14, Figure 10 · p. 15
- In scaled DQN curves, ϵ-Rainbow's anomalously low Breakout scores also occurred at smaller batch sizes and were remedied by reward horizon 1 or asynchronous optimization.The cause was unknown; reward horizon 3 usually helped.Source: Figure 11 · p. 16