Source-linked AI summary
Massively Parallel Methods for Deep Reinforcement Learning
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, David Silver
TL;DR
DQN achieved strong Atari results but required long single-machine training, motivating a scalable distributed reinforcement-learning architecture. The paper introduces Gorila, which parallelizes actors and learners and distributes the neural network and experience replay; on 49 Atari games, it outperformed single-GPU DQN on 41 games and reached comparable performance roughly ten times faster for most games.
Problem
DQN had previously been applied only to single-machine architectures, with training one Atari game taking 12–14 days on a GPU.
Method
Gorila uses parallel actors and learners together with a distributed neural network and distributed experience replay memory to implement massively distributed DQN.
Results
41 of 49 Atari games outperformed single-GPU DQN, and Gorila DQN reached single-GPU DQN performance roughly ten times faster for most games.
Takeaways & Limitations
Gorila provides a scalable deep-reinforcement-learning architecture whose performance improved with increased computation and memory.
Takeaways & Limitations
Deep Q-learning with nonlinear function approximators is highly unstable, and off-policy updates are required because replay contains experience from older behavior policies.
Abstract
from arXiv · showhide
We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.
1. Introduction
The paper addresses long single-machine training times for DQN by introducing Gorila, a massively distributed reinforcement-learning architecture. Gorila parallelizes acting and learning, distributes the neural network and replay memory, and improves Atari performance while reducing wall-time.
- DQN had previously been applied only to single-machine architectures, requiring 12–14 days on a GPU to train on one Atari game.
- Gorila uses parallel actors, parallel learners, a distributed neural network, and a distributed experience replay memory.
- Parallel actors interact with multiple environment instances, increase replay-memory capacity, and explore the state space through different policies.
- Distributed learners sample stored experience and update the value function or policy, while Q-network parameters may be distributed across machines.
- 41 of 49 Atari games outperformed single-GPU DQN, while Gorila DQN reached single-GPU DQN performance roughly ten times faster for most games.
2. Related Work
Related work spans distributed multi-agent systems, concurrent reinforcement learning, MapReduce-based batch methods, and parallel Sarsa. The paper distinguishes Gorila through its separation of acting, learning, and parameter updates and its distributed deep function approximators.
- Distributed multi-agent systems distribute control among cooperating agents sharing one environment, emphasizing teamwork and emergent group behavior.
- MapReduce has distributed matrix computations for policy evaluation, policy iteration, and value iteration, but the approach targets batch methods with linear function approximation.
- These MapReduce methods are not immediately applicable to nonlinear representations with online reinforcement learning in environments with unknown dynamics.
- Parallel Sarsa assigns each machine an agent and environment, whereas Gorila separates acting, learning, and parameter updates through client-server communication.
3. Background
The background introduces distributed neural-network training and reinforcement-learning value functions before describing DQN’s stabilizing mechanisms. DQN uses replayed transitions, separate current and target networks, and stochastic-gradient updates with epsilon-greedy action selection.
- 3.1. DistBelief: DistBelief combines model parallelism, which partitions model storage and training, with data parallelism, which trains model replicas on separate data.
- 3.1. DistBelief: The DistBelief parameter server applies replica gradients to a master model and returns updated parameters, while replicas compute gradients from minibatches.
- 3.3. Deep Q-Networks: DQN stores experience transitions in replay memory and maintains current and older target Q-networks for Bellman-error optimization.
- 3.2. Reinforcement Learning: Reinforcement learning trains an agent through sequential environment interaction, mapping observed states to actions while maximizing discounted cumulative rewards.
- 3.2. Reinforcement Learning: The action-value function Qπ(s, a) is the expected return after taking action a in state s under policy π.
- 3.2. Reinforcement Learning: Q-learning with deep neural networks is highly unstable, motivating DQN’s more stable training procedure.
- 3.3. Deep Q-Networks: DQN samples transitions uniformly from replay memory, updates current parameters with stochastic gradient descent, and selects actions using an epsilon-greedy policy.
4. Distributed Architecture
Gorila is a massively distributed reinforcement-learning architecture that separates acting, learning, experience storage, and parameter management. Its distributed DQN implementation coordinates asynchronous learners and actors through a parameter server while adding safeguards for stale or unstable updates.
- Architecture: Gorila separates reinforcement learning into parallel actors, parallel learners, a distributed neural network, and distributed experience replay memory.Actors generate behaviour, learners train from stored experience, and the distributed network represents the value function or behaviour policy.
- Actors: Each actor runs its own environment instance and generates trajectories, with approximately TNact experience produced after T time-steps.Actors may visit different parts of the state space because they follow slightly different policies.
- Experience replay memory: Experience can remain in local replay memories, scaling total capacity to MNact tuples, or be aggregated into a global distributed database.Global storage can scale independently of the number of actors but introduces additional communication overhead.
- Learners and parameter server: Learners sample minibatches from replay memory, compute off-policy DQN gradients, and send them to a parameter server for asynchronous updates.The parameter server maintains a distributed Q-network whose parameter vector is split across Nparam machines.
- Architecture: In bundled mode, each actor, replay memory, and learner form a one-to-one bundle, with communication between bundles occurring through parameter synchronization.Actors and learners periodically synchronize their Q-network replicas with the parameter server.
- Stability safeguards: Gorila DQN uses off-policy replay and safeguards against distributed-system instability by discarding stale or unusually high-loss gradients and using AdaGrad.These safeguards address disappearing nodes, network delays, and slow individual machines.
5. Experiments
The experiments evaluate Gorila DQN across 49 Atari 2600 games using a protocol closely matched to single-GPU DQN. Evaluation includes both standard randomized starts and human-gameplay starts to assess performance and generalization.
- Experimental setup: 49 Atari 2600 games from the Arcade Learning Environment were used, with the DQN preprocessing and network architecture closely followed.The games require learning directly from 210 × 160 RGB video with score changes as rewards.
- Experimental setup: The Q-network receives an 84 × 84 × 4 input formed by concatenating four previous preprocessed frames.Its architecture contains three convolutional layers, a fully connected hidden layer, and one output per valid action.
- Experimental setup: Gorila DQN uses Nparam = 31 and Nlearn = Nact = 100, with a replay memory of 1 million frames.The behaviour policy is epsilon-greedy, with epsilon annealed from 1 to 0.1 over the first one million global updates.
- Evaluation: Standard evaluation averages scores over 30 episodes with randomized null-action starts and a five-minute or 18000-frame limit.This procedure follows the established DQN evaluation protocol.
- Evaluation: Human-start evaluation uses 100 starting points sampled from professional gameplay to test generalization to states the agent may not have trained on.Only points earned by the agent are included in the averaged score.
- Evaluation: Scores are normalized so that 0 denotes a random agent and 100 denotes a professional human game player.The same starting states are used for agents and the random baseline in both evaluation procedures.
- Experimental setup: Hyperparameters were selected on Breakout, Pong, and Seaquest, then fixed across games and evaluated over five runs with different random initializations.Final results average the best periodically evaluated network from each run and compare against DQN and human scores.
6. Results
Gorila DQN improved Atari performance and generalization over single-GPU DQN while reaching comparable performance substantially faster. Its advantages were especially pronounced under human-start evaluations and with extended training.
- Training speed: Figure 5 compares time to surpass single-DQN performance with time to reach Gorila DQN’s peak performance.The red curve represents surpassing single-DQN performance; the blue curve represents reaching peak performance.
- Performance comparison: Gorila DQN outperformed single-GPU DQN on 41 of 49 Atari games under human-start evaluation.The comparison used roughly half the single-GPU DQN training time.
- Generalization: 25 games reached at least 75% of the human professional score under human-start evaluation.This evaluation began from states sampled from human gameplay.
- Training speed: Gorila DQN surpassed the best single-GPU DQN scores on 19 games within 6 hours, 23 within 12 hours, 30 within 24 hours, and 38 within 36 hours.These counts were measured under human-start evaluation.
- Training speed: Training time to reach the single-process DQN score decreased by roughly an order of magnitude.Gorila sometimes reached its best score in under two days, although most games continued improving with longer training.
7. Conclusion
The paper introduces Gorila, a massively distributed architecture for deep reinforcement learning, and applies it to an asynchronous DQN variant. Gorila DQN outperformed single-GPU DQN on 41 of 49 Atari games and is presented as a scalable approach that benefits from increased computation and memory.
- Contribution: Gorila is a massively distributed reinforcement-learning architecture that acts and learns in parallel using distributed replay memory and a distributed neural network.The paper describes this as the first massively distributed architecture for deep reinforcement learning.
- Results: Gorila DQN outperformed single-GPU DQN on 41 of 49 Atari games.The authors characterize this as the best result in the domain at that time.
- Implication: Gorila is presented as a scalable architecture whose performance improves with increased computation and memory.This conclusion follows the reported benefits of massive parallelism.
8. Appendix
The appendix presents the paper’s evaluation data, covering normalized and raw scores under null-op and human-start evaluations. It organizes these results across four tables.
- The appendix presents all data used in the paper.
- Table 1 reports normalized scores for null-op evaluation.
- Table 2 reports normalized scores for human-start evaluation.
- Table 3 reports raw scores for human-start evaluation.
- Table 4 reports raw scores for null-op evaluation.