Efficient Parallel Methods for Deep Reinforcement Learning

1 INTRODUCTION AND RELATED WORK

• Replay memory can reduce state correlations but is limited to off-policy methods, while asynchronous agents can apply stale gradients or make simultaneous parameter updates.The paper identifies these as distinct drawbacks of the two approaches.Source: §1 · p. 1
• Gorila distributes DQN across machines, where multiple learners train off-policy from a shared replay memory populated by multiple actors.The paper reports that Gorila outperformed standard DQN on Atari after four days of training.Source: §1 · p. 1
• A3C runs actor-learners asynchronously on one machine, while GA3C batches GPU action selection and learning through queues that can make training off-policy.GA3C addresses resulting instabilities with a modification to weight updates; A3C surpassed Atari state of the art at the time after four days on 16 CPU cores.Source: §1 · p. 2
• The proposed framework supports hundreds of actors on one machine with a single parameter copy and synchronous updates; its PAAC implementation achieved Atari state-of-the-art performance after a few hours.The authors also describe the framework as algorithm-agnostic and note that it avoids queues while allowing on-policy learning.Source: §1 · p. 2

2 BACKGROUND

• RL seeks a policy mapping states to actions to maximize expected cumulative reward, and neural networks represent policies explicitly or through value functions.The objective uses a discount factor between zero and one.Source: §2 · p. 2

2.1 BATCHING WITH STOCHASTIC GRADIENT DESCENT

• In SGD, a mini-batch is sampled from the training set; increasing its size raises update cost while reducing gradient variance, motivating a learning-rate adjustment with an upper bound.The text describes a trade-off between computational efficiency and sample efficiency.Source: §2.1 · p. 3

2.2 MODEL-FREE VALUE BASED METHODS

• Value-based methods learn an approximate Q-function and derive a policy by choosing the action with the highest estimated value.DQN represents Q with a convolutional neural network and updates it using Bellman-equation targets.Source: §2.2 · p. 3

2.3 POLICY GRADIENT METHODS

• Policy-gradient methods learn a parameterized policy; advantage replaces Q with Q minus state value, while A3C estimates Q with n-step returns from actor-environment interactions.The policy parameters are optimized by gradient ascent.Source: §2.3 · pp. 3–4

3 PARALLEL FRAMEWORK FOR DEEP REINFORCEMENT LEARNING

• The framework maintains multiple environment instances whose actions are generated from a policy, which may be represented implicitly by a value-based method or explicitly by a policy-gradient method.The architecture is shown in Figure 1.Source: §3 · p. 4, Figure 1 · p. 4
• Parallel environments are likely to explore different state-space locations, reducing correlations and helping stabilize training.The paper characterizes the approach as an online experience memory, sampling from the current environment distribution rather than uniformly from past experience.Source: §3 · p. 4
• A master generates actions for all environments, workers apply them and store experiences in parallel, and the master updates the policy from those experiences.Batching policy evaluation and training supports parallelization on modern compute architectures.Source: §3 · p. 4

• The implementation uses n-step advantage actor-critic with neural-network policy and value estimates, updating actor parameters by gradient ascent and critic parameters by gradient descent.Source: §4 · pp. 4–5
• Policy entropy is added to improve exploration by discouraging premature convergence to suboptimal deterministic policies.Source: §4 · p. 5
• Each of ne environments generates tmax experiences, forming batches of size ne·tmax for calculating policy and value gradients.Source: §4 · p. 5
• PAAC achieves state-of-the-art Atari 2600 performance in half the time required by GA3C and one eighth the time required by A3C.Source: §4 · p. 5

5 EXPERIMENTS

• The evaluation used 12 Atari 2600 games, standard Atari preprocessing, and a four-core Intel i7-4790K with an Nvidia GTX 980 Ti GPU.Actions were repeated four times; frames were reduced to 84×84 grayscale, and 1–30 no-op actions preceded agent control after restarts.Source: §5 · p. 5, §5.1 · p. 5

5.1 EXPERIMENTAL SETUP

• PAAC runs parallel advantage actor-critic across n_e environments, collecting actions and rewards in parallel before updating shared policy and value parameters.The algorithm repeats this process in t_max-step blocks and aggregates gradients across environments and timesteps.Source: §5.1 · p. 6
• A shared convolutional network jointly models policy and value functions, with archnips and archnature variants used to compare model sizes.The policy has a softmax output and the value function a single linear output; networks were trained with RMSProp.Source: §5.1 · p. 6

5.2 RESULTS

• After a few hours on one computer, PAAC outperforms Gorila and A3C FF in 8 games each, and matches GA3C in 2 of 9 games while surpassing it in 7.The twelve-game evaluation trained for 115 million skipped frames; Table 1 reports scores averaged over 30 runs, with A3C FF using a different start condition.Source: §5.2 · p. 6, Table 1 · p. 7
• Most tested n_e values yield similar scores at a given timestep, but n_e = 256 diverges in three of four games.Figure 3 compares scores for different n_e; the learning rate was set to 0.0007 · n_e and was not tuned separately for each batch size.Source: §5.2 · pp. 6–7, Figure 3 · p. 7
• Agent-environment interaction constrains training speed: for archnips at n_e = 32, it takes approximately 50% of the time, versus 37% for learning and action selection.Compared with archnips at n_e = 32, archnature on the GPU reduces timesteps per second by 22%; on the CPU, the reduction is 41%.Source: §5.2 · p. 7

6 CONCLUSION

• The framework supports on-policy and off-policy, value-based and policy-gradient algorithms, and can be efficiently parallelized on a GPU.The presented implementation reduces Atari 2600 training time to a few hours while maintaining state-of-the-art performance.Source: §6 · p. 8
• The authors conclude that faster training will allow these algorithms to be applied to more demanding environments and more powerful models.Source: §6 · p. 8
• The presented implementation is a parallel framework for deep reinforcement learning that can be efficiently parallelized on a GPU.Source: §6 · p. 8