PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Prioritized Experience Replay

Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, David Silver

arXiv:1803.00933v1cs.LG

TL;DR

Deep reinforcement learning has underused computational resources beyond a single machine. The paper introduces Ape-X, which distributes experience generation and prioritized replay while separating acting from learning, and reports state-of-the-art performance across discrete and continuous tasks in wall-clock speed and final performance.

  • Problem

    Deep reinforcement learning research comparatively underexplores how to harness resources beyond a single machine, despite the potential of more computation and data.

  • Method

    Ape-X separates acting from learning, with distributed actors generating experience for a shared prioritized replay memory and a learner updating the network.

  • Results

    Ape-X achieves state-of-the-art results across a wide range of discrete and continuous tasks in both wall-clock learning speed and final performance.

  • Takeaways & Limitations

    The results indicate that distributed systems are practical for research and potentially large-scale applications of deep reinforcement learning.

  • Takeaways & Limitations

    Ape-X is designed for settings where large quantities of data can be generated in parallel and is not directly applicable when data is costly to obtain.

Abstract

from arXiv · show

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

1 INTRODUCTION

The paper addresses the underexplored problem of scaling deep reinforcement learning beyond a single machine by distributing data generation and prioritized selection. It evaluates this approach across Atari and continuous-control tasks, reporting improved Atari performance with less wall-clock training time.

  • Scaling deep reinforcement learning through more computation, powerful models, and larger datasets motivates the paper’s distributed approach.
  • Current deep reinforcement learning research often optimizes performance within a single-machine budget, leaving effective use of additional resources comparatively underexplored.
  • The proposed approach distributes experience generation and prioritized selection rather than gradient computation, while remaining complementary to distributed gradient methods.
  • The architecture scales DQN and DDPG variants and evaluates them on Atari and continuous-control benchmarks.
  • The architecture achieves new state-of-the-art Atari performance in a fraction of previous state-of-the-art wall-clock time without per-game hyperparameter tuning.
  • The experiments examine scalability, replay capacity, experience recency, and heterogeneous data-generating policies across workers.

2 BACKGROUND

The background situates Ape-X within experience replay, prioritized sampling, and distributed optimization. These lines of work motivate selecting informative experiences while distinguishing data distribution from gradient computation.

  • Distributed Stochastic Gradient Descent: Distributed stochastic gradient descent accelerates neural-network training by parallelizing gradient computation through synchronous or asynchronous parameter updates.
  • Distributed Importance Sampling: Importance sampling speeds training by sampling non-uniformly and weighting updates to counteract sampling bias and reduce gradient variance.
  • Prioritized Experience Replay: Experience replay improves data efficiency by training on stored interactions, while prioritized experience replay extends this idea by emphasizing selected experiences.

3 OUR CONTRIBUTION: DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY

Ape-X extends prioritized experience replay into a distributed architecture that separates acting from learning, centralizes prioritized data, and supports diverse off-policy experience. The framework is combined with DQN and DPG variants for Atari and continuous control.

  • Ape-X extends prioritized experience replay to distributed reinforcement learning and introduces modifications intended to enable scalability.
  • Actors run environment instances and store experience, while learners sample batches from replay memory to update policy parameters.
  • In the experiments, hundreds of CPU actors generate data while one GPU learner samples useful experiences and periodically sends updated parameters to actors.
  • A shared centralized replay memory prioritizes useful data, allowing high-priority experiences identified by any actor to benefit the whole system.
  • Actors compute initial priorities to avoid the scalability problem of newly added transitions retaining maximum priority until learner updates.
  • Batched communication reduces latency requirements and can allow actors and learners to operate across different data centers without limiting performance.
  • Different exploration policies across off-policy actors broaden the diversity of jointly generated experience and can support progress on difficult exploration problems.
  • Ape-X DQN: The DQN variant combines double Q-learning, multi-step bootstrap targets, and a dueling network architecture.

4 EXPERIMENTS

Ape-X experiments evaluate distributed DQN on Atari and Ape-X DPG on continuous control, with performance improving as actor counts increase. On Atari, Ape-X achieves higher benchmark performance than reported baselines, while Ape-X DPG outperforms standard DDPG while training substantially faster.

  • 4.1 Atari: Ape-X DQN uses 360 actor machines generating approximately 50K frames per second, or approximately 12.5K transitions per second after action repetition.Actors batch experience locally before sending it to replay.
  • 4.1 Atari: Ape-X achieved substantially higher final performance and faster initial learning than several Atari baselines, including DQN, Prioritized DQN, Distributional DQN, Rainbow, and Gorila.Figure 2 compares median human-normalized scores across all 57 games and initial learning curves for six games.
  • 4.1 Atari: Ape-X DQN achieved higher median human-normalized performance than every baseline under both no-op-start and human-start evaluation.The benchmark covers 57 Atari games; human starts provide a more challenging generalization test.
  • 4.2 Continuous control: Ape-X DPG becomes increasingly effective as the number of actors increases across four continuous-control tasks, outperforming standard DDPG trained for over 10 times longer.The comparison is shown as performance over wall-clock time, with the standard DDPG reference represented by its maximum performance after five days.
  • 4.2 Continuous control: Performance consistently improves when the number of actors increases from 8 to 256, without increasing the number of learning updates performed.This scaling result is reported for Ape-X DQN on Atari experiments.

5 ANALYSIS

The analysis examines how actor count, replay capacity, recency, and policy diversity affect Ape-X performance. Increasing actors consistently helps, larger replay memories provide a small benefit on most games, and neither recency nor policy diversity alone explains the scaling results.

  • Scaling actors: Performance consistently improves as the number of actors increases from 8 to 256, while network update rate, architecture, and update rule remain unchanged.The experiments used a fixed one-million-transition shared replay memory and trained for 35 hours on six Atari games.
  • Replay capacity: Larger replay capacities provide a small benefit on most games, although Wizard Of Wor diverged with a 250K replay size.The 250K Wizard Of Wor curve is incomplete, whereas the other replay-size runs did not show divergence.
  • Replay capacity: The analysis hypothesizes that larger replay memories help retain and replay high-priority experiences for longer.With 256 actors, the replay contents are replaced much faster than in most DQN-like agents.
  • Confounding factors: Neither experience recency nor diversity of data-generating policies alone explains the observed scalability performance.The authors conclude that gathering more experience contributes substantially through better exploration and better avoidance of overfitting.

6 CONCLUSION

Ape-X is a distributed prioritized-replay framework that achieves strong performance across discrete and continuous tasks, while its applicability depends on parallel data generation.

  • Ape-X achieves state-of-the-art results across a wide range of discrete and continuous tasks in wall-clock speed and final performance.
  • The framework can combine with other off-policy reinforcement learning updates beyond the DQN and DPG applications studied here.For temporally extended methods, prioritizing sequences rather than individual transitions may be appropriate.
  • Ape-X is designed for settings where large quantities of data can be generated in parallel, including simulated environments and multi-instance real-world systems.
  • Ape-X is not directly applicable when data is costly to obtain.
  • The approach generates diverse experiences and identifies the most useful events to address exploration in large domains.
  • The architecture indicates that distributed systems are practical for research and potentially large-scale deep reinforcement learning applications.

A RECENCY OF EXPERIENCE

Increasing the number of actors makes replay contents more recent when replay capacity stays fixed, linking actor count to the effective replacement rate and data recency.

  • With fixed replay capacity, adding actors increases the rate at which replay contents are replaced.
  • Experiments with more actors contain more recent transitions generated by policies closer to the learner’s optimized parameters.
  • Duplicating added data has a similar effect to reducing replay capacity.Results with a smaller replay memory corroborate this finding.

B VARYING THE DATA-GENERATING POLICIES

The data-generating policy experiment compares six fixed ϵ values with the full ϵ range, finding a slight overall advantage for the full range that is not essential for good results.

  • The experiment tests whether assigning each actor a different ϵ contributes to scalability.
  • The fixed-policy alternative uses a small set of 6 ϵ values instead of the full range typically used.
  • Results with the full range of ϵ are overall slightly better than results with the fixed set.
  • Using the full range of ϵ is not essential for achieving good results within the distributed framework.
  • Actor-side preprocessing includes greyscaling, frame stacking, repeating actions 4 times, and clipping rewards to [−1, 1].
  • The learner begins after 50000 transitions are accumulated and uses multi-step loss with n = 3.Training uses Centered RMSProp with learning rate 0.00025 / 4, decay 0.95, epsilon 1.5e-7, gradient clipping at 40, and target updates every 2500 batches.

D CONTINUOUS CONTROL: ADDITIONAL DETAILS

The continuous-control setup benchmarks Ape-X DPG in MuJoCo using separate actor and critic networks, prioritized replay, and Gaussian exploration noise.

  • The critic uses 400 and 300 units, while the actor uses 300 and 200 units with tanh activations.The actor gradient is clipped element-wise to [−1, 1], and training uses Adam with learning rate 0.0001.
  • Replay sampling priorities use the critic’s absolute TD error with proportional prioritized sampling and priority exponent αsample = 0.6.To maintain replay capacity 10^6, transitions are evicted using proportional prioritized sampling with αevict = −0.4.
  • Ape-X DPG adds normally distributed action noise with σ = 0.3 and evaluates the noiseless deterministic policy.
  • Continuous-control benchmarking covers Humanoid and Manipulator domains in the MuJoCo physics simulator.Humanoid includes walk, run, and stand tasks; Manipulator requires catching a randomly initialized moving ball.

E TUNING

The experiments tune batch size, learning rate, prioritization, and system-update parameters across Atari and continuous-control settings. Larger batches help Atari with many actors, while extreme prioritization can destabilize continuous-control training.

  • Atari tuning: Larger Atari batch sizes improved performance with many actors, with clear benefits up to batch size 512.Tested batch sizes were 32, 128, 256, 512, and 1024.
  • Atari tuning: Atari learning rate 0.00025 destabilized some games with larger batches, while 0.00025 / 8 did not reliably improve results.
  • Continuous-control tuning: Continuous-control results were essentially consistent for prioritization exponents α in [0.3, 0.7], whereas values beyond 0.7 sometimes caused instability and divergence.Learning rates from 10^-3 to 10^-5 and batch sizes from 32 to 1024 were also tested.
  • System tuning: For many-actor experiments, actor parameter-update periods and local accumulation sizes were chosen to avoid overloading the learner and replay server.These parameters were not otherwise tuned and showed no significant observed impact on learning dynamics.

F IMPLEMENTATION

The implementation separates distributed acting, centralized prioritized replay, and GPU learning, with batching and buffering used to sustain throughput. It also addresses asynchronous execution, failures, storage, sampling, and empirical scaling across Atari games.

  • Data storage: Experience is stored in a distributed in-memory key-value replay system, while prioritized-distribution operations maintain and sample valid transition keys.The system uses custom TensorFlow operations for adding, reading, removing, and sampling batches.
  • Replay sampling: Prioritized replay samples transitions using p_k^α, with p_k = |δ_k| and α = 0 recovering uniform sampling.The distribution is updated whenever keys or priorities change.
  • Data pipeline: Background fetching, decompression, and buffering run in parallel with GPU gradient computation so training data remains available.Fetched batches are buffered in a TensorFlow queue.
  • Data ingestion: Actors construct n-step transitions in circular buffers, batch them locally, compute initial TD-error priorities, and send them to remote replay memory.Unique keys record each transition’s actor and environment step.
  • System performance: Replay-server CPU contention was the main observed bottleneck and was addressed by using sufficiently large request and response batches.Network bandwidth and shared locks were also identified as potential bottlenecks.
  • Asynchronicity: Because acting and learning are asynchronous, performance should be reported against both environment frames and learner updates rather than either alone.
  • Evaluation and scaling: Ape-X DQN training curves cover 57 Atari games against environment frames and wall-clock time, using 360 actors, one replay server, and one Tesla P100 learner.Figure 9 notes a Riverraid anomaly caused by an infrastructure error.
Loading 1803.00933v1…