PaperlayerView exact revision on arXiv

Source-linked AI summary

Asynchronous Methods for Deep Reinforcement Learning

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu

arXiv:1602.01783v2cs.LG

TL;DR

Deep reinforcement learning with deep neural networks was considered unstable for simple online algorithms. The paper uses asynchronous parallel actor-learners and finds stable training across methods and domains, with A3C surpassing Atari state of the art in half the training time.

  • Problem

    Combining simple online reinforcement learning algorithms with deep neural networks was considered fundamentally unstable, limiting straightforward deep RL training.

  • Method

    The framework asynchronously runs multiple agents on separate environment instances, using parallel actor-learners to train four standard deep reinforcement learning algorithms.

  • Results

    A3C surpasses the Atari state of the art in half the training time, while asynchronous methods train controllers across Atari, continuous motor control, and visual 3D-maze tasks.

  • Takeaways & Limitations

    Parallel actor-learners enable stable deep reinforcement learning with value-based and policy-based methods across on-policy, off-policy, discrete, and continuous settings.

  • Takeaways & Limitations

    The framework leaves potential improvements from combining experience replay and other recent reinforcement learning advances unresolved.

Abstract

from arXiv · show

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

1. Introduction

The paper replaces experience replay with asynchronous parallel agents, whose diverse trajectories decorrelate data and stabilize training while enabling on-policy deep RL. Its best method, A3C, achieves strong Atari performance efficiently and generalizes across continuous control and visually guided 3D navigation.

  • Experience replay: Experience replay reduces non-stationarity and update correlations but requires extra memory and computation and restricts learning to off-policy algorithms.Replay samples or batches data from different time steps, often generated by an older policy.
  • Asynchronous parallelism: Asynchronous execution of multiple agents across parallel environment instances decorrelates data into a more stationary process.This paradigm enables a broader spectrum of on-policy reinforcement-learning algorithms than experience replay.
  • Empirical results: Asynchronous reinforcement learning achieves better results on many Atari 2600 games in far less time than previous GPU-based algorithms.The experiments run on a single machine and use fewer resources than massively distributed approaches.
  • Empirical results: A3C masters varied continuous motor-control tasks and learns general exploration strategies in 3D mazes from visual inputs.The method succeeds across 2D and 3D games, discrete and continuous action spaces, and feedforward and recurrent agents.

2. Related Work

Prior work explored asynchronous, parallel, and distributed reinforcement learning through Gorila, parallel batch and Sarsa methods, and convergence analyses. Related approaches also parallelized evolutionary fitness evaluation for visual reinforcement learning tasks.

  • Distributed reinforcement learning: Gorila asynchronously trains reinforcement learning agents in a distributed setting using separate actors, replay memories, and learners across processes.Each learner samples from its replay memory and computes DQN-loss gradients with respect to policy parameters.
  • Parallel reinforcement learning: MapReduce parallelized batch reinforcement learning with linear function approximation, accelerating matrix operations without parallelizing experience collection or stabilizing learning.Parallel Sarsa instead used multiple separate actor-learners to accelerate training.
  • Convergence theory: Asynchronous optimization analyses established that Q-learning can converge with outdated information when it is eventually discarded and technical assumptions hold.Earlier work also studied distributed dynamic programming.
  • Evolutionary methods: Parallel evolutionary methods distribute fitness evaluations across machines or threads and have been applied to visual reinforcement learning.One example evolved convolutional neural network controllers for TORCS using fitness evaluations on 8 CPU cores in parallel.

3. Reinforcement Learning Background

The section defines reinforcement learning as policy-driven interaction with an environment to maximize discounted returns, then introduces value-based, policy-based, and actor-critic methods. It also explains how n-step returns improve reward propagation compared with one-step updates.

  • Reinforcement learning setting: An agent repeatedly observes states, selects actions under policy π, receives rewards and successor states, and continues until reaching a terminal state.The policy maps states to actions within the standard discrete-time interaction setting.
  • Returns and value functions: The discounted return Rt = P∞ k=0 γ^krt+k measures accumulated rewards, and the agent aims to maximize its expected return from each state.The discount factor satisfies γ ∈ (0, 1].
  • Returns and value functions: Action values quantify expected returns for state-action choices, while Q∗(s, a) is the maximum action value achievable by any policy and Vπ(s) is the expected return under π.These definitions distinguish optimal action values from policy-specific action and state values.
  • Value- and policy-based methods: Value-based methods approximate Q∗(s, a) with a function approximator, whereas policy-based methods directly parameterize π(a|s; θ) and apply gradient ascent on E[Rt].Q-learning is presented as an example of value-based learning, and REINFORCE as an example of policy-based learning.
  • Multi-step learning: n-step Q-learning propagates one reward directly to n preceding state-action pairs, potentially improving efficiency over one-step methods that update only the pair producing the reward.The n-step target combines rewards through time t+n−1 with a bootstrapped value at t+n.
  • Actor-critic methods: Actor-critic methods use a learned value estimate as a baseline, reducing policy-gradient variance and interpreting Rt − bt as an advantage estimate.The advantage is defined as A(at, st) = Q(at, st) − V(st).

4. Asynchronous RL Framework

The framework trains deep neural network policies with asynchronous actor-learners running on multiple CPU threads, using parallel exploration and accumulated updates to improve stability and efficiency. It instantiates four asynchronous RL algorithms and finds shared-statistics RMSProp more robust than the alternatives on a subset of Atari games.

  • Framework: Four asynchronous variants cover one-step Sarsa, one-step Q-learning, n-step Q-learning, and advantage actor-critic.The framework targets reliable deep-policy training without large resource requirements.
  • Framework: Multiple CPU threads on one machine enable asynchronous actor-learners, avoid communication costs, and support Hogwild!-style updates.Unlike Gorila, the framework uses multiple threads on a single machine rather than separate machines and a parameter server.
  • Framework: Parallel actor-learners explore different parts of the environment, making online parameter updates less correlated and removing the need for experience replay.Different exploration policies can increase diversity across threads.
  • Framework: Accumulating gradients over multiple timesteps reduces actors overwriting each other’s updates while trading computational efficiency against data efficiency.The asynchronous Q-learning variant also uses a shared, slowly changing target network.
  • Framework: Policy entropy regularization improves exploration by discouraging premature convergence to suboptimal deterministic policies.A3C shares non-output convolutional features between its policy and value-function outputs in practice.
  • Optimization: Shared-statistics RMSProp is considerably more robust than SGD with momentum and RMSProp without shared statistics on a subset of Atari 2600 games.The framework investigates all three optimization algorithms.

5. Experiments

Experiments evaluated the asynchronous methods across Atari, TORCS, continuous-control, and visual 3D-maze tasks. The results show successful Atari training, strong A3C performance, reasonable visual-maze behavior, and robustness to parallelism and hyperparameter variation.

  • Atari: All four asynchronous methods successfully trained neural-network controllers on Atari and tended to learn faster than DQN.DQN used one Nvidia K40 GPU, whereas asynchronous methods used 16 CPU cores on five Atari games.
  • Atari: A3C significantly improved the state-of-the-art average score across 57 Atari games while using half the training time.A3C agents trained for four days on 16 CPU cores; other agents trained for 8 to 10 days on Nvidia K40 GPUs.
  • Continuous control: A3C was evaluated on continuous MuJoCo motor-control tasks because it extends readily to continuous actions, using either physical state or pixels.The tasks covered rigid-body physics with contact dynamics, including manipulation and locomotion.
  • Labyrinth: A3C learned a reasonable strategy for exploring randomly generated 3D mazes from only 84 × 84 RGB images, reaching a final average score of around 50.The Labyrinth task rewarded finding apples and portals in new mazes generated at the start of each episode.
  • Parallelism: Asynchronous one-step Q-learning and Sarsa exhibited superlinear speedups, with parallel actor-learners often reducing the data required to reach a given score.The authors attribute this effect to multiple threads reducing bias in one-step methods.
  • Stability and robustness: Across five Atari games, A3C showed a wide range of learning rates in which all random initializations achieved good scores, indicating robustness to learning rates and initial weights.The study trained each asynchronous algorithm with 50 learning rates and random initializations.

6. Conclusions and Discussion

The paper concludes that asynchronous actor-learners enable stable deep reinforcement learning across algorithmic families and domains, while identifying experience replay and algorithmic or architectural combinations as promising improvements.

  • Main findings: Asynchronous versions of four standard reinforcement learning algorithms trained neural network controllers stably across value-based and policy-based, on-policy and off-policy, discrete and continuous settings.The framework was evaluated across a variety of domains.
  • Main findings: Parallel actor-learners stabilised learning for the three value-based methods considered, demonstrating that online Q-learning can be stable without experience replay.The authors clarify that this does not imply experience replay is generally useless.
  • Future improvements: Experience replay could improve data efficiency by reusing old data, potentially accelerating training where environment interaction costs more than model updates.The authors specifically identify domains like TORCS as a setting where this trade-off matters.
  • Future improvements: Combining existing reinforcement learning methods and recent deep reinforcement learning advances with the asynchronous framework offers immediate opportunities to improve the proposed methods.The paper also contrasts forward-view corrected n-step targets with backward-view eligibility traces.
  • Future improvements: Complementary neural-network changes, including dueling architectures and spatial softmax, could improve value estimation and feature-coordinate representation.The dueling architecture separates state-value and advantage streams.

Supplementary Material for "Asynchronous Methods for Deep · 7. Optimization Details

The asynchronous framework evaluates lock-free stochastic gradient descent and RMSProp variants, including per-thread and shared RMSProp state. Their sensitivity was compared across learning rates, random initializations, reinforcement-learning methods, and four Atari games.

  • 7. Optimization Details: The framework investigates asynchronous stochastic gradient descent and RMSProp without locks to maximize throughput across many threads.Both optimization algorithms are implemented in a lock-free asynchronous setting.
  • 7. Optimization Details: Momentum SGD independently updates shared parameters using each thread’s accumulated gradients, momentum, learning rate, and a lock-free parameter update.The update is m_i = αm_i + (1 − α)∆θ_i followed by θ ← θ − ηm_i.
  • 7. Optimization Details: The supplement notes that standard non-centered RMSProp has not been extensively studied in asynchronous optimization.RMSProp is described as widely used in deep learning, but comparatively understudied asynchronously.
  • 7. Optimization Details: Two asynchronous RMSProp variants differ in whether the elementwise squared-gradient moving average is maintained per thread or shared asynchronously.The per-thread version is called RMSProp; the shared version is called Shared RMSProp.
  • 7. Optimization Details: The three asynchronous optimization algorithms were compared for sensitivity to learning rates and random network initializations.The comparison includes RMSProp, Shared RMSProp, and asynchronous momentum SGD.
  • 7. Optimization Details: 50 experiments per curve used different random learning rates and initializations to compare Async n-step Q and Async Advantage Actor-Critic across four games.The games were Breakout, Beamrider, Seaquest, and Space Invaders.

8. Experimental Setup

Experiments used 16 asynchronous actor-learners on one GPU-free machine, with fixed update and optimization settings across Atari and TORCS. Value-based methods used shared target networks and randomized exploration, while advantage actor-critic used entropy regularization and repeated random initializations.

  • Compute and optimization: 16 actor-learner threads ran on a single machine without GPUs across the Atari subset and TORCS experiments.Updates occurred after every 5 actions (tmax = 5 and IUpdate = 5), using shared RMSProp optimization.
  • Value-based methods: 40,000 frames was the update interval for the shared target network used by all three asynchronous value-based methods.The target network was shared across value-based methods.
  • Exploration: 0.4, 0.3, 0.3 were the probabilities for sampling exploration rates ϵ1, ϵ2, ϵ3, respectively.The values were annealed over the first four million frames from 1 to 0.1, 0.01, and 0.5, respectively.
  • Actor-critic and repetitions: β = 0.01 was the entropy-regularization weight for advantage actor-critic in all Atari and TORCS experiments.The setup included 50 experiments for five Atari games and every TORCS level, each with a different random initialization.

9. Continuous Action Control Using the MuJoCo Physics Simulator

Asynchronous advantage actor-critic was applied to MuJoCo continuous-control tasks with state or pixel inputs and a continuous-action entropy cost. It found solutions across all domains, with most state-based tasks solvable within a few hours on CPU.

  • Experimental setup: The MuJoCo setup closely matched the discrete-action experiments, differing mainly in the requirements of continuous-action domains.The physics models and task objectives were near-identical to those examined by Lillicrap et al. (2015), but rewards and performance were generally not comparable.
  • Inputs and architecture: Experiments used joint positions, velocities, and task targets as physical-state inputs, while pendulum, pointmass2D, and gripper were also trained from RGB pixels.Low-dimensional state inputs were mapped through one hidden layer containing 200 ReLU units.
  • Exploration: A differential-entropy cost on the actor’s normal output distribution encouraged exploration, using a constant multiplier of 10^-4 across all tasks.This was the continuous-action counterpart to the entropy cost used in the discrete-action experiments.
  • Results: The asynchronous advantage actor-critic algorithm found solutions for all MuJoCo domains, with most state-based domains solvable within a few hours.Learning curves were evaluated against wall-clock time.
  • Results: All experiments, including those using pixel observations, ran on CPUs.The reported CPU execution included experiments solving domains directly from pixel-based observations.
Loading 1602.01783v2…