PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU

Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, Jan Kautz

arXiv:1611.06256v3cs.LG

TL;DR

Deep RL training generates small batches sequentially, making GPU resources difficult to utilize efficiently. The paper introduces GA3C, a hybrid CPU/GPU A3C architecture with queues and dynamic scheduling, and reports substantially faster data processing than CPU A3C. Its computational analysis also identifies trade-offs involving throughput, batching, agent count, and policy lag.

  • Problem

    Sequential data generation, small batches, and shared inference and training workloads can severely under-utilize GPUs in deep RL.

  • Method

    GA3C combines CPU/GPU A3C with queues, dynamic scheduling, automatic tuning, and computational analysis of throughput and resource-allocation trade-offs.

  • Results

    ~45× faster for larger DNNs and ~6× faster for small DNNs, GA3C generates and consumes training data faster than the CPU counterpart.

  • Takeaways & Limitations

    GA3C scales with DNN size more efficiently than CPU A3C and its findings may apply to similar asynchronous RL algorithms.

  • Takeaways & Limitations

    The analysis focuses on A3C, while policy lag can cause instability and large batches trade lower update rates for greater learning stability.

Abstract

from arXiv · show

We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU's computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C .

1 INTRODUCTION

Deep RL reduced reliance on hand-crafted features, but its sequential data-generation process creates GPU-utilization challenges. The study implements and tunes CPU and hybrid CPU/GPU A3C systems, with GA3C substantially faster for both small and large DNNs.

  • Deep RL replaced task-specific features with DNN-based value and policy approximators, enabling learning from raw pixels and super-human Go performance.
  • Sequential interaction means training data are generated during learning, while small models and batches create GPU contention and under-utilization.
  • The study implements CPU and GPU A3C versions in TensorFlow and tunes them to approximately replicate published Atari 2600 scores.
  • ~6× faster for small DNNs and ~45× for larger DNNs, GA3C generates and consumes training data faster than the CPU counterpart.

2 RELATED WORK

Related work shows that deep RL progress depends on both algorithmic advances and systems optimization. Distributed and specialized systems can improve results or training speed, but require substantial computational resources and memory.

  • DQN introduced a general deep RL approach using experience replay, improving reliability while increasing computational cost and memory footprint.
  • AlphaGo combined algorithmic and hardware specialization to achieve strong results through large-scale distributed inference and training.
  • Gorilla DQN used 100 concurrent actors on 31 machines, 100 learners, and a central parameter server, demonstrating scalable distributed RL.
  • These distributed systems achieved better results in less time but incurred increased computational load, memory footprint, and cost.

3 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C)

A3C uses actor-critic learning with a shared DNN for policy and value estimation, updating parameters from short asynchronous experience sequences. Its sequential training process limits naive GPU utilization.

  • 3.1 REINFORCEMENT LEARNING BACKGROUND: An RL agent repeatedly observes states, selects actions under a policy, receives rewards, and continues until termination or a time limit.
  • 3.1 REINFORCEMENT LEARNING BACKGROUND: Policy-based model-free learning seeks parameters for a policy π that maximizes expected reward, using a neural network as function approximator.
  • 3.1 REINFORCEMENT LEARNING BACKGROUND: The accumulated return discounts future rewards by γ, with γ ∈ (0, 1].
  • 3.1 REINFORCEMENT LEARNING BACKGROUND: A learned value baseline reduces policy-gradient variance, forming the critic alongside the actor policy.
  • 3.2 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C): A3C uses one DNN to approximate both policy and value functions, with convolutional layers followed by a fully connected layer and separate outputs.
  • 3.2 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C): The policy cost combines log policy probability, an advantage term, and entropy regularization weighted by β.
  • 3.2 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C): Training collects gradients from both cost functions and updates parameters using non-centered RMSProp.
  • 3.2 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC (A3C): Shared gradients across agent threads are known to be more robust than separated gradients.

4 HYBRID CPU/GPU A3C (GA3C)

GA3C reorganizes A3C around one GPU DNN, CPU agents, predictors, trainers, and queues to improve GPU utilization. Its dynamic scheduling balances throughput against policy lag and convergence stability.

  • 4.1 GA3C ARCHITECTURE: Predictors batch immediately available requests into GPU inference queries, while multiple predictors can hide latency.Trainers similarly process queued batches and may run in parallel; excessively large merged batches reduce convergence speed.
  • 4.1 GA3C ARCHITECTURE: GA3C uses one GPU DNN while CPU agents queue policy requests and experience batches for centralized prediction and training.Predictors batch inference requests; trainers submit experience batches for GPU model updates.
  • 4.2 PERFORMANCE METRICS AND TRADE-OFFS: TPS measures model updates per second, PPS measures prediction-query throughput, and balanced operation approximately satisfies PPS ≈ TPS × tmax.With four action repetitions, frames per second is 4×PPS.
  • 4.2 PERFORMANCE METRICS AND TRADE-OFFS: Computational tuning affects convergence: more agents can increase queue delay, while larger batches improve GPU occupancy but can reduce update rate and convergence speed.The relevant configuration must therefore be evaluated using both TPS and learning curves.
  • 4.2 PERFORMANCE METRICS AND TRADE-OFFS: Dynamic adjustment of NT, NP, and NA is proposed because the TPS-maximizing configuration depends on environment load, DNN size, and hardware.A fixed rule of two predictors and two trainers only occasionally matches the most efficient configuration.
  • 4.4 POLICY LAG IN GA3C: GA3C introduces policy lag because queued experiences may be consumed after model parameters have changed, producing gradients that can destabilize learning.A small ϵ prevents zero action probabilities from creating infinite logarithms and also improves stability, removing the necessity of gradient clipping.

5 ANALYSIS

The analysis profiles GA3C across hardware, model sizes, resource configurations, and learning settings, showing that throughput depends on balancing asynchronous components and GPU utilization. Dynamic configuration can approach optimal learning behavior, while implementation overhead and hardware-dependent scaling remain important constraints.

  • Experimental setup: GA3C experiments profile systems with and without automatic adjustment of agents, trainers, predictors, and queue sizes.The study uses GPU-enabled systems and CUDA-based profiling to examine system dynamics and convergence.
  • Resource utilization: Increasing agents raises TPS until NA =128, where diminishing returns appear; NA =128 with NP = NT = 2 delivers ∼4× CPU-only speed-up.The reported configuration achieves the highest consistent TPS on System I.
  • Model scaling: GA3C’s speed advantage grows with DNN size: the largest network is approximately 45× faster than CPU A3C, while GPU slowdowns range from 2.2× to 4.9× across systems.More recent Maxwell GPUs scale better with larger networks than the older Kepler system.
  • Implementation overhead: Profiling finds substantial overhead: prediction waits average 108ms with only 10% spent in GPU inference, while 59% of an 11.1ms training update is overhead.The paper suggests lower-level implementations could reduce these costs, but leaves that investigation for future work.
  • Resource utilization: Balanced predictor and trainer configurations matter: the best settings usually use four or fewer predictors and NP : NT ratios near 1 : 2, 1 : 1, or 2 : 1.A 14% TPS gap separates the best and worst configurations shown, while oversized training queues can throttle prediction.
  • Dynamic configuration: Dynamic configuration tracks the optimal learning curve and achieves nearly identical scoring despite starting from NT = NP = NA = 1.The search procedure changes configurations once per minute and incurs slightly fewer played frames.

6 CONCLUSION

GA3C combines hybrid CPU/GPU execution with flexible resource allocation to accelerate A3C and scale more efficiently as DNN controllers grow. The open-source implementation supports further research into computational strategies for asynchronous deep RL.

  • GA3C achieves significant speedups over the CPU A3C implementation through flexible allocation of available computational resources.
  • GA3C scales with DNN size more efficiently than the CPU A3C implementation, supporting exploration of larger DNN controllers for real-world RL problems.
  • Open-sourcing GA3C enables researchers to investigate deep RL computation and test solutions combining CPU and GPU resources.
Loading 1611.06256v3…