Source-linked AI summary
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, Koray Kavukcuoglu
TL;DR
Training one reinforcement learning agent across many tasks creates major scalability demands in data volume and training time. The paper introduces IMPALA, which decouples acting and learning and uses V-trace for off-policy correction. IMPALA achieves high throughput and improved multi-task performance, including positive transfer across tasks.
Problem
Training a single agent on many tasks is challenging because existing methods can require a billion frames and multiple days for one domain.
Method
IMPALA combines distributed actor-learner training with the V-trace off-policy actor-critic algorithm to correct policy lag between actors and learners.
Results
IMPALA achieved better performance than A3C-based agents in multi-task settings, including 49.4% vs. 23.8% human normalised score on DMLab-30.
Takeaways & Limitations
IMPALA enables efficient investigation of new ideas at small and large scale and supports large-scale multi-task reinforcement learning.
Takeaways & Limitations
V-trace estimates the policy gradient for a truncated-policy target, with accuracy depending on the bias V^π̄ρ − V^π being small when truncation is sufficiently large.
Abstract
from arXiv · showhide
In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach.
1. Introduction
The paper targets reinforcement learning across diverse tasks with one agent, where the data and training demands make scalability a central challenge. IMPALA combines a scalable actor-learner architecture with V-trace and demonstrates high-throughput, multi-task learning.
- Training one agent across tens of domains is impractical when single-domain methods can require a billion frames and multiple days.
- IMPALA scales to thousands of machines by having actors send trajectories to a central learner, which performs GPU-based updates on trajectory batches.
- V-trace corrects the off-policy discrepancy caused when actor policies lag behind the learner policy during gradient calculation.
- 250,000 frames per second makes IMPALA over 30 times faster than single-machine A3C while also improving data efficiency and robustness.
- The evaluation trains one agent on DMLab-30 and Atari-57, demonstrating multi-task reinforcement learning across diverse environments.
2. Related Work
Prior distributed reinforcement learning methods use asynchronous workers, replay, evolutionary processes, or GPU batching, but synchronization and environment-speed variation can limit efficiency. IMPALA instead decouples acting from learning and uses V-trace to address the resulting off-policy instability.
- Distributed A3C and Gorila scale reinforcement learning through asynchronous stochastic gradient descent with multiple workers.
- Batched A2C is limited because the slowest environment in each batch determines the duration of the entire batch step.
- GA3C decouples acting from gradient calculation with dynamic batching, but its actor-learner asynchrony causes learning instabilities.
- Rendering-time variance within a batch can reduce GPU utilisation when actors synchronize every step or every n steps.
- IMPALA addresses this instability with the more principled V-trace algorithm, which performs off-policy correction for its decoupled architecture.
- Retrace requires state-action-value functions Q for correction, whereas V-trace is based on the state-value function V.
3. IMPALA
IMPALA separates experience generation from learning: distributed actors produce trajectories, while one or more learners update the policy off-policy using batches. This design reduces communication overhead and supports GPU acceleration and distributed training.
- IMPALA uses an actor-critic setup in which actors generate trajectories and learners use them to learn the policy off-policy.
- Actors periodically synchronize local policy µ with learner policy π, send trajectory data through a queue, and learners update π on batches from many actors.
- V-trace corrects policy lag between actors and learners while preserving extremely high data throughput and data efficiency.
- Actors can be distributed across many machines, while the learner can be accelerated with GPUs and distributed learners for large neural networks.
- Sending observations rather than parameters or gradients often lowers communication overhead while retaining fault tolerance like distributed A3C.
- Batching entire trajectories lets IMPALA parallelize more computation than online agents such as A3C.
4. V-trace
V-trace is IMPALA’s off-policy actor-critic correction for learning from trajectories generated by a behavior policy while evaluating a target policy. Its truncated importance weights separate the value-function fixed point from convergence speed and preserve the on-policy update as a special case.
- Off-policy correction: V-trace enables learning the value function V π from trajectories generated by a different behavior policy µ.This addresses the policy lag created by decoupled acting and learning in IMPALA.
- V-trace target: The n-step V-trace target combines discounted rewards, a bootstrapped value estimate, and truncated importance-sampling corrections.The target is defined for V(xs) using trajectories generated by actors following µ.
- On-policy special case: When data are on-policy and c̄ ≥ 1, V-trace reduces to the on-policy n-step Bellman update, allowing one algorithm for on- and off-policy data.The method also reduces to TD(λ) in the on-policy infinite-horizon case when the additional λ formulation is used.
- Fixed point: With infinite ρ̄, V-trace converges to the target policy’s value function; with finite ρ̄, its fixed point lies between the behavior and target policies.As ρ̄ approaches zero, the fixed point approaches the behavior policy’s value function.
- Truncation effects: The truncation levels c̄ and ρ̄ control different properties: c̄ affects convergence speed, whereas ρ̄ determines the value-function fixed point.The product of c coefficients controls how temporal-difference updates propagate backward, motivating c̄ as variance reduction.
- Actor-critic algorithm: The actor-critic implementation updates value parameters toward V-trace targets and policy parameters using an importance-weighted policy-gradient estimate.The estimate uses qs, constructed from the next-state V-trace estimate, with a value baseline optionally subtracted to reduce variance.
5. Experiments
Experiments evaluate IMPALA's computational performance, single-task convergence and stability, V-trace corrections, and multi-task transfer on DMLab-30 and Atari-57. Across these settings, IMPALA combines high throughput with strong performance, data efficiency, robustness, and positive transfer.
- 5.1. Computational Performance: IMPALA achieves 250,000 frames/sec, or 21 billion frames/day, in the distributed multi-machine setup.Single-machine experiments also report IMPALA as the highest-performing method on both evaluated tasks.
- 5.2.1. CONVERGENCE AND STABILITY: In 4 of 5 DeepMind Lab tasks, IMPALA or batched A2C reaches the best final average return, while IMPALA exceeds A3C during most training.IMPALA outperforms synchronous batched A2C on 2 of 5 tasks while achieving higher throughput.
- 5.2.1. CONVERGENCE AND STABILITY: IMPALA achieves higher scores over more hyperparameter combinations than A3C, indicating greater robustness to hyperparameter choices.The comparison sorts final performance across hyperparameter combinations from high to low.
- 5.3.1. DMLAB-30: On DMLab-30, multi-task IMPALA outperforms IMPALA experts throughout training and shows positive transfer on language and laser tag tasks.The deep IMPALA variant also outperforms the shallow network throughout training and in final performance.
- 5.3.2. ATARI: On Atari-57, a single multi-task IMPALA agent reaches 59.7% median human normalised score while remaining competitive with the A3C shallow expert baseline.The result covers all 57 games despite their diverse visual appearance and game mechanics.
6. Conclusion
IMPALA combines a scalable distributed agent architecture with the V-trace off-policy algorithm. Together, they improve data efficiency, stability, and performance across large-scale multi-task Deep-RL settings.
- IMPALA and V-trace provide a highly scalable distributed framework for Deep-RL.The architecture is designed to use compute efficiently at both small and large scale.
- 49.4% vs. 23.8% human normalised score on DMLab-30, compared with A3C-based agents.The paper reports superior performance for IMPALA in this multi-task setting.
- IMPALA's DMLab-30 multi-task training produced positive transfer between individual tasks.The multi-task setting outperformed the expert-training setting.
- IMPALA was evaluated on both DMLab-30 and Atari-57 large-scale multi-task settings.The authors describe it as the first Deep-RL agent successfully tested in such settings, to their knowledge.
A. Analysis of V-trace
The V-trace analysis characterizes an off-policy operator with a unique fixed point and convergence properties under stated assumptions. Its two truncation levels separately control bias, variance, and convergence speed.
- With finite ρ̄, V-trace evaluates a policy between the behavior policy µ and target policy π.Larger ρ̄ reduces off-policy bias but increases variance; near-zero ρ̄ approaches V µ.
- The c̄ truncation level controls the contraction modulus and convergence speed without changing the fixed point.The fixed point depends on ρ̄, not c̄, while truncating c̄ reduces variance from coefficient products.
- The V-trace operator is a contraction under the stated condition 1 − (1 − γ)β < 1.Its contraction modulus is bounded below one, yielding a unique fixed point.
- The operator's unique fixed point is V π̄ρ, the value function of the policy π̄ρ.The fixed-point identity follows because the expression is the Bellman equation for V π̄ρ.
- Under finite tabular spaces, infinite visitation, and Robbins-Munro stepsizes, Vk converges almost surely to V π̄ρ.The convergence argument applies stochastic approximation to the contraction operator's fixed point.
B.1. Final training scores on DMLab-30
This section presents DMLab-30 training-score figures and records the experimental settings and hyperparameter ranges used in the associated experiments.
- Figure B.1 reports human normalised scores across all DMLab-30 tasks.
- Table C.1 reports Atari scores after 200M environment steps, with up to 30 no-ops at episode starts.
- The experiments specify fixed settings, swept hyperparameter ranges, and the action set used across DeepMind Lab tasks.The appendix includes reward preprocessing and fixed model hyperparameters.
E.2. V-trace Stability Analysis
The appendix compares V-trace with alternative value-estimation and off-policy-correction variants. Its stability analysis focuses on policy lag, replay, and hyperparameter combinations.
- V-trace is much more stable across a wide range of parameter combinations than ε-correction and pure on-policy learning when using replay.
- The appendix evaluates multiple variants for estimating the state-action value function, using averages over the top three runs and all runs.
F. Population Based Training
The Population Based Training setup uses a 20-million-frame burn-in and compares a discovered learning-rate schedule with linear annealing. The figure highlights the best parameter-sweep run.
- 20 million frames are used as a burn-in period without evolution to stabilize training and avoid rapid initial adaptation that hinders diversity.
- After 5,000 episode rewards are collected, the mean capped human normalised score is calculated and a random population instance is selected.
- A selected instance is copied when its score exceeds the comparison threshold by more than an absolute 5%.
- The figure compares the PBT-discovered learning-rate schedule with linear annealing and marks the best parameter-sweep run in red.
G. Atari Experiments
The Atari experiments use feed-forward agents with standardized frame preprocessing, while expert agents and multi-task agents follow specified training configurations. Hyperparameters for the Atari experiments are summarized in a dedicated table.
- All Atari agents use only a feed-forward network and preprocess frames as described in Mnih et al. (2016).
- IMPALA and A3C use the same hyperparameters when training expert agents on each game.
- The shared expert-agent hyperparameters were tuned with a shallow A3C network on six Atari games.The games were Breakout, Pong, Space Invaders, Seaquest, Beam Rider, and Q*bert.
- A dedicated table lists the hyperparameters used for the Atari experiments.