Source-linked AI summary
Agent57: Outperforming the Atari Human Benchmark
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Charles Blundell
TL;DR
RL agents have struggled to achieve human-level performance consistently across all 57 Atari games, especially where exploration and long-term credit assignment are difficult. Agent57 combines a family of exploratory-to-exploitative policies with adaptive policy selection and separate intrinsic/extrinsic value-function parameterization. It achieves performance above the human benchmark on all 57 games, while relying on substantial computation and leaving data efficiency as an open direction.
Problem
Prior RL agents performed strongly on many Atari games but failed on several challenging games, and none exceeded 100% HNS across all 57 games with one hyperparameter set.
Method
Agent57 trains a family of policies with different exploration rates and discount factors, adaptively prioritizes them, and separately parameterizes intrinsic- and extrinsic-reward value functions.
Results
Agent57 achieves performance above the human benchmark across all 57 Atari games, including 100% capped human normalized score.
Takeaways & Limitations
Balancing exploration, exploitation, and long-term credit assignment can produce uniform human-benchmark performance across the Atari 57-game suite.
Takeaways & Limitations
The method uses substantial computation, and improving its data efficiency remains an open research direction.
Abstract
from arXiv · showhide
Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.
1. Introduction
The Atari benchmark tests general RL competency across diverse games, but prior agents failed on some challenging games. Agent57 addresses exploration, long-term credit assignment, and training stability through adaptive policy selection and reward-value parameterization.
- The Arcade Learning Environment evaluates general AI competency across a diverse set of engaging and challenging Atari 2600 games.
- No single RL algorithm had exceeded 100% HNS on all 57 games with one hyperparameter set; MuZero and R2D2 surpassed it on 51 and 52 games.
- Failures on games such as Skiing, Solaris, Private Eye, Montezuma’s Revenge, Pitfall!, and Venture expose long-term credit-assignment and exploration challenges.
- NGU allocates equal experience to policies despite games requiring different exploration levels, motivating adaptive allocation of network capacity and data collection.
- Agent57 improves stability by decomposing intrinsic and extrinsic reward contributions in the state-action value function across varied intrinsic-reward scales.
2. A meta-controller: an adaptive mechanism to select
Agent57 controls the exploration–exploitation trade-off by selecting among policies with different exploration rates and discount factors throughout training. Longer backpropagation-through-time windows also improve long-term credit assignment.
- The meta-controller chooses which policies, parameterized by exploration rate and discount factor, to prioritize during training.
- Adaptive policy prioritization lets the agent dedicate shared resources to exploration or exploitation as training progresses.
- A backpropagation-through-time window twice the R2D2 baseline improved long-term credit assignment in Solaris while maintaining or improving other-game performance.
- These improvements transformed NGU into Agent57, which outperformed the human baseline uniformly across all 57 Atari games.
2. Background: Never Give Up (NGU)
NGU combines curiosity-driven exploration with distributed deep RL by training a family of policies using different intrinsic-reward scales, exploration rates, and discount factors. Its shared value-function architecture can become unstable when reward scales and variances differ.
- NGU combines curiosity-driven exploration with a distributed deep RL architecture descended from R2D2.
- Intrinsic reward combines rapidly vanishing episodic novelty with slowly vanishing lifelong novelty to encourage exploration within episodes and throughout training.
- NGU adds N intrinsic-reward scales β_j r_i,t to extrinsic reward and learns associated state-action value functions with exploration rates β_j and discount factors γ_j.
- Higher β_j values encourage exploration, while lower values encourage exploitation; discount factors provide longer horizons for exploitative policies and shorter horizons for exploratory policies.
- NGU trains a recurrent network indexed by policy choice, but learning can become unstable when reward scales, sparsity, or noise differ substantially.
- The distributed agent separates data collection from learning: independent actors feed prioritized transition sequences into a central replay buffer sampled by the learner.
3. Improvements to NGU
The improvements to NGU combine adaptive selection across a family of policies with separate intrinsic and extrinsic value-function parameterizations. These changes target policy specialization, dynamic exploration, and more stable learning.
- State-Action Value Function Parameterization: The state-action value function is split into extrinsic and intrinsic components, Q(x, a, j; θ) = Q(x, a, j; θe) + βjQ(x, a, j; θi).The two components use separately parameterized neural networks and rewards, while sharing the same target policy.
- State-Action Value Function Parameterization: Separate value networks allow adaptation to the scale and variance of each reward component while keeping optimizer states separate.The paper states that this parameterization is theoretically equivalent to optimizing the original value function with combined reward re + βjri under simple gradient descent.
- Adaptive Exploration over a Family of Policies: NGU jointly trains policies with different exploration rates and discount factors, while a meta-controller adaptively selects which policies to prioritize during training and evaluation.The policy family is indexed by pairs (βj, γj), and selection can allocate network capacity and data collection toward policies relevant to the task.
- Adaptive Exploration over a Family of Policies: A non-stationary sliding-window UCB bandit chooses one policy arm at the beginning of each episode using extrinsic episode returns as its reward signal.The sliding window allows adaptation because the agent’s rewards change throughout training, while εUCB-greedy exploration occasionally selects a random arm.
- Adaptive Exploration over a Family of Policies: The same meta-controller idea can adjust discount factors in R2D2 (Retrace) even without intrinsic rewards, producing the R2D2 (bandit) variant.R2D2 (Retrace) uses the same RL loss as Agent57, while R2D2 (bandit) adds joint training over several discount-factor policies.
4. Experiments
Agent57’s experiments evaluate its components on all 57 Atari games and a 10-game challenging set, emphasizing performance across difficult exploration and long-term credit-assignment tasks. The results show that adaptive exploration, separate intrinsic and extrinsic value networks, and longer backpropagation windows each improve performance and together support consistent 100% CHNS.
- Summary of the Results: Agent57 achieves 100% CHNS across all 57 Atari games, with mean and median scores exceeding those of NGU and R2D2.The capped score emphasizes performance on games below the human benchmark, while the mean and median summarize broader performance.
- Summary of the Results: The challenging set contains 10 games combining hard exploration with long-term credit-assignment demands.The set includes Beam Rider, Freeway, Montezuma’s Revenge, Pitfall!, Pong, Private Eye, Skiing, Solaris, Surround, and one additional game listed in the experiments.
- Ablations: Each Agent57 improvement increases final performance on the challenging set, and all are needed for consistent 100% CHNS.Figure 4 tracks performance as the improvements are incorporated on top of NGU.
- State-Action Value Function Parameterization: Separate intrinsic and extrinsic value networks improve robustness across intrinsic-reward scales and avoid worse performance on any challenging-set game.Removing this improvement causes Agent57’s performance to drop by more than 20%.
- Long-Term Credit Assignment: A longer backpropagation-through-time window is initially slower but yields greater stability and a slightly higher final score for both R2D2 and Agent57.The comparison uses windows of 80 versus 160, with the effect especially visible on Solaris.
- Adaptive Exploration: Adaptive exploration improves R2D2’s final performance by close to 20% CHNS and allows per-task adjustment of exploration and discount settings.A fixed γ = 0.9999 surpasses the human baseline on Skiing but is unstable across the full game set; the meta-controller avoids that compromise.
5. Conclusions
Agent57 combines several improvements to address exploration, exploitation, and long-term credit assignment across Atari games. The paper reports performance above the human benchmark on all 57 games, while noting substantial computational demands and remaining performance gaps.
- The paper frames Agent57 as balancing exploration, exploitation, and long-term credit assignment across a diverse game set.
- The approach improves Never Give Up with separate state-action value parameterizations, adaptive novelty preference and discount, and longer Retrace backpropagation windows.
- Agent57 uses substantial computation and scales with increasing computation, motivating further work on data efficiency.
- An average capped human normalized score of 100% is reported, but the authors state that Atari research still has room to improve efficiency and general performance.
B. Extrinsic-Intrinsic Decomposition
The paper shows that intrinsic and extrinsic value functions can be learned separately and combined to recover value iteration for their mixed reward. This decomposition is extended to transformed operators and Retrace-based neural-network optimization.
- An intrinsically motivated agent represents reward as a linear combination of extrinsic reward re and intrinsic reward ri.
- Separately computed extrinsic and intrinsic state-action values can be mixed to produce the same behavior as directly computing values for re + βri.
- The decomposition extends to transformed value functions, which converge to the optimal value function for the mixed reward under the stated scheme.
- Retrace provides off-policy evaluation or control, and its value-iteration scheme converges to the optimal state-action value function under the cited conditions.
- In neural-network practice, sampled Retrace targets are fitted with online and target networks because the function space and full operator evaluation are impractical.
E. Implementation details of the distributed setting
The distributed Agent57 system uses actors, an evaluator, a learner, replayed sequences, and separate intrinsic and extrinsic losses. Multi-armed bandits select policy parameters for actors and evaluation.
- Replay buffer: Replay stores fixed-length transition sequences, with experiments using trace length 160 and replay period 80 or trace length 80 and replay period 40.
- Actors: Actors select a pair (γj, βj) at each episode beginning through a multi-armed bandit and keep that pair fixed throughout the episode.
- Actors: The 256 actors collect ϵ-greedy Atari experience and send transitions with priorities to the replay buffer.
- Evaluator: The evaluator alternates between training the bandit with undiscounted episode returns and greedy evaluation using the selected index.
- Learner: The learner samples replay sequences, computes separate transformed Retrace losses for intrinsic and extrinsic weights, and optimizes them with Adam.
F. Network Architectures
Agent57 uses a policy family spanning exploratory and exploitative behavior, with parameter pairs controlling novelty preference and discount. The implementation also specifies distributed training and Atari preprocessing settings.
- The policy family encourages both highly exploitative and highly exploratory policies by varying β with a sigmoid schedule.
- The γ values pair long horizons with exploitative policies and shorter horizons with exploratory policies because extrinsic rewards are sparse while intrinsic rewards are dense.
- Agent57 uses N = 32 policy-parameter pairs, with γ0 = 0.9999, γ1 = 0.997, and γ2 = 0.99.
- The reported implementation includes Atari preprocessing hyperparameters, Agent57 hyperparameters, and swept hyperparameter ranges.
H.1. Atari 10: Table of Scores for the Ablations
The ablations compare backpropagation window lengths and identity versus h-transform mixes on a 10-game challenging set, alongside Agent57 learning curves on Atari57. The analysis indicates that combining extrinsic and intrinsic rewards matters more than the specific mixing form.
- Figures 12 and 13 compare short versus long backpropagation windows and identity versus h-transform mixes on the 10-game challenging set.
- Identity and h-transform mixes do not seem to differ in performance on the 10-game challenging set.
- Combining extrinsic and intrinsic rewards is identified as the important factor, regardless of whether the combination is linear.
- At extreme β values, transform and normal value-iteration schemes converge toward the same policy because h^-1 is strictly increasing.
- Figure 14 presents Agent57 learning curves on Atari57.
H.6. Videos
The videos demonstrate Agent57 across all 57 Atari games and illustrate how its policy exploration, value-function parameterization, adaptive discounting, and long-horizon credit assignment operate in selected games.
- Agent57 has an example video for each Atari 57 game in which it surpasses the human baseline.
- Ice Hockey videos contrast exploratory policies that seek new configurations with exploitative policies that pursue goals.
- James Bond videos compare R2D2 retrace at about 30,000 points with R2D2 bandit at around 140,000 points.
- The James Bond adaptive mechanism selects policies with very high discount factors to achieve the higher-scoring behavior.
- Solaris videos illustrate long-term credit assignment, linking grid-screen actions with rewards received many time steps later.