Source-linked AI summary
Bigger, Better, Faster: Human-level Atari with human-level efficiency
Max Schwarzer, Johan Obando-Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, Pablo Samuel Castro
TL;DR
Sample-efficient online RL must scale to challenging tasks despite limited environment interaction. BBF addresses this by combining larger value-estimation networks with design choices that support efficient training, achieving super-human Atari 100K performance with substantially improved compute efficiency.
Problem
How can deep RL networks be scaled when samples are scarce, given that naïve scaling can rarely maintain performance on Atari 100K?
Method
BBF combines a larger Impala-CNN-based value network with receding update horizons, increasing discount factors, periodic resets, and other training design choices.
Results
BBF achieves super-human Atari 100K performance with only roughly two hours of gameplay and about 6 hours on a single GPU, while offering a favorable performance–compute trade-off.
Takeaways & Limitations
BBF shows that larger networks and replay ratios can be used effectively in sample-efficient RL when supported by suitable training techniques.
Takeaways & Limitations
Atari 100K covers only 26 of the 55 ALE games and omits sticky actions, so benchmark results do not fully represent the broader ALE setting.
Abstract
from arXiv · showhide
We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this scaling in a sample-efficient manner. We conduct extensive analyses of these design choices and provide insights for future work. We end with a discussion about updating the goalposts for sample-efficient RL research on the ALE. We make our code and data publicly available at https://github.com/google-research/google-research/tree/master/bigger_better_faster.
1. Introduction
Deep RL can achieve human or super-human performance, but such successes often require enormous networks and environment experience. BBF targets sample-efficient online RL on Atari 100K, achieving super-human performance while improving computational efficiency over EfficientZero.
- Large-scale RL successes often depend on large neural networks and enormous numbers of environment samples.The paper notes that human players would require tens of thousands of years to gather experience comparable to OpenAI Five or AlphaGo.
- 5× fewer environment samples are required by BBF than SR-SPR to reach human-level performance across 26 games, while representative model-free methods require at least 16× more samples.
- Scaling environment samples is especially difficult in real-world domains such as healthcare and robotics, while offline RL can struggle with distribution shifts or limited performance.
- Sample-efficient online RL methods are increasingly important, although problem variability makes a single solution unrealistic.Established benchmarks can still provide transferable insights for challenging, sample-scarce domains.
- BBF is a model-free RL agent for Atari 100K that achieves super-human performance, defined as an interquartile mean human-normalized score above 1.0.Atari 100K constrains agents to roughly two hours of gameplay; BBF is also described as more computationally efficient than EfficientZero.
2. Background
The paper frames RL as learning policies and value functions in a Markov Decision Process, with value-based methods approximating optimal action values. Its background also covers replay, multi-step learning, and prior architectural advances relevant to BBF.
- RL foundations: An RL problem is modeled as an MDP with states, actions, transitions, and rewards, while a policy maps states to action distributions.
- RL foundations: A policy’s value is its expected discounted reward, and the optimal policy maximizes this quantity over all policies.The discount factor weights rewards received later in a trajectory less heavily.
- Value-based RL: Value-based methods approximate optimal Q-values through the Bellman recurrence and derive actions by selecting the highest-valued action.
- Prior methods: DQN established deep value-based RL on the ALE using a network with three convolutional and two fully connected layers to approximate Q.
- Prior methods: Rainbow combines prioritized experience, n-step learning, distributional RL, double Q-learning, dueling architecture, and NoisyNets.Prior work identified multi-step learning as one of Rainbow’s most crucial components.
- Learning mechanisms: Replay buffers improve sample efficiency by reusing past experience in mini-batch updates, with replay ratio measuring updates per environment interaction.DQN uses replay ratio 0.25, whereas some sample-efficient Rainbow-based agents use 1.
3. Related Work
Atari 100K was introduced as a sample-efficiency benchmark that motivated data augmentation, self-supervision, resets, and model-based approaches. Related scaling work suggests larger networks can help, but the paper asks how to scale them when samples are scarce.
- Sample-efficient RL: DrQ uses data augmentation, while DER and DrQ(ε) improve existing model-free algorithms primarily through hyperparameter changes.
- Sample-efficient RL: SPR adds a BYOL-based self-supervised temporal consistency loss and data augmentation, while SR-SPR adds periodic network resets.SR-SPR achieved state-of-the-art performance on Atari 100K at the time described.
- Model-based RL: EfficientZero learns a discrete-action latent dynamics model and selects actions with lookahead MCTS, while IRIS uses an autoencoder and autoregressive Transformer world model.
- Scaling in deep RL: Prior deep RL studies explored how network size and structure affect performance, including scaling Impala-style ResNet features and large offline-RL networks.
- Scaling in deep RL: Online RL work has also scaled dynamics models and transformer encoders, but the paper focuses specifically on sample efficiency rather than offline RL or generalization.
4. Method
BBF investigates how to scale value-estimation networks under the 100K-step Atari constraint. It combines a wider Impala-CNN with resets, receding update horizons, discount-factor increases, regularization, and high replay ratios.
- BBF targets Atari 100K, where agents receive 100K environment steps, by asking how networks can be scaled when samples are scarce.
- BBF achieves super-human Atari 100K performance with about 6 hours on a single GPU, using previously released components and IQM evaluation.The implementation is based on Dopamine, and IQM averages the middle 50% of runs across games and seeds.
- BBF uses replay ratio 8 to balance the computation of its larger network, while also reporting replay ratio 2 results for lower computational cost.
- Larger network: BBF uses a 15-layer Impala-CNN and scales each layer’s width by 4×, selecting 4× because 8× performed comparably while costing more.
- Receding update horizon: The update horizon decreases exponentially from 10 to 3 over 10K gradient steps after each reset, outperforming fixed n = 3 and n = 10 schedules.The annealing phase remains 25% of training because resets occur every 40K gradient steps.
5. Analysis
The analysis finds that BBF’s gains depend on coordinated design choices rather than network scale alone. Its performance remains strong across replay ratios, unseen games, and sticky-action settings, with a favorable compute trade-off.
- Self-supervision: Removing the self-supervised SPR objective causes substantial performance degradation, highlighting self-supervision as an important component in sample-constrained learning.
- Replay ratio: BBF improves with replay ratio and remains roughly 0.45 IQM above SR-SPR across replay ratios, although the linear interaction between replay ratio and network scaling remains unexplained.
- Target networks: EMA target networks become critical with network scaling at replay ratios 2 and 8, despite earlier Atari 100K methods often omitting them.
- BBF’s design choices generally provide greater benefit on held-out games, and sticky actions do not significantly affect performance, although held-out games are more challenging.
- Reset strength: Periodic resets and more aggressive perturbations improve performance, with their benefits confirmed on both Atari 100K games and 29 unseen games.
- Scale: Naively scaling SR-SPR’s CNN is insufficient: SR-SPR performance collapses as network size increases, while the smaller Impala-CNN ResNet performs better across width scales.
- Computational efficiency: BBF takes 10 hours using one CPU and half an A100 GPU, providing the best GPU-hour performance–compute trade-off among the compared agents.
6. Revisiting the Atari 100k benchmark
The Atari 100K benchmark remains informative because aggregate human-level performance does not imply human-level performance on every game, and its limited game coverage raises generalization questions. BBF addresses these concerns by performing strongly on sticky-action and held-out games while extending sample-efficiency comparisons beyond 100K steps.
- Benchmark scope: IQM ≥1.0 after 100K steps does not mean any agent achieves human-level performance on all 26 Atari 100K games.The optimality gap remains above zero without dramatically more than two hours of data.
- Benchmark scope: The benchmark’s 26 games exclude 29 games from the full 55-game ALE suite and omit sticky actions, creating a test of generalization.Sticky actions repeat the previous action with 25% probability.
- Generalization: BBF’s performance is not significantly harmed by sticky actions, and its design choices generally provide greater benefits on the harder held-out games.The held-out games are significantly more challenging than the Atari 100K games, including for DQN (Nature).
- Sample efficiency: BBF roughly matches DQN’s performance at 256 hours using only two hours of gameplay on the standard Atari setting.This comparison motivates matching Rainbow’s final performance with two hours of gameplay as a new milestone.
- Beyond 100K steps: At replay ratio 2, BBF remains ahead of SPR and SR-SPR through one million environment steps, while those methods stagnate or lose their relative advantage.BBF was trained without changing parameters, including resets, past 100K steps.
- Beyond 100K steps: BBF at replay ratio 2 can outperform most recent algorithms after 20K–50K steps, and 40K steps require only half an A100 GPU for one hour.Those comparison algorithms did not use sticky actions.
7. Discussion and Future Work
BBF reaches super-human ALE performance with two hours of gameplay while improving the use of large networks and replay ratios in a computationally efficient setting. The results motivate broader benchmark evaluation and further study of reset mechanisms and self-supervised objectives.
- Conclusions: BBF achieves super-human ALE performance with only two hours of gameplay and handles larger networks and replay ratios better than prior agents.The conclusion identifies these capabilities as crucial for network expressivity and learning efficiency.
- Future work: Periodic resets enable BBF’s use of large replay ratios by balancing catastrophic forgetting and network plasticity.The authors identify more targeted alternatives to full-network resets as a future research direction.
- Benchmark directions: Atari 100K remains relatively cheap while challenging, and expanding its scope can support continued scientific progress despite apparent saturation.The paper presents this expanded scope as a starting point for future research.
- Future work: The paper calls for continued progress toward human-level performance across all tasks with human-level or superhuman efficiency.This is stated as the broader research goal motivating future work.
- Benchmark directions: BBF’s weakened configuration at replay ratio 2 outperforms recent algorithms by 50K steps, showing that the benchmark can expose progress at very small data budgets.The comparison appears in the learning curves for BBF, SR-SPR, and SPR.
A. Additional Results
Additional results examine how BBF’s scaling behaves across network widths and against Noisy Nets. Larger networks improve TD-error and gradient-related measures, but higher environment returns depend on BBF rather than scaling alone.
- Aggregate results: Aggregate scores and metrics for BBF and competing methods are reported across the 26 Atari 100K games.The table averages results across different numbers of seeds by method.
- Network scaling: Larger ResNet encoders consistently have lower TD errors, higher gradient norms, and higher parameter norms.These trends appear across width scales on the 26 Atari 100K games.
- Network scaling: Only BBF consistently translates the scaling-related parameter and gradient trends into higher environment returns.The large TD-error difference between BBF and SR-SPR is attributed to BBF’s shorter update horizon.
- Ablations: The appendix also compares BBF at replay ratio 2 with and without Noisy Nets on the 26 Atari 100K tasks.