PaperlayerView exact revision on arXiv

Source-linked AI summary

Observe and Look Further: Achieving Consistent Performance on Atari

Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Večerík, Matteo Hessel, Rémi Munos, Olivier Pietquin

arXiv:1805.11593v1cs.LGcs.AIstat.ML

TL;DR

Deep RL has not consistently learned human-level policies across diverse Atari games, where reward variation, long horizons, and exploration pose combined challenges. The paper addresses them with a transformed Bellman operator, temporal consistency loss, and human demonstrations. It exceeds average-human performance on 40 of 42 games and completes the first level of MONTEZUMA’S REVENGE.

  • Problem

    Deep RL has not consistently achieved human-level policies across Atari games, which require handling diverse rewards, long horizons, and efficient exploration.

    Source: §1 · p. 1

  • Method

    The algorithm combines a transformed Bellman operator, temporal consistency loss, and human demonstrations with distributed online experience.

    Source: §1 · p. 2

  • Results

    40 of 42 Atari games exceeded average-human performance, and the algorithm completed the first level of MONTEZUMA’S REVENGE.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results show one algorithm addressing reward variation, long horizons, and exploration across a broad Atari benchmark.

    Source: §5 · p. 8

Abstract

from arXiv · show

Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of $\gamma = 0.999$ (instead of $\gamma = 0.99$) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters. Furthermore, it is the first deep RL algorithm to solve the first level of Montezuma's Revenge.

1 Introduction

Consistent human-level performance across Atari games remains open because agents must handle diverse rewards, long horizons, and efficient exploration together. The paper combines methods for these challenges and reports human-level performance on 40 of 42 games.

1 Introduction

  • Consistent performance across Atari remains open because agents must handle reward distributions, distant rewards, and sparse-reward exploration together.Reward clipping can alter optimal policies, while some rewards in MONTEZUMA’S REVENGE may be separated by hundreds of time steps.Source: §1 · pp. 1–2
  • The algorithm combines a transformed Bellman operator, temporal consistency loss, and demonstrations with distributed online experience to address reward variation, long horizons, and exploration.It uses unaltered rewards and a discount factor of γ = 0.999, extending the effective planning horizon by an order of magnitude.Source: §1 · pp. 1–2
  • 40 of 42 Atari games exceeded average-human performance, and the algorithm completed the first level of MONTEZUMA’S REVENGE.The evaluation used the same hyperparameters across 42 games.Source: §1 · p. 2

2 Related work

Prior work advanced deep Q-networks, expert-demonstration learning, and methods for stabilizing learning with unclipped rewards or high discounts. The paper situates its approach among these lines of work.

2 Related work

  • Expert-demonstration methods range from initialization and imitation losses to replay-buffer initialization, while DQfD combines temporal-difference and supervised classification losses.Earlier methods used handcrafted features and were not applied to large MDPs; deep networks enabled work on Atari and robotics.Source: §2 · p. 2
  • Deep Q-network extensions introduced prioritized replay and Ape-X DQN, which separates data collection across actors from learning by a central learner.Ape-X uses a central prioritized replay buffer.Source: §2 · p. 2
  • Prior approaches addressed high-discount instability with constrained TD updates on toy domains, while PopArt adaptively rescales targets to process unclipped rewards.Durugkar and Stone constrained updates to be orthogonal to the direction of maximum change of the next state; PopArt is described by van Hasselt et al. [19].Source: §2 · p. 2

3 Algorithm

The algorithm combines a transformed Bellman operator, temporal-consistency loss, and Ape-X DQfD, targeting reward-distribution robustness, longer discount horizons, and exploration with expert demonstrations.

3.1 DQN Background

  • DQN approximates the optimal action-value function with a neural network, while Bellman fixed-point iteration converges to Q* because T is a γ-contraction.The algorithm builds on this DQN background.Source: §3.1 · p. 3

3.2 Transformed Bellman Operator

  • Reward clipping stabilizes DQN but can change optimal policies; Th instead compresses action-value targets and, under stated conditions, retains the optimal policy.The chosen h reduces target scale, is Lipschitz continuous, and has a closed-form inverse.Source: §3.2 · pp. 3–4

3.3 Temporal consistency (TC) loss

  • As γ approaches 1, similar adjacent-state values can trigger catastrophic TD backups; the TC loss penalizes updates that change next-state action-value estimates.This keeps updated estimates consistent over time with the transformed operator Th.Source: §3.3 · p. 4

3.4 Ape-X DQfD

  • Ape-X DQfD combines the transformed operator and TC loss with DQfD and distributed prioritized replay, using actor and expert replay buffers plus supervised imitation loss.Training mixes actor and expert transitions from the outset at 75%/25%; imitation loss applies only to the best expert episode.Source: §3.4 · pp. 4–5

4 Experimental evaluation

On 42 Atari games, Ape-X DQfD with the deeper architecture exceeded average-human performance on 40 games; experiments also examined policy quality, component ablations, and alternatives. The results show stronger coverage across games alongside trade-offs in peak performance and the effects of demonstrations and training components.

4 Experimental evaluation

  • Evaluation covered 42 Arcade Learning Environment games using no-op-start and human-start tests; Ape-X experiments used 128 actors, compared with 360 in reported Ape-X DQN results.The study compared against Ape-X DQN, DQfD, Rainbow DQN, average-human scores, and expert scores.Source: §4 · p. 5, §4.1 · p. 6

4.1 Benchmark results

  • With the deeper architecture, Ape-X DQfD exceeded an average human on 40 of 42 games and reached a 702% median human-normalized score; Pitfall's 3,997 remained below 6,464 average-human points.The first result is shown for no-op starts; the Pitfall comparison is reported separately in the benchmark discussion.Source: Table 1 · p. 6, §4.1 · p. 6
  • The standard architecture exceeded average-human performance on 39 of 42 games, versus at most 32 for previous approaches, improving score-distribution tails while lowering some peak scores.The authors describe the result as increased consistency across games.Source: §4.1 · p. 6

4.2 Imitation vs. inspiration

  • Ape-X DQfD found more time-efficient policies than experts in all three illustrated games, exceeding the expert on HERO, matching it sooner on MONTEZUMA’S REVENGE, but trailing on MS. PACMAN.The figure compares cumulative undiscounted episode return over time for the best expert and agent episodes.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Ablation study

  • Across six sparse-, dense-reward, and strong-performing games, removing the transformed operator worsened performance; removing TC loss led to late collapse; demonstrations helped sparse games but could hurt SEAQUEST.Removing the imitation loss alone had no significant performance effect in this setup.Source: §4.3 · pp. 7–8

4.4 Comparison to related work

  • PopArt learned in some games but performed significantly worse overall, while constrained TD updates did not work in this setup.The PopArt experiment combined it with the standard Bellman operator; Figure 4 experiments ran for 40 hours and used different scales from Figure 3.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

The paper presents a deep RL algorithm combining approaches for diverse rewards, long horizons, and sparse-reward exploration. On Atari, it exceeds average-human performance on 40 of 42 games and completes Montezuma’s Revenge’s first level.

  • The algorithm exceeds average-human performance on 40 of 42 Atari games and is the first deep RL algorithm to complete Montezuma’s Revenge’s first level.Source: §5 · p. 8
  • The algorithm addresses diverse reward distributions, longer time horizons, and sparse-reward exploration with a transformed Bellman operator, temporal consistency loss, and distributed RLED framework.Source: §5 · p. 8
  • The paper reports human-level performance across a wide variety of MDPs on the Atari 2600 benchmark.Source: §5 · p. 8

A Transformed Bellman Operator in Stochastic MDPs

The appendix gives contraction conditions for the transformed Bellman operator in stochastic MDPs. It also notes that the discount factor used in practice exceeds the stated bound, leaving those practical contraction properties for future investigation.

B Learner algorithm

The learner updates its action-value estimate using sampled transitions and Adam steps, while maintaining target-network updates and replay priorities.

C Experimental setup

The experiments use the Arcade Learning Environment and differ from common Atari defaults in episode termination and reward preprocessing. They also use a higher discount factor and expert demonstrations.

  • The evaluation uses the Arcade Learning Environment and differs in several key respects from common Atari training defaults.Source: C Experimental setup · p. 13
  • Training episodes end at game over or after 50,000 steps, rather than at life loss.The setup follows Hester et al. [7] for termination and uses the episode length reported by Horgan et al. [8].Source: C Experimental setup · p. 13
  • The experiments use raw game rewards without clipping them to [−1, 1].Source: C Experimental setup · p. 13
  • The temporal consistency loss supports γ = 0.999 and an effective planning horizon of 1000 steps, compared with 100 steps at γ = 0.99.Source: C Experimental setup · p. 13
  • Expert demonstrations are included in the TD loss so the algorithm receives rewarding transitions without discovering them through ε-greedy exploration.Source: C Experimental setup · p. 13

D Full experimental results

The full-results materials include training curves for all 42 games and score tables for no-op-start and human-start evaluations. The tables report scores from best checkpoints evaluated over 200 episodes.

D Full experimental results

  • Figure 5 presents training curves across all 42 games for the standard and slightly deeper network architectures.The standard architecture is also used in the baselines.Source: Figure 5 · p. 14
  • The score tables report best-checkpoint evaluations over 200 episodes for both no-op starts and human starts.Table 3 covers no-op starts; Table 4 covers human starts.Source: Table 3 · p. 15, Table 4 · p. 16
  • Figure 6 reports human-relative scores for the deeper Ape-X DQfD under no-op starts.The score is computed relative to average human and random scores.Source: Figure 6 · p. 15
  • Figure 7 reports human-relative scores for the deeper Ape-X DQfD under human starts.The score is computed relative to average human and random scores.Source: Figure 7 · p. 16

E Experimental setup & hyper parameters

The experimental materials report the expert player's performance and the demonstrations available per game, alongside the network architectures and hyperparameters used.

E Experimental setup & hyper parameters

  • The expert-player performance and available demonstrations are reported per game, with a note that the total number of demonstration episodes or trajectories is very low.The per-game information is presented in Table 5.Source: Table 5 · p. 17
  • The study uses the standard dueling architecture and a slightly wider and deeper version.Figure 8 depicts both architectures; the standard version is the dueling architecture of Wang et al. [22].Source: Figure 8 · p. 18
  • Table 6 lists the study's hyperparameters.Source: Table 6 · p. 18

F Videos

The paper provides videos comparing an agent episode with an expert episode.

F Videos

  • Videos are provided comparing an agent episode with an expert episode.Source: F Videos · p. 19
  • The comparison pairs an agent's episode with an expert's episode.Source: F Videos · p. 19
  • The paper points readers to YouTube for the episode comparisons.Source: F Videos · p. 19

Related papers

Loading 1805.11593v1…