PaperlayerView exact revision on arXiv

Source-linked AI summary

Count-Based Exploration with Neural Density Models

Georg Ostrovski, Marc G. Bellemare, Aaron van den Oord, Remi Munos

arXiv:1703.01310v2cs.AI

TL;DR

The paper asks how density-model quality and the mixed Monte Carlo update affect pseudo-count exploration in non-tabular reinforcement learning. It uses PixelCNN-based bonuses with multiple agent architectures and finds broad Atari improvements, with Monte Carlo returns especially important in sparse-reward settings.

  • Problem

    The paper examines how density-model quality and the mixed Monte Carlo update affect exploration, extending practical count-based exploration beyond tabular settings.

  • Method

    The authors use PixelCNN to generate pseudo-count exploration bonuses and combine them with DQN and Reactor agents.

  • Results

    PixelCNN improves exploration across hard Atari games, while the mixed Monte Carlo return is especially effective in sparse-reward settings.

  • Takeaways & Limitations

    PixelCNN pseudo-counts provide a simpler, more general exploration setup compatible with value-function and policy-based reinforcement-learning algorithms.

  • Takeaways & Limitations

    Reactor may not fully exploit the exploration bonus in very sparse settings because cautious Retrace(λ) learning can discard off-policy trajectories.

Abstract

from arXiv · show

Bellemare et al. (2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning. This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge. We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count. In particular, we examine the intrinsic difficulties in adapting Bellemare et al.'s approach when assumptions about the model are violated. The result is a more practical and general algorithm requiring no special apparatus. We combine PixelCNN pseudo-counts with different agent architectures to dramatically improve the state of the art on several hard Atari games. One surprising finding is that the mixed Monte Carlo update is a powerful facilitator of exploration in the sparsest of settings, including Montezuma's Revenge.

1. Introduction

The paper investigates whether better density models improve exploration and whether prior modeling assumptions and the mixed Monte Carlo update are essential. It adapts pseudo-count exploration with PixelCNN while addressing practical challenges in model choice, training, and use.

  • Practical count-based exploration remains scarce for large state spaces because fully Bayesian methods are often intractable and tabular counts do not apply with value-function approximation.
  • Bellemare et al. defined pseudo-counts from a density model trained on experienced states and used them to improve Atari exploration, including substantial progress on Montezuma’s Revenge.
  • Their approach assumed a learning-positive density model, one online update per sample, and an effective step-size decaying as n^-1, while also using a mixed Monte Carlo/Q-Learning update.
  • The paper asks whether better density models improve exploration, whether the modeling assumptions can be relaxed, and what role the mixed Monte Carlo update plays.
  • PixelCNN introduces practical challenges because computing prediction gain requires two evaluations and one model update per agent step, while online neural training risks overfitting and catastrophic forgetting.
  • Neural optimization also constrains the step-size regime, potentially conflicting with the pseudo-count theory’s requirement that the density model’s learning rate decay over time.

2. Background

Pseudo-count exploration converts density-model learning progress into an intrinsic reward bonus, while image density models and multi-step updates provide the relevant modeling and learning context.

  • 2.1. Pseudo-Count and Prediction Gain: Prediction gain compares the model’s probability for a state before and after training on that same state, providing an approximation to the pseudo-count.
  • 2.1. Pseudo-Count and Prediction Gain: A pseudo-count generalizes visitation counts by using a density model’s recoding probability, and under assumptions it grows approximately linearly with real counts.
  • 2.1. Pseudo-Count and Prediction Gain: The pseudo-count defines an exploration bonus added to extrinsic reward, incentivizing the agent to re-experience surprising situations.
  • 2.2. Density Models for Images: CTS models Atari frames using location-dependent L-shaped filters and was applied to 3-bit greyscale, 42 × 42 downsampled images.
  • 2.2. Density Models for Images: PixelCNN is a fully convolutional neural density model that predicts pixel probabilities conditional on preceding pixels using masked convolutional filters.
  • 2.3. Multi-Step RL Methods: Multi-step reinforcement-learning methods interpolate between one-step Q-Learning and Monte Carlo updates by incorporating information from future transitions.
  • 2.3. Multi-Step RL Methods: The mixed Monte Carlo update combines TD and Monte Carlo errors, whereas Retrace(λ) uses truncated importance-sampling ratios to mix TD errors from future steps.

3. Using PixelCNN for Exploration

The paper adapts PixelCNN pseudo-counts to practical Atari exploration by reconciling theoretical density-model requirements with neural-network training constraints. Online training, a slim architecture, and decayed prediction-gain scaling produce robust exploration behavior while preserving training stability.

  • Design requirements: PixelCNN pseudo-count exploration requires online, learning-positive density estimation with approximately n^-1 step-size decay and lightweight computation at every agent step.These requirements conflict with practical neural-model preferences for randomized mini-batches and fixed optimization regimes.
  • Design requirements: A slim PixelCNN with 2 gated residual blocks and 16 feature maps processes 42 × 42, 3-bit greyscale frames for DQN-comparable computation.Vanilla PixelCNN uses 15 residual blocks with 128 feature maps, so the proposed architecture is substantially smaller.
  • Training the density model: Online PixelCNN training remains as robust on temporally correlated state sequences as on randomized sequences after minor optimization-hyperparameter tuning.This avoids mini-batch updates, which can create incompatible optimizer-state statistics between training streams and cause slower or unstable learning.
  • Training the density model: A constant learning rate of 0.001 achieved the lowest final training loss and was selected for greater robustness to the initial learning rate.A decaying schedule of 0.1 · n^-1/2 achieved the same lowest loss in the reported comparison.
  • Training the density model: PixelCNN rapidly learns state-space regularities, with loss spikes becoming associated with visually meaningful events such as episode starts.On PONG, spikes coincide with the distinctive green frame sometimes appearing at episode start.
  • Computing the pseudo-count: Scaling prediction gain by c · n^-1/2 preserves the required decay while retaining the constant learning rate, and c = 0.1 works across a broad range of Atari games.The model is robust across 1–2 orders of magnitude in c, and this configuration gives the best exploration results on hard games such as MONTEZUMA’S REVENGE.
  • Computing the pseudo-count: Because neural optimization can produce negative prediction gain by overshooting a local loss minimum, the method thresholds prediction gain at zero.This workaround addresses the difficulty of guaranteeing learning-positiveness in a deep neural density model.

4. Exploration in Atari 2600 Games

PixelCNN pseudo-count bonuses improve exploration across DQN and Reactor agents, especially on hard Atari games, while their effectiveness depends on the agent architecture and learning dynamics.

  • The PixelCNN pseudo-count augments extrinsic reward with an inverse-square-root exploration bonus, clipped so total reward remains within [−1, 1].
  • DQN-PixelCNN uses one-frame online PixelCNN updates that simultaneously produce the prediction gain used for exploration.
  • On MONTEZUMA’S REVENGE, both intrinsic-reward agents vastly outperform DQN; DQN-PixelCNN also substantially outperforms DQN and DQN-CTS on other hard exploration games.
  • Across 57 Atari games, DQN-PixelCNN outperforms DQN-CTS in 52 games by maximum score and 51 by AUC.
  • The PixelCNN bonus improves Reactor on a broad range of games, including better AUC-based sample efficiency than vanilla Reactor.
  • On sparse-reward hard exploration games, Reactor appears unable to fully exploit the bonus, whereas its Retrace(λ) learning may discard off-policy trajectories.

5. Quality of the Density Model

PixelCNN differs from CTS not only in expressiveness but also in the prediction-gain signals it produces, yielding smoother bonuses with stronger responses to rare events.

  • PixelCNN’s generative samples are somewhat higher quality than CTS samples, but generative quality does not necessarily imply better probability estimates.
  • PixelCNN produces consistently lower, smoother prediction gains than CTS, with more pronounced peaks at infrequent events.
  • These prediction-gain characteristics make the bonus less harmful in easy exploration games while preserving a strong signal for novel or rare events.
  • Because PixelCNN uses a non-decaying step-size, it remains mildly surprised by significant state changes and supplies persistent milestone rewards.
  • PixelCNN’s convolutional structure improves sample efficiency relative to vanilla CTS, although it explains only part of the overall advantage.

6. Importance of the Monte Carlo Return

The mixed Monte Carlo update is especially important when exploration rewards are transient and extrinsic rewards are sparse, although it can also harm learning in some settings.

  • The mixed Monte Carlo update helps intrinsically motivated agents rapidly propagate transient exploration rewards and bridge long horizons with rare rewards.
  • The Monte Carlo return increases variance and can prevent convergence to the optimal policy during off-policy training.
  • MMC can speed training and improve final performance, or hurt performance, with MMC alone explaining much of DQN-PixelCNN’s difference from DQN in both cases.
  • MMC and the PixelCNN reward bonus can have a compounding effect, as illustrated by H.E.R.O.
  • On the hardest sparse-reward games, neither DQN, MMC alone, nor the exploration bonus alone makes significant progress; their combination produces rapid training progress and high performance.
  • The exploration bonus densifies the reward landscape, while MMC lets the agent learn from its transient signal faster than one-step methods.

7. Pushing the Limits of Intrinsic Motivation

Increasing the prediction-gain scale produces stronger, faster exploration, but excessive scaling can harm stability and long-term performance. Remarkably, curiosity alone can train agents that achieve substantial environment scores.

  • A 12-order-of-magnitude increase in prediction-gain scale leaves performance fairly robust while trading off exploration against exploitation.Increasing the scale primarily changes the balance between seeking maximal reward and optimizing the current policy.
  • 900 in GRAVITAR, 6,600 in MONTEZUMA’S REVENGE, 39,000 in PRIVATE EYE, and 1,500 in VENTURE surpass the state of the art by substantial margins.Higher prediction-gain scales strengthen exploration and can produce record peak scores.
  • Aggressive scaling accelerates exploration and peak performance but can deteriorate stability and long-term performance.With non-decaying prediction gain and reward clipping, an overly large bonus can become effectively constant and lose its usefulness.
  • Curiosity-only training produces a high-performing agent, measured by the environment reward.The environment reward is ignored during training, while the exploration reward alone drives learning.
  • Exploration-bonus agents continue exploring, with different seeds making learning progress at very different times than vanilla DQN.

8. Conclusion

PixelCNN provides a more useful exploration bonus than previous density models in a simpler, online setup compatible with multiple RL architectures. It improves Atari exploration, learning speed, and stability, while Monte Carlo returns are important for sparse exploration.

  • PixelCNN’s greater accuracy and expressiveness produce a more useful exploration bonus than previous density models.
  • PixelCNN can be trained completely online in a simpler, more general setup and works with both value-function and policy-based RL algorithms.
  • PixelCNN pushes state of the art on hard Atari exploration problems while improving learning speed and baseline-agent stability across many games.
  • On sparse-reward games, the PixelCNN reward alone can produce significant scores, yielding an intrinsically motivated agent.
  • Monte Carlo returns support effective exploration by speeding learning with transient rewards and bridging gaps when extrinsic and intrinsic rewards are extremely sparse.
  • The implementation uses a lightweight PixelCNN with 7 × 7 and 1 × 1 masked convolutions, residual blocks, and 42 × 42 greyscale inputs quantized to 8 bins.
  • The model is trained online from Atari frames using RMSProp with momentum 0.9, decay 0.95, and epsilon 10^-4.

B. Methodology

Performance is evaluated using undiscounted per-episode training return averaged over 1M environment frames. Algorithm comparisons use area under the learning curve to capture speed and stability as well as performance.

  • Performance graphs report undiscounted per-episode return averaged over 1M environment frames per data point.
  • Algorithm-comparison graphs use area under the curve, which captures learning speed and stability that maximum score can underestimate.

C. Convolutional CTS

The convolutional CTS experiment separates PixelCNN’s architectural effects from its density-model quality. Convolution helps make CTS softer, but PixelCNN’s accuracy remains necessary for its large exploration gains on sparse-reward games.

  • DQN-PixelCNN outperforms DQN-CTS in most of 57 Atari games, motivating tests of whether expressiveness or convolution causes the improvement.
  • The comparison evaluates DQN, DQN-MC, DQN-CTS, DQN-ConvCTS, and DQN-PixelCNN on six example games.
  • On dense-reward Q*BERT and ZAXXON, MMC drives most improvement while exploration bonuses hurt performance; convolutional CTS resembles PixelCNN more than regular CTS.
  • Convolution contributes to PixelCNN’s softer behavior, but its accuracy advantage more clearly distinguishes novel from explored states and enables more effective exploration.

D. The Hardest Exploration Games

This section compares exploration agents on hard Atari games, including sparse-reward settings, across individual games and the full 57-game suite. It also examines agent architectures and the role of the mixed Monte Carlo update.

  • Game taxonomy: The Atari taxonomy distinguishes Human-Optimal, Score Exploit, Sparse, and Dense reward or exploration categories.“Human-Optimal” denotes human-level-or-higher performance by DQN-like agents, while “Score Exploit” denotes superhuman scores that need not reflect human play.
  • Comparative results: Table 2 compares DQN-CTS and DQN-PixelCNN with previously published agents on seven hard-exploration, sparse-reward Atari games.The reported scores for the authors’ agents are maximum scores over 150M training frames, averaged over three seeds.
  • Agent comparisons: The experiments compare DQN, DQN-CTS, DQN-ConvCTS, and DQN-PixelCNN training performance.
  • Agent comparisons: Training curves compare DQN, DQN-CTS, and DQN-PixelCNN across all 57 Atari games.
  • Agent comparisons: The study also compares DQN, DQN-PixelCNN, Reactor, and Reactor-PixelCNN across all 57 Atari games.
  • Monte Carlo update: Training curves compare DQN and DQN-PixelCNN with and without the mixed Monte Carlo update across all 57 Atari games.
Loading 1703.01310v2…