PaperlayerView exact revision on arXiv

Source-linked AI summary

EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Justin Fu, John D. Co-Reyes, Sergey Levine

arXiv:1703.01260v2cs.LG

TL;DR

Sparse rewards make exploration difficult, while novelty methods based on generative models can struggle with high-dimensional observations. EX2 instead trains discriminative exemplar models, connects them to implicit density estimation, and combines them with count-based exploration. It achieves competitive benchmark results, including strong performance on complex egocentric vizDoom observations.

  • Problem

    Sparse-reward exploration is difficult, and novelty methods often require generative or predictive observation models that are hard to train for complex high-dimensional inputs.

  • Method

    EX2 trains exemplar classifiers to distinguish each visited state from other states, uses their outputs for implicit density estimation, and converts novelty into an exploration bonus.

  • Results

    EX2 achieves competitive results across benchmark tasks, including good performance on complex egocentric vizDoom observations where explicit generative density estimators struggle.

  • Takeaways & Limitations

    Discriminative exemplar models provide a scalable exploration strategy that can handle rich sensory inputs without generative modeling of complex images.

  • Takeaways & Limitations

    For high-dimensional states, direct input noise may leave the data manifold, requiring learned latent-space smoothing; exemplar models can also overfit or underfit.

Abstract

from arXiv · show

Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with count-based exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.

1 Introduction

Deep reinforcement learning can represent complex policies, but sparse rewards still make meaningful exploration difficult. EX2 addresses this challenge by estimating novelty with discriminative exemplar models rather than explicit generative models.

  • Deep reinforcement learning handles complex tasks with rich policy representations, but sparse rewards can prevent meaningful policy learning.The passage cites Atari games and simulated-robot control as examples of complex tasks.
  • Generative novelty methods have succeeded on simple synthetic images but struggle with complex naturalistic observations such as egocentric vizDoom images.
  • EX2 estimates novelty by training classifiers to distinguish each visited state from other previously observed states.Its intuition is that states that are easier to distinguish are more likely to be novel.

2 Related Work

Prior exploration methods address high-dimensional or sparse-reward environments through visitation approximations, randomized value functions, or hierarchical actions. EX2 instead uses discriminative novelty estimation with an implicit density-modeling connection.

  • Classical visitation-count methods provide theoretical guarantees in finite MDPs but are difficult to extend to high-dimensional or continuous states.
  • Randomized value functions scale to complex state spaces but do not provide explicit novelty-seeking behavior.
  • Hierarchical reinforcement learning facilitates exploration by using temporally extended actions or skills.
  • Concurrent discriminative exploration work predicts next-state features from actions, whereas EX2 additionally models novelty through implicit density estimation.

3 Preliminaries

Reinforcement learning seeks a policy maximizing expected discounted rewards through interaction with an unknown MDP. Count-based exploration adds bonuses for rarely visited states, while pseudo-counts approximate visitation using density estimates.

  • An MDP is defined by state and action spaces, unknown transitions and rewards, and an initial-state distribution accessible through interaction.
  • The reinforcement-learning objective is to find the policy maximizing expected discounted cumulative reward.
  • Count-based exploration encourages rarely visited states by adding reward bonuses based on state-action visitation counts.
  • Pseudo-count methods replace impractical explicit counts with density estimates, typically learned using generative models.

4 Exemplar Models and Density Estimation

EX2 uses exemplar classifiers to distinguish individual states from background data and recover implicit density estimates. Noise injection, including learned latent-space smoothing, makes these estimates useful in continuous and high-dimensional observation spaces.

  • 4.1 Exemplar Models: An exemplar model trains one discriminator per data point to distinguish that positive exemplar from the remaining dataset.
  • 4.1 Exemplar Models: EX2 applies exemplar models to reinforcement-learning exploration and amortizes their computation with a single exemplar-conditioned network.
  • 4.1 Exemplar Models: Each discriminator is trained on balanced exemplar and background examples to estimate the probability that an input is the exemplar.
  • 4.2 Exemplar Models as Implicit Density Estimation: At an optimal discriminator, evaluating the discriminator on its own exemplar recovers the exemplar’s underlying data density.
  • 4.2 Exemplar Models as Implicit Density Estimation: Adding noise to exemplars produces smoothed density estimates equivalent to kernel density estimation, including an RBF kernel for Gaussian noise.
  • 4.3 Latent Space Smoothing with Noisy Discriminators: For high-dimensional images, direct state noise can leave the data manifold, so EX2 injects noise into a learned latent space instead.
  • 4.3 Latent Space Smoothing with Noisy Discriminators: The latent-space objective balances classification accuracy against information transmission so noise captures informative variation while remaining on the state manifold.
  • 4.3 Latent Space Smoothing with Noisy Discriminators: Limited discriminator capacity provides an additional source of density smoothing by averaging outputs for states that are difficult to distinguish.

5 EX2: Exploration with Exemplar Models

EX2 converts discriminative novelty estimates into intrinsic rewards by training classifiers on visited states and replay-buffer negatives, then optimizing the policy with the augmented reward.

  • EX2 trains a discriminator for each current trajectory state using replay-buffer states as negative examples.The algorithm samples trajectories, trains each D_s, and scores the current state for novelty.
  • The augmented reward combines the environment reward with a novelty bonus scaled by β.The paper specifies R′(s, a) = R(s, a) + βf(D_s(s)).
  • Policy optimization improves π_i against the augmented reward, after which the new trajectories are added to the replay buffer.This creates the next iteration’s reference distribution for negative sampling.
  • EX2 uses −log p(s) as a heuristic bonus and can alternatively use the count-based bonus 1/N(s) in discrete domains.Here N(s) is approximated from the replay-buffer distribution, with buffer size n.

6 Model Architecture

The architecture reduces the cost of per-state exemplar discrimination through amortization and K-exemplar batching, while retaining a continuum between state-specific and shared discriminators.

  • Architecture: Neural-network exemplar models use convolutional architectures for image observations and amortize computation across exemplars.The motivation is to reduce the cost of training many large per-exemplar classifiers.
  • Amortized model: The amortized model conditions one classifier on the exemplar x∗ instead of training a separate classifier for every exemplar.Its latent discriminator is conditioned on an encoded exemplar representation, enabling generalization to new states.
  • K-Exemplar model: The K-Exemplar model samples uniformly from a batch of K exemplars and interpolates between one discriminator per state and one discriminator for all states.K = 1 gives the more powerful state-specific model, whereas K = # states gives the weaker shared model.
  • Relationship to GANs: The K-exemplar setup resembles a GAN because one discriminator can classify current-batch states against previous states, but the interaction is cooperative rather than adversarial.The policy is rewarded for making states easier for the discriminator to identify, rather than for fooling it.
  • Architecture: Figure 1 depicts amortized and K-exemplar architectures, with noise injected after the encoder or shared layers.The experiments do not tie the encoders in the amortized architecture.

7 Experimental Evaluation

EX2 is evaluated on low-dimensional control and high-dimensional visual tasks against naïve, explicit-density, and hashing-based exploration methods. It produces sensible density estimates, competitive control results, and especially strong performance on DoomMyWayHome+.

  • Baselines: EX2 is compared with random exploration, KDE, VIME, hashing, and baseline TRPO across the benchmark suite.The two EX2 variants are K-exemplar and amortized.
  • Tasks: The evaluation spans three low-dimensional tasks for implicit density estimation and four image-based tasks where generative modeling is difficult.The low-dimensional set includes a sparse-reward 2D maze, SwimmerGather, and SparseHalfCheetah; the visual set includes Atari and vizDoom.
  • Low-dimensional tasks: On the 2D maze, EX2 produces reasonable, somewhat smoothed density estimates relative to the empirical visitation distribution.Both EX2 variants perform similarly to VIME and KDE, while TRPO with Gaussian exploration cannot find the sparse-reward goal.
  • Low-dimensional tasks: On SwimmerGather, EX2, VIME, and Hashing outperform naïve TRPO and KDE, while amortized EX2 significantly outperforms all other methods on SparseHalfCheetah.These results are reported for medium-dimensional sparse-reward continuous-control tasks.
  • Image-based tasks: EX2 matches the best prior methods on Atari and greatly exceeds prior methods on DoomMyWayHome+, which includes camera motion, partial observability, and extremely sparse rewards.The DoomMyWayHome+ result demonstrates coherent exploration through multiple rooms toward the goal.

8 Conclusion and Future Work

EX2 uses discriminative exemplar models to assign novelty bonuses and connects these models to implicit density estimation. It performs comparably to prior state-of-the-art methods on continuous-control tasks, scales to images, and surpasses prior generative methods on egocentric Doom navigation, while retaining sensitivity to overfitting and underfitting.

  • Conclusion: EX2 assigns novelty bonuses with discriminatively trained exemplar models and connects exemplar modeling to implicit density estimation.The density-estimation connection motivates EX2 as an approximation to pseudo-count exploration.
  • Conclusion: EX2 achieves comparable results to prior state-of-the-art methods on continuous-control tasks in low-dimensional environments.
  • Conclusion: EX2 scales to rich sensory inputs such as images without reconstructing samples for training its density-estimation model.
  • Conclusion: EX2 exceeds prior generative methods on domains with complex observation functions, including the egocentric Doom navigation task.
  • Future Work: Exemplar models can overfit or underfit, but a single smoothing hyperparameter interpolates between these extremes without changing the model.The paper identifies automatic adjustment of this smoothing factor as future work.

A.1 Noisy Discriminators

The noisy discriminator trains on balanced positive exemplar and negative data distributions, with noise enabling density estimation in continuous domains. Its optimal discriminator and encoder are characterized analytically through the training objective.

  • Noisy Discriminators: The noisy discriminator objective is optimized over the discriminator D(z) using the encoder distribution q(z|x).
  • Noisy Discriminators: The noisy discriminator uses balanced positives sampled from exemplar data and negatives sampled from p_X(x), with labels y = 1 and y = 0.
  • Noisy Discriminators: Differentiating the objective with respect to D(z) and setting the result to zero yields the optimal noisy discriminator.
  • Noisy Discriminators: The section also derives an optimal encoder for any discriminator to explain how the objective shapes the encoding distribution.

A.2 K-Exemplar Model

The K-exemplar model associates each discriminator with a batch of K positive exemplars and samples those positives uniformly. This interpolates between single-state and all-state discrimination while preserving a density-estimation interpretation.

  • K-Exemplar Model: The K-exemplar model samples positive examples uniformly from a batch B of K exemplars for each discriminator.
  • K-Exemplar Model: The optimal K-exemplar discriminator follows from differentiating the training objective with respect to D*_B(x).
  • K-Exemplar Model: For a discriminator trained with K uniformly sampled positives, the optimal discriminator evaluated at a positive is D*_B(x) = 1 / (1 + K P_X(x)).
  • K-Exemplar Model: Noisy K-exemplar extensions replace the single-exemplar positive distribution with the uniform batch distribution P_B(x).

A.3 Task Descriptions

The experiments cover sparse-reward navigation, locomotion, collection, Atari, and image-based Doom tasks. They use shared architectures across task families, with convolutional processing for images and task-specific EX2 hyperparameters.

  • Task Descriptions: The task suite includes 2D Maze, SparseHalfCheetah, SwimmerGather, Doom MyWayHome+, Freeway, Frostbite, and Venture.
  • Task Descriptions: 2D Maze provides reward only near the goal, requiring exploration of novel maze regions to find the sparse reward.
  • Task Descriptions: Doom MyWayHome+ uses sparse visual feedback from an RGB 32 x 32 image and places the agent in the room furthest from the vest.
  • Architectures: Non-image tasks use identical fully connected policy and exemplar architectures, while image tasks use convolutional architectures.
  • Architectures: Image-based policy networks use convolutional layers followed by fully connected layers, with Atari additionally using the last 4 grayscale 42 x 42 frames.
  • Hyperparameters: EX2 exploration-bonus weights are tuned to match task reward magnitudes through a rough grid search over candidate hyperparameters.
Loading 1703.01260v2…