PaperlayerView exact revision on arXiv

Source-linked AI summary

State Entropy Maximization with Random Encoders for Efficient Exploration

Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee

arXiv:2102.09430v4cs.LG

TL;DR

Efficient exploration remains difficult in high-dimensional RL observations, and prior novelty rewards do not provide a principled measure of effective exploration. RE3 estimates state entropy with k-nearest neighbors in a fixed random convolutional representation, improving sample-efficiency across model-free and model-based benchmarks while supporting diverse reward-free behavior.

  • Problem

    High-dimensional observations make efficient exploration challenging, while novelty-based intrinsic rewards leave effective exploration without a principled quantitative measure.

  • Method

    RE3 maximizes a k-nearest neighbor state-entropy estimate in the low-dimensional representation space of a randomly initialized encoder fixed throughout training.

  • Results

    RE3 significantly improves sample-efficiency for model-free and model-based RL across DeepMind Control Suite, MiniGrid, and Atari benchmarks.

  • Takeaways & Limitations

    RE3 provides a compute-efficient exploration approach without representation learning or additional exploration models, while supporting diverse reward-free behaviors and downstream policy improvement.

Abstract

from arXiv · show

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks. Source code and videos are available at https://sites.google.com/view/re3-rl.

1. Introduction

Exploration methods encourage diverse state visitation, but principled quantification of effective exploration remains unresolved in high-dimensional environments. RE3 addresses this by maximizing state entropy in a fixed random-encoder representation and reports improved efficiency across RL settings.

  • High-dimensional observations remain a major challenge for efficient exploration in deep reinforcement learning.
  • Novelty-based intrinsic rewards encourage diverse state visitation but leave effective exploration without a principled quantitative definition.
  • RE3 estimates state entropy with a k-nearest neighbor estimator in the representation space of a randomly initialized encoder fixed throughout training.
  • RE3 significantly improves sample-efficiency for model-free and model-based RL on DeepMind Control Suite, MiniGrid, and Atari benchmarks.
  • RE3 avoids representation learning and additional exploration models while outperforming schemes that use them.
  • RE3 is compute-efficient because it requires neither gradient computations nor updates for additional representation learning.
  • RE3 supports diverse behavior learning without extrinsic rewards and improves downstream sample-efficiency through policy fine-tuning.

2. Related Work

Prior exploration methods use action or parameter noise, action-entropy maximization, diverse goals, or novelty-based intrinsic rewards. RE3 instead applies k-nearest neighbor state entropy estimation in a fixed random-encoder representation, illustrated through state-neighborhood and reward-computation diagrams.

  • Exploration algorithms diversify behavior through action or parameter noise, action-space entropy maximization, diverse goals, and prediction-error rewards.
  • State entropy methods estimate or maximize state-distribution entropy using density approximations or k-nearest neighbor estimators.
  • Figure 1 compares k-nearest neighbors in random-encoder representation space with neighbors in ground-truth state space on Hopper.
  • The random encoder’s representation captures state-similarity information without representation learning.
  • Unlike contrastive-representation approaches, RE3 uses a fixed random encoder to obtain a stable entropy estimate without representation learning.
  • Figure 2 shows intrinsic rewards computed from distances to k-nearest neighbors in fixed random-encoder representations and combined with extrinsic rewards when available.
  • A separate RL encoder is used by the policy that maximizes expected reward.

3. Method

RE3 estimates state entropy in a low-dimensional representation space produced by a fixed random encoder, using k-nearest-neighbor distances as intrinsic rewards. Storing representations in the replay buffer supports stable, precise, and compute-efficient entropy estimation for online exploration and unsupervised pre-training.

  • Preliminaries: High-dimensional control tasks are modeled as POMDPs and reformulated as MDPs by stacking consecutive observations into states.The RL policy maximizes expected return from accumulated rewards.
  • State entropy estimation: The k-nearest-neighbor entropy estimator approximates differential entropy from distances among sampled points when the underlying density is unavailable.The estimator is motivated by the difficulty of estimating high-dimensional probability densities directly.
  • Random Encoders for Efficient Exploration: RE3 computes state distances in the low-dimensional representation space of a randomly initialized encoder whose parameters remain fixed throughout training.The method uses the encoder representation yi = fθ(si) and the distance to its k-nearest neighbor to form an entropy-based intrinsic reward.
  • Random Encoders for Efficient Exploration: Fixed representations make pairwise distances stable during training, while replay-buffer storage avoids repeatedly encoding high-dimensional states at every RL update.The stored representations also allow distances to be computed against all replay-buffer entries, supporting precise and compute-efficient entropy estimation.
  • The RE3 objective: RE3 combines intrinsic rewards with extrinsic rewards for online RL and can set extrinsic rewards to zero during unsupervised pre-training before downstream fine-tuning.The exploration–exploitation tradeoff is controlled by a nonnegative βt that decays exponentially during training.

4. Experiments

The experiments test RE3 across continuous and discrete control, comparing its sample-efficiency, representation choices, computational cost, pre-training benefits, and robustness. Across these settings, RE3 improves exploration efficiency for model-free and model-based methods while supporting diverse behavior and practical computation.

  • DeepMind Control Suite: RE3 consistently improves RAD and Dreamer sample-efficiency across DeepMind Control Suite locomotion tasks.RAD + RE3 reaches an average episode return of 601.6 on Cheetah Run Sparse, where RAD and DrQ fail to solve the task.
  • Representation comparisons: Fixed random representations outperform continually learned or ImageNet-pre-trained representations for state entropy maximization.The results support stable entropy estimation without representation learning or data-trained encoders.
  • Exploration baselines: RAD + RE3 generally outperforms RAD combined with RND or ICM, which add trainable exploration models and can suffer instability.The comparison attributes RE3’s advantage to stable state entropy estimation without additional exploration models.
  • Compute-efficiency: RE3 improves sample-efficiency without sacrificing compute-efficiency by avoiding gradient updates and repeated representation forward passes.RAD + SE w/ Random uses 7.233e+16 FLOPs at 500K Hopper Hop steps, while RAD + SE w/ Inverse dynamics requires roughly twice as many.
  • Unsupervised pre-training: RE3 pre-training produces more diverse behaviors and improves downstream sample-efficiency after fine-tuning.The reported pre-training duration is 500K environment steps, compared with 5M steps reported for prior work.
  • Discrete control: RE3 also improves discrete-control exploration, including A2C on MiniGrid, and is evaluated with Rainbow across six Atari games.In DoorKey-8x8, A2C + RE3 reaches 0.49 average episode return at 2.4M steps, versus 0.20 for A2C + ICM.

5. Discussion

RE3 is presented as a simple exploration method that combines k-nearest-neighbor state entropy with a fixed random encoder and supports both model-free and model-based RL. The discussion emphasizes stable, compute-efficient estimation while identifying pre-trained representations and architectural choices as future directions.

  • Applicability: RE3 is compatible with both model-free and model-based reinforcement-learning algorithms.
  • Method: RE3 maximizes a k-nearest-neighbor state entropy estimate in a fixed representation space produced by a randomly initialized encoder.The encoder remains fixed throughout training and is intended to capture state similarity without representation learning.
  • Scope: RE3 encourages exploration through stable and compute-efficient state entropy estimation without claiming that learned representations or additional models are unnecessary.
  • Future directions: The study leaves pre-trained fixed representations for visually complex domains and architecture-specific entropy estimation as future research directions.The paper notes that pre-training would require access to environments and broad state distributions.

A.2. Implementation Details for Model-free RL

The model-free implementation combines RE3 with established RL implementations, normalized nearest-neighbor intrinsic rewards, specified encoder architectures, and comparisons against several exploration baselines. Additional procedures cover unsupervised pre-training, learned representation baselines, and reproducibility details.

  • Base algorithms: Model-free experiments use publicly released RAD and DrQ implementations with random crop and random shift augmentation, respectively.Full hyperparameters are provided in Table 1.
  • RE3: RE3 uses a normalized nearest-neighbor intrinsic reward with k = 3 and searches β0 ∈ {0.05, 0.25} and ρ ∈ {0.00001, 0.000025}.The logarithm is omitted for simplicity in DeepMind Control Suite experiments, with similar results reported when it is retained.
  • Architecture: The model-free encoder has four convolutional layers followed by ReLU activations, a normalized fully connected layer, and a 50-dimensional tanh output.All convolutional layers use 3 × 3 kernels and 32 channels; only the first layer uses stride 2.
  • Pre-training: Unsupervised pre-training maximizes intrinsic reward for 500K environment steps before downstream fine-tuning for another 500K steps.Parameters are initialized from the pre-trained policy during downstream learning.
  • Reproducibility: The authors note that differences from original reported results can arise from different random seeds and provide source code and scripts for reproducibility.
  • Baselines: Baselines include RND, ICM, contrastive learning, inverse dynamics prediction, and pre-trained ImageNet or ATC encoders.RND and ICM use additional predictor models, while representation-learning baselines train separate encoders for entropy estimation.

A.3. Implementation Details for Model-based RL.

The model-based implementation combines RE3 with Dreamer using normalized nearest-neighbor intrinsic rewards, a Dreamer-compatible convolutional encoder, and implementation-specific hyperparameters. The authors also report discrepancies from the original Dreamer results associated with implementation differences or random seeds.

  • Intrinsic reward: Dreamer + RE3 uses normalized nearest-neighbor intrinsic rewards, omits the logarithm for simplicity, and uses a large value of the intrinsic-reward normalization parameter for trajectory-segment training.
  • Architecture: The Dreamer-compatible encoder has four convolutional layers with channels {32, 64, 128, 256}, followed by a normalized fully connected layer and a 50-dimensional tanh output.All convolutional layers use 4 × 4 kernels and stride 2.
  • Reproducibility: The experiments use the newer Dreamer implementation, and the authors attribute differences from original results to implementation differences or random seeds.Full source code and reproduction scripts are provided.
  • Hyperparameters: Tables 2 and 1 specify hyperparameters for Dreamer + RE3 and RAD + RE3 experiments, respectively.
  • Numerical stability: The authors found that 32-bit precision is necessary to avoid NaNs during intrinsic-reward normalization because the running standard deviation is very small.

B. Additional Experimental Results on DeepMind Control Suite

Additional DeepMind Control Suite experiments show that RE3 improves sample efficiency on several tasks while maintaining performance on dense-reward tasks. Figure 14 reports locomotion performance as means and standard deviations across five runs.

  • Results: RE3 improves sample efficiency on tasks including Reacher Hard and Hopper Stand.
  • Results: RE3 does not degrade performance on dense-reward tasks such as Cartpole Balance.
  • Evaluation: Figure 14 summarizes locomotion-task performance using the mean and standard deviation across five runs.

C. Details on MiniGrid Experiments

The MiniGrid experiments use a fixed convolutional encoder for state-entropy rewards, with implementation choices focused on stability, noise reduction, and environment-specific tuning.

  • The intrinsic reward is log(||y_i − y_k-NN i||2 + 1), where the added 1 provides numerical stability.
  • Averaging distances to the k nearest neighbors rather than using only the kth neighbor reduces noise and improves MiniGrid performance.
  • RE3 hyperparameters are selected by searching β over nine values from 0.00001 to 0.1, with the best result reported.
  • The MiniGrid encoder uses three convolutional layers with 16, 32, and 64 filters, ReLU activations, and max pooling after the first layer.
  • Table 3 reports MiniGrid hyperparameters, which are mostly unchanged across environments except for evaluation frequency and intrinsic-reward weight β.

D. Details on Atari Experiments

The Atari experiments use Rainbow with RE3-based state-entropy rewards and retain the released implementation’s architecture while tuning the intrinsic-reward weight.

  • The Atari implementation uses the publicly available Rainbow repository and its default hyperparameters.
  • The intrinsic reward is log(||y_i − y_k-NN i||2 + 1), with the added 1 used for numerical stability.
  • Rainbow + SE with RE3 searches β over {0.0001, 0.001, 0.01} and reports the best result for each environment.
  • Rainbow + SE with Contrastive searches β over {0.0001, 0.001} and reports the best result for each environment.
  • The Atari encoder retains the released architecture and adds a linear layer to reduce latent-representation dimensionality.
  • Figure 15 reports Atari performance using means and standard deviations across three runs.

F. Calculation of Floating Point Operations

The FLOP analysis compares the computational cost of exploration variants under explicit counting assumptions and shows that the random-encoder variant adds substantially less neural-network computation.

  • The analysis uses forward-pass operation counts for encoder layers E and MLP layers M, with backward passes counted as twice the forward-pass cost.
  • The FLOP comparison assumes one update per iteration, identical entropy and RL encoders, and negligible non-network computation such as k-NN search.
  • RAD requires bF(E + M) + 2bB(E + M) + (E + M) FLOPs per iteration.
  • RAD + SE with Random adds only E extra FLOPs per iteration, compared with 4bE for Contrastive and more than 6bE for Inverse dynamics.
  • 1.569e+15 FLOPs are required for the stated distance computations over 250000 training steps with m = 512, |B| = 100000, and d = 50.

G. Comparison to State Entropy Maximization with Contrastive Encoder for MiniGrid Pre-training

The MiniGrid comparison evaluates RE3 against contrastive state-entropy pre-training and attributes the contrastive method’s weaker performance to augmentation incompatibility with compact embeddings.

  • The comparison omits intrinsic rewards during contrastive fine-tuning, following Liu and Abbeel, although including them produced very similar performance.
  • The contrastive encoder does not work well for state-entropy estimation because its image-specific augmentations are incompatible with MiniGrid’s compact embeddings.
  • RE3 avoids the need for carefully chosen data augmentations by using a random encoder.
  • RE3 outperforms contrastive state-entropy pre-training on MiniGrid navigation tasks when the contrastive method uses random-shift augmentation.
  • Figure 16 reports navigation performance using means and standard deviations across five runs.
Loading 2102.09430v4…