PaperlayerView exact revision on arXiv

Source-linked AI summary

RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

Roberta Raileanu, Tim Rocktäschel

arXiv:2002.12292v2cs.LGcs.AI

TL;DR

Sparse-reward exploration is difficult, particularly in procedurally generated environments where agents rarely revisit states. RIDE introduces an intrinsic reward based on impactful changes in a learned state representation and outperforms existing methods, especially on procedurally generated tasks. Its reward remains substantial during training and favors interactions with controllable objects.

  • Problem

    Sparse-reward reinforcement learning is challenging, and existing exploration methods fall short when procedurally generated environments make repeated state visits unlikely.

  • Method

    RIDE rewards actions that produce significant changes in a learned state representation, trained with forward and inverse dynamics models.

  • Results

    RIDE outperforms IMPALA, Count, RND, and ICM particularly in procedurally generated sparse-reward environments and explores them more efficiently.

  • Takeaways & Limitations

    RIDE provides a persistent exploration signal that rewards impactful, controllable interactions rather than diminishing during training.

  • Takeaways & Limitations

    Future work should distinguish desirable from undesirable impact so exploration can avoid large but task-irrelevant environmental changes.

Abstract

from arXiv · show

Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

1 INTRODUCTION

Sparse-reward reinforcement learning requires exploration beyond extrinsic feedback, especially in procedurally generated environments where repeated state visits are unlikely. RIDE addresses this setting by rewarding impactful changes in a learned state representation and outperforms existing exploration methods, particularly in procedurally generated environments.

  • Sparse rewards can make standard reinforcement learning inapplicable because agents may need many episodes before encountering useful feedback.
  • Intrinsic motivation encourages exploration by rewarding novel states or improved knowledge and forward prediction of environment dynamics.
  • Procedurally generated environments preserve the task while changing the environment each episode, making repeated visits to the same state unlikely.
  • RIDE rewards actions that produce impactful changes in the agent’s learned representation of the environment state.
  • RIDE is evaluated against state-of-the-art intrinsic-reward methods on visual singleton environments and challenging procedurally generated grid-world tasks.
  • RIDE outperforms existing exploration methods particularly in procedurally generated environments, while its intrinsic reward does not diminish during training and favors controllable objects.

2 RELATED WORK

Prior exploration methods primarily encourage novelty or learning environment dynamics, but generalization to new environment instances remains limited. RIDE differs from closely related approaches by learning an agent-influence-sensitive state representation rather than relying on fixed representations.

  • Exploration research includes bonuses based on surprise, information gain, curiosity, empowerment, diversity, and feature control.
  • Intrinsic motivation methods broadly use count-based novelty bonuses or curiosity-based rewards tied to learning environment dynamics.
  • Count-based methods estimate state visitation novelty in high-dimensional spaces using density models or related exploration bonuses.
  • Curiosity-based methods reward prediction error or uncertainty about action consequences, often using learned latent state representations.
  • Many successful exploration methods are trained and tested on the same environment and therefore do not generalize to new instances.
  • Compared with related methods using fixed state representations, RIDE learns representations through forward and inverse dynamics constrained toward agent-influenced elements.

3 BACKGROUND: CURIOSITY-DRIVEN EXPLORATION

The paper formalizes episodic reinforcement learning with intrinsic and extrinsic rewards and reviews curiosity-based representation learning. Pathak et al.’s models learn controllable and agent-relevant state features, while curiosity rewards forward-prediction error.

  • The reinforcement-learning setting is a single-agent Markov Decision Process with states, actions, stochastic transitions, policies, and rewards.
  • The total reward combines intrinsic and extrinsic components, with a weighting hyperparameter controlling their relative importance.
  • The agent maximizes reward over a finite episodic horizon using a discount factor and an expectation over policy and environment outcomes.
  • Pathak et al. learn a latent state representation that models elements the agent can control or that can affect the agent.
  • A forward dynamics model predicts the next latent state from the current representation and action, while training minimizes prediction error.
  • An inverse dynamics model predicts the agent’s action from consecutive latent states, and curiosity rewards squared distance between actual and predicted next embeddings.

4 IMPACT-DRIVEN EXPLORATION

RIDE defines intrinsic reward from the magnitude of consecutive changes in a learned state representation, while episodic counts discourage repeatedly exploiting the same transitions. The method is designed to favor impactful, agent-controllable changes for exploration in sparse-reward settings.

  • Impact-driven reward: RIDE rewards actions according to the Euclidean distance between consecutive learned state representations.The representation is learned with forward and inverse dynamics models.
  • Episodic control: RIDE combines impact-driven changes with episodic visitation counts to discourage back-and-forth exploitation of high-distance state sequences.The reward is divided by the number of visits to the next state during the current episode.
  • Optimization: The learned state representation is used to construct the exploration bonus, but its embedding, forward, and inverse-model parameters are not updated by the RL reward signal.This separation prevents the policy objective from directly reshaping the representation used for intrinsic reward.
  • Design properties: RIDE does not reward features that the agent cannot influence, making its exploration robust to distractors and other inconsequential environmental variation.Unlike prediction-error bonuses, its intrinsic reward is unlikely to vanish merely because a forward model becomes accurate.

5 EXPERIMENTS

The experiments evaluate RIDE against standard reinforcement learning and intrinsic-reward methods on procedurally generated MiniGrid tasks and singleton visual environments. Across the MiniGrid tasks, RIDE is more sample efficient and solves challenges that other methods fail to solve.

  • Evaluation settings: RIDE is evaluated on procedurally generated MiniGrid environments and singleton environments with high-dimensional observations.The evaluation includes MiniGrid, Mario, and VizDoom, using IMPALA as the base RL algorithm.
  • MiniGrid environments: MiniGrid is a partially observable N × N grid in which each tile contains at most one object, including walls, doors, keys, balls, boxes, and goals.The task suite includes MultiRoomNXSY, KeyCorridorS3R3, and ObstructedMaze2Dlh.
  • Results: RIDE solves some very challenging MiniGrid tasks where other methods receive no reward even after training on over 100M frames.IMPALA cannot learn in any of the environments because the extrinsic reward is too sparse.

6 RESULTS AND DISCUSSION

RIDE is evaluated across MiniGrid, intrinsic-reward analyses, exploration without extrinsic feedback, and high-dimensional Mario and VizDoom tasks. It is more sample efficient on MiniGrid, maintains intrinsic reward during training, and explores procedurally-generated mazes more effectively than the other methods considered.

  • MiniGrid: RIDE is more sample efficient than the other exploration methods across all considered MiniGrid tasks, with the performance gap increasing as task difficulty rises.RIDE solves some challenging tasks where other methods receive no reward after more than 100M frames.
  • Analysis of the Intrinsic Reward: On MultiRoomN7S4, RIDE’s intrinsic reward is more structured for opening doors, entering new rooms, and turning at decision points.The analysis averages intrinsic rewards for each action type over 100 episodes on a random maze.
  • Analysis of the Intrinsic Reward: RIDE’s bonus remains higher and continues changing after 100M frames, whereas ICM, RND, and Count rewards converge to very low values early.The paper relates this persistence to continued encouragement of actions that change the local environment.
  • No Extrinsic Reward: On a singleton ObstructedMaze2Dlh environment, Count, RND, and IMPALA can also solve the task, highlighting the additional challenge created by procedural generation.The comparison uses the same task with the agent spawned in the identical maze and object configuration each episode.
  • No Extrinsic Reward: In procedurally-generated MultiRoomN10S6 mazes, RIDE reaches all rooms, while Count, RND, and ICM reach only the 4th, 3rd, and 2nd rooms.These agents were trained for 50M frames with intrinsic reward only; in singleton mazes, all models showed effective exploration.
  • Mario and VizDoom: On Mario, all evaluated methods learn similarly good policies after 1M frames without intrinsic reward, while on VizDoom RIDE performs as well as ICM and other baselines fail.With both intrinsic and extrinsic reward in Mario, ICM converges later and to a lower value than the other evaluated methods.

7 CONCLUSION AND FUTURE WORK

RIDE is an intrinsic reward that promotes exploration through impactful changes in a learned state representation and remains effective in procedurally generated sparse-reward environments. The authors identify several directions for improving impact measurement, temporal scope, diversity, and safety.

  • RIDE rewards actions that substantially change the environment state in a learned latent space.It is compatible with intrinsic or extrinsic rewards and any deep RL algorithm.
  • RIDE attracts agents to states where they can affect the environment and continues providing an exploration signal after long training.
  • RIDE significantly outperforms IMPALA, Count, RND, and ICM in procedurally generated sparse-reward environments.The approach is also reported to explore these environments more efficiently than other exploration methods.
  • Future improvements include symbolic impact measures, longer-term effects, diverse changes, and distinguishing useful from undesirable impact.The authors also suggest hierarchical controllers could use different impact types as skills or low-level policies.

A.1 NETWORK ARCHITECTURES

The experiments use shared convolutional-recurrent policy and value architectures, with preprocessing adapted to the observation settings of the evaluated environments.

  • All models use the same policy and value network architecture, combining convolutional layers with a 256-unit LSTM.MiniGrid uses three convolutional layers, while environments from Pathak et al. use four.
  • Singleton visual-input environments use grayscale 42 × 42 frames concatenated across the current and previous three observations.Training also uses action repeat of four to reduce overfitting.

A.2 HYPERPARAMETERS

Hyperparameters are selected through grid searches over optimization, intrinsic-reward, and entropy coefficients, with learning rates annealed linearly to zero.

  • Learning rates, batch sizes, and unroll lengths are selected by grid search across all experiments.The searched learning rates are [0.0001, 0.0005, 0.001], batch sizes are [8, 32], and unroll lengths are [20, 40, 100, 200].
  • Table 2 summarizes hyperparameters common to all experiments.
  • The best intrinsic-reward coefficient is 0.1 for ICM and RND and 0.005 for Count across all environments.
  • The best entropy coefficient is 0.0001 for ICM, RND, and Count across all environments.

A.3 MINIGRID ENVIRONMENT

MiniGrid is a partially observable, egocentric grid world with sparse rewards, discrete objects and colors, and a small action set. The evaluated tasks require navigation, object manipulation, unlocking doors, or avoiding obstacles.

  • MiniGrid consists of a partially observable N×N grid whose tiles contain at most one object.Possible objects include walls, doors, keys, balls, boxes, and goals.
  • MiniGrid provides seven actions: turning, moving, picking up, dropping, toggling, and ending an episode.Moving forward is blocked by occupied tiles and closed doors; toggle opens doors when the agent faces them.
  • Observations are egocentric 7×7×3 tensors encoding object type, color, and door state rather than RGB images.The agent cannot see through walls or closed doors.
  • MultiRoomNXSY: MultiRoomNXSY requires reaching a green goal in the most distant of randomly oriented connected rooms, with difficulty increasing in X and Y.The agent receives positive reward on reaching the goal and otherwise receives zero reward at episode termination.
  • KeyCorridorS3R3: KeyCorridorS3R3 requires finding a hidden key, unlocking a door, and picking up an object behind it.Episodes end with success or after 270 steps.
  • ObstructedMaze2Dlh: ObstructedMaze2Dlh requires retrieving a box from a 3×3 maze through locked, obstructed doors and hidden keys.Episodes end with success or after 576 steps.
  • DynamicObstacles: DynamicObstacles requires navigating to a fixed goal while avoiding moving obstacles, with collisions causing a -1 penalty and episode termination.

A.4 VIZDOOM ENVIRONMENT

RIDE’s impact-driven intrinsic reward and episodic count term support exploration in procedurally generated MiniGrid tasks. Ablations and behavioral analyses show that RIDE explores farther, reaches sparse rewards, and prioritizes controllable object interactions.

  • Ablations: RIDE combines impact-driven exploration with episodic state counts, while the ablations remove one or both components or entropy regularization.OnlyEpisodicCounts uses counts alone; NoEpisodicCounts uses impact without counts; NoEntropyNoEpisodicCounts also removes entropy regularization.
  • Ablations: OnlyEpisodicCounts solves none of the tasks, whereas NoEpisodicCounts solves easier tasks but needs more interactions than RIDE and fails on the hardest domain.NoEntropyNoEpisodicCounts either converges to a suboptimal policy or completely fails.
  • State visitation: On MultiRoomN12S10, NoEpisodicCounts visits ≤10 different states per episode, while RIDE increases toward ∼100 for an optimal policy.The comparison indicates that NoEpisodicCounts revisits some states during training.
  • State visitation: After 100m frames on MultiRoomN12S10, RIDE reaches the farthest room, while Count explores about half the maze and RND and ICM remain in the first two rooms.The environment provides very sparse extrinsic reward, making the exploration comparison especially demanding.
  • Intrinsic reward: RIDE assigns substantially more intrinsic reward to object interactions than to movement or turning, whereas RND rewards actions more uniformly.Examples include opening boxes or doors, picking up or dropping keys, and picking up the ball.
  • Limitations and generalization: RIDE solves DynamicObstacles despite large intrinsic rewards for chasing obstacles, but the method may still favor undesirable environment-changing actions.The experiment tests a stated limitation of impact-driven exploration rather than establishing that the issue never occurs.
Loading 2002.12292v2…