PaperlayerView exact revision on arXiv

Source-linked AI summary

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

Simone Parisi, Victoria Dean, Deepak Pathak, Abhinav Gupta

arXiv:2111.13119v1cs.LG

TL;DR

Task-agnostic exploration commonly assumes isolated environments and no prior experience, while real exploration builds across environments. C-BET learns exploration without extrinsic goals and transfers the policy to new tasks; the authors report stronger or competitive performance across unseen environments, including difficult MiniGrid transfer cases.

  • Problem

    Common task-agnostic exploration assumes isolated environments and agents without prior experience, whereas agents encounter multiple environments over time and can use what they have learned.

    Source: §1 · p. 1

  • Method

    C-BET learns a task-agnostic exploration policy from one or multiple environments without extrinsic goals, then transfers it as a fixed guide while task-specific policies are learned.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 10

  • Results

    Across unseen environments, C-BET outperforms or competes with prior methods; in MiniGrid MultiEnv transfer, it performs best and is the only method reported to transfer to MultiRoom-N6.

    Source: §1 · p. 2, §4.1.2 · p. 8

  • Takeaways & Limitations

    The authors conclude that task-agnostic exploration can use prior experience by learning across environments and transferring exploration policies to unseen ones.

    Source: §1 · p. 1, §5 · p. 10

    Exploration may be misaligned with task goals, and C-BET’s reliance on (pseudo)counts makes its current formulation more suited to discrete spaces.

    Source: §5 · p. 10

Abstract

from arXiv · show

Common approaches for task-agnostic exploration learn tabula-rasa --the agent assumes isolated environments and no prior knowledge or experience. However, in the real world, agents learn in many environments and always come with prior experiences as they explore new ones. Exploration is a lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of task-agnostic exploration. In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner. Later on, the agent effectively transfers the learned exploration policy to better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present a simple yet effective approach to learning task-agnostic exploration policies. Our key idea is that there are two components of exploration: (1) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent's belief; (2) an environment-centric component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs. We also introduce benchmarks and metrics for evaluating task-agnostic exploration strategies. The source code is available at https://github.com/sparisi/cbet/.

1 Introduction

The paper argues that isolated, tabula-rasa exploration misses how agents use prior experience across environments. It proposes learning task-agnostic exploration across environments and transferring the learned policy to new ones.

1 Introduction

  • Common task-agnostic methods assume isolated environments and agents with no prior knowledge, unlike the continual process of learning from new environments.The proposed setup learns without extrinsic goals, then transfers the exploration policy to new environments.Source: §1 · p. 1
  • Existing intrinsic-reward approaches mainly encourage exploration of unseen states based on the agent’s beliefs, but may overlook objects that remain interesting across environments.The paper contrasts agent-centric exploration with environment-centric interest in components such as a doorbell switch.Source: §1 · p. 2
  • C-BET combines agent-centric search for surprises with environment-centric interest in impactful changes, and performs more effectively in multi-environment learning while outperforming or competing with prior methods on unseen environments.The paper frames this as a transfer-exploration setup using policies learned from one or multiple environments.Source: §1 · p. 2

2 Preliminaries and Related Work

The paper situates exploration in reinforcement learning among intrinsic-reward methods and transfer approaches. It distinguishes its task-agnostic transfer of exploration policies from methods that transfer task-specific behaviors.

2 Preliminaries and Related Work

  • In an MDP, an agent selects actions from observed states, receives new states and rewards, and aims to maximize return; exploration is challenging when rewards are absent or sparse.The paper reviews intrinsic motivation, optimism, curiosity, bootstrapping, empowerment, and methods for incrementally learning tasks.Source: §2 · p. 2
  • Intrinsic-reward approaches such as counts and curiosity are generally agent-centric, while C-BET also learns environment-centric exploration without requiring a model or extrinsic rewards.C-BET encourages actions that cause interesting environmental changes.Source: §2 · p. 3
  • Prior transfer, continual-learning, and meta-RL methods typically transfer task-specific policies or representations, whereas this paper transfers task-agnostic exploration policies to unseen environments.The train/test distinction is used for evaluation, although the authors note that the real world has no explicit such boundary.Source: §2 · pp. 2–3

3 Learning to Explore

C-BET learns an exploration policy from task-free interaction, combining novelty with interest in rare environmental changes. It transfers that fixed exploratory component to new tasks while learning a separate task-specific policy.

3 Learning to Explore

  • C-BET first learns an environment-centric exploration policy through interaction without tasks, then uses it to guide exploration as task-specific policies are learned in new environments.The framework separates an intrinsic-reward-driven interest policy from the task policy driven by extrinsic rewards.Source: §3 · p. 3

3.1 Interestingness of State-Action Pairs

  • The intrinsic reward combines novelty for less-visited states with rewards for rare environmental changes, while actions that do not change the environment receive low interest.The paper’s reward visualization contrasts state-count rewards and an L2 change norm with C-BET’s higher reward for picking up a key.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2 Exploration Learning

  • Intrinsic rewards can vanish as counts grow, making any policy optimal if all rewards reach zero; episodic count resets can also repeatedly reward early trajectories over unexplored branches.In the chainworld example, episodic counts make the looping route through B and C preferable to visiting D.Source: §3.2 · pp. 4–5, Figure 5 · p. 5
  • C-BET randomly resets counts at each step with probability p ≤ 1 − γ_i, so counts may persist across episode boundaries and remain sensitive to rarer states and changes.The authors interpret γ_i as implying a life expectancy of 1/(1 − γ_i) steps and state that resets should not occur more frequently than that.Source: §3.2 · p. 5
  • The count-based MDP is non-stationary because rewards depend on visit history, although the authors report that intrinsic rewards change slowly enough for this not to be an issue in practice.The paper notes that prediction-error and count-based intrinsic rewards in other exploration settings also change over time.Source: §3.2 · p. 5

3.3 Exploration Transfer

  • At transfer, the fixed pre-trained exploration policy is added to the task policy’s action scores, while the task-specific value and policy are learned from extrinsic rewards.The transferred component can initially guide behavior; as extrinsic rewards are learned, the task policy can become greedier toward them, and exploration may be decayed if goals are misaligned.Source: §3.3 · pp. 5–6

4 Experiments

The experiments evaluate task-agnostic exploration and transfer in MiniGrid and Habitat, using interaction, task success, and scene coverage to compare policies. C-BET consistently outperforms baselines in transfer, including across unseen Habitat scenes.

4 Experiments

  • The experiments compare Count, RND, RIDE, and Curiosity across MiniGrid and Habitat, using panoramic views to encode environment changes.The paper presents the experiments as an evaluation framework for transfer exploration, for which it says standard benchmarks, setups, metrics, and baselines were lacking.Source: §4 · p. 6

4.1 MiniGrid Experiments

  • MiniGrid compares many-to-many and one-to-many pre-training with tabula-rasa learning, measuring unique interactions, goal-state visits, and extrinsic return.The environments are procedurally generated gridworlds with sparse rewards, partial observations, and object interactions such as picking up keys to open doors.Source: §4.1 · pp. 6–7

4.1.1 MiniGrid Pre-Training Results

  • After MultiEnv pre-training, C-BET interacts with the environment and finds goal states more often than baselines, including in environments with unseen dynamics.Count has high average interactions and success, but its success is concentrated in environments seen during pre-training; predictive-model baselines perform poorly.Source: §4.1.1 · pp. 7–8, Figure 7 · p. 7

4.1.2 MiniGrid Transfer Results

  • In MiniGrid, C-BET performs best in transfer and tabula-rasa settings, uniquely transfers to MultiRoom-N6, and alone solves the hardest ObstructedMaze task in tabula-rasa.RIDE, Curiosity, and RND generally perform worse in transfer than tabula-rasa, while no method learns MultiRoom-N12-10.Source: §4.1.2 · p. 8, Figure 8 · p. 8

4.2.2 Habitat Transfer Results

  • C-BET transfers across seven unseen Habitat scenes, uniformly discovering more states than every baseline; Figure 10 reports over twice the unique-state count.The evaluation uses episodes with fixed steps and measures how much of each scene is covered.Source: Figure 10 · p. 9, §4.2.2 · p. 9

4.2 Habitat Experiments

  • Habitat uses one-to-many transfer: policies are pre-trained with intrinsic rewards in one scene, then evaluated without further learning by scene coverage over seven random seeds.Coverage is measured from the agent’s true Cartesian state, which is not available to the agent; faster, larger, and more uniform coverage indicates better exploration.Source: §4.2 · p. 9

4.2.1 Habitat Pre-Training Results

  • During Habitat pre-training, C-BET covers twice as much of Apartment 0 as all baselines and explores almost all of it uniformly.The policy is pre-trained in Apartment 0 before transfer to new scenes.Source: §4.2.1 · p. 9, Figure 9 · p. 9, Figure 11 · p. 9

5 Discussion

The discussion presents a two-phase, task-agnostic exploration approach that rewards rare states and environment changes, then transfers the learned policy. Its scope is limited by possible misalignment between exploration and task goals and by the use of count-based rarity measures.

5 Discussion

  • The two-phase framework pre-trains exploration without extrinsic goals and transfers policies that favor rare environment changes, such as opening a door.The approach estimates rarity of states and changes from interaction rather than using additional models.Source: §5 · p. 10
  • The method transfers policies learned from environmental interestingness rather than task-specific policies, and uses task-agnostic interaction during pre-training.The authors identify this as a distinction from prior transfer and continual-learning algorithms.Source: §5 · p. 10
  • Exploration may conflict with task goals, and C-BET’s count-based rarity measures are better suited to discrete than continuous spaces.Curiosity during pre-training could attract an agent to dangerous states or harmful objects that it should avoid.Source: §5 · p. 10

A.1 Algorithms Details

  • C-BET combines rewards for rare state changes and rarely visited next states, with count resets during pre-training; the evaluated alternatives use counts or prediction-based intrinsic rewards.The appendix defines Count, RND, Curiosity, and RIDE reward formulations alongside C-BET.Source: §A.1 · p. 15

A.2 Hyperparameter Details

  • The appendix specifies shared policy and value architectures and uses a 0.001 count-reset probability, corresponding to an approximate 1,000-step life expectancy.The count-reset choice is motivated by the largest environment horizon of 640 steps.Source: §A.2 · p. 15

A.3 Compute Details

The appendices report compute requirements, describe the benchmark environments and plotting procedure, and detail Habitat’s view encoding and static hashing.

A.3 Compute Details

  • A MiniGrid pre-training run takes about eight hours for 50M steps; a Habitat run takes about 45 hours for 2M steps on the reported cluster.Habitat uses four IMPALA actors to prevent memory errors, while MiniGrid uses 40.Source: §A.3 · p. 16

A.4 MiniGrid Environments Details

  • MiniGrid tasks vary in object interactions, room layouts, and episode horizons, with procedurally generated grids and seven discrete actions.The tasks include unlocking doors, revealing keys in boxes, moving objects, and navigating multiple rooms.Source: §A.4 · p. 16

A.5 MiniGrid Plot Smoothing Details

  • MiniGrid plots average seven seeds and smooth results over 300 epochs, while Habitat state counts need no smoothing because they are available at every step.MiniGrid smoothing accommodates episode returns that can only be computed at episode end.Source: §A.5 · p. 17

A.6 Habitat Environments Details

  • Habitat uses 64×64×3 egocentric views for policy input and state counts, plus a 360° panoramic view for counting environment changes; static ternary hashing maps views to codes.The hash outputs −1, 0, or 1 using thresholds on g(x), and preliminary results favored this encoding over SimHash.Source: §A.6 · p. 17, Figure 15 · p. 17

B Ablation Study on MiniGrid

The ablations identify C-BET’s strongest design as combining agent-centric state counts with environment-centric change counts, resetting counts, and pre-training across multiple environments. They also show how view choice and reward construction shape transfer.

B Ablation Study on MiniGrid

  • C-BET combines change-count and state-count reward terms; placing them within a square root rewards transitions only when both counts are low.Squaring the reward further penalizes transitions as either count increases.Source: B Ablation Study on MiniGrid · p. 18, Figure 16 · p. 18

Aim of the Study

  • The ablations test count resets, state and change encodings, reward combinations, and pre-training environments, evaluating transfer by average success over 100 episodes.Success rates are averaged over seven seeds; plots report per-environment results and tables average across setups.Source: Aim of the Study · p. 18

B.1 State-Only Counts: Egocentric vs Panoramic Views, Squaring vs Classic Reward

  • 28.3%: egocentric state views had the highest average success rate across three setups, versus 24.9% for panoramic views, which did better in MultiEnv than SingleEnvs.Squaring the state-only reward performed slightly worse, with policies appearing to overfit to training environments.Source: Figure 17 · p. 19, §B.1 · p. 19

B.2 C-BET: Different Reward Combinations

  • 33.64%: Eq. (3) achieved the highest success rate and was the only reward with good transfer to ObstructedMazes, especially after MultiEnv pre-training.The sum-and-product combination was second at 31.9%, but did not transfer to ObstructedMazes after MultiEnv pre-training.Source: Figure 18 · p. 20, §B.2 · p. 20

B.3 C-BET: Egocentric Views

  • Using egocentric views to encode changes decreased every reward’s performance; panoramic changes better encode environment changes, so C-BET’s advantage comes from combining them with egocentric state counts.Neither state-only nor change-only counts matched C-BET’s success rate and transfer generalization.Source: §B.3 · p. 21, Figure 19 · p. 21

B.4 C-BET: Why Do ‘Change-Only Counts’ Perform Poorly?

  • Panoramic change-only counts can teach policies not to turn in SingleEnvs, while egocentric change-only counts favor turning; C-BET moves less and interacts more.With egocentric changes, turning is consistently rewarded, whereas other actions can produce no change; panoramic change-only counts overfit to moving forward.Source: §B.4 · pp. 21–22, Figure 20 · p. 22

B.5 Counts: No Resets vs Random Resets

  • Without resets, intrinsic rewards decay as counts grow and transfer worsens, especially after DoorKey pre-training; C-BET still transfers across environments from MultiEnv.Decay is particularly pronounced in DoorKey, where changes are sparse and states are similar.Source: Figure 22 · p. 23, §B.5 · p. 23, Figure 21 · p. 23

B.6 Conclusion of the Study

  • Combining egocentric state counts with panoramic change counts, random resets, and multi-environment pre-training supports the strongest generalization in the ablations.The study identifies this combination as C-BET’s key contribution; transfer is harder from environments with similar states or sparse changes.Source: §B.6 · p. 24

C MiniGrid Pre-Training Supplemental Results

The supplemental results motivate evaluating exploration through both environmental interaction and task success, based on the idea that seeking rare changes can help agents find extrinsic rewards sooner.

C MiniGrid Pre-Training Supplemental Results

C.1 Unique Interactions per Episode at Pre-Training and at Offline Transfer

C-BET encourages more unique interactions during pre-training and transfers this behavior to new environments. Its interaction pattern varies with the opportunities to manipulate objects.

C.1 Unique Interactions per Episode at Pre-Training and at Offline Transfer

  • C-BET encourages more unique interactions than the baselines during pre-training and offline transfer, and interacts in all transfer environments.Among baselines, Count interacts similarly but generalizes less well.Source: §C.1 · p. 25, Figure 23 · p. 25, Figure 24 · p. 25
  • C-BET’s transfer behavior includes interactions with unfamiliar dynamics, such as toggling boxes to reveal keys in ObstructedMazes.Most Count-baseline interactions occur in ObstructedMazes after MultiEnv training.Source: §C.1 · p. 25
  • C-BET interacts more in environments with many pickable objects and less in MultiRooms, where there is nothing to pick.The offline-transfer recap compares unique interactions across environments.Source: Figure 24 · p. 25

C.2 Extrinsic Return at Pre-Training and Success at Offline Transfer

C-BET shows increasing extrinsic return during pre-training and succeeds in many environments at the start of transfer, especially after MultiEnv pre-training. Baseline Count succeeds in some environments but generalizes less well.

C.2 Extrinsic Return at Pre-Training and Success at Offline Transfer

  • Count succeeds in some environments but overfits to pre-training environments and generalizes less well than C-BET.C-BET’s pre-training return trend is linked in the passage to finding goal states through better exploration.Source: §C.2 · p. 26
  • C-BET alone shows increasing extrinsic return across all pre-training setups, and its return is recorded only as a proxy for success.Agents receive no extrinsic rewards during pre-training.Source: Figure 25 · p. 26, §C.2 · p. 26
  • At the beginning of transfer, C-BET succeeds in many environments, especially when pre-trained on MultiEnv.These transfer evaluations occur without further extrinsic-reward training.Source: Figure 26 · p. 26, §C.2 · p. 26

C.3 The Problem of Vanishing Intrinsic Rewards at Pre-Training

During pre-training, common intrinsic rewards decay toward zero, while C-BET rewards increase; unlike Count with resets, C-BET also succeeds in unseen environments.

C.3 The Problem of Vanishing Intrinsic Rewards at Pre-Training

  • Count resets keep C-BET rewards from decaying, providing meaningful feedback as the agent learns to interact with the environment.The passage contrasts this with intrinsic rewards that vanish over time and prevent further learning.Source: §C.3 · p. 27
  • Count with resets generalizes poorly and overfits to training environments, whereas C-BET succeeds in unseen environments.Source: §C.3 · p. 27
  • C-BET rewards increase during pre-training as RIDE, Curiosity, and RND rewards quickly decay; Count rewards also vanish, but to a lesser extent.Figure 27 attributes C-BET's increasing rewards to count resets.Source: Figure 27 · p. 27, §C.3 · p. 27

D Noisy Environment Supplemental Results

In noisy MultiRoom environments, count resets matter for learning, while noise has different effects on C-BET and Count: it slightly hurts C-BET but benefits Count.

D Noisy Environment Supplemental Results

E Habitat Supplemental Results

C-BET's exploration transfers across scenes, including with egocentric change encoding, although panoramic encoding performs better.

E.1 C-BET With Egocentric Change

  • With egocentric change encoding, C-BET still visits more of Apartment 0 during pre-training and more of all scenes at transfer than the baselines, but performs less well than with panoramic views.The comparison supports the role of combining agent-centric state counts with environment-centric change counts.Source: §E.1 · p. 29, Figure 29 · p. 29

E.2 Scene Coverage at Transfer on All Scenes

  • C-BET outperforms baselines at transfer across scenes, visiting smaller scenes completely and most larger scenes within one episode.The evaluation uses 100 episodes per scene, 50,000 total steps, and a fixed starting location; darker red cells indicate higher visitation rates.Source: Figure 30 · p. 30

Apt 0

  • C-BET is the only method reported to visit Room 0, Room 1, Room 2, Office 3, and FRL Apt. 0 completely, with uniform visitation counts across scenes.Source: Figure 30 · p. 30
Loading 2111.13119v1…