PaperlayerView exact revision on arXiv

Source-linked AI summary

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Akhil Bagaria, Anita De Mello Koch, George Konidaris

arXiv:2609.36473v1cs.AIcs.LG

TL;DR

Existing option discovery often targets entire states, limiting how broadly options apply. This paper selects relevant features behind novelty spikes to form abstract subgoals, and its method outperforms flat RL and a state-reaching hierarchical baseline across three sparse-reward, image-based domains.

  • Problem

    Most option discovery methods target entire states rather than the relevant state features, making subgoals less abstract.

    Source: §1 · p. 1

  • Method

    The method attributes novelty spikes to relevant features and builds subgoal classifiers that attend only to those features.

    Source: §1 · p. 2

  • Results

    Across three image-based domains, Abstract Subgoals achieves the best final performance, outperforming flat RL and a state-reaching hierarchical baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The discovered subgoals depend on few features and are achieved across qualitatively different states, enabling skill reuse.

    Source: §1 · p. 2

    The method relies on external feature extraction that is biased toward object-centric features.

    Source: §5 · p. 9

Abstract

from arXiv · show

Temporal abstraction via options can improve exploration in large environments. However, existing option discovery algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the agent, and impedes progress on its primary task of reward maximization. We introduce an algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.

1 INTRODUCTION

The introduction argues that state-reaching options target entire states, limiting transfer and narrowing where a skill applies. It proposes discovering subgoals from novelty by selecting relevant features, then reports better performance than flat RL and a state-reaching hierarchical baseline across three image-based tasks.

Motivation

  • State-reaching methods target entire states, tying skills to irrelevant details and limiting transfer while shrinking the region where each subgoal applies.Recreating every aspect of a state can also be unscalable in large, realistic domains.Source: §1 · pp. 1–2

Approach

  • The method attributes novelty spikes to a small set of relevant features and builds subgoal classifiers that attend only to those features.The classifiers provide option-training rewards, and a high-level policy selects among options to maximize overall reward.Source: §1 · p. 2

Evaluation

  • Across MINIGRID-KEYCORRIDOR, VISUALTAXI, and MONTEZUMASREVENGE, the method outperforms flat RL and a state-reaching hierarchical baseline.The experiments use image-based observations and sparse rewards.Source: §1 · p. 2

Contributions

  • The central contribution is learning which features matter for each subgoal, yielding options that can be reused in different contexts.The introduction frames feature selection as a missing step in existing option discovery methods.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background frames options as temporally extended actions organized around subgoals, then reviews novelty, feature attribution, and prior approaches to skill discovery. It positions selecting relevant features for each subgoal as a way to generalize beyond whole-state targets.

Subgoal Options

  • An option is a temporally extended action with an initiation function, an intra-option policy, and a termination condition; its horizon is capped at H steps.A subgoal classifier can define termination and provide a binary pseudo-reward for training the option policy.Source: §2 · pp. 2–3
  • Universal Value Function Approximators share goal-conditioned value functions and policies across options by conditioning them on each option’s subgoal.This replaces a separate function approximator for every option.Source: §2 · p. 3

Novelty-based exploration

  • Novelty bonuses can guide exploration, but novelty-driven agents do not by themselves learn options for transfer and high-level planning.The paper uses novelty as a substrate for option discovery and says its framework is agnostic to the novelty estimator.Source: §2 · p. 3

Gradient-based feature attribution

  • Shapley values measure each feature’s marginal contribution averaged over feature orderings, while DeepSHAP approximates them for neural networks by backpropagating output attributions to pixels.Exact computation requires evaluating the value function on exponentially many feature subsets.Source: §2 · p. 3

Option discovery methods

  • Option discovery methods use extrinsic reward, empowerment, spectral objectives, or combinations of goal-conditioned learning and novelty, but most still target entire states.The paper treats feature representation as orthogonal to these learning objectives.Source: §2 · p. 3

Learning Subgoal Achievement Classifiers

  • Hand-designed subgoal classifiers use privileged information, while learned classifiers often attend to all features; the paper instead selects a small relevant set for each subgoal.Mutual-information methods can abstract away uncontrollable features but not irrelevant controllable ones.Source: §2 · pp. 3–4

Options for feature attainment

  • Prior feature-attainment methods assume factored state representations and target individual features or combinations, whereas this approach selects features from pixels based on novelty.The paper also connects learned subgoal classifiers to predicates used in symbolic planning.Source: §2 · p. 4

3 METHOD: ABSTRACT SUBGOAL DISCOVERY

The method discovers options by identifying which features account for novelty spikes, then building subgoal classifiers that match only those features. It alternates between selecting options, exploring from their subgoals, and discovering new options.

3 METHOD: ABSTRACT SUBGOAL DISCOVERY

  • Novelty spikes trigger feature attribution; the relevant features define a subgoal classifier, and an option policy is learned to achieve that subgoal.The process repeats as the agent explores from promising subgoals and discovers new interesting states.Source: §3 · p. 4

3.1 OPTION SELECTION AND EXPLORATION

  • Each episode, the agent selects an option, navigates to its subgoal, and explores; novelty spikes trigger discovery using lower-novelty baseline states from a nearby time window.The agent selects baseline states without replacement from a window around the novel state, choosing states whose novelty is sufficiently lower.Source: §3.1 · pp. 4–5
  • The option-selection policy restricts choices to subgoals deemed reachable, then balances expected extrinsic reward against the value of exploring from each subgoal.A bandit samples options proportionally to utility; α and λ control the tradeoff between extrinsic and intrinsic rewards.Source: §3.1 · pp. 4–5

3.2 IDENTIFYING RELEVANT FEATURES

  • Candidate features come from contour detection or pretrained segmentation; counterfactual substitution tests feature relevance by replacing one feature with its baseline value, assuming roughly independent contributions.Alternatively, gradient-based attribution aggregates pixel attributions over each feature and applies a threshold to identify relevant features.Source: §3.2 · p. 5

3.3 SUBGOAL CLASSIFIER CONSTRUCTION

  • The classifier stores the novel state and checks only selected features, allowing other state details to vary; its match test becomes the option’s termination condition.It is non-parametric and uses a feature extractor that selects relevant indices in factored states or crops image bounding boxes.Source: §3.3 · p. 6

3.4 PUTTING IT TOGETHER

  • The full loop alternates between exploiting existing options and exploring from their subgoals; sufficient novelty spikes lead to feature selection, classifier construction, and a new option.The new option is added to the repertoire for future selection.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments test whether feature attribution identifies novelty-relevant image features and yields classifiers that generalize across irrelevant state differences. Abstract Subgoals achieves the best final performance across all three domains.

4 EXPERIMENTS

  • The online algorithm extracts candidate features from a novelty-associated state, selects relevant features, and constructs a classifier that becomes an option’s termination condition.The option’s policy and value function are updated using the classifier’s binary terminating pseudo-reward.Source: §4 · p. 7, §4.2 · p. 7

4.1 EXPERIMENTAL DESIGN

  • MiniGrid-KeyCorridor, VisualTaxi, and Montezuma’s Revenge test attribution, cross-state classifier generalization, and exploration performance in sparse-reward image domains.The first two tasks provide no reward until the entire task is completed.Source: §4 · p. 6, §4.1 · p. 7

4.2 IMPLEMENTATION DETAILS

  • Contour detection and counterfactual substitution serve MiniGrid and VisualTaxi, while SAM and DeepSHAP handle Montezuma’s visual complexity.Intra-option policies use goal-conditioned R2D2 with hindsight experience replay.Source: §4.2 · pp. 7–8

4.3 QUALITATIVE RESULTS

  • DeepSHAP highlights room-identifying features and the sword-near-skulls state while ignoring lives and score, isolating meaningful image features.The sword-near-skulls state precedes extrinsic reward.Source: §4.3 · p. 8, Figure 3 · p. 8
  • A classifier attending only to the player’s position in one screen region fires despite changes in room layout, score, lives, and enemy positions, with a median of 71.4 ± 11 triggers.Some classifiers are selective, while others apply broadly across states.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 QUANTITATIVE RESULTS

  • Abstract Subgoals achieves the best final performance in all three domains; Pixel Equality underperforms in two tasks where it cannot learn transferable option policies.CFN outperforms vanilla R2D2, and Abstract Subgoals improves over both CFN and Pixel Equality.Source: Figure 5 · p. 9, §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

The paper presents feature-attributed abstract subgoal options as reusable skills and reports more efficient exploration and better task performance than flat RL and a state-reaching HRL baseline. It also identifies limitations in feature extraction and subgoal usefulness.

5 DISCUSSION AND CONCLUSION

  • The feature-attribution abstraction enables more efficient exploration and better task performance than flat RL and a state-reaching HRL baseline.The method identifies features responsible for novelty and builds subgoal classifiers that attend only to those features.Source: §5 · p. 9
  • The method relies on external feature extraction biased toward object-centric features, and some subgoals target uncontrollable features such as moving enemies.The authors suggest end-to-end disentangled representation learning as a possible replacement and off-policy controllability estimation as a possible filter.Source: §5 · p. 9
  • The paper frames its key insight as identifying why an experience was interesting, rather than merely trying to recreate it.It presents this as a step toward agents that build abstract, reusable skills from experience.Source: §5 · p. 9

A NOTATION

B ILLUSTRATION OF CORE INSIGHT

  • The illustration contrasts the most novel state sN with the most boring state sB in a trajectory and uses their feature values to examine the novelty difference.The example has three features per state; the paper notes that it can handle input images but shows state features for simplicity.Source: B ILLUSTRATION OF CORE INSIGHT · p. 16

C MONTEZUMASREVENGE IMPLEMENTATION DETAILS

The Montezuma’s Revenge implementation soft-modulates option selection, normalizes CFN observations, and filters nearby redundant subgoals.

C MONTEZUMASREVENGE IMPLEMENTATION DETAILS

  • For Montezuma’s Revenge, the option’s initiation function soft-modulates its utility in the policy over options instead of using a strict threshold.The implementation scales U(o) by Vβo(s).Source: C MONTEZUMASREVENGE IMPLEMENTATION… · p. 16
  • Observation whitening is applied only to CFN, reducing the influence of bright objects on pixel attribution without changing the policy or value networks.The example given is a white skull.Source: C MONTEZUMASREVENGE IMPLEMENTATION… · p. 16
  • A κ-window filter discards a novel state when an existing classifier triggers nearby in the trajectory, reducing redundant subgoal coverage.Classifiers that trigger throughout the full window are excluded from filtering so broad triggers do not prevent new subgoals.Source: C MONTEZUMASREVENGE IMPLEMENTATION… · p. 16

D.1 HYPERPARAMETER SENSITIVITY

In Montezuma’s Revenge, most tested threshold configurations reach meaningful scores, with lower feature-attribution thresholds performing consistently better and intrinsic-signal thresholds producing mixed results.

D.1 HYPERPARAMETER SENSITIVITY

  • Most tested configurations reach 1000+ return; lower σpatch values consistently perform better, while σstate results vary during training.The sweep tests both multipliers at 1σ, 2σ, and 3σ with three random seeds per configuration; the main experiments use σstate = σpatch = 1.Source: Figure 7 · p. 17, §D.1 · p. 17

D.2 DECOUPLING DISCOVERY FROM REACHABILITY

A fixed-subgoal experiment separates classifier reachability from subgoal discovery in Montezuma’s Revenge. Abstract Subgoals outperform Pixel Equality under this setup, consistent with the concern that narrow subgoal regions impede option-policy learning.

D.2 DECOUPLING DISCOVERY FROM REACHABILITY

  • With the same fixed subgoals, Abstract Subgoals finishes at approximately 1,100 return while Pixel Equality achieves 0.Subgoal discovery is disabled during training, isolating the methods’ ability to reach the shared subgoals.Source: Figure 8 · p. 17, §D.2 · p. 18
  • The authors identify two entangled effects in the earlier comparison: classifier reachability and which subgoals each method discovers.They suggest the abstract method’s advantage may arise because the non-abstract subgoal regions are too small for effective policy learning.Source: §D.2 · pp. 17–18
  • The experiment uses trigger states from a separate discovery-only stage and trains both methods only to reach that fixed set.Abstract Subgoals constructs classifiers with SAM/DeepSHAP feature attribution, while Pixel Equality uses full-image matching.Source: §D.2 · p. 18

D.3 SENSITIVITY TO NOVELTY ESTIMATOR

The comparison finds similar subgoal classifiers from CFN and RND, though CFN attribution is more concentrated. In Montezuma’s Revenge, discovered-option counts grow rapidly at first and then decelerate toward saturation.

D.3 SENSITIVITY TO NOVELTY ESTIMATOR

  • The sensitivity test compares subgoals discovered independently by CFN and RND, rather than comparing their novelty judgments on identical states.Each estimator selects its own most-novel states during exploration.Source: §D.3 · p. 18
  • Both novelty estimators identify largely similar relevant features, while CFN separates relevant from irrelevant features more cleanly than RND.RND attribution becomes more diffuse outside the starting room.Source: §D.3 · p. 18

D.4 GROWTH IN NUMBER OF DISCOVERED OPTIONS

  • In MONTEZUMASREVENGE, Figure 10 tracks option count, which follows a logarithmic trend: it grows rapidly initially, then continuously decelerates, starting to saturate around 200 options.Source: §D.4 · p. 18

E DISCOVERED SUBGOALS IN MINIGRID

The MINIGRID examples show classifiers attending to selected image patches and recognizing subgoals across differing configurations of irrelevant features. The appendix also documents shared, domain-specific, and method-specific hyperparameter settings.

E DISCOVERED SUBGOALS IN MINIGRID

  • The locked-room classifier fires across key positions and door configurations, illustrating generalization over combinations of irrelevant features.The other examples combine location with the yellow-door state or hallway key position.Source: E DISCOVERED SUBGOALS IN MINIGRID · p. 18, Figure 11 · p. 20
  • CFN and RND produce similar subgoal classifiers, although CFN’s attribution maps are more concentrated and RND’s are more diffuse.Figure 9 compares each estimator’s independently selected most-novel state and attribution map.Source: Figure 9 · p. 19

F HYPERPARAMETERS

  • The hyperparameter appendix provides shared settings, flat-baseline settings, policy settings, option-discovery settings, and Montezuma-specific classifier settings.Methods were tuned independently except Pixel Equality, which reuses Abstract Subgoals’ hyperparameters to isolate the classifier effect.Source: F HYPERPARAMETERS · p. 21, Table 2 · p. 21, Table 3 · p. 21
Loading 2609.36473v1…