PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on intrinsic motivation in reinforcement learning

Arthur Aubret, Laetitia Matignon, Salima Hassas

arXiv:1908.06976v2cs.LGcs.AI

TL;DR

DRL still faces challenges including sparse-reward exploration and learning abstract, reusable skills. This survey categorizes intrinsic motivations, analyzes their role and limitations across DRL challenges, and proposes a compression-based developmental architecture; it concludes that these approaches may support broader task solving while leaving important challenges unresolved.

  • Problem

    DRL faces sparse-reward exploration difficulties and challenges in learning abstract, reusable skills and representations.

  • Method

    The paper surveys intrinsic motivation in DRL, categorizes its types, analyzes current methods and research questions, and proposes a compression-based developmental architecture.

  • Results

    The survey identifies prediction error, novelty search, and information gain as knowledge-gain approaches for exploration, each with distinct limitations and advantages.

  • Takeaways & Limitations

    The proposed developmental architecture organizes intrinsic-reward components around information abstraction and may allow agents to tackle many reviewed tasks.

  • Takeaways & Limitations

    The review is not exhaustive, focuses on intrinsic motivation applied to DRL, and notes challenges involving density, predictive, and mutual-information models.

Abstract

from arXiv · show

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be addressed, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment which can be addressed by intrinsic motivation (IM). In this article, we provide a survey on the role of intrinsic motivation in DRL. We categorize the different kinds of intrinsic motivations and detail for each category, its advantages and limitations with respect to the mentioned challenges. Additionnally, we conduct an in-depth investigation of substantial current research questions, that are currently under study or not addressed at all in the considered research area of DRL. We choose to survey these research works, from the perspective of learning how to achieve tasks. We suggest then, that solving current challenges could lead to a larger developmental architecture which may tackle most of the tasks. We describe this developmental architecture on the basis of several building blocks composed of a RL algorithm and an IM module compressing information.

1 Introduction

The introduction frames sparse exploration, skill abstraction, state representation, and curriculum learning as central DRL challenges. It surveys intrinsic motivation as a way to address these challenges and organizes current approaches toward a developmental learning architecture.

  • 1 Introduction: DRL faces challenges in exploration, skill abstraction, state representation, and curriculum learning.These challenges concern sparse rewards, reusable high-level decisions, meaningful state features, and acquiring related complex skills.
  • 1 Introduction: Intrinsic motivation can improve exploration, incremental skill learning, skill selection, and state representation without additional expert supervision.Its reward function is more general and can therefore generalize across environments.
  • 1 Introduction: The survey examines intrinsic motivation characteristics, its role in DRL, its relevance to DRL challenges, and its limitations.The authors explicitly identify these four aspects as the article’s focus.
  • 1 Introduction: The paper unifies intrinsic-motivation approaches through compression theory and proposes a developmental architecture composed of related learning components.The review focuses on DRL applications, current research directions, limitations, and perspectives rather than providing an exhaustive literature survey.
  • 1 Introduction: The paper reviews intrinsic motivation through knowledge acquisition and skill learning before analyzing challenges, task performance, and developmental integration.The organization proceeds from background and classification to methods, challenges, achievements, and a proposed framework.

2 Definitions and Background

This section introduces reinforcement-learning, information-theoretic, goal-parameterized, intrinsic-motivation, and empowerment concepts used throughout the survey. It contrasts task-provided rewards with internally generated motivation and describes empowerment as control over reachable environmental states.

  • 2 Definitions and Background: An MDP models interactions among states, actions, transitions, and rewards, with the objective of maximizing cumulative long-term reward.The agent repeatedly observes a state, takes an action, receives a transition and reward, and follows a policy mapping states to actions.
  • 2 Definitions and Background: RL seeks an optimal policy, while value functions Vπ(s) and Qπ(s, a) measure expected discounted returns from states or state-action pairs.Neural networks often approximate these quantities in continuous or very large state spaces.
  • 2 Definitions and Background: Goal-parameterized RL conditions policies on goals, enabling generalization across a goal space when that space is well constructed.UVFA concatenates a goal representation with the agent’s observation and represents the policy as π(s, g).
  • 2 Definitions and Background: Entropy measures uncertainty, while mutual information measures information about one variable contained in another and the corresponding reduction in disorder.Mutual information is symmetric, zero for independent variables, and can be conditioned on a third variable.
  • 2 Definitions and Background: Intrinsic motivation drives behavior through internally compared information rather than direct environmental feedback or task-assigned reward.The paper contrasts this with extrinsic motivation, whose reward function is externally specified for a task.
  • 2 Definitions and Background: Empowerment measures an agent’s control over its environment by favoring states with diverse and predictable reachable outcomes.The paper presents empowerment as a local utility based on the agent’s ability to influence future states.

3 Challenges of RL tackled with IM

The survey identifies four DRL challenges that intrinsic motivation can address: sparse rewards, state representation, temporal abstraction, and curriculum learning. These challenges arise because extrinsic rewards provide limited information for exploration, representation learning, abstract decisions, and goal ordering.

  • 3.1 Sparse rewards: Sparse rewards provide feedback only after long sequences of specific actions, leaving agents without local indications for improving their policies.This makes environments such as Montezuma’s Revenge difficult for standard exploration policies, which may never discover rewards.
  • 3.1 Sparse rewards: Reward shaping can ease sparse-reward learning, but designing intermediary rewards often introduces errors and requires expert knowledge.
  • 3.2 Building a good state representation: A useful state representation should be Markovian, preserve policy value, generalize well, and remain low-dimensional, yet sparse rewards and noisy observations hinder learning it.Representations learned from task rewards depend on the task, whereas independently learned representations can transfer across tasks.
  • 3.3 Temporal abstraction of actions: Options reduce decision frequency and ease credit assignment by grouping low-level actions into high-level actions with potentially different execution times.They can also make sparse-reward exploration easier because one exploratory option may directly lead to a reward.
  • 3.3 Temporal abstraction of actions: Learning reusable options remains difficult because manually defining intra-option policies requires expert knowledge, while reward-learned options are task-specific.
  • 3.4 Learning a curriculum: Curriculum learning organizes tasks or examples in a meaningful order, but standard approaches commonly rely on predefined task sequences or expert scores.Without prior knowledge, a robot may fail to learn a long sequence such as grasping and moving a cube.
  • Summary: The survey frames these challenges as sharing a common source: extrinsic reward is a poor information source, motivating the use of additional outcomes through intrinsic motivation.

4 Intrinsic motivation embedded into RL

The survey embeds intrinsic motivation into reinforcement learning by extending the agent’s internal state and reward computation, then classifies intrinsic motivations around knowledge acquisition and skill learning. It connects these categories to exploration, reusable abstractions, and developmental learning.

  • 4.1 A new model of RL with intrinsic rewards: The proposed RL framework separates external task dynamics from an internal component that computes states and secondary, intrinsic reward signals.The critic computes intrinsic reward and handles credit assignment, while decisions may be high-level choices decomposed into low-level actions.
  • 4.1 A new model of RL with intrinsic rewards: Intrinsic motivation is incorporated by treating the agent’s internal processes as part of the MDP environment and by distinguishing primary from secondary reward signals.
  • 4.2 Classification of the use of IM in RL: The survey proposes a classification that emphasizes knowledge acquisition and skill learning, extending earlier knowledge-based and competence-based categories.
  • 4.2 Classification of the use of IM in RL: Knowledge acquisition motivates agents to discover environmental information and can support exploration in sparse-reward settings through novelty or information-gain rewards.
  • 4.2 Classification of the use of IM in RL: Skill learning concerns constructing efficient, task-independent, reusable skills and choosing which skills to learn through a curriculum.The survey distinguishes motivation for representing skills from motivation for selecting skills.
  • 4.2 Classification of the use of IM in RL: The classification also acknowledges social motivations in cooperative games, where rewards depend on feedback from another agent rather than directly on human feedback.

5 Knowledge acquisition

The survey organizes knowledge-acquisition approaches around exploration, state representation, and empowerment, emphasizing how intrinsic rewards support these challenges while exposing important limitations.

  • Knowledge acquisition: Knowledge-acquisition methods address exploration, state representation, and empowerment, with exploration receiving the most substantial research attention.The survey notes that state representation can benefit from active knowledge search and that empowerment can produce interesting behaviors.
  • Exploration: Prediction-error rewards measure the distance between actual and predicted next states, but their usefulness depends critically on learning a relevant state encoding.The forward model predicts the next encoded state from the current encoded state and action; learning the encoding function is identified as the main challenge.
  • Exploration: Intrinsic Curiosity Module learns an action-controllable representation and excludes action-independent noise from its prediction error, enabling exploration without extrinsic reward in VizDoom and Super Mario Bros.In Super Mario Bros., the agent crossed 30% of the first level without extrinsic reward, although the representation may lose consequences that emerge several steps later.
  • Exploration: Count-based exploration is efficient in tabular environments but becomes difficult in large or continuous spaces because the agent rarely revisits exactly the same state.Hashing and density-model adaptations address large state spaces, while RND estimates novelty through prediction error with lower complexity and better final results than pseudo-count approaches.
  • Exploration: Exploration methods trade off modeling choices: count-based approaches suit fully observable MDPs, prediction error is simple but representation-sensitive, and information gain handles stochasticity but is harder to compute.The survey identifies exploration as the largest use case for intrinsic motivation and compares these method families by their practical strengths and limitations.
  • Empowerment: Empowerment rewards states where the agent controls its environment, but its complexity and reliance on environment models leave its potential insufficiently demonstrated.Experiments report diverse learned trajectories and improved task learning after empowerment pretraining, although the learned tasks remain relatively simple.

6 Skill learning

The survey examines skill learning with intrinsic motivation, covering unsupervised skill abstraction, expert-guided rewards, and curriculum-based goal selection. It compares their benefits for transfer, temporal abstraction, exploration, and learning efficiency, while identifying assumptions and limitations.

  • Skill abstraction: Intrinsic motivation can produce reusable skills without hand-engineering every task, enabling agents to acquire options through exploration.The survey contrasts learned skills with manually specified behaviors and emphasizes their potential reuse across tasks.
  • Skill abstraction: Skill abstraction learns representations of diverse skills, often using intrinsic rewards for intra-option policies while extrinsic rewards pursue a global task.This separates option learning from task achievement and supports learning on two timescales.
  • Skill abstraction: Expert-guided intrinsic rewards can support hierarchical action decomposition, but pivotal-state and salient-event heuristics are not general enough for environments without corresponding bottlenecks.An empty room may lack bottleneck states even though useful skills, such as reaching a corner, remain learnable.
  • Skill abstraction: Unsupervised goal-generation methods trade off continuous interpolation against compact option spaces, with state-based methods requiring suitable metrics and information-theoretic methods risking forgetting and stochastic skills.The survey identifies different limitations for continuous goal embeddings and trajectory-partitioning approaches.
  • Curriculum learning: Curriculum methods select objectives at an appropriate difficulty to accelerate learning across goals while avoiding unlearnable, mastered, or forgotten tasks.The survey discusses simple sampling, learning-progress bandits, and adversarial task generation as alternative selection mechanisms.
  • Curriculum learning: Goal-space design affects exploration: state-related goals can make skills useful, while adversarial methods require good representations and may neglect areas relevant to an external task.The survey reports that future or final states often work well because they provide novelty-like exploration effects and support generalization over goals.

7 Limitations and challenges of the methods

The survey identifies limitations in intrinsic-motivation methods involving stochasticity, long-term exploration, representations, and task assumptions. It also highlights unresolved challenges in scaling these methods to complex environments and integrating skills, exploration, and representations.

  • Environment stochasticity: Prediction-error rewards can be attracted to stochasticity, including white-noise effects, because they reward prediction error rather than its improvement.Information-gain methods may handle stochasticity better, but they face practical difficulty and computational overhead.
  • Environment stochasticity: Count-based methods handle some stochastic transitions but may remain attracted to states that are newly randomized on every visit.Their visit counts increase for recurring outcomes, but not for continually novel random states.
  • Environment stochasticity: Learning progress can avoid stochasticity in goal-space exploration, but the reviewed approach only handles a discrete goal space.This limits its direct applicability when goals vary continuously.
  • Long-term exploration: Intrinsic-motivation methods remain weak on long-term information search and very sparse-reward benchmarks such as Pitfall!.The survey reports that existing methods hardly achieve a score on Pitfall! and do not exceed 10000 on Montezuma’s revenge.
  • Long-term exploration: Intrinsic rewards can themselves be sparse, so exploration is difficult when the agent must discover the intrinsic reward before using it for guidance.The survey identifies partial observability as one source of intrinsic-reward sparsity.
  • State representation: State representations and intrinsic motivation remain intertwined, with current work differing in object-centered, disentangled, and predefined representations.The survey suggests cross-fertilizing these approaches to address their limitations.
  • Skill learning: Bottom-up skill learning can improve exploration and transfer, but existing approaches rely on task sequences, expert scores, strong assumptions, or task decomposition.These dependencies constrain autonomous skill acquisition and reuse.

8 Review of tasks involving IM

The reviewed intrinsic-motivation methods are evaluated across locomotion, manipulation, navigation, and first-person navigation tasks. Their strengths differ: skill acquisition handles harder locomotion, exploration discovers areas, while partial observability and object-oriented behavior remain difficult.

  • Locomotion: Locomotion exploration methods solve relatively easy tasks, whereas skill-acquisition methods learn forward movement on harder morphologies such as Ant.Ant has a 111-dim observation space and an 8-dim action space.
  • Manipulation: Manipulation studies remain limited: exploration methods generally touch or move some objects, while skill acquisition lacks object-oriented objectives.Manipulation is also a standard setting for curriculum-learning algorithms.
  • Navigation: Navigation is the most broadly tested task and includes exploration, skill-learning, and curriculum-based methods.Exploration discovers new areas, while curriculum methods can capture different reward modes.
  • First-person navigation: First-person navigation is challenging because agents receive only partial visual observations and must infer their true state.Existing work efficiently explores these environments, but count-based methods and skill learning remain insufficiently studied.

9 Analysis

The analysis unifies many intrinsic-motivation approaches as compression-based modules that provide rewards to reinforcement-learning policies. It proposes combining these modules with state representation, hierarchical reinforcement learning, and continual developmental learning, while acknowledging substantial assumptions and implementation challenges.

  • Compression and mutual information: Across knowledge acquisition and skill learning, mutual information appears central to expanding agent abilities.The survey connects empowerment, goal-conditioned paths, information gain, and related exploration methods to mutual information.
  • Compression and mutual information: Prediction error is related to mutual information but is limited by forward models’ inability to encode stochasticity.The survey presents prediction error as functionally related to information-gain approaches.
  • Unified architecture: The reviewed models commonly contain a policy maximizing an intrinsic reward and a compression module computing that reward, often through mutual information.Neural networks are used to generalize the compression function across large state spaces.
  • Assumptions: This unified view assumes properties such as limited stochasticity, Markovian dynamics, independent observation features, metric-like actions, and hierarchical task structure.These assumptions underlie compression, representation learning, and curriculum-learning components.
  • Developmental learning: The proposed developmental architecture combines intrinsic motivations, state representation, and hierarchical reinforcement learning to address multiple task-learning challenges.The architecture is intended to support continual skill creation and broader task coverage, but implementation requires solving the challenges identified earlier.
  • Developmental learning: The survey notes that current work rarely combines learned representations, knowledge-based intrinsic motivations, curriculum motivations, and multi-level reinforcement learning in one agent.This gap motivates the proposed integration.

10 Conclusion

The survey organizes intrinsic motivation into knowledge acquisition and skill learning, relating both categories to major deep reinforcement learning challenges and a broader developmental architecture. It identifies method-specific limitations and proposes combining compatible intrinsic-motivation blocks with reinforcement learning.

  • 10 Conclusion: The survey identifies sparse-reward exploration, skill hierarchies, state representation, and curriculum learning as major deep reinforcement learning challenges.These challenges concern exploration, credit assignment, transfer, learning speed, and acquisition of complex skills.
  • 10 Conclusion: Intrinsic motivations are classified into two broad categories: knowledge acquisition and skill learning.Knowledge acquisition concerns environmental information, while skill learning concerns discovering and abstracting skills.
  • 10 Conclusion: Knowledge-acquisition methods target exploration, state representation, and controllability through prediction error, novelty search, information gain, and empowerment.Prediction error depends on a good state representation, novelty search can be computationally inefficient, and information gain is difficult to compute but avoids the white-noise problem.
  • 10 Conclusion: Skill-discovery methods use goal-based or diversity-based abstractions, while curricula can focus learning on intermediate-difficulty goals and accelerate skill acquisition.Goal sampling can simulate novelty search, learning progress requires a discrete goal space, and adversarial methods can handle continuous goal spaces but are harder to train end-to-end.
  • 10 Conclusion: The survey reports that exploration methods struggle with stochastic environments, sparse intrinsic rewards, partial observability, credit assignment, and catastrophic forgetting across many tasks.It connects these issues to needs for better representations, planning, goal-parameterized exploration, skill hierarchies, and methods addressing catastrophic forgetting.
  • 10 Conclusion: The authors characterize intrinsic-motivation types as compatible blocks in a larger developmental architecture extending frameworks such as IMGEP.The proposed architecture combines reinforcement-learning algorithms with intrinsic-motivation modules that compress information.
Loading 1908.06976v2…