PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Motivation in Reinforcement Learning: A Research Agenda for Adaptive Self-Organisation

Anatoly Belikov

arXiv:2609.17325v1cs.AI

TL;DR

The paper asks whether intrinsic motivation can produce adaptive behaviour and higher-level organisation in artificial neural networks. It compares intrinsic objectives and their failure modes, then argues that complementary objectives may mitigate some limitations.

  • Problem

    The paper asks whether adaptation, memory, communication, functional differentiation, and higher-level organisation can emerge in non-biological recurrent neural networks.

    Source: §1.1 · p. 2

  • Method

    The article compares intrinsic objectives and their failure modes, reviews reward computation including world models, and outlines a research agenda for complementary per-agent objectives.

    Source: §1 · p. 2, §1.1 · p. 3, §4 · p. 12

  • Results

    The analysis finds that existing intrinsic objectives have characteristic failure modes, but some objectives may complement one another and mitigate their limitations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose studying whether environmental constraints and global communication channels can support development and self-organisation in intrinsically motivated agents.

    Source: §1 · p. 2

    A policy-level mutual-information reward may fail to shift the policy-induced state-visitation distribution toward broader coverage.

    Source: §8.1 · p. 25

Abstract

from arXiv · show

Biological cells can be viewed as individual, interacting agents whose collective dynamics give rise to adaptive behaviour at multiple levels of organisation, from individual cells through tissues to whole multicellular organisms. In this perspective and tutorial article we discuss whether intrinsic rewards in artificial neural systems can support adaptation, functional specialisation and higher-level self-organisation without a shared external objective. We review empowerment, curiosity, learning progress, information gain, unsupervised skill discovery, mutual information estimation and the use of world models for intrinsic reward computation. Particular attention is given to failure modes showing when such objectives do not produce sustained exploration or increasingly complex behaviour. We argue that more capable systems may require complementary objectives, communication, memory, learning at multiple temporal scales and environmental constraints. Based on this perspective, we outline three experimental directions. These include a resource-constrained environment in which otherwise stable behavioural attractors become unsustainable, allowing us to test whether environmental constraints can mitigate characteristic failure modes of intrinsic objectives. The network of recurrent agents with per-agent intrinsic rewards, and a hierarchical world-model agent in which exploratory motor competence develops before goal-directed behaviour. These experiments are intended to test whether intrinsic learning can lead to adaptive organisation at progressively higher levels.

1. Introduction

The article asks whether intrinsic rewards can support adaptive behaviour and higher-level organisation in artificial systems, drawing motivation from biological adaptation. It compares intrinsic objectives and their failure modes, then outlines research directions for testing whether complementary objectives and biologically inspired structures can support self-organisation.

1. Introduction

  • Intrinsic rewards are computed from what an agent observes during its lifetime, and the article asks whether they can support adaptation and higher-level organisation in artificial neural networks.The authors compare existing objectives and analyse failure modes, while considering environmental constraints and biological structures such as global communication channels.Source: §1 · p. 2, §1.1 · p. 2

1.1. Article structure and contribution:

  • The article compares existing objectives and their failure modes, and proposes a research agenda for studying whether complementary per-agent objectives can produce higher-level adaptation.Its tutorial also derives key identities, discusses corner cases, and reviews world models for intrinsic reward computation.Source: §1.1 · pp. 2–3

2. Motivation from biology

  • Biological motivation spans multiple levels: molecular networks show memory and conditioning, somatic cells learn and communicate electrically, and tumors switch metabolic pathways under changing conditions.Source: §2 · pp. 3, 6

2.1. Learning hierarchy

  • The article asks whether intrinsic motivation in RL could yield level 2 and 3 learning; examples include bumblebee ball play without external reward and Portia spiders learning to hunt a new spider by trial and error.The classification is heuristic because no universally accepted hierarchy covers all forms of learning and cognition.Source: §2.1 · pp. 7, 9

3. Basic mathematical definitions

The tutorial introduces entropy and mutual information as tools for describing uncertainty and action–state relationships, then applies them to reinforcement learning and empowerment. It also presents inverse-model estimation and explains continuous-action cases in which empowerment estimates can become problematic.

3. Basic mathematical definitions; Mutual information

  • Pointwise mutual information can be negative, while its expectation, mutual information, is non-negative.The section introduces entropy for discrete random variables and mutual information between random variables.Source: Mutual information · p. 11, §3 · p. 11

4. Empowerment in reinforcement learning

  • Empowerment is defined as action-to-future-state channel capacity, while policy optimization uses on-policy mutual information as a lower bound on it.The action–future-state relationship is conditioned on the current state; removing that conditioning can lead to a different, possibly degenerate solution.Source: §4 · pp. 12–13
  • The mutual-information reward favors diverse achievable future states with predictable outcomes given the action.The tutorial expresses the reward using pointwise mutual information between action and next state, conditioned on the current state.Source: §4 · pp. 13–14

4.1. Inverse model trick

  • An inverse model approximates the action distribution given current and next states, enabling a likelihood-based lower-bound estimate of on-policy empowerment.The method uses positive on-policy samples and is described as analogous to the ELBO objective in a variational autoencoder [17].Source: §4.1 · pp. 15–16
  • A single-normal predictor can average across multiple future-state modes, yielding diffuse estimates; mixture distributions are a possible alternative.The forward-model approach requires learning the future-state distribution, which the tutorial says is usually harder than learning the inverse model.Source: §4.1 · pp. 16–17, Figure 6 · p. 17

4.2. Continuous(differential) entropy

  • In continuous settings, exact action reconstruction can make empowerment infinite, and increasing action range can also drive it upward.The tutorial suggests constraining action range or covariance, or adding noise to observations or actions, to keep entropy and mutual information finite.Source: §4.2 · p. 18

5. Multistep and single-step empowerment

The section shows how information carried jointly across actions can be missed by single-step empowerment. It defines multi-step variants and contrasts marginal and causal objectives, including examples of their behavior and failure modes.

5. Multistep and single-step empowerment

  • In the two-step XOR example, each action reveals zero information locally, but a recurrent agent that remembers both actions obtains 1 bit from the final state.Summing single-step quantities can therefore miss information carried jointly by multiple actions.Source: §5 · pp. 18–19
  • k-step empowerment measures mutual information between an action sequence and a later state, while trajectory empowerment also includes intermediate states.Including intermediate states makes action recovery easier, so trajectory empowerment is at least as large as the final-state-only quantity.Source: §5 · p. 19

5.1. Marginal and Causal objective

  • The marginal objective accounts for action-sequence likelihood across possible trajectories, a computation that is generally intractable without approximation.With an environment model, Monte Carlo virtual rollouts can approximate that likelihood; the causal objective instead uses causal action entropy.Source: §5.1 · pp. 19–20
  • In the card-and-notebook example, copying cards can yield up to k bits, while a notebook can make a uniform random-symbol policy optimal under the stated objective.The causal objective can also be negative when its inverse model cannot reconstruct an action from the observed states.Source: §5.2 · p. 20

6. Local Optima

Intrinsic objectives can trap agents in controllable niches, while resource constraints and other agents may disrupt those attractors or create new ones. The section considers these mechanisms as ways to change otherwise persistent behavior.

6. Local Optima

  • Empowerment can trap an agent in a controllable object with many states, much as prediction-error rewards can trap agents at noise sources.A switch with many settings can offer easy manipulation and immediate, predictable feedback.Source: §6 · p. 21
  • Energy costs and recharge states could make toy-camping unsustainable and give an empowerment maximiser reasons to recharge and optimize actions for long-term reward.The proposed setup adds an energy coordinate; low energy makes control noisy or disables actions.Source: §6 · p. 21
  • In that setup, the agent may repeat a forage, play or act, and return-to-base cycle.The text likens this pattern to behavior of a living organism.Source: §6 · p. 21
  • Other agents may disturb controllable niches through transition variability or resource competition, but predictable or submissive opponents can leave empowerment high.Agents may also partition controllable objects into private niches; non-stationary dynamics could independently make camping impossible over time.Source: §6 · pp. 21–22
  • Other agents can either destabilize simple niches or create new intrinsically rewarding attractors, depending on their behavior, environmental constraints, and the objective.The text also suggests that multi-agent worlds may provide a natural brake on camping, while directly analogous resource constraints may be harder to introduce in other settings.Source: §6 · pp. 21–22

7. Communication and collective intrinsic motivation

Communication can extend empowerment beyond physical actuators, but independent agents may maximize their own control without producing useful collective behavior. The section considers communication channels and the conditions that might support higher-level coordination.

7. Communication and collective intrinsic motivation

  • Communication can extend an agent’s action channel, so empowerment may include causal influence mediated by other agents, not only direct physical actions.The section distinguishes self-, transfer-, and assistance-empowerment as different possible objectives.Source: §7 · pp. 22–23
  • Agents can gain high empowerment by exchanging arbitrary messages when one agent responds predictably, even if the exchange produces no useful collective behavior.In the two-agent example, a predictable response to a random message allows both agents to achieve maximum reward with a uniform message distribution.Source: §7 · p. 22
  • Local self-empowerment does not automatically optimize joint empowerment, and communication alone may be insufficient for stable higher-level organization.The biological comparison also identifies adhesion, functional differentiation, and conflict-limiting mechanisms as features combined with communication in complex multicellular organization.Source: §7 · p. 23
  • Biological communication examples include chemical signaling, electrical signals, and macroscopic electric fields, while optical signaling has a less established role.The text describes quorum sensing across species and kingdoms and notes electric gradients in embryogenesis.Source: §7 · pp. 23–24
  • A global channel aggregating and broadcasting agents’ messages might support more complex behavior in self-empowerment systems.The proposal defines an aggregated message from individual messages and broadcasts it to all agents.Source: §7 · p. 24

8. MINE and InfoNCE

The section describes MINE and InfoNCE as mutual-information estimators and discusses how estimation choices affect their use for intrinsic rewards. It highlights a policy-level failure mode for one information-based reward.

8.1. Mutual information neural estimation is defined as this objective:

  • MINE estimates mutual information using the Donsker–Varadhan representation and a neural network trained to maximize a function T over joint and marginal samples.For empowerment, the estimator takes action, next state, and current state as inputs because the quantity is conditioned on the current state.Source: §8.1 · pp. 24–25
  • For empowerment, the MINE objective estimates action–next-state mutual information conditioned on the current state, but shuffled states and actions instead estimate an unconditional quantity.The passage says the trained T function gives a point estimate up to an additive constant; shuffling approximates an unconditional next-state distribution rather than one conditioned on the current state.Source: §8.1 · p. 25
  • The policy-level failure mode is that the reward may not shift visitation toward broader coverage, instead favoring a few states that are easy to predict.The passage says H(S′) is fixed across states in a minibatch, leaving inverse-model optimization as the available option in that term.Source: §8.1 · p. 25

8.2. InfoNCE

  • InfoNCE scores positive joint-distribution pairs against negatives sampled from the marginal and can be viewed as a multiclass classification objective.The passage describes this estimator as often more stable than MINE and gives the lower bound I(X; Y) ≥ log N − L_InfoNCE.Source: §8.2 · p. 26

9. WORLD MODELS

World models infer state from observations and predict its evolution under actions, enabling policy training in virtual episodes. The section describes their components and discusses how action-conditioned contrastive prediction relates to empowerment and intrinsic reward.

9. WORLD MODELS

  • World models infer state from observations and predict state evolution given actions, allowing policy training in virtual episodes.The article introduces Dreamer v3 and TWISTER as prominent models.Source: §9 · p. 26
  • DREAMER/TWISTER components include a context encoder, dynamics predictor, observation encoder and decoder, plus models for embeddings, rewards, and episode continuation.The figure is identified as showing key components; the prose describes context and latent state representations and their associated predictions.Source: §9 · pp. 26, 28, Figure 7 · p. 27

9.1. Action Conditioned (AC-CPC)

  • Dreamer and TWISTER differ in their latent-state inference, while TWISTER’s AC-CPC adds a contrastive embedding loss to Dreamer’s reconstructive loss.Dreamer’s prior is updated to match an observation encoder that uses context; TWISTER’s observation encoder uses the observation.Source: §9.1 · p. 28
  • The mutual-information quantity considered for AC-CPC can reward predictable future trajectories even when the action is unknown.The derivation distinguishes global state-pool negatives from empowerment’s state-conditioned alternatives.Source: §9.1 · pp. 29–30
  • Using global-pool negatives in AC-CPC can let a discriminator distinguish future states without actions, whereas empowerment requires hard negatives from the same initial state under alternative actions.The text says world models can generate realistic virtual episodes for such comparisons.Source: §9.1 · p. 30

10. Diversity is all you need

DIAYN learns unsupervised skills by conditioning a policy on a sampled latent code and maximizing mutual information between skills and states, while keeping actions diverse. The section also describes structural variants and notes that skill discrimination alone does not ensure useful goal-directed behavior.

10. Diversity is all you need

  • DIAYN learns skills by conditioning a policy on a sampled code and maximizing mutual information between skill and state while maximizing action entropy and minimizing action–skill information given state.The state-conditioned skill predictor provides a practical reward estimate; estimating the state distribution conditioned on skill can be difficult for high-dimensional observations.Source: §10 · pp. 30, 32
  • The objective encourages broad state visitation while making states informative about skills and allowing varied actions within each state-skill pair.Its entropy terms correspond to visiting many states, concentrating each skill in fewer states, and allowing actions to vary by skill.Source: §10 · p. 31
  • DIAYN may avoid visiting the same state under different skills, so it does not by itself ensure arbitrary goals are reachable or skills are usefully combined.The section suggests external reward, curiosity, or structured skill codes as ways to address the repeated-state issue.Source: §10 · p. 32
  • Structural DIAYN can split the skill code across time so skills share a final state while reaching it through distinguishable trajectories.The policy uses both code parts, while early and late discriminators predict different parts of the code.Source: §10 · pp. 32–33

11. Curiosity and Learning progress

The section compares curiosity signals based on prediction error, novelty, and learning progress, then discusses information gain and slow features as ways to distinguish useful exploration from noise. It also identifies scale and persistence issues that can undermine intrinsic rewards.

11. Curiosity and Learning progress

  • Curiosity rewards can combine surprise, novelty, and learning progress, with prediction error and state visitation as basic measures of surprise and novelty.Prediction error uses a forward model; novelty is high for new states and encourages greater state-visitation entropy.Source: §11 · p. 33, §11.1 · p. 33

11.1. Prediction error and Novelty

  • Batch-normalized novelty can keep ranking episodes even as their absolute diversity and coverage decline.The problem arises when current-batch statistics rescale small differences between increasingly similar episodes.Source: §11.1 · p. 33

11.2. Learning progress

  • Learning progress measures change in the forward model’s parameter distribution and can be approximated by improved prediction error.Unlike prediction error and novelty, learning progress is described as less prone to noise-staring behavior.Source: §11.2 · p. 34

12. Information gain

  • Information gain about world state can separate uncertainty reducible through observation from irreducible noise, supporting rewards for informative exploration.The section cautions that information-gain rewards can still lead to “camping on a toy.”Source: §12 · pp. 34–35

13. SFA

  • Slow Feature Analysis could augment action-conditioned embeddings with slowly changing, controllable features for intrinsic rewards over longer timescales.The proposed SFA objective combines slowness, variance, and correlation losses.Source: §13 · p. 36

14. Predictive information bonus and MDL

The section proposes combining predictive information with diversity and cost objectives to favor structured, varied image generation, while explaining that fixed objectives can still be exploited or exhausted. It also discusses VAE-based description-length approximation and limitations of empowerment and DIAYN.

14. Predictive information bonus and MDL

  • Mutual information between image patches can reward structured, variable images, unlike blank images or independent noise, while a cost term discourages trivial pixel changes.The proposed combined reward adds empowerment, structure, and diversity rewards and subtracts action or complexity cost.Source: §14 · pp. 36–37
  • Predictive information and global diversity are complementary, but neither guarantees open-ended image generation or prevents cycling among a finite set of patterns.Repeated textures or high-frequency signals may score well without being semantically interesting; a continually expanding set of challenges is proposed as one requirement for open-endedness.Source: §14 · p. 37

14.1. VAE and MDL

  • A VAE’s evidence lower bound provides a lower bound on log likelihood and can approximate description length through negative log likelihood.The section describes the variational posterior as an approximation to the intractable model posterior.Source: §14.1 · p. 38

15. Problems with Empowerment and DIAYN

  • Single-step empowerment can be short-sighted, while DIAYN learns skills without specifying their order or goal use; state partitions may also fail to reach arbitrary goals.The section suggests combining rewards or using temporal hierarchy, with DIAYN skills as low-level actions and empowerment for higher-level selection.Source: §15 · pp. 38–39, Figure 8 · p. 39

16. Research agenda

The research agenda proposes experiments to test whether intrinsic objectives, environmental constraints, communication, memory, and temporal hierarchy can support adaptive organization. Its three directions cover constrained environments, recurrent modules with local rewards, and a two-level world-model agent.

16. Research agenda

  • The agenda proposes testing whether intrinsic motivation methods can produce biological-like adaptability across levels of organization.It connects the proposal to intrinsic motivation and creativity based on discovering novel but learnable regularities.Source: §16 · p. 40

16.1. Environment for intrinsic reward evaluation

  • The proposed predator–prey environment uses scarce energy that reduces action capacity and adds motor noise, while food or prey restores energy without direct reward.It is designed to test whether intrinsic-reward combinations maintain stable energy levels; planned measures include energy, toy time, coverage, and recovery.Source: §16.1 · pp. 40–41

16.2. Recurrent model

  • The recurrent-module experiment gives each local agent its own intrinsic reward and allows communication, without a network-level reward or gradients passing between agents.Shared parameters and distinct local states, inputs, and network positions are intended to test whether roles and higher-level adaptive organization emerge without permanent agent identities.Source: §16.2 · pp. 41–42

16.3. Automatic curriculum in a two-level world model agent

  • The two-level agent first develops low-level motor competence through exploratory rewards, then trains goal-conditioned behavior under high-level target selection.The high-level target is held for multiple steps; the experiment tests whether temporal hierarchy addresses DIAYN’s lack of skill ordering and short-horizon empowerment’s limited temporal reach.Source: §16.3 · pp. 43–44

17. Mathematical Appendix

The appendix gives the mutual-information chain rule and outlines a Donsker–Varadhan lower-bound estimator, expressing mutual information through KL divergence and optimizing its function T.

17.0.1. Chain rule for Mutual information

  • The chain rule decomposes I(X, Y; Z) into I(X; Z) plus I(Y; Z|X).The entropy terms H(Z|X) cancel when the two terms are added.Source: §17.0.1 · p. 44

17.0.2. Donsker-Varadhan MI

  • The Donsker–Varadhan construction forms a Gibbs density by weighting q(u) with e^T(u) and normalizing it by Z = E_Q e^T(u).This yields a valid probability density g(u).Source: §17.0.2 · p. 44
  • Mutual information is represented as the KL divergence between the joint density and the product of the marginal densities.The appendix describes the resulting KL and mutual-information bounds as lower bounds, and says T can be optimized by gradient ascent.Source: §17.0.2 · p. 45

Related papers

Loading 2609.17325v1…