PaperlayerView exact revision on arXiv

Source-linked AI summary

Belief-Based Maximum Occupancy Principle and Active Inference

Manolis Mylonas, Rubén Moreno Bote

arXiv:2609.39342v1q-bio.NCcs.AI

TL;DR

The paper addresses how intrinsic-motivation agents can act under partial observability, where decisions require beliefs about hidden states. It extends MOP and reformulates Active Inference’s Expected Free Energy for belief-state planning; in the food-source experiments, MOP’s behavior shifts with energy while Active Inference and Empowerment favor a single source.

  • Problem

    The paper studies how intrinsic-motivation approaches can support adaptive, goal-directed behavior when agents must act under partial observability.

    Source: §1 · pp. 1–2

  • Method

    The paper extends MOP to belief-state POMDPs and reformulates Expected Free Energy as a Bellman equation for offline value iteration.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Results

    MOP balances exploration at high energy with increasingly goal-directed behavior as energy falls, while Active Inference tends to remain near one food source.

    Source: §4 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The comparison shows that MOP, Active Inference, and Empowerment induce different patterns of exploration and resource use in the studied setting.

    Source: §1 · p. 2, §4 · p. 7, §5 · p. 9

    The reported experiments use a simple two-food-source grid world, and the expected persistence of the frameworks’ qualitative similarities in richer environments remains prospective.

    Source: §5 · p. 9

Abstract

from arXiv · show

Intrinsic motivation plays a central role in adaptive and goal-directed behavior by conferring agents reward-independent objectives and biases useful to act in noisy and uncertain environments. Active Inference addresses the problem of acting in a partially observable environment through a principled framework for belief updating and action selection. A key component of Active Inference is the specification of prior preferences, which shapes behavior by encoding desirable future outcomes. An intrinsic motivation approach called the Maximum Occupancy Principle (MOP) proposes that agents act so as to maximize occupancy over future paths of states and actions, with no preferences or epistemic targets. Despite its simple formulation, MOP gives rise to rich and adaptive behaviors that combine exploratory variability with goal-directed dynamics. In this work, we extend MOP to partially observable environments and introduce a Bellman reformulation of the Expected Free Energy for Active Inference, both incorporating belief-based inference over hidden states as part of the agent state. The Bellman formulation enables tractable offline computation via value iteration over the full belief-state space. We compare the resulting behaviors in a set of minimal experimental settings with uncertain food sources. We find that MOP agents switch between goal-directed (food seeking) behavior and exploration between different food sources, depending on their energy available and their belief state. In contrast, Active Inference agents mostly inhabit regions around a single food source, a strategy having both high pragmatic and epistemic value. We finally compare with Empowerment, which is shown to be qualitatively similar to Active Inference.

1 Introduction

Intrinsic motivation accounts for adaptive behavior that reward maximization alone does not explain. This work extends the Maximum Occupancy Principle (MOP) to partial observability and compares its behavior with Active Inference and Empowerment.

1 Introduction

  • MOP maximizes the entropy of future action-state paths, favoring diverse trajectories and producing variable yet goal-directed behavior without explicit rewards.The introduction also describes life-preserving behavior as avoiding terminal states where no further path entropy can be generated.Source: §1 · p. 2
  • Active Inference combines information-seeking with prior preferences, while Empowerment maximizes mutual information between actions and future states.Both are presented as intrinsic-motivation alternatives to purely extrinsic reward-based decision making.Source: §1 · p. 2
  • The paper extends MOP to POMDPs using beliefs over hidden states and reformulates Expected Free Energy as a Bellman equation for offline value iteration.It compares the resulting behaviors with Active Inference and Empowerment in a two-food-source grid world.Source: §1 · p. 2
  • MOP exploration varies with energy and belief, whereas Active Inference and Empowerment tend to inhabit one food source.The comparison examines behavioral richness, spatial extent, and whether goal-directed action emerges without explicit encoding.Source: §1 · p. 2

2 Partial Observability Formulation

The formulation represents a partially observable world through an observable state and a belief over hidden food availability. Planning uses the resulting state-belief pair as the agent’s sufficient statistic.

2.1 A two-food-source partially observable grid-world environment

  • The example is a 5 × 5 grid world with two food sources, observable position and energy, and hidden binary variables for food availability.The setup is specialized to this environment, although the paper states its framework applies to general POMDPs.Source: §2.1 · p. 2, §2 · p. 2

2.2 Environment Dynamics

  • Spatial motion is deterministic, while energy changes depend on the new position and hidden food state; reaching zero energy leaves only the stay action.The transition model factors into energy, spatial, and food-state transitions.Source: §2.2 · p. 3
  • Food sources evolve independently, with dynamics that may depend on visitation but not on the agent’s energy.The formulation considers both location-independent dynamics and depletion at a visit.Source: §2.2 · p. 3

2.3 Observation Model

  • Observations are fully informative at a food source and uninformative away from it, preserving partial observability in a simple environment.The observation model assumes conditional independence given location and hidden state.Source: §2.3 · p. 4

2.4 Belief-State Representation

  • After an action and observation, the agent updates its probability distribution over hidden food states using Bayes’ rule.The initial belief is factorized, and the formulation states that this factorization is preserved by updates.Source: §2.4 · p. 4
  • Beliefs are discretized onto a finite grid so standard dynamic programming can operate over the full agent state.The belief is otherwise continuous, and the full state is represented by observable state s and belief b.Source: §2.4 · p. 4

2.5 Predictive Model

  • The sufficient statistic for planning is (s, b), which summarizes the available information and replaces unknown hidden states with beliefs.The predictive model gives the distribution of next observable states and beliefs following an action.Source: §2.5 · p. 4

3 Intrinsic motivation models

The paper formulates MOP and Active Inference over state-belief pairs, alongside Empowerment as an information-theoretic control objective. Its methods use belief updates and dynamic programming to define policies under partial observability.

3.1 Maximum Occupancy Principle

  • MOP optimizes expected policy entropy and state-transition entropy over future paths, with terminal states assigned zero value.The optimal policy is derived from the value function over observable states and beliefs.Source: §3.1 · p. 5

3.2.1 Variational Free Energy

  • Variational Free Energy balances prior-deviation complexity against observation-fit accuracy, and its minimizer yields the belief update used in the grid world.The paper states that minimizing VFE is mathematically equivalent to its belief-propagation equations.Source: §3.2.1 · pp. 5–6

3.2 Sophisticated Inference

  • Active Inference selects actions to minimize Expected Free Energy, balancing pragmatic risk against epistemic ambiguity resolution.Its prior preferences are uniform over location-energy states except for low preference assigned to zero energy.Source: §3.2 · p. 5, §3.2.2 · p. 6

3.2.2 Expected Free Energy

  • Expected Free Energy is formulated as a Bellman equation, enabling offline value iteration over the full state-belief space instead of exponentially scaling deep tree search.The discount factor ensures a unique fixed point and a time-stationary policy; the softmax parameter d controls policy randomness.Source: §3.2.2 · p. 6

3.3 Empowerment

  • Empowerment chooses actions to maximize mutual information between an n-step action sequence and the resulting future observation.This quantity measures the agent’s influence over what it will perceive.Source: §3.3 · p. 6

4 Results

In the two-food-source experiments, MOP explores between food sources, while Active Inference settles near one source and Empowerment explores locally. MOP’s behavior shifts with energy; Active Inference’s exploration-survival trade-off is controlled by inverse temperature d.

Independent food transition dynamics

  • MOP visits both food sources and switches between them, whereas Active Inference settles near one source and Empowerment shows limited local exploration.The comparison covers independent and depletion food dynamics; Empowerment’s finite planning horizon limits its evaluation of distant alternatives.Source: §4 · pp. 6–7, Figure 2 · p. 7

Energy-dependent behavioral variability

  • Around E = 15, MOP shifts from broad exploration at high energy to concentrated, food-seeking behavior at low energy.High-energy behavior includes more uniform visitation and frequent switching; at low energy, trajectories become more directed toward food sources.Source: §4 · p. 7

Active Inference and the exploration-survival trade-off

  • As d increases, Active Inference becomes more conservative: state-visitation entropy falls from 2.7 to 0.7 nats between d = 0.5 and d = 10, while survival rises monotonically.Its occupancy narrows toward a single food source; d sets the exploration-survival trade-off externally.Source: §4 · pp. 7–8, Figure 3 · p. 8
  • At d = 0.8, MOP and Active Inference have similar policy entropy and comparable lifespans, but MOP switches between food sources 13.3 times per thousand steps versus 3.8 for Active Inference.MOP lies off Active Inference’s survival-versus-state-entropy trade-off curve, combining exploration and survival in a way no tested precision matches.Source: §4 · p. 8

5 Discussion

The discussion presents belief-based MOP and Active Inference as approaches for partially observable settings, and reformulates Expected Free Energy as a Bellman equation. It interprets their behaviors as combining exploration with continued viability, while noting that the experimental example is intentionally simple.

Contributions and interpretation

  • The paper extends MOP and Active Inference to partially observable settings and reformulates Expected Free Energy as a Bellman equation for offline value iteration over belief states.This replaces the original framework’s tree search with dynamic programming and yields time-stationary policies applicable to POMDPs.Source: §5 · pp. 8–9
  • The discussion connects both frameworks to exploration and continued viability, while distinguishing Active Inference’s explicit risk term from MOP’s implicit avoidance of terminal states.Under MOP, terminal states collapse future path entropy; under Active Inference, risk favors preferred outcomes and avoids undesirable ones.Source: §5 · p. 9

Scope and broader implications

  • The authors identify the toy example’s simplicity as a scope boundary and expect, rather than demonstrate, qualitative similarities in more complex stochastic environments.Source: §5 · p. 9
  • Active Inference biases beliefs toward confident states through prior preferences, whereas MOP does not encode that bias and has more neutrality-preserving belief evolution.The authors suggest entropy-based control objectives offer a unifying perspective on intrinsic motivation in partially observable environments.Source: §5 · p. 9

A.1 Definition of various state spaces

The supplied definition material identifies “Propagated Belief” as a belief updated one step forward.

A.1 Definition of various state spaces

  • “Propagated Belief” is identified as a one-step-forward belief update.Source: §A.1 · p. 11

A.2 Environment Implementation

Table 1 is identified as listing experimental parameters, and the reported parameter changes affect conservativeness without changing the methods’ relative comparison.

A.2 Environment Implementation

  • Table 1 lists parameters used in the experiments.Source: Table 1 · p. 11
  • Changing Emax, Egain, and food-regeneration probabilities modulates agent conservativeness but leaves the relative comparison between methods qualitatively unchanged.Source: §A.2 · p. 11

A.3 Discretization of beliefs

The authors discretize beliefs on a uniform grid, balancing approximation detail against the size of the belief-state space. They use Δ=0.1 and report that it preserves the value function relative to a finer grid.

A.3 Discretization of beliefs

  • Reducing Δ from 0.1 to 0.01 increases the number of belief-grid combinations by 100×, so Δ=0.1 is chosen to keep the state space tractable.The two independent beliefs make the number of combinations scale as (1/Δ + 1)^2.Source: §A.3 · p. 11
  • When propagated beliefs fall between grid points, value iteration estimates successor values by linear interpolation between neighboring grid points.For example, b′_1=0.84 is interpolated between stored values at 0.8 and 0.9.Source: §A.3 · p. 11
  • At three representative states, the Δ=0.1 value-function estimates closely track the Δ=0.01 curve, with negligible differences relative to the value range.The comparison fixes b2=0.5 and varies b1.Source: §A.3 · p. 11, Figure 4 · p. 12

A.4 Beliefs in the two-food-source partially observable grid-world environment

In the two-food-source environment, beliefs depend on whether the agent reaches a food source: observations there are fully informative, while beliefs diffuse toward the prior away from food.

A.4 Beliefs in the two-food-source partially observable grid-world environment

  • At food source i, observation ωi is fully informative about the hidden state hi of that source.Beliefs saturate when the next position coincides with the food source.Source: §A.4 · p. 12
  • Belief saturation occurs only when the agent's next position x′ coincides with food source xf,i.Source: §A.4 · p. 12
  • Away from food source xf,i, the belief diffuses toward the prior.Source: §A.4 · p. 12

A.5 Variational free energy and belief propagation

The paper connects variational free energy minimization to belief propagation in the two-food-source grid-world. The resulting update is prediction-driven away from food and collapses to the observed-consistent state at a food source.

A.5 Variational free energy and belief propagation

  • The variational distribution Q(h′) is compared with a predictive prior induced by transition dynamics and an observation model to establish the belief-update rule.The observation model depends on the next state and hidden food state.Source: §A.5 · p. 12
  • Away from food, non-informative observations leave the posterior belief entirely driven by transitions, recovering pure prediction under the dynamics.Source: §A.5 · p. 12
  • At a food location, deterministic observations force the posterior to a point mass on the observed-consistent hidden state, matching the belief update used in the paper.The authors describe variational free energy minimization as a unified principle for these belief updates.Source: §A.5 · p. 13

A.6 Empowerment

Empowerment is computed independently at each state-belief pair by optimizing information between action sequences and future observations, while MOP and EFE use value iteration. The appendices explain EFE's risk and ambiguity terms, compare its behavior across policy precisions, and show that its terminal boundary value changes both behavior and computation.

A.7 Implementation of MOP, EFE and Empowerment

  • MOP and EFE use value iteration over the state-belief space, whereas empowerment enumerates action sequences and applies Blahut–Arimoto independently at each state.MOP and EFE use γ = 0.99; EFE policy stochasticity is controlled by inverse temperature d.Source: §A.6 · p. 13, §A.7 · p. 14

A.6 Empowerment

  • Empowerment maximizes mutual information between action sequences and future observations separately for each state-belief pair, but its |A|^n sequence enumeration makes longer horizons intractable.The Blahut–Arimoto algorithm solves each state independently, without information from neighboring belief states.Source: §A.6 · pp. 13–14

A.8 Explaining the EFE equation

  • EFE’s risk term penalizes predicted states that diverge from prior preferences, while its ambiguity term favors food locations because observations there reveal the hidden food state.At food, observation entropy is zero; away from food, it reaches its maximum in this grid world.Source: §A.8 · pp. 14–15

A.9 Stochastic and deterministic policies of Active Inference

  • At matched lifespan, EFE spends 41% of its time on food versus MOP’s 26%, while MOP spends 42% at least two steps away versus EFE’s 22%.At d ≤1, EFE visits both sources in 80–85% of episodes and covers a median of 22–23 grid cells; at d ≥2 it increasingly commits to one source. MOP makes 13.3 trips between sources per thousand steps, more than any EFE precision.Source: §A.9 · pp. 15–16, Figure 6 · p. 16

A.10 Boundary condition on the calculation of the EFE

  • G+ = 100 preserves stochastic exploration while ranking higher-energy states above near-death states; G+ = 0 encourages starvation, whereas G+ ≈1382 produces strong risk aversion and near-deterministic behavior.Large d makes the three boundary conditions nearly indistinguishable; larger G+ also slows value-iteration convergence.Source: §A.10 · pp. 16–17, Figure 7 · p. 17
Loading 2609.39342v1…