PaperlayerView exact revision on arXiv

Source-linked AI summary

Expected free energy as an information constraint on the Bethe Lagrangian

Wouter M. Kouw

arXiv:2608.17167v1cs.ITcs.AIeess.SYstat.ML

TL;DR

Expected free energy lacks the Kullback–Leibler structure needed for variational message passing. This paper replaces it with a constrained Bethe formulation whose solved information multiplier recovers expected-free-energy policies at γ = 1 and supports information-seeking behavior across tasks.

  • Problem

    Expected free energy includes unobserved-outcome expectations that break its Kullback–Leibler structure and hinder variational message-passing updates.

  • Method

    The paper minimizes a constrained Bethe free energy with mutual-information lower bounds, yielding policies whose KKT multiplier is solved rather than tuned.

  • Results

    The constrained Bethe agent matches expected-free-energy policies at γ = 1, seeks informative cues, and reallocates effort from novelty to salience as it learns.

  • Takeaways & Limitations

    The information demand selects among policies spanning inactive, interior, and saturated epistemic regimes while retaining exact expected-free-energy behavior at unit multiplier.

Abstract

from arXiv · show

Active inference selects actions by minimising an expected free energy functional over predicted futures. However, adding an expectation over yet-unobserved outcomes means the free energy functional no longer has a Kullback-Leibler structure, which hinders message passing treatments of inference procedures. We propose an alternative formulation based on a Bethe free energy functional, fully supporting inference by message passing. The epistemic drive is maintained by imposing an information constraint, next to normalisation, marginalisation and form constraints, insisting that the mutual information between future observations, states and parameters given actions must be at least as large as the entropy of the goal prior. For a specific value of the corresponding Karush-Kuhn-Tucker multiplier, the stationary point of this constrained Bethe Lagrangian recovers the expected free energy solution. We show that, as the information demand is varied, the solved multiplier moves through its inactive, interior, and saturated regimes. In the inactive regime the agent's epistemic drive switches off entirely, while in the saturated regime it is maximal. We compare the performance of the constrained Bethe agent on three tasks against EFE and Q-MDP.

1 Introduction

The paper frames active inference as expected-free-energy minimisation that balances exploration and exploitation, but EFE’s future-outcome expectation prevents direct Bethe-style message passing. It instead formulates active inference as constrained Bethe free-energy minimisation with a mutual-information requirement tied to goal-prior entropy.

  • Motivation: Expected free energy balances exploration and exploitation in active inference, unlike conventional reinforcement-learning objectives.The paper presents EFE as the functional minimised for adaptive decision-making under uncertainty.
  • Motivation: EFE lacks Kullback–Leibler structure because it averages over yet-unobserved future outcomes, preventing immediate distribution of stationary points over factor graphs.By contrast, Bethe free energy has a KL-divergence structure whose terms distribute cleanly over nodes and edges.
  • Contribution: The proposed formulation minimises a constrained Bethe free-energy functional rather than imposing a point-mass constraint.This follows Van de Laar et al.’s direction while changing the constraint used for active inference.
  • Contribution: The information constraint requires mutual information among observations, states and parameters given actions to be at least the entropy of the goal prior.The mutual information is averaged over the variational posterior over actions.

2 Problem statement

The paper formulates expected free energy for planning over predicted futures, where risk and ambiguity jointly balance preference satisfaction and information gain. However, averaging over unobserved outcomes breaks the KL-divergence structure required for variational message passing, motivating constrained Bethe free-energy minimisation.

  • Planning model: The agent plans over a finite-horizon state-space model with discrete actions, observations, states, transition parameters B, and emission parameters A.Predicted observations receive goal priors, and the planning model shares A and B across time.
  • Expected free energy: For independent control priors, EFE selects controls according to qEFE(uk) ∝ p(uk) exp(−G(uk)), where G(uk) decomposes into risk and ambiguity.Risk measures divergence from the goal prior, whereas ambiguity penalises outcomes that are uninformative about latent states and parameters.
  • Expected free energy: EFE combines exploitation and exploration in one objective by expressing negative expected free energy as goal cross-entropy plus predictive mutual information.The mutual information is between latent variables and predicted outcomes.
  • Message-passing limitation: Averaging over unobserved outcomes means the EFE functional is not a KL divergence and therefore does not decompose into factorwise energies and edgewise entropies.Consequently, its stationary points do not correspond to variational message updates.
  • Constrained Bethe formulation: The proposed remedy is to recover the EFE solution by minimising a Bethe free-energy functional under an appropriate set of constraints.The subsequent section develops this constrained formulation.

3 Constrained Bethe free energy

The constrained Bethe formulation fixes model-consistent variational factors and imposes an information floor tied to goal-prior entropy, leaving policy beliefs as the main free variables. Its KKT stationary policy exactly recovers expected-free-energy control when the information multiplier is γ_k = 1, while shared-horizon relaxation enables a one-dimensional solve.

  • Constraints: Four constraint families enforce normalisation, marginalisation consistency, model-matched form constraints, and sufficient expected information gain.The form constraints fix current-state and parameter beliefs and make transition and likelihood factors match the generative model.
  • Information constraint: The information floor requires predicted observations to provide joint state-parameter information at least equal to goal-prior entropy, addressing novelty and salience.Because it is evaluated under the form constraints, the information demand is a fixed function of the policy and is derived from preferences rather than hand-tuned.
  • Bethe collapse: Under the form constraints, the Bethe free energy collapses to a functional of the policy belief because transition and likelihood energy terms cancel their entropy terms.Only the policy prior and goal factors survive after conditioning on a policy rollout.
  • Stationary policy: γ_k = 1 makes every stationary constrained-Bethe policy coincide exactly with the expected-free-energy policy.In general, γ_k is solved implicitly by the information constraints and complementary slackness rather than tuned as a hyperparameter.
  • Horizon relaxation: A shared horizon multiplier γ replaces the K-dimensional root-find with a 1-dimensional one, but relaxes per-step floors because information can be unevenly distributed across steps.The shared-multiplier policy family still contains the expected-free-energy policy at γ = 1.
  • KKT regimes: If the information constraint is already satisfied at γ = 0, it is inactive and the agent has no epistemic drive; otherwise γ binds at an interior value up to γmax.The experiments set γmax = 10^3, and expected information gain is monotonic in γ within the stationary-policy exponential family.

4 Experiments

Across a T-maze, cue-grid, and gate-cue task, the constrained Bethe agent is evaluated against standard EFE and Q-MDP. Results show systematic information seeking, three multiplier regimes, and adaptive allocation between cue salience and transition novelty.

  • Experimental setup: The experiments comprise a canonical T-maze, an information-gathering cue grid, and a gate-cue task with an unknown transition.All experiments use 200 trials from fixed seeds, except the Figure 3b floor sweep, which uses 100 trials per floor value.
  • T-maze: In the T-maze, the constrained agent visits the cue every trial and reaches the rewarding arm in the minimum two steps.Q-MDP reaches the cue almost as often but does not value its information and gambles on the wrong arm in 18% of trials.
  • Information-gathering grid: In the cue grid, the constrained agent visits the cue every trial and reaches the rewarding goal on 84.0±2.6% of trials.Horizon-induced dithering in the cue’s dead-end corridor causes residual failures, while EFE never visits the cue and gambles at a nearby goal.
  • Information-floor sweep: Varying the information floor exposes inactive, interior, and saturated Karush–Kuhn–Tucker regimes for the solved multiplier.The multiplier is zero when the floor is met, crosses the expected-free-energy value γ = 1 at a unique interior root, and saturates beyond the most informative rollout.
  • Gate-cue learning: In the gate-cue task, novelty initially drives probing, then effort shifts to cue seeking as the Dirichlet posterior concentrates.The agent probes every time for the first 29 trials, with 34% correct, then visits the cue on every subsequent trial; gate-belief error falls from 0.83 to 0.18.
  • Gate-cue learning: A novelty-equipped EFE planner makes the same novelty-to-salience shift earlier, achieving 95.5% correct overall versus 90.5% for the saturated agent.The salience-only control is always correct, the reset-posterior agent probes indefinitely, and Q-MDP performs at chance (49%).

5 Discussion

The discussion contrasts fixed-weight epistemic priors in EFE planning with an inequality-constraint formulation whose dual determines whether the epistemic drive switches off, binds, or saturates. This preserves an ordinary variational-free-energy framing while making the epistemic drive a solved constraint mechanism.

  • Epistemic priors: EFE planning can be formulated as variational inference using a model augmented with preference and epistemic priors.Epistemic priors encode information seeking as an additional model factor.
  • Epistemic priors: Fixed-weight epistemic priors preserve an ordinary variational free-energy objective by incorporating the epistemic drive into the generative model.This formulation treats the epistemic drive as part of the model rather than as a separately solved constraint.
  • Solved dual: The proposed inequality constraint has a solved dual, so complementary slackness determines whether the epistemic drive switches off, binds, or saturates.These are the drive’s inactive, binding, and saturated regimes.

6 Conclusion

The conclusion frames action selection as constrained Bethe free-energy minimisation, with a solved information multiplier yielding a policy family that contains the expected-free-energy policy. The constrained Bethe agent matches a standard expected-free-energy planner to machine precision on the T-maze.

  • Action selection is posed as constrained minimisation of a Bethe free energy on a goal-augmented planning graph.
  • A single per-step information constraint and solved Karush–Kuhn–Tucker multiplier yield a one-parameter family of policies.The multiplier is solved rather than tuned.
  • At unit multiplier, the policy family contains the expected-free-energy policy exactly.
  • The constrained Bethe agent matches a standard expected-free-energy planner to machine precision on the T-maze.

A Appendix: proofs of theoretical results

This appendix defines the planning graph’s clusters and edges, then shows how form constraints restrict the constrained family. Under these restrictions, the policy belief q(uτ) is the only free factor, with all other beliefs determined as marginals.

  • Planning graph and constrained family: The planning graph comprises cluster nodes V and edges E representing shared variables.Composite clusters emit transition x_k, emission y_k, control u_k, or goal g_k variables; singleton prior clusters are named by their variables.
  • Planning graph and constrained family: Fixing q(x_t), q(B), and q(A) imposes structured mean-field factorisation while holding parameter beliefs at past-graph values.Joining control clusters into q(uτ) selects a policy region, and pinning rollout conditionals adds a form constraint.
  • Planning graph and constrained family: The resulting constrained family has q(uτ) as its only free factor, with all cluster and edge beliefs given by its corresponding marginals.The restrictions jointly determine the remaining beliefs from the policy belief.

A.1 Proof of Lemma 1

The proof shows that fixing parameter beliefs severs all cycles in the factorisation, making the Bethe entropy exact on the resulting tree. Substitution then cancels matching factors and recovers the variational free energy expression in (12), up to model normalisation.

  • A.1 Proof of Lemma 1: Fixing parameter beliefs severs the parameter-sharing cycles, so family (20) factorises along a tree conditional on uτ.The equality chains enter only through fixed q(B) and q(A).
  • A.1 Proof of Lemma 1: Because the factorisation is a tree, the Bethe entropy decomposition is exact, and the Bethe free energy equals the joint variational free energy.This equality is evaluated on the marginals of (20), with controls collected in the policy cluster.
  • A.1 Proof of Lemma 1: Substituting (20) and the goal-augmented model (1) cancels every factor against its counterpart in p, including priors and pinned transition and emission conditionals.The prior cancellations involve p(xt | Dt), p(B | Dt), and p(A | Dt), while transition and emission terms cancel pairwise.
  • A.1 Proof of Lemma 1: Taking the expectation under (20) yields equation (12), up to the additive normalisation constant of p.This completes the proof of the lemma.

A.2 Proof of Theorem 1

Theorem 1 follows by varying the constrained Lagrangian with respect to the policy distribution, normalising the resulting solution, and deriving control marginals and dual feasibility from the constraints.

  • Form constraints make the Lagrangian linear in q(uτ) with fixed coefficients.
  • Setting the policy variation to zero and enforcing normalisation yields equation (14).
  • Control marginals are obtained by marginalising the policy cluster.
  • The dual signs and feasibility conditions γk ≥0 are the Karush–Kuhn–Tucker conditions for inequalities (11).Lemma 2 resolves these conditions.

A.3 Proof of Proposition 1

The proof characterizes the shared information multiplier through a horizon-level constraint and establishes a three-regime KKT trichotomy. Convexity and strict monotonicity of expected information gain guarantee a unique interior multiplier, while excessive information demands drive saturation toward maximally informative rollouts.

  • Determining the dual: A single multiplier γ aggregates per-step information constraints across the planning horizon while preserving the KKT structure.The resulting per-step stationarity system has a covariance-matrix Jacobian, making ln Z convex and its gradient monotone.
  • Information multiplier: If β ≤ g(0), the constraint is inactive, γ = 0, and the agent carries no epistemic drive.This is the slack-constraint regime of the KKT trichotomy.
  • Information multiplier: If g(0) < β < Smax, strict monotonicity makes g(γ) = β have exactly one root with γ > 0.Strictness follows because dg/dγ is positive unless S is constant on the support of the base measure.
  • Information multiplier: If β ≥ Smax, no finite multiplier can satisfy the information floor, so γ saturates at γmax and concentrates the policy on the most informative rollouts.In this regime the feasible set is empty and no finite KKT point exists.
  • Information multiplier: The expected information gain g(γ) is non-decreasing, starts at g(0) = E¯p[S], and approaches Smax as γ tends to infinity.The limiting policy concentrates on rollouts attaining the maximum information value, although Smax is not reached at finite γ.
Loading 2608.17167v1…