Sophisticated Inference

• Policy-based active inference requires specifying allowable policies in advance, limiting scalability because only a relatively small number can be evaluated.The paper proposes deep search over all allowable future action sequences.Source: Introduction · p. 2
• Sophistication means rolling out beliefs about beliefs, whereas naive inference considers action consequences but not how actions affect beliefs.The paper develops this distinction through mathematical and numerical analyses.Source: Introduction · p. 3
• Sophisticated inference recovers Bayes-adaptive reinforcement learning (Åström, 1965; Ghavamzadeh et al., 2016; Ross et al., 2008) in the zero-temperature limit.Unlike arbitrary reward functions, its expected free energy additionally mandates seeking information about the world beyond what a particular task requires.Source: Introduction · p. 3
• In the three-armed bandit, active inference favors the central arm to reduce loss risk and resolve uncertainty about which outer arm pays off.After learning which arm rewards, the agent can commit to it; the example contrasts with choices having equal expected Bayesian payoff.Source: Introduction · p. 4
• Recursively evaluating each move’s expected free energy lets an agent update beliefs under plausible outcomes and search deeply over ordered action-outcome sequences.The search proceeds over future belief states, not states alone.Source: Introduction · p. 5

Introduction

• Considering how actions shape subsequent beliefs and epistemic actions makes the agent more confident in its choices and can make the central arm more attractive.The central arm is favored because the average expected free energy of its subsequent actions is low.Source: Introduction · p. 5
• The paper outlines active inference, develops recursive expected-free-energy planning, and illustrates sophisticated versus unsophisticated agents in bandit, T-maze, navigation, and novelty-seeking problems.The introduction identifies the bandit or T-maze comparison as illustrative simulations and navigation and novelty seeking as examples of deeper problems.Source: Introduction · pp. 5–6
• Recursive expected free energy enables efficient search through deep policy trees by evaluating only paths with sufficiently high predictive posterior probability.The restricted search propagates beliefs—probabilities—rather than value functions.Source: Introduction · p. 6

Active inference and free energy minimisation

• Policies are treated as random variables, so planning optimizes posterior beliefs over action sequences and selects an action from the most likely policy.The approach also optimizes beliefs over hidden states using an evidence bound.Source: Active inference and free energy… · p. 6

Discrete state-space models

• Generalised free energy combines each policy’s variational free energy with its expected free energy, which determines policy priors and is minimized at the policy horizon.The variational component scores evidence from observed outcomes; the expected component sets priors over policies.Source: Discrete state-space models · p. 8

Perception as inference

• Optimizing beliefs over hidden states makes variational free energy equivalent to negative log evidence; mean-field variational Bayes can approximate these beliefs through coordinate descent.An alternative minimizes variational free energy using iterated gradient flows.Source: Perception as inference · pp. 8–9
• Amortized inference uses a learned recognition model to map observations to approximate-posterior sufficient statistics, but precludes online inference and may seem biologically implausible.The text allows that some brain structures might learn to infer from processes implemented elsewhere.Source: Perception as inference · pp. 9–10

Planning as inference

• The policy posterior is a softmax over empirical, variational, and expected free energy; expected free energy can be decomposed into risk and ambiguity.The risk-and-ambiguity formulation is presented as an intuitive way to unpack expected free energy.Source: Planning as inference · p. 11
• Policy selection minimizes risk and ambiguity: risk favors states conforming to prior preferences, while ambiguity reduction favors informative, unambiguous outcomes.Risk can be expressed over outcomes when the generative model is unknown or preferences are specified only over outcomes.Source: Planning as inference · p. 11
• Expected free energy evaluates policies by combining extrinsic value for preferred outcomes with intrinsic value from information gain about hidden states or model parameters.Extrinsic value is expected log prior preference; intrinsic value is expected information gain, also described as salience for hidden states and novelty for parameters.Source: Planning as inference · p. 12
• Expected free energy encompasses information gain, risk-sensitive control, expected utility, and Bayesian decision theory as special cases under different uncertainty and preference assumptions.Removing prior preferences yields information gain; removing risk while retaining preferences yields risk-sensitive policies or KL control; removing intrinsic value yields expected utility.Source: Planning as inference · p. 14
• Relaxing fixed action-sequence priors to conditionally independent actions simplifies planning through Markovian belief propagation, at the cost of considering potentially many action sequences.The described scheme updates beliefs over hidden states and actions from each new observation and is exact because it makes no mean-field approximations.Source: Planning as inference · pp. 14–15
• The recursive formulation performs a deep search over plausible action–outcome paths, terminating branches when an action or outcome is unlikely.Belief propagation restricts evaluation to paths with nontrivial probability; the schematic depicts this branching search and its termination.Source: Planning as inference · pp. 15, 16, 18
• The search evaluates distributions over outcomes conditioned on plausible hidden states, rather than enumerating outcomes themselves.This simplification relies on having a generative model that generates outcomes.Source: Planning as inference · p. 16
• Each next action’s expected free energy combines its risk and ambiguity with the average expected free energy of future beliefs under counterfactual outcomes and actions.The empirical prior over the next action is formed recursively by averaging expected free energy from the node’s children and applying a softmax.Source: Planning as inference · pp. 16–17
• The recursion operates on functionals of belief distributions over states, whereas the Bellman equation operates directly on functions of states.Source: Planning as inference · p. 17

Exploration and exploitation in a T-maze

• A sophisticated agent chose a confident epistemic policy, then switched to exploiting the left arm once it learned reward was always there.The switch occurred when the expected utility of going directly to the baited arm exceeded the cue’s epistemic value.Source: Simulations · p. 21, Exploration and exploitation in a T-maze · p. 22
• The unsophisticated agent was less confident and, despite finding the cue attractive, switched to exploitation a few trials earlier by underestimating its epistemic value.It also tended to remain at the center, avoiding aversive outcomes.Source: Exploration and exploitation in a T-maze · p. 22
• Without epistemic value, the Bayesian-risk agent exposed itself to risk by foregoing arm visits, then exploited the left arm with relatively low confidence after learning its location.The starting and cue locations were equally attractive, and the agent lacked a sense that choosing the cue was the right kind of behaviour.Source: Exploration and exploitation in a T-maze · pp. 22–23
• With planning horizon two, the rat switches from exploratory to exploitative behavior at trial 16; with horizon one, ambiguity about early actions yields the switch slightly earlier, at trial 10.Under Bayesian risk, the rat sometimes stays at the central location and learns nothing about the prevailing context.Source: Exploration and exploitation in a T-maze · p. 24
• When the instructional cue costs one, the depth-two sophisticated agent still visits it and switches to exploitation at trial 12, while the one-step agent stays put and forgoes rewards.The sophisticated agent’s exploratory phase is slightly shortened; the unsophisticated agent avoids the aversive cue but learns nothing about the maze.Source: Exploration and exploitation in a T-maze · pp. 24–25

Deep planning and navigation

• With a planning horizon of four or more, the rat escapes local minima and follows the shortest path to its target; shorter horizons leave it stuck beyond nearby aversive barriers.The rat knew the aversive locations, and its goal-seeking path could include excursions away from the target.Source: Deep planning and navigation · pp. 27–28
• With preferences for a particular location removed, the rat makes 64 consecutive moves and nearly every location is explored as previously visited locations lose novelty; Bayesian risk instead returns it home after the first move.It then sits at its original location for 64 trials.Source: Deep planning and navigation · pp. 28–29
• With only novelty incentives, the agent explored nearly every location in 64 moves while avoiding revisits; the Bayesian-risk agent returned to its safe starting location and learned no location-outcome mapping.The novelty-driven agent updated its beliefs about aversive outcomes after each move.Source: Deep planning and navigation · p. 29, Figure 6 · p. 29
• After four maze exposures, the agent learned enough about aversive locations to plan and execute the shortest path to its preferred target.With location preferences restored, exploration focused on locations sufficient to map the route to the goal.Source: Deep planning and navigation · p. 30
• The numerical analyses show that expected free energy supplements Bayesian risk with epistemic affordance through salience about hidden states or novelty about hidden model parameters.The simulations illustrate novelty-driven learning about aversive locations and preference-guided exploration for navigation.Source: Deep planning and navigation · p. 30

Conclusion

• Recursive expected free energy instigates a deep tree search for planning as inference, accumulating expected free energies along plausible future paths.Source: Conclusion · p. 31
• Sophistication means beliefs about beliefs, formed by accumulating predictive posterior representations of time-sensitive hidden states and their uncertainty.Source: Conclusion · p. 31
• The search propagates beliefs about the consequences of successive actions, evaluating future belief paths rather than relying on sampling or random searches.Source: Conclusion · pp. 31–32

Limitations

• The scheme is illustrated only on a simple T-maze and navigation problem, while continuous real-world states and actions pose a scaling challenge for discrete-state models.Source: Conclusion · p. 31, Limitations · p. 33

Conclusion

• The navigation example solves a learning problem of the kind usually addressed by reinforcement learning, from variational principles without backward induction or belief-free schemes such as Q-learning.Source: Conclusion · p. 31
• 1.0486·10^10 horizon-four and 1.0737·10^15 horizon-six potential paths yielded only a tiny fraction evaluated in the navigation example.The paper says several hundred paths are usually evaluated, taking a few hundred milliseconds on a personal computer.Source: Conclusion · p. 32

Limitations

• The state-action factorization assumes every action is allowed from every state, a strong assumption the authors consider apt for many applications.Source: Limitations · p. 32
• A hierarchical extension could model slower higher-level transitions and optimize trajectories within each level using the sophisticated scheme.Source: Limitations · p. 33

Bayes optimality

• Generalized free energy upper-bounds Bayesian risk under a generative model whose policy priors lead to states with precise likelihoods.Bayesian risk is the divergence between predictive outcomes and prior preferences.Source: Appendix · p. 34
• Minimizing generalized free energy minimizes policy complexity, aligns predictive hidden-state posteriors with the generative model, and makes policies and outcomes independent.The resulting independence means final outcomes do not depend on initial conditions.Source: Appendix · p. 35

Expected free energy

• Expected free energy for a policy upper-bounds a mixture of expected information gain and Bayesian risk.The appendix expresses free energy over policies in terms of expected free energy before deriving this bound.Source: Appendix · pp. 35–36

Nonequilibrium steady-state

• Conditional independencies separating policies from outcomes via hidden states permit a family of steady-state solutions, with the Bayes-optimality solution as a canonical case.At steady state, the final distribution does not depend on initial observations.Source: Appendix · p. 36
• When policy surprisal corresponds to Gibbs free energy, the final distribution attains steady state; at β = 1, Gibbs free energy becomes expected free energy, while at β = 0 the steady state minimizes divergence between predictive and target distributions.Source: Appendix · pp. 36–38

Active inference

• General steady-state agents appear Bayes optimal in exploration and exploitation, with Bayesian risk defined by the generative model’s negative log evidence.Such agents appear to respond to epistemic affordances.Source: Appendix · p. 39

Empowerment

• Empowerment minimizes expected ambiguity while maximizing policy entropy, keeping options open; it is a special case of active inference when all policies are equally risky.Empowerment is the mutual information between policies and final states, given initial observations.Source: Appendix · p. 39

Information bottleneck

• The information bottleneck combines compression of initial outcomes into hidden states with information gain about final outcomes, and at relative precision 1 is an upper bound on expected information gain.Its average over initial observations prevents direct use for online planning as inference, though it can learn fixed outcome-action policies; it is a special case of active inference when Bayesian risk is ignored.Source: Appendix · p. 40

Self-organisation

• When expected free energy averaged over policies is small, predictive hidden states converge to prior or preferred distributions and consequent outcome uncertainty becomes small.In the limit, hidden-state predictions equal the prior and outcomes have no uncertainty.Source: Appendix · p. 41

Self-evidencing

• When expected free energy averaged over policies is small, expected log evidence and mutual information between predicted states and their outcomes are large.In the limit, expected log evidence is maximized and outcomes are unambiguous given hidden states.Source: Appendix · p. 41

Appendix

• Expected free energy has complementary decompositions in terms of risk and ambiguity, or information gain and entropy.Both formulations tend to increase mutual information between hidden states and outputs while minimizing entropy or Bayesian risk.Source: Appendix · p. 42
• Hidden states form a Markov blanket between policies and outcomes; at steady state, risk becomes mutual information between policies and hidden states.The diagrams assume steady state and omit dependencies on initial observations.Source: Appendix · p. 42
• The diagrams present Bayesian decision theory, including KL control and Bayesian risk, and optimal Bayesian design as complementary imperatives.Figure 1 highlights components involved in special cases such as optimal Bayesian decisions and design.Source: Appendix · p. 42
• Figure 8 clarifies relationships among conditional entropy and mutual information terms in the free energy functional.It presents complementary decompositions in terms of risk and ambiguity, and information gain and entropy.Source: Appendix · p. 42

Software note

• The belief updates are generic and can be implemented using standard routines.The routines are available as Matlab code in the SPM academic software.Source: Software note · p. 43
• The simulations can be reproduced and customized through a graphical interface by selecting the T-maze or Navigation demo.The interface is opened by typing >> DEM.Source: Software note · p. 43