PaperlayerView exact revision on arXiv

Source-linked AI summary

Active inference on discrete state-spaces: a synthesis

Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, Karl Friston

arXiv:2001.07203v2q-bio.NC

TL;DR

Active inference has developed rapidly, making its relation between normative principles, process theories, and practical implementation difficult to retain. This paper synthesizes active inference for discrete state-space models, derives neuronal dynamics from first principles, and relates them to biological processes; it concludes that the resulting process theory has theoretical and face validity, while requiring rigorous empirical validation.

  • Problem

    Recent theoretical advances have made it difficult to maintain a comprehensive overview of active inference’s process theory and practical implementation.

  • Method

    The paper mathematically synthesizes active inference on discrete state-space generative models, deriving its process theory and neuronal dynamics from first principles.

  • Results

    The process theory’s neuronal dynamics correspond to known empirical phenomena, extend predictive-coding theories, and produce electrophysiological responses resembling empirical measurements.

  • Takeaways & Limitations

    The synthesis provides a practical guide for simulating experimental behaviour and a pointer toward in-silico neurophysiological predictions that can be tested empirically.

  • Takeaways & Limitations

    A complete set of mechanisms by which biological agents perform structure learning has not yet been laid out.

Abstract

from arXiv · show

Active inference is a normative principle underwriting perception, action, planning, decision-making and learning in biological or artificial agents. From its inception, its associated process theory has grown to incorporate complex generative models, enabling simulation of a wide range of complex behaviours. Due to successive developments in active inference, it is often difficult to see how its underlying principle relates to process theories and practical implementation. In this paper, we try to bridge this gap by providing a complete mathematical synthesis of active inference on discrete state-space models. This technical summary provides an overview of the theory, derives neuronal dynamics from first principles and relates this dynamics to biological processes. Furthermore, this paper provides a fundamental building block needed to understand active inference for mixed generative models; allowing continuous sensations to inform discrete representations. This paper may be used as follows: to guide research towards outstanding challenges, a practical guide on how to implement active inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions.

1 Introduction

Active inference is presented as a normative framework for perception, action, planning, decision-making, and learning, with discrete state-space models addressing processes naturally represented as discrete alternatives. The paper synthesizes the theory, derives neuronal dynamics, and connects implementation to biological processes and empirical predictions.

  • 1 Introduction: Active inference uses variational free energy to fit internal models to past sensations and expected free energy to evaluate future actions against preferences.An internal forward generative model predicts sensory input and supports inference about its causes.
  • 1 Introduction: Earlier formulations used continuous random dynamical systems, whereas many neural and decision-making processes are better framed with discrete hierarchical representations.Examples include visual working memory, place-cell state estimation, language, and tasks involving discrete alternatives.
  • 1 Introduction: Active inference compares favourably to reinforcement learning in the game of Doom.
  • 1 Introduction: The paper provides a mathematical synthesis of discrete state-space active inference, derives neuronal dynamics from first principles, and relates them to biological processes.It is also positioned as a practical implementation guide and a source of in-silico neurophysiological predictions.
  • 1 Introduction: The process theory formalizes an action-perception cycle in which agents infer latent causes, plan preferred future actions, act, and learn contingencies through synaptic plasticity.The cycle repeats across perception, planning, decision-making, action, and slower-timescale learning.

2 Active inference

Active inference explains perception and action through complementary minimization of variational and expected free energy. Its first-principles account rests on generative models, statistical-physics assumptions, and a process theory linking beliefs to biological dynamics.

  • 2 Active inference: Variational free energy measures the fit between an internal generative model and sensory observations, while expected free energy evaluates actions by their ability to reach preferred states.Preferences are states an agent believes it is likely to work toward.
  • 2 Active inference: The generative model is a joint probability distribution over hidden states and sensory consequences, allowing inference of likely hidden causes from sensations.Beliefs about hidden states are updated so they closely match the causes of observed outcomes.
  • 2 Active inference: Minimizing variational free energy improves the generative model’s predictions, enabling expected-free-energy planning that realizes preferences and reduces expected future surprise.
  • 2 Active inference: Active inference is grounded in statistical physics under non-equilibrium steady-state and Markov-blanket assumptions, under which internal states can parameterize Bayesian beliefs about external states.
  • 2 Active inference: The paper distinguishes active inference as a principle derived from assumptions about living systems from process theories hypothesized to implement it computationally and biologically.These process theories generate predictions about neuronal dynamics and experimentally elicited electrophysiological responses.

3 Discrete state-space generative models

The paper represents discrete-time decision problems with finite state-space generative models, typically formulated as POMDPs. These models encode initial-state beliefs, state transitions, state–outcome likelihoods, policies, and approximate posterior beliefs.

  • Model structure: A discrete generative model is a joint probability distribution over hidden states, outcomes, and variables causing outcomes.States unfold over time and generate observations; the likelihood matrix A encodes state–outcome probabilities.
  • Model structure: The simplest model studied is a POMDP with initial-state beliefs D, transition matrix B, and likelihood matrix A.This model supports later generalisations to Markov decision processes and hierarchical probabilistic models.
  • Notation: Policies π are finite sequences of actions, while S and O denote finite sets of possible hidden states and outcomes.The notation distinguishes random variables from the possible values they can take.
  • Motivation: The framework is motivated by discrete, hierarchical representations and sequential computation in processes including working memory, place-cell estimation, language, attention, and perception.The paper derives its process theory first for the simplest tractable discrete model before discussing more complex extensions.
  • Inference and learning: Active inference operates across timescales by inferring hidden states and policies quickly while learning environmental contingencies more slowly.The slower parameters include contingencies such as the likelihood matrix A.

4 Variational Bayesian inference

Variational Bayesian inference makes posterior estimation tractable by replacing an intractable exact posterior with an optimised approximate posterior. Minimising variational free energy jointly improves inference about latent causes and model evidence, using a structured mean-field factorisation for exposition.

  • Posterior inference: Exact posterior computation is intractable for complex generative models, so the paper optimises an approximate posterior by minimising KL divergence.The KL divergence measures discrepancy between probability distributions and provides the variational route to inference.
  • Free energy: Minimising variational free energy approximates the true posterior while keeping sensory surprise low and maximising evidence for the generative model.This connects approximate Bayesian inference with active inference’s objective.
  • Free energy: Variational free energy decomposes into complexity and accuracy, balancing deviation from prior beliefs against fit to observed data.The complexity term favours explanations requiring minimal updating from the prior, whereas accuracy measures how well beliefs explain observations.
  • Approximate posteriors: The paper uses a structured mean-field approximation to obtain tractable free-energy expressions and derive neuronal dynamics.Different posterior factorisations trade inference quality against computational complexity.
  • Approximate posteriors: The chosen factorisation is primarily didactic, while the marginal approximation is described as more biologically plausible and closer to accurate Bethe-style inference.The paper notes that the cited software implementation uses the marginal approximation.

5 Perception

Perception is formulated as policy-dependent hidden-state estimation through variational free-energy minimisation. The resulting gradient-descent and softmax dynamics have Bayesian message-passing interpretations and can be mapped onto neuronal population activity.

  • State estimation: Perception is state estimation: for each policy, the agent minimises variational free energy to infer past, present, and future hidden states.The resulting posterior is contingent on pursuing the given policy.
  • Neuronal dynamics: Neuronal dynamics perform gradient descent on free energy, with state estimates represented as a softmax of accumulated negative free-energy gradients.The gradient can therefore be interpreted as a prediction error.
  • Neuronal dynamics: The softmax maps unconstrained activity variables to state probabilities whose components sum to one.This makes it a natural choice for categorical state estimation.
  • Temporal interpretation: The dynamics are faster than observation sampling and are linked to behaviour-relevant processing in working memory and visual perception.They are also consistent with predictive processing, while using a different generative-model-based prediction error than predictive coding schemes.
  • Biological interpretation: Interpreting latent activity as membrane potentials and softmax state estimates as firing rates supports simulated responses including place-cell activity, mismatch negativity, and evidence accumulation.The paper lists several additional electrophysiological phenomena, including phase precession and theta–gamma coupling.
  • Message passing: Under the mean-field approximation, state-estimation dynamics coincide with variational message passing; under Bethe approximation, they coincide with belief propagation.This formally connects active inference with established approximate-inference algorithms.

6 Planning, decision-making and action selection

Planning and decision-making arise by evaluating expected free energy across policies, while action selection follows posterior beliefs about those policies. This framework balances preference-directed risk minimization with ambiguity reduction and supports biologically plausible decision dynamics.

  • Planning and decision-making: Expected free energy scores possible future courses of action relative to prior preferences, yielding a posterior distribution over policies.Policies are selected through inference over predicted future states and preferences.
  • Planning and decision-making: The most likely policies minimise expected free energy, balancing exploitative risk minimization with explorative ambiguity minimization.Risk concerns conformity with preferred states, whereas ambiguity concerns uncertainty about predicted outcomes.
  • Action selection: Approximate posterior beliefs over policies support Bayesian model averaging to infer likely actions and policy-independent hidden states.These averages may be implemented through neuromodulatory mechanisms.
  • Action selection: The policy softmax provides a smooth, biologically plausible approximation to winner-take-all decision architectures.This links active-inference policy selection to established computational-neuroscience models of selective attention and recognition.
  • Computational cost: Planning can become computationally prohibitive because the number of action sequences grows combinatorially with deeper futures.Occam-window pruning helps but cannot handle large policy spaces from deep trees and long temporal horizons; hierarchical models offer one proposed solution.
  • Action-perception cycle: Variational and expected free-energy minimization are complementary: accurate generative models support effective future planning and preference realization.The action-perception cycle repeatedly infers latent causes, plans, and realizes preferred actions.

7 Properties of the expected free energy

Expected free energy can be decomposed into risk, ambiguity, information gain, and evidence-related terms. These decompositions connect policy selection to preference alignment, uncertainty reduction, and established ideas from psychology, economics, and engineering.

  • Overview: Expected free energy is introduced as a fundamental construct connecting active inference with theories in neuroscience and engineering.The section unpacks its principal mathematical and conceptual features.
  • Risk and ambiguity: Policy selection minimises risk and ambiguity: risk aligns predicted states with preferred states, while ambiguity concerns uncertainty about future outcomes.The risk component relates to KL control and risk-sensitive policies.
  • Information-theoretic decomposition: Expected free energy combines expected log evidence with expected information gain, linking uncertainty resolution about outcomes and hidden states to prior beliefs.The evidence term is evaluated under predicted future outcomes, while the information-gain term makes that evaluation informed by encountered outcomes.
  • Outcome-based risk: When preferences are outcome-based, state risk can be replaced by outcome risk under the stated small-divergence approximation.This formulation is useful when the generative model is unknown or during structure learning.
  • Extrinsic and intrinsic value: Extrinsic value corresponds to expected log evidence, whereas intrinsic value is the expected information gain about hidden states or model parameters.These correspond respectively to reward or utility, and to salience or novelty across related literatures.

8 Learning

Learning in discrete-state active inference treats synaptic weights as beliefs about environmental contingencies and updates them through variational free-energy gradients. The resulting Dirichlet accumulation becomes increasingly confident under stable environmental structure and is formally identical to associative or Hebbian plasticity.

  • Synaptic learning: Active-inference learning models synaptic plasticity as updating beliefs about environmental contingencies such as likelihood and transition matrices.Synaptic weights can perform matrix multiplication to predict outcomes from beliefs about hidden states.
  • Conceptual synthesis: Figure 4 relates variational and expected free energy to planning, perception, information gain, risk, ambiguity, and preference-directed action.Its special-case decompositions connect expected free energy with established constructs across psychology, machine learning, economics, and engineering.
  • Timescales: The learning dynamics operate more slowly than perceptual inference because they accumulate evidence across multiple state-outcome pairs.For simulations, this supports a one-step belief update at the end of each observation trial.
  • Variational derivation: The synaptic-plasticity dynamics follow gradient descent on variational free energy with respect to the likelihood parameters.The derivation uses the KL divergence between Dirichlet distributions and an outer-product representation.
  • Belief updating: Updating Dirichlet parameters counts observed mappings between hidden states and observations, producing a rule formally identical to associative or Hebbian plasticity.Prior beliefs are replaced by approximate posterior beliefs for the subsequent trial.
  • Environmental stability: Repeated observations increase confidence in the likelihood matrix while the environment remains relatively constant.Bayesian model reduction is proposed to reverse this accumulation and support adaptation to changing environments.

9 Structure learning

This section treats structure learning as learning the variables and dependencies of a generative model, using Bayesian model comparison, reduction, and expansion to adapt model complexity.

  • Structure learning concerns learning the variables and functional dependencies of an agent’s generative model.
  • The paper presents Bayesian model reduction and expansion as complementary approaches, while noting that biological structure-learning mechanisms remain incomplete.
  • Bayesian model comparison scores competing generative models against observations and selects the model with the greatest evidence.
  • Bayesian model reduction: Bayesian model reduction compares a full model with a simpler prior and removes redundant parameters when the reduced model has more evidence.
  • Bayesian model expansion: Bayesian model expansion adds states when accuracy gains sufficiently outweigh increased complexity, supporting generalisation and concept learning.

10 Discussion

The discussion reconnects active inference’s underlying principle with its process theory, neuronal dynamics, implementation, and outstanding challenges. It emphasizes empirical validation, generative-model identification, model extensions, and scalability limits.

  • The paper clarifies active inference by relating its assumptions, discrete-state process theory, and accompanying neuronal dynamics to biological interpretation.This synthesis addresses the difficulty of connecting recent theoretical advances with practical implementation.
  • The proposed neuronal dynamics are presented as compatible with known empirical phenomena, extensions of predictive coding, and the free energy principle.The principle is framed as avoidance of surprising states.
  • Rigorous empirical validation of synthetic electrophysiological responses requires specifying biological agents’ generative models and comparing alternatives against measured choice behaviour.Bayesian model comparison is identified as one route to selecting an appropriate model.
  • Finding the generative model that best explains observable data is identified as a key challenge for accurately simulating belief updating and behaviour in silico.The paper notes that models may also be specified manually for particular purposes.
  • Complex behavioural data may require extending active inference beyond currently implemented discrete, continuous, and mixed state-space models to forms such as Markov decision trees and Boltzmann machines.New generative models would need corresponding process theories that produce testable predictions and reproduce observed behaviour in silico.
  • Scaling active inference to many degrees of freedom may make policy-tree search intractable because the current pruning strategy is unlikely to reduce the search space sufficiently.The discussion links this issue to understanding how variational free energy principles might support scaling and how biological agents search deep policy trees.

11 Conclusion

The conclusion presents the paper as a synthesis of active inference assumptions, discrete-state process theory, and biologically interpreted neuronal dynamics. It also identifies challenges in empirical model selection and scaling behaviour with many degrees of freedom.

  • The paper summarizes active inference assumptions, process-theory details, and the relationship between neuronal dynamics and biological processes.These processes are illustrated under discrete state-space generative models and cover action, perception, planning, decision-making, learning, and structure learning.
  • Outstanding challenges include identifying process theories consistent with active inference, refining them through Bayesian model comparison, and scaling inference to behaviour with many degrees of freedom.The conclusion frames these as theoretical-neuroscience challenges.

Software availability

The supplementary material describes implementation routes for discrete-state belief updating and extensions to richer generative models. It covers learned parameters, policy confidence, factorized states, and hierarchical timescales.

  • Software availability: The generic belief-updating scheme can be implemented with standard routines in SPM’s Matlab software, with simulation examples accessible through its graphical interface.The cited routines include spm_MDP_VB_X.m, and examples can be accessed by typing DEM.
  • Extended generative models: More complex discrete-state models extend belief updating by allowing transition probabilities and initial-state beliefs to be learned over time.The paper’s main treatment learned A while keeping B and D fixed; the extension introduces priors over B and D.
  • Policy selection: Policy posteriors can include an inverse-temperature parameter γ, interpreted as confidence in policy selection and linked to postsynaptic responses to dopaminergic input.This extension is presented as enabling simulation of biologically plausible dopaminergic effects.
  • Factorized models: Multiple hidden factors represent independently evolving state streams that interact to generate outcomes, supporting integration of sensory information from distinct senses.The paper refers to graphical representations of these factorized models.
  • Deep temporal models: Deep temporal models nest generative-model layers hierarchically across different timescales, with lower-level outcomes serving as inputs to higher-level models.Belief updating proceeds at lower levels conditioned on higher-level distributions, followed by higher-level updating that treats lower levels as outcomes.

B Expected free energy

Expected free energy links steady-state active inference to policy selection by relating predictive distributions to preferences and outcome uncertainty. The framework also decomposes this objective into risk, ambiguity, and novelty-related terms, with extensions to multiple future time-steps.

  • Steady-state formulation: Active inference considers systems that self-organise to general steady-states, where policies generate final states through conditional dependencies involving hidden-state Markov blankets.General steady-states allow probabilistic trajectories rather than a unique path from initial to final states.
  • Objective function: The Gibbs energy G(π; β) combines KL divergence from preferred state and parameter distributions with a likelihood-based outcome term weighted by β.At β = 1, Gibbs energy becomes expected free energy.
  • Steady-state criterion: Policy surprisal and Gibbs energy are equal on average under Q if and only if the system reaches steady-state.The derivation rewrites expected Gibbs energy as policy surprisal plus a KL divergence between aggregate predictive and preferred distributions.
  • Precision parameter: When β > 0, policies favour lower outcome entropy conditional on states, producing itinerant behaviour; β = 0 corresponds to KL or risk-sensitive control.KL control can also arise when the likelihood entropy is constant across policies.
  • Active-inference consequence: For systems under NESS dynamics and a Markov blanket, the most likely trajectories minimise expected free energy, corresponding to β = 1.General steady-states are associated with Bayes-optimal exploration and exploitation under the generative model’s negative-log-evidence risk.
  • Decomposition and computation: The expected free-energy expression separates risk over states or outcomes, ambiguity from likelihood entropy, and novelty as expected information gain about model parameters.The appendix provides analytical expressions for these components and derives novelty using Dirichlet parameter updates and digamma-function identities.
Loading 2001.07203v2…