PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep active inference agents using Monte-Carlo methods

Zafeirios Fountas, Noor Sajid, Pedro A. M. Mediano, Karl Friston

arXiv:2006.04176v2q-bio.NCcs.AIstat.ML

TL;DR

Deep active inference has largely been implemented in low-dimensional, discrete tasks, while some deep approaches approximate or reduce expected free energy. This paper introduces agents using multiple Monte-Carlo methods and reports disentangled representations and planning despite temporary visual loss.

  • Problem

    Active-inference implementations have largely been restricted to low-dimensional, discrete tasks, and the scalability of existing approaches to more complex domains remains unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper combines Monte-Carlo tree search, a habitual policy network, MC dropout, and precision modulation in a deep active-inference architecture.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    The precision factor enhances disentanglement, and Animal-AI agents simulate future plans despite temporary suspension of visual input.

    Source: §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The authors connect precision modulation with attention-like uncertainty shaping and more statistically independent latent features.

    Source: §5 · p. 9

    The paper does not establish whether full expected-free-energy agents outperform simpler reward-seeking or other model-based reinforcement-learning agents on reward-maximization tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Active inference is a Bayesian framework for understanding biological intelligence. The underlying theory brings together perception and action under one single imperative: minimizing free energy. However, despite its theoretical utility in explaining intelligence, computational implementations have been restricted to low-dimensional and idealized situations. In this paper, we present a neural architecture for building deep active inference agents operating in complex, continuous state-spaces using multiple forms of Monte-Carlo (MC) sampling. For this, we introduce a number of techniques, novel to active inference. These include: i) selecting free-energy-optimal policies via MC tree search, ii) approximating this optimal policy distribution via a feed-forward `habitual' network, iii) predicting future parameter belief updates using MC dropouts and, finally, iv) optimizing state transition precision (a high-end form of attention). Our approach enables agents to learn environmental dynamics efficiently, while maintaining task performance, in relation to reward-based counterparts. We illustrate this in a new toy environment, based on the dSprites data-set, and demonstrate that active inference agents automatically create disentangled representations that are apt for modeling state transitions. In a more complex Animal-AI environment, our agents (using the same neural architecture) are able to simulate future state transitions and actions (i.e., plan), to evince reward-directed navigation - despite temporary suspension of visual input. These results show that deep active inference - equipped with MC methods - provides a flexible framework to develop biologically-inspired intelligent agents, with applications in both machine learning and cognitive science.

1 Introduction

Active inference unifies perception and action under free-energy minimization, but implementations have largely remained limited to low-dimensional, discrete tasks. This paper proposes a deep architecture using Monte-Carlo methods to address planning and scalability challenges while retaining alignment with active inference.

1 Introduction

  • Active inference frames biological intelligence as perception and action jointly minimizing free energy across time, yet implementations have largely focused on low-dimensional, discrete state spaces.The framework is presented as a Bayesian account of biological intelligence.Source: §1 · p. 1
  • Prior deep agents have addressed continuous tasks, but some approximate or reduce expected free energy, and their scalability to more complex domains remains unclear.The proposed extension estimates all expected-free-energy summands with a single deep neural architecture.Source: §1 · p. 2
  • The architecture introduces Monte-Carlo tree search for free-energy-optimal policy selection, a habitual network to approximate the optimal policy distribution, MC dropout for parameter-belief updates, and precision modulation over state transitions.The authors frame these choices as supporting scalability and biological plausibility.Source: §1 · p. 2

2 Active Inference

Active inference agents infer a generative model from observations and choose actions to reduce uncertainty about the environment. The section connects surprise minimization to variational free energy and policy selection to expected free energy.

2 Active Inference

  • Agents sample the environment to calibrate a generative model that explains observations and act to reduce uncertainty about the environment.The model represents hidden states, observations, policies, and action-conditioned generative distributions.Source: §2 · p. 2
  • Current-observation surprise is addressed by optimizing model parameters with a tractable loss whose right-hand side is variational free energy.Variational free energy is also described as negative evidence lower bound.Source: §2 · p. 2
  • For future observations, agents select policies associated with the lowest expected free energy, then sample actions from a distribution over policies.The action-selection distribution uses a softmax function.Source: §2 · p. 2
  • Figure 1 maps the generative, recognition, and attention components, shows ancestral-sampling calculations for expected free energy, and depicts habitual-network-guided tree exploration.Its schematic distinguishes generative-model arrows, recognition-model arrows, and top-down attention by color.Source: Figure 1 · p. 3

3 Deep Active Inference Architecture

The architecture combines neural-network amortization and Monte-Carlo sampling to calculate expected free energy, select actions, and regulate transition-model precision. MCTS supports deliberative action selection, while a habitual network approximates the action distribution.

3 Deep Active Inference Architecture

  • The architecture parameterizes observation and transition models with θ_o and θ_s, and state and action amortization with φ_s and φ_a.The action amortization parameters belong to the habitual network.Source: §3 · p. 3

3.1 Calculating variational and expected free energy

  • The model computes variational free energy at each time step, while action probability sums the probabilities of policies beginning with that action.The derivation assumes normally distributed states and Bernoulli observations, with expectations over the encoder distribution estimated from one sample.Source: §3.1 · p. 3
  • Expected free energy includes desired-observation likelihood, state-uncertainty reduction, and parameter-uncertainty reduction, but two terms require Monte-Carlo reformulation for tractable computation.The reformulated terms use ancestral samples and MC dropout for neural-network parameters.Source: §3.1 · p. 4

3.2 Action selection and the habitual network

  • MCTS searches possible state-action trajectories to select actions using expected free energy, while a habitual network approximates the resulting action distribution.The habitual network is trained to reproduce the last executed action, selected by sampling the MCTS-derived distribution.Source: §3.2 · pp. 4–5

3.3 State precision and top-down attention

  • State-transition precision varies with policy-belief updates: it is high when posterior and prior policy beliefs are similar, acting as top-down attention over which transitions to learn precisely.The precision mechanism is described as incentivizing disentangled latent states and gradually increasing as the habitual network better approximates the action distribution.Source: §3.3 · p. 5

4 Results

The results evaluate the agent in dSprites and Animal-AI, covering environment learning, disentangled representations, planning, and comparison with model-free reinforcement learning. DAIMC performed better than DQN and A2C, matched PPO2, and planned during temporary visual-input loss.

4 Results

  • Evaluation used dSprites object sorting and Animal-AI preferences tasks, encoding reward as preferred future observations; Animal-AI also tested temporary visual-input loss.The dSprites task used shape-dependent rewards for moving objects across the bottom border.Source: §4 · pp. 5–6

4.1 Learning environment dynamics and task performance

  • Active-inference variants collected similar rewards, explored more, and reconstructed expected reward better than the reward-maximizing baseline; all predicted five steps ahead with loss similar to the baseline.The variants also reconstructed current observations.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Disentanglement and transition learning

  • Active-inference agents using transition precision ωt produced more disentangled latent representations, and their total-correlation ranking matched their reward-reconstruction ranking.The authors suggest disentanglement may help explain differences in reward prediction.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Navigation and planning in reward-based tasks

  • In dSprites, behavior shifted from epistemic exploration to reward seeking; after 700 iterations, agents planned reward-maximizing moves, with peak performance at Tdec ≈0.8.In Animal-AI, agents planned around obstacles and maintained simulated plans during temporary visual-input loss; performance worsened as the loss lasted longer, while shorter decision thresholds preserved it longer.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 Comparison with model-free reinforcement learning agents

  • With 2M learning iterations, DAIMC performed better than DQN and A2C, matched PPO2, and showed substantially less variance across runs than PPO2.The authors describe this comparison as illustrating applicability, not as a thorough benchmark against state-of-the-art RL agents.Source: §4.4 · p. 8

5 Concluding Remarks

The concluding remarks frame the architecture as biologically inspired through hierarchical generative modeling. They discuss MCTS, precision modulation, the scope of performance comparisons, and possible extensions.

5 Concluding Remarks

  • The hierarchical generative model contextualizes lower-level state representations with a higher-level policy network, reflecting a separation of temporal scales.The authors present this structure as part of a flexible framework for biologically inspired agents.Source: §5 · p. 8
  • Figure 5 compares DAIMC with DQN, A2C, and PPO2, showing means and standard deviations across multiple training runs.The authors say a more thorough comparison with reward-maximizing agents remains future work and that they did not focus on benchmarking against state-of-the-art RL agents.Source: Figure 5 · p. 9, §5 · p. 9
  • MCTS prunes policy paths whose expected free energy is much higher than an upper confidence bound, reducing the paths that must be evaluated.The authors relate this resource-limited search to biological planning and to selecting trajectories according to their prior probability of occurring.Source: §5 · p. 9
  • The authors associate precision factor ωt with attention because it incorporates prior uncertainty and encourages statistical independence between features.They link uncertainty modulation to noradrenaline for state transitions and dopamine for policy selection.Source: §5 · p. 9
  • Future extensions include testing larger networks or LSTMs, episodic memory, alternative task objectives, and the relevance of disentangled latents to functional segregation in the brain.The authors also identify more thorough performance comparisons with reward-maximizing agents as future work.Source: §5 · p. 9

6 Broader impact

The authors present the agent as a possible tool for biologically inspired research and realistic behavioral experiments. They also flag potential misuse in modeling people's cognitive biases.

6 Broader impact

  • The architecture may support brain-inspired simulations, behavioral experiments, and hypotheses about physiology and cognitive structure-function relationships.The authors propose manipulating priors to study how brain damage could affect cognitive function, building on discrete-state active-inference work.Source: §6 · p. 10
  • The authors warn that bad actors could potentially exploit the architecture to model and profit from people's cognitive biases.Source: §6 · p. 10

8 Supplementary material

The supplementary material provides an expected-free-energy derivation, glossary, training details and algorithm, model parameters, and examples of plans and latent traversals.

8.1 Expected free energy derivation

  • The EFE derivation combines rewritten terms after Bayes inversion and factorization; transition, encoder, and MC-dropout samples estimate entropy terms, whose difference is mutual information.The estimator is described as noisy but fast and suitable for training.Source: §8.1 · p. 15

8.2 Glossary of terms and notation

8.3 Training Procedure

  • Regularization ramps γ from 0 to 0.8 and improves convergence speed, while the transition and action networks each use a learning rate of 10^-4.The reported runtime varies by environment and training setup, from about 9 hours for Animal-AI off-policy training to 8 days on-policy.Source: §8.3 · p. 17

8.4 Training algorithm

  • The on-policy algorithm samples observations, runs the planner, updates the habitual network and precision, applies an action, and trains the transition and state models.The description applies to both environments and can also be used with batch sizes above one.Source: §8.4 · p. 17

8.5 Model parameters

  • Both environments use 10-dimensional latent states and depth-3 MCTS rollouts, with 300 loops for dSprites and 100 for Animal-AI.The networks use convolutional, deconvolutional, fully connected, and dropout layers; Animal-AI has image-shaped inputs and three action outputs.Source: §8.5 · p. 18, Figure 6 · p. 18

8.6 Examples of agent plans

8.7 Examples of traversals

  • Figure 9 shows latent-space traversals, distributions across 1,000 random observations, and correlations between latent dimensions and six environment features.The six features comprise five dSprites factors and reward, encoded in the top pixels of s9.Source: Figure 9 · p. 20
Loading 2006.04176v2…