Deep Active Inference

1. Introduction

• Active inference minimizes an upper variational bound on average surprise, connecting its functional form to variational inference in deep generative models.The optimization for active agents depends on true world dynamics, which the agent usually cannot access.Source: §1 · p. 1
• Deep and recurrent neural networks approximate world dynamics, while evolution strategies estimate gradients for the non-differentiable objective without requiring the world’s motion equations or derivatives.The objective is the variational bound averaged over a population of agents.Source: §1 · p. 2
• The paper pairs active inference with machine-learning methods to build agents that pursue goals while learning generative models of themselves and their surroundings.The authors illustrate the agent with mountain car and state that the approach can scale to complex environments.Source: §1 · p. 2

2. Active Inference

• Homeostatic motivation treats survival as keeping internal states within bounds, approximated by minimizing sensory entropy under an almost-constant sensory mapping.The mapping’s sensitivity is assumed to remain nearly constant over encountered states.Source: §2 · pp. 2–3
• The agent uses a deep, recurrent generative model of sensory observations and latent world states, with a variational density whose parameters are accessible through sensory states, model parameters, and neural activity.The model factorizes into a likelihood and a prior over states; the variational density may be represented by its means and standard deviations.Source: §2 · pp. 3–4
• Because computing surprise requires marginalizing over potentially many hidden states, active inference minimizes a simpler upper bound, variational free energy.The integral becomes extremely hard to solve when the latent-state space is high-dimensional.Source: §2 · p. 4
• Optimizing the variational density approximates the posterior over hidden states from observations, while optimizing generative-model parameters improves evidence for the model.The paper calls these processes perception and perceptual learning, respectively.Source: §2 · pp. 4–5
• With actuators, the agent can also minimize free energy by changing its observations; assigning higher prior probability to a state makes it more likely to seek that state.The likelihood term is called accuracy, while the divergence between posterior and prior is called complexity.Source: §2 · p. 5
• Here, beliefs cover hidden world states; broader formulations can also cover generative-model parameters, often using a mean-field factorization.Source: §2 · p. 6

3. Methods

• The approach combines stochastic optimization of deep generative models with evolution strategies to implement active inference.Source: §3 · p. 6
• Evolution strategies are used as an efficient, scalable optimizer for non-differentiable objective functions.Source: §3 · p. 6
• The mountain car demonstration targets reaching a specific goal while concurrently learning a generative model of the environment.Source: §3 · p. 6

3.1.1. Fully Connected Layer.

• A fully connected layer applies a nonlinear function to a weighted input sum plus bias; its weights and biases are tunable parameters.Source: §3.1.1 · pp. 6–7

3.1.2. Deep Feedforward Networks.

• A deep feedforward network stacks layers sequentially, with intermediate outputs forming hidden layers and the final function producing the output.Source: §3.1.2 · p. 7
• Deep feedforward networks can approximate functions without requiring the agent’s generative model to share the true process’s model family.The cited approximation result requires a linear output layer, at least one nonlinear hidden layer, and sufficient hidden-layer width.Source: §3.1.2 · p. 7

3.1.3. Recurrent Neural Networks.

• A recurrent network feeds back samples from its current state distribution as inputs for the next state, supporting temporal dependencies.The paper notes that this simple recurrent architecture can learn long-range dependencies given sensible initialization.Source: §3.1.3 · p. 7

• The car starts at x = −0.5 with zero velocity in the discrete-time environment.Source: §3.2 · pp. 8–9
• The car’s physical state comprises its action state, position, and velocity, and the environment dynamics are summarized by update equations.Source: §3.2 · p. 8
• The environment’s forces include a position-dependent downhill force, motor force controlled by action, and friction that varies linearly with velocity.Source: §3.2 · pp. 8–9
• The car must initially move uphill away from its target to build momentum, because its motor limit cannot overcome the downhill force at x = 0.At x = 0, the downhill force is −0.05 while motor force is limited to (−0.03, 0.03).Source: §3.2 · p. 9

3.3.1. Sensory Inputs.

• The agent receives noisy position input, can add a nonlinear transformed position channel to test complex modeling, and lacks direct velocity measurements.Velocity must therefore be inferred from sensory sequences that depend on position.Source: §3.3.1 · p. 10

3.3.2. The Agent’s Generative Model of the Environment.

• A state-transition model represents learned world dynamics, while state-conditioned likelihoods generate sensory observations; sampling from the learned model produces simulated processes.The transition depends on the current state, and the model’s observation likelihoods factor across the observables.Source: §3.3.2 · pp. 11–12

3.3.3. Variational Density.

• An amortized inference network approximates the posterior over hidden states from previous state and current sensory input, enabling fast inference but limiting context sensitivity.The paper attributes the limitation to representing the mapping with a static nonlinear function.Source: §3.3.3 · pp. 12–13

3.3.4. Action States.

• A neural action function approximates action states, allowing time-invariant parameters for the generative model, variational density, and actions to be optimized together efficiently.The paper names this class of agents for using deep networks to approximate both the variational-density statistics and action states.Source: §3.3.4 · pp. 13–14

3.3.5. Goal Directed Behavior.

• Goal-directed behavior is specified by encoding position in a latent state and setting its prior to favor the target position during the final timesteps.The example runs for 30 timesteps and sets the target to x = 1.0 for at least the last ten.Source: §3.3.5 · pp. 14–15

3.3.6. The Free Energy Objective.

• Minimizing variational free energy improves the generative model and posterior approximation, while optimizing action parameters steers the agent toward states expected under its model.Source: §3.3.6 · p. 16

• The free-energy estimate uses parallel processes, sampling actions, environment transitions, observations, and latent states to accumulate likelihood and KL terms.The algorithm identifies likelihood as accuracy and KL divergence as complexity.Source: §3.4 · p. 17
• Evolution strategies optimize a parameter population’s expected free energy, avoiding the need to backpropagate through unknown or nondifferentiable environmental dynamics.The method optimizes the population density’s sufficient statistics rather than a single parameter set.Source: §3.4 · p. 17
• Parameter samples from a Gaussian population are used to approximate gradients of expected free energy with respect to the population statistics.The population standard deviations are parameterized for stability, with a positive minimum to avoid division by zero and sign switches.Source: §3.4 · p. 18
• The estimated gradients update the population statistics with ADAM until expected free energy converges.Source: §3.4 · pp. 18–19

3.5. Constrained Sampling from the Learned Generative Model.

• Constrained sampling uses MCMC to infer sensory channels from specified observations, with multiple samples providing estimates of their uncertainty.The agent can estimate its homeostatic channel given proprioceptive and spatial inputs, or infer position and homeostatic channels given proprioceptive inputs.Source: §3.5 · p. 19
• MCMC chains may fail to converge unless initialized sufficiently close to a solution.The cited account of the method also conditions its convergence claim on unobserved channels not starting too far from their actual values.Source: §3.5 · pp. 19–20
• The algorithm repeatedly encodes partly randomized observations, generates denoised samples, resets known inputs, and re-encodes until inferred causes converge.The variational density encodes observations into the learned low-dimensional representation; repeated samples are intended to recover likely causes under the generative model.Source: §3.5 · pp. 19–20

3.6. Experimental Set Up and Parameters.

• The free-energy estimate uses one process per population sample, with 10^4 population samples; more processes at fixed total simulations worsen convergence.The authors attribute the poorer convergence to reduced parameter-space coverage while total stochastic variability remains approximately constant.Source: §3.6 · p. 20
• The experiments ran on a desktop PC with a 2013 NVIDIA GTX Titan GPU, and Table 1 lists parameter values for algorithms 2 and 3.Source: §3.6 · p. 20
• The implementation and figure-reproduction scripts were made available as code written in Python 2.7 using Theano for GPU-optimized tensor operations.Source: §3.6 · pp. 20–21

4. Results

• After about 250 population-density updates, the agent discovers that first moving away from its target builds momentum and produces a rapid decline in free energy.Training converges to a plateau before this strategy shift; the optimization takes less than 0.4 seconds per iteration and uses less than 300 MB of GPU memory.Source: §4 · p. 21
• After about 30,000 training steps, the agent swings left to gain momentum, reaches x = 1.0, and stays there by counteracting the downhill force.The leftward start allows it to overcome the steep slope near x = 0 despite limited motor force.Source: §4 · p. 21
• Samples from the learned generative model reproduce the agent’s proprioceptive and position channels, as well as the setting’s irrelevant nonlinear homeostatic channel.The reported samples are noisy but consistent with actual behavior, and averaging across processes can reduce variability.Source: §4 · pp. 21–22
• With a proprioceptive timecourse shifted back ten steps as input, nine of ten constrained samples resemble the actual trajectory and unconstrained model samples; one process does not converge.The authors suggest that an MCMC chain may need initialization sufficiently close to its solution.Source: §4 · p. 22

5.1. A Scalable and Flexible Active Inference Implementation.

• The agent solves mountain car while learning a generative model of itself and its environment, without access to the true generative process or its derivatives.Deep and recurrent neural networks approximate the environment dynamics, while a black-box optimiser estimates gradients.Source: §5.1 · pp. 22–23
• After 30,000 training steps, using mean population-density parameters, the trajectory first leads uphill to the left, away from the target, to gain momentum and overcome the steep slope at x = 0.The model’s samples show prior expectations and trajectory-conditioned expectations for proprioceptive input oa, position ox, transformed position oh, and homeostatic state s1.Source: Figure 6 · p. 24, Figure 7 · p. 25, Figure 8 · p. 26

5.2. Comparison to Original Implementation.

• Unlike the original implementation, this agent uses discrete timesteps and must learn both successive-position dynamics and an approximation of the world’s generative process.Evolution strategies estimate action-related gradients without access to the true sensory derivatives.Source: §5.2 · p. 27

5.3. Comparison to a recent ”Action-Oriented” Implementation of Active Inference.

• Compared with the continuous-time Action-Oriented approach, the agent learns a flexible model of the dynamics and does not require a proprioceptive channel to reach goals.Restricting action learning to proprioceptive prediction errors might hinder or delay learning in more complex environments.Source: §5.3 · pp. 27–28

5.4. Action Function as State-Action Policy.

• The agent’s flexible state representation can retain a history-dependent velocity estimate, supporting different actions at the same position depending on acquired momentum.It learns to move left initially, then accelerate right at that same position to climb the slope.Source: §5.4 · p. 28

5.5. Emergent Curiosity.

• Optimising the generative model gives the agent epistemic value, so it seeks to learn environmental statistics even when its actions cannot influence them.Source: §5.5 · p. 28

5.6. Constrained Sampling for Understanding the Learned Models.

• Constrained samples can reveal the agent’s beliefs, but its inferred dynamics deviate from the true dynamics because it experiences only a small subset of the phase space.The authors suggest that noisy environments with larger stochastic fluctuations could expose it to more states and support a more complete model.Source: §5.6 · p. 29

5.7. Benefits of the Evolution Strategies Optimiser.

• Evolution strategies explore the environment through parameter-space search, so the agent can learn a nearly deterministic policy without forced random actions.The authors also state that full-free-energy gradient estimates are resilient to long horizons and sparse rewards.Source: §5.7 · p. 29

5.8. Variational Bayesian Perspective on Evolution Strategies.

• The population density can be viewed as a variational distribution over model parameters, making the objective a joint variational free energy over parameters and latent states.This interpretation treats population dynamics as ensemble-based parameter optimisation, with parameter beliefs minimising free energy integrated over time.Source: §5.8 · pp. 29–30

5.9. Towards more flexible Expectations.

• Current goals rely on explicit priors on the first latent dimension, which the authors describe as ad hoc and restricted.They propose sampling from the generative model and penalising deviations from target distributions to express richer expectations.Source: §5.9 · pp. 30–31

5.10. A first step from Artificial Life to Artificial General Intelligence.

• The proposed agent learns an environmental model while achieving prespecified goals encoded as prior expectations on perceived states.The authors frame this as a flexible, scalable implementation of active inference grounded in a homeostatic argument.Source: §5.10 · p. 31

7.1. Performance without an explicit proprioceptive channel.

• Without a proprioceptive channel, the agent learns its prior-instilled goal while building a generative model of exteroceptive sensations.After 30,000 training steps, it swings left and then directly upward to the goal at x = 1.0, and its model sample corresponds to its trajectory.Source: §7.1 · p. 35, Figure 10 · p. 37, Figure 11 · p. 38
• The convergence plot for the agent without proprioceptive feedback includes a right-panel enlargement of the red-shaded region.The proprioceptive-surprise-only agent's convergence plot likewise enlarges its red-shaded region.Source: Figure 9 · p. 35, Figure 12 · p. 39

7.2. Performance with purely proprioceptive action.

• Using action only to suppress proprioceptive surprise severely impairs convergence and prevents successful goal achievement and learning about the environment.The action-function update ignores direct changes in other sensory modalities caused by actions.Source: §7.2 · pp. 35–36
• The proprioceptive-only agent remains at its initial position after 30,000 training steps and shows no clear behavioral strategy.Its sampled generative model lacks correspondence to its actual trajectory.Source: Figure 13 · p. 40, Figure 14 · p. 41