PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Inference in Robotics and Artificial Agents: Survey and Challenges

Pablo Lanillos, Cristian Meo, Corrado Pezzato, Ajith Anil Meera, Mohamed Baioumy, Wataru Ohata, Alexander Tschantz, Beren Millidge, Martijn Wisse, Christopher L. Buckley, Jun Tani

arXiv:2112.01871v1cs.ROcs.AIcs.LG

TL;DR

Robotics requires state-estimation and control under uncertainty, while exact Bayesian inference can be intractable. This paper surveys active inference theory and robotics implementations, including experiments and connections to other frameworks. The reviewed experiments report deep AIF successes on challenging reinforcement-learning benchmarks, alongside unresolved modelling and implementation challenges.

Abstract

from arXiv · show

Active inference is a mathematical framework which originated in computational neuroscience as a theory of how the brain implements action, perception and learning. Recently, it has been shown to be a promising approach to the problems of state-estimation and control under uncertainty, as well as a foundation for the construction of goal-driven behaviours in robotics and artificial agents in general. Here, we review the state-of-the-art theory and implementations of active inference for state-estimation, control, planning and learning; describing current achievements with a particular focus on robotics. We showcase relevant experiments that illustrate its potential in terms of adaptation, generalization and robustness. Furthermore, we connect this approach with other frameworks and discuss its expected benefits and challenges: a unified framework with functional biological plausibility using variational Bayesian inference.

I. INTRODUCTION

The paper surveys active inference for robotics and artificial agents, presenting its mathematical background, applications, experiments, benefits, and challenges. AIF frames estimation and control as optimizing one objective: a free-energy bound.

I. INTRODUCTION

  • The survey reviews AIF work in robotics and artificial agents and discusses its benefits and challenges as a unified framework for estimation, control, planning, and learning.It also provides mathematical and theoretical background and organizes relevant work across state-estimation, control, planning, and high-level cognitive skills.Source: §I · p. 1, A. Overview · p. 1
  • AIF links perception to prior knowledge and action: predicted states can shape perception and trigger actions that make incoming sensations better fit the internal model.The paper uses the Dallenbach illusion and escalator effect to illustrate these roles of prior information and predicted world states.Source: §I · p. 1, Figure 1 · p. 1

B. Paper structure

  • The paper structure covers the common ground, mathematical descriptions of state-estimation, control, planning, and learning, robotic experiments, and connections to other frameworks.It also discusses benefits and challenges to using AIF as a modelling technique for robotic systems.Source: B. Paper structure · p. 2, §II · p. 2

II. ACTIVE INFERENCE COMMON GROUND

  • AIF addresses estimation and control through one free-energy objective, updating internal states and generating actions that make the world more predictable.The framework encodes intentions as preferences that influence both state-estimation and control.Source: §II · p. 2

A. From Bayesian Inference to the Free Energy Principle

The section motivates variational inference as a tractable alternative to computing an often-intractable Bayesian posterior. It explains how variational free energy supports optimization-based inference and how common approximations simplify estimation and control.

1) Variational Free Energy:

  • Variational inference approximates the often-intractable posterior p(x|y) by optimizing an auxiliary density q(x) to minimize its KL divergence from that posterior.The section presents this as a tractable solution to inference, particularly in continuous spaces.Source: 1) Variational Free Energy · p. 3
  • Minimizing variational free energy bounds surprise and turns Bayesian inference into optimization, while AIF jointly infers state and action by optimizing that objective.The VFE is an upper bound on surprise, F ≥ −log p(y).Source: 1) Variational Free Energy · p. 4

2) Mean-field and Laplace approximation:

  • The paper chooses a Gaussian variational density tracked by its mean and variance; mean-field and Laplace approximations make estimation and control a quadratic optimization problem that minimizes prediction error.Other forms of variational approximation are outside the paper's scope.Source: 2) Mean-field and Laplace approximation · p. 4

B. Generative models

The generative model represents the agent’s internal approximation of the world and can be specified or learned. Perception and action then use the same variational free-energy objective to update beliefs and compute control.

B. Generative models

  • The generative model is the agent’s internal representation of the process that generates data, and it may be designed by an expert or learned through interaction.It represents states and observations through a joint distribution, with transition and observation models.Source: B. Generative models · p. 4
  • In state-space form, internal-state dynamics use prior information or preferences, while the observation model relates sensory output to internal states.The model functions may be explicit or approximators such as neural networks.Source: B. Generative models · p. 4

C. Perception and action

  • AIF optimizes VFE through both perception and action: perception estimates states and parameters, while action computes a control signal that can resolve uncertainty and obtain observations aligned with beliefs or intentions.The paper notes that action is encoded in the observation y(u) in the original formulation, and both equations are usually solved by gradient descent.Source: C. Perception and action · p. 4

III. STATE-ESTIMATION

The section presents state-estimation as Bayesian filtering under the free-energy framework, estimating both states and confidence. It also identifies sensitivity to assumed noise smoothness and open questions about guarantees as challenges for real-robot use.

III. STATE-ESTIMATION

  • State inference estimates the mean state and its precision using gradients of free energy, with the state dynamics expressed as a differential equation.The section frames this as analogous to applying a Bayesian filter.Source: §III · pp. 4–5

A. State-estimation minimizes the prediction error

  • Under the stated prediction-error definitions, the state gradient of free energy is the gradient of a weighted sum of squared sensory and state-model prediction errors.The paper connects this formulation to predictive coding.Source: A. State-estimation minimizes the… · p. 5

B. Confidence in estimation

  • In continuous time, generalized coordinates track state-trajectory probability densities and capture noise smoothness, but estimation quality is highly sensitive to assumed smoothness, especially when it is low.Higher-order derivatives beyond accelerations are problematic on real robots when noise smoothness is unknown.Source: B. Confidence in estimation · p. 5
  • Future work includes estimating colored-noise smoothness; despite some stability and convergence guarantees, optimality guarantees remain an open area.The passage identifies scope for further proofs of optimality.Source: B. Confidence in estimation · p. 5

IV. CONTROL

The control formulation uses variational free energy for state estimation and action, while planning extends the framework to future trajectories. It introduces expected free energy to evaluate plans over future states and observations.

IV. CONTROL

  • The control formulation uses variational free energy for state estimation and action, with actions changing sensory input and preferences encoded in internal dynamics.Actions influence free energy indirectly through sensory input; being away from a preferred state increases prediction error and generates actions toward the goal.Source: §IV · p. 5

V. PLANNING

  • Planning models future states and observations conditional on action plans, addressing tasks where actions have delayed consequences.The section introduces discrete-time optimization, expected free energy, and a method for finding an optimal control plan.Source: §V · pp. 5–6

A. Discrete-time optimization under the Markov assumption

  • Planning assumes a discrete-time POMDP in which observations depend on current hidden states and states depend on the previous state and action.The Markov assumptions let inference use current observations and the previous state and action rather than the full history.Source: A. Discrete-time optimization under… · p. 6

B. Expected Free Energy

  • Expected free energy measures average free energy over future states and observations conditioned on a plan, with preferences encoded in a biased generative model.Unlike instantaneous free energy, it includes an expectation over future observations, which are unknown and inferred under the plan.Source: B. Expected Free Energy · p. 6

C. Finding the optimal plan

The section derives plan selection from expected free energy and describes learning approaches for active inference models. It also notes that discrete-state methods face expressiveness and scalability constraints.

C. Finding the optimal plan

  • Under time-factorized variational densities, trajectory expected free energy decomposes into timestep terms, and the optimal plan is a softmax distribution over summed plan-conditioned expected free energy.The decomposition relies on the POMDP’s statistical factorization.Source: C. Finding the optimal plan · pp. 6–7
  • Expected free energy combines extrinsic goal or reward value with intrinsic information gain, which favors informative states and exploration.The information-gain term measures divergence between posterior and prior beliefs about the state trajectory.Source: C. Finding the optimal plan · p. 7

VI. LEARNING

  • Generative models can be learned without an expert designer, using discrete-state approaches or function approximations for adaptive control and planning.The section identifies likelihoods, transition models, approximate posteriors, and desired observation distributions as learnable components.Source: §VI · p. 7

A. Discrete-state-space

  • Discrete-state models support explicit evaluation in small environments, but low-dimensional state requirements and exponential policy enumeration limit scalability and fine-grained continuous settings.The cited discussion describes direct enumeration as rapidly restricting this approach to relatively simple tasks; hierarchies or sparse matrices are proposed as possible avenues for future work.Source: A. Discrete-state-space · p. 7

B. Deep active inference

Deep active inference uses function approximators to scale inference and control, and optimizes expected free energy for planning; hierarchical recurrent models extend predictive behavior across time. The survey describes their learning procedures and robotic applications.

B. Deep active inference

  • Deep active inference uses general function approximators, including Gaussian processes and neural networks, for scalable distribution modeling, trading away performance and convergence guarantees and interpretability.Deep neural networks can in theory represent arbitrary distributions given sufficient depth and width.Source: B. Deep active inference · p. 8

1) Adaptive control with deep AIF:

  • For adaptive control, a learned forward model predicts sensory output from internal state; weighted prediction errors update state estimates and the action through model Jacobians.The approach can use self-supervised learning and extend to multimodal inputs.Source: 1) Adaptive control with deep AIF · p. 8

2) Planning with deep AIF:

  • Deep AIF plans by learning densities, parameters, and policies through expected-free-energy optimization, with reported performance matching or potentially exceeding deep-RL state of the art.Proposed estimators include bootstrapping, black-box optimization, and imagined future rollouts; model-based methods have matched state-of-the-art model-based RL on several MDP and POMDP tasks.Source: 2) Planning with deep AIF · p. 8
  • When generative and posterior model parameters are explicit, expected free energy also includes parameter information gain, encouraging exploration described as artificial curiosity or intrinsic motivation.This is characterized as novelty seeking.Source: 2) Planning with deep AIF · p. 8

VII. HIERARCHICAL REPRESENTATION

  • PV-RNN extends recurrent predictive coding with time-varying latent variables, error regression during action generation, and multiple timescales.Its layers use deterministic and random latent variables, with the lowest layer predicting sensory observations; predicted proprioception can provide target joint angles for motor control.Source: §VII · p. 9
  • PV-RNN alternates top-down generation and bottom-up inference; BPTT updates adaptation variables and connectivity weights until free energy is minimized, after which actions can be generated.Generation sweeps from higher to lower layers and forward through time; inference sweeps from lower to higher layers and backward through time.Source: §VII · pp. 9–10
  • Hierarchical recurrent models have been implemented in imitation, object-manipulation planning, and navigation combining active inference with reinforcement learning.Source: §VII · p. 10

VIII. ROBOTIC EXPERIMENTS

The survey situates robotic active inference alongside earlier predictive robot models and describes its potential under uncertain dynamics. It also notes that meaningful generative models and prior preferences remain a design and learning challenge.

VIII. ROBOTIC EXPERIMENTS

  • Early hierarchical RNNs generated visuo-proprioceptive movement patterns and adapted in real time to environmental changes, but used deterministic rather than Bayesian dynamics,,.Later work described exact-formalism reaching with a simulated 7-DOF arm and deployed active inference on humanoid and industrial robots,,,.Source: §VIII · p. 10
  • Active inference treats actions as realizing sensory consequences of prior causes, avoiding the need for precise inverse dynamics, which can be problematic to compute.The approach instead shifts the challenge to designing or learning meaningful generative models and prior beliefs.Source: §VIII · p. 10
  • The reviewed experiments cover estimation, adaptive control, fault-tolerant control, planning, and complex cognition.Source: §VIII · p. 10

A. Estimation

The survey reviews estimation and adaptive-control experiments, from multisensory body estimation to learned controllers and pixel-based methods. Reported demonstrations include robustness and adaptation, alongside trade-offs in model learning and state accuracy.

A. Estimation

  • Multisensory body estimation improved configuration refinement and adapted to strong visual-tactile perturbations or missing inputs, while remaining susceptible to visual-tactile illusions,.The forward model was learned with Gaussian processes, and estimation minimized variational free energy.Source: A. Estimation · p. 10
  • DEM-based estimation outperformed Kalman filtering for state estimation and an Unknown Input Observer for input estimation under colored noise; quadrotor learning produced accurate output predictions versus state-of-the-art identification methods, [10].The survey also reports a stable parameter-estimation proof.Source: A. Estimation · pp. 10–11

3) With function learning:

  • At 24.79 m/s2 gravity, M-AIF achieved better end-effector error performance than the Panda built-in controller; tests also showed adaptation to input noise and internal parameter changes.The figure compares the built-in controller with proprioceptive and multisensory AIF; the survey also reports improved performance under gravity and other parameter changes.Source: Figure 4 · p. 11, 3) With function learning · p. 12, 1) Without learning · p. 12

B. Adaptive control

  • On a physical iCub, active inference supported upper-body reaching and head-object tracking, with reported noise robustness and potential for counteracting unexpected environmental or model changes.Cartesian- or image-space goals were translated into joint space using the inverse Jacobian, avoiding complex inverse dynamical models for control.Source: B. Adaptive control · p. 11

1) Without learning:

  • The model-free Active Inference Controller for a 7-DOF arm outperformed MRAC across adaptability, tuning effort, disturbance rejection, computational effort, and pick-and-place performance with unknown object weights,.It transferred from simulation to a real robot without controller retuning while preserving compliant torque-command behavior.Source: B. Adaptive control · p. 11

2) With precision learning:

  • Online precision learning improved response and robustness versus manually tuned AIC, but target-biased state beliefs made reconstruction inaccurate until the target was reached.Learned covariance matrices converged above the controlled system's Gaussian noise level.Source: 2) With precision learning · p. 12

3) With function learning:

  • Function learning can avoid designing observation and dynamics models, but still requires learning state forward dynamics and the observation-model Jacobian.Pixel-AIF used convolutional networks for visual inputs and velocity control; M-AIF extended generative model learning to high-dimensional multimodal torque control,.Source: 3) With function learning · p. 12
  • The learned observation-model Jacobian has a physical interpretation as the change in perception or action with respect to joints; a virtual-arm visualization shows elbow- and shoulder-related changes [21].Source: 3) With function learning · p. 12

4) In combination with discrete planning:

The section surveys AIF applications in adaptive manipulation and fault recovery, reward-based planning, and intention-blended imitation. Across these settings, it describes adaptive responses to changing sensory input, sparse rewards, and conflicting intentions.

4) In combination with discrete planning:

  • Behaviour-tree priors guide AIF during mobile manipulation, allowing robots to perform, repeat, or skip actions as conditions change while providing safety guarantees and goal convergence.Experiments used two mobile manipulators in simulation and on real robots in retail tasks.Source: 4) In combination with discrete planning · p. 12

C. Fault-tolerant control

  • AIF-based fault-tolerant control detects and recovers from frozen encoders and camera misalignment without added monitoring signals or alternative controllers, but changing goals can trigger false alarms.Precision learning also supports stochastic fault detection and recovery without threshold definitions.Source: C. Fault-tolerant control · p. 13

D. Planning

  • Deep AIF optimizes a plan distribution from expected free energy estimates and executes the first action of its most likely plan.The evaluated settings include shaped rewards, sparse rewards, and reward-free maze coverage.Source: D. Planning · p. 13
  • Deep AIF reaches the Mountain Car goal in one trial, performs comparably to benchmarks on Cup Catch, and substantially outperforms SAC after 100 episodes on Half Cheetah.The comparison covers running and flipping tasks in Half Cheetah; Fig. 5 plots average return by episode.Source: D. Planning · p. 13, Figure 5 · p. 13
  • In the reward-free Ant Maze, Deep AIF explores more than a random baseline, covering a larger portion of the maze.Success is measured by percentage of maze covered.Source: D. Planning · pp. 13–14

1) Intention-blended Human-robot collaboration:

  • In imitative interaction, a less-dominant complexity term favors changing predictions to match observations, whereas a more-dominant term favors retaining the robot’s original intention.When the network predicted primitive B but observed C, the first condition adapted to C and the second maintained B.Source: 1) Intention-blended Human-robot… · p. 14

2) Self/other distinction:

The section describes a mirror-based approach to robot self/other distinction and connects active inference with classical control and reinforcement learning. It highlights both limiting cases that resemble established controllers and differences arising from AIF’s information-seeking objective and flexible goal priors.

2) Self/other distinction:

  • AIF supports mirror-based self/other distinction by using movement cues to learn visual forward kinematics and accumulate Bayesian model evidence.The approach distinguishes a robot’s simple actions from those of other agents and is presented as relevant to safe human-robot interaction.Source: 2) Self/other distinction · p. 14

A. Relationship with classical controllers

  • As τ^-1 approaches infinity, the active-inference controller becomes a PI controller with Kd = 0.The section also notes that generalized motions up to third order yield a non-zero D term.Source: A. Relationship with classical… · p. 15
  • AIF shares Bayesian-inference and optimal-control foundations with LQG, while its expected-free-energy planning also connects it to model-predictive control and control-as-inference.The section notes that trajectory expectations for planning can be approximated by Monte Carlo sampling with a forward dynamics model.Source: A. Relationship with classical… · p. 15

B. Relationship to Reinforcement Learning

  • AIF’s expected free energy adds intrinsic information-seeking to reward-based planning and permits goal priors beyond the Boltzmann form implicitly assumed by RL.When only the extrinsic term is considered, it reduces to expected reward; the model-based RL correspondence is more exact than the model-free one.Source: B. Relationship to Reinforcement… · p. 15

X. BENEFITS AND CHALLENGES

The survey presents Active Inference (AIF) as a unified perception, action, and planning framework, and discusses its potential benefits and open challenges for robotics. It reviews biological plausibility and variational Bayesian inference as additional foundations for expected robotic capabilities.

A. Unified framework

  • AIF unifies perception and action through free-energy minimization, may let agents enforce simplified models through action, and combines intrinsic and extrinsic value in planning.It also supports hierarchical generative models for complex probabilistic controllers.Source: A. Unified framework · p. 16
  • Open challenges include avoiding suboptimal convergence, deriving feedback updates for complex systems, and addressing beliefs biased toward desired states rather than actual hidden states.The authors mention possible roles for intrinsic value and action in the generative model, alongside alternative control approaches.Source: A. Unified framework · p. 16

B. Functional biological plausibility

  • AIF and predictive coding may support monitoring, self-explainability, and metacognitive abilities; prediction precision lets robots monitor uncertainty, while self-awareness and agency are proposed as opportunities for interaction.The authors connect these capabilities to possible industrial and healthcare applications and to explainability and safety.Source: B. Functional biological plausibility · p. 16
  • Generalized coordinates improve estimation accuracy in drones in wind, but their methodological value for generic robotics and alternatives using noise-filter states remain open questions.The paper describes generalized coordinates as appending states with derivatives up to sixth order.Source: B. Functional biological plausibility · p. 16

C. Variational Bayesian Inference

  • With suitable hierarchical Bayesian priors, variational inference is expected to reduce data or trial requirements, and demonstration-based curricula may help humans teach robots those priors.The authors describe obtaining suitable priors through teaching as an open research area.Source: C. Variational Bayesian Inference · p. 16
  • Prediction errors and expectations of sensor-value variance are proposed as a basis for detecting unexpected inputs, including collisions, sensor or actuator malfunctions, sampling hiccups, and unexpected user behavior.The authors present broad sensor prediction and fault detection as an expected future capability following preliminary robot-arm experiments.Source: C. Variational Bayesian Inference · p. 17
  • AIF may address long-horizon planning complexity through approximate factorization of action distributions and uncertainty-aware planning, while offering a probabilistic foundation for model-predictive control.The authors also identify prediction accuracy deterioration with planning steps as a challenge motivating these approaches.Source: C. Variational Bayesian Inference · p. 17

XI. SUMMARY

  • The survey covers AIF formulations and implementations for estimation, adaptive control, planning, and learning, with robotic experiments and connections to classical control and reinforcement learning.It discusses benefits and challenges relevant to AIF's possible role as a robotics methodology.Source: §XI · p. 17

APPENDIX

The appendix presents variational free energy (VFE) through its relation to the posterior and evidence lower bound, then describes Laplace and mean-field approximations. These approximation choices respectively represent the posterior around a MAP estimate or simplify its variational distribution by factorizing variables.

A. Variational Free Energy

  • VFE is described as the KL divergence between the variational density and true posterior minus sensory surprise, and as the negative evidence lower bound.The appendix introduces a VFE definition using a model in which action is implicit in the observation model y(u).Source: A. Variational Free Energy · p. 20

B. Laplace Approximation

  • The Laplace approximation improves on a MAP point estimate by representing the posterior as a normal distribution centered at the MAP estimate.Figure 7 compares the true posterior with the Laplace approximation and MAP estimate.Source: B. Laplace Approximation · p. 20, Figure 7 · p. 20

C. Mean field Approximation

  • The mean-field approximation simplifies computation by assuming the variational distribution factorizes into independent parts over variables.The passage describes approximating p(x|y) with q(x) under this factorization.Source: C. Mean field Approximation · p. 20

D. Generalized coordinates

Generalized coordinates represent state trajectories and model colored noise using higher-order derivatives, with reported estimation advantages in cited settings. The appendix also presents expected free energy as a planning objective decomposable into extrinsic and intrinsic value.

D. Generalized coordinates

  • Generalized coordinates represent states and their higher-order derivatives, allowing trajectory-density tracking and modeling of colored process and observation noise.The noise smoothness precision encodes correlations between noise derivatives under a Gaussian-convoluted white-noise assumption.Source: D. Generalized coordinates · p. 20
  • Generalized coordinates are reported to outperform the Kalman Filter in state estimation and unknown input observers in input estimation for colored noise, and to improve quadrotor state estimation in wind.The reported comparisons cite for colored noise and [11] for a quadrotor hovering in wind.Source: D. Generalized coordinates · p. 20

E. Expected free energy (EFE)

  • Expected free energy over a plan decomposes into extrinsic and intrinsic value, and the optimal plan is characterized by minimizing expected free energy over trajectories.The passage associates the extrinsic-value term with expected log preferences over observations.Source: E. Expected free energy (EFE) · p. 20
Loading 2112.01871v1…