PaperlayerView exact revision on arXiv

Source-linked AI summary

Habits and goals in synergy: a variational Bayesian framework for behavior

Dongqi Han, Kenji Doya, Dongsheng Li, Jun Tani

arXiv:2304.05008v1cs.LGcs.AI

TL;DR

Habitual and goal-directed behavior are often treated separately, although connecting them matters for understanding how they may share skills. This paper models them through prior and goal-conditioned posterior distributions of a shared Bayesian intention, and reports flexible goal planning without goal training in its simulations.

  • Problem

    Habitual and goal-directed behavior are commonly treated as separate problems, leaving their relationship and potential skill sharing to be addressed.

    Source: §1 · p. 2

  • Method

    The framework represents habitual behavior with a goal-independent prior over intention and goal-directed behavior with its goal-conditioned posterior, using predictive internal models for planning.

    Source: §1 · pp. 2–3, §7 · p. 17

  • Results

    The agent learned habitual behavior and an intention-conditioned predictive model, then planned for given goal observations or partial goal properties without goal training.

    Source: §7 · p. 17

  • Takeaways & Limitations

    The framework links habitual and goal-directed behavior through a shared intention representation and predictive internal model.

    Source: §7 · p. 17

    The proof of concept focuses on a T-maze and has yet to address more complex motor-control tasks.

    Source: §7 · p. 18

Abstract

from arXiv · show

How to behave efficiently and flexibly is a central problem for understanding biological agents and creating intelligent embodied AI. It has been well known that behavior can be classified as two types: reward-maximizing habitual behavior, which is fast while inflexible; and goal-directed behavior, which is flexible while slow. Conventionally, habitual and goal-directed behaviors are considered handled by two distinct systems in the brain. Here, we propose to bridge the gap between the two behaviors, drawing on the principles of variational Bayesian theory. We incorporate both behaviors in one framework by introducing a Bayesian latent variable called "intention". The habitual behavior is generated by using prior distribution of intention, which is goal-less; and the goal-directed behavior is generated by the posterior distribution of intention, which is conditioned on the goal. Building on this idea, we present a novel Bayesian framework for modeling behaviors. Our proposed framework enables skill sharing between the two kinds of behaviors, and by leveraging the idea of predictive coding, it enables an agent to seamlessly generalize from habitual to goal-directed behavior without requiring additional training. The proposed framework suggests a fresh perspective for cognitive science and embodied AI, highlighting the potential for greater integration between habitual and goal-directed behaviors.

1 Introduction

Habitual and goal-directed behavior are often treated as separate systems, despite a proposed overlap in the low-level motor skills they use. The paper unifies them through a Bayesian intention variable, using its prior for habitual behavior and its goal-conditioned posterior for goal-directed behavior.

1 Introduction

  • The framework is motivated by the separation of habitual and goal-directed systems and the possibility that they share low-level motor skills.The paper proposes that each system may leverage actions learned by the other.Source: §1 · p. 2
  • The framework represents habitual and goal-directed behavior as, respectively, the prior and goal-conditioned posterior distributions of a shared latent intention z.A shared function maps intention and contextual information to actions.Source: §1 · pp. 2–3
  • Predictive coding uses a compact intention z to predict future sensory observations, making it plausible to search for intentions that lead to a goal.The paper describes minimizing variational free energy to support this framework.Source: §1 · p. 3
  • The KL-divergence between the posterior and prior provides a theoretical link by bounding their difference and balancing model fit against latent-representation complexity.The paper presents this term as a basis for relating the two behaviors.Source: §1 · p. 3
  • Proof-of-concept simulations report that neuronal stochasticity does not harm behavioral efficiency while enhancing diversity, and that predictive coding supports flexible goal planning.The experiments use an embodied robot agent.Source: §1 · p. 3

2 Background

The background introduces reinforcement learning and distinguishes model-free from model-based approaches. It then describes predictive coding, the free energy principle, and active inference as accounts of prediction and action.

2.1 Reinforcement learning

  • Reinforcement learning seeks a policy that maximizes expected cumulative reward, with model-based methods learning environmental dynamics and model-free methods learning a direct state-to-action mapping.Both approaches balance exploration and exploitation.Source: §2.1 · pp. 3–4

2.2 Predictive coding

  • Predictive coding describes an internal generative model that predicts incoming sensory information, with prediction errors sent upward to refine the model.The account also describes top-down predictions and Bayesian comparison with sensory data.Source: §2.2 · p. 4

2.3 Free energy principle and active inference

  • The free energy principle frames biological systems as minimizing free energy, while active inference uses internal predictions to guide actions that reduce prediction error.Active inference is presented as Bayesian decision-making based on the free energy principle.Source: §2.3 · p. 4

3 Methods

The methods model intention as a latent variable and use it to connect habitual learning with goal-directed planning. They describe habitual behavior first, then train and use the model for goal-directed behavior.

3 Methods

  • The methods assume the agent first learns habitual behavior and then conducts goal-directed planning.Source: §3 · p. 4

3.1 Model details

  • A 2-dimensional latent variable, called intention, feeds a recurrent model, while a shared policy network maps prior or posterior states to actions.Source: §3.1 · pp. 4–5

3.2.1 Behaving

  • Habitual behavior is learned in an episodic reinforcement-learning setting, with actions computed from the prior recurrent state.Source: §3.2.1 · p. 5, §3.1 · p. 5

3.2.2 Training

  • Training updates the networks end-to-end from replayed sequences using prediction-error, complexity, and policy losses, with SAC as the base reinforcement-learning algorithm.Source: §3.2.2 · p. 6

3.3 Goal-directed behavior

  • Goal-directed planning optimizes intention against future observation predictions, enabling zero-shot planning from learned habitual behavior and goals expressed as images, image regions, or colors.Source: §3.3 · pp. 7–8

3.4 On the complexity term in learning habitual behaviors

  • The complexity term connects posterior-policy training to habitual behavior by making the combined objective optimize the prior-based habitual policy.Source: §3.4 · p. 8

4 Results

In the T-maze, the agent developed diverse, efficient habitual routes and then used inferred intentions to pursue varied goals without extra training. Goal-directed trials succeeded across all five tested cases.

4.1 Environment

  • The T-maze starts each trial at a fixed bottom position, offers two exits, and gives the agent 360° RGB observations and continuous-valued motor actions.Reaching an exit ends the trial; hitting a wall incurs a negative reward.Source: §4.1 · pp. 8–9

4.2 Learning diverse and effective habitual behavior

  • After 400,000 environment steps, the agent developed efficient routes to both exits; early intention noise selected the exit, with RNN-state branching emerging after several steps.The branching was attributed to randomness in the latent intention.Source: §4.2 · p. 10
  • Ablations indicate that three framework components are crucial for developing more diverse and efficient habitual behavior.The study evaluated diversity and efficiency metrics for habitual behavior.Source: §4.2 · p. 10

4.3 Flexible goal-directed behavior immediately transferred from habitual behavior

  • The framework transfers habitual control to goal-directed planning without extra training, inferring intention from future observations and using the same policy network for actions.A KL constraint keeps inferred intention from moving too far from the habitual prior.Source: §4.3 · p. 11
  • Goal-directed success was 97.9% (a), 97.4% (b), 100% (c), 99.5% (d), and 100% (e) across the five tested goal cases.The cases included reaching a place, observing a color more, and avoiding a color.Source: §4.3 · p. 11
  • With inferred goal-directed intention, predicted observations matched the place goal more than 10 steps ahead and reflected color goals, including avoiding blue.The blue-avoidance case also led the agent to an exit.Source: §4.3 · p. 12
  • Because goal-directed planning is constrained by habitual low-level motor skills, goals unreachable through those habits may remain unattainable.The authors describe this constraint as reducing the search space of possible actions.Source: §4.3 · p. 12

5 Related work in AI

The related work contrasts approaches to embodied intelligence, control, and behavior, positioning Bayesian behavior as a framework for goal-free learning that can later support goal-directed planning.

5.1 Embodied artificial general intelligence

  • The one-big-net framework emphasizes an RNN that can reuse knowledge across tasks, while LeCun’s framework uses energy minimization; Bayesian behavior explicitly predicts sensations and represents their uncertainty with z.The related work notes that prediction’s role in goal-directed planning is not explicitly specified for the one-big-net framework.Source: §5.1 · pp. 12–13

5.2 Model-based RL

  • Model-based RL uses learned environment models for dreaming, planning, or supporting model-free learning; this framework instead uses active inference to minimize free energy with respect to a goal.The cited variational-RNN approaches focus on single-goal tasks, whereas this framework’s stochasticity can support randomly pursuing multiple goals.Source: §5.2 · p. 13

5.3 Variational Bayes in deep learning and RL

  • Variational Bayes has been used in deep learning and RL to learn latent representations, but the related work says its use to model habits and goals had not been discussed in prior deep-learning studies.The framework is presented as a new decision-making and control application within the variational Bayes family.Source: §5.3 · p. 13

5.4 Goal-conditioned RL

  • Unlike goal-conditioned RL, which trains with episode goals and rewards for achieving them, this framework trains without assigned goals and tests with goals.Its training objective maximizes habitual-behavior rewards and minimizes expected free energy.Source: §5.4 · p. 13

5.5 Deep active inference

  • Deep active inference uses an environment model to infer actions toward desired outcomes, while the proposed framework infers an intention latent state rather than actions directly.The intention can be viewed as a high-level action in hierarchical RL.Source: §5.5 · pp. 13–14

5.6 Control as probabilistic inference

  • Control as probabilistic inference casts optimal control as inference over latent system states, while the implementation uses SAC to learn habitual behavior.The cited discussion says CPI implementations became model-free because they do not consider detailed action outcomes.Source: §5.6 · p. 14

5.7 Generalization in RL

  • The authors characterize learning without goals and testing with goals as a potentially novel RL generalization setting, distinct from prior work on changes in states, dynamics, observations, or rewards.They qualify the novelty claim with “as far as we are aware.”Source: §5.7 · p. 14

5.8 Self-discovery of goals or skills

  • Variational skill discovery learns diverse, predictable action primitives using latent skill labels and information-theoretic objectives, whereas the framework is related to goal-conditioned work without training a goal-achieving agent.Mendonca et al. train a goal-achieving agent with an additional goal-achievement reward; this framework does not train for goal achievement.Source: §5.8 · p. 14

6 Discussion

The framework computes intention z through a fast forward posterior during habitual behavior and through active inference during goal-directed behavior. This supports using learned low-level skills for goals not assigned during training.

6.1 Computing the Bayesian latent state z

  • The prior of z is a diagonal unit Gaussian for diverse, unconditioned goal choices, while its posterior is aligned with future states and is computed differently for learning habits and planning toward goals.The two posterior-computation routes optimize model parameters and biases during habitual learning or optimize z during active inference.Source: §6.1 · p. 14
  • During habitual behavior, the posterior uses the previous RNN state, current observation, and next-step observation, binding intention to state transitions without an assigned goal.This forward computation is designed to support later planning and keep habitual behavior computationally efficient.Source: §6.1 · p. 14
  • For goal-directed behavior, active inference searches for z that minimizes free energy with respect to the goal, using a backward process rather than encoding the goal as an input.The learned world model and compact intention representation support imagined future planning; this route is more computationally demanding and can target varied observation properties.Source: §6.1 · pp. 14–15

6.2 Toward understanding behavior

  • Stochasticity in hidden z, rather than motor actions, can develop diverse intentions while a learned mapping from z to actions produces effective low-level movements.The account attributes this to a learning process that regularizes z’s stochasticity.Source: §6.2 · p. 15
  • The prior and goal-conditioned posterior of z connect habitual control and goal-directed planning through the KL-divergence term in variational free energy, supporting motor-skill sharing.Habitual behavior uses the unconditioned prior; goal-directed behavior uses hindsight goal-related information to determine the posterior.Source: §6.2 · p. 15
  • Goal achievement without goal-specific training is framed as searching an internal predictive model over abstract intentions rather than searching all motor actions.The account compares specifying a goal objective to designing a language-model prompt, while noting that this framework’s training does not involve goals.Source: §6.2 · p. 15

6.3 The frame problem of AI

The discussion presents backward intention inference as a possible response to the frame problem, while identifying goal selection as unresolved. It also describes how repeated goal-directed inference may be amortized into habitual behavior and relates free-energy minimization to predictive coding and information bottleneck.

6.3 The frame problem of AI

  • Because goals can be diverse and complex, treating them as direct model inputs is vulnerable to the frame problem; backward inference of goal-directed intentions is proposed as a potential solution.The account presents this as a potential solution for complex environments, not a demonstrated guarantee.Source: §6.3 · p. 15

6.4 Where comes the goal

  • The framework assumes goals are assigned by the programmer and leaves modeling intrinsic goal selection to future research.A proposed possibility is a meta-habit for selecting goals that enhances agent or population fitness.Source: §6.4 · p. 15

6.5 Mutual conversion between habitual and goal-directed behaviors

  • The discussion contrasts the usual progression from goal-directed behavior to habit with a reversed schema, and describes amortizing repeated goal-directed intentions into fast habitual behavior.A feedforward network trained on experience with a repetitive goal predicts intention from the current RNN state; the reported experiment says it yields habitual behavior pursuing that goal.Source: §6.5 · p. 16

6.6 Predictive coding and information bottleneck

  • The account relates free-energy minimization to predictive coding and the information bottleneck: compact z encodes environmental and goal information while supporting future-observation prediction.With a unit-Gaussian prior, minimizing expected free energy is described as a special case of information bottleneck that limits input-to-latent information while increasing latent-to-prediction information.Source: §6.6 · p. 16

6.7 Hierarchy

The discussion treats z as a higher-level representation, describes uncertainty-sensitive arbitration between habitual and goal-directed intentions, and suggests possible relevance to neurological conditions.

6.7 Hierarchy

  • The RNN state h and intention z are described as two task-representation levels, with z more abstract at 2 dimensions compared with h at 256 dimensions.The authors identify intrinsic model hierarchy as an area for future work.Source: §6.7 · p. 16

6.8 Arbitration between habitual and goal-directed behaviors

  • In the proposed arbitration, the intention with smaller variance is more dominant, connecting preference between prior and goal-directed z to uncertainty.The passage describes this as the mixed behavior’s computation and notes a related latent-variable method in variational autoencoders.Source: §6.8 · pp. 16–17

6.9 Insights for neurological diseases

  • The framework is proposed as a tool for studying behavioral diversity in ASD; for Parkinson’s disease, the authors suggest impaired goal-directed planning may relate to high uncertainty in goal-directed intention.They raise reducing that uncertainty through medicine, deep brain stimulation, or sensory stimuli as a research possibility, not an established treatment.Source: §6.9 · p. 17

7 Conclusion

The framework treats habits and goals as prior and posterior distributions of an intention variable, combining model-free learning with active inference and predictive planning. In simulated T-maze experiments, it produced habitual motor behavior and goal-directed planning, while the authors note important limits in task scope and reachable states.

7 Conclusion

  • The framework models habitual and goal-directed behavior as prior and posterior distributions of intention, respectively, and uses predictive internal models for goal-directed planning.It combines model-free deep reinforcement learning for habitual motor skills with active inference for flexible behavior.Source: §7 · p. 17
  • The agent learned smooth bifurcating habitual behavior through online deep RL and used its predictive internal model to plan for goal observations or partial goal properties without goal training.The internal model predicts future observations conditioned on intention z.Source: §7 · p. 17
  • The authors identify stochasticity with proper regularization as enabling diverse, effective motor actions, and variational Bayesian methods as supporting shared low-level skills across behavior types.They also propose that predictive coding enables goal-directed behavior by inferring neural state z values associated with desired goal properties.Source: §7 · p. 17
  • The authors characterize the variational free energy complexity term as bridging habitual and goal-directed behavior.Source: §7 · p. 18
  • The proof of concept focuses on a T-maze, and the framework may fail to reach states or places outside the agent’s habitual behavior.The authors say additional learning or raw-action search may be needed, at greater time and resource cost.Source: §7 · p. 18
  • The authors identify intrinsic goal generation, hierarchical models, and integration of language and sound modalities as directions for future research.They suggest pretrained models such as GPT and CLIP could support more flexible goal modalities.Source: §7 · p. 18

A.1 Convolutional and de-convolutional networks structure

The appendix identifies tables specifying the convolutional and de-convolutional network structures used for image encoding and decoding.

A.1 Convolutional and de-convolutional networks structure

  • The paper identifies the convolutional and de-convolutional networks as the image-encoding and image-decoding structures, respectively.Source: §A.1 · p. 23
  • Table 1 specifies the structure of the convolutional neural network.Source: Table 1 · p. 23, §A.1 · p. 23
  • Table 2 specifies the structure of the de-convolutional neural network.Source: Table 2 · p. 23, §A.1 · p. 23

A.2 Repetitive goal converts to habit

The authors show that repeated goal-directed planning can be amortized into habitual behavior by learning an estimate of intention z from goal-directed trials. In the T-maze example, the resulting behavior consistently selects the top-right exit.

A.2 Repetitive goal converts to habit

  • A feedforward network is trained to estimate z from goal-directed planning trials, illustrating how a repeated goal can become fast habitual control.The example uses the same top-right-exit goal observation for the 10 planning trials.Source: §A.2 · p. 23
  • After amortized inference replaces the prior and posterior z values, the habitual behavior always chooses the top-right exit.The estimate is trained from 10 goal-directed trials for that exit; Figure 7 plots 30 deterministic-action trials before and after.Source: §A.2 · p. 23, Figure 7 · p. 23

A.3 Weighting of the complexity term

The complexity-term weight βz trades prediction accuracy against complexity. In the reported sweep, βz = 100 yielded the best overall behavior efficiency and diversity, while very large values were associated with very low diversity.

A.3 Weighting of the complexity term

  • Higher βz corresponds roughly to lower complexity and less accurate prediction, while lower βz corresponds to the reverse trade-off.Source: §A.3 · pp. 23–24
  • βz = 100 yielded the best overall behavior efficiency and diversity among the tested settings.The sweep tested βz values of 1, 10, 100, 1,000, and 10,000.Source: §A.3 · p. 24, Figure 8 · p. 24
  • When βz is very large, the agent tends to acquire habitual behavior with very low diversity.Source: §A.3 · p. 24

Related papers

Loading 2304.05008v1…