Source-linked AI summary
Deep Interactive Bayesian Reinforcement Learning via Meta-Learning
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, Katja Hofmann
TL;DR
Agents must maximize online return while learning about initially unknown other agents, but computing Bayes-optimal adaptation is generally intractable. MeLIBA meta-learns an approximate belief model and a policy conditioned on that belief; the authors report improved performance over model-free and other belief-based alternatives, especially when opponent type and rewards are not tightly coupled.
Problem
Agents must maximize online return while learning about other agents whose strategies are initially unknown, and exact IBRL solutions are intractable in most settings.
Source: §1 · p. 1, §2.3 · p. 2Method
MeLIBA jointly meta-trains a sequential, hierarchical VAE belief inference model and a policy conditioned on its approximate beliefs for a given prior over other agents.
Source: §1 · p. 1, §4.3 · p. 5Results
Takeaways & Limitations
Maintaining explicit beliefs over other agents helps adaptation compared with model-free approaches, especially when agent type and rewards are not tightly coupled.
Source: §6 · p. 7The setting resets other agents at each episode’s start and does not model their learning across episodes.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Agents that interact with other agents often do not know a priori what the other agents' strategies are, but have to maximise their own online return while interacting with and learning about others. The optimal adaptive behaviour under uncertainty over the other agents' strategies w.r.t. some prior can in principle be computed using the Interactive Bayesian Reinforcement Learning framework. Unfortunately, doing so is intractable in most settings, and existing approximation methods are restricted to small tasks. To overcome this, we propose to meta-learn approximate belief inference and Bayes-optimal behaviour for a given prior. To model beliefs over other agents, we combine sequential and hierarchical Variational Auto-Encoders, and meta-train this inference model alongside the policy. We show empirically that our approach outperforms existing methods that use a model-free approach, sample from the approximate posterior, maintain memory-free models of others, or do not fully utilise the known structure of the environment.
1 INTRODUCTION
Agents must adapt to unknown others while maximizing online return, making effective use of prior knowledge. MeLIBA meta-learns approximate belief inference and a policy that conditions on those beliefs, improving performance over several alternatives.
1 INTRODUCTION
- Ad-hoc adaptation requires learning about other agents while maximizing online return, particularly when data, time, and computation are limited.Prior knowledge about other agents’ strategies can support faster adaptation.Source: §1 · p. 1
- IBRL formalizes acting under uncertainty about other agents’ strategies, but exact solutions are intractable in most interesting problems and existing approximations are restricted to small environments and simple models.The framework balances exploration for information against exploitation for expected reward.Source: §1 · p. 1
- MeLIBA meta-learns approximate belief inference and a policy conditioned on the resulting beliefs to produce tractable, approximately Bayes-optimal agents for complex multi-agent settings.Both components are jointly meta-trained on a given prior over other agents.Source: §1 · p. 1
- The belief model combines sequential and hierarchical VAEs to represent permanent and temporal components of other-agent models, and the policy conditions on their latent variables.The inference model and policy are jointly meta-trained on a given prior distribution.Source: §1 · p. 1
- MeLIBA improves performance over sampling from approximate beliefs or using a memory-equipped model-free policy, and results support factoring the model according to environment and agent structure.The authors report that this approach can learn adaptive policies.Source: §1 · p. 1
2 BACKGROUND
The background formulates interaction as decision-making under uncertainty over other agents’ policies. The ideal belief-conditioned policy adapts within an episode, but computing it is generally intractable and the assumed prior excludes cross-episode learning by other agents.
2.1 Environment
- The setting is a Markov game with shared states, joint actions, transition dynamics, and individual rewards; agents choose actions simultaneously and may condition on interaction history.The described environment assumes full observability of states and other agents’ actions.Source: §2.1 · p. 2
2.2 Objective
- The objective is to maximize agent 1’s online return within a finite horizon while learning about unknown opponents drawn from a prior over their policies.Agent 1 has no control over the other agents’ policies and evaluates performance in expectation over the prior.Source: §2.2 · p. 2
2.3 Interactive Bayesian RL
- IBRL maintains a posterior over opponents’ strategies from interaction history and conditions actions on both the environment state and the current belief.The resulting belief is part of the hyper-state, and information gathering trades off against exploitation according to expected return within the horizon.Source: §2.3 · p. 2
- A Bayes-optimal policy maximizes expected return in the belief MDP and adapts to other agents given a prior over their strategies, but computing it analytically is intractable except for the smallest tasks.Existing approximation methods are also restricted to small environments.Source: §2.3 · p. 2
2.4 Prior Beliefs over Other Agents
- The prior allows opponents to adapt within a game, but their memory resets between games, so learning across episodes is outside the modeled setting.Adding a prior over opponents’ learning algorithms is described as possible but would make inference and planning harder.Source: §2.4 · p. 2
3 RELATED WORK
Related work addresses adaptive behavior through Bayesian RL, agent models, and meta-learning. MeLIBA focuses on online Bayes-optimal adaptation with explicit beliefs over potentially non-stationary opponents, rather than relying on sampled beliefs or conflating agent structure with the environment.
3 RELATED WORK
- The IBRL formulation explicitly models beliefs over other agents, whereas modeling agents as part of the environment conflates sources of stochasticity; existing IBRL approximations remain restricted to small settings or restrictive assumptions [11, 13, 25, 26].MeLIBA targets online return while learning, rather than behavior that cannot generally be executed from the first timestep because opponents’ strategies are unknown.Source: §3 · p. 3
- MeLIBA represents agent types as learned continuous latent variables and separates shared behavior from permanent and temporal agent-specific features, building on character and mental-state modeling.Unlike the observational setting of Rabinowitz et al., MeLIBA trains a policy to interact with and influence others using inferred beliefs.Source: §3 · p. 3
- Unlike Papoudakis and Albrecht [39], who assume Markov opponents and condition policies on sampled beliefs, MeLIBA models history-dependent opponents and conditions on beliefs to account for uncertainty.The authors state that sampling from the approximate belief does not permit learning a Bayes-optimal policy.Source: §3 · p. 3
- Q-Mixing transfers Q-values to new opponent mixtures and uses an opponent classifier, while MeLIBA can represent continuous distributions over other agents’ policies.Q-Mixing is described as relying on a discrete, small set of other agents.Source: §3 · p. 3
- The model-free recurrent alternative conditions on interaction history, but the authors report it performs worse than explicitly modeling other agents’ actions as an inductive bias.Domain-specific adaptive methods have also been developed for games including Avalon, Hanabi, and Poker.Source: §3 · p. 3
4 META LEARNING INTERACTIVE BAYESIAN AGENTS
MeLIBA meta-learns approximately Bayes-optimal agents using a hierarchical sequential VAE to infer beliefs about other agents and a policy conditioned on those beliefs. Its design models both persistent agent traits and changing behavior, then trains inference and policy components together.
4 META LEARNING INTERACTIVE BAYESIAN AGENTS
- MeLIBA jointly meta-trains a hierarchical sequential VAE for beliefs over other agents and a policy conditioned on environment states and approximate beliefs.The method targets approximately Bayes-optimal adaptation to other agents.Source: §4 · p. 3
4.1 Modelling Other Agents
- Each other agent is represented by a persistent character and a changing mental state, allowing the model to represent policies conditioned on interaction history.The mental state can encode observations such as counts of others’ cooperation.Source: §4.1 · p. 4
4.2 Approximate Belief Inference
- Because exact optimization is intractable, the model uses a sequential VAE to infer beliefs and predicts other agents’ future actions from the current posterior.The action-prediction term factors across agents under the independence assumption, and a recurrent model represents changing future mental states.Source: §4.2 · p. 4
- A hierarchical latent structure lets temporal states depend on permanent types, while using the previous posterior as the prior encourages beliefs to change slowly over time.The encoder produces the permanent latent from earlier layers and the temporal latent from deeper layers.Source: §4.2 · p. 4
4.3 Meta-Learning Bayes-Adaptive Policies
- The policy receives both the environment state and the approximate belief, so it can account for uncertainty about other agents when choosing actions.The Gaussian posterior is represented by its mean and variance.Source: §4.3 · p. 5
4.4 Putting Everything Together
- The implementation uses a hierarchical encoder, recurrent action decoder, and belief-conditioned policy, alternating PPO policy updates with VAE updates.The RL loss is not backpropagated through the encoder, which the authors say saves computation.Source: §4.4 · p. 5
5 EMPIRICAL EVALUATION
The evaluation examines MeLIBA’s latent structure and compares it with adaptive-policy baselines and architectural ablations. In the Game of Chicken and Treasure Hunt, results highlight the value of modeling opponents’ strategies and the limits of simpler or model-free alternatives.
Game of Chicken
- The 13-round Game of Chicken requires inferring and remembering one of three Tit-for-Tat strategies; the Bayes-optimal strategy initially swerves, then exploits the inferred type.The three opponents swerve after the other player has swerved once, twice, or three times in a row.Source: Game of Chicken · p. 5
- MeLIBA separates opponent types after 1–2 timesteps, and its temporal latent learns to count swerves.The separation occurs after the other agent begins swerving; latent-belief uncertainty is high initially and decreases as information arrives.Source: Game of Chicken · p. 6
- In Game of Chicken, RL2 and MeLIBA quickly learn Bayes-optimal behavior, while MeLIBA-m cannot solve the task and LIOM performs poorly.MeLIBA-m uses only a fixed latent and a feed-forward decoder; the authors attribute its failure to an inappropriate opponent model, and suggest LIOM’s sampling and small latent space may destabilize PPO.Source: Game of Chicken · p. 6, Figure 2 · p. 6
5.2 Treasure Hunt
- Treasure Hunt tests predictive structure in a 10 × 10 grid with a 100-step horizon, three hard-coded opponent strategies, and comparisons against model-free RL2, LIOM, and an average policy.The evaluation also studies architectural choices by ablating hierarchical and sequential structure.Source: §5 · p. 5, §5.2 · p. 6
- In Treasure Hunt, MeLIBA, MeLIBA-m, and LIOM show no significant performance difference; a permanent latent can represent agent type and the last coin collected.The authors suggest MeLIBA-m when opponents’ actions are known not to depend on history, and report no performance loss from using full MeLIBA when that is unknown.Source: §5.2 · p. 7
- MeLIBA significantly outperforms RL2 in Treasure Hunt, while RL2 adapts to some extent but performs better than the average policy.A classifier finds that RL2’s recurrent state represents opponent strategy to some extent, but this representation does not translate into adaptation that beats the feed-forward average policy.Source: §5.2 · p. 7
6 DISCUSSION & CONCLUSIONS
MeLIBA meta-learns approximate Bayes-optimal policies and hierarchical representations of other agents, supporting adaptation across several environments. The conclusion highlights explicit beliefs’ advantage in some settings and identifies evolving agents as future work.
6 DISCUSSION & CONCLUSIONS
- MeLIBA meta-learns approximately Bayes-optimal policies for a given distribution of other agents and scales to complex multi-agent environments.Its model learns a hierarchical latent representation separating agents’ permanent types from temporal internal states.Source: §6 · p. 7
- Explicit beliefs over other agents help compared with model-free approaches, especially when agent type is not tightly coupled to rewards.The paper reports adaptation through learning latent representations by predicting other agents’ future actions and conditioning the policy on them.Source: §6 · p. 7
- The authors also point to learning more informative priors from expert data or pre-trained agents, and to applications involving human cooperation or simulated traffic.If such data are unavailable, they suggest training all agents simultaneously to generalize to agents unseen during training.Source: §6 · p. 7
- The current setting assumes other agents can adapt within an episode but reset at its beginning.The authors identify simultaneous training of all agents to generalize to unseen agents as a future direction; they note that maintaining diversity and avoiding overfitting are difficult.Source: §6 · p. 7
- Modeling agents that learn over time remains future work, with possible approaches including inferring a known learning algorithm or adding a third latent level.Such an extension would require reasoning about how the agent’s actions and learning affect the other agents’ learning.Source: §6 · p. 7
A ELBO DERIVATION
The appendix derives an ELBO for predicting another agent’s future actions. The derivation expands the left-hand side across timesteps and uses the previous posterior as the next prior, with a Gaussian initialization.
A ELBO DERIVATION
- The appendix derives the ELBO for modeling another agent’s future actions.The derivation is presented as a sequence of expansions and timestep updates.Source: A ELBO DERIVATION · p. 10
- The left-hand side is expanded before the derivation is repeated across subsequent timesteps.The text specifies repeating steps (18)–(22) for t + 1 through H.Source: A ELBO DERIVATION · p. 10
- For the right-hand side, the previous posterior becomes the new prior, initialized at the first timestep with univariate Gaussian priors N(0, 1).The posterior is written as q(m, m_t | τ: t−1).Source: A ELBO DERIVATION · p. 10
B.1 Game of Chicken
The Chicken Game appendix examines latent-variable ablations and learning curves. Its findings connect architecture choices to how interaction history and temporal structure are represented.
B.1 Game of Chicken
- In latent-variable ablations, the permanent latent m counts interactions less consistently than the temporal latent m_t.The non-hierarchical model also places some temporal structure in m, unlike the desired separation in MeLIBA.Source: §B.1 · p. 11
- All MeLIBA architectures solve the Chicken Game except the version using only a permanent latent variable m.The failure is attributed to a mismatch: the other agent conditions on interaction history, while the feed-forward action decoder uses only the last state and actions.Source: §B.1 · p. 11
- LIOM performs better with RL-loss backpropagation or a larger latent dimension, but remains below MeLIBA.The authors suggest that passing sampled latents rather than their mean and variance may make policy inputs noisy and training unstable.Source: §B.1 · p. 11
B.2 Treasure Hunt
The Treasure Hunt appendix specifies a deterministic gridworld with coin-collection and banking rewards, alongside hard-coded agent behaviors. It also describes learning-curve and agent-type classification procedures.
B.2 Treasure Hunt
- The Treasure Hunt gridworld uses a 10 × 10 grid, five discrete actions, deterministic transitions, and rewards for collisions and coin collection or delivery.Rewards are −0.1 for collisions, +0.1 for picking up a coin, +1 for delivery to the matching bank, and −1 for delivery to the wrong bank.Source: §B.2 · p. 11
- Hard-coded agents head directly to their preferred coin color without avoiding other-colored coins or the other agent.If they collect a coin of the wrong color, they deliver it to the correct bank before returning to their preferred color.Source: §B.2 · p. 11
- Treasure Hunt learning curves use three seeds per method with shaded 95% confidence intervals, while type-classifier accuracy is tested on 20 rollouts.The classifier is fit using 80 test rollouts per trained model; the shaded areas show one standard deviation across three trained models.Source: §B.2 · p. 11
- Table 1 reports runtimes in hours, which the authors describe as rough guides for comparing relative method runtimes.Parallel experiments on each GPU contributed to runtime variation.Source: Table 1 · p. 12
B.3 Runtimes
Runtime figures are rough guides because experiments ran in parallel on each GPU, causing variation; they are useful for comparing methods' relative runtimes.
B.3 Runtimes
- Parallel experiments on each GPU caused runtime variation, so the reported runtimes are rough guides for comparing methods' relative times.The authors ran between 1 and 4 experiments per GPU in parallel.Source: §B.3 · p. 12
B.4 Hyperparameters
Table 2 lists MeLIBA hyperparameters, and the accompanying text describes encoder dimensions, recurrent-network settings, and hyperparameter selection.
B.4 Hyperparameters
- Table 2 lists the MeLIBA hyperparameters.Source: §B.4 · p. 12, Table 2 · p. 13
- The first encoder layer separately maps state/action/rew to 32/16/16 units in Chicken Game and 64/16/16 in Treasure Hunt.Source: §B.4 · p. 12
- MeLIBA uses GRUs in recurrent network parts, and the hidden size of its recurrent layer is denoted R.Source: §B.4 · p. 12
- RL2 training used a three-layer recurrent network with hidden size 128, and hyperparameters were selected by a simple linesearch.The authors refer to a table for the values considered.Source: §B.4 · p. 12