Source-linked AI summary
Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models
Xi Xiong, Lu Liu, Yiming Li, Jakob Foerster
TL;DR
In partially observable offline MARL, agents lack global states and limited data coverage can make learned world models unreliable. This paper infers latent states and iteratively refines a Bayesian world model using MPIL, establishing a monotone regret bound and reporting strong policy performance across benchmark settings.
Problem
Offline MARL must learn policies when agents have only decentralized observations, while limited dataset coverage can make model-based methods unreliable and existing approaches leave world-model reliability and refinement underexplored.
Source: §1 · p. 1Method
The framework infers latent global states from local observation-action histories and uses MPIL to assess and iteratively refine a Bayesian posterior world model using offline data.
Source: §1 · p. 2, §4.1 · p. 4Results
The method achieves the highest normalized return in eight of twelve experimental settings and ranks second in the other four; MPIL-guided refinement improves model prediction and policy performance on partially observable multi-agent MuJoCo tasks.
Source: §1 · p. 2, §6.1 · p. 7, §7 · p. 9Takeaways & Limitations
The analysis connects world-model reliability and offline dataset size and coverage to policy regret in partially observable offline MARL.
Source: §1 · p. 2The stated regret result assumes expected MPIL contracts at a parametric rate and the concentrability coefficient is uniformly bounded across dataset sizes.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
Model-based offline multi-agent reinforcement learning (MARL) provides a promising approach to multi-agent decision-making under partial observability, where the global state is unavailable from offline data. However, limited dataset coverage can lead to substantial out-of-distribution (OOD) model errors, which may mislead policy optimization and degrade downstream performance. Existing model-based offline MARL methods typically rely on access to global states or mitigate model errors through pessimistic or conservative policy optimization, leaving the reliability and refinement of learned world models less explored. We propose a Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data. Building on posterior information loss, we extend this criterion to the multi-agent setting as multi-agent posterior information loss (MPIL), which provides a reliability measure for world model refinement and offline evaluation. Theoretically, we establish an upper bound on policy regret that is monotonic in MPIL and further characterize its dependence on the size and state-joint-action coverage of the offline dataset. Experiments on partially observable multi-agent variants of D4RL MuJoCo tasks demonstrate competitive policy performance across diverse offline data conditions, while MPIL-guided refinement improves world model prediction and downstream policy performance. These results support a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.
1 INTRODUCTION
Offline MARL must learn effective policies from historical data when agents lack access to global states. This work introduces a latent Bayesian world model refined with MPIL and connects model reliability and offline data coverage to policy regret.
1 INTRODUCTION
- Offline MARL remains challenging when decentralized observations do not reveal the global state needed for reliable coordination and policy learning.Offline learning avoids the safety risks and trial-and-error costs of extensive online interaction when an accurate simulator is unavailable.Source: §1 · p. 1
- Limited offline-data coverage can make learned models unreliable, allowing policy optimization to exploit errors in out-of-distribution regions.Existing model-based approaches primarily mitigate these errors through pessimistic or conservative policy optimization.Source: §1 · p. 1
- Prior methods mainly mitigate model errors during policy optimization rather than refine the model; SOReL links discrepancy to regret but is single-agent and does not use PIL for refinement.The paper identifies a lack of a principled criterion linking world-model reliability to policy performance and safety.Source: §1 · p. 2
- The framework infers latent global states from local observation-action histories, models posterior uncertainty, and iteratively refines the world model for policy learning and offline regret estimation.The refinement, learning, and assessment loop uses no further interaction with the true environment.Source: §1 · p. 2
- MPIL-guided refinement improves world-model prediction and policy performance on partially observable multi-agent MuJoCo benchmarks, while estimated regret tracks true regret as offline data increases.The theory bounds true regret by a monotonic function of MPIL and characterizes its dependence on dataset size and state–joint-action coverage.Source: §1 · p. 2
2 PRELIMINARIES
The preliminaries frame cooperative partial-observation problems as Dec-POMDPs and define the dynamics and observation components of a world model. The offline dataset consists of local observations, joint actions, and shared rewards, not global states.
2 PRELIMINARIES
- A Dec-POMDP models cooperative decision-making in which agents receive local observations and share a system-level objective.Each agent selects actions from its local history, while the agents’ actions form a joint action and the environment supplies a shared reward.Source: §2 · p. 2
- The world model combines a transition-reward model with an observation model for generating agents’ local observations.The transition-reward and joint-observation distributions are represented by the model components described in the preliminaries.Source: §2 · p. 2
- The offline dataset contains local observations, joint actions, and shared rewards, with no true global state available during training or execution.The dataset is indexed by transitions, with N denoting the number of transitions.Source: §2 · p. 2
3 RELATED WORK
Related work covers model-free and model-based offline RL, policy assessment, and posterior information loss. The paper targets the underexplored task of assessing and refining world models from fixed offline data in partially observable multi-agent settings.
3 RELATED WORK
- Model-free offline RL methods address distribution shift with constraints, conservative value estimation, or pessimism, but do not explicitly model dynamics for reasoning about long-horizon consequences.Multi-agent extensions address coordination through CTDE, decentralized optimization, and global-to-local regularization.Source: §3 · p. 3
- Model-based offline RL uses learned dynamics for imagined-trajectory policy optimization, but many methods assume global states are available, limiting their fit to partially observable offline MARL.Prior work uses concatenated local observations [27] or latent representations from observation-action histories [29] [30].Source: §3 · p. 3
- Policy assessment methods estimate or select policies, while SOReL uses PIL to connect posterior model discrepancy with regret but does not update model parameters with PIL.The paper’s MPIL-guided framework assesses and iteratively refines latent world models without access to global states.Source: §3 · p. 3
4 METHODOLOGY
The framework infers latent global states from decentralized histories, learns a Bayesian world model, and uses MPIL to assess and refine it before offline policy optimization. It combines history-based representation learning, posterior modeling, iterative refinement, and CTDE policy learning.
4 METHODOLOGY
- The methodology first infers latent global states and learns a Bayesian world model, then uses MPIL for model assessment and refinement before cooperative policy optimization.The framework is depicted as a comparison of traditional and MPIL-guided model-based offline MARL.Source: §4 · p. 3, Figure 1 · p. 3
4.1 LATENT GLOBAL STATE INFERENCE
- A GRU infers a compact, history-dependent latent global state from joint observation-action histories, while an auxiliary decoder reconstructs observations to encourage observation-relevant representations.The resulting latent-state-augmented dataset is used for world-model learning.Source: §4.1 · p. 4
4.2 WORLD MODEL FORMULATION
- The Bayesian world model represents transition-reward dynamics jointly and local observation dynamics separately, with local observations conditionally independent given latent state and joint action.The resulting posterior characterizes uncertainty over the complete world model, which is trained using posterior-averaged NLL.Source: §4.2 · p. 4
- MPIL measures posterior-averaged discrepancy between true and learned world-model distributions over dataset-covered state-action pairs, and posterior-averaged NLL provides a tractable way to estimate it.Under a fixed offline data distribution, changes in NLL across refinement stages track changes in MPIL.Source: §4.2 · pp. 4–5
4.3 LEARNING AN APPROXIMATELY BAYES-OPTIMAL POLICY
- Refinement warm-starts the ensemble, incorporates successive offline-data subsets, and minimizes empirical NLL while monitoring MPIL changes on a fixed held-out validation set.Refinement stops when the change meets a convergence tolerance or all data subsets have been incorporated.Source: §4.3 · pp. 5–6
- After refinement, SAC under CTDE optimizes cooperative policies with a centralized critic using inferred latent states and joint actions, while decentralized actors use local observation histories.The resulting policy is evaluated fully offline using regret.Source: §4.3 · p. 6
5 REGRET ANALYSIS
The section connects posterior world-model reliability, measured by MPIL, to policy regret and offline evaluation, while characterizing how dataset size and coverage shape guarantees.
5 REGRET ANALYSIS
- The analysis bounds Bayes-optimal policy regret monotonically in MPIL and characterizes how the guarantee depends on offline data size and coverage.These results connect world-model reliability to policy-performance guarantees.Source: §5 · p. 6
5.1 CONTROLLING REGRET WITH THE MPIL
- Under Assumption A.1 and bounded coverage, MPIL upper-bounds the regret of the policy learned from the world model, and smaller MPIL yields a tighter guarantee.The concentrability coefficient bounds distribution shift between policy-induced state–action distributions and the offline data distribution.Source: §5.1 · p. 6
5.2 EXPECTED REGRET UNDER MPIL CONTRACTION
- Expected regret is O(N^-1/2) when expected MPIL contracts at a parametric rate and coverage remains uniformly bounded.The guarantee tightens with more data only when policy-relevant state–joint-action regions remain sufficiently covered.Source: §5.2 · pp. 6–7
5.3 EMPIRICAL REGRET APPROXIMATION
- Because true regret cannot be evaluated directly, the section defines a normalized empirical approximation using returns predicted by the learned posterior world model.The approximation supports fully offline policy-quality evaluation but does not replace theoretical regret.Source: §5.3 · p. 7
6 EXPERIMENTS
Experiments evaluate policy performance, offline regret estimation, and MPIL-guided refinement on partially observable multi-agent D4RL tasks. The method performs consistently across data conditions, while refinement improves model predictions and policy returns.
6 EXPERIMENTS
- The evaluation uses 12 task–dataset combinations across Halfcheetah, Hopper, and Walker2d, with random, medium, full-replay, and expert data and no global states.Policies are compared with BC, OMAR, and MOMA-PPO.Source: §6 · p. 7, §6.1 · p. 7
6.1 OFFLINE POLICY PERFORMANCE
- The method achieves the highest normalized return in eight of twelve settings and ranks second in the remaining four, with clear advantages on random and full-replay data.The comparison covers normalized performance on multi-agent D4RL datasets.Source: §6.1 · p. 7, Table 1 · p. 8
6.2 VALIDATION OF MPIL AND APPROXIMATE REGRET
- Across Walker2d dataset sizes, δMPIL and approximate regret decrease early and stabilize near 0.5M samples, tracking the normalized true-regret trend.The stabilization provides a practical indication of data sufficiency without online interaction.Source: Figure 2 · p. 8, §6.2 · p. 8
6.3 EFFECT OF MPIL-GUIDED WORLD MODEL REFINEMENT
- On Walker2d, MPIL-guided refinement reduces JNRMSE on mixed validation data, while refined-model policies achieve higher returns across all four dataset types.The prediction-error improvement is particularly evident under the mixed shifted data distribution; policy gains are especially pronounced on full-replay data.Source: §6.3 · pp. 8–9, Figure 3 · p. 9
7 CONCLUSION
The conclusion presents a Bayesian latent-world-model framework for fully offline MARL under decentralized partial observability, with MPIL connecting model reliability to regret and refinement. It identifies generalization and cross-task model reuse as future directions.
7 CONCLUSION
- The framework infers latent states from local histories, uses MPIL to refine and evaluate posterior world models, and establishes a monotone regret bound dependent on MPIL and dataset size and coverage.Experiments show competitive policy performance across dataset qualities and improvements in prediction and downstream performance from MPIL-guided refinement.Source: §7 · p. 9
- Future work will study zero-shot generalization to unseen agent configurations, interaction structures, and environment dynamics, and shared or conditioned models across tasks.MPIL may also guide selection of unreliable regions and informative offline samples for targeted updates.Source: §7 · p. 9
A.1 ALGORITHM
The algorithm builds latent-state datasets and an initial world model, then incorporates additional offline subsets and applies an MPIL stopping criterion before sampling a posterior model and imagined trajectory.
A.1 ALGORITHM
- Construct the first latent-state dataset from the initial offline subset and learn initial ensemble parameters by minimizing the world-model loss.Source: §A.1 · p. 13
- Iteratively add offline subsets and test MPIL against the convergence tolerance, recording the terminating stage when the criterion is met.Source: §A.1 · p. 13
- At the terminating stage, construct the empirical posterior and sample a world model together with an imagined trajectory.Source: §A.1 · p. 13
A.2.1 WORLD MODEL IMPLEMENTATION
The appendix specifies a latent posterior world-model and its refinement and policy-learning procedures, alongside theoretical conditions and held-out evidence on MPIL-guided prediction.
A.2.1 WORLD MODEL IMPLEMENTATION
- A recurrent encoder maps joint observation–action histories to latent states; an ensemble of probabilistic networks models posterior uncertainty over transitions, rewards, and observations.Ensemble members are selected by holdout performance and optimized with empirical MPIL.Source: §A.2.1 · p. 13
A.2.2 WORLD MODEL REFINEMENT
- Each refinement stage warm-starts from the preceding ensemble and uses empirical MPIL optimization with holdout validation and early stopping.Source: §A.2.2 · pp. 13–14
A.2.3 POLICY LEARNING WITH THE REFINED WORLD MODEL
- Policy learning uses SAC-based CTDE, combining imagined transitions from the refined ensemble with offline data for actor and critic updates.Source: §A.2.3 · p. 14
A.2.4 BASELINE IMPLEMENTATION
- The baselines are deterministic behavior cloning, OMAR, and MOMA-PPO, whose synthetic rollouts are uncertainty-penalized and terminated when model reliability is insufficient.OMAR uses conservative critic regularization and candidate-action actor rectification; MOMA-PPO follows Barde et al..Source: §A.2.4 · p. 15
A.2.5 MULTI-AGENT, LOCAL-OBSERVATION, AND DATASET CONFIGURATION
- Decentralized actors receive local observations, while the union reconstructs the original D4RL observation; full-replay datasets contain no method-generated or evaluation-environment trajectories.Source: §A.2.5 · pp. 15–16
A.3 PROOF
- The analysis equates fixed-data NLL differences with MPIL and bounds regret under latent-sufficiency, MPIL-contraction, and policy-coverage assumptions.The coverage assumption concerns regions visited by the policies in the regret comparison, not all possible policies.Source: §A.3 · p. 16, §A.3.1 · p. 17, §A.3.2 · p. 17
A.4 WORLD MODEL PREDICTIVE PERFORMANCE
- MPIL-guided updates concentrate next-observation and reward predictions around ideal predictions on Walker2d, with reduced dispersion.The evaluation uses a held-out test set; the passage reports similar improvement for reward prediction.Source: Figure 4 · p. 20, §A.4 · p. 20