PaperlayerView exact revision on arXiv

Source-linked AI summary

NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games

Tomáš Holeček, Viliam Lisý

arXiv:2609.01549v1cs.LG

TL;DR

Competitive imperfect-information games make decentralized world-model learning difficult because opponent policies are entangled with environment dynamics. NashDreamer addresses this with centralized MARSSM training and decentralized policy optimization, improving early-to-mid training convergence while exposing posterior-collapse challenges in high-chance-stochasticity environments.

  • Problem

    Decentralized world models in competitive imperfect-information games face identifiability barriers, while opponent-induced non-stationarity complicates learning sound environment dynamics.

  • Method

    NashDreamer pairs a centralized MARSSM with decentralized, game-theoretic actor-critic optimization and can use alternatives such as RNaD or MMD.

  • Results

    NashDreamer significantly improves convergence in early-to-mid training and improves sample efficiency across evaluated domains compared with model-free baselines.

  • Takeaways & Limitations

    Centralized world modeling provides a sample-efficiency layer for imperfect-information games while remaining compatible with game-theoretic policy-gradient methods.

  • Takeaways & Limitations

    High chance stochasticity remains challenging because stabilizing regularization can induce posterior collapse or, when removed, representation drift; Leduc Poker exhibits related model errors and late-stage deterioration.

Abstract

from arXiv · show

Model-based reinforcement learning (MBRL) has achieved remarkable results in single-agent domains, yet its extension to competitive imperfect information games (IIGs) remains underexplored. In multi-agent settings, opponent-induced non-stationarity complicates the learning process, and decentralized model learning faces severe identifiability barriers, which we argue make centralized model learning a mathematical necessity. Building on this analysis, we propose NashDreamer, a principled MBRL framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent State-Space Model (MARSSM) that decouples environment dynamics from the effect of players' strategies on their individual observations. NashDreamer is designed to use arbitrary policy gradient algorithms and inherits their convergence guarantees towards Nash equilibria under an idealized model. Empirical evaluations across four benchmark games demonstrate that NashDreamer substantially improves sample efficiency over model-free baselines early in the training. Finally, we theoretically analyze the architecture's optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.

1 Introduction

NashDreamer addresses the difficulty of applying model-based reinforcement learning to competitive imperfect-information games, where opponent policies create non-stationarity and decentralized models face identifiability barriers. It proposes centralized training with a MARSSM and reports improved early sample efficiency over model-free baselines.

  • Decentralized learning cannot distinguish environment stochasticity from a shifting opponent policy, motivating centralized training as mathematically necessary for a sound stationary world model.
  • NashDreamer is a CTDE framework for two-player zero-sum imperfect-information games built around a centralized Multi-Agent Recurrent State-Space Model.
  • The MARSSM is designed to combine with game-theoretically sound policy-gradient methods and support convergence toward Nash equilibria.
  • NashDreamer improves sample efficiency over model-free baselines across varied domains, especially during early training.
  • NashDreamer is a sample-efficiency layer intended to compare policy-gradient method X against NashDreamer[X] at equal real-environment interaction budgets.

2 Background

The paper formalizes two-player zero-sum imperfect-information games and reviews Dreamer-style world modeling and game-theoretic policy optimization. Dreamer learns latent dynamics for imagination, while regularized policy methods address non-convergence in imperfect-information games.

  • Game setting: The game uses joint actions, joint observations, chance transitions, zero-sum rewards, imperfect information, and perfect recall for two players.
  • Game setting: Each player conditions policy decisions on their action-observation history and information set rather than the unobserved full history.
  • Game setting: Collective observability assumes the joint action-observation history identifies the true underlying history, without giving either player perfect information individually.
  • Dreamer background: Dreamer trains an RSSM with posterior encoding from observations and prior dynamics prediction, then generates imagined trajectories for actor-critic optimization.
  • Game-theoretic optimization: Regularized Nash Dynamics and related methods address policy-gradient cycling in imperfect-information games; RNaD provides convergence to a Nash equilibrium.

3 World Models in Games: The Necessity of Centralized Training

Decentralized world models in adversarial games cannot reliably separate opponent behavior from objective environment dynamics. Centralized training resolves this by conditioning the model on joint actions while retaining decentralized execution.

  • Without observing the opponent’s action, decentralized models face action aliasing when different opponent actions produce indistinguishable transition evidence.
  • A decentralized model can represent a joint system against one opponent rather than the objective environment dynamics, creating policy-physics entanglement.
  • Strong opponents can exploit this identifiability gap by selecting actions that maximize the mismatch between physical reality and the agent’s internal model.
  • Centralized training conditions the world model on joint actions, decoupling environment physics from the opponent’s shifting strategy while preserving ego-only observations at test time.
  • In Rock-Paper-Scissors, a tabular world model requires only 9 transitions before policy training can proceed entirely in imagination.

4 NashDreamer

NashDreamer combines a centralized MARSSM for environment dynamics with decentralized infoset representations and game-theoretic actor-critic optimization. Its theoretical analysis establishes idealized convergence while exposing posterior collapse and representation-drift trade-offs in stochastic environments.

  • Architecture: The MARSSM learns centralized dynamics from joint actions and observations, while separate recurrent infoset models represent each player’s private action-observation history.
  • Architecture: The centralized model state aggregates both players’ private observations, while each infoset model produces a fixed-size representation for decentralized decision-making.
  • Training objective: NashDreamer extends Dreamer prediction losses with rewards, termination flags, legal actions, per-player observations, and infoset reconstruction losses.
  • Imagination: During imagination, prior dynamics generate states and reconstructed observations update the infoset models, making the observation decoder an essential generative component.
  • Actor-critic: The decentralized actors use individual infosets while the centralized critic uses the joint representation; RNaD is preferred because its convergence guarantee composes with the model result.
  • Model convergence: An infoset-isomorphic model is strictly better in expected loss than branch-duplicated or information-partition-violating models, and idealized composition with exact RNaD dynamics yields Nash-equilibrium convergence.
  • Model convergence: Posterior collapse can outperform the structurally correct model under some stochastic environments, especially when chance outcomes are high-stochasticity and low-impact for prediction.
  • Model convergence: Dreamer-style MBRL faces a trade-off: KL regularization can reduce representation non-stationarity yet drive optimization toward degenerate solutions, while removing it permits representation drift.

5 Empirical Evaluation

The evaluation compares NashDreamer with matched model-free optimizers using real environment interaction, exact NashConv where tractable, and budgeted approximate NashConv on larger domains. NashDreamer generally improves early sample efficiency, while stochastic-model errors can cause deterioration.

  • Evaluation protocol: NashDreamer is evaluated against identical model-free optimizers using real environment steps, with replay-buffer RNaD as an additional comparison.The model-free baselines include RNaD variants and MMD; MMD is also run inside imagination as NashDreamer[MMD].
  • Exact NashConv domains: In Goofspiel-5, NashDreamer reaches comparable asymptotic performance using substantially fewer environment interactions than model-free RNaD.Replay failed to improve RNaD’s sample efficiency.
  • Exact NashConv domains: In Leduc Poker, NashDreamer initially outpaces the model-free baseline but later deteriorates because the world model hallucinates impossible public-card events.The structural error persists throughout training, and the one-way KL loss does not prevent it.
  • Exact NashConv domains: In Phantom Tic-Tac-Toe, NashDreamer[RNaD] reaches model-free RNaD’s final exploitability using roughly a quarter of the gradient steps.Both optimizers improve inside imagination; replay improves RNaD but remains above NashDreamer[RNaD].
  • Approximate NashConv on larger domains: Approximate NashConv on Goofspiel-13 and Battleship is a budgeted lower bound, comparable only across methods evaluated at the same best-response budget.Exact NashConv is intractable in these domains, so best responses are estimated with PPO search.
  • Approximate NashConv on larger domains: On larger domains, NashDreamer[RNaD] reaches lower approximate NashConv after 20k gradient steps than model-free RNaD reaches at 30k in Goofspiel-13.NashDreamer leads at every checkpoint in Goofspiel-13; NashDreamer[MMD] similarly surpasses MMD at 20k versus 30k in Battleship.

6 Related Work

Related work spans model-based reinforcement learning, multi-agent Dreamer extensions, and game-theoretic reinforcement learning. NashDreamer is positioned against approaches that either target single-agent or cooperative settings, lack game-theoretic guarantees, or require explicit game-tree access.

  • Model-based RL: MuZero and Dreamer learn latent world models for planning or imagination, primarily in perfect-information games and single-agent settings.NashDreamer builds on the online DreamerV3 architecture rather than the offline DreamerV4 paradigm.
  • Multi-agent Dreamer: MAMBA and DreamerCF extend Dreamer to cooperative settings, while competitive systems use opponent embeddings or single-player modeling with population-based self-play.The cited multi-agent approaches use policy-gradient actor-critics without game-theoretic convergence guarantees.
  • Game-theoretic RL: CFR, DREAM, RNaD, PSRO, and MMD represent complementary game-theoretic or model-free approaches to imperfect-information games.CFR requires explicit game-tree access, whereas DREAM removes that requirement through Monte Carlo sampling.

7 Conclusion

NashDreamer combines a centralized latent world model with decentralized game-theoretic actor-critics and improves early-to-mid training convergence. High chance stochasticity remains a major challenge because stability and posterior collapse are in tension.

  • Conclusion: NashDreamer pairs the centralized MARSSM world model with a decentralized game-theoretic actor-critic to address multi-agent identifiability barriers.The framework is designed for two-player zero-sum imperfect-information games.
  • Conclusion: High chance stochasticity remains challenging because stabilizing the model conflicts with avoiding collapse and degenerate minima.Future work also includes relaxing the t = 0 unrolling constraint and validating the architecture in massive environments.

8 Broader impacts

The paper defines the game, optimization, trajectory, and world-model concepts underlying NashDreamer’s analysis. It also specifies how latent states should preserve both game-tree structure and players’ information partitions.

  • Policy optimization: RNaD regularizes rewards against a reference policy and periodically updates that policy, yielding provable convergence to a Nash equilibrium.MMD instead regularizes policy updates toward a magnet policy and the previous iterate.
  • Latent representations: A latent trajectory records recurrent and stochastic latent states, joint actions, reconstructed observations, and latent infoset embeddings produced by model unrolling.
  • World-model assumptions: The model’s theoretical mapping excludes chance-node histories because environmental stochasticity is represented as latent transitions rather than explicit nodes.
  • Isomorphism conditions: Tree-structure isomorphism requires a bijection from non-chance histories to reachable latent states with matching action sequences and observations.Infoset isomorphism additionally requires corresponding latent infosets to preserve the game’s information equivalence relation.
  • Failure modes: Branch duplication and information-partition violations describe distinct ways a world model can misrepresent histories or players’ information despite matching physical dynamics.

B.2 Proofs

The proofs show that infoset-isomorphic models are optimal among structurally sound alternatives, while branch duplication and information-partition violations incur strictly higher losses. They also establish convergence under idealized policy optimization and expose posterior collapse as a lower-loss degenerate solution in high-entropy environments.

  • Theorem 2: Theorem 2 proves that the infoset-isomorphic model M∗ has lower expected loss than both branch-duplicated Mb and information-violating Ml.The comparison holds under any joint sampling policy µ.
  • Isomorphic model: For M∗, deterministic prediction and unifier losses vanish, leaving KL penalties equal to the environment’s inherent chance entropy.The recurrent state preserves predecessor history and joint action, reducing expected dynamics loss to H(o | ˆh).
  • Branch duplication: Branch duplication preserves reconstructions and marginal chance probabilities but creates irreducible unifier variance because identical infosets map to multiple latent targets.Its global minimum is therefore strictly higher, although the artifact can remain benign for downstream policy optimization.
  • Information partitioning: Information-partition violations alias distinct true infosets, forcing deterministic decentralized decoders to incur strictly positive reconstruction error.Individual infoset targets are theoretically necessary because centralized targets could otherwise rely on one fully informed player.
  • Policy convergence: Under a fixed converged M∗ or Mb and exact RNaD dynamics, NashDreamer converges to a Nash equilibrium of the underlying game.The guarantee assumes exact policy evaluation and no function-approximation or sampling error.
  • Posterior collapse: 0.94 versus 5.06: posterior collapse achieves lower objective loss than M∗ in a high-entropy stochastic counterexample, revealing vulnerability under static KL-balancing.The collapsed solution can exploit a degenerate representation; the authors note the same property can be constructed for DreamerV3.

E.1 Mitigating Compounding Errors in High-Dimensional Spaces

Scaling NashDreamer beyond compact infoset vectors creates challenges from raw-observation decoding, replay-induced distribution mismatch, and latent-capacity choices. The framework addresses these with latent bottlenecks, root-starting trajectories, clipped corrections, and carefully sized stochastic representations.

  • High-dimensional observations: Raw observations make MARSSM scaling vulnerable because Dreamer-style imagination normally avoids decoding pixels at every step.The proposed remedy is to predict low-dimensional observation embeddings and aggregate those embeddings in decentralized infoset networks.
  • Replay buffer design: Complete trajectories must start at t = 0 because mid-episode initialization severs the causal chain of strategic deductions in imperfect-information games.Stored latent states also suffer representation drift as network parameters evolve.
  • Replay buffer design: Off-policy mid-episode replay mismatches true infoset reach probabilities, requiring importance sampling or counterfactual unbiasing with potentially prohibitive variance.Trajectory-level importance ratios grow with game length and policy drift, motivating ratio clipping; without clipping, Battleship gradients diverge.
  • Replay buffer design: Replay effects are game-dependent: it does not improve plain RNaD in Goofspiel-5, but improves it in Leduc and Phantom Tic-Tac-Toe.In the two largest domains, replay is the weaker RNaD baseline under both head-to-head play and approximate exploitability.
  • Latent capacity: The factored stochastic state uses two categorical distributions with six categories for Leduc under minimal-capacity conditions, rather than being a structural MARSSM requirement.The authors report that a single 34-category distribution failed because of gradient dilution, while larger latent spaces remain supported when chance dynamics are unknown or large.

I Latent Error Trees

Latent Error Trees recursively compare learned latent reconstructions with reachable game states, exposing structural errors in the world model.

  • Under a flat 1 × 34 latent, the single categorical distribution exhibits severe mode collapse and fails to learn the true environment structure.
  • LE-Trees recursively unroll the game from the initial state under the current joint policy, pruning branches below 0.05 action probability.
  • Blue diamond nodes denote ground-truth states, while adjacent rectangles denote corresponding latent reconstructions.
  • Latent nodes are colored by maximum decoder-head prediction error: green is low, orange moderate, and red severe.

J Leduc Failure Modes and Latent Space Capacity

Leduc Poker exposes a stochastic-environment failure mode: deterministic structure is learned reliably, but chance transitions remain miscalibrated and latent design choices trade collapse against drift.

  • Goofspiel-5: Deterministic emissions converge in Goofspiel-5, with terminal flags and legal actions within 1,000 steps, observations about an order of magnitude later, and reward last.
  • Leduc Poker: In Leduc Poker, terminal flags and legal actions converge within 1,000 steps, while observation and reward errors remain nonzero and chance transitions stay miscalibrated.
  • The diagnostics show deterministic components are learned without major issues, whereas chance stochasticity can produce incorrect or degrading representations localized in chance transitions.
  • Posterior collapse: At the initial K = 30 node, the flat latent collapses near 1.7, the factored latent reaches 2.9, and the one-way variant reaches 3.45 against log 30 = 3.401.
  • Posterior collapse: At the public K = 4 node, the flat, factored, and one-way latents reach 1.386, 1.45, and 2.4 respectively, reversing the initial-node ordering.
  • Posterior collapse: No configuration is faithful at both nodes: flat latents collapse at large K, factored latents remain approximate, and removing regularization induces drift.

K.5 Chance-Outcome Calibration in Leduc Poker

Chance-outcome calibration separates latent configurations more clearly than deterministic emissions: factored latents perform best at chance nodes, while all variants preserve terminal and legal-action predictions.

  • Calibration: At deterministic nodes, factored and one-way variants reach zero error within roughly 10,000 steps, while the flat latent settles near 0.11 on both statistics.
  • Calibration: At chance nodes, factored, flat, and one-way variants achieve L1 distances of 0.16, 0.45, and 0.63, with unmatched masses of 0.07, 0.44, and 0.63.
  • Calibration: The factored variant is best at chance nodes and exact at deterministic ones, with residual error reflecting probability misallocation rather than failure to represent outcomes.
  • Rollout diagnostics: Terminal flags and legal-action masks reach zero within roughly 1,000 steps for all configurations, so Leduc failure is not loss of terminal or action tracking.
  • Diagnostic interpretation: Rollout validity and probability calibration are complementary: a model may generate realizable states while visiting them at incorrect frequencies.
  • Comparative structure: NashDreamer’s LE-Tree exactly matches reachable dynamics, whereas DreamerV3 retains artificial transition stochasticity.
  • Policy evaluation: NashDreamer with RNaD converges to the pure Nash equilibrium, while REINFORCE’s entropy bonus prevents concentration on required pure strategies.

Q Head-to-Head Evaluation

Head-to-head evaluation shows an early-to-mid-training advantage in Battleship but parity in stochastic Goofspiel-13, while exploitability can disagree with pool-specific win rates.

  • Battleship 5 × 5: NashDreamer[RNaD] beats every method at all Battleship 5 × 5 checkpoints, but its margin over model-free RNaD declines from 66.7% to 56.7%.
  • Stochastic Goofspiel-13: NashDreamer[RNaD] and RNaD are at parity in stochastic Goofspiel-13, with model-based win rates of 50.3%, 49.9%, and 49.7%.
  • Exploitability: In Goofspiel-13, best-response search separates the methods at 30K, producing values of 0.168 and 0.356 despite head-to-head parity.
  • Exploitability: In Battleship at 30K, NashDreamer[RNaD] wins cross-play but is more exploitable at 1.169 than MMD at 1.022 and NashDreamer[MMD] at 0.817.
  • Interpretation: Head-to-head results measure performance against a particular opponent pool, so the study reports exact NashConv or approximate exploitability where available.

R World-Model Warm-Up Ablation

The warm-up sweep shows that warm-up length primarily affects early training stability, with effects depending on the policy optimizer. Longer warm-ups do not improve final policy quality in these Leduc experiments.

  • Setup: Warm-up length is a stabilizing mechanism rather than a theoretical requirement, swept over 0, 250, 500, 1000, and 2000 gradient steps.Both actor-critics inside imagination were run at each setting, with 1000 steps used in other Leduc experiments.
  • RNaD: NashDreamer[RNaD] is invariant to warm-up length, with no setting separating beyond the spread across training seeds.The Leduc model remains badly learned at every setting, and longer warm-up does not improve the asymptote.
  • MMD: For MMD, 0 and 250 warm-up steps accelerate early convergence relative to longer settings and model-free MMD.Warm-ups of 500 steps or more are indistinguishable from the model-free baseline.
  • Interpretation: MMD’s lack of gain on small games is explained by convergence inside the warm-up window, leaving imagination no additional behavior to accelerate.This interpretation is consistent with the observed separation only during the early phase at the two shortest warm-ups.
Loading 2609.01549v1…