Source-linked AI summary
Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions
Ondřej Kubíček, Viliam Lisý, Tuomas Sandholm
TL;DR
Many game-playing successes use discrete actions, while some adversarial domains require continuous actions and mixed strategies. MMPO learns Gaussian-mixture policies with magnetic regularization; the paper reports convergence in settings where gradient descent fails and poker performance on par with Slumbot.
Problem
Many successful game-playing methods focus on discrete actions, while continuous-action approaches may depend on expert-designed discretizations or be sample inefficient.
Source: §1 · p. 1Method
MMPO uses a Gaussian-mixture policy with magnetic KL regularization, trained through self-play for continuous or mixed discrete-and-continuous actions.
Source: §1 · p. 2, §3.3 · p. 5Results
Takeaways & Limitations
The results demonstrate MMPO's applicability to games combining discrete and continuous actions and to full-size heads-up no-limit Texas hold’em.
Source: §1 · p. 2The regularized Gaussian solution is not necessarily a Nash equilibrium of the original game because the feasible strategy space is limited by σmin.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Most successes of superhuman game-playing algorithms are in games with discrete actions, yet in auctions, robotics, sports, or trading, actions are nearly continuous. Prior techniques either rely on expert-designed discretizations or are sample inefficient. We present a scalable policy-gradient algorithm for large sequential games with continuous or mixed discrete and continuous actions. It combines magnetic mirror descent with a mixture of Gaussians reparametrization, trained via self-play. We show that it approximates equilibrium in games where gradient descent fails. In sequential games, it outperforms neural fictitious self-play and matches or outperforms the final strategies of policy space response oracles with 3.5--5.5$\times$ fewer samples. In heads-up no-limit Texas hold'em, it performs on par with Slumbot.
1 INTRODUCTION
Game-playing successes have largely focused on discrete actions, while domains such as robotics and economics require continuous or nearly continuous actions. The paper introduces MMPO for large sequential continuous-action games and reports convergence where standard gradient descent fails, alongside poker performance on par with Slumbot.
1 INTRODUCTION
- Continuous or nearly continuous actions arise in domains including robotics and economics, unlike the discrete action spaces of many successful game-playing algorithms.The introduction also names sports and trading as examples of nearly continuous-action domains.Source: §1 · p. 1
- Adversarial games with hidden information or simultaneous moves may require randomizing among actions to prevent players from becoming predictable.The introduction contrasts these settings with single-player or cooperative reinforcement learning, where an optimal deterministic strategy may exist.Source: §1 · p. 1
- Prior continuous-action approaches either discretize actions, limiting solution quality through expert-chosen abstractions, or have difficulty modeling mixed equilibria, scaling to sequential games, or using samples efficiently.The cited approaches are described as having these limitations [Kroer & Sandholm, 2015; Schulman et al., 2017; Hsieh et al., 2019; Domingo-Enrich et al., 2020; Martin & Sandholm, 2023; 2025; Heinrich & Silver, 2016; Lanctot et al., 2017].Source: §1 · p. 2
- MMPO approximates Nash equilibria in large sequential zero-sum games with continuous actions by learning Gaussian-mixture strategies and using regularization that converges where standard gradient descent may fail.It jointly learns Gaussian means, standard deviations, and the distribution over components.Source: §1 · p. 2
- In heads-up no-limit Texas hold’em, self-play training on a single GPU produced a strategy on par with the latest Slumbot.The authors also report training via self-play for a few days with minimal domain knowledge and no subgame solving, card abstraction, or fixed bet abstraction.Source: §1 · p. 2
2 BACKGROUND
The background formulates two-player zero-sum games through action spaces and a payoff function, then describes mixed strategies, equilibrium, and exploitability as a comparison measure.
2 BACKGROUND
- A two-player zero-sum game is represented by action spaces for both players and a bounded, continuous payoff function; one player maximizes payoff while the other minimizes it.The action spaces are compact and non-empty, and players choose actions simultaneously.Source: §2 · p. 2
- A mixed strategy is a Borel probability measure over a player's actions, and the Nash equilibrium is a strategy pair from which neither player can improve by unilateral deviation.The equilibrium is characterized as a saddle point of the payoff.Source: §2 · p. 2
- NashConv, also called exploitability, is used to compare baselines and serves as a distance from Nash equilibrium.Source: §2 · p. 2
3 MAGNETIC MIXTURE POLICY OPTIMIZATION
The method builds a trainable Gaussian-mixture policy for continuous-action games, using KL-based magnetic regularization to address cases where ordinary gradient descent fails. Its categorical and Gaussian components support mixed strategies and can be combined with discrete actions.
3 MAGNETIC MIXTURE POLICY OPTIMIZATION
- The Gaussian-mixture approach replaces fixed pure-action discretization with trainable distributions, enabling gradient estimation and entropy regularization while targeting sequential games.The authors state that adding a secondary KL-divergence loss enables convergence in some games where standard gradient descent fails.Source: §3 · p. 3
3.1 CATEGORICAL HEAD
- In the categorical-head setup, K available actions form a matrix game, and the policy learns a distribution over those actions.The matrix game's utility is the expected value when an action is played.Source: §3.1 · p. 3
3.2 GAUSSIAN HEAD
- For a single Gaussian policy, adding magnet KL regularization improves gradient descent's convergence properties, although its regularized solution is not necessarily the original game's Nash equilibrium.The Gaussian strategy is parameterized by its mean and standard deviation; the feasible strategy space is limited by the positive lower bound σmin, so it lacks the Quantal Response Equilibrium guarantee described for MMD [Sokota et al., 2023].Source: §3.2 · pp. 3–4
- Under convex-concave game assumptions, replacing the magnet with the previous fixed point yields a sequence converging to an ϵ-equilibrium, with ϵ bounded by the tunable σmin.Source: §3.2 · p. 4
3.3 MIXTURE OF GAUSSIANS
- MMPO represents a policy as Gaussian components plus a categorical distribution over them, and uses an upper bound on mixture KL divergence as a secondary regularization loss.The final loss combines PPO-like losses with magnet regularization for both the categorical distribution and Gaussian components.Source: §3.3 · p. 5
- Increasing the number of Gaussian components broadens the strategies the model can represent but also increases the samples required for training.The model can learn to ignore components that are not useful for a decision.Source: §3.3 · p. 5
- MMPO also supports games combining discrete and continuous actions; discrete actions simply have no corresponding Gaussian mean or standard deviation.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments test MMPO across convergence, one-shot and sequential games, and full-size heads-up no-limit Texas hold’em. They report convergence benefits from magnet regularization, faster progress toward strong sequential strategies, and performance on par with Slumbot.
4.1 CONVERGENCE OF THE REGULARIZED GAUSSIAN HEAD
- In three games with pure equilibria, regularized projected gradient descent converged to Nash where ordinary gradient descent failed.With neural policies and on-policy updates, magnet regularization improved performance by at least an order of magnitude, but sampling noise prevented exact convergence.Source: Figure 1 · p. 6, §4.1 · p. 6, Figure 2 · p. 6
4.2 ONE-SHOT GAMES
- In one-shot games with non-finite-support equilibria, finite mixtures are limited by their number of Gaussians, while PSRO converged closer in Glicksberg-Gross.PSRO adds one Gaussian strategy per iteration, allowing it to mix among more strategies than the fixed mixture model.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
4.3 SEQUENTIAL GAMES
- MMPO reached strategies close to equilibrium in all tested sequential games, and reached strong strategies faster than PSRO in wall time and environment interactions.With more training time, PSRO converged closer to equilibrium; negative approximate Blotto exploitabilities are only lower bounds on true exploitability because PPO computes the best response.Source: Figure 4 · p. 7, §4.3 · p. 7
4.4 HEADS-UP NO LIMIT TEXAS HOLD’EM
- After four days of self-play, MMPO performed on par with Slumbot in heads-up no-limit Texas hold’em.Across 900,000 hands, its average win rate was 13.5 ± 36.7 mbb/hand; its final local best response value was −1622.7 ± 158.9.Source: Figure 5 · p. 8, §4.4 · p. 8
5 RELATED WORK
Prior work addresses game solving through regret minimization, strategy ensembles, policy gradients, and action abstraction, but these approaches have limitations for scalable continuous-action games.
5 RELATED WORK
- CFR extensions such as Deep CFR, ReBeL, and Dream rely on discrete action spaces, and scalable CFR for continuous actions had not yet been developed to the authors’ knowledge.CFR minimizes counterfactual regret at each decision point for both players.Source: §5 · p. 8
- Fictitious self-play and double oracle mix ensembles of strategies and train best responses, but can be sample-inefficient because best-response training is a bottleneck and often requires hundreds of responses.The methods can be adapted to continuous actions by modifying best-response search.Source: §5 · p. 8
- Policy-gradient methods use Gaussian reparameterization, while prior Gaussian mixtures targeted strategy variability rather than adversarial opponents and adversarial extensions used a single Gaussian.PPO and MAPPO lack convergence guarantees and may yield exploitable self-play strategies; magnetic mirror descent and regularized Nash dynamics address convergence in discrete-action games.Source: §5 · pp. 8–9
- Action abstraction removes actions or discretizes them; uniform grids can scale poorly in extensive-form games, though Lipschitz utilities permit bounded abstraction error.Poker abstraction methods and dynamic bet abstractions have also been developed, with the latter requiring a tractable game for CFR.Source: §5 · p. 9
6 CONCLUSION
The conclusion presents MMPO as an extension of regularized policy gradients to continuous-action games, including games with mixed action types. It reports convergence in cases where plain gradient descent diverges and poker performance on par with a strong bot.
6 CONCLUSION
- MMPO extends regularized policy-gradient techniques to continuous-action games and represents strategies as mixtures of Gaussians, including for games with discrete and continuous actions.The conclusion also reports convergence with a single Gaussian in games where plain gradient descent diverges.Source: §6 · p. 9
- MMPO is reported to be efficient and robust across several domains and baselines, and on par with a strong poker bot in heads-up no-limit Texas hold’em.The poker strategy was trained in self-play without subgame-solving and with minimal domain-specific knowledge.Source: §6 · p. 9
AI USE STATEMENT
The statement describes AI assistance across experiments, proof development, literature exploration, and writing. The authors say AI did not generate the paper’s main ideas, section drafts, experimental data, or citations, and that they reviewed the AI-assisted work.
AI USE STATEMENT
- AI tools assisted with programming, experiment scheduling, result management, plotting, and summarization.The statement lists Claude, OpenAI, Cursor, and other models used for these tasks.Source: AI USE STATEMENT · p. 10
- AI tools helped flesh out and formalize proof details and explore related literature.The statement names the models used for each task.Source: AI USE STATEMENT · p. 10
- AI tools assisted with improving formulation and the paper’s overall flow.The statement names Gemini and Claude models for writing assistance.Source: AI USE STATEMENT · p. 10
- The authors say AI did not generate the main ideas, draft sections, produce experimental data, or generate citations, and that they reviewed AI-assisted work by testing implementations and verifying proof material.They take responsibility for the final content, including AI-assisted text, claims, and artifacts.Source: AI USE STATEMENT · p. 10
A PROOFS
The proofs establish convergence properties for regularized updates under stated assumptions and give an exploitability bound for fixed-magnet iteration. The appendix also provides the rules for the experimental games.
A PROOFS
- Strong monotonicity and Lipschitz continuity of the regularized field support a contraction argument, yielding PGDA convergence to the unique variational-inequality solution.The proof states a learning-rate condition for contraction and convergence.Source: A PROOFS · pp. 16–18
- Under its stated assumptions, the variational inequality has a unique solution when η exceeds LV/δR, and the solution is a fixed point of the update operator.If the solution is interior to Π, then VR(π*) = 0.Source: A PROOFS · p. 17
- With fixed σmin > 0 and η > LV/σmin^2 in a convex-concave game with continuous second derivatives, fixed-magnet iteration converges to a solution with bounded exploitability.The appendix states the bound in Theorem 2, but the supplied passage does not include its value.Source: A PROOFS · p. 20
B GAMES
- The appendix provides rules for all games used in the experiments.Source: B GAMES · p. 20
B.1 ONE-SHOT GAMES
The one-shot and sequential test games cover continuous-action equilibria with both pure and mixed strategies. The suite also includes finite-mixture and discretization cases that expose limits of those representations.
B.1 ONE-SHOT GAMES
- Continuous matching pennies and rotational games have pure Nash equilibria at a1=a2=0 where gradient descent does not converge; circle-game equilibria have K+1 uniformly spread points, using K=3.In the two-point game, both players have a Nash equilibrium mixing 30% playing -1 and 70% playing 1.Source: §B.1 · p. 21
- The Glicksberg-Gross and circle games have equilibria that a finite mixture of Gaussians cannot reach exactly.The two-point game is a favorable case for mixtures, which can approximate its two pure strategies well.Source: §B.1 · p. 21
B.2 SEQUENTIAL GAMES
- Sequential tests include continuous-bet Kuhn and Leduc Poker, Sequential Blotto over N turns, and heads-up no-limit Texas Hold’em with bets treated as continuous.In Hold’em, a sampled real-valued bet is rounded to the nearest integer for play.Source: §B.2 · pp. 21–22
C EXPERIMENTAL DETAILS
The experimental appendix documents baselines, evaluation procedures, and computational resources, then reports sampled-gradient and wall-time comparisons. These additional tests examine convergence with noisy gradients and training speed against competing methods.
C.1 PROXIMAL POLICY OPTIMIZATION
- PPO is used as a baseline, an evaluation best-response algorithm, and an oracle for NFSP and PSRO; its policy uses a single Gaussian head.The PPO critic is trained with Generalized Advantage Estimate (GAE).Source: §C.1 · p. 22
C.2 MAGNETIC MIRROR DESCENT
- MMD adds KL divergence between the current strategy and magnet strategy to the loss, and trains MMPO’s categorical distribution over Gaussian components.MMD is also used as a solver for uniformly discretized action spaces.Source: §C.2 · p. 22
C.3 NEURAL FICTITIOUS SELF-PLAY
- NFSP alternates between training best responses and training an average strategy from replayed trajectories, while PSRO adds best responses to a growing strategy portfolio and solves its payoff matrix.The NFSP best response uses PPO rather than DQN.Source: §C.3 · p. 22, §C.4 · p. 23
C.5 RANDOMIZED POLICY NETWORKS
- Randomized policy networks encode a strategy implicitly in network weights by mapping input noise to sampled actions, while local best response estimates a lower bound on exploitability.The local best response uses fold, call, pot, and all-in actions and assumes both players check thereafter when estimating action values.Source: §C.5 · p. 23, §C.6 · p. 24
C.7 COMPUTATIONAL RESOURCES
- Experiments generally used a single CPU core; heads-up no-limit Texas Hold’em was trained on an H200 GPU for four days.The final Hold’em evaluation used 8,976 CPU-hours.Source: §C.7 · p. 24
D.1 REGULARIZED GAUSSIAN HEAD
- With sampled gradients, magnet regularization keeps exploitability roughly 10^-2 across games, while unregularized gradient descent diverges to highly exploitable strategies.Noise in gradient estimation limits convergence arbitrarily close to Nash equilibrium; strategies also gradually diverge without the magnet in continuous matching pennies.Source: §D.1 · p. 25, Figure 6 · p. 25, Figure 7 · p. 25
D.2 WALL-TIME COMPARISON
- MMPO is as fast as PPO in wall time and converges much quicker than NFSP and PSRO, consistent with the environment-interaction comparison.The comparison is reported for one-shot and sequential games.Source: §D.2 · p. 25, Figure 8 · p. 26, Figure 9 · p. 26
D.3 EFFECT OF THE DISCRETIZATION AND AMOUNT OF COMPONENTS IN THE MIXTURE
The appendix compares mixture size and uniform discretization, finding different outcomes across games and a trade-off between strategy capacity and training cost. It also details the action representation and training choices for the Hold’em experiment.
D.3 EFFECT OF THE DISCRETIZATION AND AMOUNT OF COMPONENTS IN THE MIXTURE
- MMPO reliably converges in the two-point game with three Gaussian components, but two components can be sensitive to initialization; more components also require more samples and slow training.The optimal component count can therefore depend on the available budget.Source: §D.3 · p. 27
- In matching pennies, symmetric uniform discretization can find equilibrium, while Gaussian sampling noise limits MMPO’s convergence regardless of component count.The discretized-action comparison is shown in Figure 10.Source: §D.3 · p. 26, Figure 10 · p. 27
- In Glicksberg-Gross, discretization performs best with six bins and worsens with more, while MMPO is less sensitive to entropy because Gaussians can shift to use probability more effectively.Similar behavior for discretization appears in some sequential games; the comparison is shown in Figure 11.Source: §D.3 · pp. 26–27, Figure 11 · p. 27
E.1 EXPERIMENTAL DETAIL
- For Hold’em, the method uses Fold, Call/Check, and All-in as discrete actions, with five Gaussian components for bets expressed as pot fractions.The component means are initialized across the interval [1/3, 3] geometrically; sampled bets are rounded to legal amounts for training.Source: §E.1 · p. 27
- The Hold’em setup adds logit-based action clipping, 1% random categorical exploration with clipped importance-sampling correction, and distinct magnet and entropy coefficients for categorical and Gaussian heads.The coefficients for the Gaussian head were reduced after early experiments found the magnet loss outweighed the advantage gradient.Source: §E.1 · p. 28
E.2 ADDITIONAL RESULTS
The additional Hold’em results describe card-conditioned betting patterns, checkpoint performance, and self-play action coverage. They report differences from GTO Wizard and show that later checkpoint gains are not uniform across evaluation comparisons.
E.2 ADDITIONAL RESULTS
- The small- and big-blind betting tables average probabilities across suits and distinguish pairs, offsuit hands, and suited hands; MMPO differs from GTO Wizard in folding patterns.GTO Wizard never folds suited small-blind hands and folds offsuit hands more aggressively; MMPO never folds in the described big-blind spot, unlike GTO Wizard.Source: §E.2 · pp. 28–29, Figure 12 · p. 29, Figure 13 · p. 30
- Later checkpoints improve against older checkpoints, but Slumbot performance does not improve substantially; the gains appear mainly in reduced showdown losses, while folding rewards remain roughly unchanged.The authors identify this pattern as strategy non-transitivity in imperfect-information games.Source: §E.2 · p. 29, Figure 14 · p. 31
- Final-checkpoint self-play bets and pot sizes show no blind spots on decisions caused by static discretization.The bet and pot distributions are displayed in Figures 15 and 16.Source: §E.2 · p. 29, Figure 15 · p. 31, Figure 16 · p. 31