SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

Bo Liu; Leon Guertler; Simon Yu; Zichen Liu; Penghui Qi; Daniel Balcells; Mickel Liu; Cheston Tan; Weiyan Shi; Min Lin; Wee Sun Lee; Natasha Jaques

Source-linked AI summary

SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, Natasha Jaques

arXiv:2506.24119v3Jun 30, 2025cs.AIcs.CLcs.LG

TL;DR

Reasoning RL for language models is limited by human-curated data and domain-specific reward engineering. SPIRAL replaces these requirements with online multi-turn zero-sum self-play and role-conditioned advantage estimation, achieving broad reasoning transfer and outperforming supervised fine-tuning on expert game trajectories.

Problem
Reasoning RL depends on human-curated problem-answer pairs and domain-specific reward engineering, limiting the scalability and diversity of training challenges.
Method
SPIRAL trains a shared language model through fully online multi-turn zero-sum self-play across games, using distributed actor-learners and role-conditioned advantage estimation.
Results
Up to 10.5% improvement across 8 reasoning benchmarks surpasses supervised fine-tuning on 25,000 expert game trajectories across diverse model architectures.
Takeaways & Limitations
Different games develop transferable spatial, probabilistic, and strategic skills that combine synergistically for reasoning improvement.
Takeaways & Limitations
The authors identify future expansion to cooperative games, partial observability, and games targeting specific reasoning weaknesses as open directions.

Abstract

from arXiv · show

Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approaches depend on human-curated problem-answer pairs and domain-specific reward engineering. We introduce SPIRAL, a self-play framework where models learn by playing multi-turn, zero-sum games against continuously improving versions of themselves, generating an automatic curriculum of stronger opponents, and eliminating the need for human supervision. To enable this self-play training at scale, we implement a fully online, multi-turn, multi-agent reinforcement learning system for LLMs and propose role-conditioned advantage estimation (RAE) to stabilize multi-agent training. SPIRAL produces reasoning capabilities that transfer broadly, improving performance by up to 10% across a suite of 8 reasoning benchmarks on 4 different models spanning Qwen and Llama model families, outperforming supervised fine-tuning on 25,000 expert game trajectories. Multi-game training (TicTacToe, Kuhn Poker, Simple Negotiation) yields the strongest results, with improvements observed across both base and instruction-tuned models. Analysis of chain-of-thought traces reveals that games develop distinct cognitive patterns that transfer to improve reasoning performance, with different games developing complementary strengths. Even models which have already been trained on reasoning tasks using RLVR, like DeepSeek-R1-Distill-Qwen-7B, still benefit from our approach. These results demonstrate that zero-sum games naturally develop transferable reasoning capabilities across diverse model architectures and training stages, highlighting a promising direction for autonomous reasoning development. Our code can be found in https://github.com/spiral-rl/spiral.

1 INTRODUCTION

SPIRAL addresses the scalability limits of reasoning RL by replacing human-curated rewards and datasets with self-play in evolving zero-sum language games. Its multi-game training transfers broadly to reasoning benchmarks while RAE stabilizes multi-agent learning.

Current reasoning RL depends on carefully engineered rewards, domain-specific datasets, and expert supervision, making broader scaling increasingly unsustainable.
SPIRAL generates unlimited training data through game dynamics and continuously raises difficulty through self-play against evolving opponents.
Up to 10% improvement across 8 reasoning benchmarks surpasses supervised fine-tuning on 25,000 expert game trajectories.
Multi-game training combines complementary skills from TicTacToe, Kuhn Poker, and Simple Negotiation to improve reasoning across model families.
RAE prevents thinking collapse, in which models abandon reasoning traces after 200 steps and lose generalization.

2 RELATED WORK

Related work shows that LLM reasoning RL remains tied to human-curated tasks and reward design, while prior self-play and game-based methods differ in supervision, online adaptation, or game scope. SPIRAL combines fully online multi-agent training with distributed self-play across multiple games.

Reinforcement Learning for LLM Reasoning: RLVR improves chain-of-thought reasoning but depends on human-curated problem sets and domain-specific reward engineering.
Self-Play and Multi-Agent RL for LLMs: Prior LLM self-play methods include offline single-game training, predefined human task distributions, and single-turn coding-task generation.
The SPIRAL Framework: SPIRAL’s framework uses an actor-learner system, vectorized environments, one policy for both roles, and role-conditioned advantages for on-policy updates.
Self-Play and Multi-Agent RL for LLMs: SPIRAL uses fully online, full-parameter multi-agent RL through a distributed actor-learner architecture across multiple games.
LLMs in Gaming: Game-based LLM research has studied evaluation, single-agent multi-turn RL, and transfer from visual-spatial games to mathematical reasoning.

3 THE SPIRAL FRAMEWORK

SPIRAL trains a shared language-model policy through multi-turn, two-player zero-sum games, using distributed online self-play and role-conditioned advantage estimation. Terminal game outcomes provide the learning signal, while role-specific baselines reduce variance from asymmetric positions.

Formulation: SPIRAL represents each game as a two-player zero-sum Markov game with complete contexts as states and complete multi-token responses as actions.
Benefits of multi-turn, zero-sum games: Zero-sum games provide terminal rewards only, forcing models to maintain context, plan ahead, and adapt across sequential turns.
Self-play: A shared policy plays both roles, with role conditioning enabling distinct strategies and stronger opponents as the policy improves.
RL objective: Monte Carlo REINFORCE updates use player-specific turns, but evolving opponents create high variance and a non-stationary environment.
Role-conditioned advantage estimation: RAE centers returns around role-specific expectations, reducing variance caused by positional asymmetries and preserving genuine learning signals.
Implementation: The online system uses distributed actors, vectorized multi-game self-play, and a centralized learner to update full sequences with role-conditioned advantages.

4 EXPERIMENTAL RESULTS

SPIRAL improves reasoning through multi-game self-play, with transferable patterns, adaptive opponents, complementary game skills, and stabilized training from RAE.

Multi-game SPIRAL improves Qwen3-4B-Base average benchmark performance from 34.0% to 44.5% (+10.5%).
SPIRAL consistently outperforms supervised fine-tuning on 25,000 expert game trajectories across all 8 reasoning benchmarks.
Understanding Why SPIRAL Works: Case-by-Case Analysis transfers from games to mathematics at 72% to 71%, while Pattern Recognition rises from 35% to 45%.
Understanding Why SPIRAL Works: Self-play maintains 50-52% win rates, whereas fixed-opponent training reaches 62.5%, indicating exploitation rather than continued learning.
Understanding Why SPIRAL Works: Multi-game agents achieve 59.5% average performance, exceeding the best single-game specialist at 52.9%.The evaluated games target spatial reasoning, probabilistic inference, and strategic optimization.
Without RAE, reasoning traces collapse from approximately 2,000 to near-zero characters, while RAE improves general reasoning performance from 40% to 47%.RAE centers returns around role-specific baselines to reduce gradient variance.

5 CONCLUSION

SPIRAL develops transferable reasoning through competitive self-play without human-curated data, with multi-game training combining distinct spatial, probabilistic, and strategic skills.

SPIRAL enables reasoning development through competitive self-play without human-curated data.
Multi-game training improves reasoning benchmarks by up to 10.5% and surpasses supervised fine-tuning on 25,000 expert game trajectories.
Different games develop distinct transferable skills that combine synergistically across spatial, probabilistic, and strategic reasoning.
Future work could expand to cooperative and partially observable games and design environments targeting specific reasoning weaknesses.

REPRODUCIBILITY STATEMENT

The authors provide experiment code and training settings, while using LLMs only for language refinement and related-work search.

Experiment code and training settings are provided, and experiments were run on an 8 H100 GPU cluster.
LLMs were used only for grammar, phrasing, readability, and related-work search; the authors report sole responsibility for scientific contributions.

B LIMITATIONS

SPIRAL’s main limitations concern dependence on engineered game environments, substantial compute, restricted evaluation coverage, and risks that become more important in complex settings.

Reliance on Designed Game Environments: Engineered games remain a core dependency, shifting rather than eliminating the need for design choices that may encode biases or heuristics.The experiments use relatively simple, densely rewarded games, leaving scaling to open-ended sparse-reward environments unresolved.
Computational Cost and Scalability: Each experimental run required 8 H100 GPUs for approximately 25 hours, and gains began to plateau with extended training.The authors therefore identify computational efficiency and architectural improvements as necessary for further progress.
Evaluation and Transferability: Evaluation primarily used academic benchmarks such as MATH and GPQA, which measure formal and scientific reasoning but do not capture the full spectrum of self-play.This constrains conclusions about broader reasoning transfer.
Evaluation and Transferability: Strict zero-shot evaluation tests direct transfer but may not fully reveal the models’ potential because target-domain fine-tuning was not explored.The authors note that fine-tuning on a small number of target examples could potentially improve performance.
Potential for Reward Hacking: SPIRAL remains susceptible to reward hacking, including exploiting game bugs or discovering repetitive strategies that maximize scores without learning intended reasoning.The authors did not observe significant instances, but retain this risk for more complex environments with difficult reward shaping.

D.3 EVALUATION SETTINGS

The evaluation tests whether gameplay-induced reasoning transfers to nongame benchmarks under zero-shot conditions, using mathematics and general reasoning tasks with standardized sampling settings.

Evaluation Protocol: Models are evaluated zero-shot on established benchmarks to test direct transfer from game training to general problem-solving.Base models receive few-shot examples following the Qwen3 Report settings.
Math Benchmarks: Mathematical evaluation spans MATH500, OlympiadBench, Minerva Math, AIME24, AIME25, and AMC23 across algebra, geometry, and competitive mathematics.AVG@32 is reported for AIME24, AIME25, and AMC23, while other mathematics benchmarks use PASS@1.
General Reasoning Benchmarks: General reasoning evaluation uses GPQA-Diamond for graduate-level science questions and MMLU-Pro for multidisciplinary knowledge.These benchmarks complement the mathematical tasks with scientific and broad knowledge-oriented reasoning.
Reasoning Trace Analysis: Checkpoint analysis tracks case-by-case reasoning on Minerva Math Problem 135, showing later structured reasoning with clear case separation and systematic analysis.The reported pattern is described as emerging from game playing and transferring to mathematical problem solving.

E.2 EXTENDED ANALYSIS OF RAE ABLATION STUDY

The extended ablation shows that RAE improves early self-play learning and prevents the reasoning and optimization collapse observed with vanilla REINFORCE. Additional experiments compare generalization, scaled supervised baselines, and robustness across seeds.

Game Performance: RAE reaches a 35% game win rate rapidly, whereas vanilla REINFORCE learns more slowly.
Math Reasoning Collapse: Without RAE, math reasoning performance crashes from 35% to 12% around step 150, a 66% relative decrease.
Gradient Stability: Without RAE, policy gradients become erratic before collapsing near zero after step 200, while RAE maintains norms around 0.1.
Interpretation: Self-play alone is insufficient for stable reasoning development; role-specific variance reduction prevents convergence to degenerate policies that abandon reasoning.
Complexity Generalization: SPIRAL averages 36.1% on more complex OOD environments, compared with 18.4% for SFT, while averaging 47.1% versus SFT’s 39.1% in training settings.
Benchmark Configurations: Single-game SPIRAL averages 40.0-41.4%, exceeding SFT’s 38.4%, while multi-game training reaches 42.3-42.7%.
Data Scaling: Doubling SFT data to 52k trajectories leaves Qwen3-4B Average at 39.7% versus 39.7%, while SPIRAL continues to outperform SFT.
Robustness: Three-seed reruns show SPIRAL consistently outperforming SFT with narrow confidence intervals.

E.7 SPIRAL AS PART OF THE MID-TRAINING STAGE

The paper evaluates SPIRAL within broader training pipelines and analyzes how reasoning patterns discovered in games transfer to mathematics. It reports strongest performance when SPIRAL follows RLVR and identifies both broadly transferable and domain-specific patterns.

Mid-Training Integration: SPIRAL followed by RLVR outperforms Base→RLVR, while RLVR→SPIRAL achieves the highest average performance at 48.1.
Mid-Training Integration: SPIRAL-Multi raises DeepSeek-Distill-Qwen-7B’s average score to 59.3, whereas standard SFT causes performance regression.
Instruction Models: For Qwen3-4B-Instruct-2507, SPIRAL improves the original model by 2% and exceeds SFT by 3.6%, reaching 75.51% average performance.
Analysis Procedure: The analysis discovers patterns bottom-up, compares shared and domain-specific strategies, and classifies examples across the observed traces.
Transfer Analysis: Transfer analysis uses 290 winning Kuhn Poker games, 46,792 mathematical attempts, and checkpoints at steps 0, 128, and 400.
Transfer Patterns: Case-by-case analysis transfers from 72% in games to 71% in mathematics, reflecting domain-agnostic systematic enumeration.
Transfer Patterns: Expected-value calculation transfers from 78% in games to 28% in mathematics, while pattern recognition rises from 35% to 45%.

F.7 VALIDATION METHODOLOGY

The validation methodology combines sampling, success stratification, manual checking, and scale validation before relating training-game capabilities to reasoning skills. The selected games target spatial, probabilistic, and multi-constraint reasoning.

Validation Steps: The study samples 50 random trajectories per checkpoint to reduce selection bias while preserving computational feasibility.
Validation Steps: Separate successful and failed attempts identify strategies associated with problem-solving rather than merely frequent patterns.
Validation Steps: Manual spot-checks compare GPT-4.1 classifications against raw traces, and all 46,792 mathematical traces are classified for scale validation.
Conclusion: The methodology reports quantitative evidence that competitive gameplay develops reasoning skills applicable beyond the training domain.
Training Games: TicTacToe targets spatial pattern recognition, while Kuhn Poker targets expected-value reasoning and decisions under uncertainty.
Training Games: Simple Negotiation develops multi-constraint optimization through resource trading, opponent-preference modeling, and multi-step proposals.

G.2 OUT-OF-DISTRIBUTION EVALUATION GAMES

The evaluation games test whether skills learned from training games generalize to novel mechanics. Each game isolates a distinct transfer target, including dynamic planning, sequential risk assessment, and communication under asymmetric information.

Evaluation Scope: The evaluation suite tests whether learned skills generalize to novel game mechanics.
Dynamic Spatial Reasoning: Snake tests whether static spatial patterns transfer to trajectory planning and dynamic obstacle avoidance.
Risk-Reward Reasoning: Pig Dice tests whether probabilistic reasoning transfers to sequential risk assessment and expected-value calculation.
Communication Strategy: Truth and Deception tests whether negotiation skills transfer to communication strategy under asymmetric information.
Evaluation Scope: Together, the games probe which cognitive skills generalize beyond training contexts and whether SPIRAL develops capabilities beyond game-specific tactics.