MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs

1 INTRODUCTION

• LLMs’ ability to cooperate and compete in multi-agent systems is a frontier for AI because many real-world scenarios involve agents interacting over long horizons.The introduction names negotiation, strategic gameplay, and collaborative software development as examples.Source: §1 · p. 2
• Long-horizon credit assignment and variance in advantage estimates complicate RL for multi-turn, multi-agent tasks with heterogeneous roles, asymmetric information, and different payoff scales.Actions can earn immediate rewards while jointly contributing to a final sparse reward, making turn-level attribution difficult.Source: §1 · p. 2
• MARSHAL combines turn-level advantage estimation for fine-grained credit assignment with agent-specific advantage normalization to stabilize training across heterogeneous roles.The framework learns through self-play in cooperative and competitive games.Source: §1 · p. 2
• Up to 10.0% on AIME, 7.6% on GPQA, and 3.5% on average across all benchmarks are zero-shot performance improvements reported when MARSHAL agents are integrated into multi-agent systems.MARSHAL agents trained from Qwen3-4B also show up to 28.7% performance improvements in three held-out games.Source: §1 · p. 2
• The paper reports experiments and ablations showing strategic ability and multi-agent reasoning that generalize to held-out games and multi-agent LLM systems.The evaluated systems include AutoGen and MAD.Source: §1 · p. 2

2 PRELIMINARY

• Static-task RL assigns a terminal reward after the model generates a complete Chain-of-Thought response, framing the process as one turn of token generation.The supplied passage introduces this setup but ends before completing its description.Source: §2 · p. 2
• Figure 2 contrasts player-trajectory generation through self-play with GRPO’s naive advantage estimation and MARSHAL’s estimation for multi-turn, multi-agent credit assignment.The figure places trajectory generation on the left, naive GRPO estimation in the middle, and MARSHAL estimation on the right.Source: Figure 2 · p. 3
• In the token-level MDP, the state is the previously generated tokens, the action is the next token, and the policy aims to maximize final sparse reward.A task question q conditions the token-level policy.Source: §2 · p. 3
• In strategic games, an entire game is an episode modeled as a turn-level MDP, with each turn’s action being a multi-token output generated by the autoregressive policy.The game state can include the board configuration or cards in hand.Source: §2 · p. 3

3.1 SELF-PLAY WITH GRPO

• MARSHAL adapts GRPO by treating self-play game trajectories as grouped responses, assigning their terminal returns across turns and normalizing separate game groups independently.Source: §3.1 · pp. 3–4

3.2 ADVANTAGE ESTIMATION FOR MULTI-TURN, MULTI-AGENT LEARNING

• MARSHAL sums rewards from each turn onward before normalizing cumulative returns, avoiding global normalization of heterogeneous intermediate rewards and centering advantages around batch-return means.The paper describes this as providing a more stable learning signal for multi-turn decision-making.Source: §3.2 · p. 4
• Because returns can depend on player role, MARSHAL groups trajectories by role and normalizes turn advantages against that role’s mean cumulative reward.Source: §3.2 · p. 4

3.3 GAME ENVIRONMENTS

• Training/evaluation pairs are Tic-Tac-Toe/Connect Four, Kuhn Poker/Leduc Hold’em, and Mini Hanabi/Simple Hanabi, spanning competitive and cooperative settings.The held-out games are selected as more complex evaluations of generalization.Source: §3.3 · p. 5

3.4 REWARD DESIGN

• Game outcomes provide the primary training signal, supplemented by format-validity rewards and a turn-level verbosity penalty.Valid format earns +0.05; invalid actions incur -10.0 and terminate the game, while the verbosity penalty scales linearly beyond a threshold with lmin=11, lmax=2048, and α=0.5.Source: §3.4 · p. 5

4.1 EXPERIMENTAL SETUP

• Qwen-4B specialists were trained on Tic-Tac-Toe, Kuhn Poker, or Mini Hanabi, while a generalist trained on all three was evaluated against strong opponents and on held-out games.The evaluation also tests zero-shot transfer in MAD and AutoGen, qualitative reasoning patterns, failure modes, and ablations.Source: §4.1 · pp. 5–6

4.2 STRATEGIC ABILITY

• The generalist achieved 28.7% improvement on Leduc Hold’em and 22.9% on Simple Hanabi, while specialists generalized to more complex held-out games.The Tic-Tac-Toe specialist also improved smoothly on the out-of-distribution Mini Hanabi task.Source: Figure 3 · p. 6, §4.2 · p. 6, Figure 4 · p. 6

4.3 GENERALIZATION TO MULTI-AGENT SYSTEMS

• 3.51% average gain over original Qwen3-4B across all benchmarks, with 7.57% on GPQA-Diamond in MAD and 10.00% on AIME in AutoGen.The generalist performs robustly across both competitive MAD and cooperative AutoGen settings.Source: Table 1 · p. 7, §4.3 · p. 7

4.4 REASONING PATTERN AND FAILURE MODE ANALYSIS

• Competitive-game training yielded role-aware behavior that transferred from choosing a defensive second move in Tic-Tac-Toe to taking the negative-side role in MAD.The analysis presents these as qualitative examples of emergent reasoning patterns.Source: §4.4 · p. 7, Table 2 · p. 8
• Cooperative Hanabi training fostered intent recognition that transferred to inferring collaborator uncertainty in AutoGen, including from a missing TERMINATE token.The paper characterizes this as a component of Theory of Mind.Source: §4.4 · pp. 7–8
• On GPQA-Diamond in MAD, MARSHAL reduced Inter-Agent Misalignment by 11.5% and System Design Issues by approximately 7%.The largest Inter-Agent Misalignment reductions came from fewer Task Derailment and Ignored Other Agent’s Input failures.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 ABLATION STUDIES

• Fixed-opponent training overfit static opponents and environments, with the Kuhn Poker specialist showing severe degradation on non-poker games.Table 3 compares self-play with the fixed-opponent variant using average normalized game returns.Source: Table 3 · p. 9, §4.5 · p. 9
• Removing turn-level advantage estimation or agent-specific normalization degraded performance, especially in long-horizon or competitive games with distinct player returns.Normalization’s effect was mild for the cooperative Hanabi specialist, whose players have similar return distributions.Source: Table 4 · p. 9, §4.5 · p. 9, Figure 6 · p. 9

LLM-based multi-agent systems

• Existing LLM-based multi-agent systems design cooperative or competitive workflows around fixed models, whereas this work trains LLMs for integration into those frameworks.Examples include AutoGen and CAMEL for cooperation, MetaGPT and ChatDev for software development, and MAD for debate (Li et al., 2023; Hong et al., 2024; Qian et al., 2023; Liang et al., 2023).Source: §5 · p. 10

Reinforcement learning for LLMs

• Most RL for LLMs targets single-turn or single-agent tasks; MT-GRPO addresses turn-level credit assignment but normalizes turn rewards turn-by-turn.MARSHAL instead uses a sum-then-normalize approach for robustness across strategic games with varying turn counts.Source: §5 · p. 10

Self-play training of LLMs

• Prior LLM self-play work targets alignment or strategic play in specific games, while SPAG and SPIRAL study reasoning transfer through zero-sum games and single-agent evaluation.MARSHAL covers cooperative and competitive games and evaluates generalization in multi-agent systems; its agent-specific normalization is analogous to SPIRAL’s role-conditioned advantage estimation (Cheng et al., 2024; Liu et al., 2025).Source: §5 · p. 10

6 DISCUSSION

• The authors argue that self-play in cooperative and competitive strategic games cultivates multi-agent reasoning and that game-honed skills translate to general multi-agent systems.They describe self-play as a scalable paradigm for training LLM agents.Source: §6 · p. 10
• The study uses two-player games as efficient prototypes, while scaling to larger N-player environments raises challenges in non-stationarity, population diversity, and credit assignment.The authors say these challenges warrant dedicated future investigation.Source: §6 · p. 10
• The authors identify complex social sandboxes, including simulated software engineering and collaborative research, as directions for training agents in practical domains.Source: §6 · p. 10

• The authors publicly provide the code, model checkpoints, and training scripts needed to reproduce the paper's findings.Source: REPRODUCIBILITY STATEMENT · p. 11
• The repository includes configurations for replicating the key experiments.Source: REPRODUCIBILITY STATEMENT · p. 11
• The stated reproducibility resources are available in the MARSHAL GitHub repository.Source: REPRODUCIBILITY STATEMENT · p. 11

ETHICS STATEMENT

• The authors identify the dual-use nature of enhanced LLM-agent capabilities as the work's primary ethical consideration.They note possible malicious applications alongside goals of beneficial cooperation and competition.Source: ETHICS STATEMENT · p. 11
• The authors released code and models to encourage further study into the safety and alignment of more capable agents.Source: ETHICS STATEMENT · p. 11

A USE OF LLMS

• LLMs were used as writing assistants for grammar, clarity, and readability; the authors state that the core research is their original work.Source: A USE OF LLMS · p. 15

B IMPLEMENTATION DETAILS

• MARSHAL is built on ROLL; its stack uses vLLM for rollout inference, Megatron-LM for distributed training, and OpenSpiel and VS-Bench for game environments.ROLL provides support for agentic, multi-turn rollouts.Source: B IMPLEMENTATION DETAILS · p. 15
• Training uses Qwen-4B, online self-play updates, batch sizes of 128 trajectories for specialists and 384 for the generalist, and 200 optimization steps.The Adam optimizer uses a cosine schedule, warming up for 10 steps to 1 × 10−6 before decaying to 0.Source: B IMPLEMENTATION DETAILS · p. 15
• Self-play rollout generation uses nucleus sampling with temperature 0.6, Top-P 0.99, and Top-K 100 to encourage diverse, coherent strategy exploration.Source: B IMPLEMENTATION DETAILS · p. 15
• Models were trained on a single server with 8 NVIDIA H100 GPUs.Source: B IMPLEMENTATION DETAILS · p. 15

C HYPERPARAMETERS

• The authors use a unified hyperparameter configuration for all trained models and provide the complete settings in Table 5.Source: C HYPERPARAMETERS · p. 15

D EVALUATION DETAILS

• Competitive games are tested against fixed opponents, including MCTS agents for Tic-Tac-Toe and Connect Four and Nash-equilibrium policies for Kuhn Poker and Leduc Hold’em.The Connect Four MCTS agents use 10 or 100 simulations; Tic-Tac-Toe agents use 100 or 1,000.Source: D EVALUATION DETAILS · p. 15
• Competitive-game performance averages returns across both player positions, Hanabi uses standard self-play return, and every game setting evaluates 1,000 games.Source: D EVALUATION DETAILS · p. 15
• Downstream evaluation covers mathematics and general QA benchmarks, using MASLab for multi-agent tests and a maximum output length of 8,192 tokens.Single-agent benchmarks use Qwen2.5-Math evaluation scripts or lm-evaluation-harness.Source: D EVALUATION DETAILS · p. 16

E SCALING TO LARGER MODELS

• On held-out complex games, MARSHAL-8B significantly improves over the Qwen3-8B base model, raising Leduc Hold’em from 7.26% to 53.89% and Simple Hanabi from 4.55% to 37.27%.Source: E SCALING TO LARGER MODELS · p. 16
• 2.60% average improvement in competitive MAD and 3.90% in cooperative AutoGen: MARSHAL-8B outperforms the base model in both settings.On AIME, MAD rises from 70.00% to 80.00%, while AutoGen rises from 60.00% to 70.00%.Source: E SCALING TO LARGER MODELS · p. 17, Table 7 · p. 17

F COMPARISON WITH SPIRAL: DECOUPLING ALGORITHM AND GAME

• The controlled comparison varies either the advantage estimator, replacing MARSHAL with SPIRAL’s RAE, or the game mix, restricting MARSHAL training to competitive games.These variants isolate algorithm and cooperative-game contributions, respectively.Source: F COMPARISON WITH SPIRAL: DECOUPLING… · p. 17

F.1 STRATEGIC ABILITY COMPARISON

• 54.33% to 24.93% in Mini Hanabi and 36.75% to 5.53% in Simple Hanabi: replacing MARSHAL with RAE sharply reduces cooperative-game performance.Training only on competitive games also yields low Hanabi scores.Source: §F.1 · p. 17, Table 8 · p. 18

F.2 GENERALIZATION TO MULTI-AGENT SYSTEMS

• 75.96% average: full MARSHAL leads the competitive MAD comparison, while competitive-only training performs comparably in that competitive framework.For cooperative AutoGen, competitive-only training averages 80.34% versus MARSHAL’s 82.15%; the authors conclude both cooperative games and MARSHAL’s credit-assignment algorithm matter.Source: §F.2 · p. 18, Table 9 · p. 18, Table 10 · p. 18

G.1 THEORETICAL DIFFERENCES

• MARSHAL’s cumulative-return normalization is presented as addressing variable trajectory lengths and preserving long-term action effects, unlike MT-GRPO’s turn-by-turn reward normalization.The authors argue later-turn batch sizes shrink under MT-GRPO as shorter trajectories end.Source: §G.1 · pp. 18–19

G.2 EMPIRICAL COMPARISON

• 50.48% to 36.67% in Mini Hanabi: MT-GRPO underperforms MARSHAL; MARSHAL also has higher averages in MAD (75.01% vs. 73.77%) and AutoGen (80.15% vs. 79.10%).The game comparison includes training and held-out games; downstream comparisons use Tic-Tac-Toe specialists.Source: §G.2 · p. 19, Table 11 · p. 19, Table 12 · p. 19

H HYPERPARAMETER ANALYSIS ON RESPONSE LENGTH PENALTY

• Removing the length penalty increases average response length from 1,700 to 1,954 tokens and raises overlong-response failures from 9.6% to 20.4%.Mini Hanabi performance falls from 50.48% to 38.18%; α = 1.0 retains comparable strategic performance to the default α = 0.5.Source: H HYPERPARAMETER ANALYSIS ON RESPONSE… · p. 20, Table 13 · p. 20

I GAME OBSERVATION AND PROMPT

• Game prompts supply state and action information tailored to each game; Hanabi hides the agent’s own hand while showing other players’ hands and game tokens.Mini Hanabi is used for training, while the more complex Simple Hanabi variant is used for out-of-distribution evaluation.Source: I GAME OBSERVATION AND PROMPT · pp. 22, 24, 25

J FULL RESULTS

• The appendix identifies tables reporting raw, unnormalized results for training and testing game returns and single-agent, MAD, and AutoGen benchmarks.Source: J FULL RESULTS · p. 26, Table 14 · p. 26, Table 15 · p. 27