PaperlayerView exact revision on arXiv

Source-linked AI summary

Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Play

Xiachong Feng, Deyi Yin, Xiaocheng Feng, Yi Jiang, Libo Qin, Yangfan Ye, Lei Huang, Weitao Ma, Qiming Li, Yuxuan Gu, Bing Qin, Lingpeng Kong

arXiv:2604.17696v1cs.AI

TL;DR

Game self-play can develop reasoning, but existing terminal outcomes do not identify which reasoning patterns transfer beyond game semantics. STRATAGEM modulates trajectory reinforcement using abstraction and evolution signals, yielding consistent cross-task improvements, particularly on competition-level mathematics, while its evaluation remains bounded by the tested environments, model families, and external evaluator dependency.

  • Problem

    Existing self-play approaches rely on terminal game outcomes and do not distinguish transferable reasoning patterns from game-specific heuristics, while domain specificity and contextual stasis hinder reasoning transfer.

  • Method

    STRATAGEM selectively reinforces trajectories using φ to measure abstraction and ψ to reward reasoning that progressively adapts and deepens across turns.

  • Results

    STRATAGEM produces consistent improvements across mathematical reasoning, general reasoning, and code generation, with pronounced gains on competition-level mathematics; ablations and human evaluation support both components' contributions.

  • Takeaways & Limitations

    The results suggest that trajectory structure, not only game outcomes, carries a signal associated with transferable abstract and progressive reasoning.

  • Takeaways & Limitations

    Experiments use three text-based games and Qwen3-4B variants, while φ and ψ are computed with GPT-4, creating an external API dependency.

Abstract

from arXiv · show

Games offer a compelling paradigm for developing general reasoning capabilities in language models, as they naturally demand strategic planning, probabilistic inference, and adaptive decision-making. However, existing self-play approaches rely solely on terminal game outcomes, providing no mechanism to distinguish transferable reasoning patterns from game-specific heuristics. We present STRATAGEM, which addresses two fundamental barriers to reasoning transfer: domain specificity, where learned patterns remain anchored in game semantics, and contextual stasis, where static game contexts fail to cultivate progressive reasoning. STRATAGEM selectively reinforces trajectories exhibiting abstract, domain-agnostic reasoning through a Reasoning Transferability Coefficient, while incentivizing adaptive reasoning development via a Reasoning Evolution Reward. Experiments across mathematical reasoning, general reasoning, and code generation benchmarks demonstrate substantial improvements, with particularly strong gains on competition-level mathematics where multi-step reasoning is critical. Ablation studies and human evaluation confirm that both components contribute to transferable reasoning.

1 Introduction

Games support reasoning development, but terminal-reward self-play does not distinguish transferable reasoning from game-specific heuristics. STRATAGEM addresses domain specificity and contextual stasis by selectively reinforcing abstract, adaptive trajectories, improving performance across reasoning and code-generation benchmarks.

  • Games demand strategic planning, probabilistic inference, and adaptive decision-making, making them useful environments for cultivating general reasoning.
  • Existing game-based self-play can improve reasoning without curated datasets, but terminal rewards provide no mechanism to separate transferable patterns from game-specific heuristics.
  • STRATAGEM identifies domain specificity and contextual stasis as barriers to transferring reasoning learned through games.
  • STRATAGEM shows strong transfer across mathematical reasoning, general reasoning, and code generation, especially on competition-level mathematics requiring multi-step reasoning.
  • The Reasoning Transferability Coefficient φ measures abstraction level, while the Reasoning Evolution Reward ψ incentivizes progressive reasoning development.

2 Preliminaries

The paper formulates multi-turn reasoning as a Markov decision process and competitive interaction as a two-player zero-sum Markov game. A shared role-conditioned policy generates full responses while players alternate turns under environment transitions.

  • Multi-turn reasoning is formulated as an MDP with states, actions, transitions, rewards, and a discount factor.
  • States represent complete contexts, while actions correspond to full responses rather than individual tokens.
  • At each turn, the model generates a response containing reasoning and an executable action.
  • Competitive interactions use a two-player zero-sum Markov game with opposed rewards and environment transitions.
  • 2.2 SPIRAL: SPIRAL uses one role-conditioned policy for alternating players and separate baselines for each game-role pair to handle asymmetric expected returns.

3 Method

STRATAGEM selectively modulates self-play trajectory advantages to favor abstract, transferable reasoning and progressive adaptation. Its two signals target domain specificity and contextual stasis through complementary reinforcement mechanisms.

  • Overview: STRATAGEM extends self-play with trajectory advantage modulation using transferability and evolution signals.The method computes game-based advantage and adjusts it with φ and ψ.
  • Motivation: Domain specificity anchors game reasoning to game-specific concepts, terminology, and heuristics instead of abstract, domain-agnostic patterns.The paper contrasts game-specific heuristics with reusable frameworks such as case enumeration and expected value.
  • Motivation: Contextual stasis arises because static game contexts do not require reasoning to adapt as intermediate conclusions reshape the problem.The evolution signal is designed to encourage reasoning that deepens, adapts, and remains coherent across turns.
  • Reasoning Transferability Coefficient: The Reasoning Transferability Coefficient φ multiplicatively scales game advantage according to reasoning abstraction and cross-domain transfer potential.Its dimensions include abstraction level, structural clarity, and principle orientation, scored on a low/medium/high discrete scale.
  • Reasoning Evolution Reward: The Reasoning Evolution Reward ψ additively rewards reasoning deepening, strategy adaptation, and logical coherence within trajectories.Its zero-centered scoring penalizes degradation and reduces variance in policy-gradient estimates.
  • Training Procedure: Only trajectories exhibiting both abstract and progressive reasoning receive maximal reinforcement.The training procedure incorporates both signals during trajectory advantage modulation.

4 Experiment

The experiments train STRATAGEM with Qwen3-4B-Base on three complementary text-based zero-sum games and evaluate transfer across mathematics, general reasoning, and code generation.

  • Game Environments: The evaluation uses Tic-Tac-Toe, Kuhn Poker, and Simple Negotiation to cover spatial, probabilistic, and strategic reasoning.These TextArena games provide naturally verifiable win/loss rewards.
  • Training Configuration: STRATAGEM builds on SPIRAL with Qwen3-4B-Base as the base model.Training uses β = 0.2, GPT-4 scoring, two NVIDIA A100 GPUs, and approximately 30 GPU-hours per run.
  • Evaluation Metrics: Transfer is evaluated on mathematical reasoning, general reasoning, and code generation benchmarks using zero-shot prompting.The benchmark suite includes MATH500, OlympiadBench, Minerva Math, AIME’24, AIME’25, AMC’23, GPQA, MMLU-Pro, and HumanEval pass@1.

5 Results

STRATAGEM consistently improves mathematical, general-reasoning, and code-generation performance over Qwen3-4B-Base and SPIRAL, with especially strong competition-mathematics gains. Ablations, human evaluation, evaluator agreement, and generalization tests support contributions from both modulation components.

  • Main Results: STRATAGEM consistently outperforms Qwen3-4B-Base and SPIRAL across mathematical reasoning, general reasoning, and code generation.The strongest gains occur on competition-level mathematics, where multi-step reasoning is critical.
  • Ablation Study: Removing ψ drops AIME24 by 6.70% and AMC-23 by 7.50%, while ψ improves 8 of 9 benchmarks.The ablation identifies φ with abstraction and ψ with adaptive reasoning, indicating complementary roles.
  • Human Evaluation: Human evaluators score STRATAGEM 4.06 for abstraction and 4.18 for progression, versus baseline scores of 2.48 and 2.32.The variant without ψ retains competitive abstraction at 3.82 but has lower progression at 3.36.
  • Evaluation Validation: Cross-evaluator agreement exceeds κ = 0.60 and Spearman correlation = 0.70 across GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 Flash.Together with human agreement of Krippendorff’s α ≈0.75, this supports evaluator-consistent scoring of φ and ψ.
  • Generalization: STRATAGEM outperforms SPIRAL on Snake (+0.20), Pig Dice (+0.20), and Truth and Deception (+0.08).Multi-game training is best on 6 of 9 benchmarks, and gains persist with Qwen3-4B-Instruct initialization.

6 Related Work

The paper builds on game-based self-play research and examines whether STRATAGEM’s transfer-oriented training generalizes across games, training mixtures, and base-model initializations.

  • Game-Based Self-Play: Prior systems use games and self-play as environments for developing strategic, probabilistic, and adaptive reasoning in language models.The related work includes both classic game-playing systems and recent LLM game agents.
  • Case Study: STRATAGEM’s Tic-Tac-Toe case study contrasts baseline reset behavior with reasoning that tracks state and adapts strategy.The comparison illustrates the targeted distinction between game-specific repetition and transferable reasoning patterns.
  • Training Diversity: Multi-game training achieves the best performance on 6 of 9 benchmarks, particularly on competition-level mathematics.This supports combining reasoning patterns from diverse games for robust generalization.
  • Initialization Robustness: STRATAGEM provides consistent gains over SPIRAL across mathematical, general, and code benchmarks with Qwen3-4B-Instruct.The result indicates trajectory-advantage modulation is not tied to a single base-model initialization.

7 Conclusion

STRATAGEM learns transferable reasoning by selectively reinforcing trajectories with abstract and adaptive patterns. Across several benchmarks, it improves over base models and SPIRAL, especially on competition-level mathematics, while evaluations support the abstraction and progression of its reasoning.

  • 7 Conclusion: STRATAGEM selectively reinforces trajectories exhibiting abstract and adaptive reasoning patterns to learn transferable reasoning.It uses a Reasoning Transferability Coefficient for domain specificity and a Reasoning Evolution Reward for contextual stasis.
  • 7 Conclusion: STRATAGEM addresses domain specificity and contextual stasis through the Reasoning Transferability Coefficient (φ) and Reasoning Evolution Reward (ψ).
  • 7 Conclusion: STRATAGEM shows consistent improvements across mathematical reasoning, general reasoning, and code generation, including gains over base models and SPIRAL.
  • 7 Conclusion: The strongest gains occur on competition-level mathematics, where multi-step reasoning is critical.
  • 7 Conclusion: Ablation, human evaluation, and evaluator agreement support the conclusion that STRATAGEM cultivates abstract and progressive reasoning.
  • 7 Conclusion: The results suggest that trajectory structure, rather than outcome alone, carries transferable reasoning signal.The conclusion motivates richer environments, compositional curricula, and lightweight local reward models.

Limitations

The study is limited to three text-based TextArena games and Qwen3-4B base and instruction-tuned variants. Its transferability evaluation also relies on GPT-4 to compute φ and ψ, creating an external API dependency.

  • Limitations: Experiments use three text-based games from TextArena, limiting the breadth of environments represented.Broader environments, including complex multi-agent scenarios or richer state spaces, may diversify learned reasoning patterns.
  • Limitations: The experiments cover Qwen3-4B base and instruction-tuned variants, leaving larger backbones and additional model families untested.
  • Limitations: φ and ψ are computed with GPT-4, introducing an external API dependency into training.Distilling the evaluator into a lightweight local reward model is identified as a next step toward self-contained training.

A Detailed Experimental Results

STRATAGEM achieves the highest performance on 8 of 9 benchmarks, with particularly strong gains on competition-level mathematics and consistent improvements on general reasoning and code generation.

  • A Detailed Experimental Results: STRATAGEM achieves the highest performance on 8 out of 9 benchmarks.The comparisons use Qwen3-4B-Base and SPIRAL, reporting benchmark accuracies and improvement deltas.
  • A Detailed Experimental Results: AIME24 improves 2×, from 10.00% to 20.00%, while AMC-23 improves by 10 percentage points.
  • A Detailed Experimental Results: On Minerva Math, STRATAGEM reaches 41.50%, slightly below SPIRAL at 42.30%, while improving 17.2 percentage points over baseline.
  • A Detailed Experimental Results: STRATAGEM consistently outperforms both the baseline and SPIRAL on general reasoning benchmarks.
  • A Detailed Experimental Results: HumanEval (pass@1) improves by 10 percentage points over baseline, indicating stronger programming performance after game-based training.

B Ablation Study Details

The ablation study isolates the Reasoning Evolution Reward (ψ), which captures dynamic reasoning development across game trajectories. Removing ψ generally harms performance, especially on competition-level mathematics, with Minerva Math as the sole reported exception.

  • B Ablation Study Details: The ablation compares full STRATAGEM with a variant lacking ψ to isolate the reward’s contribution.ψ captures the dynamic quality of reasoning development across game trajectories.
  • B Ablation Study Details: Removing ψ lowers AIME24 by 6.70%, from 20.00% to 13.30%, and AMC-23 by 7.50%, from 60.00% to 52.50%.
  • B Ablation Study Details: AIME25 decreases by 3.30% and MATH500 by 1.40% without ψ.
  • B Ablation Study Details: With ψ, GPQA improves by 1.01% and MMLU-Pro by 0.91%.
  • B Ablation Study Details: Minerva Math is the only exception, with ψ producing a 1.10% decrease.
  • B Ablation Study Details: Performance improves across 8 out of 9 benchmarks with ψ, supporting its value for extended multi-step reasoning and strategic adaptation.

C Parameter Sensitivity Analysis

The analysis identifies β = 0.20 as the strongest overall setting for the Reasoning Evolution Reward, while showing moderate-range robustness and task-specific preferences. The section also documents evaluation prompts, human assessment procedures, and qualitative evidence that STRATAGEM improves abstraction and reasoning progression.

  • Parameter Sensitivity Analysis: β = 0.20 achieves the best performance on 6 of 9 benchmarks.Performance remains relatively stable across β ∈ [0.10, 0.20].
  • Parameter Sensitivity Analysis: β = 0.30 continues improving AIME24, whereas Minerva Math peaks at β = 0.10.These results indicate task-specific preferences, although β = 0.20 provides the best overall balance.
  • Human Evaluation: Human evaluation samples 50 traces from Kuhn Poker and Tic-Tac-Toe for four models, with five blind expert annotators scoring abstraction and progression on 1–5 Likert scales.Table 8 reports mean scores and standard errors across annotators and samples.
  • Qualitative Analysis: Across three game types, STRATAGEM improves abstraction and progression by targeting domain-agnostic concepts and progressive state-aware reasoning.The qualitative cases connect formal frameworks, opponent-intent modeling, historical context, and strategic communication with mathematical reasoning transfer.

J Out-of-Distribution Evaluation Games

The out-of-distribution evaluation tests whether cognitive skills learned in training games transfer to games with novel mechanics. The selected games probe spatial planning, sequential risk assessment, and communication under asymmetric information, within a self-play framework that supplies verifiable rewards.

  • Evaluation Design: Each unseen OOD game tests whether cognitive skills from training games transfer to novel mechanics.The evaluation therefore targets generalization beyond the games used during training.
  • Snake: Snake tests whether Tic-Tac-Toe pattern recognition transfers to trajectory planning and dynamic obstacle avoidance.Players collect apples while avoiding walls, themselves, and opponents.
  • Pig Dice: Pig Dice tests whether Kuhn Poker probabilistic reasoning transfers to sequential risk assessment and expected-value calculation.Players decide whether to bank points or continue rolling despite the risk of losing turn points.
  • Truth and Deception: Truth and Deception tests whether negotiation skills transfer to communication strategy under information asymmetry.The Guesser uses strategic questioning to identify the truth while the Deceiver attempts to mislead.
  • Self-Play Foundation: SPIRAL provides the self-play foundation, using zero-sum games with automatically verifiable win, loss, or draw rewards and a continually improving self-opponent.Role-conditioned baselines and turn-indexed policy-gradient aggregation account for differing player roles.
  • STRATAGEM Motivation: STRATAGEM addresses domain specificity with φ and contextual stasis with ψ through trajectory advantage modulation.The two signals respectively measure abstraction and reward adaptive reasoning development.
Loading 2604.17696v1…