PaperlayerView exact revision on arXiv

Source-linked AI summary

SPICE: Self-Play In Corpus Environments Improves Reasoning

Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, Jason Weston

arXiv:2510.24684v1cs.CL

TL;DR

Existing language-model self-play methods face hallucination amplification and information symmetry, motivating a grounded route to sustained self-improvement. SPICE uses one model as a document-mining Challenger and task-solving Reasoner, with adversarial dynamics generating an automatic curriculum. It achieves consistent gains across mathematical and general reasoning benchmarks on multiple model families, while corpus grounding supplies the external signal for continued improvement.

  • Problem

    Existing self-play methods can plateau or collapse because generated questions and answers amplify hallucinations and the generator and solver share the same knowledge base.

  • Method

    SPICE uses one model in Challenger and Reasoner roles, mining raw documents for diverse tasks and document-grounded answers that the Reasoner solves without document access.

  • Results

    SPICE achieves consistent improvements across mathematical and general reasoning tasks and multiple model families, including average gains of +8.9% in mathematical reasoning and +9.8% in general reasoning.

  • Takeaways & Limitations

    Document grounding enables sustained self-improvement by providing continually novel, verifiable knowledge and supporting progressively harder self-generated goals.

Abstract

from arXiv · show

Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement.

1 Introduction

Existing language-model self-play methods can plateau or collapse because synthetic questions and answers amplify hallucinations and share the same information. SPICE addresses these barriers by grounding self-play in raw documents, using a Challenger and Reasoner to generate and solve diverse tasks, and improves mathematical and general reasoning across models.

  • Motivation: Ungrounded language-model self-play can plateau or collapse through hallucination amplification and information symmetry, which limit challenge diversity and factual reliability.Variational synthesis can preserve diversity but remains bounded by the model’s initial knowledge coverage.
  • SPICE: A single model acts as a Challenger that mines documents for grounded tasks and a Reasoner that solves them without document access.Information asymmetry creates challenge, while raw documents provide questions and document-extracted answers without predefined labels.
  • SPICE: The Challenger targets the frontier of Reasoner capability, while diverse multiple-choice and free-form formats support verification across domains without specialized tools.The Challenger is rewarded for high variance in Reasoner success, and the Reasoner is rewarded for correct answers.
  • Results: SPICE improves mathematical and general reasoning consistently across model families, with average gains of +8.9% and +9.8%, respectively.Reported model-level gains range from +5.7% to +11.9%, and SPICE surpasses standard RLVR and pure self-play baselines in all listed cases.
  • SPICE: SPICE treats a large document corpus as an external knowledge source for sustained self-improvement beyond the model’s intrinsic knowledge.The framework is presented as a shift from closed-loop self-play toward interaction with verifiable knowledge in document corpora.

2 SPICE: Self-Play In Corpus Environments

SPICE uses one shared model as both a document-grounded Challenger and a document-free Reasoner, training the roles jointly through self-play. Corpus documents supply verifiable tasks and information asymmetry, while role-specific rewards create an evolving curriculum.

  • Framework: SPICE alternates one model between Challenger and Reasoner roles, generating document-grounded questions before solving them without document access.The shared model is optimized jointly across both roles.
  • Challenger: The Challenger samples corpus passages and produces valid multiple-choice or typed-answer questions with gold answers extracted from documents.Typed free-form answers may be integers, expressions, or strings.
  • Challenger: The Challenger’s variance-based reward is highest when the Reasoner has a 50% pass rate, favoring tasks that are challenging yet solvable.Tasks that are too easy or too hard receive lower rewards, producing an automatic curriculum.
  • Reasoner: The Reasoner receives a binary correctness reward for matching the document-extracted gold answer while relying only on the question and internalized knowledge.A rule-based verifier checks mathematical equivalence and other answer types.
  • Optimization: SPICE maximizes expected Challenger and Reasoner rewards jointly using separate role-specific DrGRPO advantages.The advantages are centered around role-specific expectations without standard-deviation normalization.

3 Experimental Results

SPICE improves reasoning across four base-model families and mathematical and general reasoning tasks, while analyses link gains to Challenger co-training, corpus grounding, and progressively harder tasks. The experiments evaluate broad benchmark performance and the dynamics underlying these improvements.

  • Main results: SPICE consistently outperforms all baselines across four model families and improves both mathematical and general reasoning tasks.Table 1 compares SPICE with base, self-play, and other baselines across the evaluation suite.
  • Learning dynamics: Co-training the Challenger alongside the Reasoner is essential for maximizing gains, whereas a fixed Challenger challenges the Reasoner less effectively.Without Challenger training, the Reasoner improves more slowly.
  • Ablations: Corpus grounding raises performance to 43.9% compared with 40.7% without access to external documents.The grounded setting provides continuous access to diverse document contexts.
  • Learning dynamics: As training progresses, Challenger tasks shift from surface-level information toward deep comprehension and multi-step reasoning, alongside more sophisticated Reasoner problem-solving patterns.The analyses use the same document or question across training steps to track evolution.

4 Ablations

The ablations show that corpus composition, mixed task types, Challenger co-training, and variance-based reward design shape SPICE’s reasoning improvements and curriculum quality.

  • 4 Ablations: Co-training the Challenger with the Reasoner improves performance more than using a fixed Challenger.Without Challenger training, the Reasoner is not challenged enough and improves more slowly.
  • 4.1 Corpus Distribution: Combining NaturalReasoning and Nemotron-CC-Math produces the best overall performance across mathematical and general reasoning tasks.NaturalReasoning gives the largest general-reasoning gains, while Nemotron-CC-Math gives the largest mathematical gains individually.
  • 4.2 Task Type: Mixing multiple-choice and free-form questions achieves the best overall performance, although free-form-only training gives the highest math gains.MCQs provide reliable verification, while free-form questions encourage flexible reasoning.
  • 4.2 Task Type: The Challenger’s task complexity advances from surface-level fact extraction to deep comprehension and multi-step reasoning as training progresses.Later tasks can require angular-size relationships and proportional reasoning to reach document-stated values.
  • 4 Ablations: The Challenger’s reward function critically influences how task difficulty is calibrated during self-play.The ablation compares four reward strategies, with the paper’s Variance reward outperforming previously proposed rewards.

5 Related Work

Related work spans reinforcement learning for reasoning, multi-agent language-model RL, autonomous self-play, and synthetic question generation. SPICE combines these directions by grounding adversarial self-play in document corpora rather than engineered environments or static datasets.

  • Reinforcement Learning for LLM Reasoning: RL with verifiable rewards has recently been used to improve language-model reasoning capabilities.The cited examples include OpenAI o1 and DeepSeek-R1.
  • Multi-Agent RL for Language Models: Multi-agent RL for language models remains technically challenging, leading prior work to simplify architectures, environments, or interaction settings.SPIRAL demonstrates transferable reasoning in zero-sum games but requires carefully designed game environments.
  • Self-Play for Autonomous Improvement: Self-play has produced strong game-playing systems and can create automatic curricula through intrinsic motivation.Language-model self-play later extended the paradigm to alignment and autonomous improvement.
  • Synthetic Question Generation from Corpora: Synthetic question-generation methods either remain bounded by initial datasets or produce static datasets rather than adaptive curricula.SPICE instead mines corpus contexts for continually changing, document-grounded tasks.

6 Conclusion

SPICE addresses hallucination and information symmetry in ungrounded self-play by using document corpora as an external environment. Its Challenger–Reasoner dynamics generate increasingly difficult goals and support broad reasoning improvements without explicit human supervision.

  • 6 Conclusion: SPICE is presented as a self-play reinforcement-learning framework designed to overcome hallucination and information symmetry.It outperforms pure ungrounded self-play according to the conclusion.
  • 6 Conclusion: A large document corpus supplies a continually changing external environment for adversarial Challenger–Reasoner interaction.The Challenger generates tasks while the Reasoner solves them, allowing the system to create increasingly challenging goals.
  • 6 Conclusion: SPICE achieves strong improvements across mathematical and general reasoning benchmarks compared with state-of-the-art self-play methods.The conclusion attributes broad transfer to training signals mined from corpus knowledge.
  • 6 Conclusion: The framework opens avenues for self-improvement without explicit human supervision.This conclusion is framed as a shift from closed-loop self-play to corpus-grounded adversarial learning.

A Evaluation Settings

The evaluation uses zero-shot transfer to standard benchmarks, mostly with greedy decoding for reproducibility. AIME evaluations instead average 32 temperature-0.6 samples, while other listed mathematical and general benchmarks use their specified accuracy protocols.

  • A Evaluation Settings: All models are evaluated zero-shot to test transfer of corpus-grounded self-play reasoning to standard benchmarks without task-specific adaptation.Most evaluations use greedy decoding at temperature 0 for reproducibility.
  • A Evaluation Settings: AIME’24 and AIME’25 use average accuracy over 32 sampling runs with temperature 0.6.This protocol is intended to better assess performance on challenging competition problems.
  • A Evaluation Settings: Other mathematical benchmarks report pass@1 accuracy with greedy decoding, including MATH-500, OlympiadBench, Minerva Math, GSM8K, and AMC.The passage specifies benchmark-level evaluation protocols rather than task-specific adaptation.
  • A Evaluation Settings: General reasoning evaluation covers GPQA-Diamond, SuperGPQA, MMLU-Pro, and BBEH.These benchmarks span graduate-level science, 285 disciplines, rigorous multiple-choice reasoning, and extended hard reasoning tasks.

B Training Configuration Details

The experiments use fixed training budgets, distributed infrastructure, several optimization components, and a balanced 20,000-document corpus spanning mathematical and general reasoning.

  • Training Configuration Details: All compared methods use 640 iterations and batch size 128, with R-Zero reported at its best performance because it degraded after five iterations.Absolute Zero applies separate penalties for incorrect well-formatted responses and formatting errors.
  • Training Configuration Details: Experiments use a distributed actor-learner architecture with vLLM inference and Math-Verify for answer-equivalence checking.Math-Verify handles fractions, decimals, algebraic expressions, and other mathematical formats.
  • Training Configuration Details: Training uses 8 H200 GPUs, a 1e-6 constant learning rate, DrGRPO without KL regularization, and 128 trajectories per gradient update.Gradient checkpointing, flash attention, and ZeRO Stage 2 support memory-efficient training.
  • Training Configuration Details: R-Zero trains the Challenger and Reasoner with separate rollout counts, while DrGRPO computes advantages without standard-deviation normalization.The advantage is defined as ˆAi = Ri − mean({Rj}) to reduce difficulty-related bias in gradient updates.
  • Training Configuration Details: The corpus contains 20,000 documents split evenly between Nemotron-CC-Math and NaturalReasoning, sampled uniformly across 640 iterations.Each document is used approximately 2–3 times, covering STEM, humanities, and social sciences.

C Additional Results and Analysis

The appendix provides benchmark-level ablation results to examine how design choices affect individual reasoning capabilities.

  • Additional Results and Analysis: The appendix reports comprehensive benchmark-level results for all ablation studies presented in Section 4.These results are intended to expose effects of individual design choices.
  • Additional Results and Analysis: The ablations analyze how different design choices affect specific reasoning capabilities.The appendix complements aggregate results with more targeted analysis.
  • Additional Results and Analysis: The reported ablation results support examining component-level effects rather than relying only on overall performance.This follows from the appendix’s stated focus on how design choices affect capabilities.

C.1 Comprehensive Benchmark Results

Detailed ablations show that corpus composition, response format, and Challenger reward strategy produce distinct benchmark-specific strengths, while their combination supports broad performance.

  • Comprehensive Benchmark Results: The appendix reports complete ablation results across all 11 evaluation benchmarks, revealing patterns hidden by averaged scores.Tables 6, 7, and 8 provide the full breakdowns.
  • Corpus Composition: Mathematical benchmarks favor Nemotron-CC-Math, with gains up to +10.2 points on AIME25, while general reasoning favors NaturalReasoning, including +11.9 on GPQA-Diamond.NaturalReasoning also leads by +3.9 on BIG-Bench Extra Hard.
  • Corpus Composition: The combined corpus achieves the highest GSM8K score at 92.7%, suggesting synergistic effects for certain problem types.This result appears in the corpus-composition ablation.
  • Response Format: MCQs reach 61.0% on AMC, whereas free-form responses lead on MATH500 at 82.0% and AIME25 at 23.6%.The formats show complementary strengths across answer-selection and detailed-solution tasks.
  • Challenger Reward Strategy: Variance-based rewards improve all benchmarks over the Absolute Zero baseline, with largest gains of +4.0 on BIG-Bench Extra Hard, +5.7 on AIME25, and +5.5 on MMLU-Pro.The approach appears particularly effective for multi-step reasoning.
  • Comprehensive Benchmark Results: The ablations indicate that SPICE’s design choices contribute distinct strengths that combine into a well-rounded reasoning system.The appendix characterizes these components as working synergistically toward state-of-the-art performance.

D Challenger Reward Functions

SPICE compares Challenger rewards based on task difficulty and uses a smooth variance-based objective centered on a 50% Reasoner pass rate.

  • Challenger Reward Functions: Figure 7 compares threshold, Absolute Zero, R-Zero, and SPICE reward functions as functions of the Reasoner pass rate p.The strategies encode different task-selection objectives during self-play.
  • Variance-Based Reward: SPICE’s variance-based reward peaks at the optimal variance σ2_opt = 0.25, achieved when the Reasoner succeeds on 50% of tasks.The temperature parameter τ = 0.01 controls the sharpness of the reward peak.
  • Variance-Based Reward: The variance-based reward decreases smoothly for tasks that are either easier or harder than the Reasoner’s capability frontier.Unlike the threshold reward, it avoids discontinuities that could destabilize training.
  • Variance-Based Reward: Unlike R-Zero, the Gaussian form provides better gradient signals near the optimum.This supports smoother optimization around the target difficulty.
  • Empirical Comparison: The variance-based approach yields superior performance across all benchmarks, supporting balanced-difficulty training as the optimal learning regime.The result is reported in the reward-strategy ablation.

E Prompt Templates

The prompt templates guide the Challenger to generate difficult, self-contained questions through document analysis, complexity enhancement, and self-testing. They also support format selection, model-specific answer generation, and structured JSON output.

  • Question Format Selection: The Challenger selects MCQ or free-form questions by evaluating whether documents support plausible distractors or precise deterministic answers.The selection prompt returns a structured JSON response specifying the recommended format.
  • Answer Generation: Model-specific answer templates preserve architecture-specific reasoning instructions while requiring structured JSON output for consistent answer extraction.Qwen3 uses chat-style step-by-step reasoning with boxed answers, whereas OctoThinker uses a conversational format emphasizing internal reasoning.
  • Question Generation: Question-generation prompts identify complex relationships and require synthesis across multiple concepts rather than isolated factual retrieval.They target relationships spanning sections, multi-step procedures, interacting principles, and implicit conclusions.
  • Difficulty Enhancement Process: The hardening process adds complexity layers, blocks shortcuts, and explicitly requires multi-step reasoning before question generation.The template asks what simple version to avoid, which concepts to connect, and how to ensure reasoning is necessary.
  • Self-Contained Requirements: Generated questions must be self-contained, standalone, and free of references to the source document.Specialized terms should be defined within the question when needed for clarity.
  • Multiple Choice Design: The MCQ workflow requires four balanced, plausible options and a self-test that rejects questions solvable through shortcuts or fewer than three reasoning steps.The template checks whether options reveal the answer, permit eliminating distractors too easily, or require insufficient reasoning.
Loading 2510.24684v1…