SPICE: Self-Play In Corpus Environments Improves Reasoning

1 Introduction

• Ungrounded self-play can plateau or collapse through compounding hallucinations and shared knowledge that limits challenge, motivating diverse, verifiable external feedback.Even variational synthesis remains bounded by its initial coverage, a compressed representation of pretraining data.Source: §1 · p. 2
• SPICE has one model act as Challenger and Reasoner: the Challenger mines raw corpus documents for questions and gold answers unseen by the Reasoner, creating information asymmetry and grounding.The document corpus supplies continual novelty and anchors questions and answers in real-world content.Source: §1 · p. 2
• In corpus-grounded self-play, the Challenger generates frontier-level problems and the Reasoner is rewarded for correct answers, creating an adaptive curriculum from raw documents without predefined questions or labels.During training, the fixed Reasoner’s pass rate decreases from 55% to 35%, while the fixed Challenger’s pass rate increases from 55% to 85%.Source: §1 · p. 2
• +8.9% average mathematical reasoning and +9.8% general reasoning gains show SPICE improves across domains, outperforming standard RLVR and ungrounded self-play baselines across four model variants.General reasoning gains span MMLU-Pro, GPQA-Diamond, SuperGPQA, and BBEH.Source: §1 · p. 2
• SPICE treats a large document corpus as an external knowledge source for sustained self-improvement, shifting from closed-loop self-play toward open-ended interaction with verifiable knowledge.Without corpus grounding, models show limited improvement and fail to maintain adaptive challenge; grounding supports stable advancement by mining novel contexts.Source: §1 · pp. 2–3

2.1 Notations and Preliminaries

• The Challenger sees a sampled corpus document while the Reasoner sees only the question, creating information asymmetry between the roles.Source: §2.1 · p. 3

2.2 Challenger: Document-Grounded Task Generation

• Document-grounded questions use varied formats and extracted gold answers, while a variance-based reward favors tasks the Reasoner solves about half the time.The reward peaks at 1.0 when response variance is 0.25, corresponding to a 50% pass rate; tasks that are too easy or too hard receive lower rewards.Source: §2.2 · pp. 3–4

2.3 Reasoner: Solving Tasks Without Document Access

• Without document access, the Reasoner answers the Challenger’s question using step-by-step reasoning and receives a binary correctness reward based on answer equivalence.The answer is prompted in boxed tags, and a rule-based verifier checks it against the document-extracted gold answer.Source: §2.3 · p. 4

2.4 Training with Role-Specific Advantages

• SPICE jointly optimizes shared model weights with DrGRPO and separate role-specific advantages, centering each role’s returns without standard-deviation normalization.This is intended to make gradient updates reflect learning signals rather than difficulty-induced noise.Source: §2.4 · p. 4, Algorithm 1 SPICE · p. 5

2.5 Implementation

• The implementation builds on Oat’s distributed actor-learner architecture and uses vLLM for inference during both role-generation phases.The cited implementation passage attributes the actor-learner interfaces to Oat and the distributed architecture to Espeholt et al. (2018).Source: §2.5 · p. 4

Algorithm 1 SPICE

• Actors alternate between generating questions with document access and answering without it, then send trajectories to a learner that updates the model using DrGRPO.Reasoner responses both determine the Challenger’s variance reward and form the Reasoner’s training group; Math-Verify checks mathematical equivalence and exact-matches other answer types.Source: Algorithm 1 SPICE · p. 5

3.1 Setup

• Experiments compare four Qwen3 and OctoThinker base models against pretrained, fixed-strong-Challenger, ungrounded R-Zero, and code-focused Absolute Zero baselines.Evaluation covers mathematical and general reasoning benchmarks, using GPT-4o for answer-equivalence checking.Source: §3.1 · p. 6

3.2 Quantitative Analysis

• Across steps 200–640, Reasoner pass rate falls from 55% to 35% against a fixed step-200 Reasoner as Challenger checkpoints improve, but rises from 55% to 85% against a fixed step-200 Challenger as Reasoner checkpoints improve.Evaluations used 128 documents with 128 generation attempts each; in full SPICE, simultaneous evolution drives mutual improvement beyond either role in isolation.Source: §3.2 · p. 6
• Co-training the Challenger with the Reasoner is essential for maximizing gains; without Challenger training, the Reasoner is insufficiently challenged and improves more slowly.Source: §3.2 · p. 6
• SPICE outperforms baselines, improving over base models by +9.1 (Qwen3-4B-Base), +5.7 (Qwen3-8B-Base), +10.5 (OctoThinker-3B-Hybrid-Base), and +11.9 (OctoThinker-8B-Hybrid-Base).Improvements appear across both mathematical and general reasoning tasks.Source: §3.2 · p. 6, Table 1 · p. 7
• With corpus grounding, performance reaches 43.9%, compared with 40.7% without external documents.Diverse document contexts provide near-inexhaustible question material.Source: §3.2 · p. 7

3.3 Qualitative Analysis

• On the same documents, Challenger tasks evolve from surface-level questions to deeper comprehension and multi-step reasoning as the Reasoner improves.Early tasks are limited in difficulty because the Reasoner cannot learn from tasks that are too difficult.Source: §3.3 · p. 7
• Given the same question, the Reasoner develops more sophisticated problem-solving, decomposing problems, validating intermediate steps, and self-correcting inconsistencies.The passage presents this progression as authentic reasoning rather than mere memorization.Source: §3.3 · p. 7

4 Ablations

• Against a fixed step-200 Reasoner, later Challenger checkpoints lower pass rate from 55% to 35%, while a fixed Challenger sees later Reasoner pass rate rise from 55% to 85%.The evaluations compare SPICE checkpoints at steps 200–640 on 128 documents.Source: Figure 3 · p. 8
• With corpus grounding, performance improves steadily to 43.9%, compared with 40.7% without it; a fixed Challenger also improves less than full adversarial training.Figure 4 compares training dynamics for corpus grounding and Challenger training.Source: Figure 4 · p. 8

4.1 Corpus Distribution

• Using NaturalReasoning yields the largest general-reasoning gains, Nemotron-CC-Math yields the largest math gains, and combining both gives the best overall performance.The datasets target general reasoning and math, respectively.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 Task Type

• Free-form-only questions yield the highest math gains, but combining MCQ and free-form achieves the best overall performance.The authors attribute the benefit to MCQs enabling reliable verification and free-form questions encouraging flexible reasoning; the Challenger’s questions also progress from fact extraction to angular-size and proportional reasoning.Source: §4.2 · p. 8, Table 3 · p. 8, Figure 5 · p. 9

4.3 Challenger’s Reward

• The variance-based Challenger reward achieves the best performance across all metrics, with learning favored when the Reasoner has a balanced success rate.Unlike rewards based on failure, solvability thresholds, or agreement with a single answer mode, variance captures the full spread across multiple Reasoner samples.Source: §4.3 · p. 9, Table 4 · p. 9
• The Reasoner’s depicted progression moves from intuitive guessing to angular-size principles, ratio setup, algebraic manipulation, and solution verification.This evolution is shown in Figure 6.Source: Figure 6 · p. 10

Reinforcement Learning for LLM Reasoning

• RLVR can elicit chain-of-thought reasoning with rule-based rewards, but depends on human-curated problem sets and domain-specific reward engineering.SPICE instead generates problems dynamically through corpus interactions, aiming to enable learning without human curation.Source: §5 · p. 10

Multi-Agent RL for Language Models

• Prior multi-agent LLM work makes technical compromises, and SPIRAL requires designed game environments; SPICE grounds adversarial dynamics in corpus contexts.Examples include RNNs instead of transformers and simplified environments that avoid full autoregressive generation; SPICE enables automatic curriculum emergence without engineered environments.Source: §5 · pp. 10–11

Self-Play for Autonomous Improvement

• Pure self-play methods face limits: R-Zero degrades after 3-4 iterations, while Absolute Zero and SQLM remain confined to coding tasks.R-Zero’s pseudo-label accuracy drops from 79% to 63%; other approaches rely on offline updates, human-seeded task distributions, or data-free training.Source: §5 · p. 11

Synthetic Question Generation from Corpora

• SPICE mines corpus contexts online to calibrate tasks to the Reasoner, unlike corpus-mining methods that produce static datasets or require rule-based filters.WebInstruct (Yue et al., 2024) requires rule-based filters, while General-Reasoner (Ma et al., 2025) and NaturalReasoning (Yuan et al., 2025) create static offline datasets.Source: §5 · p. 11

6 Conclusion

• SPICE outperforms pure ungrounded self-play by using a large document corpus as an external environment for self-improvement.The conclusion frames this as a way to address hallucination and information symmetry.Source: §6 · p. 11
• The large document corpus provides a near-inexhaustible, continually changing environment through which LLMs interact with web documents.Source: §6 · p. 11
• Adversarial dynamics let the Challenger generate tasks and the Reasoner solve them, creating increasingly challenging goals for the system.Source: §6 · p. 11
• SPICE achieves strong improvements on mathematical and general reasoning benchmarks compared with state-of-the-art self-play methods.Source: §6 · p. 11
• Mining training signals from corpus knowledge develops reasoning capabilities that transfer broadly across domains.Source: §6 · p. 11
• The authors present corpus-grounded adversarial learning as a path toward self-improvement without explicit human supervision.They describe it as opening new avenues beyond closed-loop self-play.Source: §6 · p. 11

Evaluation Protocol

• All models are evaluated zero-shot; most use greedy decoding at temperature 0, while AIME’24 and AIME’25 average accuracy across 32 runs at temperature 0.6.The AIME protocol follows SimpleRL (Zeng et al., 2025); other evaluations follow protocols from Ma et al. (2025).Source: A Evaluation Settings · p. 17
• Mathematical benchmarks use pass@1 with greedy decoding except AIME’24 and AIME’25, and GPT-4o verification checks extracted, normalized answers for equivalence.Benchmarks include MATH-500 (Hendrycks et al., 2021), OlympiadBench, Minerva Math (Lewkowycz et al., 2022), GSM8K (Cobbe et al., 2021), and AMC.Source: A Evaluation Settings · p. 17

General Reasoning Metrics

• General reasoning evaluation uses greedy decoding and exact match on extracted answer choices, with consistent prompts and answer formats across models.Benchmarks are GPQA-Diamond, SuperGPQA (Du et al., 2025), MMLU-Pro (Wang et al., 2024), and BBEH (Kazemi et al., 2025).Source: A Evaluation Settings · p. 17

B Training Configuration Details

• 640 iterations and batch size 128 were used across methods; R-Zero is reported at its best performance after degrading after 5 iterations.Absolute Zero assigns -0.5 to incorrect but well-formatted responses and -1 to formatting errors.Source: Table 5 · p. 17
• 8 H200 GPUs per experiment, a 1e-6 learning rate, and DrGRPO without KL regularization (β = 0) define the training setup.Experiments use Oat, vLLM (Kwon et al., 2023), and Math-Verify; each gradient update processes 128 trajectories.Source: B Training Configuration Details · pp. 17–18
• R-Zero uses 5 Reasoner and 4 Challenger rollouts with uncertainty reward peaking at 50% accuracy (Huang et al., 2025), while Absolute Zero uses 1 rollout and 8 learnability attempts.R-Zero's reward is 1 − 2|p − 0.5|; Absolute Zero's is 1 − p for 0 < p < 1, with zero reward for trivial or impossible tasks.Source: B Training Configuration Details · p. 18
• DrGRPO subtracts the mean reward from each reward to avoid difficulty bias in gradient updates.R-Zero validates binary correctness against majority-voted pseudo-labels with Math-Verify; Absolute Zero checks Python value equality against ground-truth outputs, and their invalid-response penalties differ.Source: B Training Configuration Details · p. 18
• 20,000 documents comprise equal shares of Nemotron-CC-Math (Mahabadi et al., 2025) and NaturalReasoning (Yuan et al., 2025).Documents are sampled uniformly and used approximately 2–3 times over 640 iterations, spanning STEM, humanities, and social sciences.Source: B Training Configuration Details · p. 18

• The appendix breaks down all Section 4 ablation studies by benchmark to reveal their effects on specific reasoning capabilities.Source: C Additional Results and Analysis · p. 18

Comprehensive Benchmark Results

• The full ablation results span all 11 evaluation benchmarks and reveal patterns beyond the averages; the paper concludes that SPICE’s design choices combine distinct strengths into a well-rounded reasoning system.Source: §C.1 · pp. 18–19
• Nemotron-CC-Math yields up to +10.2 points on AIME25, NaturalReasoning is favored on general reasoning (GPQA-Diamond +11.9; BIG-Bench Extra Hard +3.9), and combined data scores 92.7% on GSM8K.Source: §C.1 · p. 18
• MCQs score 61.0% on AMC, free-form reaches MATH500 82.0% and AIME25 23.6%, while the mixed approach achieves optimal GSM8K performance, indicating complementary format strengths.The analysis suggests task diversity aids word-problem comprehension.Source: §C.1 · pp. 18–19
• Variance-based rewards improve all benchmarks versus the absolute zero baseline, especially BIG-Bench Extra Hard (+4.0), AIME25 (+5.7), and MMLU-Pro (+5.5).They appear particularly effective for problems requiring multi-step reasoning.Source: §C.1 · p. 19

D Challenger Reward Functions

• Maximum reward occurs at a 50% reasoner success rate, with σ2 = p(1 −p), σ2 opt = 0.25, and temperature τ = 0.01.The optimal variance is achieved at p = 0.5; τ controls the sharpness of the reward peak.Source: D Challenger Reward Functions · p. 19
• Figure 7 contrasts SPICE’s variance-based reward with threshold, Absolute Zero’s 1 −p, and R-Zero’s 1 −2|p −0.5| rewards.The figure describes the threshold reward as 0 for trivial or impossible tasks and 1 otherwise, while R-Zero peaks at maximum uncertainty; SPICE’s Gaussian form provides better gradient signals near the optimum.Source: D Challenger Reward Functions · pp. 19–20, Figure 7 · p. 20
• SPICE’s reward decreases smoothly for easier and harder tasks, creating a curriculum centered at the model-capability frontier.Unlike the threshold reward, it has no discontinuities that could destabilize training.Source: D Challenger Reward Functions · p. 20
• The variance-based reward yields superior performance across all benchmarks, supporting training on problems with balanced difficulty.The passage attributes the gains to focusing training on problems at the edge of the reasoner’s capability.Source: D Challenger Reward Functions · p. 20

Question Format Selection

• The Challenger selects MCQ or free-form questions based on document content, and assigns free-form answers an integer, expression, or string type.The selection prompt returns its recommendation as structured JSON.Source: E Prompt Templates · p. 20, Figure 9 · p. 21

Question Generation

• Question-generation prompts target complex relationships and multi-step reasoning, requiring self-contained questions solvable without the source document.For MCQs, the prompt guides distractor creation and self-tests difficulty; free-form prompts require unambiguous, extractable answers.Source: E Prompt Templates · p. 20

Answer Generation

• Reasoner answer templates vary by model family: Qwen3 uses chat-style instructions for step-by-step reasoning and boxed answers, while OctoThinker uses a conversational format.The templates aim to support consistent answer extraction while preserving each model family’s performance characteristics.Source: E Prompt Templates · p. 20, Figure 8 · p. 21

Question Generation

• MCQ instructions require connecting concepts across document sections, documenting the reasoning path, and testing whether options permit shortcuts.Questions should resist lookup, keyword matching, and partial-reasoning shortcuts; distractors should reflect plausible errors or incomplete understanding.Source: E Prompt Templates · p. 22
• Free-form prompts tailor question construction to the answer type and require precise answer formatting and step-by-step solution verification.They specify formats such as whole-number integer answers and concise string answers, and ask the Challenger to verify the solution matches the requested type.Source: E Prompt Templates · p. 23