PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini

arXiv:2605.31433v1cs.CL

TL;DR

Open-ended language-model training still depends on curated prompts, reference answers, or stronger judges because many tasks lack uniquely verifiable answers. SCOPE uses document-grounded co-evolving task generation, retrieval-based solving, and frozen self-judging to improve open-ended performance by +5.4 to +10.4 points across three 7–8B models while matching or exceeding curated-prompt training.

  • Problem

    Open-ended tasks lack unique correct answers, so existing training relies on human-curated supervision or access to stronger-model judges.

  • Method

    SCOPE co-evolves a Challenger that generates document-grounded tasks with a Solver that answers through retrieval, while a frozen model writes rubrics and judges responses.

  • Results

    SCOPE improves open-ended performance by +5.4 to +10.4 points across three 7–8B model families and matches or exceeds GRPOdata trained on ∼9K curated prompts.

  • Takeaways & Limitations

    Training exclusively on open-ended tasks transfers to short-form QA, where SCOPE surpasses GRPOdata across all three model backbones.

  • Takeaways & Limitations

    The multi-stage pipeline requires more compute than single-stage curated-prompt training, limiting experiments to 7–8B models and leaving larger-scale performance open.

Abstract

from arXiv · show

Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.

1 Introduction

Open-ended language-model training remains dependent on curated supervision or frontier-model judges because such tasks lack uniquely verifiable answers. SCOPE addresses this bottleneck with data-free self-play that co-evolves document-grounded task generation and multi-turn retrieval-based solving, achieving substantial gains across open-ended and short-form QA benchmarks.

  • Motivation: Open-ended tasks lack unique correct answers, making rule-based verification unsuitable and motivating rubric-based evaluation.Existing rubric-based methods still depend on curated prompts or frontier-model judges, leaving external supervision unresolved.
  • Method: SCOPE is the first framework to extend data-free self-play to open-ended tasks by co-evolving a Challenger and Solver from the same base model.The Challenger generates document-grounded tasks near the Solver’s capability frontier, while the Solver answers through multi-turn retrieval; a frozen base-model copy serves as Judge.
  • Results: +10.4 points is the largest reported open-ended gain, achieved by Qwen2.5-7B from 24.4→34.8 across eight benchmarks.SCOPE matches or exceeds GRPOdata trained on ∼9K curated prompts despite using no curated prompts.
  • Generalization: +13.8 is the largest short-form QA gain, with SCOPE surpassing GRPOdata across all three model families despite training exclusively on open-ended tasks.On creative writing, SCOPE surpasses GRPOdata on all three backbones by up to +7.4, while GRPOdata falls below the base model on two.

2 Background

SCOPE assumes an unlabelled document corpus and a pretrained language model that performs multi-turn retrieval before answering tasks. A task-specific judge evaluates rollouts with weighted rubric criteria, and GRPO updates the policy using group-relative rewards and a clipped, KL-regularized objective.

  • Problem setup: The model performs multi-turn retrieval over an unlabelled document corpus, gathers supporting evidence, and produces an answer from each rollout.For task q, the policy generates G rollouts and each answer is parsed from its rollout.
  • Rubric-based judging: A judge evaluates each rollout against task-specific rubrics comprising natural-language criteria with nonnegative importance weights.Each criterion receives a binary verdict, collected into a grade vector.
  • Rubric-based judging: The rubric reward aggregates criterion verdicts as a weighted average.Criterion weights are constrained to be nonnegative.
  • Policy optimization: GRPO estimates advantages from within-group reward statistics without a learned value function and samples G rollouts for each prompt.The policy is optimized using group-relative advantages computed from rollout rewards.
  • Policy optimization: The policy maximizes a clipped surrogate objective with KL regularization against a reference policy.The objective uses an importance-sampling ratio, clipping range ϵ, and KL regularization strength β.

3 SCOPE

SCOPE uses two co-evolving policies—a Challenger that creates document-grounded tasks and a Solver that answers them—plus a fixed copy of the base model as Judge. Iterative quality filtering, frontier-targeted difficulty, rubric-based rewards, and length control sustain learning while preventing task and response inflation.

  • Roles: SCOPE initializes Challenger, Solver, and Judge from the same base model, but updates only the two policies while keeping the Judge fixed.The Challenger generates document-grounded tasks, the Solver answers via retrieval-augmented generation, and the Judge writes rubrics, grades responses, and gates task quality.
  • Training loop: Each self-play iteration alternates Challenger training with Solver training on Judge-filtered, moderate-difficulty tasks.Tasks failing quality gates or falling outside the mean-score window [ℓ, u] = [0.2, 0.8] are discarded.
  • Challenger training: The Challenger reward combines format compliance with difficulty feedback targeting the Solver’s capability frontier, peaking when the mean Solver grade is ¯g = 1 2.The Challenger is optimized with GRPO using M0 as the reference policy.
  • Solver training: The Solver reward combines rubric accuracy, format compliance, and search usage, while a cosine length penalty discourages answer inflation beyond a soft token limit.The penalty is absent below ℓsoft and decays toward a near-zero floor at ℓhard.
  • Self-improvement: SCOPE sustains self-improvement by grounding tasks and rubrics in source documents, creating learnable information that the Solver does not obtain from the task alone.As the Solver closes the hidden-rubric gap on current tasks, the Challenger must co-evolve to maintain the learning signal.

4 Experiments

Across three instruction-tuned models, SCOPE improves open-ended performance substantially, matches or exceeds curated-prompt training, and transfers to short-form QA despite training exclusively on open-ended tasks. Gains are strongest on research-intensive tasks, while reward-design ablations show that both quality gates and length penalties are important.

  • Experimental setup: SCOPE evaluates Qwen2.5-7B-Instruct, Qwen3-8B, and OLMo-3-7B-Instruct on eight open-ended and seven short-form QA benchmarks.Open-ended tasks cover deep research, scholarly QA, planning, user assistance, and creative writing; short-form QA covers general and multi-hop reasoning.
  • Open-ended results: +10.4, +7.8, and +5.4 points are gained by Qwen2.5-7B, OLMo-3-7B, and Qwen3-8B, respectively, with improvements monotonic across iterations.The corresponding average scores are 24.4 →34.8, 30.7 →38.5, and 37.7 →43.1.
  • Open-ended results: +11.1 points are gained on Deep Research and Scholarly QA averaged across models, compared with +8.4 on planning, +4.3 on creative writing, and +3.0 on user assistance.The largest improvements occur on research-intensive tasks, consistent with SCOPE’s document-grounded retrieval and synthesis training focus.
  • Open-ended results: SCOPE matches GRPOdata on average open-ended scores, reaching 34.8/33.4 on Qwen2.5, 43.1/41.5 on Qwen3, and 38.5/39.0 on OLMo-3.GRPOdata trains on approximately 9K curated prompts, whereas SCOPE uses fully self-generated tasks and rubrics.
  • Short-form QA transfer: +7.8–13.8 points are gained on short-form QA, with SCOPE reaching 59.7 on Qwen2.5, 49.2 on OLMo-3, and 61.8 on Qwen3.SCOPE also gains +9.2 on general QA and +11.0 on multi-hop QA, surpassing GRPOdata across all three models.
  • Reward design ablation: At iter-3, removing quality gates drops Qwen3-8B to 37.4 and removing the length penalty collapses it to 32.8, versus 43.1 for SCOPE.Both ablations fall below the 37.7 base model, exposing distinct failure modes prevented by the two reward components.

5 Analysis

SCOPE’s sustained improvement depends on Challenger–Solver co-evolution, which keeps tasks near the Solver’s frontier. Gains reflect improvements in both retrieval and synthesis, while rubric generation—not grading—is the main self-judging bottleneck.

  • Co-evolution is necessary: SCOPE gains +3.4 points across open-ended benchmarks, versus +0.8 for a frozen Challenger, showing that sustained improvement requires co-evolution.The gap widens to 2.6 points by iter-3 because static tasks become too easy for the improving Solver.
  • Co-evolution is necessary: Co-evolution keeps mean rubric scores near the 0.5 difficulty sweet spot, whereas a frozen Challenger reaches 0.71 by iter-3.The rising score indicates that the Solver has outgrown the static Challenger’s tasks, weakening the training signal.
  • Both retrieval and synthesis improve: SCOPE improves both retrieval and synthesis across all four benchmarks, with the larger gain tracking each task’s bottleneck.On HotpotQA, retrieval contributes +3.6 versus +0.9 from synthesis; on ResearchQA, the gains are +2.2 and +3.5, respectively.
  • Rubric quality matters more than grading: Judge effectiveness depends more on rubric generation than grading, with performance stable across grader sizes but dropping sharply with a 4B rubric generator.The experiment scales rubric generation and grading from 4B to 32B while holding the other role at 8B.
  • Rubric quality matters more than grading: 4B rubrics often omit document-specific facts, whereas 8B and 32B rubrics ground criteria in concrete source details such as dates, dollar amounts, and named entities.The larger rubrics produce largely overlapping requirements, with 32B differing mainly on analytical criteria.

6 Related Work

Related work extends data-free self-play to LLM post-training and develops rubric-based rewards for open-ended tasks. Existing self-play methods co-evolve task generation and solving but reward code execution or math, while rubric methods generate or learn task-specific evaluation criteria.

  • Data-free self-play: Data-free self-play has demonstrated complex strategy discovery without human data, and recent LLM methods co-evolve task generation and solving via GRPO.Absolute Zero and R-Zero reward code execution or math.
  • Rubric rewards for open-ended tasks: Rubric-based reinforcement learning replaces scalar rewards with task-specific evaluation criteria for open-ended tasks.Prior approaches generate rubrics with off-the-shelf models, update them on-policy using search-grounded knowledge, or extract goals and rubrics from conference papers via frontier models.

7 Conclusion

SCOPE extends data-free self-play to open-ended tasks by using document-grounded rubric decomposition and a frozen self-judge for reward signals. Across three 7–8B model families, it improves open-ended performance by +5.4 to +10.4 points and matches or exceeds GRPOdata trained on ∼9K curated prompts without curated data or frontier-model supervision.

  • 7 Conclusion: SCOPE extends data-free self-play to open-ended tasks through document-grounded rubric decomposition.A frozen self-judge writes task-specific criteria from source documents, providing the reward signal for open-ended tasks.
  • 7 Conclusion: +5.4 to +10.4 points open-ended performance improvement across three 7–8B model families.SCOPE matches or exceeds GRPOdata trained on ∼9K curated prompts without curated data or frontier-model supervision.

A Limitations · B Additional Results

SCOPE’s multi-stage training pipeline requires more compute than single-stage GRPO, limiting experiments to 7–8B models and leaving larger-scale performance unresolved. It may broaden access to self-improvement but risks generating sensitive, biased, or inappropriate training tasks.

  • A Limitations: SCOPE’s multi-stage pipeline requires more compute than single-stage GRPO on curated prompts.The overhead comes from Challenger training, rubric generation, and task filtering.
  • A Limitations: Experiments were limited to 7–8B models because of the pipeline’s compute overhead.
  • A Limitations: Whether SCOPE’s gains hold at larger scales, such as 32B models, remains an open question.
  • A Limitations: SCOPE targets the post-data regime, where curated supervision has been exhausted.The paper argues that trading additional compute for continued improvement may therefore be justified.
  • A Limitations: SCOPE may reduce dependence on human-curated prompts and frontier-model judges.This could make self-improvement more accessible to open-weight models and groups without proprietary supervision pipelines.
  • A Limitations: Data-free self-play may generate sensitive, biased, or inappropriate tasks from source-corpus content.Examples include tasks synthesising medical misinformation or reproducing stereotypes.

B.1 Training Dynamics … B.4 Reference Comparison with Related Methods

Across training, SCOPE improves steadily through Challenger-driven difficulty adaptation, with gains strongest for weaker models and sustained beyond three iterations. Domain mixing compounds benefits, while reference comparisons show competitive performance despite reliance on neither curated prompts nor frontier-model supervision.

  • B.1 Training Dynamics: SCOPE matches or exceeds GRPOdata at every iteration except OLMo-3-7B iteration 3, where GRPOdata leads by 0.5 points.Per-iteration gains diminish, while weaker models benefit more: Qwen2.5-7B gains +10.4 versus Qwen3-8B’s +5.4.
  • B.1 Training Dynamics: Rubric reward dips at iteration boundaries as the Challenger generates harder tasks near Solver score τ=0.5, while total reward rises more steadily.The dips occur for Qwen3-8B and OLMo-3-7B, whereas Qwen2.5-7B does not exhibit them.
  • B.1 Training Dynamics: All Solvers initially reduce search calls, then recover toward three searches per rollout as they learn retrieval improves answer quality.OLMo-3-7B contracts most sharply, from approximately 3.5 to 1.9 searches, before climbing steadily.
  • B.2 Extended Training Iterations: Average Qwen3-8B performance rises from 37.7 to 44.8 through six iterations, with diminishing but positive gains; iterations 1–3 provide +5.4 of +7.1.Training uses identical hyperparameters across the extended six-iteration run.
  • B.3 Leave-One-Out Domain Ablation: Every leave-one-out variant improves monotonically, but the full four-domain mixture’s margin over the best ablation widens from 0.24 to 0.99 to 1.55 across iterations 1–3.The widening gap indicates compounding cross-domain benefits during training.
  • B.3 Leave-One-Out Domain Ablation: Long-form QA is the most foundational domain: removing it causes a −3.0-point average drop and the worst scores on four of eight benchmarks.The other domains have smaller average impacts, from −1.5 to −1.9, with concentrated effects on related benchmarks.
  • B.4 Reference Comparison with Related Methods: Reference comparisons are uncontrolled because related methods use curated or externally sourced prompts and differ in tools, supervision, and task setup, so results are reported descriptively.All methods are re-evaluated with SCOPE’s retrieval endpoint and the same LLM judge while preserving original prompt and tool-calling conventions.
  • B.4 Reference Comparison with Related Methods: SCOPE scores 43.1 on average, within 1.0 point of DR Tulu at 44.1, ahead of WebExplorer at 39.0 and WebThinker-R1 at 26.5, and leads on four of eight benchmarks.Its gap is concentrated on research-heavy benchmarks, while it leads on user-facing and creative tasks without curated prompts or frontier-model supervision.

C Detailed Training Algorithm · D Theoretical Justification of the Difficulty Reward

SCOPE alternates Challenger and Solver training through rollout collection, quality gating, rubric generation, grading, and difficulty filtering. Its difficulty reward targets maximum feedback variance, while the filter prevents variance collapse at extreme task difficulties.

  • C Detailed Training Algorithm: Algorithm 1 specifies the complete SCOPE loop, including rollout collection, quality gating, rubric generation, and difficulty filtering.The loop requires corpus D, base model M0, iterations T, batch size B, and difficulty window [ℓ, u].
  • C Detailed Training Algorithm: SCOPE initializes Challenger C0, Solver S0, and Judge J from M0, then repeatedly samples documents and generates document-grounded tasks.Challenger rollouts produce task pairs (oC, q), while Solver training uses tasks generated by the updated Challenger.
  • C Detailed Training Algorithm: Quality-gated Challenger tasks receive generated rubrics, N Solver rollouts, and reward RC before Challenger updates via GRPO.Tasks failing quality gates receive only the format reward rfmt(oC).
  • C Detailed Training Algorithm: The Solver stage retains quality-gated tasks whose mean rubric score satisfies ℓ ≤ ḡ ≤ u, then trains on the resulting difficulty-filtered set.The algorithm grades N Solver rollouts for each generated rubric before adding (q, G) to T.
  • D Theoretical Justification of the Difficulty Reward: The difficulty reward fdiff(ḡ; τ) peaks at mean rubric score ḡ = τ = 1⁄2, targeting maximum feedback variance.Under GRPO, group-relative advantages use centered and standardized rewards.
  • D Theoretical Justification of the Difficulty Reward: The filter ḡ ∈ [ℓ, u] with ℓ = 0.2 and u = 0.8 ensures µq(1−µq) ≥ 0.16, preventing variance collapse at difficulty extremes.This filter complements the difficulty reward by maintaining a lower bound on feedback variance.

E Training Hyperparameters and Configuration … F.2 Solver Format and Search Rewards

The paper specifies a shared multi-GPU training implementation and decomposes rewards into format compliance and retrieval incentives. Challenger and Solver rollouts receive structured format scores, while Solver search is separately rewarded up to three retrieval steps.

  • E Training Hyperparameters and Configuration: All three models use the same optimizer and schedule on 6 NVIDIA H100 GPUs, with model-specific tool-calling syntax and verl, FSDP, and SGLang infrastructure.Qwen3-8B and OLMo-3-7B use native tool-calling syntax; Qwen2.5-7B uses a custom XML format.
  • F Reward Function Details: Reward implementation converts raw rollout outputs into component scores that are combined into final scalar rewards.The pipeline covers computation from decoded rollout output through reward aggregation.
  • F.1 Challenger Format Reward: The Challenger format reward averages think, tool, and structure components, each measuring a distinct requirement on multi-turn outputs.Think measures tagged reasoning turns, tool measures valid search tags relative to expected search turns, and structure checks the final task XML.
  • F.1 Challenger Format Reward: The Challenger format score is rfmt = (think + tool + structure)/3, and zero-format rollouts are excluded from difficulty estimation and receive zero total reward.This makes basic output validity a prerequisite for difficulty-based reward assignment.
  • F.2 Solver Format and Search Rewards: The Solver format reward averages reasoning-tag compliance, valid non-final search calls, and final-answer compliance across the decoded conversation.The format components assess whether each assistant turn includes reasoning, whether non-final turns search validly, and whether the final response satisfies the answer condition.
  • F.2 Solver Format and Search Rewards: The Solver format score is F(a) = (think + tool + answer)/3.The three components are averaged rather than summed directly.
  • F.2 Solver Format and Search Rewards: The separate search reward counts valid search tags across all assistant turns and saturates after 3 retrieval steps, encouraging multi-hop search without rewarding excessive retrieval.Unlike the format tool component, it includes the final turn and uses a denominator of 3.

G Task Generation Pipeline … I.2 Solver Examples

SCOPE generates and filters document-grounded tasks through co-evolving Challenger–Solver interactions, using standardized prompts and rubric-based judging. Across iterations, Challengers create increasingly retrieval-dependent tasks while Solvers improve from repetitive searching and flat answers to distinct retrieval threads and structured synthesis.

  • G Task Generation Pipeline: Each iteration creates 2,000–3,000 Challenger prompts with stratified 4:3:2 allocations for 1/2/3 search turns and shuffled, non-overlapping data.Prompts pair sampled corpus documents with task types and use a per-iteration seed.
  • G Task Generation Pipeline: Candidate tasks are quality-gated, assigned 3–5 task-specific rubrics, evaluated with 4 Solver rollouts, and retained only when mean rubric score ḡ lies in [0.2, 0.8].The Judge checks entity identifiability and source relevance before rubric generation and difficulty filtering.
  • G Task Generation Pipeline: The pipeline targets 5,120 filtered tasks per iteration, generating 200–500 candidates per batch and retaining at most one task per source document.This corresponds to 256 batches × 20 steps and enforces source diversity.
  • H Prompt Templates: Challenger and Solver prompts share XML formats for task emission, answer submission, and judging, while search-action syntax adapts to each model’s native tool-calling convention.The Challenger selects among five training task types, with creative writing defined but excluded.
  • H Prompt Templates: Quality gates assign binary 0-or-1 scores for entity identifiability and source relevance, while the Judge grades each rubric independently with strict binary scoring.Rubric-generation prompts produce 3–5 task-specific criteria from the task and source document.
  • I Qualitative Analysis: The qualitative analysis systematically compares Challenger and Solver rollouts across iterations to characterize these changes in task difficulty, retrieval, and synthesis.The reported examples cover both retrieval behavior and organization of retrieved evidence.
  • I.1 Challenger Examples: By iteration 3, Challengers use retrieved external content to create tasks with information asymmetry that requires Solver retrieval, unlike iteration-1 tasks answerable from the source alone.Retrieved comparators can implicitly split tasks into multiple subproblems requiring research and synthesis.
  • I.2 Solver Examples: Iteration-1 Solvers issue generic or repeated queries and produce flat fact lists, whereas iteration-3 Solvers pursue distinct search threads and synthesize coherent, structured responses.Examples show progression from event-level repetition to event–mechanism–consequence retrieval and from chronology to thematic analysis.
Loading 2605.31433v1…