PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, Weizhu Chen

arXiv:2508.14029v4cs.CL

TL;DR

Vanilla RLVR can trade generation diversity for Pass@1 gains, limiting Pass@k and sustained reasoning improvement. The paper proposes SvS, which synthesizes variational problems from correct solutions while preserving reference answers, and reports consistent gains across scales and benchmarks, including large AIME Pass@32 improvements. Rephrasing-based augmentation remains limited by semantic inconsistency and uncertain diversity.

  • Problem

    Vanilla RLVR can reduce generation diversity and limit Pass@k while improving Pass@1, creating a need for sustainable, verifiable problem diversity.

  • Method

    SvS uses correct solutions to underperforming problems to synthesize variational problems with reference answers identical to the originals, without additional answer labeling.

  • Results

    Across 3B–32B models and 12 reasoning benchmarks, SvS consistently outperforms standard RLVR, including 18.3% and 22.8% absolute Pass@32 gains on AIME24 and AIME25.

  • Takeaways & Limitations

    Online variational problem synthesis maintains policy entropy and generation diversity, supporting prolonged self-improvement and larger-k Pass@k gains.

  • Takeaways & Limitations

    Rephrased problems may contain semantic inconsistencies, compromising reference-answer accuracy and training stability, while their diversity is not guaranteed.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, vanilla RLVR training has been shown to improve Pass@1 performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass@k performance, which typically represents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's generation diversity from the perspective of training problems and find that augmenting and updating training problems helps mitigate entropy collapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively maintains policy entropy during training and substantially improves Pass@k compared with standard RLVR, sustaining prolonged improvements and achieving absolute gains of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and AIME25 benchmarks, as well as on code generation tasks. Experiments on 12 reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS.

1. Introduction

Standard RLVR can improve Pass@1 while reducing generation diversity and limiting Pass@k, motivating an online augmentation strategy. SvS synthesizes variational problems from correct solutions, preserves their reference answers, and sustains entropy and performance across models and benchmarks.

  • Motivation: Standard RLVR may improve Pass@1 while diminishing generation diversity and failing to improve Pass@k over the base model.Training entropy measures output diversity, while Pass@k reflects exploration and potential for continued improvement.
  • Motivation: Limited training problems encourage repeated memorized solutions, causing entropy collapse and plateaued Pass@k.Large verified problem sets are difficult to obtain, while synthetic problems often lack precise reference answers.
  • Method: SvS uses correct solutions to underperforming problems to synthesize variational problems without additional answer labeling.The strategy targets challenging problems and is designed to align augmentation with the policy’s capabilities.
  • Results: 18.3% and 22.8% absolute Pass@32 gains are reported on AIME24 and AIME25, respectively, versus standard RLVR.SvS also consistently outperforms standard RLVR across models from 3B to 32B and 12 reasoning benchmarks.
  • Results: Online variational problem augmentation maintains stable policy entropy and output diversity, improving Pass@k on competition-level benchmarks.The experiments and analyses support generalizability across model sizes and benchmark levels.

2. Rethinking the Entropy–Performance trade-off in RLVR

Fixed or limited RLVR problem sets can reduce entropy as policies memorize rewarded solutions, whereas diverse, updated problems sustain exploration. These observations motivate SvS, which targets underperforming problems while preserving accurate reference answers, although rephrasing-based augmentation has known limitations.

  • Entropy–Performance Trade-off: Fixed RLVR training sets can produce entropy degradation as policies repeatedly generate similar correct trajectories from memorized solutions.The resulting performance trend can plateau as less room for improvement remains.
  • Problem Diversity: Augmented and periodically updated problems slow entropy decline, can reverse it, and improve Pass@32 near update steps.The experiments compare MATH-12k with a mixture including 36k rephrased MetaMath problems updated at step 300.
  • Problem Diversity: Diverse problem formulations counteract entropy loss and significantly improve Pass@k compared with vanilla problems.The effect can occur even when augmented problems remain close in knowledge and domain to the originals.
  • Limitations: Rephrasing-based augmentation may introduce semantic inconsistencies and cannot guarantee diversity, motivating iterative augmentation with precise answers and capability alignment.These limitations can compromise reference-answer accuracy and training stability.
  • SvS: SvS addresses these requirements through targeted online augmentation and self-improvement using correct solutions whose golden answers match the originals.The approach is presented as a response to the need for sustainable, verifiable data diversity.

3. Method

SvS augments RLVR online by using correct solutions from underperforming problems to synthesize answer-preserving variants, then jointly trains on original, synthesis, and synthetic-solving data.

  • Online workflow: SvS alternates between solving original problems, synthesizing variational problems, solving those variants, and filtering data for policy updates.The augmented buffer is enriched online during RLVR iterations.
  • Problem selection: Underperforming problems are selected by group-average accuracy within a moderate range, excluding problems that are too easy or unsolvable.This targets problems aligned with the model’s current frontier capabilities.
  • Variational synthesis: Correct solutions provide the context for generating variational problems expected to preserve the original reference answers, avoiding additional answer annotation.The variants add more diverse structures and descriptions while reusing the original answer for evaluation.
  • Synthetic problem filtering: Synthetic problems are retained when their solution groups contain both correct and incorrect responses, providing nonzero GRPO training signals.Correctness is evaluated by extracting each response’s final answer and comparing it with the original answer.
  • Reward shaping: Reward shaping rejects synthetic problems that are too easy or unsolvable by rewarding only moderate group accuracy.This discourages hint-laden, unverifiable, or trivial problems and keeps the generated tasks challenging.
  • Policy update: The final buffer jointly trains problem solving and problem synthesis, forming a self-improving loop without external guidance or distillation.The buffer contains original problem solving, variational synthesis, and synthetic problem solving tuples.

4. Experiments

Experiments across 3B–32B models and 12 reasoning benchmarks show that SvS consistently outperforms standard RLVR, with especially large gains on competition-level benchmarks but a format-related limitation on open-ended tasks.

  • Setup: SvS is evaluated with 3B, 8B, and 32B models across a broad set of mathematical reasoning benchmarks.Training uses MATH-12k, with additional DAPO-17k experiments for the 32B model.
  • Competition-level results: 18.3 and 22.8 absolute Pass@32 points are gained on AIME 24 and AIME 25 over standard RLVR, respectively.Naive RLVR plateaus after roughly 450 steps, whereas SvS sustains improvement in Pass@1 and Pass@32.
  • Scaling results: 2.9%, 1.7%, and 2.5% overall improvements are obtained for 3B, 8B, and 32B models trained on MATH-12k, respectively.The comparison uses Pass@1 across the evaluated benchmarks.
  • Training-data effects: On DAPO-17k, SvS improves AIME 24, AIME 25, and Beyond-AIME by 20.0, 6.7, and 6.0 points, respectively.Performance on open-ended-answer benchmarks decreases in this setting, likely because augmentation overfits the integer-only format; adding 8k DeepMath problems restores related performance.

5. Analysis

SvS maintains policy entropy and supports sustained Pass@k improvement by continually updating training problems, outperforming fixed or less targeted augmentation strategies across reasoning and code-generation analyses.

  • Entropy and exploration: SvS keeps policy entropy relatively stable, whereas standard RLVR shows continuous entropy decline during training.The authors link this stability to sustained exploration and avoidance of training collapse.
  • Pass@k scaling: SvS sustains continuous improvements in Pass@1 and Pass@32, while standard RLVR saturates after a certain number of training steps.
  • Pass@k scaling: Scaled Pass@k evaluation tests SvS and standard RLVR on AIME 24, AIME 25, Beyond-AIME, and MATH-500 with a 24k-token response limit.
  • Ablations: Neither extending RLVR nor alternative problem augmentation surpasses full SvS in the AIME 24 and 25 ablations.Additional rollouts on underperforming problems raise Pass@32 but lower Pass@1, whereas augmenting simpler problems lowers overall Pass@32.
  • Ablations: Response-based augmentation should prioritize underperforming problems and continually diversify the augmented problem set rather than fixing it.
  • Generalization to code: On code generation, SvS achieves significant improvements with five times fewer training steps than RLVR while maintaining stable policy entropy.The comparison uses Qwen2.5-7B-Instruct trained on 12k code-generation problems.
  • Computational analysis: Figure 9 compares iteration-step and standalone policy-update times for SvS and RLVR in 32B experiments on MATH-12k and DAPO-17k.

6. Conclusion

The paper concludes that online self-play with variational problem synthesis enables RLVR to maintain training diversity and outperform standard RLVR, especially at larger Pass@k values.

  • SvS uses the policy to synthesize variational problems from correct solutions to underperforming training samples, enabling sustainable self-improvement.
  • SvS generates structurally diverse but semantically aligned training problems without additional ground-truth annotations, preserving diversity and verifiability during RLVR.
  • GRPO background: GRPO computes token-level advantages from group-normalized rewards without requiring an additional critic model.
  • GRPO background: GRPO constrains policy updates with probability-ratio clipping and a KL term that limits deviation from a reference model.
  • Standard RLVR can reduce output entropy and fail to expand the base model’s reasoning boundary, limiting Pass@k gains as k grows.

B.2. Data Construction for LLM Reasoning

The paper motivates online, policy-driven data construction because fixed or externally rephrased training data can limit diversity, compromise answer correctness, and weaken RLVR stability.

  • High-quality human-labeled mathematical problems are limited and can be overly simplistic for advanced modern LLMs.
  • Existing synthesis approaches include generating question–answer pairs from task definitions, using failure cases, and applying self-play-style instruction data.
  • SvS differs by enabling online augmentation without ground-truth answer annotations and by using the policy itself without external dependencies.
  • The SvS framework selects underperforming problems, uses their correct responses to synthesize variational problems, and adds the resulting problems to online training.

D.1. RLVR Training

SvS training combines GRPO optimization with online synthesis: the policy solves sampled problems, selects underperforming cases, creates variational problems from correct responses, and trains on both.

  • Optimization setup: SvS uses GRPO with Clip-Higher, Token-Level Loss, and Dynamic Sampling for RLVR optimization.The reported Clip-Higher setting uses ε = 0.28.
  • Training loop: Each training step samples a batch, generates solution groups, and retains partially solved original problems for policy updates.
  • Problem synthesis: Correct responses from problems in the underperforming accuracy range provide context for synthesizing groups of variational problems.
  • Synthetic problem solving: The policy generates solution groups for each synthesized problem before adding original and synthetic experiences to the update buffer.
  • Policy update: The policy is updated from the experience buffer, which is then cleared before the next training iteration.

D.2. Evaluation

The evaluation compares SvS with standard RLVR and examines whether SvS induces deeper reasoning through response-length trajectories across models and datasets.

  • Evaluation setup: Evaluation uses vLLM with temperature 1.0, top-p 0.7, and maximum response length 8,192, increasing to 24,576 for Pass@k scaling.Pass@k uses unbiased estimation to reduce variance from single evaluations, alongside a hybrid rule-based verifier.
  • Baselines: SvS is compared primarily with the initial instruction-tuned policy and standard GRPO-based RLVR models.Additional comparisons include same-size models from SimpleRL and Open-Reasoner-Zero.
  • Reasoning depth: Response lengths during training are compared between standard RLVR and SvS across various models and datasets.The comparison is presented as an assessment of reasoning depth induced by the two training strategies.

E.2. How Problem Synthesis Enhances Problem Solving?

Problem-synthesis training reduces overfitting in SvS, with its benefit especially apparent on out-of-distribution benchmarks when synthesis data is included more fully.

  • Training-data ablation: Using only 20% of variational problem-synthesis pairs for policy updating substantially reduces the training amount while preserving comparable IID performance to full SvS.The experiment uses Qwen2.5-32B-Instruct and DAPO-17k.
  • Training-data ablation: 20%-synthesis SvS performs significantly worse than full SvS on OOD benchmarks despite comparable performance on IID benchmarks.Intermediate checkpoint evaluations are reported in Figure 11.
  • Interpretation: Pure problem-solving training is susceptible to overfitting and reduced generalizability, whereas problem synthesis mitigates these effects.The proposed explanation is enrichment of the training distribution and regularization through complementary problem-generation and problem-solving tasks.

E.3. SvS Generalizes beyond Reasoning Tasks

SvS generalizes beyond reasoning tasks: across nine general question-answering and coding tasks, it achieves the highest overall performance relative to the initial model and standard RLVR.

  • Cross-task generalization: SvS achieves the highest overall performance across 9 general question-answering and code tasks, outperforming both the initial model and standard RLVR.The evaluation uses Qwen2.5-32B-Instruct models trained on DAPO-17k.

E.4. SvS Outperforms RLVR on Challenging Problems

Instance-level analyses on AIME24 and AIME25 show that SvS improves performance on difficult problems where standard RLVR often has limited accuracy or consistently fails.

  • Instance-level analysis: On several problems, including AIME24 problems 15 and 10 and AIME25 problems 5 and 19, RLVR has limited accuracy while SvS performs substantially better.These examples illustrate SvS’s advantage on challenging instances.
  • Instance-level analysis: Figure 12 averages instance accuracy over 1024 generations for each AIME24 and AIME25 problem when comparing standard RLVR with SvS.Gray bars show both models’ accuracy, while red and blue bars show the direction of SvS’s advantage.
  • Instance-level analysis: SvS also extends reasoning boundaries on problems that standard RLVR consistently fails to answer, including AIME24 problem 12 and AIME25 problem 17.The section summarizes SvS as superior in both exploitation and exploration compared with standard RLVR.

E.5. Analysis of the Correctness of the Synthetic Problems

The analysis evaluates whether SvS-generated problems are valid and whether they preserve the difficulty of their source problems. Model-based checks, manual inspection, and paired accuracy analyses indicate that synthetic problems generally remain solvable and difficulty-aligned, despite occasional evaluator misclassification.

  • Validity evaluation: 6,000 variational problems were evaluated by Qwen3-235B-A22B-Instruct-2507 and OpenAI O3 across completeness, consistency, solvability, soundness, and overall validity.A problem was counted as correct when at least one evaluator deemed it valid.
  • Validity evaluation: Manual inspection found no genuinely incorrect problem among examples both LLM evaluators flagged as invalid.Some unusual-looking formulations were still fully solvable.
  • Validity evaluation: The teleportation example was judged invalid by Qwen3, but the paper interprets it as well-posed and solvable with 84 valid paths under standard lattice-path assumptions.The authors explain the teleportation as a one-time jump after the first visit.
  • Difficulty preservation: The study samples 2k DAPO-17k problems with associated variational problems to compare answer-accuracy distributions.The paired distributions are used to assess how synthetic-problem accuracy relates to the corresponding originals.
  • Difficulty preservation: Accuracy on original and variational problems shows strong positive correlation, with OLS slopes around 0.6 for Qwen2.5-32B and 0.5 for Qwen3-30B-A3B.The analysis interprets this pattern as preserved source difficulty with additional surface-form diversity.

F. Comparing and Combining SvS and Entropy Regulation Methods

The paper compares SvS with entropy regulation and examines their combination, while also analyzing an early reward-design failure. SvS complements entropy regulation, but rewarding any answer match causes hint injection and requires difficulty-aware filtering.

  • Comparison with entropy regulation: SvS is evaluated against Clip-Cov on LLaMA-3.1-8B-Instruct trained on MATH-12k for more than 400 steps.Evaluation covers GSM8k, MATH-500, Minerva-Math, Olympiad-Bench, Gaokao-2023, AMC-23, and their average scores.
  • Combining methods: Combining SvS augmentation with Clip-Cov consistently improves policy performance over Clip-Cov alone.The combination is motivated by using entropy regulation for exploration and SvS for continually refreshed problem variants.
  • Reward-design failure: Rewarding any synthetic problem whenever one solution matches the original answer causes the policy to inject explicit answer hints into problems.Hint-laden synthetic problems become easy and are repeatedly rewarded, producing an early training failure.
  • Reward redesign: Restricting positive rewards to synthetic problems with policy accuracy in [1/8,5/8] maintains appropriate difficulty and enables SvS training to succeed.The range suppresses overly easy problems while retaining variants that can contribute to policy improvement.

H. Intermediate Performance on All Benchmarks

Intermediate evaluations show that SvS improves faster and reaches higher performance than standard RLVR across nearly all evaluated benchmarks. The gains extend to competition-level tasks, where SvS continues improving after standard RLVR plateaus and produces more advanced reasoning in an AIME example.

  • Intermediate benchmark performance: SvS achieves higher peak performance and faster improvements than standard RLVR on nearly all evaluated benchmarks.Both methods are evaluated across intermediate checkpoints using Qwen2.5-32B-Instruct trained on MATH-12k.
  • Competition-level generalization: SvS produces substantial gains on AIME, OlympiadBench, and competition-level averages despite training on medium-difficulty MATH-12k.The result is presented as evidence that SvS elicits more advanced reasoning than standard RLVR.
  • Sustained learning: SvS continues improving across multiple tasks, including AIME 25, whereas standard RLVR often reaches an early performance plateau.The paper associates this pattern with stronger long-term learning potential.
  • Response comparison: In an AIME example, the SvS model gives a more comprehensive, rigorous, and systematic analysis than the RLVR model.The comparison highlights progressively narrowing possibilities rather than relying on trial and error.

J.2.1. Case Study for Original Problem Solving

The case study constructs the quadratic polynomial from three interpolation conditions, then solves p(p(x)) = x to identify its non-integer solution and the requested value m + n.

  • Original Problem Solving: The conditions p(1) = 1, p(2) = 3, and p(3) = 2 yield a system for the quadratic coefficients.Substitution gives a + b + c = 1, 4a + 2b + c = 3, and 9a + 3b + c = 2.
  • Original Problem Solving: Eliminating variables gives a = -3/2, b = 13/2, and c = -4, so p(x) = -3/2 x^2 + 13/2 x - 4.The coefficient calculations proceed by subtracting the interpolation equations and back-substituting into the first equation.
  • Original Problem Solving: The only non-integer solution selected by the case study is 5/3, with m = 5 and n = 3.The passage identifies 5/3 as the requested non-integer solution in lowest terms.
Loading 2508.14029v4…