PaperlayerView exact revision on arXiv

Source-linked AI summary

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang

arXiv:2610.04299v1cs.LGcs.AIcs.CL

TL;DR

Repeated self-evolution can reverse early gains, motivating the question of how to sustain improvement. R-Quest guides self-evolution with validity and task-novelty feedback; across two model families it leads evaluated methods on 12 benchmarks and sustains gains over ten rounds.

  • Problem

    The paper asks why continued self-evolution can reverse early gains and how to sustain self-improvement.

    Source: §1 · p. 1

  • Method

    R-Quest trains the solver to reject invalid questions and uses validity judgments and sampled pairwise task-novelty comparisons to guide questioner rewards.

    Source: §1 · p. 2

  • Results

    Across two model families, R-Quest achieves the highest domain-average scores on 12 benchmarks and exceeds R-Zero by 17.32 points at the same tenth-round iteration on Qwen3-4B-Base.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support validity and task-novelty assessment as a practical route to sustained self-improvement.

    Source: §1 · p. 2

Abstract

from arXiv · show

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.

1 INTRODUCTION

Repeated self-evolution can reverse early gains, and the paper identifies invalid and repeated questions as quality bottlenecks. R-Quest uses validity and task-novelty feedback; across two model families it leads evaluated methods on 12 benchmarks and sustains gains over ten rounds.

1 INTRODUCTION

  • Invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in solver training data.The comparison study finds that repairing invalid questions improves solver performance under matched training settings.Source: §1 · p. 2
  • Lexical repetition penalties can miss variants of the same mathematical task, allowing repeated task structures to concentrate in training data.Different wording, notation, or numerical parameters can make equivalent tasks evade repetition penalties.Source: §1 · p. 2
  • R-Quest uses solver validity judgments and sampled pairwise task-novelty comparisons to guide questioner rewards toward valid, novel questions.The feedback signals are generated during self-evolution without further calls to stronger external models.Source: §1 · p. 2
  • Across two model families, R-Quest achieves the highest domain-average scores on 12 benchmarks and leads R-Zero by 17.32 points at the same tenth-round iteration on Qwen3-4B-Base.R-Quest reaches its best mathematical performance in round ten.Source: Figure 1 · p. 1, §1 · p. 2

2 PRELIMINARY STUDY

The preliminary study describes the questioner–solver cycle and examines two sources of training-data quality problems: invalid questions and repeated mathematical task types.

2.1 BACKGROUND: THE QUESTIONER–SOLVER FRAMEWORK

  • The questioner and solver share an initialization and train alternately with GRPO; the questioner rewards uncertainty and penalizes lexical-cluster repetition.Questions passing an intermediate-consistency filter train the solver using majority-answer pseudo-labels.Source: §2.1 · pp. 2–3

2.2.1 INVALID QUESTIONS IN SOLVER TRAINING

  • Valid-question rates fell from 76.5% in round one to 39.5% in round five; after consistency filtering, round-five validity was 27.64%.The study sampled 200 questions from each R-Zero round.Source: §2.2.1 · p. 3
  • With 1,273 matched questions and the same 766 valid items, repaired-data training reached 49.91 versus 46.54 on seven math benchmarks.The remaining 507 invalid questions in the original dataset were replaced by repaired counterparts.Source: §2.2.1 · p. 3

2.2.2 REPETITION IN GENERATED QUESTIONS

  • The three largest task types made up 59.5% of the fifth-round sample, while BLEU split the largest type across 22 clusters.Such fragmentation can weaken the cluster-size-based repetition penalty.Source: Figure 4 · p. 3, §2.2.2 · p. 4

3 METHOD

R-Quest adds validity and task-novelty feedback to questioner–solver self-evolution. It trains the solver to reject invalid questions, uses pairwise judgments to suppress repeated task types, and applies both signals during alternating updates.

3 METHOD

  • R-Quest uses solver validity judgments and frozen-model novelty comparisons to guide questioner rewards during alternating questioner–solver updates.Novelty comparisons assess whether questions share the same mathematical task structure, not merely similar wording.Source: §3 · p. 4, §3.2 · p. 4

3.1 VALIDITY-AWARE SOLVER INITIALIZATION

  • The validity-aware solver is trained to answer valid questions or return INVALID for flawed ones, with a penalty discouraging rejection of valid questions.Its training data combines questions sampled across five R-Zero iterations with verified reference answers.Source: §3.1 · p. 4

3.2 QUESTION NOVELTY ASSESSMENT VIA SAMPLED PAIRWISE JUDGMENTS

  • The frozen base model judges sampled question pairs as the same type when their mathematical setup and task match, despite wording or numerical changes.Sampling K comparisons per candidate reduces comparison cost from O(B^2) to O(BK); with K = 8, the stated pass probability is approximately 43% at a same-type fraction of 10%.Source: §3.2 · pp. 4–5

3.3 SELF-EVOLUTION WITH VALIDITY AND NOVELTY FEEDBACK

  • For each generated question, the questioner receives validity feedback and, for questions passing the validity check, a reward based on novelty and answer uncertainty.The validity gate classifies a question as invalid when more than half of sampled responses return INVALID; the questioner is updated with GRPO.Source: §3.3 · p. 5
  • Solver training uses candidate questions passing a majority-vote validity check and answer-consistency filtering, while mixing in fixed initialization data to preserve invalid-question recognition.Questioner and solver updates alternate.Source: §3.3 · p. 5

4 EXPERIMENTS

Across ten rounds and multiple reasoning domains, the experiments show that R-Quest sustains gains while improving question validity and limiting repeated mathematical tasks. The ablations and quality analyses examine the contributions of validity and novelty feedback.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers two model families and 12 benchmarks; methods alternate questioner and solver GRPO updates, with a 10,000-candidate-question budget per round.The benchmarks span mathematical reasoning, general-domain reasoning, and code generation; the reported domain checkpoints have the highest average benchmark score across evaluated rounds.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • R-Quest scores 51.92 in round ten, 17.32 points above R-Zero at the same iteration; every R-Quest checkpoint exceeds R-Zero’s best score.R-Zero peaks at 49.10 in round three and falls to 34.60 by round ten, below the base model’s 46.17.Source: Figure 5 · p. 6, §4.2 · pp. 6–7
  • R-Quest improves average code-generation scores over the base models by 3.90 points on Qwen3-4B-Base and 5.23 points on OctoThinker-3B, and has the highest general-reasoning average on both backbones.Table 2 reports code-generation pass@1 and general-domain reasoning results.Source: Table 2 · p. 7, §4.2 · p. 7

4.3.1 COMPONENT ABLATIONS

  • Removing either validity or novelty feedback lowers performance across all three domains, while a frozen validity judge underperforms full R-Quest.The authors report that continuous validity training through self-evolution is essential.Source: Table 3 · p. 7, §4.3.1 · pp. 7–8

4.3.2 VALIDITY ASSESSMENT DURING SELF-EVOLUTION

  • R-Quest’s invalid-question recall reaches 86.36% in round ten, while R-Zero’s remains between 10.23% and 26.14% across ten rounds.R-Quest’s balanced accuracy rises from 64.80% after initialization to 72.93% in round ten.Source: Figure 6 · p. 8, §4.3.2 · p. 8
  • R-Quest’s question validity rate rises from 88.0% in round one to 94.5–97.5% in rounds four through ten, while R-Zero’s falls from 76.5% to 49.0%.The analysis samples 200 unfiltered questions per method and round.Source: Figure 6 · p. 8, §4.3.2 · p. 8
  • Among valid sampled questions, R-Quest retains 54.69% majority-answer accuracy in round ten, compared with 2.06% for R-Zero.Accuracy is measured against independently verified references, with no credit for missing answers or unresolved judgments.Source: Figure 6 · p. 8, §4.3.2 · p. 8

4.3.3 QUESTION DIVERSITY DURING SELF-EVOLUTION

  • R-Zero’s top-five task-group share rises from 17.0% to 69.0% across rounds one to five, while R-Quest with K = 8 reaches 24.0% in round five and maintains higher performance.Groups are based on underlying mathematical construction and requested task; the figure also plots mathematical performance.Source: §4.3.3 · pp. 8–9, Figure 7 · p. 9

5 RELATED WORK

The related work situates R-Quest among self-evolving systems that generate training tasks and among approaches targeting validity and diversity. It distinguishes its feedback signals and their role in self-contained evolution.

Self-improving Reasoning Models

  • Self-evolving reasoning research includes label-free reinforcement learning and self-play frameworks that train question generators and solvers using model-derived feedback.Other cited work grounds learning in games, documents, vision tasks, and tool environments.Source: §5 · p. 9

Question Validity and Diversity in Self-evolution

  • Prior validity work studies synthetic mathematical data, while VHG uses the stronger external model GPT-5.4 to verify problems during self-play.R-Quest teaches the solver to reject invalid questions through mid-training, avoiding stronger-model calls during later self-evolution.Source: §5 · p. 9
  • Compared with representation- or code-embedding-based novelty methods, R-Quest asks whether question pairs express the same mathematical task and uses a frozen copy of the base model for feedback.The related work notes that R-Diverse uses an additional code generator and code encoder.Source: §5 · p. 9

6 CONCLUSION

The conclusion identifies invalid questions and repeated mathematical tasks as bottlenecks to sustained self-evolution. R-Quest uses validity and task-level novelty feedback, and reports strong benchmark averages and sustained gains.

6 CONCLUSION

  • Invalid questions and repeated mathematical tasks are identified as two bottlenecks to sustained self-evolution.Source: §6 · p. 10
  • R-Quest trains the solver to recognize and reject invalid questions and uses validity and task-level novelty feedback to guide question generation.Source: §6 · p. 10
  • Across two model families and 12 benchmarks, R-Quest achieves the highest domain-average scores in mathematics, code generation, and general reasoning, and sustains gains over ten rounds on Qwen3-4B-Base.Further analyses report stronger validity assessment, less task repetition, and more reliable majority-vote supervision across rounds.Source: §6 · p. 10

AI USE STATEMENT

The statement describes generative-AI use in the research workflow and summarizes materials supplied for reproducing the training and evaluation pipeline. The appendices also document dataset construction and answer-review procedures.

AI USE STATEMENT

  • Generative AI was used for synthetic data generation, model-based annotation and verification, evaluation, and limited language editing; the authors reviewed the manuscript and take responsibility for it.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper and appendices provide information to reproduce the main training and evaluation pipeline, including prompts, rewards, training settings, baselines, benchmarks, and metrics.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATASET CONSTRUCTION

  • The validity-aware initialization dataset contains 2,240 questions, assembled from sampled R-Zero training questions across five rounds after deduplication and review.The passage points to separate tables for dataset statistics and error types among invalid questions.Source: §A.1 · p. 14, Table 4 · p. 14, Table 5 · p. 14

A.2 ANNOTATION AND ANSWER REVIEW

  • Annotation first classifies questions for validity, then independently solves valid questions for reference answers; uncertain or unverifiable cases are reviewed or excluded.The procedure also reviews reference answers and excludes unresolved cases.Source: §A.2 · pp. 14–15

A.3 REWARD AND TRAINING SETTINGS

The validity-aware solver learns to answer valid questions and reject invalid ones, with rewards that distinguish correct solving from correct rejection. Semantic novelty judgments compare question pairs by their underlying exercise pattern, not just their wording.

A.3 REWARD AND TRAINING SETTINGS

  • The solver is trained to return a mathematical answer for valid questions and INVALID for invalid questions.The prompt asks it to assess whether a problem is well-defined, consistent, and solvable as stated.Source: §A.3 · pp. 15–16
  • Correct answers and correct rejections receive 1, rejecting a valid question receives −0.5, and other boxed responses receive 0.Responses with a missing or empty answer box receive −0.1.Source: §A.3 · p. 16
  • The solver is initialized from the checkpoint at step 15 after eight responses per question are grouped for GRPO advantages with a KL penalty to the base model.The validity-aware initialization trains Qwen3-4B-Base and OctoThinker-3B-Hybrid-Base with GRPO.Source: §A.3 · pp. 15–16

B.1 TASK SPECIFICATION

  • A pair is SAME_TYPE when its questions share a distinctive mathematical setup and requested task, despite differences in local details.Broad topic or similar wording alone is not sufficient for a SAME_TYPE judgment.Source: §B.1 · p. 17

B.2 EFFECT OF RANDOM SAMPLING

Randomly checking K references gives a scalable novelty gate whose strength depends on K; the paper also specifies how exercise-pattern concentration is grouped and measured, alongside its question-generation and solving prompts.

B.2 EFFECT OF RANDOM SAMPLING

  • 66.3% to 44.0% and 16.8% to 2.8%: approximate pass probabilities fall as K increases from 8 to 16 for patterns with 5% and 20% shares, respectively.The gate rejects a candidate if any of its K sampled references is judged the same exercise type.Source: §B.2 · p. 17, Figure 8 · p. 17
  • K = 16 lowers top-five pattern concentration but yields worse mathematical performance than K = 8, indicating that stricter suppression can limit exploration of challenging variants.The default K = 8 balances suppression of recurring patterns with room for continued exploration.Source: §B.2 · p. 18

B.3 EXERCISE-PATTERN CONCENTRATION ANALYSIS

  • Concentration is measured as the mean top-five share across three random orderings of 200 records, grouped by distinctive mathematical setup and requested task.Each question joins one existing group or establishes a new one; wording and local details alone do not define a new pattern.Source: §B.3 · p. 18

C.1 TASK SPECIFICATIONS

  • The generation prompt requests a new, non-trivial problem aimed at difficulty below a 30% solve rate among advanced high-school students, while the solving prompt requests a boxed answer.The paper uses the R-Zero question-generation and problem-solving instructions.Source: §C.1 · p. 19

C.2 SAMPLING AND TRAINING SETTINGS

Question evaluation combines validity voting and answer consistency; retained questions provide pseudo-labels for solver training, which also mixes in replay examples.

C.2 SAMPLING AND TRAINING SETTINGS

  • The questioner reward uses nine validity responses, rejects questions with a majority of INVALID outputs, and evaluates passing questions with ten fresh solutions.Questions failing the required output format receive a reward of −1.Source: §C.2 · p. 19
  • Solver data retains questions with answer consistency s(q) ∈ [0.3, 0.8] and a non-empty majority-vote answer as the pseudo-label.The same sequence of nine validity responses followed by ten solutions is used for candidate training questions.Source: §C.2 · p. 19
  • Replay examples make up 10% of the mixed solver dataset, while new questions use a reward of 0.9 for pseudo-label agreement and 0.1 for the required response format.Question evaluation and GRPO rollouts use temperature 1.0 and a maximum response length of 4,096 tokens; both models use a learning rate of 10^-6 and KL coefficient of 10^-2.Source: §C.2 · p. 19

D BASELINE DETAILS

The evaluated baselines include untrained base models, R-Zero, OCNR, and R-Diverse; the latter two add distinct novelty or repetition controls to the R-Zero framework.

D BASELINE DETAILS

  • Qwen3-4B-Base and OctoThinker-3B are evaluated without additional training to establish pre-self-evolution performance.Source: D BASELINE DETAILS · p. 20
  • OCNR adds a one-class detector novelty score to the R-Zero questioner reward and generates 20,000 candidates per round.The detector distinguishes solver-training questions from withheld questions using solver representations.Source: D BASELINE DETAILS · p. 20
  • R-Diverse penalizes repetition using solution-program similarity within batches and across rounds, with historical examples forming 30% of solver data from round two onward.Its implementation uses Qwen2.5-Coder-7B and Jina-Code-Embeddings-1.5B for the representations and similarity comparisons.Source: D BASELINE DETAILS · p. 20

E EVALUATION BENCHMARKS

Evaluation covers mathematical reasoning, general-domain reasoning, and code generation; the detailed ablations show that full R-Quest achieves the highest average performance in all three domains.

E EVALUATION BENCHMARKS

  • Evaluation spans mathematical reasoning, general-domain reasoning, and code generation despite self-evolution focusing on mathematical questions.Source: E EVALUATION BENCHMARKS · p. 21
  • Seven mathematical benchmarks use mean@32 for AMC and both AIME tests, and greedy-decoding accuracy for the others, with Qwen3-32B judging answer equivalence.Source: E EVALUATION BENCHMARKS · p. 21
  • Generalization is evaluated on MMLU-Pro, SuperGPQA, and BBEH using exact-match accuracy under greedy decoding.The benchmarks cover academic-domain questions, graduate-level disciplines, and logical, spatial, and multi-step reasoning tasks.Source: E EVALUATION BENCHMARKS · p. 21
  • Code generation uses HumanEval+ and MBPP+, with greedy decoding and pass@1 reported on the full EvalPlus test suites.The benchmarks assess Python function completion and short programs from natural-language task descriptions.Source: E EVALUATION BENCHMARKS · p. 21

F.1 DETAILED ABLATION RESULTS

  • Full R-Quest achieves the highest average performance in all three domains; removing either feedback lowers code and general-reasoning scores, while validity removal most affects OlympiadBench.Individual ablations still score higher on some benchmarks.Source: Table 9 · p. 22, §F.1 · p. 22
Loading 2610.04299v1…