PaperlayerView exact revision on arXiv

Source-linked AI summary

R-Zero: Self-Evolving Reasoning LLM from Zero Data

Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, Dong Yu

arXiv:2508.05004v4cs.LGcs.AIcs.CL

TL;DR

Existing self-evolving LLM training depends on costly, labor-intensive human-curated tasks and labels, limiting scalable autonomous improvement. R-Zero instead co-evolves a Challenger and Solver from a single base model to generate training data autonomously, improving reasoning across backbones and domains, including a +6.49-point gain for Qwen3-4B-Base on math benchmarks.

  • Problem

    Existing self-evolving LLM training relies on costly, labor-intensive human-curated tasks and labels, limiting scalable autonomous improvement.

  • Method

    R-Zero co-evolves independently optimized Challenger and Solver models from one base LLM, generating challenging questions and solver training data without external data.

  • Results

    R-Zero consistently improved reasoning across backbones and domains, with Qwen3-4B-Base gaining +6.49 points on average across math benchmarks after three iterations.

  • Takeaways & Limitations

    R-Zero supports a model-agnostic, self-supervised route to iterative reasoning improvement without externally curated training data.

  • Takeaways & Limitations

    R-Zero depends on objectively verifiable correctness signals, making application to open-ended or subjective tasks difficult.

Abstract

from arXiv · show

Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.

1 INTRODUCTION

R-Zero addresses the scalability bottleneck of human-curated training data by enabling a base LLM to self-evolve from zero external data through co-evolving Challenger and Solver roles. Experiments show iterative, model-agnostic reasoning improvements, including a +6.49-point gain on Qwen3-4B-Base math benchmarks after three iterations.

  • Self-evolving LLMs autonomously generate, refine, and learn from their own experiences, offering a scalable pathway toward artificial superintelligence (Tao et al., 2024; Tan et al., 2024).
  • Human-curated tasks and labels are costly, labor-intensive, difficult to scale, and a fundamental bottleneck for training self-evolving LLMs.These resources provide supervision for fine-tuning and reinforcement learning with verifiable rewards (RLVR) (Shao et al., 2024; DeepSeek-AI et al., 2025).
  • Prior label-free reinforcement-learning methods derive rewards from model outputs, including sequence-level confidence scores and output entropy.Examples using confidence scores include (Li et al., 2025a; Prabhudesai et al., 2025; Huang et al., 2025), while entropy-based approaches include (Agarwal et al., 2025; Cheng et al., 2025).
  • R-Zero initializes one base model as independently optimized Challenger and Solver roles that co-evolve during reinforcement learning.The Challenger generates tasks near the Solver’s current capability edge, while the Solver learns by solving increasingly difficult Challenger-generated tasks.
  • +6.49 points: Qwen3-4B-Base’s average math-benchmark score increased after three self-evolution iterations.The experiments also report significant improvements on complex general-domain tasks, indicating that math-focused training can transfer beyond mathematics.

2 METHOD

R-Zero is a fully automated framework in which a Challenger and Solver, initialized from the same base LLM, co-evolve through iterative question generation, filtering, and training. The Challenger targets questions near the Solver’s capability boundary, while the Solver learns from these curated challenges using verifiable rewards.

  • Framework overview: R-Zero co-evolves a Challenger and Solver from the same base LLM through an iterative curriculum-generation loop.The Challenger generates difficult questions, after which filtered question-answer pairs train the Solver.
  • Challenger training: The Challenger is trained with GRPO to generate mathematical questions that are challenging yet solvable for the current Solver.Its uncertainty reward favors questions where the Solver’s empirical accuracy approaches 50%.
  • Challenger training: The Challenger’s reward combines Solver uncertainty with a BLEU-based repetition penalty, while malformed outputs receive zero reward before further scoring.The reward pipeline also applies a structural format check requiring <question> and </question> tags.
  • Curriculum generation: Candidate questions enter the Solver curriculum only when majority-vote correctness lies within the specified interval, removing tasks that are too easy or too hard.Filtering also improves pseudo-label quality by excluding ambiguous, ill-posed, or unreliable low-consistency items.
  • Solver training: The Solver is fine-tuned with GRPO on curated challenges using a binary verifiable reward for matching the majority-vote pseudo-label.The resulting policy update improves the Solver’s ability to answer difficult questions generated by its co-evolving Challenger.

3 EXPERIMENTS

R-Zero improves mathematical reasoning across model architectures and scales, while its self-generated math curriculum also transfers to general-domain reasoning. Experiments use held-out test sets and evaluate final tabular results after 45 training steps.

  • Experimental setup: Experiments evaluate Qwen3-4B/8B-Base and OctoThinker-3B/8B, covering scale within one family and transfer across architectures.The Qwen3 models are from Yang et al. (2025), and the OctoThinker models are from Wang et al. (2025b).
  • Mathematical reasoning: +5.51 points on Qwen3-8B-Base and +2.68 points on OctoThinker-3B after three R-Zero iterations demonstrate consistent mathematical gains across scales and architectures.Qwen3-8B-Base rises from 49.18 to 54.69, while OctoThinker-3B rises from 26.64 to 29.32.
  • Evaluation protocol: All tabulated results use 45 training steps and held-out test sets, while figures evaluate solver checkpoints every 15 steps.The evaluation code was adopted from General-Reasoner (Ma et al., 2025) and rerun for consistency.
  • Mathematical reasoning: +3.7 points on Qwen3-4B-Base and +1.52 points on Qwen3-8B-Base arise at the first R-Zero iteration over the Base Challenger baseline.This validates the contribution of RL-based Challenger training to generating a more effective curriculum than a non-trained generator.
  • General-domain reasoning: +5.13 points on Qwen3-8B-Base and +3.65 points on OctoThinker-3B show that math-focused R-Zero training transfers to general-domain reasoning.The transfer appears across all tested models and follows the performance patterns observed in mathematical reasoning.

4 ANALYSIS

R-Zero’s analyses show that its components improve data quality and performance, but iterative self-improvement eventually becomes unstable as generated-question difficulty rises and pseudo-label accuracy falls. R-Zero also complements supervised fine-tuning, with sequential application outperforming concurrent data mixing.

  • Ablation study: Removing the Repetition Penalty significantly degrades performance, indicating that diverse Challenger questions are crucial for effective Solver training.The ablation disables the Challenger’s repetition penalty and compares performance on Qwen3-4B-Base.
  • Ablation study: Disabling Task Filtering causes a notable performance drop, including a general-domain average decline of over 6 points.Filtering calibrates curriculum difficulty and removes low-consistency, potentially ambiguous or ill-posed questions as implicit quality control.
  • Iteration scaling dynamics: Performance initially improves across model sizes but eventually degrades, with larger models delaying rather than preventing collapse.The 0.6B model peaks at Step 15, whereas the 4B model rises for three iterations before sharply dropping at Step 60.
  • Co-evolutionary dynamics: 79.0% pseudo-label accuracy falls to 63.0% by the third iteration as Challenger questions become progressively more difficult.A static Solver’s performance drops from 59.0% on Step 15 questions to 47.0% on Step 45 questions against consistent GPT-4o ground truth.

5 RELATED WORK

Related work pursues label-free LLM reasoning through self-generated rewards, dual-role self-play, and reinforcement learning with verifiable rewards. These approaches span confidence- or consistency-based signals, agent interaction, and applications across code and other task domains.

  • Self-Play: Self-play assigns models dual roles to create self-improvement loops without human data, with especially fruitful applications in verifiable code generation (Chen et al., 2024; Zhang et al., 2024; Lin et al., 2025; Pourcel et al., 2025).In code generation, a “Coder” agent’s program is verified by a “Tester” agent’s unit tests.
  • Reinforcement Learning with Verifiable Rewards: Reinforcement Learning with Verifiable Rewards enhances LLMs across diverse tasks, including relation extraction, interactive GUI navigation, and search-engine utilization (Li et al., 2025d; DeepSeek-AI et al., 2025; Shao et al., 2024; Shi et al., 2025b; Jin et al., 2025).Early implementations relied on rule-based verifiers, while the supplied passage indicates that newer work has begun exploring further developments.

6 LIMITATION

R-Zero’s applicability is limited by its reliance on objectively verifiable domains and deterministic evaluation signals for reliable Challenger–Solver training feedback. Its current labeling and verification strategies may not generalize across all task types, motivating more robust mechanisms.

  • R-Zero relies on objectively verifiable domains and deterministic evaluation signals to generate reliable training feedback through Challenger–Solver co-evolution.
  • Its current labeling and verification strategies may not generalize to all task types.
  • More robust and broadly applicable labeling mechanisms could expand the range of domains where R-Zero can be effectively applied.

7 CONCLUSION AND FUTURE WORK … B.1 TRAINING HYPERPARAMETER

R-Zero is presented as a fully autonomous self-evolving framework whose Challenger and Solver co-evolve to generate a self-generating curriculum without data dependency. The paper reports improved reasoning capability across multiple domains and identifies efficiency, robust labeling, and domain expansion as future directions.

  • 7 CONCLUSION AND FUTURE WORK: R-Zero is a fully autonomous self-evolving framework designed to overcome data dependency.Its curriculum is generated through the interaction of a Challenger and Solver.
  • 7 CONCLUSION AND FUTURE WORK: The Challenger and Solver co-evolve to create a self-generating curriculum.
  • 7 CONCLUSION AND FUTURE WORK: Experiments demonstrate that R-Zero improves LLM reasoning capability across multiple domains.
  • 7 CONCLUSION AND FUTURE WORK: Future work includes improving efficiency, exploring more robust labeling techniques, and expanding R-Zero to new domains.
  • B.1 TRAINING HYPERPARAMETER: The training-hyperparameter section summarizes critical algorithmic hyperparameters for both Solver and Challenger training stages.
  • B.1 TRAINING HYPERPARAMETER: All experiments used BFloat16 (BF16) mixed precision and FlashAttention 2.

B.1.1 SOLVER TRAINING … 3. Final Penalty Calculation

The implementation specifies separate solver and challenger training settings, exact generation and evaluation prompts, and a BLEU-based repetition penalty computed through hierarchical clustering. Together, these components define the training, judging, diversity-control, and penalty-calculation procedures.

  • B.1.1 SOLVER TRAINING: Solver training uses a global batch size of 128, 5 rollouts, and rollout temperature 1.0.
  • B.2 PROMPT TEMPLATES: The prompt templates require step-by-step reasoning with the final answer in \boxed{} and direct the Challenger to generate novel, difficult competition-math questions in a strict format.The solver prompt inserts a problem statement, while the Challenger outputs one question enclosed in <question> tags and no extra text.
  • B.3 GPT-4O JUDGE PROMPT: GPT-4o evaluates mathematical answers by comparing a model-generated answer with the benchmark ground truth and returning only Yes or No.The evaluation uses a dedicated math-answer-checker configuration and supports complex final answers.
  • B.4 REPETITION PENALTY IMPLEMENTATION: The Challenger’s repetition penalty discourages semantically similar questions within a batch by clustering questions according to BLEU-score similarity.
  • 1. Pairwise Distance Calculation via BLEU Score: Pairwise question distance is defined as one minus sentence BLEU, using NLTK’s first smoothing method and whitespace tokenization without further normalization.
  • 2. Agglomerative Clustering: Similar questions are grouped with scikit-learn agglomerative hierarchical clustering using a precomputed distance matrix and average linkage.
  • 3. Final Penalty Calculation: The final repetition penalty for each question is determined by the relative size of its assigned cluster within the total batch.For question xi in cluster Ck, the calculation uses |Ck| and the batch size B.

C EVALUATION BENCHMARK

The evaluation uses a comprehensive benchmark suite split between mathematical reasoning and general-domain reasoning, testing whether math-focused training improvements generalize across domains.

  • Evaluation scope: The evaluation is divided into mathematical reasoning and general-domain reasoning categories to test both core performance and cross-domain generalization.The question-generator prompt focuses primarily on mathematical problem-solving, while the evaluation examines whether resulting reasoning improvements transfer to other domains.
  • Mathematical Reasoning: Mathematical reasoning is evaluated on seven benchmarks: AMC, Minerva, MATH-500, GSM8K, OlympiadBench, AIME-2024, and AIME-2025.GPT-4o programmatically verifies complex final answers against ground truth; mean@32 is reported for AMC and AIME.
  • General Domain Reasoning: General-domain reasoning is evaluated with MMLU-Pro and SuperGPQA, covering challenging multitask and graduate-level questions across 285 disciplines.SuperGPQA questions are verified as unsearchable on the web to isolate reasoning ability from simple knowledge recall.
  • General Domain Reasoning: General-domain results follow Ma et al. (2025) using its experimental setup, prompts, and evaluation code, with Exact Match accuracy from greedy decoding.This preserves a standardized evaluation procedure for the general-domain benchmarks.

D PARAMETER SHARING BETWEEN CHALLENGER AND SOLVER

An ablation compares standard two-model R-Zero with Single-R-Zero, which uses one shared-parameter model for both Challenger and Solver roles. Separating the roles is reported as crucial for performance and stability.

  • D Parameter Sharing Between Challenger and Solver: The ablation tests Single-R-Zero, a unified shared-parameter model that generates the curriculum and learns from it, against the standard two-model framework.The study evaluates whether independent Challenger and Solver models are necessary for R-Zero’s success.
  • D Parameter Sharing Between Challenger and Solver: 49.12 peak performance is achieved by two-model R-Zero, which improves through more iterations and collapses after iteration three, whereas Single-R-Zero peaks after iteration one.The reported findings identify role separation as important for both performance and stability.

E BEYOND LABEL NOISE: UNPACKING THE ROOTS OF INSTABILITY

The analysis finds that declining pseudo-label quality is associated with performance collapse but is neither its sole nor universal trigger. It instead points to broader instability from training exclusively on self-synthesized data, including degenerative feedback effects.

  • Pseudo-label quality and instability: The absolute percentage of label noise is not the sole determinant of instability, so degraded pseudo-label quality cannot fully explain the collapse.The study tests pseudo-label degradation as a primary failure mode of self-correction but concludes it is neither the primary nor the sole driver.
  • Pseudo-label quality and instability: Pseudo-label accuracy consistently declines across iterations, but no universal accuracy threshold triggers performance collapse; the 0.6B model declines at 70.6% accuracy.Table 7 reports pseudo-label data quality at each model’s collapse onset, showing that collapse can begin while accuracy remains relatively high.
  • Model collapse from self-synthesized data: Training exclusively on self-synthesized data may induce model collapse through a degenerative feedback loop that reduces diversity or amplifies model biases (Tan et al., 2024; Shumailov et al., 2024; Dohmatob et al., 2024b; Zhou et al., 2025c; Seddik et al., 2024; Dohmatob et al., 2024a; Briesch et al., 2023; Zheng et al., 2025a).This proposed mechanism presents a broader and more fundamental challenge than label noise alone.

F THEORETICAL ANALYSIS … I.2 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS

R-Zero is theoretically motivated to generate maximally informative tasks by targeting a Solver success probability of 50%, and its algorithm co-evolves Challenger and Solver models through filtered self-play and verifiable rewards. Experiments indicate that removing the math-only prompt improves broader reasoning performance for sufficiently capable models, while the method builds on GRPO and RLVR.

  • F THEORETICAL ANALYSIS: The uncertainty reward r_uncertainty ∝ 1 − 2|p̂(x; S_ϕ) − 1/2| is maximized when the Solver’s success probability is 50%, targeting tasks at its capability frontier.The analysis connects frontier training efficiency to the KL divergence D_KL(S_ϕ||S*) and shows that this divergence is lower-bounded by reward variance, which is maximal at p̂ = 0.5.
  • G ALGORITHM OF R-Zero: R-Zero alternates Challenger training, Solver dataset construction, and Solver GRPO training during each self-play iteration.The algorithm samples question groups, applies format and repetition-based filtering, creates majority-vote pseudo-labels, retains sufficiently consistent examples, and updates the Solver using binary correctness rewards.
  • G ALGORITHM OF R-Zero: The algorithm constructs Solver data by sampling candidate questions, majority-voting pseudo-labels, estimating correctness, filtering by threshold δ, and training with binary rewards through GRPO.This pipeline uses N candidate questions and m Solver samples per question, then updates S_ϕ on curated pairs.
  • H GENERALIZING BEYOND THE MATH DOMAIN: The generalization experiment removes the math-specific prompting constraint to generate commonsense, logical-reasoning, and other non-mathematical questions.It tests whether gains persist when supervision is more heterogeneous rather than restricted to structured mathematical formats.
  • H GENERALIZING BEYOND THE MATH DOMAIN: The “R-Zero (remove math prompt)” setting achieves the highest general-benchmark performance for the 8B model, while broader training also improves mathematics benchmarks.This supports transfer beyond pure mathematics and indicates that the method benefits from sufficient model capacity and more diverse training data.
  • I PRELIMINARIES: The preliminaries situate R-Zero within recent reinforcement-learning methods for fine-tuning large language models.The paper reviews GRPO and RLVR as the two methodologies most relevant to its framework.
  • I.1 GROUP RELATIVE POLICY OPTIMIZATION: GRPO normalizes rewards within response groups to stabilize policy optimization without a separately learned value function.Its clipped surrogate objective increases probabilities for responses with positive relative advantages, while clipping and a β-weighted KL penalty limit aggressive policy drift.
  • I.2 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS: RLVR fine-tunes models with binary rewards from a rule-based verifier that deterministically checks response correctness.This reward structure is especially suited to mathematics and code generation and underlies the Solver’s reward mechanism.
Loading 2508.05004v4…