Source-linked AI summary
Guided Self-Evolving LLMs with Minimal Human Supervision
Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, Dong Yu
TL;DR
Unguided self-evolving LLMs can plateau or degrade because self-play reinforces drift, bias, and diversity collapse. R-FEW adds lightweight human anchoring to Challenger generation and mixed, difficulty-based Solver training, producing stable iterative gains across reasoning benchmarks. Its reported results show that minimal human data can approach or surpass systems trained with much larger labeled datasets.
Problem
Unguided self-evolving language models can plateau or degrade as concept drift, diversity collapse, and biased self-reinforcement destabilize training.
Method
R-FEW grounds Challenger-generated questions with a few human anchor examples and trains the Solver online on mixed human and synthetic data using a curriculum.
Results
+3.0 points: Qwen3-8B-Base improves over R-Zero on math tasks and performs on par with General-Reasoner despite using far less human data.
Takeaways & Limitations
Minimal human grounding supports stable, scalable co-evolution and can approach or exceed systems trained on substantially larger labeled datasets.
Abstract
from arXiv · showhide
AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.
1 Introduction
Self-evolving language-model systems aim to improve from their own experiences, but unguided self-play can plateau or degrade as biases and low-diversity behaviors accumulate. R-FEW addresses this instability with limited human grounding and reports iterative reasoning gains.
- 1 Introduction: Self-play is intended to let models improve by competing against themselves and generating automatic feedback.This paradigm is motivated by successful self-play mechanisms in other AI settings.
- 1 Introduction: Unguided language-model self-play often plateaus or deteriorates because concept drift and diversity collapse reinforce biased, low-entropy behavior.Repeated self-play can amplify spurious correlations and biased reasoning patterns, reducing factual and semantic validity.
- 1 Introduction: R-FEW combines few-shot human anchoring for Challenger generation with online curriculum training for the Solver.The Challenger samples zero to five anchor examples, while the Solver is trained on generated data under an adaptive curriculum.
2 Preliminaries: Self-Play For Data-Free Training
Data-free LLM self-play casts training as a Challenger-Solver process in which generated problems, solver responses, and rewards drive reinforcement-learning updates. Existing frameworks differ mainly in how they shape Challenger and Solver rewards to control difficulty, answerability, uncertainty, and supervision.
- 2.1 Unified View of Self-Play Objectives and Reward Design: A Challenger generates tasks, a Solver answers them, and a verifier or self-consistency mechanism produces rewards for updating both agents.The Challenger conditions on historical context, while the Solver conditions on the generated question.
- 2.1 Unified View of Self-Play Objectives and Reward Design: The unified objective treats Challenger generation as an adaptive curriculum that elicits Solver reasoning improvement through reinforcement learning.This formulation covers language self-play methods under a common objective.
- 2.1 Unified View of Self-Play Objectives and Reward Design: Self-play frameworks instantiate different Challenger and Solver rewards, each encoding a distinct learning bias.Reward choices determine how systems encourage difficulty, uncertainty, answerability, informativeness, or stability.
- 2.1 Unified View of Self-Play Objectives and Reward Design: Absolute Zero uses a fully zero-data regime with linear difficulty shaping, while R-Zero uses uncertainty-driven curricula with verifiable pseudo-labels.R-Zero additionally uses a repetition penalty to discourage duplicate questions.
- 2.1 Unified View of Self-Play Objectives and Reward Design: SQLM targets a difficulty sweet spot using answer success statistics, whereas SPICE uses response variance and source-text supervision.SPICE treats response variance as an indicator of appropriate question difficulty.
3 Method
R-FEW combines few-shot human grounding for Challenger-generated tasks with an online curriculum Solver trained on synthetic and human examples. The loop selects moderately uncertain, increasingly challenging problems to support stable self-evolution.
- 3.1 Overview: R-FEW grounds synthetic task generation in a small pool of human anchor examples while retaining the self-evolving self-play loop.The Challenger samples up to five in-context demonstrations, with k = 0 recovering the data-free R-ZERO setting.
- 3.2 Few-Shot Grounded Challenger: The Challenger is optimized with uncertainty- and diversity-related rewards, with a supervised warm-up helping it follow the required output format.The warm-up addresses instruction-following difficulties caused by longer prompts.
- 3.2 Few-Shot Grounded Challenger: The Challenger generates questions near the Solver’s current ability frontier by targeting medium uncertainty and difficulty shaping.Anchor proximity encourages exploration near human data while allowing autonomous difficulty progression.
- 3.3 Online Curriculum Solver: The Solver aggregates synthetic questions and human anchors, estimates success rates, and filters both sources to a mid-difficulty quantile interval.The reported interval is [0.3, 0.7], selecting tasks intended to be challenging yet solvable.
- 3.3 Online Curriculum Solver: The Solver reward combines curriculum-weighted solution accuracy with an upweighted human-anchor term to reduce forgetting.The human-data coefficient is set to λ = 2.0 in practice.
4 Experiments
R-FEW is evaluated against self-evolving baselines across mathematical and general reasoning benchmarks, with additional ablations and analyses of domain effects and training stability. It consistently improves performance with 1% or 5% human data while reducing instability associated with unguided self-play.
- 4.2.1 Comparison with Baseline Methods: Qwen3-8B-Base reaches 55.1 with 1% data and 56.7 with 5%, surpassing General-Reasoner’s 56.0.The comparison uses substantially less human data than General-Reasoner.
- 4.2.2 Ablation Studies: Disabling challenger training reduces Math and General averages by 1.9 and 1.0 points, while removing warm-up or curriculum learning shows similar degradation.Math performance is more sensitive than general-domain performance.
- 4.2.3 Domain Corelation with Human Data: Domain-sampled human data most improves the matching MMLU-Pro category, with math additionally showing broad usefulness across complex reasoning domains.The analysis identifies math–physics and business–economics as strongly connected domain pairs.
- 4.2.4 Improved Stability and Reduced Reward Hacking: R-Zero’s diversity drops from 35 to below 20 in its first 50 steps, while later diversity gains coincide with verbosity-driven length inflation.R-FEW maintains stable diversity and length during self-evolution.
- 4.2.4 Improved Stability and Reduced Reward Hacking: Both methods generate increasingly difficult questions, but R-Zero’s increase is primarily caused by longer outputs whereas R-FEW maintains length while increasing challenge.Difficulty is measured as the percentage of questions answered incorrectly by Qwen3-8B-Base after Gemini-2.5-Pro relabeling.
5 Related Work
Related work extends self-play from games to language models and connects it with reinforcement-learning approaches for improving reasoning. These systems use self-generated competition, feedback, or externally defined verifiers to guide model behavior.
- Self-Play and Self-Evolving LLMs: AlphaZero-style self-play inspired language-model systems that continually refine reasoning without curated human data.The paradigm improves a policy through self-play matches and was later extended to LLM reasoning.
- Reinforcement Learning for LLM Reasoning: RLHF and related methods improve LLM behavior through preference optimization, while RLVR supplies scalar feedback from verifiers or environments.RLVR examples include search-based reasoning and semantically aware free-form generation.
6 Conclusion and Future Work
R-FEW enables autonomous LLM improvement with minimal human supervision through grounded question generation and online curriculum training, while supporting stable co-evolution and strong reasoning gains.
- R-FEW combines few-shot-grounded question generation with an online solver curriculum to produce stable, scalable co-evolution.The framework uses only 1–5% human data while approaching or surpassing systems trained with much larger labeled datasets.
- R-FEW achieves strong gains across mathematical and general reasoning tasks despite minimal human supervision.
- The authors identify efficiency, richer verification, and open-ended domains without objective correctness signals as future research directions.
A.1 Training Details
The training procedure alternates Challenger and Solver updates, using fixed rollout, batch-size, and learning-rate settings across 100 Solver steps and 50 Challenger steps.
- The Challenger trains for 50 steps and the Solver for 100 steps in iterative alternating updates.The Challenger is updated for 5 steps followed by 10 Solver-training steps rather than alternating every step.
- Both models use batch size 512, eight rollouts, and a learning rate of 5e-7.
A.2 Training Hyperparameter
The training setup includes a 100-step Qwen3-8B-Base Solver curve alternating with Challenger training, alongside mixed-precision implementation details.
- Figure 5 shows the Qwen3-8B-Base Solver training curve over 100 steps while alternating with the Challenger.
- Experiments use BF16 mixed precision and FlashAttention 2.
A.2.1 Solver Training
Solver training uses a global batch size of 512, eight rollouts, and rollout temperature 1.0.
- The Solver uses a global batch size of 512.
- The Solver generates 8 rollouts.
- The rollout temperature is 1.0.
A.2.2 Challenger Training
The challenger-training setup specifies rollout and batching parameters, prompt templates for problem generation, and GPT-4o-based answer evaluation.
- The configuration uses a global batch size of 512, eight rollouts, and rollout temperature 1.0.
- The solver prompt requests step-by-step reasoning with the final answer enclosed in \boxed{}, using a problem statement as input.
- The challenger prompt asks an expert problem setter to create brand-new, non-trivial reasoning problems across academic subjects.
- Challenger generation is conditioned on five example questions, with each output required inside {question} and {/question} tags.
- Mathematical answer correctness is programmatically judged by GPT-4o using an equivalence prompt limited to trivial simplifications.