Source-linked AI summary
DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution
Shengda Fan, Xuyan Ye, Yankai Lin
TL;DR
Existing LLM self-play systems face instability from solver-dependent Questioner objectives and noisy pseudo-labels for Solver training. DARC decouples question generation from Solver updates and uses a document-augmented teacher to supervise a question-only student. Across its evaluations, DARC improves reasoning performance and approaches fully supervised models without human annotations.
Problem
Existing coupled self-play systems face objective drift for the Questioner and noisy self-generated pseudo-labels for the Solver, making stable self-evolution difficult.
Method
DARC trains a difficulty-aware Questioner with fixed corpus-grounded supervision, then distills document-augmented teacher outputs into a question-only student Solver.
Results
DARC consistently improves heterogeneous Solver backbones, with training validation rewards rising without degradation and document augmentation producing win rates above 50% in short- and medium-context regimes.
Takeaways & Limitations
DARC provides a decoupled curriculum that generalizes across Solver sizes and more effectively induces transferable reasoning abilities than standard next-token fine-tuning.
Takeaways & Limitations
DARC relies on external corpora, uses inevitably noisy pseudo-labels, and is primarily designed for domains with verifiable answers.
Abstract
from arXiv · showhide
Self-play with large language models has emerged as a promising paradigm for achieving self-improving artificial intelligence. However, existing self-play frameworks often suffer from optimization instability, due to (i) non-stationary objectives induced by solver-dependent reward feedback for the Questioner, and (ii) bootstrapping errors from self-generated pseudo-labels used to supervise the Solver. To mitigate these challenges, we introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process. First, we train the Questioner to synthesize difficulty-calibrated questions, conditioned on explicit difficulty levels and external corpora. Second, we train the Solver with an asymmetric self-distillation mechanism, where a document-augmented teacher generates high-quality pseudo-labels to supervise the student Solver that lacks document access. Empirical results demonstrate that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models. Moreover, DARC consistently outperforms all baselines and approaches the performance of fully supervised models without relying on human annotations. The code is available at https://github.com/RUCBM/DARC.
1 Introduction
DARC addresses instability in LLM self-play by decoupling Questioner and Solver training and using document-grounded asymmetric distillation. It improves reasoning across model families, outperforming self-evolving baselines and approaching fully supervised performance without human annotations.
- Human-annotated data scarcity motivates self-evolution methods that can improve LLM capabilities without human data.
- Coupled self-play destabilizes optimization because the Questioner tracks a moving Solver-dependent target while the Solver learns from noisy pseudo-labels.
- DARC separates Questioner and Solver training into sequential stages, avoiding real-time dependence on Solver performance during question generation.
- A document-augmented teacher uses majority voting to supervise a question-only student Solver, reducing label noise and self-confirmation bias.
- 10.9 points average accuracy improvement over base models establishes DARC as model-agnostic across Qwen and LLaMA backbones.
- DARC approaches General-Reasoner trained on 232K WebInstruct examples despite using no human annotations.
2 Related Work
Prior work uses reinforcement learning, self-play, and synthetic data to scale reasoning supervision, but general-reasoning self-play remains vulnerable to unstable rewards and weak supervision.
- External verifiers provide reliable rule-based rewards but are largely restricted to deterministic domains such as mathematics and code.
- Self-supervision methods use confidence maximization or majority-vote pseudo-labels, yet remain dependent on curated question sets.
- General-reasoning questioner–solver self-play often suffers from non-stationary rewards and unstable optimization unlike code-centric systems with executable feedback.
- Synthetic-data pipelines expand supervision through bootstrapped instructions or corpus-driven question-answer generation, but often rely on superior external models for quality control.
3 Methodology
DARC decomposes self-evolution into sequential Questioner and Solver stages, then analyzes why decoupling mitigates instability in coupled self-play.
- Framework overview: DARC separates controllable question generation from offline Solver training, followed by a theoretical analysis of optimization stability.The framework uses explicit difficulty levels and an external corpus for the Questioner, then asymmetric self-distillation for the Solver.
- 3.1 Questioner: Difficulty-Aware Generation: The Questioner generates corpus-grounded questions conditioned on a document d and target difficulty τ.Its policy q ∼ Qθ(· | d, τ) is trained to match specified difficulty while remaining grounded in the source document.
- 3.1 Questioner: Difficulty-Aware Generation: The difficulty estimator D(q) uses the empirical success rate of a fixed Solver over N sampled answers, with majority voting providing the pseudo-label a∗.The resulting estimator is D(q) = ŝ, and the reward encourages alignment between D(q) and τ after grounding verification.
- 3.2 Solver: Offline Curriculum Learning via Asymmetric Self-Distillation: After freezing the Questioner, DARC orders generated questions from easy to hard by τi and trains the Solver progressively on this offline curriculum.The curriculum is constructed after Questioner training rather than through an online Questioner–Solver loop.
- 3.2 Solver: Offline Curriculum Learning via Asymmetric Self-Distillation: Asymmetric self-distillation uses a document-augmented teacher to generate majority-vote pseudo-labels, while a student Solver learns without document access.Samples with vote agreement below γ are discarded to reduce label noise; the student is optimized with a correctness reward.
- 3.3 Theoretical analysis: The coupled-self-play analysis shows that Solver updates can reverse the Questioner’s ascent direction, making a current-objective update decrease the next objective.The instability arises from objective drift induced by Solver updates, not stochastic noise or large learning rates; DARC instead uses a fixed difficulty estimator and explicit curriculum scheduling.
4 Experiments
DARC is evaluated across mathematical and general reasoning benchmarks, backbone families, training dynamics, and targeted analyses. It consistently improves reasoning, remains competitive with supervised or weakly supervised alternatives, and shows stability, cross-solver generalization, and benefits from its core components.
- Evaluation Setup: DARC is evaluated on mathematical and general reasoning benchmarks using Qwen- and LLaMA-family backbones.The evaluation includes MATH-500, GSM8K, OlympiadBench, Minerva Math, AMC, MMLU-Pro, SuperGPQA, and GPQA-Diamond.
- Main Results: 10.9 points: DARC improves average benchmark scores over base models while outperforming label-free baselines across three backbones.DARC remains competitive with weakly supervised R-Few while requiring no human annotations.
- Training Stability: DARC's validation reward rises steadily without degradation, while curriculum transitions produce transient training-reward drops at steps 32 and 64.The drops coincide with transitions from Easy to Medium and Medium to Hard, while validation reward contrasts with collapse reported in prior self-play systems.
- Main Results: DARC approaches the performance of General-Reasoner, despite General-Reasoner using the full 232K WebInstruct dataset with supervision.General-Reasoner is identified as a supervised reference and is not directly comparable to self-evolving baselines under the same supervision regime.
- Asymmetric Self-Distillation: Document augmentation improves supervision in short- and medium-context regimes, but gains diminish beyond 5K tokens because excessive length may dilute the supervisory signal.Table 2 reports Avg@8 win rates above 50% for document-augmented prompting in the shorter regimes.
- Questioner Analysis: Solver accuracy decreases monotonically from Easy to Hard across backbones, indicating solver-independent difficulty rankings for generated questions.The resulting difficulty partition is described as solver-agnostic and input-driven.
- Questioner Analysis: Question sets generated by a Qwen3-4B-Base Questioner improve both larger 8B and smaller 1.7B Solvers, demonstrating cross-solver generalization.Further tuning on human-annotated data yields additional gains, indicating that the synthesized curriculum is complementary to human supervision.
5 Conclusion
DARC combines decoupled training with asymmetric self-distillation to stabilize LLM self-evolution. Experiments suggest it consistently outperforms existing baselines.
- DARC adopts decoupled training and asymmetric self-distillation to stabilize self-evolving.
- DARC consistently outperforms existing baselines across the reported experiments.
Limitations
DARC has scope and supervision constraints despite its progress toward stable LLM self-evolution. Its applicability is limited by external-corpus dependence, noisy pseudo-labels, and reliance on verifiable answers.
- DARC relies on an external corpus to ground both the Questioner and Solver, constraining fully data-free applications.
- Asymmetric self-distillation produces inevitably noisy pseudo-labels, potentially limiting further performance gains.
- DARC is primarily designed for domains with verifiable answers, restricting applicability to open-ended tasks.
Ethical Statement
The work studies LLM self-evolution as an offline research methodology rather than a deployed user-facing system. It uses public corpora without collecting user data, while acknowledging residual corpus-content risks.
- The models operate in an offline experimental environment and are not used for autonomous decision-making.
- The study uses publicly available corpora, does not collect user data, and reports only aggregated benchmark results.
- ChatGPT assisted only with language refinement and presentation clarity, not ideas, experiments, analysis, interpretation, or conclusions.
- Experiments use 8 NVIDIA A800 GPUs and set the maximum prompt length to 8,192 tokens and generation limit to 4,096 tokens.
A.2 Questioner Training Details
Questioner training uses external documents and multiple sampled candidates to construct valid difficulty-conditioned questions and support pseudo-label reliability. A binary judge checks document grounding rather than answer correctness.
- Questioner training uses 10,000 Nemotron-CC-Math documents and 10,000 DataComp-LM documents for one epoch with global batch size 16.
- Each document-difficulty pair yields G = 8 candidate questions, while each question receives N = 8 privileged-Solver trajectories for pseudo-labeling.
- The judge provides binary feedback on document grounding, not answer labels or knowledge distillation.
- The final Solver training set retains valid-format questions from up to 60,000 generated candidates without additional filtering.
- Pseudo-label reliability is balanced with training efficiency using N = 8 rollouts and acceptance threshold γ = 0.3.
B Effect of Curriculum Learning in Training Solver
The study isolates curriculum ordering by comparing curriculum learning with random shuffling on the same offline question set. Curriculum learning improves early-stage sample efficiency without increasing the number of active prompts.
- Validation reward reaches 0.7 in 24 steps with curriculum learning, compared with 32 steps under random shuffling.
- The efficiency gain is not attributable to accessing more active prompts, whose numbers remain comparable across settings.
C Training Dynamics of the Questioner
Questioner training shows rapidly increasing early reward followed by a plateau, while KL loss rises gradually before stabilizing. These trends indicate stable optimization dynamics approaching a steady regime across both model scales.
- Questioner reward increases rapidly early in training and then plateaus for both model scales.The diagnostic considers Qwen3-4B and Qwen3-8B backbones.
- KL loss rises gradually before stabilizing during GRPO training.
- The combined reward and KL trends suggest optimization gradually approaches a steady regime.
D Case Study of Generated Questions
The case study evaluates Questioner outputs under Easy, Medium, and Hard conditions using an input document and the specified Solver–Questioner prompting setup. Generated questions become progressively more difficult while remaining self-contained and document-inspired.
- Case Study of Generated Questions: Easy, Medium, and Hard conditions produce questions with progressively higher solving difficulty.The Easy question uses more direct shallow reasoning, whereas Medium and Hard questions add nuanced constraints and more competitive distractors.
- Case Study of Generated Questions: The generated questions remain self-contained even though the Questioner does not access the source document during generation.
- Questioner Prompt: The prompt requires one self-contained question and its correct answer inspired by the input material.
- Questioner Prompt: The question and solution process must match the target difficulty level and answer-type constraints.
- Questioner Prompt: Questions must be self-contained, unambiguous, variable-complete, uniquely answerable, and not copied from the source material.
- Questioner Prompt: The output schema includes analysis, question, intermediate_results, answer, solving_time_estimate, required_concepts, and potential_errors.