Source-linked AI summary
DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory
Xincheng Wei, Yifan Ding, Yoshua Li, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Wenjian Ding, Yao Zhang
TL;DR
Self-play needs guidance that targets unresolved reasoning weaknesses without relying on external task resources. DiagEvo derives that guidance from recurring solver failures through hierarchical error memory, mixed generation, and double-confidence filtering, and it outperforms the compared baselines across the evaluated solvers and benchmarks.
Problem
Self-play guidance based on difficulty, learnability, or diversity does not specify which recurring reasoning weaknesses later rounds should target, while guided methods use external task resources.
Method
DiagEvo diagnoses recurring error causes from failed solver trajectories, stores them as Active or Mastered in hierarchical memory, mixes targeted generation with exploration, and filters conflicting pseudo-labels.
Results
DiagEvo achieves the highest overall score among compared methods across Qwen3-4B, Qwen3-8B, and OctoThinker-8B; on Qwen3-8B, its mathematical reasoning average is 72.3%, 4.5 points above R-Zero.
Takeaways & Limitations
Solver failure history can serve as an internal curriculum signal for subsequent self-play rounds without external task resources.
Takeaways & Limitations
Confidence filtering measures agreement among solver responses rather than ground-truth correctness, so a shared error can still pass the filter.
Abstract
from arXiv · showhide
Self-play is an effective paradigm for language-model self-evolution, but without guidance, solver performance can plateau or decline across rounds. Unguided methods steer question generation with signals such as difficulty, learnability, or diversity. These signals keep questions challenging and varied but do not specify which unresolved reasoning weaknesses later rounds should target. Guided methods obtain direction from external task resources, including human examples, document corpora, or specified difficulty targets, and therefore rely on task information supplied outside the self-play loop. We show that the needed direction can instead be derived from the solver's own failure history. We introduce DiagEvo, whose diagnostician extracts recurring error causes from this history and stores them in a hierarchical error-cause memory. The memory groups related causes under skill nodes and tracks each as Active or Mastered according to self-consistency on targeted questions. The challenger uses these states and recurrence counts to balance cause-targeted generation with free exploration. Double-confidence filtering retains intermediate-difficulty questions only when the most common solver answer has a clear vote lead. DiagEvo derives its curriculum from information produced during self-play, without external task resources. With the default 4B diagnostician, DiagEvo outperforms every baseline in mean accuracy across all nine benchmarks for each of the three solvers: Qwen3-4B, Qwen3-8B, and OctoThinker-8B. On Qwen3-8B, it reaches 72.3% mean accuracy across five mathematical reasoning benchmarks, 4.5 percentage points above R-Zero. Its mean accuracy across all nine benchmarks is 57.4%, 1.1 percentage points above DARC. Ablations show that the hierarchical error-cause memory and double-confidence filtering both contribute to these gains.
1 Introduction
DiagEvo addresses the difficulty of sustaining learnable, targeted self-play by deriving curriculum guidance from recurring solver failures rather than external task resources. It combines hierarchical error memory, mixed generation, and confidence filtering to improve performance across the evaluated solvers.
- Self-play must keep questions near the solver’s competence boundary: difficult enough to expose weaknesses but learnable enough to provide stable training signals.
- Label-free difficulty feedback estimates how hard questions are but does not identify recurring reasoning causes, allowing surface complexity to grow without targeted practice.
- Guided self-play uses human examples, document corpora, difficulty labels, or privileged teachers, so continued improvement depends on information outside self-play.
- DiagEvo compares failed and agreeing solver trajectories to identify transferable error causes and tracks their recurrence for future curriculum generation.
- DiagEvo organizes causes into Active and Mastered states, mixes cause-targeted generation with free exploration, and filters high-conflict pseudo-labels before solver training.
- On Qwen3-8B, DiagEvo reaches 72.3% average mathematical reasoning accuracy, 4.5 points above R-Zero, while using no external task resources.
2 Related Work
Prior work uses self-play, interaction memory, curriculum learning, and pseudo-label safeguards to improve training, but DiagEvo applies structured failure memory directly to question generation within self-play.
- Label-free self-play updates challenger and solver models without human annotations, but questions may become too easy or difficult and pseudo-labels less reliable.
- Language-agent methods store successful and failed interactions, summaries, or skills to guide task-solving decisions or policy updates.
- Curriculum methods control example exposure, filter learnable questions, or create answer-preserving variants for reinforcement learning.
- DiagEvo builds structured memory from solver failures, tracks causes as Active or Mastered, and uses those states to target subsequent questions.
- Majority voting can reinforce shared solver errors, motivating safeguards based on vote-share thresholds or response confidence.
- One DiagEvo round combines challenger generation, double-confidence filtering, memory maintenance, hierarchical grouping, and state- and frequency-based scheduling.
3 Method
DiagEvo turns failed solver trajectories into a hierarchical, frequency-aware curriculum that combines targeted and exploratory question generation with filtered pseudo-label training. Diagnosis then updates the memory for subsequent rounds.
- Round structure: The challenger and solver update alternately: the challenger receives memory, while the solver trains on generated question–pseudo-label pairs and supplies failed trajectories.
- Question generation: Question generation combines targeting of Active causes with free exploration to address known weaknesses while maintaining broad coverage.
- Frequency-driven generation: Active episodes count recurring diagnosed failures, and promotion to Mastered resets a cause’s active frequency.
- Frequency-driven generation: More active-cause failures increase targeted generation, while zero failures produce only free exploration; parameter k controls the balance.
- Question generation: Cross-state stitching pairs an Active cause with a Mastered cause from the same skill node during targeted generation.
- Solver optimization: Double-confidence filtering retains intermediate-difficulty questions only when self-consistency is within bounds and the leading vote clearly exceeds the second.
- Solver optimization: Construction responses create pseudo-labels and select training pairs, whereas separate optimization responses provide policy-gradient signals and diagnostic failures.
- Memory maintenance: Memory maintenance extracts, deduplicates, assigns, and consolidates causes, with the diagnostician making semantic decisions sequentially.
4 Experiments and Analysis
DiagEvo is evaluated across three solvers, nine reasoning benchmarks, and component ablations. Its default configuration achieves the strongest overall results, while experiments identify contributions from diagnosis-guided generation, hierarchical memory, filtering, and round-wise dynamics.
- Main Results: DiagEvo with the default 4B diagnostician exceeds every baseline in overall score on all three solvers.The evaluation compares label-free and externally supervised methods across the benchmark suite.
- Main Results: 72.3% mathematical reasoning and 38.8% general reasoning on Qwen3-8B exceed R-Zero by 4.5 and 2.6 points, respectively.The corresponding margins over DARC are 1.2 and 1.0 points.
- Main Results: 53.5% and 41.9% overall scores on Qwen3-4B and OctoThinker-8B exceed DARC by 1.3 points.These are the reported overall scores for the other two solvers.
- Main Results: Mathematical averages rise by 1.0, 1.0, and 1.2 points when the diagnostician scales from 4B to 235B-A22B across the three solvers.On OlympiadBench, the gains are 2.4, 2.4, and 2.8 points; general-reasoning averages change by only 0.2 to 0.5 points.
- Cross-Domain Generalization: General-reasoning averages improve despite mathematical-only training, rising to 38.8%, 35.0%, and 28.4% for Qwen3-8B, Qwen3-4B, and OctoThinker-8B.The corresponding starting averages are 33.2%, 27.8%, and 10.7%.
- Ablation Studies: Freezing the challenger reduces the mathematical average by 3.8 points, while the full mixed generation policy outperforms both single-mode variants.The tested values of k change the mathematical average by at most 1.0 point.
- Ablation Studies: Removing memory-state updates or cross-state stitching lowers both mathematical and general averages, while same-skill-node stitching exceeds random pairing by 1.0 point mathematically.The state-update ablation combines promotion and reactivation effects.
- Ablation Studies: Adding the relative confidence constraint raises the mathematical average by a further 1.4 points after absolute confidence filtering.Both constraints improve performance over no filtering.
5 Conclusion
DiagEvo converts solver failure history into a curriculum through hierarchical error memory, mixed generation, and double-confidence filtering. Across three solvers, it achieves the highest overall scores among compared methods, with ablations and round analyses supporting the value of its components.
- Conclusion: DiagEvo diagnoses recurring solver error causes and organizes them in Active and Mastered hierarchical memory states.These states and recurrence frequencies guide cause-targeted generation and free exploration, while filtering removes high-conflict questions before training.
- Conclusion: DiagEvo with the default 4B diagnostician achieves the highest overall score across Qwen3-4B, Qwen3-8B, and OctoThinker-8B.Its training loop uses no external task resources.
- Conclusion: 72.3% on Qwen3-8B mathematical reasoning is 4.5 points above R-Zero, while mixed generation, memory updates, cross-state stitching, and filtering all contribute to performance.The round analysis shows five successive rounds of improvement.
A Limitations and Future Work
DiagEvo’s reported limitations concern pseudo-label verification, training-horizon selection, and evaluation scope. The authors identify self-play signals and broader domains as directions for future work.
- Pseudo-label confidence: Double-confidence filtering measures agreement among solver responses, not ground-truth correctness, so shared errors can still pass.Future work could use other verification signals produced during self-play to identify such cases.
- Training horizon: DiagEvo fixes the number of co-evolution rounds in advance, although the seven-round analysis peaks mathematically at round 5.Future work could use changes in Active causes and solver self-consistency to determine whether another round is useful.
- Evaluation scope: DiagEvo’s curriculum is built from mathematical questions, so the evaluation does not test direct curriculum construction in other domains.The authors propose extending the diagnosis and memory loop to open-ended tasks with long interaction sequences, including multi-turn tool use.
B Mixed Generation Distribution
The challenger mixes free exploration with cause-targeted generation, while alternating challenger updates and solver-training-set construction. Its reward favors questions near the solver’s competence boundary, and memory transitions use self-consistency and recurring failures.
- Mixed generation: The challenger samples questions from a mixed distribution combining free exploration with targeting of causes in hierarchical memory.The targeted component conditions generation on an error cause and the memory state.
- Round structure: Each round uses one batch to update the challenger and another candidate pool to construct and filter the solver training set.The current challenger generates the update batch; after GRPO, the updated challenger generates curriculum candidates.
- Quality reward: 50% self-consistency maximizes the uncertainty reward and represents the solver’s competence boundary.Here, self-consistency is the frozen solver’s majority-answer vote share.
- Quality reward: Agglomerative clustering applies larger repetition penalties to larger question clusters, while format-invalid questions receive zero reward.The complete reward uses the R-Zero setting λ = 1.
- State–frequency transitions: For an Active cause, promotion requires targeted candidates and mean self-consistency at least θup; later failures can reactivate it as Mastered-state transitions are committed.Promotion resets frequency, whereas reactivation starts a new episode weighted by newly matched failures.
E Training Hyperparameters
DiagEvo uses shared GRPO settings for challenger and solver, alongside co-evolution, filtering, curriculum, and memory-maintenance configurations. Its additional diagnosis and memory components add limited runtime overhead.
- GRPO Optimization: Both challenger and solver use R-Zero’s reported optimization hyperparameters, while role-specific batch and update settings are separate.Experiments run on one node with 8 NVIDIA H200 GPUs using BF16 mixed precision and FlashAttention 2.
- Co-evolution Settings: Each round optimizes the challenger for up to 6 GRPO steps with global batch size 256 and response-group size eight.The solver samples 12 construction responses per candidate to determine pseudo-labels and confidence-based filtering.
- Control Hyperparameters: Reward, double-confidence filtering, curriculum transitions, and memory maintenance are controlled by dedicated hyperparameters reported in Table 5.The diagnostician routing and retrieval procedure is specified separately in Appendix J.
- Diagnosis Configuration: The default diagnostician is Qwen3-4B-Instruct-2507, using only questions, pseudo-labels, and trajectories produced during self-play.The diagnostician and embedding model are frozen and general-purpose; text-embedding-v4 supports error-cause retrieval.
- Runtime: 94.2% of total runtime comes from question sampling, pseudo-label construction, and GRPO updates shared with R-Zero.Diagnosis and memory maintenance account for 5.7% of total time, indicating a small added overhead.
F Baseline Score Validation
The authors validate baseline comparability by reproducing base models and R-Zero under the same solvers, benchmarks, and evaluation protocol. Aggregate and per-benchmark deviations remain small.
- Aggregate Validation: Base-model averages differ by at most 0.1 points between reported and reproduced results.The comparison uses the same solvers, benchmarks, and evaluation protocol.
- Aggregate Validation: R-Zero scores differ by at most 0.5 points between the reported and reproduced evaluations.Per-benchmark deviations remain below 1.0 points.
- Per-Benchmark Validation: Per-benchmark deviations remain below 1.0 points, supporting comparability of the baseline scores.The reproduced evaluation was conducted under the authors’ infrastructure and protocol.
G Hyperparameter Sensitivity
Sensitivity experiments examine the exploration–exploitation parameter k and filtering threshold τ using Qwen3-8B-Base with the default 4B diagnostician. Performance is best near the reported τ setting and changes with the exploration schedule.
- Exploration–Exploitation Schedule: Smaller k values favor earlier exploitation, whereas larger values preserve more exploration in the frequency-driven schedule.Table 8 varies k under Qwen3-8B-Base.
- Filtering Threshold: Performance falls on either side of τ = 1.6 in double-confidence filtering.The τ = 1.6 row repeats the full configuration for reference.
- Filtering Threshold: Stricter filtering lowers performance consistently with removing too many training questions, although retention rates are not directly measured.The supplied sensitivity passage presents this as a consistency explanation rather than a directly measured mechanism.
H Diagnostic Evaluation Protocols
The evaluation measures per-round performance, memory growth, pseudo-label reliability, and generated-question properties, while PCA visualizes coverage by generation mode. Results indicate that memory-guided generation increases difficulty without requiring longer questions and combines complementary exploration modes.
- Per-round Performance: Per-round mathematical reasoning averages compare memory-guided and memory-free question generation while holding double-confidence filtering fixed.Rounds 6 and 7 are additionally evaluated under the same protocol.
- Oracle Agreement: Oracle agreement measures pseudo-label reliability by comparing solver majority-vote labels with GPT-5 majority-vote answers on retained training questions.Each GPT-5 oracle answer uses 16 sampled responses, and human inspection of 50 questions per round found no oracle errors.
- Question Properties: Without memory, average question length grows from 45.6 to 168.9 words, while 2-gram diversity rises only after questions become much longer.This timing links the late diversity increase to added length.
- Question Properties: With memory, average length stays near 73 words after round 2 while 2-gram diversity rises from 33.3 to 36.5.Both variants generate harder questions, but the memory-guided increase occurs at stable length.
- Coverage Visualization: PCA colors questions by free-exploration or cause-targeted generation across rounds 3–5.Free exploration covers a broad stable region, while cause-targeted generation moves into new regions as the memory gains error causes.
- Coverage Visualization: The two generation modes have complementary roles: free exploration maintains broad coverage, while cause-targeted generation follows current solver error causes.The passage connects this pattern with performance loss when either mode is removed.
I Case Study: From Failure to Curriculum
The case study follows a diagnosed reasoning failure from solver trajectories into hierarchical memory, targeted question generation, and eventual mastery. It also specifies how DiagEvo routes generation modes and maintains error causes during runtime.
- Failure and diagnosis: The diagnostician identifies a transferable failure: accepting algebraic candidates without checking their validity under the original domain restrictions.This cause is stored as Active under the skill node Checking Conditions after Algebraic Steps, alongside a related Mastered cause.
- Cause-targeted generation: Cross-state stitching combines an Active cause with a related Mastered cause to generate one composite question testing both reasoning elements.The mode is activated when the sampled Active cause shares a skill node with at least one Mastered cause.
- Cause-targeted generation: The generated question tests the same corrective action under a new surface form and enters the solver update only if it passes double-confidence filtering.The example requires propagating 0 < t < 1 and recognizing that equality at t = 1 is impossible.
- Behavior and state: 48.2% to 65.6% to 75.0%: rising self-consistency on targeted questions exceeds θup = 70%, promoting the diagnosed cause to Mastered.The later example shows the solver rejecting an invalid candidate after substituting solutions into the original equation.
- Runtime routing and maintenance: Free exploration uses no memory, while single-cause generation targets an Active cause whose skill node contains no Mastered causes.Runtime prompts preserve the question-generation role and format while inserting the relevant skill node and error cause; code handles retrieval and routing, while the diagnostician extracts and consolidates causes.