PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory

Xincheng Wei, Yifan Ding, Yoshua Li, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Wenjian Ding, Yao Zhang

arXiv:2609.00768v1cs.AI

TL;DR

Self-play needs guidance that targets unresolved reasoning weaknesses without relying on external task resources. DiagEvo derives that guidance from recurring solver failures through hierarchical error memory, mixed generation, and double-confidence filtering, and it outperforms the compared baselines across the evaluated solvers and benchmarks.

  • Problem

    Self-play guidance based on difficulty, learnability, or diversity does not specify which recurring reasoning weaknesses later rounds should target, while guided methods use external task resources.

    Source: §1 · pp. 1–2

  • Method

    DiagEvo diagnoses recurring error causes from failed solver trajectories, stores them as Active or Mastered in hierarchical memory, mixes targeted generation with exploration, and filters conflicting pseudo-labels.

    Source: §1 · pp. 2–3, §3.2 · p. 5

  • Results

    DiagEvo achieves the highest overall score among compared methods across Qwen3-4B, Qwen3-8B, and OctoThinker-8B; on Qwen3-8B, its mathematical reasoning average is 72.3%, 4.5 points above R-Zero.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Solver failure history can serve as an internal curriculum signal for subsequent self-play rounds without external task resources.

    Source: §5 · p. 10

    Confidence filtering measures agreement among solver responses rather than ground-truth correctness, so a shared error can still pass the filter.

    Source: §3.3 · p. 6, A Limitations and Future Work · p. 12

Abstract

from arXiv · show

Self-play is an effective paradigm for language-model self-evolution, but without guidance, solver performance can plateau or decline across rounds. Unguided methods steer question generation with signals such as difficulty, learnability, or diversity. These signals keep questions challenging and varied but do not specify which unresolved reasoning weaknesses later rounds should target. Guided methods obtain direction from external task resources, including human examples, document corpora, or specified difficulty targets, and therefore rely on task information supplied outside the self-play loop. We show that the needed direction can instead be derived from the solver's own failure history. We introduce DiagEvo, whose diagnostician extracts recurring error causes from this history and stores them in a hierarchical error-cause memory. The memory groups related causes under skill nodes and tracks each as Active or Mastered according to self-consistency on targeted questions. The challenger uses these states and recurrence counts to balance cause-targeted generation with free exploration. Double-confidence filtering retains intermediate-difficulty questions only when the most common solver answer has a clear vote lead. DiagEvo derives its curriculum from information produced during self-play, without external task resources. With the default 4B diagnostician, DiagEvo outperforms every baseline in mean accuracy across all nine benchmarks for each of the three solvers: Qwen3-4B, Qwen3-8B, and OctoThinker-8B. On Qwen3-8B, it reaches 72.3% mean accuracy across five mathematical reasoning benchmarks, 4.5 percentage points above R-Zero. Its mean accuracy across all nine benchmarks is 57.4%, 1.1 percentage points above DARC. Ablations show that the hierarchical error-cause memory and double-confidence filtering both contribute to these gains.

1 Introduction

DiagEvo addresses the difficulty of sustaining learnable, targeted self-play by deriving curriculum guidance from recurring solver failures rather than external task resources. It combines hierarchical error memory, mixed generation, and confidence filtering to improve performance across the evaluated solvers.

  • Self-play must keep questions near the solver’s competence boundary: difficult enough to expose weaknesses but learnable enough to provide stable training signals.Source: §1 · pp. 1–2
  • Label-free difficulty feedback estimates how hard questions are but does not identify recurring reasoning causes, allowing surface complexity to grow without targeted practice.Source: §1 · p. 2, Figure 1 · p. 2
  • Guided self-play uses human examples, document corpora, difficulty labels, or privileged teachers, so continued improvement depends on information outside self-play.Source: §1 · p. 2
  • DiagEvo compares failed and agreeing solver trajectories to identify transferable error causes and tracks their recurrence for future curriculum generation.Source: §1 · pp. 2–3
  • DiagEvo organizes causes into Active and Mastered states, mixes cause-targeted generation with free exploration, and filters high-conflict pseudo-labels before solver training.Source: §1 · pp. 2–3
  • On Qwen3-8B, DiagEvo reaches 72.3% average mathematical reasoning accuracy, 4.5 points above R-Zero, while using no external task resources.Source: §1 · p. 3

2 Related Work

Prior work uses self-play, interaction memory, curriculum learning, and pseudo-label safeguards to improve training, but DiagEvo applies structured failure memory directly to question generation within self-play.

  • Label-free self-play updates challenger and solver models without human annotations, but questions may become too easy or difficult and pseudo-labels less reliable.Source: §2.1 · p. 3
  • Language-agent methods store successful and failed interactions, summaries, or skills to guide task-solving decisions or policy updates.Source: §2.2 · p. 3
  • Curriculum methods control example exposure, filter learnable questions, or create answer-preserving variants for reinforcement learning.Source: §2.2 · p. 3
  • DiagEvo builds structured memory from solver failures, tracks causes as Active or Mastered, and uses those states to target subsequent questions.Source: §2.2 · p. 3
  • Majority voting can reinforce shared solver errors, motivating safeguards based on vote-share thresholds or response confidence.Source: §2.3 · p. 3
  • One DiagEvo round combines challenger generation, double-confidence filtering, memory maintenance, hierarchical grouping, and state- and frequency-based scheduling.Source: Figure 2 · p. 4

3 Method

DiagEvo turns failed solver trajectories into a hierarchical, frequency-aware curriculum that combines targeted and exploratory question generation with filtered pseudo-label training. Diagnosis then updates the memory for subsequent rounds.

Round structure

  • The challenger and solver update alternately: the challenger receives memory, while the solver trains on generated question–pseudo-label pairs and supplies failed trajectories.Source: §3 · p. 4

Question generation

  • Question generation combines targeting of Active causes with free exploration to address known weaknesses while maintaining broad coverage.Source: §3.1 · p. 4

Frequency-driven generation

  • Active episodes count recurring diagnosed failures, and promotion to Mastered resets a cause’s active frequency.Source: §3.1 · pp. 4–5
  • More active-cause failures increase targeted generation, while zero failures produce only free exploration; parameter k controls the balance.Source: §3.1 · p. 5

Question generation

  • Cross-state stitching pairs an Active cause with a Mastered cause from the same skill node during targeted generation.Source: §3.1 · p. 5

Solver optimization

  • Double-confidence filtering retains intermediate-difficulty questions only when self-consistency is within bounds and the leading vote clearly exceeds the second.Source: §3.2 · pp. 5–6
  • Construction responses create pseudo-labels and select training pairs, whereas separate optimization responses provide policy-gradient signals and diagnostic failures.Source: §3.2 · pp. 5–6

Memory maintenance

  • Memory maintenance extracts, deduplicates, assigns, and consolidates causes, with the diagnostician making semantic decisions sequentially.Source: §3.3 · p. 6

4 Experiments and Analysis

DiagEvo is evaluated across three solvers, nine reasoning benchmarks, and component ablations. Its default configuration achieves the strongest overall results, while experiments identify contributions from diagnosis-guided generation, hierarchical memory, filtering, and round-wise dynamics.

Main Results

  • DiagEvo with the default 4B diagnostician exceeds every baseline in overall score on all three solvers.The evaluation compares label-free and externally supervised methods across the benchmark suite.Source: §4.2 · p. 7, §4.1 · p. 7
  • 72.3% mathematical reasoning and 38.8% general reasoning on Qwen3-8B exceed R-Zero by 4.5 and 2.6 points, respectively.The corresponding margins over DARC are 1.2 and 1.0 points.Source: §4.2 · p. 7
  • 53.5% and 41.9% overall scores on Qwen3-4B and OctoThinker-8B exceed DARC by 1.3 points.These are the reported overall scores for the other two solvers.Source: §4.2 · p. 7
  • Mathematical averages rise by 1.0, 1.0, and 1.2 points when the diagnostician scales from 4B to 235B-A22B across the three solvers.On OlympiadBench, the gains are 2.4, 2.4, and 2.8 points; general-reasoning averages change by only 0.2 to 0.5 points.Source: §4.2 · p. 7

Cross-Domain Generalization

  • General-reasoning averages improve despite mathematical-only training, rising to 38.8%, 35.0%, and 28.4% for Qwen3-8B, Qwen3-4B, and OctoThinker-8B.The corresponding starting averages are 33.2%, 27.8%, and 10.7%.Source: §4.2 · p. 8

Ablation Studies

  • Freezing the challenger reduces the mathematical average by 3.8 points, while the full mixed generation policy outperforms both single-mode variants.The tested values of k change the mathematical average by at most 1.0 point.Source: §4.3 · p. 8
  • Removing memory-state updates or cross-state stitching lowers both mathematical and general averages, while same-skill-node stitching exceeds random pairing by 1.0 point mathematically.The state-update ablation combines promotion and reactivation effects.Source: §4.3 · p. 8
  • Adding the relative confidence constraint raises the mathematical average by a further 1.4 points after absolute confidence filtering.Both constraints improve performance over no filtering.Source: §4.3 · p. 8

5 Conclusion

DiagEvo converts solver failure history into a curriculum through hierarchical error memory, mixed generation, and double-confidence filtering. Across three solvers, it achieves the highest overall scores among compared methods, with ablations and round analyses supporting the value of its components.

Conclusion

  • DiagEvo diagnoses recurring solver error causes and organizes them in Active and Mastered hierarchical memory states.These states and recurrence frequencies guide cause-targeted generation and free exploration, while filtering removes high-conflict questions before training.Source: §5 · p. 10
  • DiagEvo with the default 4B diagnostician achieves the highest overall score across Qwen3-4B, Qwen3-8B, and OctoThinker-8B.Its training loop uses no external task resources.Source: §5 · p. 10
  • 72.3% on Qwen3-8B mathematical reasoning is 4.5 points above R-Zero, while mixed generation, memory updates, cross-state stitching, and filtering all contribute to performance.The round analysis shows five successive rounds of improvement.Source: §5 · p. 10

A Limitations and Future Work

DiagEvo’s reported limitations concern pseudo-label verification, training-horizon selection, and evaluation scope. The authors identify self-play signals and broader domains as directions for future work.

Pseudo-label confidence

  • Double-confidence filtering measures agreement among solver responses, not ground-truth correctness, so shared errors can still pass.Future work could use other verification signals produced during self-play to identify such cases.Source: A Limitations and Future Work · p. 12

Training horizon

  • DiagEvo fixes the number of co-evolution rounds in advance, although the seven-round analysis peaks mathematically at round 5.Future work could use changes in Active causes and solver self-consistency to determine whether another round is useful.Source: A Limitations and Future Work · pp. 12–13

Evaluation scope

  • DiagEvo’s curriculum is built from mathematical questions, so the evaluation does not test direct curriculum construction in other domains.The authors propose extending the diagnosis and memory loop to open-ended tasks with long interaction sequences, including multi-turn tool use.Source: A Limitations and Future Work · p. 13

B Mixed Generation Distribution

The challenger mixes free exploration with cause-targeted generation, while alternating challenger updates and solver-training-set construction. Its reward favors questions near the solver’s competence boundary, and memory transitions use self-consistency and recurring failures.

Mixed generation

Round structure

  • Each round uses one batch to update the challenger and another candidate pool to construct and filter the solver training set.The current challenger generates the update batch; after GRPO, the updated challenger generates curriculum candidates.Source: C Challenger Update and Quality Reward · p. 13

Quality reward

  • 50% self-consistency maximizes the uncertainty reward and represents the solver’s competence boundary.Here, self-consistency is the frozen solver’s majority-answer vote share.Source: C Challenger Update and Quality Reward · p. 13
  • Agglomerative clustering applies larger repetition penalties to larger question clusters, while format-invalid questions receive zero reward.The complete reward uses the R-Zero setting λ = 1.Source: C Challenger Update and Quality Reward · p. 13

State–frequency transitions

  • For an Active cause, promotion requires targeted candidates and mean self-consistency at least θup; later failures can reactivate it as Mastered-state transitions are committed.Promotion resets frequency, whereas reactivation starts a new episode weighted by newly matched failures.Source: D State–Frequency Transition Rule · pp. 13–14

E Training Hyperparameters

DiagEvo uses shared GRPO settings for challenger and solver, alongside co-evolution, filtering, curriculum, and memory-maintenance configurations. Its additional diagnosis and memory components add limited runtime overhead.

GRPO Optimization

  • Both challenger and solver use R-Zero’s reported optimization hyperparameters, while role-specific batch and update settings are separate.Experiments run on one node with 8 NVIDIA H200 GPUs using BF16 mixed precision and FlashAttention 2.Source: E Training Hyperparameters · p. 14, Table 3 · p. 14

Co-evolution Settings

  • Each round optimizes the challenger for up to 6 GRPO steps with global batch size 256 and response-group size eight.The solver samples 12 construction responses per candidate to determine pseudo-labels and confidence-based filtering.Source: E Training Hyperparameters · p. 14, Table 4 · p. 15

Control Hyperparameters

  • Reward, double-confidence filtering, curriculum transitions, and memory maintenance are controlled by dedicated hyperparameters reported in Table 5.The diagnostician routing and retrieval procedure is specified separately in Appendix J.Source: E Training Hyperparameters · p. 15, Table 5 · p. 15

Diagnosis Configuration

  • The default diagnostician is Qwen3-4B-Instruct-2507, using only questions, pseudo-labels, and trajectories produced during self-play.The diagnostician and embedding model are frozen and general-purpose; text-embedding-v4 supports error-cause retrieval.Source: E Training Hyperparameters · p. 15

Runtime

  • 94.2% of total runtime comes from question sampling, pseudo-label construction, and GRPO updates shared with R-Zero.Diagnosis and memory maintenance account for 5.7% of total time, indicating a small added overhead.Source: E Training Hyperparameters · p. 15, Table 6 · p. 16

F Baseline Score Validation

The authors validate baseline comparability by reproducing base models and R-Zero under the same solvers, benchmarks, and evaluation protocol. Aggregate and per-benchmark deviations remain small.

Aggregate Validation

Per-Benchmark Validation

G Hyperparameter Sensitivity

Sensitivity experiments examine the exploration–exploitation parameter k and filtering threshold τ using Qwen3-8B-Base with the default 4B diagnostician. Performance is best near the reported τ setting and changes with the exploration schedule.

Exploration–Exploitation Schedule

Filtering Threshold

  • Performance falls on either side of τ = 1.6 in double-confidence filtering.The τ = 1.6 row repeats the full configuration for reference.Source: G Hyperparameter Sensitivity · p. 17, Table 9 · p. 17
  • Stricter filtering lowers performance consistently with removing too many training questions, although retention rates are not directly measured.The supplied sensitivity passage presents this as a consistency explanation rather than a directly measured mechanism.Source: G Hyperparameter Sensitivity · p. 17

H Diagnostic Evaluation Protocols

The evaluation measures per-round performance, memory growth, pseudo-label reliability, and generated-question properties, while PCA visualizes coverage by generation mode. Results indicate that memory-guided generation increases difficulty without requiring longer questions and combines complementary exploration modes.

Per-round Performance

  • Per-round mathematical reasoning averages compare memory-guided and memory-free question generation while holding double-confidence filtering fixed.Rounds 6 and 7 are additionally evaluated under the same protocol.Source: H Diagnostic Evaluation Protocols · p. 17

Oracle Agreement

  • Oracle agreement measures pseudo-label reliability by comparing solver majority-vote labels with GPT-5 majority-vote answers on retained training questions.Each GPT-5 oracle answer uses 16 sampled responses, and human inspection of 50 questions per round found no oracle errors.Source: H Diagnostic Evaluation Protocols · p. 17

Question Properties

Coverage Visualization

  • PCA colors questions by free-exploration or cause-targeted generation across rounds 3–5.Free exploration covers a broad stable region, while cause-targeted generation moves into new regions as the memory gains error causes.Source: Figure 5 · p. 18, H Diagnostic Evaluation Protocols · p. 18
  • The two generation modes have complementary roles: free exploration maintains broad coverage, while cause-targeted generation follows current solver error causes.The passage connects this pattern with performance loss when either mode is removed.Source: H Diagnostic Evaluation Protocols · p. 18

I Case Study: From Failure to Curriculum

The case study follows a diagnosed reasoning failure from solver trajectories into hierarchical memory, targeted question generation, and eventual mastery. It also specifies how DiagEvo routes generation modes and maintains error causes during runtime.

Failure and diagnosis

  • The diagnostician identifies a transferable failure: accepting algebraic candidates without checking their validity under the original domain restrictions.This cause is stored as Active under the skill node Checking Conditions after Algebraic Steps, alongside a related Mastered cause.Source: I Case Study: From Failure to Curriculum · p. 18

Cause-targeted generation

Behavior and state

  • 48.2% to 65.6% to 75.0%: rising self-consistency on targeted questions exceeds θup = 70%, promoting the diagnosed cause to Mastered.The later example shows the solver rejecting an invalid candidate after substituting solutions into the original equation.Source: I Case Study: From Failure to Curriculum · p. 19, Figure 6 · p. 19

Runtime routing and maintenance

  • Free exploration uses no memory, while single-cause generation targets an Active cause whose skill node contains no Mastered causes.Runtime prompts preserve the question-generation role and format while inserting the relevant skill node and error cause; code handles retrieval and routing, while the diagnostician extracts and consolidates causes.Source: J Prompt Templates and Memory Routing · p. 20

Related papers

Loading 2609.00768v1…