PaperlayerView exact revision on arXiv

Source-linked AI summary

When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain

Chenxu Wang, Chaozhuo Li, Xinze Shi, Songyang Liu, Kyrie You Wu, Ziluowen Luo, Shun Zhang, Chenxi Li, Litian Zhang

arXiv:2609.36535v1cs.CL

TL;DR

Self-evolution can improve LLMs using their own generated data, yet performance may plateau and decline while existing remedies often target isolated components. This paper introduces learnable information gain and ATRI, which reweights samples and halts training when gain stays low. The authors report that ATRI mitigates degeneration and improves self-evolution.

  • Problem

    Self-evolution can plateau and degrade, while existing remedies often address the Questioner or Solver separately despite their coupling.

    Source: §1 · pp. 1–2

  • Method

    The paper defines learnable information gain from new-round data's NLL under a previous-round fit and uses ATRI to reweight samples and halt after two consecutive low-gain rounds.

    Source: §1 · pp. 2–3, §3.1.2 · p. 4

  • Results

    The authors report that ATRI mitigates degeneration and improves self-evolution.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Tracking learnable information gain is presented as a way to observe saturation and forecast degradation before it occurs.

    Source: §1 · p. 2

    The stopping threshold is an empirical choice, with insensitivity established only for α ∈ [0.05, 0.30]; 0.1 is a trade-off rather than a derived optimum.

    Source: K Limitations and Broader Impact · p. 22

Abstract

from arXiv · show

Self-evolution lets large language models (LLMs) improve iteratively using their own generated data, but often suffers from self-evolution degeneration: performance improves, plateaus, then declines. Existing methods address this issue at the component level, targeting either the Questioner or the Solver, and overlook that self-evolution is a tightly coupled system. We propose a holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round. Theoretically, this gain equals the Kullback-Leibler divergence between the two rounds' data distributions plus their entropy change. Practically, it is estimated by fitting a small language model to the previous round and scoring new data via negative log-likelihood. Based on this diagnostic, we propose ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when information gain remains low. Experiments on popular datasets demonstrate the superiority of our proposal.

1 Introduction

Self-evolution can improve LLMs using their own generated data, but performance may plateau and decline, while existing responses focus on isolated components. This section introduces a system-level diagnostic based on learnable information gain and ATRI, which uses that signal to regulate training.

1 Introduction

  • Self-evolution degeneration describes solver performance that improves early, plateaus, and ultimately degrades across rounds.Proposed explanations include reward hacking, distributional or diversity collapse, and a closing generation–verification gap.Source: §1 · p. 1
  • Existing remedies often target either the Questioner or Solver, although changes at one stage may exacerbate degeneration elsewhere in the coupled system.For example, downstream reward-variance filtering may be limited if upstream answer generation has already degraded.Source: §1 · p. 2
  • The paper defines learnable information gain as novel round content that can be parameterized into the model and contribute to performance improvement.Under an exact fit, it decomposes into the KL divergence between consecutive rounds' data distributions plus their entropy change.Source: §1 · p. 2
  • ATRI estimates gain by scoring new-round data with a distribution fitted to the previous round, then reweights novel samples and halts after two consecutive low-gain rounds.The fitted distribution is implemented as a small language model retrained each round.Source: §1 · p. 2
  • The authors evaluate ATRI on several reasoning datasets and report that it forecasts degeneration before it occurs and alleviates degeneration.The introduction presents the framework as consolidating signals throughout the training pipeline rather than monitoring components separately.Source: Figure 1 · p. 2, §1 · p. 3

2 Preliminary Analysis

The analysis compares held-out accuracy with four partial training signals and learnable information gain during 15 rounds of R-Zero training on Qwen3-4B-Base with MATH. Accuracy declines after round 4, while the partial signals do not reliably indicate that decline and information gain reaches zero before the peak.

2 Preliminary Analysis

  • Accuracy peaks at round 4 and declines through round 15, while reward variance, policy entropy, distinct-n-gram ratio, and 1 − self-BLEU fluctuate without signaling degeneration.The experiment records held-out accuracy and these four signals across 15 rounds of R-Zero training on Qwen3-4B-Base with MATH.Source: §2 · p. 3
  • Ct reaches zero at round 3 and then fluctuates around zero, anticipating the accuracy peak and tracking the loss of new learnable information.The setup counts an answer as correct only when it exactly matches the ground-truth answer.Source: §2 · p. 3

3 Methodology

ATRI adds a shared learnable-information-gain module to the Questioner–Solver loop: it scores new data against a proxy of the previous round, reweights both updates, and helps decide when to stop.

3.1.1 General Training Reward for Questioner

  • The Questioner reward favors questions the Solver answers correctly about half the time, penalizes within-batch BLEU-similar questions, and zeroes questions that fail format checks.The combined reward is normalized into the GRPO advantage.Source: §3.1.1 · p. 4

3.1.2 Learnable Information Gain Estimation

  • A small proxy language model fits prior-round question–answer text, not the downstream task, and uses length-normalized NLL to mark text atypicality.The proxy learns recurring patterns in the previous round, making those patterns easier to predict.Source: §3.1.2 · pp. 4–5
  • A question’s information gain is its prediction difficulty relative to the previous-round mean; positive gain indicates difficulty above that baseline.Positive gain suggests the question may elicit content beyond what earlier questions covered.Source: §3.1.2 · p. 5

3.1.3 Training Objective Function

  • Questioner updates use positive information-gain weights to rescale each question’s GRPO advantage.Questions with negative gain receive no weight.Source: §3.1.3 · p. 5

3.2 Solver Training with the Learnable Information Gain

  • After the Questioner update, the Solver trains on fresh answers to retained questions, with positive answer-side information gain rescaling its GRPO advantage.Questions too easy or too hard for the current Solver are filtered out.Source: §3.2 · pp. 5–6

3.3 Iterative Co-Evolution Paradigm with Early Warning

  • Round-level Ct compares average proxy costs of complete question–answer pairs across rounds; near-zero gain indicates little new content, and under exact fit equals KL divergence plus entropy change.The proxy scores each complete pair as text.Source: §3.3 · p. 6
  • ATRI stops when Ct stays below τ = 0.1 C1 for two rounds; in vanilla runs, Ct reached zero at round 3, accuracy peaked at 4, and decline began at 9.The paper describes continued training after Ct reaches zero as fitting samples the model can already generate, while accuracy degrades.Source: §3.3 · p. 6

4 Experiments

ATRI performed best across both tested base models and mathematical and general reasoning benchmarks. Across existing methods, learnable-information-gain monitoring anticipated accuracy peaks, and experiments examined sample selection, component contributions, robustness, cost, and stopping.

4.1 Experimental Setup

  • The experiments used seven math and three general-reasoning benchmarks, comparing ATRI with six self-evolution baselines on the same base models and evaluation suite.Qwen3-4B-Base and Qwen3-8B-Base were used; reported performance was averaged over five training runs.Source: §4.1 · p. 7

4.2 Main comparison with existing self-evolution methods

  • ATRI achieved the best aggregate performance on both base models, across mathematical and general reasoning benchmarks.The authors attribute the gain to within-round informative-sample weighting and cross-round stopping before degradation.Source: §4.2 · p. 7

4.3 The three-phase lifecycle recurs across existing methods

  • Across six baselines, Ct fell below its alarm threshold one to four rounds before the accuracy peak, with longer leads for methods with longer pre-peak plateaus.The trajectories followed a rise, plateau, and decline; Ct reflects redundancy between consecutive rounds and changes before accuracy decline becomes visible.Source: §4.3 · pp. 7–8, Figure 5 · p. 8, Figure 4 · p. 8
  • 19.8 points (pass@8), 36.5% (path entropy), and 46.6% (distinct correct paths) fell after the accuracy peak, consistent with Phase III solution-distribution contraction.The measurements used eight sampled solutions per problem on the same 500 held-out problems at each checkpoint.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Direction-aware selection of external samples

  • Cd-only selection beat four alternatives at every budget; adding ct led from 1,500 samples, with the widest margin at 3,000.Cd targets samples resembling model failures, while ct removes samples the model has already absorbed; samples are added when both gains are positive.Source: §4.4 · p. 9

4.6 Robustness and computational cost

  • 4.1% to 1.3% was the measured Ct overhead across three scales; ATRI was robust within sensible threshold and proxy-size ranges, with only the most aggressive setting losing accuracy.Stopping too early costs more than stopping slightly late; the default proxy size sits at the elbow between underfitting and a 2.5-fold proxy cost.Source: §4.6 · p. 9, Table 5 · p. 9

4.5 Contribution of each component

  • 3.37 Math AVG separated full ATRI from removing both controls; stopping contributed more, weighting and filtering added 1.06 Math AVG, and the external phase added 2.12.Weighting and filtering contributed under an identical stopping round.Source: Table 4 · p. 9, §4.5 · p. 9

4.7 Case study: applying the diagnostic to existing methods

  • 39.01 to 46.32 was the average-accuracy change with passive Ct stopping, placing the result within 0.83 of the oracle test-best checkpoint without held-out validation data.The stopping rule fires after Ct stays below the threshold for two consecutive rounds.Source: §4.7 · p. 10

5 Conclusion

The paper frames self-evolution as an information-bounded process and uses learnable information gain to characterize round-level learning. It concludes that ATRI can regulate sample weighting and stopping, while appendices specify the diagnostic, proxy choices, and extensions to other objectives.

5 Conclusion

  • The paper proposes learnable information gain as a system-level diagnostic and ATRI to reweight samples within rounds and stop evolution when gain remains low.Under an exact fit, the gain decomposes into the KL divergence between rounds plus their entropy change; experiments report mitigated degeneration and improved self-evolution.Source: §5 · p. 10

A Related Work

  • Recent work identifies several contributors to collapse: reward-hacking problem generators, task gates that affect stability more than reward design, and rise-then-collapse dynamics; each study examines one component or setting.Repeated self-distillation progressively limits solution expressiveness; the paper relates Phase III to this result.Source: A Related Work · p. 13

B Information-theoretic Boundary of Self-Evolution

  • Under the stated assumption that generation and training use no signal about the target capability beyond current parameters, the loop cannot increase information about that capability.The paper calls this constraint Generative Containment; external data or ground-truth verification can break the assumption and add information.Source: B Information-theoretic Boundary of… · pp. 13–14

C Within-round Decomposition of Ct

  • Ct decomposes into length-weighted question novelty and solution novelty conditioned on the question, using per-token NLL averages.The question weight is λe = |q|/(|q| + |s|); the decomposition is exact because per-token averages are not additive.Source: C Within-round Decomposition of Ct · p. 14

D Proxy Model Capacity Selection

  • 52.73 (70M), 53.55 (160M), and 53.62 (410M) were the Math AVG results in the proxy sweep; 160M sits at the stated elbow between underfitting and higher compute.The 410M proxy costs about 2.5 times as much as 160M, while proxies above 1B begin to compete with main-model fine-tuning.Source: D Proxy Model Capacity Selection · pp. 14–15
  • The proxy restarts from pretrained Pythia each round and never sees main-model parameters, so Ct depends on consecutive datasets and proxy capacity rather than the main model or training setup.Pythia's fixed corpus, recipe, and tokenizer allow controlled capacity variation and comparable encoding costs across sizes.Source: D Proxy Model Capacity Selection · p. 15

E Extensions of the Calibration Loss

E.3 DPO

  • In DPO, each preference pair is weighted by the gain of its preferred solution; the negative branch needs no separate weight because it supplies contrast.The general calibrated form multiplies the base objective weight by the positive part of the gain, where the base weight depends on the objective.Source: §E.3 · pp. 15–16, §E.4 · p. 16

F Implementation Details

The implementation uses GRPO for the main models, a small proxy fitted once per round, and an alarm threshold based on first-round gain. Evaluation and training protocols are specified for reproducible baseline comparisons.

Main model fine-tuning

  • The Questioner and Solver use GRPO with AdamW, one epoch per round, batch size 128, and sequence length capped at 2,048 tokens.The optimizer uses learning rate 1 × 10^-5, β1=0.9, β2=0.95, weight decay 0.1, 50-step warmup, and bfloat16.Source: F Implementation Details · p. 16

Generation and verification

  • Generation uses temperature 0.7 and top-p 0.95, with eight Solver answers per question; verifier-positive answers form the training data.Answers are normalized before matching, and the ATRI loop uses majority-vote pseudo-labels without a learned reward model.Source: F Implementation Details · p. 16

Proxy fine-tuning and stopping

  • The Pythia-160M proxy is freshly initialized and fine-tuned for one epoch per round; the closed loop stops when Ct is below 0.1 C1 for two consecutive rounds.Proxy training adds 1.4 PFLOPs per round on Qwen3-8B-Base configurations.Source: F Implementation Details · p. 16

Baselines and evaluation

  • Baselines are rerun on matched base models, generation budget, verifier, and evaluation protocol; reported accuracies use greedy decoding and means over five runs.Math AVG and Overall AVG are unweighted means over seven and ten benchmarks respectively; Math AVG standard deviations are within ±0.4.Source: F Implementation Details · p. 16

G Generality Across Model, Task, and Proxy Families

Lifecycle and alarm behavior were tested beyond the main Qwen mathematical-reasoning setting, and proxy-family substitution was evaluated on an unchanged trajectory. The reported alarms remained close to the performance peaks across these settings.

Generality across model and task

Proxy-family generality

  • With OPT-125M replacing Pythia-160M on the same Qwen3-4B/MATH trajectory, the alarm fired at the same round and proxy sample weights had Spearman correlation 0.913.Full ATRI scores changed by at most 0.14 points.Source: G Generality Across Model, Task, and… · p. 17

H Robustness of the Stopping Decision

The stopping rule remains stable under reduced per-round data, surface-level controls, task shifts, and token-window training. Proxy refitting is needed after a task switch to keep the alarm aligned with the performance peak.

H Robustness of the Stopping Decision

  • Four stopping rules are compared on six 15-round baseline runs: final-round, validation-based, oracle, and passive Ct stopping.The supplied text describes the comparison and checkpoint-selection procedures but not the accuracy values reported in Table 7.Source: H Robustness of the Stopping Decision · p. 17, Table 7 · p. 17
  • 77.6% of trials using 25% of the original per-round data reproduce the full-data stopping round, and every trial stays within two rounds.The estimate comes from 1,000 bootstrap trials at each of four data sizes.Source: H Robustness of the Stopping Decision · p. 17, Table 8 · p. 18
  • 93.1% of combined length-matching and format-standardisation trials select the original stopping round, with the Ct trajectory correlation at 0.958.Every tested control preserves the original alarm and stopping rounds; the remaining combined-control trials differ by one round.Source: H Robustness of the Stopping Decision · p. 17, Table 9 · p. 18
  • After a MATH-to-MBPP switch, the old-task proxy alarms two rounds after the performance peak, while refitting on the first MBPP round alarms one round before it.The refitted timing matches the single-task runs.Source: H Robustness of the Stopping Decision · p. 18, Table 10 · p. 18
  • Token windows of half or one average-round size stop at the same checkpoint as the round-based rule, while a two-round window stops one checkpoint later.The authors suggest a window on the order of one round of data as a reasonable default.Source: H Robustness of the Stopping Decision · p. 18

I Proofs

The appendix sets notation for the proof and establishes a boundary condition: identical empirical distributions across adjacent rounds give zero gain.

I Proofs

  • The proofs use Dt for the round-t training set and Pt−1 for the reference distribution fitted on the previous round.The encoding cost is token-normalized NLL, and the main text writes ℓ(x) and ¯ℓ(D) for costs under the proxy Mt−1.Source: I Proofs · p. 18

I.1 Boundary condition

  • Ct = 0 when Dt and Dt−1 have the same empirical distribution.The two dataset averages in the definition of Ct then coincide.Source: §I.1 · p. 18

I.2 Information-theoretic identity

The information-theoretic identity separates gain into distribution shift and entropy change, with proxy-fitting residuals affecting the estimate away from an exact fit. Under stated contraction conditions, gain decreases across a closed loop and tends to zero as successive rounds converge.

I.2 Information-theoretic identity

  • With an exact-fit reference, Ct equals KL(pt ∥ pt−1) plus the entropy change between rounds.For a general reference, the identity includes proxy residual terms; baseline subtraction cancels the reference’s fitting error at the zero point.Source: §I.2 · pp. 18–19
  • Away from zero, the gap between the general identity and the exact-fit form depends on the difference in proxy residuals across rounds.The gap is small when residuals are low and stable over the region shared by both rounds.Source: §I.2 · p. 19
  • Even with an exact fit, entropy contraction can make Ct negative; length shifts, residual differences, and finite samples can also push empirical Ct slightly below zero.When round length distributions match, the KL term is nonnegative, but the entropy change need not be.Source: §I.2 · p. 19

I.3 Why the gain decreases in a closed loop

  • If successive-round KL divergence and entropy change are each nonincreasing, Ct is nonincreasing; if both tend to zero, Ct tends to zero.These conditions describe a contracting loop under the exact-fit assumption.Source: §I.3 · p. 19
  • A closed loop can satisfy the contraction conditions as training repeatedly uses its own outputs, whereas external data can enlarge the shift KL and raise gain again.The proposition assumes an exactly fitted reference; with an underfit proxy, the comparison holds up to residual differences.Source: §I.3 · p. 19

I.4 Sample average equals the dataset gain

The proofs justify using Ct as a round-level summary and decompose it into question- and answer-level terms. They also connect low-gain training to self-distillation under stated conditions, with updates shifting probability toward high-probability outputs.

I.4 Sample average equals the dataset gain

  • Ct is the round-level summary, while ct(e) provides its sample-level decomposition.The sample-average identity justifies aggregating the sample terms into the dataset gain.Source: §I.4 · p. 20

I.5 Question–Solver decomposition

  • For samples generated by a two-step Question–Solver process, the gain decomposes into question and answer terms.The decomposition uses length-weighted dataset averages for the question and answer components.Source: §I.5 · p. 20

I.6 Connection to self-distillation in Phase III

  • When Ct ≤ 0 and the model fits the previous-round data well, SFT on current-round data is approximately a self-distillation step.The main experiments use GRPO; for positive-advantage samples, its weighted log-likelihood terms support applying the argument with sample weights.Source: §I.6 · pp. 20–21
  • In this regime, repeated self-referential updates shrink the effective hypothesis space and suppress correct but low-probability solution paths.Sampling and verifier filtering restrict training to high-probability outputs, so the update shifts probability mass toward that region.Source: §I.6 · p. 21

J Directional information gain and external-phase training

The external-phase extension continues training after closed-loop information gain saturates, using a directional criterion alongside novelty to select samples that target model weaknesses.

J Directional information gain and external-phase training

  • The external phase starts after C_t falls below threshold τ for two consecutive rounds, then draws training samples from the external pool instead of D_t.It terminates when C_t and directional gain remain below threshold simultaneously over an extended period.Source: J Directional information gain and… · p. 21
  • Directional gain complements novelty because a sample outside D_{t−1} may reinforce capability the model already handles rather than target its weakness.The directional score identifies whether an external sample resembles examples the model gets wrong.Source: J Directional information gain and… · p. 21
  • Only external samples that are both novel relative to D_{t−1} and more similar to the model’s incorrect behavior pass the dual gate.A positive novelty score alone can admit samples the model already handles; positive directional gain indicates similarity to errors.Source: J Directional information gain and… · p. 21
  • The dual gate also applies to tool-feedback and human-feedback samples because both criteria depend only on sample text.Source: J Directional information gain and… · p. 22

K Limitations and Broader Impact

The discussion bounds ATRI by assumptions about round structure, proxy fit, external data, and verifier reliability, while describing compute and evaluation implications of information-gain control.

K Limitations and Broader Impact

Related papers

Loading 2609.36535v1…