Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

1 Introduction

• With certain verification methods, relative GV-Gap increases monotonically with pre-training flops.Figure 1 depicts this pattern; the authors conjecture a linear relationship with the logarithm of pre-training flops.Source: Figure 1 · p. 2, §1 · p. 2
• The study addresses uncertainty from process confounders and prior work's limited coverage of model families and verification methods.The introduction also notes that synthetic-data self-training can degrade performance, while reliable verification has shown promise in preventing collapse.Source: §1 · pp. 1–2
• GV-Gap measures the performance gain from reweighting generations by self-verification scores and is reported to measure self-improvement more accurately than post-update performance differences.Source: §1 · p. 2
• Cross-verification gaps rise with verifier capability and fall with generator capability, while many models fail on tasks beyond their reasoning or planning capabilities.Source: §1 · p. 2
• Iterative self-improvement drives GV-Gap to saturation at zero within a handful of rounds, independent of model capacity, as effective diversity degrades.Source: §1 · p. 2
• GV-Gap need not correlate positively with generation accuracy, and an ensemble of verification methods can enhance self-improvement.Source: §1 · p. 3

2 A Dissection of the Self-improvement Framework

• The framework generates candidate responses, scores them with a model-based proxy utility, then updates the model using the self-verified responses.The proxy utility can be derived from a verifier model’s prompted quality score.Source: §2 · p. 3
• The update projects a verifier-score-reweighted generation distribution back into the model class, with thresholded rejection sampling as one example.Rejection sampling uses indicator weights above a threshold; top-n filtering sets that threshold at a score quantile.Source: §2 · pp. 3–4

2.1 Three Key Factors of Self-improvement

• Self-improvement requires the updated model to improve expected utility, with generation variability and informative verification among the conditions that can bottleneck progress.Greedy decoding leaves no generation variation to exploit, while limited verification may provide no useful weighting signal.Source: §2.1 · p. 4
• The generation-verification gap measures the improvement available from reweighting generations, and a positive correlation between verifier weights and utility guarantees a positive gap.The gap weights each prompt’s improvement by its deficiency from the best possible utility.Source: §2.1 · pp. 4–5
• The relative generation-verification gap addresses the absolute gap’s limited headroom when a generator already performs near the task maximum.At 99% accuracy, the absolute gap can be at most 0.01, and remaining errors may be subtle.Source: §2.1 · p. 5
• A high-fidelity update should closely mimic the reweighted distribution, and the paper notes that distillation error is often small for sufficiently expressive LLM classes.The framework bounds the update’s utility error by ε_update; a total-variation bound ε′ gives ε_update ≤ ε′Umax.Source: §2.1 · p. 5
• Finetuning can appear to improve performance through output-format alignment or output-length changes, so gains need not reflect underlying capability.The experiments observed the format-alignment scenario; the authors also discuss reasoning improvements as conceivable, not observed.Source: §2.1 · p. 5

3 Experiment Setup

• Experiments sample 128 responses per prompt and one verification per response, then compare accuracy after filtering with accuracy on the original generations.The main setup uses p = 0.9, t = 0.7, maximum length 512, four-shot prompts, and rejection sampling with quantile or global thresholds.Source: §3 · pp. 5–6
• Only one verification is sampled per generation for computational reasons, although the authors identify multiple verifications and verification-compute scaling as future work.Ablations over reasonable sampling temperatures reportedly preserve the same observations.Source: §3 · p. 5
• Verification mechanisms include multiple-choice labels, chain-of-thought scores, and pairwise tournament comparisons.Chain-of-thought scores can be 1–10 or binary; tournaments repeatedly compare generation pairs in single elimination.Source: §3 · p. 6
• The experiments cover Qwen-1.5, Qwen-2, Qwen-2.5, Llama-2, Llama-3, Llama-3.1, and Yi-1.5, using base models to avoid post-training confounds.Inference is performed with vLLM.Source: §3 · p. 6

4.1 Scaling Results

• Qwen-1.5 0.5B, Qwen-2 0.5B, and Llama-2 7B have non-positive gaps for nearly all verification methods despite non-trivial generation accuracy.The authors suggest these small models may lack the instruction-following and reasoning capabilities required for self-improvement.Source: §4.1 · p. 6
• CoT verification has positive gaps for medium and large models, while some MC methods yield non-positive gaps even for medium-sized models.The authors note this aligns with studies finding MC evaluation can be unreliable, especially for small models.Source: §4.1 · p. 6
• With certain verification methods, relative GV-Gap grows monotonically with pre-training flops, while absolute GV-Gap shows no similar trend.The authors hypothesize a linear relationship between relative GV-Gap and the logarithm of pre-training flops, with slopes that may differ by model family.Source: §4.1 · p. 6

4.2 Cross Verification

• In cross-verification, gaps rise with verifier capacity and fall with generator capacity.The study varies one model role at a time across Llama-2 and Qwen-2 families, using MC with quantile threshold and CoT-Score.Source: Figure 3 · p. 7, §4.2 · p. 7
• Larger verifiers may be costly, and a smaller verifier does not always yield a positive gap, leaving compute-optimal pairings unresolved.The authors note that finding the optimal verifier for each generator may require a combinatorially large number of experiments.Source: §4.2 · p. 7

4.3 Unimprovable Tasks

• On Natural Question, gaps remain below 1% or negative across models despite non-trivial generation accuracy.The analysis uses a 3,610-question test subset; Table 1 reports near-zero gaps for Qwen-2 models.Source: §4.3 · pp. 7–8, Table 1 · p. 8

4.4 Sudoku

• Around 200% improvement appears for Qwen-2 72B Sudoku models, while smaller models do not self-improve.Across the tested families, only the largest models—Qwen-1.5/2 72B and Llama 3.1 70B—show non-trivial gaps, with 50%–300% accuracy improvement.Source: Table 2 · p. 8, §4.4 · p. 8
• The authors suggest sudoku self-improvement requires reasoning and planning, and propose developing a metric to predict task-specific self-improvability.Their analysis is primarily post-hoc; they note most models may lack the capabilities needed for sudoku despite its simpler verification.Source: §4.4 · p. 8

• The gap falls nearly to zero within two or three rounds at similar rates across capacities; format adherence explains extra 7B/14B accuracy gains, not self-improvement.For 7B and 14B, iteration-one accuracy exceeds the generation accuracy plus gap at iteration zero because finetuning improves adherence to the required answer format.Source: Figure 4 · p. 9, §5 · p. 9
• With MC verification, the gap is near zero after one round, making further multi-round self-improvement unlikely.This rapid saturation holds across other MC thresholds as well.Source: §5 · p. 9
• Small-k pass@k rises across rounds, while large-k pass@k falls, indicating reduced effective generation diversity.The authors suggest that failure to verify rare but correct answers may lead to convergence on incorrect solutions.Source: §5 · p. 9
• The MATH benchmark shows the same saturation limit and decrease in effective diversity as GSM8K.The paper also reports iterative-improvement experiments with a more powerful verifier and ground-truth labels in an appendix.Source: §5 · p. 9

6.1 Generalization of Verifications

• Tournament gaps relative to the previous round generally decrease monotonically as later-stage filtering is more likely to exploit verification errors.Source: §6.1 · p. 10
• CoT-Score has a score mode of 10 across models, while Tournament shows no critical recency bias, with a first-generation preference of 0.56 ± 0.12.Source: §6.1 · p. 10
• Across most models, MC and Tournament gaps vary concavely with threshold, while CoT-Score gaps increase monotonically; optimal thresholds often agree across models but may not transfer across tasks.When tuning a large model is costly, the authors suggest first finding a threshold on a smaller model.Source: §6.1 · p. 10

6.3 Improvement via Ensemble

• Combining non-trivial-gap verifications with an AND filter improves verification performance, except for CoT on the 0.5B model, whose gap is near zero.The reported example is for the Qwen-2 family; improvements across model sizes suggest smaller-model strategies can transfer when all verifications are valid.Source: Table 3 · p. 11, §6.3 · p. 11

6.2 Correlation Studies between Verification Methods

• Verification methods differ substantially: MC has greater gap variance and can drive accuracy to an extreme in one round, while CoT slightly improves accuracy on most questions.The pattern is reported across models.Source: §6.2 · p. 11, Figure 7 · p. 11
• Verifier outputs are generally weakly correlated, and gaps do not positively correlate with generation accuracy, supporting the potential value of verifier ensembles and relative gap measurement.Source: §6.2 · p. 11

• Synthetic data supports pre- and post-training, but self-training on model-generated data has also been linked to performance degradation and model collapse.The cited literature examines strategies to counter that behavior.Source: §7 · p. 12
• Prior self-improvement studies report practical gains, but often use additional verification data, surrogate improvement metrics, or only a small number of models.This work instead aims to analyze self-improvement in a controlled, comprehensive manner.Source: §7 · p. 12
• Test-time inference work allocates more computation to generation and selection, while test-time scaling studies optimize model size and accuracy under fixed compute and verifier constraints.The authors suggest that understanding self-improvement could inform test-time scaling without an external verifier.Source: §7 · pp. 12–13
• LLM-as-a-judge uses a language model to verify another model’s or its own generation; this paper studies verification mechanisms as a component of self-improvement.Source: §7 · p. 13
• The paper’s framework relies on reweighting the generation distribution, a strategy with a history in NLP reranking applications.Source: §7 · p. 13

8 Conclusion and Discussion

• The study examines self-improvement across model families, tasks, and verification mechanisms, and identifies relative-gap scaling, iterative saturation, and ensemble verification as findings with practical implications.The authors describe the scaling analysis as primarily observational and call for a more extensive scaling-law study.Source: §8 · p. 13
• Compute-optimal methods for self-improvement across different tasks remain a critical challenge, while an inference-time scaling law is presented as a possibility.The paper also raises this possibility for cross-improvement.Source: §8 · p. 13
• Declining effective generation diversity during iterative self-improvement remains a significant obstacle, and mitigating it could offer empirical benefits.Source: §8 · p. 13
• Verification mechanisms have distinct, non-overlapping properties despite functional similarities, motivating further exploration of their composition.The authors suggest combining compositional verification could enhance self-improvement.Source: §8 · p. 13

A Verification Mechanisms

• Multiple-choice verification scores Correct versus Incorrect labels, chain-of-thought verification returns a score, and tournament verification selects winners through pairwise elimination.The paper considers CoT-Score and CoT-Binary variants.Source: A Verification Mechanisms · p. 20

B.1 Sampling Temperature

• At temperatures 1 and 0.5, relative GV-Gap increases monotonically with pre-training flops and is almost linear against log flops.The 14B model at temperature 1 is an exception; the authors attribute this possibly to added sampling noise.Source: §B.1 · p. 21, Figure 8 · p. 21

B.2 Instruct Models

• Qwen-1.5-Chat and Qwen-2.5-Instruct lack the scaling property, whereas Llama-2-Chat shows it, based on a curve fit using only three models.Qwen-1.5-Chat accuracy also does not scale monotonically with model size.Source: Figure 9 · p. 21, §B.2 · p. 22

B.3 Fixed Verifier

• With gold labels, effective diversity does not degrade during iterative training; with a stronger but imperfect verifier, the degradation remains.The authors link the latter to some wrong answers being labeled correct and then concentrated on during training.Source: §B.3 · p. 22, Figure 10 · p. 22

B.4 Variance of the Verifier

• For Qwen-1.5-72B, verifier variance is near zero for most generations, though variance behavior differs across models.The analysis uses 64 CoT-B verifications per response and measures empirical variance as p(1 − p).Source: §B.4 · p. 22, Figure 11 · p. 23

C.1 Additional Results for Section 4.1

• The gap shows no clear scaling with pre-training flops; the figure plots relative gap against log flops for three verification methods.The methods are MC with a top-0.8 quantile threshold, CoT-Score with global threshold 8, and Tournament with 5 rounds.Source: Figure 12 · p. 23, Table 4 · p. 24

C.2 Additional Results for Section 4.3

• The supplementary tables report relative gaps on GSM-8K and gaps on Natural Question, where near-zero gaps with non-trivial generation accuracy indicate a non-improvable task.Source: Table 5 · p. 25, Table 6 · p. 26

C.3 Additional Results for Section 4.4

• The Sudoku table reports generation accuracy, gap, and relative gap for the evaluated models.Source: Table 7 · p. 26

C.4 Additional Results for Section 5

• On MATH, iterative self-improvement saturates in 4 iterations with the gap diminishing to 0, and effective diversity degrades along the process.Source: §C.4 · pp. 26–27, Figure 13 · p. 27

C.5 Additional Results for Section 6.1

• The gap is plotted across thresholds for MC and CoT-Score; CoT-Binary's gap does not change with threshold.Source: Figure 14 · p. 28
• Tournament figures track filtered-dataset accuracy and gap by round, with the gap measured against the previous round's accuracy.Source: §C.5 · pp. 28–29, Figure 15 · p. 30
• The score distribution figure reports a mode of 10 for all models.Source: Figure 16 · p. 30

C.6 Additional Results for Section 6.2

• Figure 17 summarizes GSM8K gap distributions by verification method and model, using bins 0.005 wide and marking each distribution's mean and standard deviation.Source: Figure 17 · p. 34
• Additional Qwen-1.5 plots show verification-output correlations and correlations between each verification method's gap and generation accuracy.Source: §C.6 · pp. 34–35, Figure 18 · p. 35, Figure 19 · p. 36

C.7 Additional Results for Section 6.3

• Table 8 defines relative-gap thresholds on GSM-8K using either prompt-specific utility quantiles (“top n”) or a shared numeric threshold (τ = n).The table caption states that values are percentages.Source: Table 8 · p. 37

D Details on Iterative Self-improvement

• The RL update uses Reasoning Preference Optimization, equivalent to a Bradley-Terry RL update with SFT regularization, with β = 0.05 and SFT regularization coefficient 1.Source: D Details on Iterative Self-improvement · p. 37
• Each iteration samples N outputs per prompt, scores them, retains outputs above τ, and updates the model by maximum-likelihood estimation.The procedure begins from a pretrained model and repeats for T iterations.Source: D Details on Iterative Self-improvement · p. 38

E Generation and Verification Prompts

• The math prompts include binary correctness judgments, chain-of-thought scoring, and pairwise solution tournaments.The binary prompt checks both the final answer and derivation; the scoring prompt rates solutions from 1 to 10.Source: E Generation and Verification Prompts · pp. 38–39
• The appendix also gives a 4×4 Sudoku solving prompt and a validator that checks the initial clues and Sudoku constraints.Source: E Generation and Verification Prompts · pp. 39–40

F Infrastructure Statement

• Inference runs use Nvidia A100 40GiB nodes, while iterative self-improvement training uses Nvidia A100 80GiB nodes.Source: F Infrastructure Statement · p. 41