PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization

Yongcheng Zeng, Xinyu Cui, Xuanfa Jin, Qirui Mi, Guoqing Liu, Zexu Sun, Mengyue Yang, Dong Li, Weiyu Ma, Ning Yang, Jian Zhao, Jianye Hao, Haifeng Zhang, Jun Wang

arXiv:2502.05605v6cs.CLcs.LG

TL;DR

High-quality training data is scarce, and whether LLMs can refine responses without external information remains a central question. EVOLVE combines iterative preference training with Self-Refinement inference; its evolved Llama-3.1-8B model reports wins over GPT-4o and gains on reasoning benchmarks.

  • Problem

    High-quality training data is scarce, and it remains an open question whether LLMs can autonomously refine responses without external information.

    Source: §1 · pp. 1–2

  • Method

    EVOLVE iteratively combines preference training with multi-step Self-Refinement inference to generate preference data for subsequent training cycles.

    Source: §6 · p. 11

  • Results

    EVOLVE’s Llama-3.1-8B surpassed GPT-4o, with 62.3% length-controlled and 63.3% raw win rates on AlpacaEval 2 and 50.3% on Arena-Hard, and improved GSM8K and MATH.

    Source: §1 · pp. 2–3, §6 · p. 11

  • Takeaways & Limitations

    The paper investigates leveraging Self-Refinement to achieve Self-Improvement of intrinsic model abilities.

    Source: §1 · p. 2, §6 · p. 11

    On GSM8K and MATH, Self-Refinement gains become less pronounced as accuracy rises, which the authors attribute to absent domain-specific training.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Self-Refinement refers to a model's ability to revise its own responses to produce improved outputs. This capability can also serve as a fundamental mechanism for Self-Improvement, for example, by reconstructing datasets with refined results to enhance intrinsic model performance. However, our comprehensive experiments reveal that large language models (LLMs) show no clear evidence of inherent Self-Refinement and may even experience response quality degradation after Self-Refinement. To address this issue, we propose EVOLVE, a simple and effective framework for eliciting and tracking the evolution of Self-Refinement through iterative training. We first explore optimization methods during training to activate the model's Self-Refinement capability. Then, at inference, we investigate various generation strategies to further enhance and utilize Self-Refinement while supplying the necessary data for training. Through synergistic optimization of training and inference stages, we continually evolve the model's Self-Refinement ability, enabling it to better refine its own responses. Moreover, we demonstrate the potential of leveraging Self-Refinement to achieve broader Self-Improvement of intrinsic model abilities. Experiments show that the evolved Self-Refinement ability enables the Llama-3.1-8B base model to surpass GPT-4o, achieving 62.3% length-controlled and 63.3% raw win rates on AlpacaEval 2, and 50.3% on Arena-Hard. It also generalizes effectively to out-of-domain reasoning tasks, improving performance on mathematical reasoning benchmarks such as GSM8K and MATH.

1 Introduction

The paper asks whether LLMs can refine their own responses without external information, against a backdrop of scarce high-quality training data. It proposes EVOLVE to strengthen this capability through iterative training and inference, reporting benchmark gains from a Llama-3.1-8B base model.

1 Introduction

  • The paper asks whether LLMs can autonomously refine responses without external information, as Self-Improvement methods seek to use model-generated data amid scarce high-quality training data.Self-Refinement is framed as revising a model’s own outputs to improve accuracy and stability.Source: §1 · pp. 1–2
  • Preliminary experiments found no clear evidence of inherent Self-Refinement in LLMs, with response quality sometimes degrading after refinement.The study used three refinement templates across models with different architectures.Source: §1 · p. 2
  • EVOLVE investigates activating and strengthening Self-Refinement through fine-tuning, ongoing data acquisition, and iterative training updates.The paper also examines using activated Self-Refinement to improve intrinsic model capabilities.Source: §1 · p. 2
  • The work studies training methods for generating improved responses from prior outputs and strategies for using Self-Refinement to collect datasets.These studies are framed as eliciting and enhancing Self-Refinement through iterative training.Source: §1 · p. 2
  • 62.3% length-controlled and 63.3% raw win rates on AlpacaEval 2, plus 50.3% on Arena-Hard, accompanied EVOLVE’s improvement over GPT-4o and Llama-3.1-405B-Instruct and its gains on GSM8K and MATH.The reported results start from the Llama-3.1-8B base model.Source: §1 · pp. 2–3
  • EVOLVE’s framework iteratively alternates inference-based preference-data collection with preference training, filtering the data by rules or a reward model.Each iteration produces the next model in the training cycle.Source: Figure 2 · p. 3

2 Related Works

Related work connects Self-Improvement to generated data and feedback, preference optimization to dataset quality, and iterative optimization to data scarcity. This work focuses on strengthening Self-Refinement through iterative training.

Self-Improvement

Reinforcement Learning from Human Feedback (RLHF)

Data Generation and Iterative Optimization

3 Methodology

EVOLVE studies how training and inference can jointly activate and strengthen Self-Refinement through iterative training. It combines complementary SFT and PT stages with generation strategies that support data acquisition.

3.1 Training Stage of EVOLVE: Synergistic Optimization of SFT and PT

  • Only synergistic SFT and PT effectively activate and consistently strengthen Self-Refinement; alternative PT objectives yield comparable improvements.SFT alone leaves the capability limited, while response quality can oscillate without its adaptations.Source: Figure 3 · p. 4, §3.1 · p. 5
  • SFT trains the model to produce preferred responses both directly and when refining rejected responses using a refinement template.Without these SFT adaptations, response quality oscillates across inference turns.Source: §3.1 · p. 4

3.2 Inference Stage of EVOLVE: Exploring Dynamic Generation Strategies

  • The four inference strategies are parallel sampling, chained refinement, few-shot refinement using prior generations, and refinement conditioned on self-evaluations.Self-evaluation is generated from the prompt and current response, then conditions the next refinement.Source: §3.2 · pp. 5–6
  • Chain of Self-Refinement achieves the best results across iterations among the four generation strategies.It repeatedly refines the previous response using the prompt and refinement template.Source: Figure 5 · p. 6, §3.2 · p. 6
  • EVOLVE iterates preference training and data generation, with improved Self-Refinement supporting hierarchical high-quality data generation in a sustained training loop.The framework integrates findings from the training and inference phases.Source: §3.2 · p. 6

4 Experiments

The experiments evaluate EVOLVE’s iterative training and inference strategy, compare it with prior methods, test key components, and examine transfer to mathematical reasoning. EVOLVE improves benchmark performance, strengthens Self-Refinement across iterations, and generalizes to out-of-domain reasoning tasks.

4.1 Experiments Setting

  • The setup trains EVOLVE through SFT, preference training, and iterative preference-data updates, then evaluates Direct and three-round Self-Refinement on four general and mathematical benchmarks.The benchmarks are AlpacaEval 2, Arena-Hard, GSM8K, and MATH.Source: §4.1 · pp. 6–7

4.2 Boost in Model Performance with EVOLVE

  • Across iterations, EVOLVE-trained models improve in both inference performance and Self-Refinement, while the refinement gain over direct generation increases each round.Figure 6 reports responses generated from 256 UltraFeedback test samples.Source: §4.2 · pp. 7–8, Figure 6 · p. 7
  • EVOLVE delivers consistent improvements in Direct answering and a qualitative leap under Self-Refinement on AlpacaEval 2 and Arena-Hard, with more stable gains than existing methods.Table 1 compares scores under both settings, with parenthetical gains over the SFT baseline.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Impact of Key Components on EVOLVE Performance

  • Omitting DPO loss slows iterative improvement in both Direct and Self-Refinement settings, likely because direct-answer quality and the ceiling for later refinement decline.Figure 7 analyzes DPO loss’s impact on EVOLVE performance.Source: §4.3 · p. 8, Figure 7 · p. 9
  • Models trained with Parallel Sampling datasets underperform those using Self-Refinement generation, which the authors identify as essential for high-quality preference data.The authors also report clearer, more concise, and more logically coherent refined responses.Source: §4.3 · p. 9, Figure 8 · p. 9
  • Using ArmoRM produces performance comparable to Skywork Reward Model, indicating EVOLVE’s effectiveness is robust across these reward-model choices.Table 2 reports the comparison under Direct and Self-Refinement settings.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 Generalization of EVOLVE to Out-of-Domain Reasoning Tasks

  • EVOLVE’s activated Self-Refinement generalizes to GSM8K and MATH, with measurable gains despite no explicit exposure to mathematical data.As accuracy rises, gains from Self-Refinement become less pronounced; the authors attribute this to lacking domain-specific training for advanced strategies.Source: §4.4 · p. 10, Table 3 · p. 10

5 Extension: The Potential of Self-Refinement for Achieving Model Self-Improvement

The paper tests whether Self-Refinement can support model Self-Improvement through online iterative training and self-collected preference data. EVOLVE improves direct and refined responses while collecting higher-quality datasets, though unsupervised gains slow and the process introduces noise.

Rules for Achieving Model Self-Improvement

  • EVOLVE’s online setup filters four generated responses by selecting the first as rejected and the fourth as chosen, then trains on the resulting preference pairs.The described approach starts from the EVOLVE offline model and performs two online training rounds.Source: §5 · p. 10

Evolution of Self-Refinement and Dataset Augmentation

  • EVOLVE-trained models improve in both direct answering and Self-Refinement during iterative training, and Self-Refinement collects higher-quality datasets for further gains.Under unsupervised conditions, capability improvement slows.Source: §5 · p. 10, Figure 9 · p. 11

Challenges with Self-Rewarding and SPIN

  • Self-Rewarding can stagnate with low-discriminability data, while SPIN can degrade with lower-quality collections, exposing a challenge for Self-Improvement in data-scarce domains.The comparison is reported in Table 4.Source: §5 · p. 10, Table 4 · p. 11

Summary

  • Self-Refinement improves dataset quality but also introduces significant noise, leaving the Self-Improvement process complex and in need of further study.The paper additionally points to further exploration of Self-Improvement’s upper bounds.Source: §5 · pp. 10–11

6 Conclusion

EVOLVE combines iterative preference training with Self-Refinement inference, using refined outputs to build data for later training cycles. The conclusion reports gains on AlpacaEval 2 and Arena-Hard and identifies broader intrinsic Self-Improvement as a potential application.

6 Conclusion

  • EVOLVE couples iterative preference training with multi-step Self-Refinement inference, using refined outputs to create preference data for subsequent training cycles.The conclusion says this loop improves direct question-answering and Self-Refinement capabilities.Source: §6 · p. 11
  • The framework surpasses GPT-4o on AlpacaEval 2 and Arena-Hard, while the conclusion identifies intrinsic Self-Improvement as a potential application.Future work will extend the approach to more complex reasoning and broader domains, including mathematics and coding.Source: §6 · p. 11

C.2 Training Details

  • The SFT phase uses a 5 × 10^-6 learning rate, batch size 128, sequence length 1024, cosine schedule, 3% warm-up, one epoch, and AdamW.Preference training uses a 1 × 10^-6 learning rate; β is 0.01 for reward-model filtering and 0.05 for rule-based selection.Source: §C.2 · p. 17

C.3 Inference Details

  • Iterative data collection and evaluation use temperature 0.7 for direct and refined responses, while GSM8K and MATH use greedy decoding.AlpacaEval 2 generation uses temperature 0.9; MT-Bench and Arena-Hard follow their official decoding configurations.Source: §C.3 · p. 17

C.4 Evaluation Benchmarks Details

  • Evaluation covers AlpacaEval 2, MT-Bench, Arena-Hard, GSM8K, and MATH, including the full 1,319-question GSM8K test set and 1,024 MATH samples.The cited passage describes the benchmark formats and reports scores using each benchmark's evaluation protocol.Source: §C.4 · p. 17

C.5 Experiment Results on MT-Bench

  • MT-Bench scores are averaged using GPT-4 and GPT-4-Preview-1106 as judges.The cited table presents MT-Bench results.Source: §C.5 · p. 18, Table 5 · p. 18

C.6 Computational Overhead of Training and Inference

  • The computational-overhead section compares Iterative DPO and EVOLVE across two rounds of online iterative training.The supplied passage identifies the comparison and table but does not give overhead values.Source: §C.6 · p. 18, Table 6 · p. 18
  • A normalization parameter α keeps the overall loss weight equal to 1 during updates for a fair comparison.This adjustment is described in the computational-overhead section.Source: §C.6 · p. 18

C.7 Ablation Study on Parameter α

The α ablation selects 0.8 as optimal for Self-Refinement on Llama-3.1-8B Base. The accompanying derivation frames refinement preference training through a modified Bradley-Terry setup.

C.7 Ablation Study on Parameter α

  • α = 0.8 is selected because Self-Refinement performs optimally at that value on Llama-3.1-8B Base.Figure 10 presents experimental results for different α values.Source: Figure 10 · p. 18, §C.7 · p. 18

D.1 The Derivation of the Refinement Loss Function from the Bradley-Terry model Perspective

  • The modified Bradley-Terry setup conditions preference on the input, an arbitrary response, and a refinement template guiding generation of a better response.The derivation then defines a refinement preference function and combines the Self-Refinement loss with DPO loss.Source: §D.1 · p. 19
  • The refinement objective is parameterized as a policy model and estimated by maximum likelihood, with the preference problem treated as binary classification.The appendix describes deriving a negative log-likelihood loss for this formulation.Source: §D.1 · p. 19

D.3 Experiment Results

BT_EVOLVE and EVOLVE show nearly identical performance across the reported benchmarks, which the authors attribute to refinement rather than the specific algorithm. The supplied figure and table captions identify comparisons and evaluation scope but do not provide additional outcome values.

D.3 Experiment Results

  • BT_EVOLVE and EVOLVE perform nearly identically across benchmarks, which the authors attribute to the refinement principle rather than the specific algorithm.Table 7 covers AlpacaEval 2, Arena-Hard, and MT-Bench under Direct and Self-Refinement settings; Table 8 covers GSM8K and MATH.Source: Table 7 · p. 20, Table 8 · p. 20, §D.3 · p. 20

E Comparison with Open-Source Models

  • Length-controlled AlpacaEval 2 win rate improves over EVOLVE iterations and surpasses GPT-4 level for base Llama-3.1-8B versions using Self-Refinement.Figure 11 states this trend but supplies no numerical win rates.Source: Figure 11 · p. 21

F.1 DPO Loss and Generation Strategy

  • The ablation table covers AlpacaEval 2 and Arena-Hard, reporting length-controlled win rate and raw win rate comparisons.The supplied caption identifies the metrics and benchmarks but provides no cell values.Source: Table 10 · p. 21

G Quality Evaluation of Data Collected under the Self-Improvement Setting

Two reward models consistently assess Self-Refinement-collected preference data as higher quality. Further iterations indicate that EVOLVErule Self-Refinement performance has stabilized, while Direct generation continues to improve.

G Quality Evaluation of Data Collected under the Self-Improvement Setting

  • Both Skywork Reward Model and ArmoRM assessments consistently indicate higher dataset quality for the Self-Refinement generation strategy.The figure's y-axis is the percentage of samples where the chosen response receives a higher reward than the rejected response; the two reward models are used to reduce potential evaluation bias.Source: Figure 12 · p. 22

H Further Exploration of Self-Improvement with More Iterations

  • EVOLVErule Self-Refinement performance has essentially reached its upper bound, while Direct generation continues to improve and Self-Rewarding converges by Iter 3.The exploration divides the last 15K UltraFeedback samples into three 5K subsets for three additional training iterations and evaluates on the UltraFeedback test set.Source: H Further Exploration of… · p. 22, Figure 13 · p. 22

I Evaluation of Self-Refinement Capabilities in Open-Source Models

The evaluation reports that current open-source models often fail to refine responses effectively and may degrade them; the section also examines sequential refinement and self-evaluation prompts.

I Evaluation of Self-Refinement Capabilities in Open-Source Models

  • Four open-source instruction models were evaluated across three refinement templates, with two reward models used to score results.The models were Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-2-9B-Instruct, and Mistral-7B-Instruct; Figure 14 reports mean ArmoRM scores across three runs.Source: I Evaluation of Self-Refinement… · pp. 23–24, Figure 14 · p. 24
  • Current open-source models often struggle to refine responses effectively, and refinement can degrade performance.The evaluation used Skywork Reward Model and ArmoRM to reduce dependence on a single reward model.Source: Figure 14 · p. 24, I Evaluation of Self-Refinement… · p. 24

J Analysis of Various Dynamic Generation Stratedies

  • Dynamic-generation prompts supply prior responses in sequence, while Self-Evaluation retains the Chain template and adds explicit question-and-response critique.The critique covers question understanding, response quality, and suggestions for improvement.Source: J Analysis of Various Dynamic… · pp. 25–26
  • Examples show small models without domain-specific evaluation training giving superficial assessments and missing consequential errors in proposed solutions.The cited examples include an incorrect arithmetic conclusion and a mistaken age calculation.Source: Table 11 · p. 27, Table 12 · p. 28

K Qualitative Examples

The examples contrast direct and refined responses, illustrating both recipe differences and improvements in organization and reasoning.

K Qualitative Examples

  • The direct-response cactus recipe specifies 40–50% coarse sand, 30–40% pumice or perlite, and 10–20% coco coir.Calcitic lime is optional in that example.Source: K Qualitative Examples · p. 29
  • The self-refinement cactus example specifies 40–50% coarse grit or sand and 20–30% perlite or pumice, with loam or vermiculite also listed.Source: K Qualitative Examples · p. 29
  • Both cactus responses are described as valuable, and the direct answer recommends well-draining soil and keeping it slightly dry between waterings.Source: K Qualitative Examples · p. 29, Table 13 · p. 30
  • EVOLVE’s refined answers are more concise and organized than GPT-4-1106-Preview, with more coherent reasoning than direct responses and logic comparable to that baseline.Source: Table 14 · p. 31, Table 15

L Downstream Task Evaluation

Downstream evaluations found no catastrophic forgetting, with performance remaining comparable to baseline across the reported tasks.

L Downstream Task Evaluation

  • The evaluation followed lm-evaluation-harness protocols and included few-shot settings identified beneath each benchmark name.Source: L Downstream Task Evaluation · p. 32
  • Downstream-task performance remained comparable to baseline, indicating no catastrophic forgetting during post-training.The authors attribute this pattern primarily to limited relevance between training-data distribution and downstream-task skills, and note stable or slightly degraded performance may be expected.Source: L Downstream Task Evaluation · p. 32, Table 16 · p. 33

Related papers

Loading 2502.05605v6…