PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Refine: Iterative Refinement with Self-Feedback

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, Peter Clark

arXiv:2303.17651v2cs.CLcs.AIcs.LG

TL;DR

LLMs may produce intelligible but suboptimal outputs on tasks with multifaceted objectives or hard-to-define goals. Self-Refine has the same model generate feedback and iteratively refine its output, outperforming direct generation across seven diverse tasks.

  • Problem

    LLMs can fall short on tasks with multifaceted objectives or hard-to-define goals, motivating iterative refinement without relying on domain-specific training data.

  • Method

    SELF-REFINE generates an initial output, obtains feedback from the same LLM, and iteratively refines the output until a specified condition is met.

  • Results

    5-40% absolute improvement over direct generation was achieved across seven diverse tasks using strong LLMs, with improvements across all model sizes and tasks.

  • Takeaways & Limitations

    SELF-REFINE shows that strong LLMs can provide useful feedback and improve their own outputs at test time without additional training or reinforcement learning.

  • Takeaways & Limitations

    SELF-REFINE requires base models with sufficient few-shot modeling or instruction-following abilities to provide feedback and refine outputs in context.

Abstract

from arXiv · show

Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.

1 Introduction

SELF-REFINE iteratively improves LLM outputs by having the same model generate feedback and refine its draft, addressing shortcomings on complex or underspecified tasks. Across seven diverse generation tasks, it outperforms direct generation from strong LLMs, with 5–40% absolute gains and up to 13% in code generation.

  • Motivation: LLMs often produce coherent initial outputs yet struggle with intricate requirements, multifaceted objectives, and hard-to-define goals.Such settings include dialogue response generation and improving program readability, where iterative mapping from a candidate to an improved output may help.
  • Motivation: The approach draws on human iterative self-refinement, in which an initial draft is revised using self-provided feedback.Examples include revising an impolite email request and refactoring an initially quick-and-dirty program implementation.
  • Method: SELF-REFINE alternates FEEDBACK and REFINE steps, using the same model to critique an initial output and revise it until a stopping condition is met.The process runs for a specified number of iterations or until the model determines that no further refinement is needed.
  • Empirical scope and results: SELF-REFINE outperforms direct generation from GPT-3.5 and GPT-4 by 5–40% absolute across seven diverse generation tasks, improving code generation by up to 13%.The evaluation spans natural-language and source-code generation, and code models such as Codex also benefit.

2 Iterative Refinement with SELF-REFINE

SELF-REFINE iteratively generates an output, obtains feedback from the same language model, and refines the output until a task-specific stopping condition is met. It requires no training and relies on three prompts plus supervision contained in few-shot examples.

  • FEEDBACK: Feedback is prompted to be actionable and specific, identifying concrete changes likely to improve the output.Feedback examples can address multiple dimensions, such as code efficiency, readability, and overall quality.
  • Iterating SELF-REFINE: SELF-REFINE alternates FEEDBACK and REFINE steps until a task-specific stopping condition, such as a fixed timestep or feedback-derived stop score, is met.The final refined output is returned, and prior outputs and feedback are appended to later prompts to help avoid repeating mistakes.
  • 2 Iterative Refinement with SELF-REFINE: SELF-REFINE uses the same underlying language model to generate, critique, and refine outputs through repeated feedback–refinement cycles.The method uses separate prompts for initial generation, feedback, and refinement, without additional training.
  • 2 Iterative Refinement with SELF-REFINE: The approach requires no training and relies only on supervision contained in few-shot examples for its generation, feedback, and refinement prompts.Few-shot examples are task-specific input-output pairs, feedback triples, and refined-output quadruples.

3 Evaluation

Across seven diverse tasks, SELF-REFINE consistently improves the same base LLMs and outperforms previous state-of-the-art systems. Gains are especially large in preference-based tasks, while math improvements are limited by feedback’s difficulty identifying subtle errors.

  • Main results: SELF-REFINE consistently improves base models across all model sizes and outperforms previous state-of-the-art results across all tasks.Table 1 summarizes results using GPT-3.5, ChatGPT, and GPT-4 as base LLMs.
  • Code-based tasks: 8.7% absolute: GPT-4+SELF-REFINE increases Code Optimization from 27.3% to 36.0%.CODEX shows similar trends on code-based tasks.
  • Constrained Generation: SELF-REFINE gains especially in Constrained Generation because iterative refinement can repair missed concepts among many reasonable outputs.The task requires generating a sentence containing up to 30 given concepts, creating more opportunities for first-attempt omissions.
  • Preference-based tasks: 49.2%: GPT-4 preference score in Dialogue Response Generation rises from 25.4% to 74.6% with SELF-REFINE.Other preference-based tasks also show remarkable improvements across all models.

4 Analysis

The analysis shows that SELF-REFINE depends on specific feedback, benefits from repeated iterations, and is limited by weaker models’ difficulty following the refinement process. Qualitative analysis further identifies feedback accuracy as the main determinant of success, while a website-generation case suggests broader applicability.

  • The impact of the feedback quality: Specific, actionable feedback outperforms generic or absent feedback across Code Optimization, Sentiment Transfer, and Acronym Generation.Code Optimization scores fall from 27.5 with SELF-REFINE feedback to 26.0 with generic feedback and 24.8 without feedback; Sentiment Transfer falls from 43.2 to 31.2 and fails without feedback, while Acronym Generation falls from 56.4 to 48.0 without actionable feedback.
  • How important are the multiple iterations of FEEDBACK-REFINE?: Iteration generally improves output quality, with Code Optimization rising from 22.0 initially to 28.8 after three iterations and Sentiment Reversal from 33.9 to 36.8.Most gains occur in early iterations, although multi-aspect tasks can fluctuate; numerical aspect scores support balanced evaluation and output selection.
  • Does SELF-REFINE work with weaker models?: Vicuna-13B struggled to generate correctly formatted feedback and follow refinement prompts, often repeating outputs or hallucinating conversations even with Oracle or hard-coded feedback.This indicates that SELF-REFINE’s refinement process may not transfer reliably to weaker base models.
  • Qualitative Analysis: Feedback errors caused most unsuccessful refinements: 33% mislocated errors, 61% inappropriate fixes, and only 6% incorrect implementation of good feedback.These findings make accurate feedback the main failure-sensitive component of SELF-REFINE.
  • Qualitative Analysis: Successful refinements used accurate feedback for precise fixes in 61% of cases, while 33% succeeded despite partially incorrect feedback, indicating resilience to sub-optimal feedback.The refiner could sometimes correct issues even when the feedback was not fully correct.
  • Going Beyond Benchmarks: A website-generation case extended SELF-REFINE beyond benchmarks by iteratively refining HTML, CSS, and JavaScript for improved usability and aesthetics.The process began with a rudimentary design and evolved it from a user-provided high-level goal.

5 Related work

Prior refinement work spans diverse tasks and varies in feedback source, representation, and refiner type. SELF-REFINE uses natural-language feedback generated by the same pretrained language model, avoiding supervised refiner training and parameter updates.

6 Limitations and Discussion

The discussion identifies limitations involving model capabilities, access and reproducibility, language scope, and potential misuse, while noting that code and outputs are released to support reproducibility.

  • Self-Refine requires base models with sufficient few-shot modeling or instruction-following abilities to provide feedback and refine outputs in context without supervised training data.
  • The experiments use closed, costly models whose training details and biases are not fully documented, although the authors release code and model outputs for reproducibility.
  • Because the experiments cover only English datasets, the models may not provide the same benefits in other languages.
  • The approach does not explicitly guard against bad actors using prompting techniques to steer models toward more toxic or harmful text.

7 Conclusion · A Evaluation Tasks · B Broader Related Work

SELF-REFINE enables a single LLM to generate feedback and iteratively refine its outputs without additional training data or reinforcement learning, and is evaluated across diverse tasks. The paper distinguishes this structured, granular feedback-based approach from Reflexion, Self-Correction, and reinforcement-learning alternatives while releasing code, data, and prompts.

  • 7 Conclusion: SELF-REFINE uses one LLM to provide self-feedback and iteratively refine outputs without additional training data or reinforcement learning.The authors present this as a simple approach applicable across a wide variety of tasks.
  • 7 Conclusion: The authors make their code, data, and prompts anonymously available to support further research on iterative refinement.The materials are available at selfrefine.info.
  • A Evaluation Tasks: The evaluation covers multiple tasks and datasets, illustrating one refinement iteration from input x and prior output y_t to feedback fb_t and refined output y_t+1.Table 4 provides the associated datasets, sizes, task examples, and few-shot prompts used for feedback and refinement.
  • B Broader Related Work: Compared with Reflexion, SELF-REFINE corrects outputs using granular, structured, multidimensional feedback rather than seeking the next solution in ReAct-based planning.Reflexion and ReAct provide free-form reflection about execution and possible improvements, whereas SELF-REFINE is described as more precise and actionable.
  • B Broader Related Work: Compared with Self-Correction (Welleck et al., 2022), SELF-REFINE explicitly generates feedback, which the paper reports produces significantly better refined outputs.The comparison also states that Self-Correction refines with separately trained models, whereas SELF-REFINE uses instructions and few-shot prompting.
  • B Broader Related Work: SELF-REFINE avoids training a separate task-specific refiner by using instructions and few-shot prompting.This is contrasted with Self-Correction’s separate refiner or corrector for each task.
  • B Broader Related Work: Table 5 organizes related approaches into reinforcement-learning methods, trained correctors, and few-shot correctors.The caption identifies these categories by purple, orange, and green coloring, respectively, without reporting comparative cell values here.

C Human Evaluation … F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines

The paper evaluates SELF-REFINE through blind human A/B judgments, GPT-4-based structured assessment, and comparisons with few-shot and fine-tuned baselines across reasoning and programming tasks. These evaluations examine preference alignment, model-assisted judging, and performance under differing sample budgets and model architectures.

  • C Human Evaluation: SELF-REFINE is evaluated against a baseline using blind A/B judgments, where annotators select the output better aligned with the task instruction.The evaluation presents the input, instruction, and two candidate outputs without revealing which method produced them.
  • C Human Evaluation: The human-evaluation preference rate measures how often annotators select SELF-REFINE over the baseline, with evaluations conducted on 150 examples per dataset.Table 6 reports normalized preferences corresponding to SELF-REFINE’s proportion of selections over the baseline.
  • D GPT-4 Evaluation: GPT-4 is used as an evaluator by receiving structured comparison prompts that encourage deliberation and require a rationale before the final choice.This evaluation approach is demonstrated for sentiment reversal, acronym generation, and dialogue response generation.
  • D GPT-4 Evaluation: For acronym generation, GPT-4 compares candidates using ease of pronunciation, ease of spelling, relation to the title, and positive connotation.The prompt requires a short explanation and a categorical judgment about which acronym is better, equally good, or neither good.
  • E Model Key: The paper references OpenAI’s GPT-3.5 model terminology in its model key.The model-key section directs readers to the OpenAI documentation for GPT-3.5 terminology.
  • F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines: The comparison section evaluates SELF-REFINE against few-shot and fine-tuned baselines across mathematical reasoning and programming tasks, including the GSM and PIE datasets.Tables 7 and 8 present model-performance comparisons for math reasoning and program optimization, respectively.
  • F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines: On the PIE dataset, SELF-REFINE achieves superior program-optimization performance while using at most 4 samples, versus 16 or 32 samples for other models.Table 8 reports performance in percentage of programs optimized and includes human references, baseline models, fine-tuned PIE-2B and PIE-16B models, and SELF-REFINE with different LLMs.

G Evaluation of Vicuna-13b · H Additional Analysis

Vicuna-13b consistently follows task initialization prompts but struggles with feedback and refinement prompts, often producing assistant-like or unhelpful outputs. Using ChatGPT for feedback and refinement shows promise for improving Vicuna-13b’s initialization, although the supplied result is incomplete.

  • G Evaluation of Vicuna-13b: Vicuna-13b follows task initialization prompts consistently but struggles with feedback and refinement prompts, often producing assistant-like responses.The paper provides a representative example in Appendix G.
  • G Evaluation of Vicuna-13b: Using the same prompts as other models yielded limited Vicuna-13b performance, suggesting that more extensive prompt engineering may be needed.
  • G Evaluation of Vicuna-13b: Mixed-refine uses Vicuna-13b for initialization and ChatGPT for feedback and refinement, and its Math Reasoning results suggest this combination may improve performance.Vicuna-13b alone achieved 24.18% on the partially supplied Math Reasoning result.
  • G Evaluation of Vicuna-13b: In another sentiment-transfer trajectory, Vicuna-13b generated increasingly negative reviews but feedback inconsistently judged their sentiment and sometimes incorrectly cited positive wording.The trajectory includes feedback that the review was already very negative, followed by a contradictory claim that it contained positive phrases.
  • G Evaluation of Vicuna-13b: Vicuna-13b’s sentiment-transfer attempt produced empty feedback, an error, and subsequently unhelpful feedback before copying content from the prompt.The example contrasts these outputs with GPT-4 outputs.
  • G Evaluation of Vicuna-13b: Figure 6 compares preferences for SELF-REFINE outputs, the MULTI multiple-sample baseline, and ties.

H.1 Using Oracle Feedback … L.1 Method

Across its extensions and analyses, SELF-REFINE benefits from external correctness signals, supports iterative improvement beyond benchmarks, and yields statistically significant gains while exposing feedback and refinement failure modes. The paper also introduces new tasks and applies the method to recursive code-readability improvement.

  • H.1 Using Oracle Feedback: Oracle Feedback improves Math Reasoning performance by 4.8% for GPT-3 and 0.7% for GPT-4 by refining only incorrect current answers.The method uses correctness information to decide whether to proceed to refinement, indicating that external signals can help on particular tasks.
  • H.1 Using Oracle Feedback: In Dialogue Response Generation, feedback errors include 25% incorrect feedback, 30% generic feedback, and 10% incorrect scoring.These errors identify weaknesses in interpreting and understanding user inputs.
  • H.1 Using Oracle Feedback: During refinement, 10% of cases were not robust, 25% ignored feedback, and 20% introduced a new problem, while 60% were robust to incorrect or generic feedback.Acronym Generation can also fluctuate across iterations because improvements in one quality aspect may trade off against losses in another, motivating explicit numerical feedback scores.
  • I Beyond Benchmarks: Beyond benchmarks, SELF-REFINE generates actionable website-layout feedback covering colors, fonts, content, and layout, then produces refined ice-cream and photosynthesis pages.The supplied examples show concrete edits such as changing colors, increasing font sizes, adding content, and inserting dividers.
  • J Statistical Confidence Intervals: Nearly all GPT-4 gains are statistically significant, while ChatGPT gains are significant on 4 of 7 datasets and GPT-3.5 gains on 3 of 7.Table 13 reports Wilson confidence intervals and significance markers for SELF-REFINE across evaluated tasks.
  • K New Tasks: CommonGen-Hard extends CommonGen by requiring coherent sentences incorporating 20–30 concepts instead of 3–5, targeting advanced commonsense reasoning and creative problem-solving.SELF-REFINE addresses this constrained-generation challenge through iterative creation with introspective feedback.
  • L Code Readability: For code readability, SELF-REFINE uses natural-language critiques to guide edits, treating readability as distinct from correctness and important for usability, upgradability, and maintenance.The approach lets an LLM freely choose and express readability enhancements as feedback for code refinement.
  • L.1 Method: The readability method skips INIT, then recursively computes c_k+1 = critique(y_k) and y_k+1 = editor(y_k, c_k+1) for N iterations from y_0.FEEDBACK critiques the code, while REFINE applies the critique and returns the edited code for the next iteration.

L.2 Experiments … M.1 Modules

The paper applies SELF-REFINE to improve code readability and open-domain dialogue responses through iterative feedback and refinement. In code readability, SELF-REFINE achieves performance comparable to or better than human rewrites on three automatic metrics, while dialogue generation uses modular initialization, multifaceted feedback, and iterative refinement.

  • L.2 Experiments: The code-readability experiment uses a random 300-example subset of CodeNet containing hard-to-read multi-line competitive-programming snippets.A separate 60-example subset is edited by human annotators to assess human performance.
  • L.2 Experiments: Both critique and editor models use InstructGPT, with critique decoding at T = 0.0 or T = 0.7, greedy code editing, and N = 5 refinement iterations.The prompts are provided in Figures 22–23.
  • L.2 Experiments: Across iterations, the study tracks meaningful-variable ratio, comment ratio, and function-unit count for 300 CodeNet examples under critique temperatures T = 0.0 and T = 0.7.Iteration 0 is measured from the original CodeNet code, and the trajectories are plotted in Figure 11(a–c).
  • L.2 Experiments: At T = 0.7, SELF-REFINE produces more meaningful variables, more function units, and slightly more comments than human annotators, while T = 0.0 produces fewer meaningful variables and comments but more function units.These comparisons are reported for the final iteration on the 60-example subset.
  • M Dialogue Response Generation: Open-domain dialogue response generation is framed as a difficult task requiring coherent, engaging responses across diverse topics, addressed here through automatically generated feedback and iterative refinement.The paper uses SELF-REFINE to improve response quality rather than relying on one-step generation alone.
  • M.1 Modules: The dialogue pipeline initializes a response from the conversation context, then generates feedback assessing ten qualitative aspects of response quality.Feedback uses six in-context examples and often explains why a response scores poorly on a given aspect.
  • M.1 Modules: The iterate module refines outputs using the dialogue context, prior responses, and feedback so the revised response better matches the feedback.The module’s context-response-feedback-refinement example is shown in Figure 13.

M.2 Setup and Experiments … P Sentiment Reversal

Across dialogue, code optimization, math reasoning, and sentiment reversal, SELF-REFINE applies iterative feedback and refinement through few-shot prompting, with experiments evaluating its gains and task-specific implementations. The reported dialogue results show SELF-REFINE outperforming direct generation on both automatic and human evaluation, while other sections describe optimization, error correction, and sentiment-targeted refinement setups.

  • M.2 Setup and Experiments: The dialogue setup implements INIT, FEEDBACK, and ITERATE as few-shot prompts, provides three INIT in-context examples, and runs up to k = 3 iterations.INIT serves as the no-feedback baseline for the self-improvement loop.
  • M.2 Setup and Experiments: Dialogue evaluation uses the FED dataset’s eighteen fine-grained qualities with both automated metrics and human preference judgments.For human evaluation, annotators compare SELF-REFINE and INIT responses on 100 randomly selected test instances across ten response-quality aspects, with “both” available when neither is preferred.
  • M.2 Setup and Experiments: SELF-REFINE beats INIT by a wide margin on both automatic and human dialogue-response evaluation across three latest GPT model versions.The study uses GPT-3.5 as a strong direct-generation baseline and reports that SELF-REFINE produces more engaging outputs in manual analysis.
  • N Code Optimization: In code optimization, SELF-REFINE generates natural-language feedback on PIE’s optimization and feeds that feedback into REFINE for further improvement.PIE targets functionally correct programs by applying algorithmic modifications intended to improve runtime performance.
  • N Code Optimization: The code-optimization comparison reports SELF-REFINE outperforming both the direct method and a simplified feedback approach lacking introspective feedback.The accompanying table is described as a main-results and ablation analysis, highlighting the value of multi-faceted feedback.
  • O Math Reasoning: For GSM-8k math reasoning, SELF-REFINE represents solutions in Python and iteratively uses feedback to identify and correct errors in generated code.The generator produces an initial solution, FEEDBACK scans it for errors, and REFINE creates a revised solution; the correct label determines when the loop advances.
  • O Math Reasoning: Math-reasoning accuracy is evaluated as a function of SELF-REFINE iteration count on the GSM-8k benchmark.The paper presents these results in Figure 14 and uses label feedback to decide transitions within the iterative process.
  • P Sentiment Reversal: Sentiment reversal rewrites entire reviews to switch between positive and negative sentiment, using few-shot INIT, FEEDBACK, and ITERATE modules for up to k = 4 iterations.Iterations continue until the target sentiment is reached, making the task a long-form style-transfer setting rather than sentence-level transfer.

P.1 Details … S Prompts

The paper details evaluation procedures for sentiment reversal, an acronym-generation task with multifaceted feedback, and CommonGen-Hard constrained generation, while documenting prompts for all experimental tasks. These materials emphasize iterative refinement, chain-of-thought-style feedback, and task-specific evaluation criteria.

  • P.1 Details: Sentiment reversal is evaluated through automated and human preference rates for target-sentiment adherence and generation dramaticness.Automated evaluation uses few-shot GPT comparisons of which review is more positive or less boring, while target sentiment is also judged with the VADER classifier.
  • P.1 Details: Pin-pointed chain-of-thought-style feedback identifies words and phrases to alter, outperforming a “something is wrong” feedback ablation in sentiment evaluation.The ablation removes specific diagnostic guidance, whereas the full feedback explains how to reach the desired sentiment level.
  • Q Acronym Generation: Acronym Generation targets concise, memorable acronyms under tradeoffs involving length, pronunciation, and relevance, using a manually pruned dataset of 250 examples.The task’s feedback evaluates ease of pronunciation, ease of spelling, relation to the title, positive connotation, and familiarity, with reasoning generated before criterion scores and human evaluation of final quality.
  • Q Acronym Generation: The acronym feedback framework assesses each candidate across pronunciation, spelling, title relation, connotation, and recognizability.Because some criteria are difficult to quantify, the authors use few-shot demonstrations and chain-of-thought-style reasoning before assigning scores.
  • R Constrained Generation: CommonGen-Hard extends CommonGen from 3–5 to 20–30 concepts, testing coherent, grammatical generation and SELF-REFINE’s introspective iterative refinement.The increased concept load is intended to challenge commonsense reasoning and expose deficiencies in initial model outputs.
  • S Prompts: The prompt appendix provides generation, feedback, and refinement prompts for acronym generation, code tasks, constrained generation, dialogue, math reasoning, and sentiment reversal.The Base LLM uses input-output examples, feedback uses input-output-feedback triples, and refinement uses input-output-feedback-refined quadruples.
  • S Prompts: Task-specific prompts operationalize refinement through rubrics and examples, including ten desired dialogue-response traits and explicit concept-coverage checks for constrained generation.The documented prompts also include worked examples for acronym, math, and sentiment-reversal tasks.
Loading 2303.17651v2…