Source-linked AI summary
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, Rui Zhang
TL;DR
The paper asks when LLMs can correct their own mistakes despite mixed prior evidence. It critically surveys self-correction research, categorizes its research questions, and finds that reliable feedback is the central bottleneck, while external feedback and large-scale fine-tuning support success.
Problem
Prior studies provide mixed evidence about when LLMs can self-correct and often leave their research questions and evaluations insufficiently specified.
Method
The paper critically surveys prior work, categorizes self-correction research questions, specifies verification requirements, and provides an experiment-design checklist.
Results
The survey finds no reliable general-task evidence for self-correction using prompted LLM feedback, while reliable external feedback and large-scale fine-tuning support self-correction.
Takeaways & Limitations
Feedback generation is the bottleneck, so self-correction is most suitable when task verification is exceptionally easy or reliable external feedback is available.
Takeaways & Limitations
Some prior external-information studies may over-evaluate self-correction because the information could also improve initial response generation directly.
Abstract
from arXiv · showhide
Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback. However, there is still no consensus on the question of when LLMs can correct their own mistakes, as recent studies also report negative results. In this work, we critically survey broad papers and discuss the conditions required for successful self-correction. We first find that prior studies often do not define their research questions in detail and involve impractical frameworks or unfair evaluations that over-evaluate self-correction. To tackle these issues, we categorize research questions in self-correction research and provide a checklist for designing appropriate experiments. Our critical survey based on the newly categorized research questions shows that (1) no prior work demonstrates successful self-correction with feedback from prompted LLMs, except for studies in tasks that are exceptionally suited for self-correction, (2) self-correction works well in tasks that can use reliable external feedback, and (3) large-scale fine-tuning enables self-correction.
1 Introduction
Self-correction refines LLM responses during inference, but the conditions under which it reliably works remain unsettled. This survey categorizes research questions and finds success mainly with exceptionally suitable tasks, reliable external feedback, or large-scale fine-tuning.
- Self-correction and open questions: Self-correction refines LLM responses during inference using feedback from the model, external tools, external knowledge, or fine-tuning.The survey covers applications including arithmetic reasoning, code generation, and question answering.
- Survey approach: The survey categorizes self-correction research questions and proposes frameworks and a checklist for designing appropriate experiments.It argues that prior studies often leave their research questions implicit, leading to inadequate evaluations.
- Main findings: In general tasks, no prior work shows reliable successful self-correction with in-context learning, except for tasks with properties exceptionally favorable to correction.Examples include tasks whose responses are decomposable.
- Main findings: Self-correction works well when reliable external feedback is available, and large-scale fine-tuning enables self-correction.The survey identifies feedback generation as a central bottleneck for current LLMs.
- Evaluation gaps: It remains unclear whether self-correction outperforms other approaches because prior work often lacks sufficiently strong baselines.The survey identifies this comparison problem as a research question for final outputs.
- Scope: The survey primarily covers research published through May 2024, with later work not comprehensively analyzed.The survey was originally published in June 2024.
2 Self-Correction of LLMs
Self-correction refines LLM responses during inference using feedback that may come from the same model, external information, or fine-tuned models. Frameworks differ in feedback generation, refinement timing, model pairing, and the quality of initial responses.
- Self-correction refines LLM-generated responses during inference, possibly using external tools or knowledge.
- Frameworks: A common framework generates an initial response, produces feedback from the input and response, then generates a refined response using that feedback.
- Frameworks: Direct refinement improves responses without explicitly generating feedback.
- Frameworks: Post-hoc correction revises completed responses, whereas generation-time correction provides feedback on intermediate reasoning steps.
- Sources of Feedback: Feedback sources include intrinsic prompting, external tools or knowledge, oracle information, human feedback, stronger models, and fine-tuned models.
- Differences from Related Approaches: Self-consistency and generate-and-rank are treated as distinct strong baselines because they select among responses rather than refine them.
3 Research Questions
The survey separates self-correction research into distinct questions about intrinsic correction, externally assisted correction, and comparison with other methods. It argues that each question requires appropriately matched frameworks, initial responses, and evaluations.
- Research Questions: Prior studies often leave research questions underspecified, sometimes targeting different questions within one study without distinguishing them.
- Research Questions: The survey identifies three research questions: intrinsic correction, externally assisted correction, and whether final outputs outperform other methods.
- Research Questions: Best-possible initial responses are generated with best effort using information accessible to self-correction modules, including tools, knowledge, or fine-tuning.
- Requirements for Verifying RQs: RQ1 requires same-model refinement without additional information, while RQ2 and RQ3 may use external information.
- Requirements for Verifying RQs: RQ1 and RQ2 require best-possible initial responses, whereas RQ3 concerns final performance and does not require strong initial responses.
- Frameworks for Verifying RQs: The survey proposes categories of same-model self-correction aligned with different research questions and distinguishes realistic, fair, unfair, and unrealistic settings.
4 Self-Correction with Prompting
Under fair settings, prompted intrinsic self-correction generally does not reliably improve LLM responses, although decomposable tasks can make it effective.
- Intrinsic self-correction has failed to improve or has degraded performance across arithmetic reasoning, closed-book QA, code generation, planning, and graph coloring.
- Reliable feedback generation is the central bottleneck when LLMs are prompted to assess and revise their own responses.
- No major work demonstrates successful self-correction from prompted-LLM feedback in general tasks under fair settings.Prior evaluations often use oracle information or weak initial prompts that overstate self-correction gains.
- Self-correction can work in tasks with decomposable responses, where verifying individual answers is easier than generating the full response.CoVe illustrates this setting for questions requiring multiple answers, but many real-world tasks lack this property.
5 Self-Correction with External Information
External information can make self-correction effective when it supplies reliable feedback, but evaluations must distinguish improvement over best-possible initial responses from improved final performance. Fine-tuning also enables self-correction, typically with substantial training data.
- External Tools and Knowledge: Self-correction improves responses when reliable external tools or knowledge provide feedback.Examples include code interpreters, symbolic reasoners, and search engines used to validate initial responses.
- Evaluation Fairness: External information can be used unfairly when it improves correction but could also directly improve initial response generation.This distinction matters for evaluating improvement from best-possible initial responses rather than only final-output performance.
- Verifiable Tasks: Verifiable tasks support self-correction because response correctness can be checked easily without external information.Examples include constrained generation and Game of 24.
- Fine-tuning: Fine-tuning feedback or refinement models improves self-correction, with many supervised methods relying on more than 100K training instances.Some approaches use stronger models or synthetic data to construct feedback and refinement targets.
- Evaluation Fairness: Fine-tuning comparisons can be unfair when correction models receive stronger training than initial-response models.Such setups can evaluate final-output performance but not improvement from best-possible initial responses.
6 Strong Baselines
Because self-correction uses multiple LLM calls, claims about superior final outputs require comparisons with strong baselines that account for comparable computation. Relevant baselines include self-consistency and generate-and-rank methods.
- Baseline Requirements: Self-correction should be compared with strong baselines that use comparable additional LLM calls or computational cost.Many studies lack such comparisons, although some compare against self-consistency or pass@k.
- Self-Consistency: Self-consistency generates multiple responses and selects the majority-voted final answer.The approach has also been extended beyond reasoning tasks to text and code generation.
- Generate-and-Rank: Generate-and-rank methods generate multiple responses and select among them using verifiers such as self-evaluation, confidence, fine-tuned models, or external tools.
7 Summary of Our Analysis
The survey identifies feedback generation as self-correction’s central bottleneck and finds that success depends on task properties, reliable external feedback, or large training data for feedback generation.
- Summary of Our Analysis: Feedback generation is the bottleneck: LLMs can refine responses with reliable feedback but struggle to generate reliable feedback unaided.The survey recommends evaluating feedback quality directly, not only downstream refined-response performance.
- Intrinsic Self-Correction: Intrinsic self-correction is effective in tasks whose verification is much easier than the original task.Responses that are decomposable are one example of this favorable property.
- External Information: Self-correction with external information works for tasks with reliable external feedback tools.Code generation is given as an example.
- Fine-tuning: Fine-tuning supports self-correction across many tasks when large training data for feedback generation is available.Self-corrective learning and reinforcement learning are especially applicable when responses can be evaluated against ground-truth answers.
8 Checklist for Self-Correction Research
The proposed checklist addresses weaknesses in self-correction studies by requiring clear research questions and appropriate experimental reporting, while recommending direct feedback-quality evaluation.
- Checklist: The checklist requires clearly stating the target research question and self-correction framework category.
- Checklist: It provides requirements for appropriate experiments and recommended analyses, including direct evaluation of feedback quality and reporting negative results.
9 Differences from Other Survey
This survey focuses specifically on inference-time self-correction and offers a more detailed, critical analysis than broader surveys. It also extends evaluation beyond intrinsic reasoning tasks to in-context learning, external tools, and fine-tuning.
- Unlike broader surveys of self-correction, this work specifically examines inference-time self-correction.
- The survey provides a more detailed and critical analysis of prior self-correction research.
- Its coverage includes in-context learning, external tools, and fine-tuning.
10 Related Work of Self-Correction
Related work spans self-detection, human-written text editing, and self-training or self-improvement. These approaches overlap with self-correction but differ in their targets or training-time use of model-generated responses.
- Self-Detection: Self-detection uses language models, sometimes with external information, to identify mistakes across misinformation, faithfulness, harmful-content, and bias domains.
- Editing Human-Written Text: Language models have been used to edit human-written text for information updates, grammatical errors, factual errors, and code repairs.
- Self-Training or Self-Improvement: Self-training or self-improvement trains models using their own responses, including self-evaluation, self-correction, and self-generated reasoning data.
11 Future Directions
The survey identifies future directions in improving feedback, exploring tasks beyond reasoning, reducing fine-tuning data requirements, and improving pre-training for intrinsic self-correction. It also notes reinforcement-learning trends after the survey’s original coverage period.
- Improving Feedback: Improving feedback generation is a priority because in-context learning makes it difficult for LLMs to generate feedback on their own responses.The survey suggests exploring reference-free and point-wise LLM-based evaluation metrics.
- Unexplored Tasks: Intrinsic self-correction should be studied in tasks beyond the reasoning domains that dominate current research.The difficulty of self-evaluation differs across tasks, and some unexplored tasks may be especially suitable.
- Fine-tuning on Small Training Data: Future work should investigate self-correction with smaller fine-tuning datasets because feedback-generation fine-tuning often relies on large-scale human annotations.
- Pre-training for Improving Self-Correction: Pre-training strategies may improve intrinsic self-correction because large-scale fine-tuning currently improves this capability.
- The survey mainly covers work published before May 2024, while later developments are discussed only as emerging trends.
- Reinforcement learning has emerged as a self-correction trend, including training models to recognize mistakes and refine their reasoning.
- The survey aims to identify the conditions under which LLMs can correct their own mistakes.
- It responds to unclear research questions and inappropriate experiments by categorizing research questions and frameworks and providing an experimental checklist.