Source-linked AI summary
Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, William Yang Wang
TL;DR
LLMs perform well but still produce hallucinations, unfaithful reasoning, harmful content, and other errors that affect trust and deployment. This paper surveys automated-feedback self-correction, organizing methods by correction timing, feedback, and refinement strategy, and covering applications and future challenges. Its synthesis spans training-time, generation-time, and post-hoc correction, while identifying limited theoretical justification, evaluation, and task coverage as open issues.
Problem
LLMs can produce hallucinations, unfaithful reasoning, harmful content, and other undesired behaviors that undermine trust and hinder real-world applications.
Method
The paper comprehensively surveys and taxonomizes automated-feedback self-correction across correction timing, feedback sources and formats, refinement strategies, applications, and future directions.
Results
The survey organizes self-correction methods into training-time, generation-time, and post-hoc correction and summarizes applications including factual correction, reasoning, and code generation.
Takeaways & Limitations
Automated-feedback self-correction is presented as an emerging approach for improving LLM outputs with minimal human feedback across diverse applications.
Takeaways & Limitations
Current research lacks robust quantitative metrics, unified comparative evaluation, and theoretical justifications for self-correction, while broader reasoning tasks remain under-explored.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. However, their efficacy is undermined by undesired and inconsistent behaviors, including hallucination, unfaithful reasoning, and toxic content. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. Techniques leveraging automated feedback -- either produced by the LLM itself or some external system -- are of particular interest as they are a promising way to make LLM-based solutions more practical and deployable with minimal human feedback. This paper presents a comprehensive review of this emerging class of techniques. We analyze and taxonomize a wide array of recent work utilizing these strategies, including training-time, generation-time, and post-hoc correction. We also summarize the major applications of this strategy and conclude by discussing future directions and challenges.
1 Introduction
LLMs achieve strong results across many NLP tasks but can hallucinate, reason unfaithfully, generate harmful content, and violate constraints, undermining trust and real-world use. This survey examines self-correction with automated feedback as a way to reduce human intervention.
- LLMs show strong performance across diverse NLP benchmarks and abilities in understanding, generation, and reasoning.
- Hallucinations, unfaithful behavior, harmful content, and failures to follow rules undermine trust in LLMs and hinder real-world applications.
- Human-feedback methods can be costly because they require manual labor and may lack real-time capabilities.
- Self-correcting LLMs iteratively learn from automatically generated feedback signals to understand action consequences and adapt their behavior.
- Automated feedback may come from the LLM itself, separately trained models, external tools, or external knowledge sources, with strategies including self-training and generate-then-rank.
- The survey taxonomizes automated-feedback correction and discusses training-time, generation-time, post-hoc, application, and future-work perspectives.
2 A Taxonomy for Correcting LLMs with Automated Feedback
The survey frames automated-feedback correction as an interaction among a language model, critic model, and refine model, then classifies methods across five dimensions. It organizes corrected errors, feedback sources and formats, correction timing, and refinement strategies.
- The taxonomy classifies existing work by what gets corrected, feedback source, feedback format, when feedback is used, and how the model is corrected.
- 2.1 Conceptual Framework: The language model maps an input to an initial output that may contain hallucinations or incorrect reasoning.
- 2.1 Conceptual Framework: A critic model analyzes an input and output to produce feedback, such as a scalar score or natural-language critique.
- 2.1 Conceptual Framework: A refine model uses the input, output, and feedback to produce a revised output or directly update the language model through fine-tuning or reinforcement learning.
- 2.2 What to Correct?: Automated feedback targets hallucination, unfaithful reasoning, toxic or harmful content, and flawed code through verification, external guidance, process feedback, alignment, or execution.
- 2.3 Source and Format of Feedback: Feedback can be self-generated or external, and can take scalar-value or natural-language form.
- 2.5 When to correct the model with feedback?: Training-time correction optimizes model parameters before deployment, but may require fine-tuning access, available training feedback, and optimizable signals.
3 Training-Time Correction
Training-time correction modifies model parameters using human, learned, or automated feedback. The survey distinguishes direct human optimization, reward modeling, and self-training, while emphasizing automated feedback as its primary focus.
- Training-time correction uses feedback to modify model parameters during training, with human feedback, reward-model feedback, and automated feedback as three typical strategies.
- Learning from Human Feedback: Direct human-feedback optimization generates candidate outputs, collects human evaluations or refinements, and optimizes the model on the resulting data.
- Learning from Human Feedback: Reward modeling trains a model to emulate human feedback, providing consistent real-time signals without constant human involvement.
- Learning with Automated Feedback: The survey distinguishes automated feedback from human assessment and focuses on extrinsic feedback from external metrics or models and intrinsic feedback from the language model itself.
- Learning with Automated Feedback: External metric guidance commonly uses non-differentiable training because metric signals are discrete, incorporating metric scores into optimization objectives.
- Learning with Automated Feedback: Self-training bootstraps model outputs by selecting correct rationales, majority-voted reasoning paths, or critique-revision data for further fine-tuning.
4 Generation-Time Correction
Generation-time correction improves outputs during decoding by selecting among complete candidates or evaluating and steering individual reasoning steps. The survey describes feedback sources and search procedures underlying these strategies, while noting important limitations of whole-output ranking.
- Generate-then-Rank: Generate-then-Rank samples multiple candidate outputs and uses a critic model to select the best candidate.The critic maps the input and candidate generations to the best output.
- Generate-then-Rank: Generate-then-Rank has been applied to reasoning tasks by scoring candidate reasoning paths and selecting among them through ranking or voting.In Chain-of-Thought settings, each candidate contains an explanation and predicted answer.
- Generate-then-Rank: Critics for candidate selection include trained verifiers, self-verification models, and knowledge-base retrieval for assessing reasoning faithfulness.Examples include DIVERSE, Self-Verification, RR, and LEVER.
- Generate-then-Rank: Generate-then-Rank cannot pinpoint exact error locations, assess long outputs easily, or control generation before completion.The method waits until the entire output has been generated before correction.
- Feedback-Guided Decoding: Feedback-guided decoding evaluates individual reasoning steps and uses search algorithms to steer generation toward higher-quality continuations.The critic scores each step conditioned on the input and preceding steps.
- Feedback-Guided Decoding: Step-level feedback can come from human annotations, synthetic-data verifiers, external metrics, external knowledge, or self-evaluation.These sources differ in how the critic model is obtained.
5 Post-hoc Correction
Post-hoc correction intervenes after an entire output is generated, enabling iterative refinement with self-generated, external, or multi-agent feedback. The survey covers specialized tools, knowledge sources, trained critics, and debates among multiple language models.
- Overview: Post-hoc correction addresses holistic evaluation settings by intervening after the complete output and supporting detailed natural-language feedback.Feedback may identify exact errors or suggest general writing improvements.
- Self-Correction: Self-correction uses one language model to generate an output, critique it, and refine it iteratively until an acceptable result or iteration limit is reached.Self-Refine implements these roles with one pretrained LLM and different prompts.
- Self-Correction: Reflexion extends self-correction with long-term memory for storing prior feedback and outputs, helping avoid repeated mistakes.It also incorporates scalar-valued and other feedback forms.
- Self-Correction: Self-correction depends on powerful models, because smaller open-source models may struggle to refine outputs even when correct feedback is provided.The survey notes explicit training as a possible response.
- External Feedback: External tools provide specialized feedback for coding, logical reasoning, factuality, and other correction tasks.Examples include program executors, symbolic solvers, retrievers, search engines, calculators, and trained models.
- Multi-Agent Debate: Multi-agent debate uses multiple LLM instances that propose and debate responses over several rounds before producing a common answer.LM vs LM applies the approach to factual-error detection, while other work studies bargaining interactions.
6 Applications
Automated correction is applied across factuality, reasoning, code generation, open-ended generation, translation, and summarization. The survey highlights strong activity in arithmetic reasoning and code repair, while identifying limitations in feedback quality and task coverage.
- Factual Correction: Automated correction supports factual-error detection and correction, with retrieved facts providing evidence when external tools are available.The survey describes self-correction as foundational to fact-correction and fact-checking systems.
- Reasoning Tasks: Reasoning correction is difficult because most reasoning tasks lack reliable references for checking outputs.Multi-hop question answering may require both factual and reasoning correction.
- Reasoning Tasks: Reasoning-error detection uses debate, self-refinement, and decoding algorithms such as beam search to guide reasoning toward a correct direction.The LLM can be prompted to check or correct errors in step-by-step reasoning.
- Reasoning Tasks: Self-correction is well studied for arithmetic reasoning because intermediate steps are comparatively easy to verify.Deductive reasoning has received some attention, whereas inductive and abductive reasoning remain under-explored.
- Code Synthesis: Code correction uses compiler warnings, runtime errors, test outputs, and program explanations as feedback for revising generated programs.These signals are fed back into the LLM to guide correction.
- Code Synthesis: Self-repair in code is bottlenecked by feedback quality, with substantial improvements observed only when feedback came from expert programmers or GPT-4.This raises questions about whether the ability is emergent and how smaller models might acquire it.
- Open-ended Generation: Open-ended generation uses iterative refinement for toxic outputs, narrative quality, and dialogue responses, often relying on detailed natural-language feedback.Subjective evaluation motivates the use of richer feedback.
- Machine Translation: Machine translation correction includes automatic post-editing from human or synthetic data, alongside newer in-context learning, training-time, and decoding-time methods.These approaches target translation errors after or during generation.
7 Research Gaps and Future Directions
The survey identifies unresolved theoretical, evaluative, continual-learning, model-editing, and multimodal challenges for automated self-correction in LLMs. It proposes directions spanning better metrics and benchmarks, continual adaptation, model editing, and multimodal expansion.
- 7.1 Theoretical Justifications: Theoretical justifications for LLM self-analysis and self-improvement remain underdeveloped, motivating research into the principles underlying self-correction.The authors argue that theoretical study could provide a more transparent understanding of this ability.
- 7.2 Measuring the Ability of Self-Correction: Current evidence is primarily empirical, while unified quantitative metrics and comparative evaluations of self-correction strategies remain missing.The proposed evaluation dimensions include effectiveness, applicability, complexity, upper-bound limits, task complexity, initial error, and post-correction quality.
- 7.2 Measuring the Ability of Self-Correction: Diagnostic benchmarks could enable more standardized and objective evaluation of LLMs and self-correction strategies.The survey presents benchmark construction as a promising research direction for developing more accurate and efficient models.
- 7.3 Continual Self-Improvement: Continual self-correction could involve continuously evaluating outputs, learning from errors, updating knowledge, and adapting decision-making strategies.This direction is linked to continual learning and adaptation to novel environments and contexts.
- 7.3 Continual Self-Improvement: Continual self-improvement remains under-explored because existing self-training studies typically examine one-time correction and particular improvements, leaving continual robustness and stability uncertain.A central open question is whether correcting one behavior could unintentionally alter a previously corrected behavior.
- 7.3 Continual Self-Improvement: Combining post-hoc and training-time correction is proposed as a way to balance immediate, lower-cost intervention with more fundamental behavioral change.The survey suggests using post-hoc correction to collect training data such as frequent mistakes and their corrections.
- 7.4 Self-Correction with Model Editing: Model editing offers fine-grained corrections without extensive retraining, while analyzing edit impacts may clarify self-correction mechanisms.Methods that reduce editing side effects could help prevent new errors while resolving existing ones.
- 7.5 Multimodal Self-Correction: Self-correction research is expanding beyond text toward multimodal LLMs and image, audio, and video settings, although only a handful of studies have explored these areas.Examples include progressive image self-refinement, vision-and-language navigation, and iterative revision of adversarial prompts for text-to-image generation.
8 Conclusion
The paper surveys self-correcting LLMs with automated feedback, organizing methods into training-time, generation-time, and post-hoc correction. It also reviews applications and future challenges, and provides a continually updated reading list for the field.
- 8 Conclusion: The survey comprehensively categorizes automated-feedback self-correction into training-time, generation-time, and post-hoc strategies.It analyzes these strategies across major application areas and outlines associated future directions and challenges.
- 8 Conclusion: The paper covers applications including factual-error correction, reasoning enhancement, and code-generation improvement.Its stated goal is to serve as a resource for readers studying this rapidly evolving domain.
- 8 Conclusion: A continually updated GitHub reading list accompanies the survey to support readers interested in self-correction research.The repository is presented as an aid to the survey’s resource-building effort.