Source-linked AI summary
Large Language Models Cannot Self-Correct Reasoning Yet
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou
TL;DR
The paper asks whether LLMs can improve reasoning through self-correction without external feedback, addressing uncertainty about the method’s efficacy. It evaluates intrinsic self-correction and related evaluation setups, finding that reasoning performance generally fails to improve and can deteriorate. The authors therefore distinguish intrinsic correction from cases where valid external feedback can support improvement.
Problem
The paper addresses whether LLMs can reliably correct reasoning errors without external feedback, amid concerns about their accuracy and reasoning capabilities.
Method
The paper evaluates existing self-correction techniques for reasoning, including intrinsic correction, equivalent-response baselines, and alternative prompt designs.
Results
Current LLMs struggle to improve reasoning through intrinsic self-correction; performance can deteriorate, while multi-agent debate is inferior to self-consistency at equal response counts.
Takeaways & Limitations
Self-correction should be interpreted critically, with valid external feedback leveraged when available to improve model performance.
Takeaways & Limitations
The evaluation focuses on reasoning, so self-correction strategies may enhance performance in other domains such as preference alignment, style, or safety.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content. A contemporary methodology, self-correction, has been proposed as a remedy to these issues. Building upon this premise, this paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations. Central to our investigation is the notion of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback. In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction. Drawing from these insights, we offer suggestions for future research and practical applications in this field.
1 INTRODUCTION
The paper examines whether LLMs can intrinsically self-correct reasoning without external feedback. Across its evaluation, self-correction often fails to improve performance and can reflect weaknesses in evaluation design.
- LLM self-correction is proposed as a way to address concerns about accuracy, reasoning, and safety in generated content.
- The paper investigates whether LLMs can recognize and correct their own reasoning errors without relying on external feedback.
- Intrinsic self-correction means revising an initial response using only the model’s inherent capabilities, without external feedback.The setting is important because high-quality external feedback is often unavailable in real-world applications.
- When oracle labels are absent, prior reported self-correction improvements can disappear, and performance may even deteriorate after correction.
- With equivalent inference costs, multi-agent debate is no more effective than self-consistency despite using multiple model responses.This comparison addresses the additional responses required by self-correction.
- Some reported gains arise from sub-optimal initial prompts, since integrating more informative feedback into the original instruction can perform better than self-correction.In these cases, self-correction decreases performance.
- The paper uses these findings to encourage research on methods that can genuinely correct reasoning.
2 BACKGROUND AND RELATED WORK
This section distinguishes intrinsic from externally assisted self-correction and frames the paper’s evaluation of reasoning-focused techniques. It also highlights that prior evaluation setups may obscure the actual contribution of self-correction.
- Self-correction refers to recognizing whether an output is correct and producing a refined answer, but its definition varies across the literature.
- Internal feedback comes from the model itself, whereas external feedback can come from humans, other models, tools, or knowledge sources.
- The paper focuses on reasoning because effective self-correction requires judging answer correctness, which is often unavailable from high-quality external feedback in practice.
- The authors call self-correction without external or human feedback intrinsic self-correction and use that meaning unless oracle feedback is explicitly specified.
- The paper evaluates existing techniques across settings without oracle labels, with equal response counts, and with informative initial prompts.It summarizes corresponding evaluation issues in Table 1.
3 LLMS CANNOT SELF-CORRECT REASONING INTRINSICALLY
The evaluation compares oracle-label and intrinsic self-correction across reasoning benchmarks and models. Without oracle labels, self-correction consistently reduces accuracy, with answer changes often converting correct responses into incorrect ones.
- 3.1 EXPERIMENTAL SETUP: The evaluation uses GSM8K, CommonSenseQA, and HotpotQA, following prior self-correction benchmarks and testing multiple LLMs.GSM8K contains 1,319 test problems, CommonSenseQA uses 1,221 development questions, and HotpotQA uses 100 closed-book questions.
- 3.1 EXPERIMENTAL SETUP: The three-step procedure generates an initial answer, asks the model to review it and provide feedback, then answers the original question again using that feedback.
- 3.2 RESULTS: Oracle-label self-correction shows significant improvements, but it uses ground-truth answers to decide whether further correction is needed.If an answer is already correct, the correction loop stops.
- 3.2 RESULTS: Oracle-label results are only indicative of oracle performance because knowing the ground truth is counter-intuitive for solving reasoning problems.
- 3.3 WHY DOES THE PERFORMANCE NOT INCREASE, BUT INSTEAD DECREASE?: Without oracle labels, accuracies of all evaluated models drop across all benchmarks after self-correction, despite testing alternative feedback prompts.The models must independently decide whether to retain their previous answers.
- 3.3 WHY DOES THE PERFORMANCE NOT INCREASE, BUT INSTEAD DECREASE?: After two correction rounds, GPT-3.5 retains its initial GSM8K answer 74.7% of the time, while remaining cases more often change correct answers into incorrect ones than the reverse.The analysis attributes the pattern to difficulty judging the correctness of reasoning; CommonSenseQA answer changes can also be biased by relevant-looking false options.
4 MULTI-AGENT DEBATE DOES NOT OUTPERFORM SELF-CONSISTENCY
The paper finds that multi-agent debate offers little advantage over self-consistency at comparable inference cost. Its gains appear to reflect consistency-based selection among multiple generations rather than self-correction.
- With three responses, multi-agent debate is only slightly better than self-consistency using the same number of agents.
- At an equivalent number of responses, multi-agent debate significantly underperforms simple self-consistency with majority voting.
- The authors argue that multi-agent debate is better understood as consistency across multiple generations, differing from self-consistency mainly in its voting mechanism.
5 PROMPT DESIGN ISSUES IN SELF-CORRECTION EVALUATION
The evaluation shows that self-correction gains can result from incomplete initial prompts rather than correction itself. Fair comparisons therefore require equally informative instructions at both stages.
- Self-correction can appear beneficial when feedback prompts contain task requirements omitted from the initial-response prompt.
- In the constrained-generation example, the original prompt did not clearly require including all 20–30 input concepts.
- Adding “ALL” concept coverage to the initial prompt significantly outperformed Madaan et al.’s post-correction results, while applying their correction prompt afterward decreased performance.
6 CONCLUSION AND DISCUSSION
The paper concludes that current LLMs generally cannot improve reasoning through intrinsic self-correction and urges careful evaluation of claimed gains. It recommends external verification when valid feedback exists and fair comparisons using cost-matched baselines and balanced prompts.
- Current LLMs struggle to self-correct reasoning without external feedback, making inherent recognition and repair of reasoning mistakes overly optimistic so far.
- External feedback from code execution, tools, or trained verifiers can help LLMs verify and correct responses when valid feedback is available.
- Self-correction methods should be evaluated against strong multiple-response baselines with comparable inference costs.The paper notes that multi-agent debate is inferior to self-consistency at the same response count.
- Prompt design should provide a complete task description in the initial-response prompt and receive comparable effort to the correction prompt.
7 LIMITATIONS AND BROADER IMPACT
The evaluation focuses on reasoning, so its conclusions may not extend to all self-correction uses. The discussion also emphasizes that external feedback can materially affect self-correction performance and that misleading or obscured feedback practices create ambiguity.
- Because the work evaluates reasoning, self-correction strategies may still improve performance in other domains such as response style or safety alignment.
- LLMs may evaluate inappropriate responses more reliably than they identify errors in their reasoning.
- Self-correction performance becomes significantly weaker without external feedback and can be biased by misleading feedback.
- Ambiguity in the literature can arise when label usage or external feedback is not clearly disclosed.
REPRODUCIBILITY STATEMENT
The experiments use GPT-3.5, GPT-4, and Llama-2, with model access and prompt details documented to support reproducibility.
- Experiments use GPT-3.5 and GPT-4 through the public API, alongside the open-source Llama-2 model.
- The authors specify model kernels or experiment access times for reproducibility.
- They use prompts from prior work when possible and provide exact designed prompts in Appendix A.
A PROMPTS AND EXAMPLE OUTPUTS
The examples illustrate prompts and outputs across math reasoning, commonsense question answering, and constrained generation. They include cases where self-correction improves, harms, or leaves answers unchanged.
- Self-correction examples: A GSM8K example shows self-correction changing an incorrect answer to a correct one.
- Math reasoning: The yogurt problem output demonstrates step-by-step arithmetic and concludes with the requested boxed numerical answer.The response computes 0.5 sets per day, $2.50 per day, and $75.00 over 30 days.
- Self-correction examples: A GSM8K example shows self-correction changing a correct answer to an incorrect one.
- Self-correction examples: A GSM8K example shows self-correction leaving the answer unchanged.
- Self-correction examples: A CommonSenseQA example shows self-correction changing a correct answer to an incorrect one.
- Constrained generation: The constrained-generation examples require a paragraph containing all listed concepts, with feedback implicitly identifying missing concepts in one prompt.One example uses a prompt from Madaan et al. (2023), while another uses a casually written author prompt.