PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Cannot Self-Correct Reasoning Yet

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou

arXiv:2310.01798v2cs.CLcs.AI

TL;DR

High-quality external feedback is often unavailable, raising the question of whether LLMs can correct their own reasoning. The paper examines intrinsic self-correction and finds that current LLMs cannot improve reasoning through it.

  • Problem

    Whether LLMs can judge and correct their own reasoning matters because high-quality external feedback, such as answer correctness, is often unavailable.

    Source: §2 · p. 2

  • Method

    The paper examines LLM reasoning self-correction in the intrinsic setting, without external or human feedback.

    Source: §2 · p. 2

  • Results

    Current LLMs cannot improve their reasoning performance through intrinsic self-correction.

    Source: §6 · p. 8

  • Takeaways & Limitations

    When valid external feedback is available, the authors argue that it is beneficial to use it to enhance model performance.

    Source: §6 · p. 8

    The work evaluates reasoning, and the authors note that self-correction strategies may enhance LLM performance in other domains.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications. Nevertheless, concerns persist regarding the accuracy and appropriateness of their generated content. A contemporary methodology, self-correction, has been proposed as a remedy to these issues. Building upon this premise, this paper critically examines the role and efficacy of self-correction within LLMs, shedding light on its true potential and limitations. Central to our investigation is the notion of intrinsic self-correction, whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback. In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction. Drawing from these insights, we offer suggestions for future research and practical applications in this field.

1 INTRODUCTION

The paper defines intrinsic self-correction as revising an answer using only the model’s own capabilities, and examines whether it can improve LLM reasoning without external feedback.

1 INTRODUCTION

  • Without external feedback, LLMs struggle to self-correct reasoning, and performance often deteriorates after correction.Intrinsic self-correction is important because high-quality external feedback is often unavailable in real-world applications.Source: §1 · pp. 1–2
  • Reported self-correction improvements in Kim et al. (2023) and Shinn et al. (2023) depend on oracle labels and disappear without them.Source: §1 · p. 2
  • With an equivalent number of responses, multi-agent debate is no more effective than self-consistency for improving reasoning.Source: §1 · p. 2
  • When initial prompts are suboptimal, self-correction’s apparent gains come from more informative feedback; integrating that information initially works better, while correction decreases performance.Source: §1 · p. 2
  • The paper calls for future research into methods that can genuinely correct reasoning.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The section frames self-correction as a model recognizing and revising its own incorrect output, while distinguishing internal feedback from externally supplied feedback. It defines the paper’s focus as intrinsic reasoning self-correction without external or human feedback.

  • Self-correction asks whether models can recognize output errors and refine their answers; an ideal case revisits and corrects a mistaken reasoning step.The discussion cites Bai et al. (2022), Madaan et al. (2023), and Welleck et al. (2023, inter alia).Source: §2 · p. 2
  • The literature distinguishes feedback generated internally by the model from external feedback supplied by people, other models, tools, or knowledge sources.This distinction is pivotal because the literature uses “self-correction” ambiguously (Pan et al., 2023).Source: §2 · p. 2
  • Kim et al. (2023) and Shinn et al. (2023) use oracle correctness labels, whereas this work studies reasoning self-correction without external or human feedback.The paper terms this setting intrinsic self-correction, noting that high-quality external feedback is often unavailable.Source: §2 · p. 2
  • The paper previews three weaknesses: techniques can reduce reasoning performance without oracle labels, underperform equal-response baselines, and fare worse with informative initial-response prompts.Table 1 summarizes issues in prior LLM self-correction evaluation setups.Source: §2 · p. 2, Table 1 · p. 3

3 LLMS CANNOT SELF-CORRECT REASONING INTRINSICALLY

With oracle labels, self-correction improves benchmark results, but without labels accuracy drops across models and benchmarks, and feedback-prompt variations do not reverse the decline.

3.1 EXPERIMENTAL SETUP

  • Experiments span GSM8K, CommonSenseQA, and HotpotQA, testing GPT-3.5, GPT-4, GPT-4-Turbo, and Llama-2 for up to two self-correction rounds.The process generates an initial answer, requests feedback, then answers again; GPT-3.5 uses the full evaluation sets, while other models use random samples.Source: §3.1 · p. 3

3.2 RESULTS

  • Oracle-label self-correction significantly improves results, but these gains indicate oracle performance rather than intrinsic self-correction ability.The oracle uses ground-truth answers to stop correcting once an answer is correct, preventing correct answers from being changed to incorrect ones.Source: Table 2 · p. 4, §3.2 · p. 4, §3.3 · p. 6
  • Accuracy drops for all models across all benchmarks after intrinsic self-correction.The reported accuracy results cover GPT-3.5 and GPT-4, as well as GPT-4-Turbo and Llama-2.Source: Table 3 · p. 4, §3.2 · p. 4, Table 4 · p. 5

3.3 WHY DOES THE PERFORMANCE NOT INCREASE, BUT INSTEAD DECREASE?

  • Without oracle labels, LLMs must decide whether to keep or revise their answers, but they often cannot judge reasoning correctness reliably.The authors suggest that feedback adds a prompt that can bias a model away from an answer optimal for its initial prompt and decoding algorithm.Source: §3.2 · p. 4, §3.3 · pp. 4, 6
  • 74.7% of the time, GPT-3.5 retains its GSM8K answer; in other GSM8K cases and on CommonSenseQA, correct answers are often changed to incorrect ones.GPT-4 and GPT-4-Turbo are more likely than GPT-3.5 and Llama-2 to retain initial answers, possibly due to confidence or robustness; HotpotQA is omitted from this analysis because its sample is small.Source: §3.3 · pp. 4–5, Figure 1 · p. 6
  • Alternative self-correction prompts also consistently decrease performance without oracle labels.The experiments tested different feedback prompts for GPT-4-Turbo and Llama-2.Source: §3.2 · p. 4, Table 5 · p. 5, Table 6 · p. 5

4 MULTI-AGENT DEBATE DOES NOT OUTPERFORM SELF-CONSISTENCY

On GSM8K, multi-agent debate and self-consistency both improve over standard prompting, but debate underperforms majority-vote self-consistency when given an equivalent number of responses. The paper attributes debate’s gains to consistency across generations rather than self-correction.

4 MULTI-AGENT DEBATE DOES NOT OUTPERFORM SELF-CONSISTENCY

  • The GSM8K replication used Du et al. (2023)’s prompt with gpt-3.5-turbo-0301, three agents, and two debate rounds, testing the complete test set.The paper also reports self-consistency, which generates multiple responses and uses majority voting, following Wang et al. (2022).Source: §4 · p. 7
  • Both methods improve over standard prompting, but debate underperforms majority-vote self-consistency with an equivalent number of responses.With three responses, debate is only slightly better than self-consistency using the same number of agents.Source: §4 · p. 7
  • Debate’s gains reflect consistency across generations, not self-correction; its effect should be separated from selection among multiple generations.The distinction from self-consistency is whether voting is model-driven or based purely on counts.Source: §4 · p. 7

5 PROMPT DESIGN ISSUES IN SELF-CORRECTION EVALUATION

This section argues that self-correction evaluations depend on properly designed prompts for initial responses: requirements that can be stated upfront should not be introduced only in feedback. In the Constrained Generation example, explicitly requiring all concepts in the initial prompt outperforms prior self-correction results, while adding self-correction to that stronger prompt lowers performance.

5 PROMPT DESIGN ISSUES IN SELF-CORRECTION EVALUATION

  • Although reasoning performance decreases with all evaluated feedback-prompt types, outcomes vary with the feedback prompt, making prompt design important to fair evaluation.The section emphasizes that initial-response prompts affect how fairly self-correction improvement can be measured.Source: §5 · p. 7
  • When requirements are easy to specify, they should be included explicitly in the initial-response prompt; otherwise, gains may reflect more detailed feedback instructions rather than self-correction.Examples include requiring certain words, efficient code, or positive sentiment.Source: §5 · p. 8
  • The prompt requiring all concepts in initial responses significantly outperforms Madaan et al. (2023)’s self-correction results, while adding their self-correction prompt to those responses decreases performance.The Constrained Generation task asks for coherent sentences using all 20–30 input concepts; the original prompt did not clearly require including them.Source: §5 · p. 8

6 CONCLUSION AND DISCUSSION

Current LLMs struggle to improve reasoning through intrinsic self-correction without external feedback; the discussion recommends using available feedback and evaluating correction methods fairly.

6 CONCLUSION AND DISCUSSION

  • Current LLMs struggle to improve reasoning through intrinsic self-correction without external feedback, making inherent error recognition and repair an overly optimistic expectation.The authors urge researchers to recognize self-correction’s potential and limitations when developing future methods.Source: §6 · p. 8

Leveraging external feedback for correction

Evaluating self-correction against baselines with comparable inference costs

  • Because self-correction adds inference cost, future methods should report detailed costs and compare against strong multi-response baselines such as self-consistency.With the same number of responses, multi-agent debate’s final response was inferior to self-consistency (Wang et al., 2022); the authors also suggest alignment that raises the probability of decoding an optimal solution, reducing the need for multiple generations.Source: §6 · pp. 8–9

Putting equal efforts into prompt design

  • Prompt design should give equal effort to initial response generation and self-correction, with the complete task description included in the initial prompt.Leaving part of the task description for the feedback prompt can make measured improvements misleading.Source: §6 · p. 9

7 LIMITATIONS AND BROADER IMPACT

The paper evaluates self-correction for reasoning, while noting that its potential may differ in other domains. It also highlights ambiguity about external feedback in prior work and calls for research that genuinely improves reasoning.

7 LIMITATIONS AND BROADER IMPACT

  • The study evaluates reasoning, so it does not establish whether self-correction can improve other domains, where prior work reports style alignment and safety benefits (Bai et al., 2022; Ganguli et al., 2023; Madaan et al., 2023).The authors distinguish aligning responses with preferences from identifying reasoning errors.Source: §7 · p. 9
  • LLMs may assess whether a response is inappropriate (Ganguli et al., 2023) while struggling to identify errors in their reasoning.Source: §7 · p. 9
  • Prior work likewise finds self-correction weaker without external feedback (Gou et al., 2023; Zhou et al., 2023a) and vulnerable to misleading feedback (Wang et al., 2023a).Source: §7 · p. 9
  • The paper urges critical scrutiny because some studies obscure label usage or fail to clarify their use of external feedback, and calls for research that genuinely improves reasoning.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the models and documents model versions or experiment access times, prompt sources, and the location of designed prompts.

REPRODUCIBILITY STATEMENT

  • Experiments use GPT-3.5, GPT-4, and Llama-2, documenting specific kernels or experiment access times and providing prior-work prompts or designed prompts in Appendix A.GPT-3.5 and GPT-4 are accessed through the public API; Llama-2 is open source.Source: REPRODUCIBILITY STATEMENT · p. 9

A PROMPTS AND EXAMPLE OUTPUTS

The examples show self-correction producing different outcomes: correcting an incorrect answer, changing a correct answer to an incorrect one, or leaving an answer unchanged. They also illustrate constrained-generation prompts with different ways of expressing the concept constraint.

  • Figure 3 shows self-correction changing an incorrect GSM8K answer to a correct one.Source: Figure 3 · p. 13
  • Figures 4 and 6 show self-correction changing correct answers to incorrect ones on GSM8K and CommonSenseQA, respectively.Source: Figure 4 · p. 14, Figure 6 · p. 16
  • Figure 5 shows a GSM8K example where self-correction does not change the answer.Source: Figure 5 · p. 15
  • In the Madaan et al. (2023) constrained-generation example, the concept constraint is implicit in feedback and refinement prompts, but absent from the initial-response prompt; another example uses an authors’ casually written prompt.Source: Figure 7 · p. 17, Figure 8 · p. 17

Related papers

Loading 2310.01798v2…