PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models have Intrinsic Self-Correction Ability

Dancheng Liu, Amir Nassereldine, Ziming Yang, Chenhui Xu, Yuting Hu, Jiajie Li, Utkarsh Kumar, Changjae Lee, Ruiyang Qin, Yiyu Shi, Jinjun Xiong

arXiv:2406.15673v2cs.CLcs.AI

TL;DR

The paper examines whether LLMs can self-correct using only their own knowledge, a question challenged by recent critiques. Through theoretical analyses and experiments, it argues that intrinsic self-correction is exhibited across LLMs when prompts are fair and temperature is zero.

  • Problem

    Recent critiques question whether LLMs have intrinsic self-correction ability, despite earlier reports of improved performance with self-correction.

    Source: §1 · p. 1

  • Method

    The paper combines theoretical analyses and quantitative experiments to study intrinsic self-correction and its conditions.

    Source: §1 · p. 2

  • Results

    LLMs exhibit intrinsic self-correction when prompts are fair and temperature is zero, while higher temperature and biased prompts degrade accuracy.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings offer a perspective on intrinsic self-correction as analogous to self-verification and chain-of-thought prompting.

    Source: §1 · p. 2, §7 · p. 10

    Evaluations cover four models and six datasets, with Sections 4 and 5 focusing mainly on a subset of models and CommonSense QA.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have attracted significant attention for their exceptional abilities in various natural language processing tasks, but they suffer from hallucinations that will cause performance degradation. One promising solution to improve the LLMs' performance is to ask LLMs to revise their answer after generation, a technique known as self-correction. Among the two types of self-correction, intrinsic self-correction is considered a promising direction because it does not utilize external knowledge. However, recent works doubt the validity of LLM's ability to conduct intrinsic self-correction. In this paper, we present a novel perspective on the intrinsic self-correction capabilities of LLMs through theoretical analyses and empirical experiments. In addition, we identify two critical factors for successful self-correction: zero temperature and fair prompts. Leveraging these factors, we demonstrate that intrinsic self-correction ability is exhibited across multiple existing LLMs. Our findings offer insights into the fundamental theories underlying the self-correction behavior of LLMs and remark on the importance of unbiased prompts and zero temperature settings in harnessing their full potential.

1 INTRODUCTION

The introduction frames whether LLMs can self-correct using only their own knowledge as an open question, amid disagreement about intrinsic self-correction. The paper argues that this ability is present, with zero temperature and fair prompts identified as important conditions.

1 INTRODUCTION

  • Intrinsic self-correction uses the model’s own knowledge, unlike extrinsic self-correction, which uses external knowledge.The introduction presents whether LLMs can perform intrinsic self-correction as an open research question.Source: §1 · pp. 1–2
  • Recent studies report improved benchmark performance with self-correction, while later critiques question intrinsic self-correction’s efficacy and report QA accuracy degradation.The cited critique attributes prior results to possible confounds including external feedback, ground-truth stopping criteria, and uninformative prompts.Source: §1 · pp. 1–2
  • The authors use theoretical analyses and experiments to argue that LLMs have intrinsic self-correction ability, while initial errors reflect inherent hallucination.Their experiments use the same settings as Huang et al. (2024).Source: §1 · p. 2
  • The authors identify temperature and prompt neutrality as conditions affecting intrinsic self-correction; biased wording may prompt unwanted answer changes.They describe fair prompts as avoiding direct or indirect influence to change or maintain the initial answer.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper reports intrinsic self-correction across LLMs when prompts are fair and temperature is zero.The introduction also reports accuracy degradation when temperature is increased or prompts are biased.Source: §1 · p. 2
  • The paper presents intrinsic self-correction as analogous to chain-of-thought and self-verification prompting, which provide additional context for a final response.This framing is offered as a perspective on how intrinsic self-correction functions.Source: §1 · p. 2

2 NEW PERSPECTIVE ON INTRINSIC SELF CORRECTION

The section models intrinsic self-correction as adding reasoning context and reviewing an initial answer, with outcomes depending on prompting and the order of decisions and rationales. It argues that hallucinations can keep an LLM from reaching its full ability on the first attempt.

2.1 PRELIMINARY

  • Intrinsic SC adds prompt context, like CoT and self-verification; as temperature approaches zero, generation shifts from sampling to selecting the maximum-logit token.The section also notes that higher temperature may harm intrinsic SC through non-optimal sampling.Source: §2.1 · p. 3
  • At T = 0, a fixed prompt should yield the same response, but GPT-family models may vary, with GPT-4 showing very high answer variance.The authors suggest query optimizations and GPT-4’s internal sparse mixture-of-experts design as possible reasons.Source: §2.1 · p. 3

2.2 DOES INTRINSIC SC EXIST IN LLMS?

  • Intrinsic SC is modeled as an initial answer and rationale, a correctness review with new rationales and a decision, then a final answer using the prior information.The second-stage output can put the decision before the rationale or the rationale before the decision.Source: §2.2 · p. 4
  • With CoT initially, decision-first correction improves accuracy; without initial CoT, it may lower accuracy, while rationale-first correction improves accuracy comparably to CoT alone.The rationale-first case with CoT could perform better than decision-first correction, but the paper notes that prompting did not reliably control this output order.Source: §2.2 · p. 4
  • Rationale-first review could outperform decision-first review, but GPT-3.5 favored decision-first output and prompts changed the order in only some responses.Source: §2.2 · p. 4
  • The authors conclude that intrinsic SC exists with adequate prompting and may be as effective as other prompt-engineering techniques.Source: §2.2 · p. 4

2.3 WHY CAN’T LLM ANSWER QUESTIONS CORRECTLY IN THE INITIAL ATTEMPT?

  • Assuming a non-zero chance of hallucination, an LLM may answer incorrectly despite having the correct knowledge, lowering accuracy below its true ability.Source: §2.3 · p. 5
  • The authors conclude that intrinsic SC can reduce hallucinations and improve response quality, similarly to other prompt-engineering techniques.Source: §2.3 · p. 5

3 EXPERIMENT SETUP

The experiments evaluate self-correction across six benchmarks and several language models. The setup is used with theoretical analyses to study conditions under which self-correction works best.

3 EXPERIMENT SETUP

  • The experiments are designed to study self-correction across tasks and models and, with further theoretical analyses, examine conditions under which it thrives.Source: §3 · p. 5
  • Six benchmarks cover CommonSense QA, GSM8K, MMLU, HotPotQA, Big Bench, and SVAMP.Source: §3 · p. 5
  • The evaluation uses GPT-3.5-Turbo, GPT-4-Turbo, Llama-3.1-8B-Instruct, and Mistral-7b-instruct-v0.2.The two Llama and Mistral models run locally on a server with 2x NVIDIA A100 (80GB PCIe) GPUs.Source: §3 · p. 5

4 INCREASING TEMPERATURE MIGHT DECREASE ACCURACY DURING SC

Temperature can reduce intrinsic self-correction (SC) accuracy, though the effect differs between GPT-3.5 and GPT-4. The analysis links this effect to temperature-driven decision variability and the response structure used during SC.

4 INCREASING TEMPERATURE MIGHT DECREASE ACCURACY DURING SC

  • The temperature analysis motivates zero temperature as a way to limit randomness that can harm later SC stages.The paper notes that higher temperature can increase creativity, while affecting subsequent SC stages even when the original response is unchanged.Source: §4 · p. 5

4.1 THEORETICAL ANALYSIS

  • As temperature rises, decision variance increases, making decisions more likely to flip and potentially introducing hallucinations that reduce accuracy after SC.The analysis treats context-conditioned decisions as binary and notes that variance is highest when the positive-decision probability is 0.5.Source: §4.1 · p. 6
  • The response ordering matters: Order 1 is more temperature-sensitive than Order 2, whose post-rationale decision is closer to high-confidence copying.The paper characterizes providing a decision as harder and lower-confidence than selectively copying information from rationales.Source: §4.1 · p. 6

4.2 IMPACT OF TEMPERATURE ON SELF-CORRECTION

  • GPT-3.5’s SC ability drops as temperature increases, while GPT-4 remains robust to temperature changes on Commonsense QA.The experiments used an optimized prompt; an ablation fixing the first-stage response showed a clearer accuracy-decrement trend for GPT-3.5.Source: §4.2 · pp. 6–7, Figure 2 · p. 7

5 DESIGNING FAIR PROMPT FOR SELF-CORRECTION

The paper argues that intrinsic SC depends on fair prompting and examines how prompt bias affects answer changes and accuracy. Across the reported benchmarks, fair prompts improve accuracy, while preliminary results suggest larger models may have stronger SC ability.

5.1 THEORETICAL ANALYSIS

  • The final stage’s answer depends on the stage-2 and stage-3 prompts, because other listed factors are fixed or depend only on the stage-2 temperature.The analysis identifies τ2 and τ3 as the remaining variables determining the final answer.Source: §5.1 · p. 7

5.2 INTRINSIC SC COULD BE ACHIEVED WITH FAIR PROMPT UNDER ZERO TEMPERATURE

  • Across six datasets, fair Set 3 improves accuracy for the four evaluated models, while biased Set 1 may decrease it.On averaged results, the fair-prompt improvement is statistically significant with respect to the number of changed responses.Source: Table 2 · p. 8, §5.2 · p. 9

5.1 THEORETICAL ANALYSIS

  • Under the analysis’s assumption that biased prompts induce extra changes, bias at either prompt stage can introduce more incorrect answers, and effects accumulate when both are biased.The analysis models a γ% of answers as changed under bias and assumes no answer changes under a fair prompt.Source: §5.1 · pp. 8–9

5 DESIGNING FAIR PROMPT FOR SELF-CORRECTION

  • The prompt study compares three sets, including one that removes negative phrasing and a final set that also avoids words such as “correct” and “incorrect.”The experiments retain the first-stage prompt used by Huang et al. (2024) to examine the effect of its semantic bias.Source: §5 · p. 7, §5.2 · p. 9

5.2 INTRINSIC SC COULD BE ACHIEVED WITH FAIR PROMPT UNDER ZERO TEMPERATURE

  • Prompt bias effects vary across models and benchmarks, and the authors say they cannot yet explain or predict which specific features account for that variation.The paper reports different patterns for Commonsense QA and HotpotQA and presents a possible training-related explanation as a hypothesis.Source: §5.2 · p. 9
  • A truly unbiased prompt avoids semantically or literally positive and negative cues; the authors recommend this as a safe option for future SC studies.Examples include “find the problem” and “correct your answer.”Source: §5.2 · p. 9

5.3 ABLATION STUDY ON MODEL SIZE

  • Preliminary Qwen 2.5 results suggest larger models may have better intrinsic SC ability, but the study is too limited for a definitive claim.The ablation evaluates math and logic benchmarks across model sizes.Source: §5.3 · p. 9, Table 3 · p. 10, Figure 3 · p. 10

6 LIMITATIONS

The reported findings are limited by model and dataset coverage, narrower evaluation in some sections, and the absence of error bars.

6 LIMITATIONS

  • Evaluations cover only four models and six datasets because of computational and staffing constraints.Sections 4 and 5 focus mainly on a subset of models and CommonSenseQA.Source: §6 · p. 10
  • Sections 4 and 5 focus mainly on a subset of models and CommonSenseQA.This narrows the scope of those sections’ evaluations.Source: §6 · p. 10
  • Error bars are absent because error analysis required extensive computational and financial resources; the authors acknowledge variance in non-zero-temperature experiments.Source: §6 · p. 10

7 CONCLUSION

The paper argues that intrinsic self-correction is analogous to self-verification and chain-of-thought prompting, and reports that it appears across models under zero temperature and unbiased prompts. Its related-work discussion contrasts external assistance with intrinsic correction and reviews evidence and open questions around both approaches.

7 CONCLUSION

  • Intrinsic self-correction is reported across models when zero temperature and unbiased prompts are used.The conclusion frames it as analogous to self-verification and chain-of-thought prompting.Source: §7 · p. 10

A.1 EXTRINSIC SC

  • Extrinsic self-correction uses external knowledge, often through stronger LLMs acting as verifiers, recommenders, or revisers, sometimes with retrieval or critic data.The paper notes that reliance on external knowledge bases or larger models can limit practicality.Source: §A.1 · p. 16, §A.2 · p. 16

A.2 INTRINSIC SC

  • Intrinsic self-correction can improve responses without external resources, while prior work commonly uses the same LLM to generate feedback and revise its answer (Madaan et al., 2023).Source: §A.2 · pp. 16–17
  • Prior studies dispute intrinsic self-correction, citing oracle-label criteria and prompts that may favor refinement, while other investigations report mixed evidence (Huang et al., 2024).Jiang et al. (2024) studies selection among multiple-agent outputs; Tyen et al. (2024) reports a positive net value in correcting incorrect traces.Source: §A.2 · p. 17
  • The paper argues that a generic fair prompt can support intrinsic self-correction at zero temperature, and distinguishes its analysis of optimal conditions from Liu et al.’s convergence focus.It also notes that confidence-based correction faces questions about whether LLM confidence tracks correctness (Becker & Soatto, 2024).Source: §A.2 · p. 17

A.3 APPLICATIONS OF SELF-CORRECTION

  • Intrinsic self-correction outputs may serve as reasoning traces for fine-tuning during reinforcement learning, connecting the approach to introspective reasoning and inference scaling (Liu et al., 2023; Wang et al., 2023b).Source: §A.3 · p. 17

B DETAILS ON BENCHMARKS

The benchmark suite spans commonsense, arithmetic, subject knowledge, multi-hop reasoning, sports reasoning, and arithmetic word problems, with some evaluations restricted by resource demands.

B DETAILS ON BENCHMARKS

  • The study uses CommonSenseQA’s five-choice questions, GSM8K arithmetic and logic problems, and two MMLU subjects: Formal Logic and Conceptual Physics.The MMLU evaluation is limited to those two subjects because of resource constraints.Source: B DETAILS ON BENCHMARKS · p. 18
  • The suite also includes contextual HotpotQA for multi-hop reasoning, BIG-bench Sport feasibility judgments, and a 300-pair SVAMP test split.Source: B DETAILS ON BENCHMARKS · p. 18
  • Experiments generally use each benchmark’s full validation set, but GPT-4 temperature experiments and GSM8K evaluations use only the first 200 questions.This follows Huang et al. (2024) and reflects substantial computational requirements.Source: B DETAILS ON BENCHMARKS · p. 18

C PROOF OF PROPOSITION 2.1

The proof models a capable LLM whose generated answers can be randomly changed by hallucination, and argues that this lowers accuracy relative to its true ability. The appendix also points readers to numerical results for Figure 2A and Figure 2B.

C PROOF OF PROPOSITION 2.1

  • Under the proof’s assumptions, hallucination makes generated accuracy lower than the LLM’s true ability when that ability exceeds random guessing.True ability is represented by λ > 1/k; hallucination changes a fraction h of answers, with changes assumed equally likely to alter correct or incorrect answers.Source: C PROOF OF PROPOSITION 2.1 · pp. 18–19
  • The proof counts hλ answers changing from correct to incorrect and h(1−λ)/(k−1) changing from incorrect to correct, yielding a net accuracy decrease when λk > 1.Source: C PROOF OF PROPOSITION 2.1 · pp. 18–19

D.1 NUMERICAL RESULTS FOR FIGURE 2A AND FIGURE 2B

  • This section provides detailed numerical results for the analysis in Section 4, presented for the reader’s reference.Source: §D.1 · p. 19

D.1.1 NUMERICAL RESULTS FOR FIGURE 2A

On CommonSense QA, temperature experiments with unbiased prompts show an overall downward accuracy trend for GPT-3.5 and GPT-4, clearer for GPT-3.5 than GPT-4.

D.1.1 NUMERICAL RESULTS FOR FIGURE 2A

  • GPT-3.5 shows a clear accuracy decrement as temperature changes, while GPT-4’s trend is less distinct because of response decomposition.The reported overall trend is downward for both models.Source: §D.1.1 · p. 19
  • The tables report CommonSense QA accuracy results for the two GPT models under the temperature settings.Source: §D.1.1 · p. 19, Table 4 · p. 19, Table 5 · p. 19
  • The temperature effect is evaluated on GPT-3.5 and GPT-4 using unbiased prompts, with temperature applied throughout generation.Source: Table 4 · p. 19, Table 5 · p. 19

D.1.2 NUMERICAL RESULTS FOR FIGURE 2B

These numerical analyses examine temperature and prompt effects on intrinsic self-correction, including answer changes across models and datasets. The reported patterns link higher temperature and biased prompts with more harmful changes.

D.1.2 NUMERICAL RESULTS FOR FIGURE 2B

  • With stage one fixed at a zero-temperature response, GPT-3.5 shows an obvious accuracy-loss trend as temperature varies during self-correction, unlike GPT-4, whose trend is less distinct.The tables cover temperature applied only to the intrinsic self-correction process.Source: §D.1.2 · p. 20, Table 6 · p. 20, Table 7 · p. 20

D.2 ABLATION STUDY ON BIASED PROMPT

  • Under the biased prompt, rising temperature still trends toward lost self-correction ability for GPT-3.5, while GPT-4 remains robust; the effect depends on response decomposition, not prompt bias.The comparison is reported for the biased-prompt temperature ablation.Source: §D.2 · p. 21, Table 8 · p. 21, Table 9 · p. 21

D.3 NUMERICAL RESULTS ON THE NUMBER OF CHANGED ANSWERS

  • Biased Prompt Set 1 has more invalid answer changes (C2I) than unbiased Prompt Set 3, and invalid changes exceed valid changes (I2C).Source: §D.3 · p. 21

D.3.1 NOTATION REFERENCE

D.3.2 CHANGE OF DISTRIBUTION OF LABELS IN RESPONSE TO TEMPERATURE CHANGES

  • As temperature increases, changed responses increase, with C2I growing faster than I2C across the reported model results.The temperature analyses include GPT-3.5, GPT-4, and Mistral tables.Source: §D.3.2 · p. 22, Table 11 · p. 22, Table 12 · p. 22

D.3.3 CHANGE OF DISTRIBUTION OF LABELS IN RESPONSE TO PROMPT CHANGES

  • Switching from biased to unbiased prompts significantly decreases changed answers; biased prompts are associated with more changes and greater risk of correct answers becoming incorrect.For robust models, the reported I2C and C2I counts remain very small with the biased prompt.Source: §D.3.3 · p. 23, Table 14 · p. 23, Table 15 · p. 24

E PROMPT AND EXAMPLE OUTPUTS

This appendix presents prompts for reviewing or verifying answers and example outputs, covering commonsense QA and math problems as well as a revolving-door security question.

E PROMPT AND EXAMPLE OUTPUTS

E.1 PROMPT SET 1 HUANG ET AL. (2024) (BIASED) ON COMMONSENSE QA

  • On the leaf question, the model retained forest (C) after review while noting ambiguity in “ground” and possible variation from large compost piles.The improved response interprets “ground” as non-forested outdoor areas and says a forest remains the most consistent, extensive source of leaves.Source: §E.1 · p. 25

E.2 PROMPT SET 1 HUANG ET AL. (2024) (BIASED) ON GSM8K

  • For Jill’s annual salary, reviewing the prior answer found no calculation or reasoning problems; teaching earned $35,000/year and coaching $22,500/year, totaling $57,500/year, with final answer 57500.Source: §E.2 · p. 26

E.3 PROMPT SET 2 (LESS BIASED) ON COMMONSENSE QA

  • On the less-biased bathroom-key question, the model judged its reasoning sound and retained convenience store (B).It explains that such stores may require a purchase to obtain a restroom key.Source: §E.3 · pp. 26–27

E.4 PROMPT SET 2 (LESS BIASED) ON GSM8K

  • The response affirms the reasoning is sound and reiterates Eliza’s weekly earnings as 460, calculating regular pay for the first 40 hours at $10 per hour and overtime at 1.2 times the regular rate.Source: §E.4 · p. 27

E.5 PROMPT SET 3 (UNBIASED) ON COMMONSENSE QA

  • On the unbiased guitar question, verification endorsed making music (E) as the most appropriate answer rather than singing or hearing sounds.The response describes playing guitar as actively creating music.Source: §E.5 · p. 28

E.6 PROMPT SET 3 (UNBIASED) ON GSM8K

  • For Terry’s yogurt purchase, the response calculates 60 yogurts for 30 days, buys 15 sets of four at $5.00 each, and gives a total cost of $75.00 (answer 75).Source: §E.6 · p. 28

E.7 SECTION 2.2 CASE 2; E.8 SECTION 2.2 CASE 3; E.9 SECTION 2.2 CASE 4

  • The case examples retain work room (D) for the hostess question and bank (A) for the revolving-door question, with the latter also shown as a bare initial answer.The work-room response is verified as reasonable; the bank response is justified as controlling entry and exit flow.Source: §E.7 · p. 29, §E.8 · p. 29

E.9 SECTION 2.2 CASE 4

  • The prompt asks the model to analyze the question and answer and provide its reasoning.Source: §E.9 · p. 29
  • The prompt then asks the model to verify whether the response is the most appropriate answer and clearly state yes or no.Source: §E.9 · p. 29
  • In the example, the model explains that revolving doors at banks control entry and exit flow by allowing one person through at a time.The explanation supports choosing bank as the answer.Source: §E.9 · pp. 29–30
  • The example judges the response selecting option A, bank, to be the most appropriate answer and answers yes.Source: §E.9 · p. 30
  • The final instruction asks for the original question’s answer as a single letter from A to E, formatted in parentheses at the end.Source: §E.9 · p. 30
  • The example’s final answer is A: bank.Source: §E.9 · p. 30

F RESPONSE DIFFERENCES BY EACH OF THE MODEL

The examples show a difference in where GPT-3.5 and GPT-4 place their final judgment during response generation. Both models then receive a verification prompt and provide a justification.

F RESPONSE DIFFERENCES BY EACH OF THE MODEL

F.1 GPT-3.5

  • In the GPT-3.5 example, the initial response answers “bank,” and the verification response confirms that answer.Source: §F.1 · p. 30

F.2 GPT-4

  • The GPT-4 example gives an extended rationale for choosing a bank before the verification response reiterates the choice.Source: §F.2 · pp. 30–31
Loading 2406.15673v2…