CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing

• The introduction identifies hallucinations, faulty code, and toxic content as behaviors that undermine trust in LLMs and hinder real-world applications.Source: §1 · p. 1
• CRITIC verifies selected aspects of an initial output with external tools, then revises it from their critiques in an iterative verify-then-correct process.The introduction names search engines and code interpreters as examples of tools.Source: §1 · p. 1, Figure 1 · p. 2
• CRITIC uses in-context learning with tool interaction, requiring only text-to-text tool APIs and a few-shot demonstration rather than supplementary data or task-specific training.This design addresses the resource-intensive human annotation and data construction required by traditional mitigation approaches.Source: §1 · pp. 1–2
• For ChatGPT, CRITIC achieves 7.7 F1 enhancements across three QA tasks, 7.0% absolute gains on three mathematical reasoning tasks, and a 79.2% reduction in toxicity probability.The paper reports consistent gains over prior techniques across free-form question answering, mathematical program synthesis, and toxicity reduction.Source: §1 · p. 2
• All tested LLMs proved unreliable at validating their own results, while self-correction without external feedback could yield modest gains or worsen performance.The paper emphasizes external tool feedback as crucial for consistent self-improvement.Source: §1 · p. 2

Truthfulness Evaluation

• Open-ended truthfulness evaluation for API-accessed LLMs is less studied than task-specific hallucination detection, motivating CRITIC’s external-tool verification of black-box outputs.Earlier work largely focuses on faithfulness to provided source content, while CRITIC draws inspiration from journalistic fact-checking.Source: §2 · p. 2

Natural Language Feedback

• Feedback methods use scalar signals for reinforcement learning or ranking, or natural-language feedback for editing; LLM self-feedback remains task-limited and unreliable for truthfulness and reasoning (Kadavath et al., 2022).CRITIC addresses these limitations by empowering LLM evaluation with external tools (Fu et al., 2023).Source: §2 · p. 3

Tools Augmented Language Models

• Tool augmentation supports retrieval, search, calculation, code execution, and mathematical proving; CRITIC uses in-context learning without task-specific training (Paranjape et al., 2023).Prior work teaches tool use through pre-training, fine-tuning, or in-context learning.Source: §2 · p. 4

3 CRITIC: CORRECTING WITH TOOL-INTERACTIVE CRITIQUING

• CRITIC iteratively verifies an initial LLM output with external tools, feeds the resulting critiques back to the model, and repeats until a stopping condition is met.The cycle can run for multiple rounds.Source: §3 · p. 4

3.1 IN-CONTEXT LEARNING FOR LLMS

• CRITIC uses chain-of-thought reasoning (Wei et al., 2022) and few-shot in-context learning (Brown et al., 2020; Min et al., 2022), prompting the model with example input-output pairs before a test instance.The model completes the prompt to generate an output for the test instance.Source: §3.1 · p. 4

3.2 INTERACTION WITH EXTERNAL TOOLS

• External tools, including search engines, code interpreters, and APIs, are represented as text-to-text functions and interleaved with LLM generations in demonstrations.Search tools return parsed results for model-generated queries, while code interpreters return execution information and results.Source: §3.2 · p. 4

3.3 VERIFICATION WITH TOOL-INTERACTION

• For verification, CRITIC generates critiques using task-appropriate tools to assess attributes such as truthfulness, feasibility, or safety; tool choice can be task-dependent, heuristic, or automatic.The implementation prespecifies tools by task for evaluation, using Google for question-answering verification.Source: §3.3 · p. 5

3.4 CORRECTION WITH CRITIQUES

• The model conditions each correction on the input, its previous output, and verification critiques, which identify errors, suggest actions, or provide credible grounding.The verify-and-correct cycle can stop when critiques are satisfied, a maximum iteration count is reached, or environmental feedback arrives.Source: §3.4 · p. 5

Implementation

• QA evaluation uses 500 validation examples each from AmbigNQ, TriviaQA, and HotpotQA, reporting EM and F1.A Google-based search tool retrieves the top web page and extracts up to 400 characters; CoT initializes answers, followed by at most three corrections. Baselines include prompting, Self-Consistency, ReAct, and CRITIC without tools.Source: Implementation · pp. 5–6
• CRITIC improves over initial CoT results across QA datasets, correction settings, and LLMs; F1 gains are 5.6 and 10.3 for text-davinci-003 and 7.7 and 12.4 for ChatGPT.The paired gains are for CRITIC and CRITIC∗, respectively; CRITIC also exceeds ReAct by average F1 improvements of 5.1 and 8.2 on the two LLMs.Source: Implementation · pp. 6–7, Table 1 · p. 6

4.2 MATHEMATICAL PROGRAM SYNTHESIS

• For mathematical synthesis, CRITIC uses Python interpreter errors and execution results to revise programs, with up to four corrections and exact match evaluation on GSM8k, SVAMP, and TabMWP.Initial programs use program-of-thought; correction uses sampling with p = 0.5. Baselines include PoT and CRITIC without interpreter feedback.Source: §4.2 · pp. 7–8
• TabMWP gains from CRITIC scale with model size: +4.7, +9.4, and +16.0 for 7B, 13B, and 70B models, respectively.CRITIC and CRITIC∗ improve over PoT; without interpreter feedback, correction is limited and unstable, including a 1.8-point decrease on text-davinci-003.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 TOXICITY REDUCTION

• For detoxification, CRITIC uses PERSPECTIVE API toxicity scores to critique outputs, allows up to four iterations, and stops when overall toxicity falls below 10%.Evaluation uses 1,000 non-toxic prompts and measures maximum toxicity across 25 generations, probability of toxicity above 50%, perplexity, and distinct n-grams.Source: §4.3 · pp. 8–9
• CRITIC substantially reduces toxic generations while preserving fluency and diversity, achieving toxicity mitigation on par with supervised state-of-the-art methods without extra data or training.The results also underscore the importance of external feedback, as the LLM alone struggles to mitigate toxicity effectively.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 ADDITIONAL ABLATIONS AND ANALYSIS

• Across tasks, iterative correction generally improves results, with larger gains in the oracle setting; most benefits typically arrive within two to three rounds.Marginal returns diminish, and self-generated feedback without reliable external feedback is less effective and relatively inefficient.Source: Figure 3 · p. 6, Figure 4 · p. 7, Figure 5 · p. 8
• For QA, critique-conditioned generation beats best-of-N rejection sampling by 4.5 and 3.3 EM for the two LLMs, respectively.The authors attribute this to critiques offering actionable suggestions and credible grounding that guide new generations away from similar errors.Source: Table 1 · p. 6, §4.4 · p. 9

5 CONCLUSION

• CRITIC enables frozen LLMs to self-verify and self-correct through external interaction, without requiring further training.It uses external feedback to validate knowledge and improve answers through introspection.Source: §5 · p. 9
• Experiments across diverse tasks and LLMs demonstrate CRITIC’s effectiveness, generality, and interoperability.Source: §5 · p. 9
• CRITIC’s findings highlight LLMs’ unreliability in self-verification and the potential of external tool interaction to address it.Source: §5 · p. 9

A LIMITATIONS & FUTURE WORK

• Inference overhead grows linearly with iterations, though CRITIC can yield benefits with relatively few iterations, even one.Two correction iterations in mathematical program synthesis take about twice the PoT baseline time; ReAct and Self-Consistency also trade time for performance.Source: A LIMITATIONS & FUTURE WORK · p. 21
• CRITIC experiments rely on in-context demonstrations, and results may vary with prompt construction; future work should reduce reliance on manually crafted prompts.The authors describe their ReAct-style prompts as straightforward to construct, with a workload comparable to ReAct or PoT, and note the long re-encoded context window of demonstrations.Source: A LIMITATIONS & FUTURE WORK · p. 21
• Effectiveness on other tasks and LLMs remains uncertain, and future work could extend CRITIC beyond text to translation, proofs, simulated environments, and other modalities.The authors note that some inputs may not need or support suitable external feedback, and that explicit language evaluation may not suit every output.Source: A LIMITATIONS & FUTURE WORK · p. 21

B ETHICAL CONSIDERATIONS

• Ethical safeguards should address malicious use, transparency, bias in models and tools, and privacy and security risks from API interactions.Suggested measures include making verification and corrections understandable, evaluating and mitigating biases, and using data anonymization and secure communication protocols.Source: B ETHICAL CONSIDERATIONS · p. 21

C.1 NL FEEDBACK & SELF-CORRECTION

• Self-correction without external feedback can yield modest gains or worse performance, as LLMs often cannot accurately identify what they know.The discussion cites deterioration without oracle verification and notes that later studies on reasoning, graph coloring, and planning corroborate the unreliability of self-correction.Source: §C.1 · p. 22

C.2 UNCERTAINTY ESTIMATION FOR SELF-VERIFICATION

• Uncertainty estimation includes intrinsic probability or sampling methods and post-hoc methods, but poor calibration, free-form evaluation, and interpretability limit reliability.CRITIC addresses these issues by interacting with external tools such as search engines to improve the reliability of expressed uncertainty.Source: §C.2 · pp. 22–23

C.3 DETAILS FOR UNCERTAINTY ESTIMATION BASELINES

• The uncertainty-baseline details specify confidence calculations for language-model probabilities, entropy, self-consistency, and self-evaluation.Self-Con samples 20 CoT answers at temperature 0.5; Self-Eval uses the greedy answer and 10-shot prompts, since zero-shot performs poorly.Source: §C.3 · p. 23

C.4 THE RELATIONSHIP BETWEEN CRITIC AND RLHF

• CRITIC and RLHF target related alignment goals but differ: RLHF fine-tunes with human annotations, while CRITIC enables black-box self-correction through tools at inference.The paper cautions that direct performance comparison may mislead and identifies CRITIC’s potential to inspire or enhance RLAIF (Bai et al., 2022b).Source: §C.4 · p. 23

D.1 IS SELF-VERIFICATION RELIABLE?

• Without Oracle verification, self-feedback correction can worsen answers in QA, commonsense reasoning, and math, where external knowledge or intricate reasoning is needed.The authors attribute this risk to unstable gaps between generation, discrimination, and critique, and difficulty knowing what the model knows (Kadavath et al., 2022).Source: §D.1 · p. 24
• The reliability test uses ChatGPT-generated confidence scores on free-form QA to assess how well they discriminate correct from incorrect answers.For verification, the model critiques its proposed answer before selecting among four correctness options.Source: §D.1 · p. 24
• The confidence score is the normalized weighted sum of token probabilities for options (A), (B), (C), and (D), with option weights set from 4 to 1.This approach is suggested by (Liu et al., 2023b).Source: §D.1 · p. 24
• Answers with F1 above 0.6 count as correct, and AUROC measures discrimination; the evaluation uses three datasets with exact-match scores ranging from 30 to over 80.Fuzzy matching was judged more consistent with human evaluation than exact matching; verification accuracy is also reported for non-intrinsic methods.Source: §D.1 · p. 24
• Baselines include entropy-based intrinsic scores, Self-Con sampling, Self-Eval expressed uncertainty, and Only-True, which predicts every answer as correct.The intrinsic methods comprise LM probabilities (entropy), length-normalized predictive entropy, and maximum predictive entropy.Source: §D.1 · p. 25
• 54% verification accuracy on HotpotQA leaves Self-Eval only slightly above random guessing, while it performs worse than Only-True on TriviaQA; CRITIC outperforms prior estimation methods.The authors also characterize CRITIC’s results as generalizable and interpretable.Source: §D.1 · p. 25

D.2.1 ERROR ANALYSIS ON FREE-FORM QUESTION ANSWERING

• The authors manually analyzed 100 randomly selected HotpotQA cases; false negatives used F1 > 0.6 as the automatic correctness threshold.They annotated error types for both initial CoT and CRITIC; false negatives were human-correct predictions judged wrong by the automatic indicator.Source: §D.2.1 · p. 25, Table 6 · p. 25
• Hallucinations fell from 36% for CoT to 7% for CRITIC, though some persisted.The analysis reports that CRITIC did not eliminate hallucinations.Source: Table 6 · p. 25, §D.2.1 · p. 25
• Residual hallucinations arose when search failed to find useful evidence or CRITIC misunderstood the evidence.The paper illustrates these cases in Appendix E.Source: §D.2.1 · p. 25
• Most errors after CRITIC involved reasoning mistakes, refusals to answer, or incorrect corrections.Source: §D.2.1 · p. 25
• CRITIC refused when it could not find enough supporting evidence, a behavior the authors considered consistent with truthfulness.Source: §D.2.1 · p. 25
• Dataset ambiguities or inaccuracies appeared in 49% of CRITIC error samples as false negatives, motivating LLM-based QA evaluation (Shao et al., 2023).These cases suggest bias in evaluating free-form QA with automatic metrics such as EM and F1.Source: §D.2.1 · p. 25, Table 6 · p. 25

D.2.2 ERROR ANALYSIS ON MATHEMATICAL PROGRAM SYNTHESIS

• Intrinsic reasoning errors made up 77.4% of initial PoT errors, unreasonable outputs 16.8%, and syntax and runtime errors together 5.8%.Intrinsic errors included misunderstanding the question or omitting conditions.Source: §D.2.2 · p. 26, Table 7 · p. 26
• CRITIC corrected 57.4% of unreasonable-output errors and 35.3% of syntax errors, compared with 26.7% of intrinsic errors, where reliable feedback was unavailable.Correction rates were assessed in the non-oracle setting with up to four rounds.Source: §D.2.2 · p. 26, Table 7 · p. 26
• In a non-oracle setting, CRITIC reduced errors in initially erroneous samples by 32.2%; accuracy on originally correct outputs decreased by 4.3%, with 14.3% wrong corrections after tool feedback.Source: §D.2.2 · p. 26

D.3 DISCUSSION ON TOOL USE COSTS

• The paper describes CRITIC’s tool use as free: it used a local code interpreter for mathematical program synthesis and Google-provided Perspective API for toxicity reduction.For QA, it used a web tool to retrieve Google Search results and web pages such as Wikipedia.Source: §D.3 · p. 26
• For QA experiments, the authors cached about 9GB of web search results from January to April 2023 and planned to release the caches after anonymous review to support stability, fairness, and reproducibility.The Search Engine results in the paper came from their separately open-sourced web-tool code.Source: §D.3 · p. 26

D.4 THE SIGNIFICANCE OF EACH TOOL IN VARIOUS CONTEXTS

• Web tools take the leading role in knowledge-heavy tasks, including commonsense QA such as AmbigNQ and TriviaQA and multi-hop reasoning such as HotpotQA.CRITIC primarily used Wikipedia page browsing and Google snippets.Source: §D.4 · pp. 26–27
• For mathematical program synthesis, external knowledge is typically unnecessary, so the code interpreter acts like a calculator and supplies feedback through error messages and execution results.The paper illustrates this feedback in Appendix E.2.Source: §D.4 · p. 27

D.5 COMPLETE LLAMA-2 RESULTS

• Table 8 reports LLaMA-2 free-form question-answering results, including an oracle setting that corrects only incorrect answers and prior supervised SoTA results.The caption attributes prior supervised SoTA results to Shao & Huang (2022), Shi et al. (2023), and Zhu et al. (2021).Source: Table 8 · p. 27
• Table 9 reports LLaMA-2 results for mathematical program synthesis.Source: Table 9 · p. 28

D.6 ADDITIONAL COMPARISON WITH SELF-CORRECTION WITHOUT TOOL-USE

• Codex with PAL scores 71.3; adding Self-Refine yields 26.7 (-44.6), while oracle Self-Refine yields 76.2 (+4.9).The table identifies these as additional mathematical program synthesis results; the oracle setting applies correction only to incorrect answers.Source: Table 10 · p. 28, §D.6 · p. 28

D.7.1 FREE-FORM QUESTION ANSWERING

• Figures 6–9 plot F1 and EM across CRITIC iterations for free-form question answering with gpt-3.5-turbo and text-davinci-003.Source: Figure 6 · p. 28, Figure 7 · p. 29, Figure 8 · p. 29

D.7.2 MATHEMATICAL PROGRAM SYNTHESIS

• Figure 10 plots solve rate across CRITIC iterations on GSM8k using gpt-3.5-turbo.Source: Figure 10 · p. 30

D.7.3 TOXICITY REDUCTION

• Figure 11 plots solve rate across CRITIC iterations on GSM8k using text-davinci-003.Source: Figure 11 · p. 30
• Figures 12–13 show CRITIC iterations on toxicity reduction using gpt-3.5-turbo and text-davinci-003, respectively.Source: Figure 12 · p. 30, Figure 13 · p. 30

E QUALITATIVE EXAMPLES

• The qualitative section provides representative success and failure case analyses for each task, omitting parts deemed unimportant to save space.Source: E QUALITATIVE EXAMPLES · p. 31

E.1.1 SUCCESS CASES

• CRITIC revised the screenwriter answer from David Diamond to David Weissman, matching the gold answer with EM/F1 True/1.00.Search evidence indicated David Weissman co-wrote the film with Nicolas Cage and Tea Leoni, while David Diamond also co-wrote The Family Man.Source: §E.1.1 · pp. 31–32
• The instrument-ratio example corrected faulty reasoning and returned Badly Drawn Boy, matching the gold answer with EM/F1 True/1.00.The revised reasoning compared the members and instruments of both acts rather than accepting the proposed answer.Source: §E.1.1 · pp. 32–33
• For the Hertfordshire fire-officer question, CRITIC used appointment timing to update Darryl Keen to Alex Woodman as the current officer.The example states Keen held the role until July 2021 and Woodman was appointed in September 2021; the revised answer included both time periods but scored EM/F1 False/0.33 against an outdated gold answer.Source: §E.1.1 · p. 34

E.1.2 FAILURE CASES

• The examples distinguish evidence gaps from reasoning errors: the Olympic answer remained unsupported, while the Tennessee answer relied on evidence about the wrong entity.Both cases show that retrieved material alone did not ensure a correct answer to the question asked.Source: §E.1.2 · pp. 34–36
• For the 1904 Olympics, the evidence lacked the number of titles won by Americans, prompting “Not enough evidence”; the prediction scored EM/F1 0/0.00 against gold answer 21.Source: §E.1.2 · pp. 34–35
• For the Tennessee Volunteers question, CRITIC initially confused the SEC headquarters city with the city where the team plays, answering Knoxville rather than the gold answer Birmingham, Alabama.The example identifies the category mistake: the question asks where the team plays, not where the conference is headquartered.Source: §E.1.2 · p. 36

E.2.1 SUCCESS CASES

• In the avocado-tree example, execution raised a NameError because the code used the 10th-year fruit variable before defining it, prompting feedback to correct the calculation order.The feedback says to calculate the 8th-year amount after defining the 10th-year amount.Source: §E.2.1 · pp. 36–37
• The venue example first produced a TypeError from using undefined x; its replacement equation ran but returned 6.66667 instead of the gold answer 10.0.CRITIC also noted that the equation should set the venues’ costs equal and solve for guests.Source: §E.2.1 · pp. 37–38
• After feedback that the number of guests must be an integer, the displayed venue answer is 10.0, matching Gold Answer 10.0 and marked correct.Source: §E.2.1 · pp. 38–39

E.2.2 FAILURE CASES

• CRITIC’s verifier wrongly accepts the dance solution, although its execution returns 55.0 and a later solution matches gold answer 60.0.The listing labels this a failure case for wrong verification.Source: §E.2.2 · pp. 39–40
• The verifier treats 55 as reasonable because it lies between 0 and 100, then checks the code and concludes it is correct.Source: §E.2.2 · p. 39
• In the snowball example, the initial program times out with no answer, and the failure analysis attributes this to an infinite loop.Source: §E.2.2 · p. 40
• The proposed diagnosis says the loop runs until Jay has more than 60 snowballs and calls for converting the 15-minute melt rate to an hourly rate.Source: §E.2.2 · p. 40
• The revised snowball solution still fails, returning 3.0 hours against a gold answer of 5.0.Source: §E.2.2 · p. 41

E.3 EXAMPLES OF TOXICITY REDUCTION

• The toxicity-reduction examples section warns that it contains LLM-generated content that may be offensive or sensitive.Source: §E.3 · p. 41

E.3.1 SUCCESS CASES

• Toxicity: 0.521, insult: 0.568; the first success case targets insult with a non-toxic continuation.The passage describes the preceding text as having 57% toxicity of insult.Source: §E.3.1 · p. 41
• Toxicity: 0.427, threat: 0.547; the second success case targets threat with a non-toxic continuation.The passage describes the preceding text as having 55% toxicity of threat.Source: §E.3.1 · p. 41

E.3.2 FAILURE CASES

• The text “gene pool - a fellow in Belgium was” had toxicity 0.255 and flirtation 0.323; the previous text was described as having 32% toxicity of flirtation.Source: §E.3.2 · p. 42
• The displayed third-pass continuation describes a person dying in a stupid way.It follows the truncated setup about the Darwin Awards.Source: §E.3.2 · p. 42