PaperlayerView exact revision on arXiv

Source-linked AI summary

CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen

arXiv:2305.11738v4cs.CLcs.AI

TL;DR

LLMs can hallucinate, produce faulty code, and generate toxic content, while conventional mitigation often requires costly annotation or training. CRITIC has black-box LLMs use external tools to verify and revise outputs, and experiments show consistent gains across three tasks, with external feedback outperforming unsupported self-correction.

  • Problem

    LLMs exhibit hallucinations, faulty code, and toxic content, while existing mitigation methods require large-scale human annotation or data construction.

  • Method

    CRITIC uses in-context learning and external text-to-text tools to verify an initial output, generate critiques, and condition iterative corrections on that feedback.

  • Results

    CRITIC consistently improves performance across question answering, mathematical program synthesis, and toxicity reduction for multiple LLMs.

  • Takeaways & Limitations

    External tool feedback is crucial for consistent self-improvement because relying only on model self-correction can produce modest gains or deteriorate performance.

  • Takeaways & Limitations

    CRITIC’s effectiveness on other tasks and LLMs remains uncertain, and the experiments were limited to the textual modality.

Abstract

from arXiv · show

Recent developments in large language models (LLMs) have been impressive. However, these models sometimes show inconsistencies and problematic behavior, such as hallucinating facts, generating flawed code, or creating offensive and toxic content. Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging. Inspired by this observation, we introduce a framework called CRITIC that allows LLMs, which are essentially "black boxes" to validate and progressively amend their own outputs in a manner similar to human interaction with tools. More specifically, starting with an initial output, CRITIC interacts with appropriate tools to evaluate certain aspects of the text, and then revises the output based on the feedback obtained during this validation process. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs. Meanwhile, our research highlights the crucial importance of external feedback in promoting the ongoing self-improvement of LLMs.

1 INTRODUCTION

LLMs can produce hallucinations, faulty code, and toxic content, while conventional mitigation often requires costly annotation or task-specific training. CRITIC addresses this by using external tools to verify and correct outputs, improving performance across diverse tasks.

  • LLMs sometimes hallucinate facts, generate faulty code, or produce toxic content, undermining trust and real-world use.
  • CRITIC enables black-box LLMs to verify and rectify outputs through human-like interaction with external tools.The framework is presented as an alternative to methods requiring large-scale human annotation or data construction.
  • The verify-then-correct process can be iterated to support continuous output enhancement.
  • CRITIC improves performance across question answering, mathematical program synthesis, and toxicity reduction for multiple language models.
  • For ChatGPT, CRITIC yields 7.7 F1 enhancements across QA tasks, 7.0% absolute gains on mathematical reasoning tasks, and a 79.2% reduction in toxicity probability.

2 RELATED WORK

Related work frames CRITIC around truthfulness evaluation, natural-language feedback, tool-augmented language models, and an iterative verification-and-correction procedure. The examples illustrate how external evidence and execution feedback identify errors in answers and programs.

  • Truthfulness evaluation for open-ended text generation is less studied for black-box LLMs accessed through APIs.CRITIC addresses this gap by letting such models interact with external tools to verify their outputs.
  • Natural-language feedback is used in text editing and correction, while feedback can come from human evaluations, demonstrations, or existing corpora.
  • In QA examples, search evidence exposes an incorrect medal attribution, while interpreter feedback identifies implausible or incorrect pizza calculations.
  • Toxicity examples show CRITIC evaluating a continuation’s threat toxicity and generating a less toxic continuation.
  • Algorithm 1 generates an initial output, verifies it through tools to obtain critiques, and corrects it until the stopping condition or iteration limit is reached.
  • Tool-augmented language models use retrievers, search engines, calculators, and interpreters to supplement memorized knowledge and reasoning.

3 CRITIC: CORRECTING WITH TOOL-INTERACTIVE CRITIQUING

CRITIC combines in-context learning with text-to-text external tools to create interpretable verify-then-correct trajectories. The model conditions each correction on the original input, prior output, and tool-generated critique, enabling iterative refinement.

  • CRITIC first generates an output from the input and then repeatedly verifies and corrects it through a “Verify ⇒Correct ⇒Verify” cycle.The cycle continues until a specific stopping condition is met.
  • The framework uses chain-of-thought reasoning and few-shot in-context learning to guide frozen LLMs during inference.
  • Search engines, code interpreters, and APIs are represented as text-to-text functions interleaved with model generation.Search tools return parsed results, while interpreters return execution information and final results.
  • At each verification step, the model produces task-specific critiques of the previous output using the input, output, prompt, and external tools.Critiques can evaluate properties such as truthfulness, feasibility, or safety.
  • Corrections are generated from the input, previous output, and critiques, which identify errors, suggest actions, or provide credible grounding.

4 EXPERIMENTS

CRITIC is evaluated across question answering, mathematical program synthesis, and toxicity reduction using multiple LLMs and external verification tools. Across these settings, iterative tool-based correction consistently improves performance, with external feedback especially important for reliable correction.

  • Experimental Setup: CRITIC is evaluated on free-form question answering, mathematical program synthesis, and toxicity reduction using ChatGPT, Text-Davinci-003, and LLaMA-2 variants.The experiments use CRITIC and an oracle variant that corrects only inaccurate samples.
  • Question Answering: CRITIC improves QA over initial chain-of-thought results across datasets, settings, and LLMs, including F1 gains of 5.6 on Text-Davinci-003 and 7.7 on ChatGPT.The oracle CRITIC∗ gains are 10.3 and 12.4 F1, respectively.
  • Mathematical Program Synthesis: +4.7, +9.4, and +16.0 improvements are observed on TabMWP with 7B, 13B, and 70B models, showing stronger benefits from larger models.Without execution feedback, program correction becomes limited and unstable, including a 1.8-point decrease on Text-Davinci-003.
  • Toxicity Reduction: CRITIC substantially lowers toxic generations while preserving the vanilla LLMs’ fluency and diversity, achieving toxicity mitigation comparable to supervised methods without extra data or training.The results also identify external feedback as important because the LLM alone struggles to mitigate toxicity effectively.
  • Additional Analyses: Iterative correction generally improves outputs, but marginal benefits diminish after typically 2–3 rounds, while unreliable self-feedback produces inferior and inefficient returns.In QA, critique-conditioned generation outperforms rejection sampling by 4.5 and 3.3 EM for the two LLMs.

5 CONCLUSION

CRITIC is a plug-and-play framework that enables frozen LLMs to self-verify and self-correct through interaction with external tools. Experiments across diverse tasks and LLMs consistently show its effectiveness, generality, and interoperability.

  • CRITIC enables frozen LLMs to self-verify and self-correct by interacting with the external environment.
  • CRITIC uses external feedback to validate knowledge and improve answers through introspection without further training.
  • Experiments across diverse tasks and LLMs consistently demonstrate CRITIC’s effectiveness, generality, and interoperability.
  • The findings highlight LLMs’ unreliability in self-verification and the potential of external tool interaction to address this problem.

A LIMITATIONS & FUTURE WORK

CRITIC improves LLM outputs through tool interaction but introduces inference overhead, prompt dependence, and unresolved scope and safety boundaries. The paper also highlights transparency, bias, privacy, and the unreliability of self-correction without external feedback.

  • Inference Latency: Inference overhead grows linearly with the number of CRITIC iterations, although significant benefits can arise with even one iteration.Two correction rounds in mathematical program synthesis incur about twice the PoT baseline’s time overhead.
  • Prompt Engineering: CRITIC experiments rely on appropriate in-context demonstrations, and different prompt constructions may change the results.Future work should seek more efficient tool use without manually crafted demonstrations.
  • More Tasks and Settings: Effectiveness on other tasks and LLMs remains uncertain because suitable external feedback may not always be needed or usable.The experiments are limited to textual outputs, while language evaluation may not suit every modality.
  • Ethical Considerations: Steering LLMs toward malicious content creates risks that require detection and mitigation measures.The paper frames these risks as part of CRITIC’s ethical implications.
  • Trustworthiness and Transparency: Transparency is needed so users can understand model conclusions and verify corrections.This requirement concerns the verification and correction process itself.
  • Bias and Fairness: Both trained LLMs and external tools can introduce biases, requiring evaluation and mitigation to support fairness.The paper identifies model and tool biases as separate sources of concern.
  • Privacy and Security: External-tool API interactions raise data privacy and security concerns, motivating safeguards such as anonymization and secure communication.The paper treats protection of user information and sensitive data as a priority.
  • Self-Correction and External Feedback: Self-verification and self-correction are unreliable without external feedback, whereas CRITIC emphasizes tool-based feedback for consistent improvement.The paper attributes this unreliability to LLMs’ difficulty identifying what they know.

D MORE EXPERIMENTS AND DISCUSSION

The paper investigates whether LLMs can reliably assess the correctness of their own answers using confidence estimates. Experiments show weak self-verification, while tool interaction substantially improves factual discrimination.

  • Datasets and Metrics: The evaluation treats answers with F1 above 0.6 as correct and uses AUROC to measure discrimination.Fuzzy matching is selected as more consistent with human evaluation than exact matching for open-ended answers.
  • Self-Verification Reliability: LLMs struggle to distinguish the veracity of their own answers and cannot provide reliable confidence about what they know.The evaluation uses free-form question answering because it offers open-ended generation with clear ground truth.
  • Self-Verification Reliability: Self-Eval achieves only 54% accuracy on HotpotQA, performing slightly above random guessing and worse than Only-True on TriviaQA.Only-True has no discrimination ability, making the comparison especially unfavorable to Self-Eval.
  • Tool-Interactive Verification: CRITIC significantly improves factual discrimination through tool interaction, outperforming previous estimation methods with strong generality and interpretability.The verification procedure asks the model to critique its proposed answer and assess its correctness.

D.2.1 ERROR ANALYSIS ON FREE-FORM QUESTION ANSWERING

Manual HotpotQA error analysis shows that CRITIC reduces hallucinations and exposes evaluation problems, but reasoning, refusal, and incorrect-correction errors remain. Search failures and evidence misinterpretation explain some residual hallucinations.

  • Evaluation Design: Table 6 reports manually analyzed percentages of CRITIC and CoT success and failure modes on randomly selected HotpotQA samples.Its FN category denotes answers judged correct by humans but incorrect under the F1 > 0.6 indicator.
  • Success and Failure Modes: 36% versus 7%: CRITIC significantly reduces hallucinations but does not eliminate them.Remaining hallucinations arise when the search engine finds no useful evidence or the model misunderstands the evidence.
  • Success and Failure Modes: Most post-CRITIC errors involve reasoning mistakes, refusal to answer, and incorrect corrections.Refusal occurs when insufficient evidence supports a response and is treated as an expected truthfulness-preserving behavior.
  • Evaluation Reliability: 49% of CRITIC error samples reveal label ambiguities, inaccuracies, or outdated issues in HotpotQA.These false negatives indicate bias in automatic EM/F1 evaluation when human-correct answers are judged wrong.

D.2.2 ERROR ANALYSIS ON MATHEMATICAL PROGRAM SYNTHESIS

The error analysis categorizes initial mathematical-program errors and measures how effectively CRITIC corrects them, while also noting costs and task-specific tool choices.

  • Correction effectiveness: 57.4% of unreasonable-output errors and 35.3% of syntax errors are corrected by CRITIC.The correction settings allow up to four rounds in a non-oracle setting.
  • Error categories: 77.4% of initial PoT responses contain intrinsic reasoning errors, while 16.8% have unreasonable outputs and 5.8% have syntax or runtime errors.Intrinsic errors include misunderstanding the question or omitting conditions.
  • Correction effectiveness: CRITIC reduces errors in initially erroneous samples by 32.2% overall, but corrects intrinsic errors at only 26.7%.Reliable external feedback is harder to obtain for intrinsic reasoning errors.
  • Correction risks: Accuracy decreases by -4.3% on originally correct outputs, producing 14.3% wrong corrections after tool feedback.This result reflects applying correction in the non-oracle setting.
  • Tool use: The reported tools are free in these experiments, including the web-search infrastructure, local interpreter, and Perspective API.The paper describes tool use as requiring no charge in this setup.
  • Tool use: Web tools are especially important for knowledge-intensive and multi-hop QA, whereas code interpreters provide mathematical execution feedback without external knowledge.Tool significance varies by task.
  • Tool use: CRITIC uses a local code interpreter for mathematical program synthesis, while QA uses cached Google Search and web-page results.The interpreter supplies error messages and execution results; the web tool stores about 9GB of cached search results.

D.5 COMPLETE LLAMA-2 RESULTS

This section presents complete LLaMA-2 results for free-form question answering and mathematical program synthesis.

  • Free-form question answering: Table 8 reports LLaMA-2 results for free-form question answering, including oracle results that correct only incorrect answers.The table also compares against previous supervised state-of-the-art results.
  • Mathematical program synthesis: Table 9 reports LLaMA-2 results for mathematical program synthesis.The supplied passage identifies the table and task but provides no numerical results.

D.6 ADDITIONAL COMPARISON WITH SELF-CORRECTION WITHOUT TOOL-USE

The section compares CRITIC with self-correction without tool use and shows that tool-interactive correction is evaluated across multiple tasks and iterations.

  • Comparison with self-correction: Table 10 compares mathematical program synthesis results with PAL, Self-Refine, and an oracle Self-Refine setting.The asterisk denotes correction applied only to incorrect answers.
  • Comparison with self-correction: The reported comparison includes Codex w/ PAL at 71.3, Self-Refine at 26.7 (-44.6), and Self-Refine* at 76.2 (+4.9).These values are presented as the listed comparison results.
  • Iterative evaluation: Figures 6–9 track F1 and EM across CRITIC iterations for free-form QA with gpt-3.5-turbo and text-davinci-003.The supplied captions identify the metrics, models, and task but do not state the plotted outcomes.
  • Iterative evaluation: Figures 10–13 track solve rate or toxicity reduction across CRITIC iterations for the two language models.Figures 10–11 concern GSM8k solve rate; Figures 12–13 concern toxicity reduction.
  • Qualitative analysis: The section provides representative success and failure analyses for each task while omitting unimportant portions.The passage frames these analyses as qualitative examples accompanying the quantitative comparisons.

E.1 EXAMPLES OF FREE-FORM QUESTION ANSWERING

The examples show CRITIC using external evidence to correct factual, reasoning, and temporal errors, while also exposing cases where available evidence remains insufficient.

  • E.1.1 SUCCESS CASES: In a truthfulness case, search evidence corrects the proposed 2016 Russian national silver-medal pair and identifies Nikita Katsalapov with Victoria Sinitsina.The revised factual answer is supported by the search-based critique.
  • E.1.1 SUCCESS CASES: A factual QA example revises the answer to David Weissman, achieving EM/F1 of True/1.00.The case concerns a screenwriter with credits for “Evolution” who co-wrote The Family Man.
  • E.1.1 SUCCESS CASES: A reasoning-trace example uses searches about each band’s members and instruments to conclude that Badly Drawn Boy has the higher instrument-to-person ratio, with EM/F1 of True/1.00.The proposed answer is marked incorrect and revised to Badly Drawn Boy.

E.2.1 SUCCESS CASES

The success cases show CRITIC detecting runtime errors and unreasonable intermediate answers, then revising code until it reaches the gold answer. Its feedback identifies both execution failures and semantic plausibility problems.

  • Runtime-error correction: CRITIC identifies an undefined variable causing a NameError and recommends calculating num_of_fruits_in_10th_year before num_of_fruits_in_8th_year.The initial execution returns None, while the critique explicitly diagnoses the variable-ordering error.
  • Overall pattern: The examples include correcting both runtime errors and unreasonable values through two iterations.The section listing labels this pattern as a CRITIC success case on GSM8k.
  • Semantic correction: CRITIC flags 6.66667 guests as unreasonable because the number of guests must be an integer.The first correction produces 6.66667, which is marked incorrect against the gold answer of 10.0.
  • Semantic correction: A second iteration corrects the venue calculation, producing 10.0 against the gold answer of 10.0.The revised execution is explicitly marked correct.

E.2.2 FAILURE CASES

The failure cases show that CRITIC can validate incorrectly or produce an incorrect correction. Errors include accepting flawed code and returning a wrong result after a timeout-related diagnosis.

  • Wrong verification: CRITIC incorrectly declares flawed dance-percentage code correct, although the subsequent correction reaches the gold answer of 60.0.The initial verification says the assignment is correct, while the later output is marked correct against the gold answer.
  • Wrong verification: The dance example’s initial output is 55.0, based on subtracting 25 rather than applying 25% to the remaining students.The supplied passages show the code and its output but do not state this full arithmetic diagnosis explicitly.
  • Wrong correction: The snowball program times out because its loop is effectively infinite under the stated condition and updates.The critique identifies the loop condition as problematic and notes that the melting rate must be converted from 15-minute to hourly units.
  • Wrong correction: The proposed snowball correction returns 3.0 while the gold answer is 5.0, so the correction remains incorrect.The execution is explicitly marked false.
  • Scope warning: The section also includes an explicit warning that the surrounding content contains LLM-generated offensive or sensitive material.This warning marks a content-safety boundary for the examples.

E.3.1 SUCCESS CASES

The toxicity-reduction success cases show CRITIC generating alternative continuations that lower measured toxicity-related scores while maintaining the stated goal of fluency.

  • Reduce insult: CRITIC produces a different non-toxic continuation for an insult case, where the previous text has 57% toxicity of insult.The example is labeled a success case for reducing insult.
  • Reduce threat: CRITIC produces a different non-toxic continuation for a threat case, where the previous text has 55% toxicity of threat.The example is labeled a success case for reducing threat.

E.3.2 FAILURE CASES

The toxicity-reduction failure cases show that CRITIC does not consistently improve hard examples: one case decreases toxicity, while another remains unchanged across iterations.

  • Hard case: In the hard case, toxicity decreases to 32% by iteration 3, but the continuation is still presented within a failure-case example.The corresponding passage reports a 32% toxicity value for the previous text.
  • Hard case: The hard example begins with a Darwin Awards passage involving someone removing himself from the gene pool.The source text identifies the initial continuation and its framing.

F PROMPTS

The appendix presents few-shot prompts spanning chain-of-thought, ReAct, and CRITIC examples across question answering, program synthesis, and answer verification. These examples illustrate both initial reasoning and tool-informed correction of factual, numerical, and evaluative answers.

  • F PROMPTS: The appendix provides few-shot prompts for CRITIC and baseline methods across all evaluated tasks.
  • F.1.1 CHAIN-OF-THOUGHT (COT): Chain-of-thought prompts answer questions about people, works, places, history, geography, and shared professions through step-by-step reasoning.
  • F.1.1 CHAIN-OF-THOUGHT (COT): Additional chain-of-thought examples cover paintings, architecture, sports, composers, philosophers, magazines, occupations, rivers, and currencies.
  • F.1.2 REACT: The ReAct prompts interleave questions with searches and observations before finishing answers, including examples on Olympic skating, soccer, barium, and film roles.
  • F.1.2 REACT: The USA soccer example uses retrieved evidence to identify 2018 as the team’s last missed World Cup before returning in 2022.
  • F.1.3 CRITIC: CRITIC corrects the proposed Flintstones answer by distinguishing Elizabeth Perkins’s role as Wilma from Elizabeth Taylor’s role as Wilma’s mother, Pearl Slaghoople.
  • F.2.2 CRITIC: A mathematical-program example judges an answer reasonable after checking that the code includes the relevant information, while noting possible improvements in concision and readability.
  • F.4.2 CRITIC: CRITIC examples classify answers as probably wrong or absolutely wrong when evidence contradicts them or reasoning contains an inaccurate date.
Loading 2305.11738v4…