PaperlayerView exact revision on arXiv

Source-linked AI summary

CriticBench: Benchmarking LLMs for Critique-Correct Reasoning

Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, Yujiu Yang

arXiv:2402.14809v4cs.CLcs.AIcs.LG

TL;DR

Research on LLM critique and correction has not yet provided a comprehensive evaluation across models and tasks. CRITICBENCH assesses generation, critique, and correction across five domains, finding task-dependent correction and inconsistencies across these abilities. The results also show that weaker models can sometimes outperform stronger models in self-critique.

  • Problem

    Prior studies covered a narrow range of models and datasets and reported inconsistent findings, leaving LLM critique and correction without a comprehensive evaluation.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    CRITICBENCH evaluates generation, critique, and correction using 15 datasets across five domains and experiments with 17 LLMs.

    Source: §1 · p. 2

  • Results

    Correction varies by task, GQC capabilities show linear correlation alongside inconsistencies, and weaker models can sometimes correct stronger models’ outputs more effectively than they self-correct.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    CRITICBENCH provides a basis for evaluating and analyzing LLM generation, critique, and correction capabilities across task types.

    Source: §1 · p. 2, §5 · p. 9

    The evaluation relies on costly human annotations and would benefit from more nuanced critique metrics across tasks and evaluation settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.

1 Introduction

Prior work had not provided a comprehensive account of LLM critique and correction across models and tasks. CRITICBENCH evaluates generation, critique, and correction across five domains, revealing task-dependent correction, GQC inconsistencies, and cross-model critiquing differences.

1 Introduction

  • Earlier studies covered narrow ranges of models and datasets and reported inconsistent findings, motivating a more systematic evaluation.Source: §1 · p. 1
  • CRITICBENCH evaluates critique and correction across 15 datasets in five domains and tests 17 LLMs.The benchmark contains 3.8K instances, with responses from eight LLaMA, Vicuna, and GPT models and additional GPT-4 review.Source: §1 · p. 2
  • Correction performance varies by task: LLMs struggle more with detail-oriented algorithmic tasks than logic-centric code generation.Source: §1 · p. 2
  • Models’ correct responses are inconsistent across generation, critique, and correction, with stronger models showing greater GQC consistency.Source: §1 · p. 2
  • Stronger models critique better overall, but weaker models can evaluate stronger models’ responses and sometimes outperform them in self-critique.Source: §1 · p. 2

2 Related Work

Prior work examined reasoning prompts and tools, as well as multiple approaches to LLM critique and correction. The paper identifies a lack of comprehensive comparative evaluation and positions CRITICBENCH as a broad analysis across generation, critique, and correction.

LLM Reasoning

  • Reasoning research has explored few-shot and Chain of Thought prompting, other prompting methods, and external tools such as search engines and Python interpreters.Source: §2 · pp. 2–3

LLM Critiquing & Correction

  • Prior critique-correction studies used internal or external feedback, multiple-model debate, or trained critique models, but lacked comprehensive evaluation.Source: §2 · p. 3

Critiquing Tasks

  • Earlier critique datasets typically targeted particular tasks and models, while CRITICBENCH compares LLM generation, critique, and correction more broadly.Source: §2 · p. 3

3 The CRITICBENCH

CRITICBENCH evaluates critique and correction across diverse tasks, using varied source models and reviewed responses. Its evaluation prompts models to critique a response and then produce a correction informed by that critique.

3.1 Overview of CRITICBENCH

  • The benchmark follows four collection principles: varied task types, diverse response-generating models, acknowledged datasets, and GPT-4 plus manual data review.The overview describes these principles as supporting broad ability coverage, response variety, comparisons with generation capabilities, and data quality.Source: §3.1 · p. 3

3.2 Question Collection

  • CRITICBENCH covers mathematical, commonsense, symbolic, coding, and algorithmic tasks across 15 datasets.The collection uses GSM8K, MATH, AQuA, TabMWP; CSQA, AmbigNQ, StrategyQA, HotpotQA; Penguins, Colored Object, Date; MBPP, HumanEval; and Object Counting and Repeat Copy. Sampling mixes random subsets with complete datasets.Source: §3.2 · p. 4

3.3 Response Collection

  • Responses from GPT-3.5, GPT-4, LLaMa2, and Vicuna variants were generated with greedy decoding, filtered for valid reasoning, and sampled to balance counts across datasets.The source-model set includes LLaMa2 7B, 13B, and 70B, and Vicuna 7B, 13B, and 33B.Source: §3.3 · p. 4

3.4 Response Annotation

  • Response correctness combines rule-based matching with GPT-4 precision checks, and disagreements are resolved by manual review.The review also identified Date-dataset questions lacking correct options.Source: §3.4 · p. 4

3.5 Evalution

  • Generation and correction are measured by accuracy, while critique uses F1 to assess error identification.Accuracy is the number of correct predictions divided by the total questions; critique F1 combines precision and recall, with the paper motivating it as more robust and reproducible amid class imbalance and evaluation biases.Source: §3.5 · p. 5

4 Experiments

The experiments assess generation, critique, and correction across several model families and prompts, finding that their relationships and effectiveness vary with training strategy and task type. They also reveal inconsistencies across these capabilities and differences in how models critique one another.

4.1 Experimental Setup

  • The setup evaluates generation by accuracy, critique by F1 on fixed responses, and correction by accuracy after critique.The study uses zero-shot prompts and few-shot settings for weaker-instruction-following pretrained models.Source: §4.1 · pp. 5–6

4.2.1 RQ1: Key Factors in LLM Critical Reasoning

  • Models above 13 billion parameters surpass random-guess critique, while effective correction requires Mixtral-8×7b or larger; RLHF improves critique and correction, while CT beats GPT-3.5 in critique at 13B.Phi-2 performs worse at critique than similarly capable generators, despite excelling in generation.Source: §4.2.1 · p. 6
  • Vicuna-13b’s ZS-AO flagged 22.04% of responses as incorrect versus 4.8% with ZS-CoT, against a 51.63% actual error rate; few-shot outperformed zero-shot in critique and correction.Oracle feedback improved correction, but for LLaMa-2-70b through GPT-4, oracle corrections still fell short of direct generation.Source: §4.2.1 · pp. 6–7

4.2.2 RQ2: Correlations of GQC Abilities

  • Generation and critique show a positive linear relationship, but generation and correction are less tightly related, and models may identify errors without fixing them.Weaker models can reduce response correctness during correction.Source: Figure 6 · p. 7, §4.2.2 · p. 7

4.2.3 RQ3: Impact of Task Type

  • Critique and correction are weaker on detail-oriented algorithmic tasks; math correction improves only above 41% generation accuracy, while code correction improves once generation beats original accuracy.Symbolic reasoning also shows weaker critique than generation and correction.Source: §4.2.3 · pp. 7–8, Figure 7 · p. 8

4.2.4 RQ4: Consistency of GQC Knowledge

  • GQC knowledge remains inconsistent: as model size increases, knowledge coherence and complete failures decrease, while some models answer correctly yet misjudge response correctness.Correctly critiqued responses that are not correctly generated or corrected form a significant portion.Source: §4.2.4 · pp. 8–9

4.2.5 RQ5: Patterns of Inter-Model Critique

  • Strong models critique better overall and weak-model errors are easier to identify, yet some weaker models approach or exceed stronger models’ self-critique scores.After critique-focused training, Auto-J and UltraCM improve cross-model assessment; UltraCM approaches its self-critique level against GPT-4.Source: Figure 8 · p. 8, §4.2.5 · p. 9

5 Conclusion

The conclusion synthesizes CriticBench’s findings on GQC relationships, task-dependent performance, and cross-model critique. It emphasizes that weaker models can sometimes correct stronger models’ outputs more effectively than those models self-correct.

5 Conclusion

  • The analysis finds a linear correlation and subtle inconsistencies among generation, critique, and correction capabilities.Source: §5 · p. 9
  • Models perform better in critique and correction on logic-focused tasks than on tasks requiring attention to detail.Source: §5 · p. 9
  • Weaker models can sometimes correct stronger models’ outputs more effectively than those models self-correct.Source: §5 · p. 9

Limitations

The paper uses binary discrimination results to assess critique, citing generality and annotation concerns with finer-grained scoring. It identifies costly human annotation and the need for more nuanced critique metrics as future-work concerns.

  • Binary results are used to assess critique because fine-grained principle-based scores are task-specific and evaluation principles may vary across tasks.The paper also notes that exhaustively listing evaluation principles is idealistic.Source: Limitations · p. 9
  • Principle-based scores rely on human annotations or GPT-4 validation, which the paper says can be costly or subject to errors and biases.Source: Limitations · p. 9
  • The paper calls for evaluation methods that reduce reliance on costly human annotations and critique metrics that capture more task- and scenario-specific performance.The authors frame these as directions for future work.Source: Limitations · p. 9

Ethics Statement

The ethics statement notes that CRITICBENCH uses existing public datasets and models, with annotations from GPT-4 and human evaluation. It also flags residual response errors and potential bias risks in critique use.

Ethics Statement

  • The benchmark is constructed from existing public datasets and models, with results annotated using GPT-4 and human evaluation.Source: Ethics Statement · p. 9
  • Despite rule-based filtering and GPT-4 and human review, unpredictable errors may remain in model-generated responses.Source: Ethics Statement · p. 9
  • When GQC abilities are inconsistent and critique exceeds generation and correction, critique judgments may contain harmful biases that require careful discernment.Source: Ethics Statement · p. 9

A Statistics of CRITICBENCH

Table 2 is identified as reporting statistics for the datasets used in CRITICBENCH.

A Statistics of CRITICBENCH

  • Table 2 reports statistics for the datasets used in CRITICBENCH.Source: Table 2 · p. 15

B Annotation Example

The appendix presents examples of annotation inconsistencies and excluded invalid responses, then describes a manual quality check that identified a Date-dataset question with faulty options.

B Annotation Example

C Example of Invalid Response

  • Table 6 gives an example of an invalid response excluded from the dataset.Source: Table 6 · p. 17

D Manual Quality Check

  • Manual checks found that all Date-dataset questions about Jane and John marrying on Jan 2, 1958, had option issues; Table 7 is identified as an example with no correct options.The authors say they conducted manual checks to ensure label correctness.Source: D Manual Quality Check · p. 17, Table 7 · p. 17

E Examples of Detail-Oriented and Logic-Oriented tasks

The paper distinguishes detail-oriented tasks, which demand careful attention to question conditions, from logic-oriented tasks, which require logical reasoning. Algorithmic tasks emphasize detail, mathematics combines detail and logic, and code generation is categorized as logic-oriented.

E Examples of Detail-Oriented and Logic-Oriented tasks

F Detailed Results on Different Tasks

This section presents performance results across five reasoning task categories and gives examples of the generation, critique, and correction prompts used. The prompt examples include verdicts for factual and coding answers and a corrected factual answer.

F Detailed Results on Different Tasks

  • The cited table captions identify task-performance reporting for mathematical reasoning, commonsense reasoning, symbolic reasoning, code generation, and algorithmic tasks across Tables 10 through 14.Source: Table 10 · p. 19, Table 11 · p. 19, Table 12 · p. 20
  • Figures 9 and 10 present model performance across different task types, without the supplied captions specifying particular comparisons or outcomes.Source: Figure 9 · p. 21, Figure 10 · p. 22

G.1 Generation prompt

G.2 Critique prompt

  • The critique prompts ask whether an answer is correct, request problems when it is wrong, and require a correct-or-wrong verdict.Examples include a factual answer marked wrong and a code answer marked wrong after identifying a mismatch with the task.Source: §G.2 · pp. 25, 28

G.3 Correction prompt

  • The correction prompt asks models to improve an answer based on the critique and return a concise final answer.In the Manhattan Project example, the corrected answer gives 1942 as the start year and 1946 as the end year.Source: §G.3 · p. 32
Loading 2402.14809v4…