CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation

1 Introduction

• High-quality evaluation critiques, including scores and explanations, are expected to provide scalable feedback that guides persistent improvements in LLM generation.Source: §1 · p. 1
• Existing evaluation models struggle to produce informative critiques for pointwise grading and pairwise comparison, particularly in reference-free settings.The paper reports that reference-free critiques tend to lack fine-grained distinguishability, alongside unsatisfactory evaluation performance.Source: §1 · p. 1
• Eval-Instruct constructs informative evaluation data for pointwise grading and pairwise comparison, with and without references.It first obtains pointwise critiques using pseudo references, then applies pointwise-to-pairwise and referenced-to-reference-free prompting strategies; cross-validation is used for reference-free pairwise comparison.Source: §1 · pp. 1–2
• On three instruction-following benchmarks, CritiqueLLM outperforms all open-source baselines and is comparable to GPT-4 in system-level pointwise-grading correlations.The model is fine-tuned on data constructed by Eval-Instruct.Source: §1 · p. 2
• The authors report that CritiqueLLM's generated critiques can act as scalable feedback to improve LLM generation quality.Source: §1 · p. 2

2 Related Work

• NLU-style evaluation uses tasks such as multiple-choice QA and metrics such as accuracy or F1, which may not reflect LLMs' ability to generate responses to user queries.Source: §2 · p. 2
• NLG-style evaluation extends generation metrics, and recent LLM-based metrics are used to assess LLM performance; this paper focuses on that line of work.Researchers also train evaluation models on data collected by prompting GPT-4, aiming for lower-cost, more stable evaluations.Source: §2 · p. 2
• Against evaluation-specific models including PandaLM, JudgeLM, and AUTO-J, the authors present Eval-Instruct as addressing uninformative critiques by exploiting links across tasks and settings.They report that the resulting data improves generated critique quality.Source: §2 · p. 2

3.1 Task Definition and Method Overview

• Eval-Instruct first obtains GPT-4 pointwise grading critiques with pseudo references, then uses multi-path prompting to construct data for other evaluation tasks and settings.The method covers pointwise grading and pairwise comparison, with and without references, and fine-tunes a unified critique-generation model on the constructed data.Source: §3.1 · p. 3

3.2.1 Pseudo Reference Collection

• The authors augment seed queries across ten instruction-following tasks, collect generations from ten models, and manually check GPT-4 generations to create pseudo references.They also strictly filter the seed queries and remove a small number of failed generations, such as empty responses.Source: §3.2.1 · p. 3

3.2.2 Multi-Path Prompting

• The two prompting strategies create pairwise and reference-free critiques from referenced pointwise grading data, using alternate paths and cross-validation to filter contradictory labels.Pointwise-to-pairwise prompting injects pointwise critiques into pairwise critiques; referenced-to-reference-free prompting removes direct reference comparison while retaining informative content.Source: §3.2.2 · p. 3, Figure 1 · p. 4

3.2.3 Cross Validation

• 7.7% of evaluation data were filtered because the two paths produced contradictory comparison labels; consistent examples retain critiques from both paths.The authors report this as evidence that most constructed examples had consistent labels.Source: §3.2.3 · p. 5

3.3 Supervised Fine-Tuning

• CritiqueLLM is fine-tuned jointly on all constructed data, with simplified task prompts and pairwise examples augmented by swapping the generated texts and corresponding critique contents.The amount of reference-free pairwise data in the objective is measured after cross-validation.Source: §3.3 · p. 5

4.1 Dataset

• The three benchmarks cover the paper’s evaluation tasks and settings, with human-annotated pointwise scores or pairwise preferences across varied instruction-following scenarios.AUTO-J (Eval-P) has 1,392 pairs; LLMEval has about 100,000 pairs, from which about 1,000 were sampled for evaluation because of computational and API-cost limits.Source: §4.1 · p. 5, Table 1 · p. 5
• The benchmark data have different relationships to training: AUTO-J training data are included but its test set is excluded, while LLMEval has no similar task or data distribution.The authors state that string matching found no overlap between training and test queries.Source: §4.1 · pp. 5–6

4.2 Baselines

• Baselines include general LLMs prompted as evaluators and task-specific AUTO-J-Bilingual-6B and JudgeLM-13B models.The general baselines include ChatGPT, GPT-4, and open-source models; the selected task-specific models target particular evaluation tasks and settings.Source: §4.2 · p. 6

4.3 Implementation Details

• CritiqueLLM is trained on 8 A800 GPUs for 5 epochs, using AdamW with a peak learning rate of 6e-5 and maximum sequence length 8,192; main results use greedy decoding.Source: §4.3 · p. 6

4.4.1 Pointwise Grading

• CritiqueLLM performs comparably with GPT-4, especially in system-level pointwise correlations, and outperforms ChatGPT and all open-source baselines.System-level correlations are reported as nearly approaching 1.0, indicating near-distinction of the overall performance of eight LLMs.Source: §4.4.1 · p. 7

4.4.2 Pairwise Comparison

• CritiqueLLM beats ChatGPT and all open-source baselines in pairwise agreement and consistency, with performance comparable to GPT-4 especially in consistency.Agreement requires both consistent results after swapping text order and alignment with the human preference label.Source: §4.4.2 · p. 7, Table 3 · p. 7

4.5 Analysis on Critique Quality

• GPT-4 evaluation with human verification finds CritiqueLLM’s critiques superior to ChatGPT’s and CritiqueLLM (DP), and comparable to GPT-4’s.The comparison assesses correctness, helpfulness, and informativeness, in that priority order.Source: §4.5 · pp. 7–8, Figure 2 · p. 7

4.6 Analysis of Critique as Feedback

• CritiqueLLM’s critiques improve generated-text quality in most tasks, but seem to degrade performance in logical reasoning, mathematics, and advanced Chinese understanding.The authors attribute these cases to misleading critiques about reasoning processes and leave further investigation for future work.Source: §4.6 · p. 8
• The preliminary feedback study notes that high-quality human critiques are difficult to collect for AlignBench, stronger LLMs resist further improvement, and weaker LLMs struggle to follow critiques.The authors identify improving critique collection and helping weaker models follow critiques as future work.Source: §4.6 · p. 8

4.7 Ablation Study

• Removing cross-validation degrades performance especially in pairwise comparison, while removing critique explanations degrades both pointwise and pairwise evaluation.The authors say cross-validation filters low-quality evaluation data and explanations contribute positively to final performance.Source: §4.7 · p. 9, Table 5 · p. 9
• Self-consistency decoding can enhance pointwise grading, while greedy decoding performs best in pairwise comparison and is comparable in pointwise grading at lower computational cost.The evaluated alternatives also include beam search and nucleus sampling.Source: §4.7 · p. 9

• Eval-Instruct constructs informative evaluation data for pointwise grading and pairwise comparison, with and without references.Source: §5 · p. 9
• CritiqueLLM beats ChatGPT and all open-source baselines, with comparable performance to GPT-4 in system-level correlations of pointwise grading.Source: §5 · p. 9
• CritiqueLLM's critiques provide scalable feedback that can improve LLM generation quality.Source: §5 · p. 9

• Multi-path prompting lengthens input prompts and raises API costs when constructing evaluation data, although acquisition is single-round.Source: Limitations · p. 9
• The evaluation model has self-evaluation bias, meaning preference for texts generated by the same base model; the authors suggest using evaluators with different base models.Source: Limitations · p. 9
• The authors leave further investigation of self-evaluation bias for future work because no satisfactory solution currently exists.Source: Limitations · p. 9

• Table 6 provides prompts for query augmentation and scoring.Source: A Query Augmentation and Scoring Prompts · p. 12
• For query generation, ChatGPT receives in-context examples and detailed requirements to produce augmented queries and assign category labels.Source: A Query Augmentation and Scoring Prompts · p. 12
• During evaluation, ChatGPT assigns each query a difficulty score to balance difficulty across the augmentation dataset.Source: A Query Augmentation and Scoring Prompts · p. 12

• The appendix provides original prompts for pointwise-to-pairwise and referenced-to-reference-free strategies.Source: B Prompt Design for Eval-Instruct · p. 12
• The pointwise-to-pairwise strategy has an original prompt in Table 7.Source: B Prompt Design for Eval-Instruct · p. 12
• English translations of the strategy prompts appear in Tables 8 and 10.Source: B Prompt Design for Eval-Instruct · p. 12

C Case Study on Critique Generation

• The case study provides generated examples for both pointwise grading and pairwise comparison.The examples are presented in Tables 11 and 13, with English translations in Tables 12 and 14.Source: C Case Study on Critique Generation · p. 12
• The prompt-design tables cover converting pointwise critiques to pairwise form and converting referenced prompts to reference-free prompts.Tables 7–10 present these designs in the original and translated versions.Source: Table 7 · p. 14, Table 8 · p. 15, Table 9 · p. 16
• Reference-free pointwise examples compare critiques from ChatGPT, GPT-4, and CritiqueLLM.The English-translated examples are in Table 12.Source: Table 11 · p. 18, Table 12 · p. 19
• Reference-free pairwise examples compare critiques from ChatGPT, GPT-4, and CritiqueLLM.The English-translated examples are in Table 14.Source: Table 13 · p. 20, Table 14 · p. 21