PaperlayerView exact revision on arXiv

Source-linked AI summary

CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists

Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, Najoung Kim

arXiv:2403.18771v3cs.CL

TL;DR

LLM-as-a-Judge can produce inconsistent ratings because subjective Likert-scale criteria yield low agreement and high variance across evaluator models. CheckEval decomposes criteria into binary checklist questions, and experiments report stronger human correlation alongside improved evaluator agreement and reduced variance.

  • Problem

    Subjective Likert-scale judging can yield low agreement and high variance across evaluator models, complicating reliable evaluation of generated text.

    Source: §1 · p. 1

  • Method

    CheckEval decomposes evaluation criteria into fine-grained Boolean checklist questions for structured yes/no assessment.

    Source: §1 · p. 2

  • Results

    Across 12 LLMs and multiple datasets, CheckEval improves correlation with human evaluation over Likert-based G-Eval and SEEval, while also increasing inter-evaluator agreement and reducing score variance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Checklist decisions make the criteria met in an evaluation traceable, supporting more interpretable scores.

    Source: §1 · p. 2

    The study does not integrate evaluation signals into model training, which is outside its scope.

    Source: §2.3 · p. 2

Abstract

from arXiv · show

Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models. We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols. To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments. More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance. CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments.

1 Introduction

LLM-as-a-Judge offers a scalable alternative to human evaluation, but subjective Likert scoring can produce low agreement and high variance across evaluator models. CheckEval replaces holistic ratings with fine-grained binary checklist questions and reports improved human correlation, agreement, and consistency across evaluators.

1 Introduction

  • Human evaluation remains the gold standard but is costly, time-consuming, and difficult to scale, while ROUGE and BLEU align poorly with human judgments.These constraints motivate alternatives that can approximate human evaluation more reliably.Source: §1 · p. 1
  • Existing LLM-as-a-Judge protocols often ask models to assign subjective Likert scores, which can yield low inter-evaluator agreement and high score variance.Evaluators may struggle to distinguish high-quality texts, and results are sensitive to evaluator-model choice.Source: §1 · p. 1
  • CheckEval decomposes evaluation criteria into fine-grained Boolean questions, making decisions more explicit and tracking which criteria are met.The framework targets both explainability and consistency through structured binary responses.Source: §1 · p. 2
  • Across 12 LLM evaluators and multiple datasets, CheckEval improves correlation with human evaluation over Likert-based G-Eval and SEEval.The framework also reports higher inter-evaluator agreement and lower score variance.Source: §1 · p. 2

2 Related Work

Prior work spans LLM-based judging, decompositional evaluation, and reliability analysis. CheckEval is positioned as a fine-grained pointwise approach, while the paper distinguishes evaluation robustness from using judge outputs to train models.

2.1 LLM-as-a-Judge

  • Traditional ROUGE and BLEU have limitations tied to reference-text reliance, while checklist-based testing was previously proposed for NLP models.Source: §2.1 · p. 2
  • Pointwise LLM-as-a-Judge assigns absolute scores to individual outputs, but broad dimensions and single numeric ratings can produce inconsistent judgments.Pairwise evaluation compares two outputs but becomes computationally expensive as comparisons scale.Source: §2.1 · p. 2

2.2 Decompositional Approaches

  • Decomposing evaluation criteria into fine-grained Boolean checklists follows work showing that atomic units can reduce subjectivity in factual-consistency judgments.CheckEval applies this decompositional approach to evaluation clarity and ambiguity.Source: §2.2 · p. 2

2.3 Reliability of Evaluation

  • Reliability work considers inter-evaluator agreement and score variance alongside correlation; CheckEval reports improvements across diverse LLM evaluators.The paper focuses on evaluation robustness and explainability, not integrating evaluation signals into model training.Source: §2.3 · pp. 2–3

3 Method

CheckEval turns high-level text-quality criteria into a checklist of binary questions. It defines dimensions and sub-dimensions, expands and filters questions, then scores texts from yes/no responses while preserving criterion-level explanations.

3 Method

  • CheckEval has three stages: define evaluation dimensions, generate a checklist, and evaluate the target text with yes/no answers.The binary format is intended to clarify criteria and reduce ambiguity compared with Likert scoring.Source: §3 · p. 3

3.1 Defining Dimensions of Evaluation

  • Dimensions may come from benchmark definitions or be specified for a task, then are broken into sub-dimensions grounded in the original definitions.For summarization fluency, examples include formatting, grammar, completeness, and readability.Source: §3.1 · p. 3

3.2 Checklist Generation

  • Seed questions use yes/no answers; LLM augmentation diversifies perspectives or elaborates detail, and filtering checks alignment, dimension consistency, and redundancy.The authors use human-selected sub-dimensions because fully automated generation sometimes misaligned with benchmark criteria; they report improved human correlation after filtering.Source: §3.2 · pp. 3–4, §3.1 · p. 3

3.3 Checklist-based Evaluation

  • The evaluator answers grouped checklist questions, and the score is the uniformly weighted proportion of yes responses.Multiple questions are asked together to reduce computational cost; checklist responses make the criteria behind scores traceable without additional rationale generation.Source: §3.3 · pp. 4–5

4 Experimental Setup

The experiments compare checklist evaluation with Likert-based baselines across summarization and dialogue benchmarks, using 12 evaluator models and standardized prompting settings.

4.1 Datasets and Metrics

  • SummEval and Topical-Chat provide human-rated benchmarks for summarization and knowledge-grounded dialogue; correlations use sample-level scores for SummEval and turn-level scores for Topical-Chat.The reported metrics are Pearson’s r, Spearman’s ρ, and Kendall’s τ; QAGS is reported only in Appendix B.1 because it covers factual consistency alone.Source: §4.1 · p. 5

4.2 Baselines

  • G-Eval and SEEval are the main Likert-style baselines, with SEEval adding brief self-explanations before its rating.G-Eval is included as a widely adopted, reproducible baseline with a relatively simple setup; the comparisons also include stronger methods but are not apples-to-apples.Source: §4.2 · p. 5

4.3 Models

  • The 12 evaluators comprise open-source large, medium, and small models plus GPT-4-Turbo, GPT-4o, and GPT-4o-mini.The open-source size groups span 7–9B, 22–32B, and 70–123B models.Source: §4.3 · p. 5

4.4 Implementation Details

  • All three methods use temperature 0, one sample, and a fixed seed; CheckEval evaluates multiple checklist questions in one prompt for efficiency.Maximum lengths are 20 for G-Eval, 500 for SEEval, and 200 for CheckEval; the original baseline prompts are used.Source: §4.4 · p. 5

5 Results

Across SummEval and Topical-Chat, CheckEval generally correlates more strongly with human judgments than the Likert-style baselines and achieves higher inter-evaluator agreement. Its agreement advantage holds alongside correlation, addressing a reliability dimension that correlation alone does not capture.

5.1 Correlation with Human Evaluation

  • Mistral-Large with CheckEval reaches ρ = 0.55 on SummEval and r = 0.65 on Topical-Chat, the highest correlations among the evaluated models.Across 12 LLMs, CheckEval generally exceeds G-Eval and SEEval, with a few exceptions for Qwen2.5 and Mistral-Small; some GPT-4o, medium, and small-model results also favor CheckEval.Source: §5.1 · p. 6

5.2 Inter-evaluator Agreement (IEA)

  • For large models, CheckEval reaches IEA scores of 0.67 (α and κ) on both SummEval and Topical-Chat, compared with lower agreement for G-Eval and SEEval.This level on SummEval is comparable to human-rater agreement of κ ≈0.7; CheckEval maintains both high correlation and IEA across models and tasks.Source: Table 2 · p. 6, §5.2 · p. 7
  • The authors attribute inconsistent holistic ratings to bundled fine-grained criteria and poorly calibrated adjacent Likert scores; an appendix analysis finds the IEA gain is not solely an output-format effect.IEA treats evaluator models within a capacity group as annotators, measuring whether they assign similar ratings to the same input.Source: §5.2 · pp. 6–7

6 Human Validation

Human studies assess checklist quality and compare checklist-based scores from people and LLMs. They report high-quality augmented questions, statistically significant alignment with human judgments, and high agreement across human and LLM raters.

6 Human Validation

  • Two human studies assess checklist generation and compare LLM scores with scores from people applying the CheckEval protocol.The protocol comparison analyzes both correlation with human scores and inter-rater agreement.Source: §6 · p. 7, §6.2 · p. 8

6.1 Validation of Checklist Generation Process

  • Checklist augmentation scores exceed 90% on both datasets, while filtering scores are slightly lower but remain competitive.Annotators often expected one or two additional questions per dimension to be filtered, mostly because they overlapped semantically.Source: §6.1 · pp. 7–8
  • Removing one or two additional questions per dimension produced extremely similar Mistral-Large correlations, with only minor drops.The follow-up used human-curated filtering and indicates that the automatic filtering worked effectively in practice.Source: §6.1 · p. 8

6.2 Validation of CheckEval Protocol

  • CheckEval scores correlate significantly with human judgments, and agreement is high among human raters and between human and large-LLM groups.The correlation study sampled 20 summaries; the agreement study used 100 summaries and focused on relevance because of annotation cost.Source: §6.2 · p. 8, Figure 3 · p. 8

7 Analysis

The analysis examines evaluator-model stability, a naturalness case study, and the contribution of checklist-generation components. It reports more stable correlations for CheckEval, illustrates the interpretability of its binary decisions, and finds both augmentation and filtering contribute to performance.

7.1 Stability Analysis of Evaluation Methods

  • CheckEval has higher mean correlation and lower variance than G-Eval and SEEval on both datasets across evaluator models.Stability here concerns alignment with human annotations across evaluator models, rather than agreement on the same samples.Source: §7.1 · p. 9

7.2 Case Study

  • On a Topical-Chat naturalness example, CheckEval scores 0.88 versus 1.0 for human judgments, closer than G-Eval’s 0.25.Its yes/no subquestions identify attributes such as avoiding repetition and message clarity that contribute to the score.Source: §7.2 · p. 9, Table 6 · p. 9

7.3 Ablation Study

  • Removing either filtering or augmentation reduces overall performance, with augmentation having a slightly larger impact.The study also examines whether increasing baseline inference budgets can close the performance gap.Source: §7.3 · p. 9

8 Conclusion

CheckEval evaluates generated text by decomposing criteria into fine-grained Boolean questions. Across models and datasets, it reports stronger human alignment and evaluator agreement than Likert-based methods, alongside explainability and reduced variance.

8 Conclusion

  • CheckEval converts evaluation criteria into structured binary questions for evaluating open-ended text.The framework is described as fine-grained and Boolean QA-based.Source: §8 · p. 9
  • CheckEval achieves higher correlation with human judgments and IEA than Likert-based methods across different LLM evaluators.The conclusion also reports reduced variance across LLMs.Source: §8 · p. 9
  • CheckEval’s traceable binary decisions support explainability and analysis of attributes driving quality judgments.Source: §8 · p. 9

9 Limitations

The paper identifies limits in checklist alignment, binary judgments, task coverage, and manual checklist construction. Its appendices also describe comparison baselines, seed-question writing, and top-performing evaluators.

9 Limitations

  • Task-specific, human-written seed questions ground evaluation criteria, but misalignment may still require human involvement.The authors frame this as a challenge common to automatic evaluation, not unique to CheckEval.Source: §9 · p. 10
  • The study compares modelwise trends and Boolean checklists with Likert scales, but leaves comparisons with prompt optimization, debate, and meta-evaluator training for future work.Source: §9 · p. 10
  • Not all tasks or criteria fit yes/no decisions; mixed-quality responses two to three times longer than current benchmarks make this boundary more apparent.Source: §9 · p. 10
  • The study focuses mainly on summarization and dialogue response generation, so broader task generalization remains to be tested.The authors name story generation, long-form question answering, and machine translation as additional tasks for study.Source: §9 · p. 10
  • Checklist construction remains manual and task-specific, making the time and effort needed for new domains difficult to predict.The authors suggest exploring an NLG question database and LLM-assisted checklist assembly.Source: §9 · p. 10

A.1 Baselines

  • G-Eval is the main comparison baseline; the appendices also compare against other text-evaluation methods, including SEEval, TIGERScore, GPTScore, and HD-EVAL.G-Eval was selected for its widespread adoption and methodological similarity to CheckEval.Source: §A.1 · p. 15

A.2 Detailed Process of Seed Question Writing

  • Seed questions cover subdimensions with atomic, non-overlapping formulations; the team cross-validates them for clarity, consistency, and relevance.The authors report maintaining moderate granularity to avoid augmented questions drifting from seed intent.Source: §A.2 · p. 15

A.3 Top-3 Models per Evaluation Method

  • Top CheckEval evaluators included GPT-4-Turbo, Mistral-Large, and Gemma2-27B on SummEval, and GPT-4-Turbo, GPT-4o, and Mistral-Large on Topical-Chat.Source: §A.3 · p. 15

A.4 Human Validation

The authors describe three human validation studies using Ph.D.-level evaluators and samples from SummEval. The studies assess correlation, agreement, and checklist validation under distinct sampling and annotation conditions.

A.4 Human Validation

  • Three human studies used three Ph.D.-student-level evaluators or above, all with evaluation-research and summarization or dialogue backgrounds.The studies sampled summaries from SummEval using stratification by original human annotation scores.Source: §A.4 · p. 15
  • The correlation study sampled 20 SummEval summaries and evaluated coherence, consistency, fluency, and relevance with binary responses.Source: §A.4 · p. 16
  • The agreement study sampled 100 summaries and assessed relevance alone because that dimension required about 10,000 answers per annotator.The sample size was based on a pilot-study power analysis.Source: §A.4 · p. 16
  • In checklist validation, annotators saw the same items, with about 28 questions per dimension for SummEval and 26 for Topical-Chat.Source: §A.4 · p. 16

B.1 Additional experiments with QAGS

On QAGS, CheckEval outperforms G-Eval on correlation for most tested LLMs and has higher IEA across model groups. Additional comparisons report competitive performance, while checklist-validation agreement is quantified separately.

B.1 Additional experiments with QAGS

  • 9 of 12 LLMs show higher CheckEval correlation with human judgments than G-Eval on QAGS.The passage says this is comparable to results on the other two datasets reported in the main text.Source: §B.1 · p. 16
  • CheckEval achieves higher IEA than G-Eval across all model groups on QAGS.Source: §B.1 · p. 17

B.3 Checklist Validation

  • Exact Match IAA was 0.677 for SummEval and 0.634 for Topical-Chat; it was chosen over Fleiss’s Kappa given yes-heavy responses and fewer than 30 items per dimension.Source: §B.3 · p. 17

B.2 Comparative performance of various LLM-as-a-Judge methods

  • CheckEval performs well overall on both datasets and remains competitive with newer methods; Table 10 compares a broader set of LLM-as-a-Judge methods.The authors emphasize reliability and consistency across LLMs, rather than proposing the best-performing judge.Source: Table 10 · p. 17, §B.2 · p. 17

C.1 Analysis of Performance on High and Low-Quality Texts

This analysis compares CheckEval with G-Eval across text-quality levels and tests whether weighting, binarization, or greater inference budget changes the findings. It also reports checklist question-count changes and describes the evaluation and question-augmentation prompts.

C.1 Analysis of Performance on High and Low-Quality Texts

  • CheckEval correlates more strongly with human judgments than G-Eval across all dimensions for high-quality texts, including fluency on SummEval and engagingness and naturalness on Topical-Chat.For SummEval fluency, correlations are 0.34 vs. 0.16; for Topical-Chat engagingness, 0.60 vs. 0.42, and naturalness, 0.44 vs. 0.35.Source: §C.1 · p. 17, Figure 4 · p. 18
  • For low-quality texts, CheckEval generally retains stronger correlations but drops in SummEval fluency and Topical-Chat groundedness, where its binary scores can be less discriminative.The authors suggest mismatches between benchmark definitions and human annotation practices may contribute; uniformly low yes/no scores can also obscure differences among poor responses.Source: §C.1 · pp. 17–18

C.2 Does CheckEval need question weighting?

  • Learned question-specific weights produced mixed results across evaluator models, so the authors did not use weighted aggregation in their reported results.The weights came from linear regression on 20% of SummEval data, repeated across five random seeds.Source: §C.2 · p. 18, Table 11 · p. 18

C.3 Does Binarizing Likert-Scale Outputs Close the IEA Gap?

  • Binarizing G-Eval’s Likert outputs improves its IEA scores, but a large gap to CheckEval remains across model groups, so output format alone does not explain the difference.The experiment tested both assignments of the middle Likert value to binary 0 or 1.Source: §C.3 · p. 19, Table 12 · p. 19

C.4 Does Increasing the Inference Budget Strengthen the Baselines?

  • Sampling and averaging three diverse G-Eval outputs on Topical-Chat changed correlations minimally and did not close the gap with CheckEval.Pearson’s r changed from 0.6387 to 0.6389, and Spearman’s ρ from 0.6169 to 0.6176.Source: §C.4 · p. 19, Table 16 · p. 20

D The number of questions at each stage

Evaluation Prompt for SummEval

Augmentation - Question Diversification Prompt

  • The Topical-Chat prompt supplies conversation history, a fact, and a candidate response; the augmentation prompt generates sub-questions from seed questions and key components.The evaluation and augmentation prompts are shown in Figures 6–8; the paper also identifies a filtering prompt in Figure 9.Source: F Prompts · p. 20, Evaluation Prompt for Topical-Chat · p. 24, Figure 6 · p. 24
Loading 2403.18771v3…