Source-linked AI summary
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
TL;DR
As LLMs increasingly replace human judges in NLP evaluation, the validity and reproducibility of that practice remain uncertain. The paper introduces JUDGE-BENCH and evaluates 11 open-weight and proprietary LLMs across 20 human-annotated datasets, finding substantial variation in agreement with humans and recommending task-specific validation before deployment.
Problem
LLM-based NLP evaluation raises questions about validity, systematic bias, data leakage, and reproducibility, especially for proprietary models.
Method
The paper introduces JUDGE-BENCH, comprising 20 diverse human-annotated datasets, and evaluates 11 open-weight and proprietary LLMs as judges.
Results
Agreement with human judgments varies widely across datasets, properties, data sources, and annotator expertise, although some tasks show reliable alignment.
Takeaways & Limitations
LLM judges should be validated and calibrated against task-specific human judgments before deployment.
Takeaways & Limitations
Correlation with human judgments may validate shared biases rather than evaluator reliability, so human annotators may also require validation.
Abstract
from arXiv · showhide
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators.
1 Introduction
LLM-based evaluation can reduce effort but raises concerns about validity, bias, data leakage, and reproducibility. This paper addresses these concerns by testing whether current LLMs can approximate human judgments across diverse datasets and evaluation settings.
- Motivation: LLMs are increasingly used to replace human judgments for NLP evaluation, despite unresolved validity and reproducibility concerns.LLM ratings can reduce evaluation effort and are claimed to be reliable across rounds, but proprietary models may change or disappear.
- Study scope: 11 open-weight and proprietary LLMs are evaluated on 20 datasets spanning tasks, properties, prompt styles, judgment types, and human-annotator expertise.The resulting JUDGE-BENCH release contains over 70,000 test instances with associated human judgments and an extensible codebase.
- Findings: LLMs align well with human judgments on some tasks, such as instruction following, but their performance is inconsistent across and within annotation tasks.The study therefore examines reliability across varied evaluation conditions rather than treating LLM judging as uniformly effective.
- Findings: Chain-of-Thought prompting does not reliably improve agreement between LLM and human judgments.The finding is reported as consistent with recent related work.
- Findings: GPT-4o aligns better with humans in some settings, but the gap with large open-source models is small.This result is presented as promising for the reproducibility of future evaluation efforts.
2 Construction of JUDGE-BENCH
JUDGE-BENCH organizes human-annotated English data to test LLM evaluation across data sources, tasks, properties, judgment formats, and annotator expertise. Its schema is designed to support systematic comparison and future extension.
- Data sources: JUDGE-BENCH distinguishes model-generated items, which evaluate NLP systems, from human-generated items, which assess properties such as grammaticality or toxicity.The distinction also permits analysis of whether LLMs favor machine-generated outputs.
- Evaluated properties: The benchmark covers properties including grammaticality, toxicity, coherence, factual consistency, verbosity, plan success, and mathematical-reasoning correctness.Some properties recur across tasks, while others are task-specific.
- Dataset design: All 20 datasets use a precise schema, retain available individual annotations, and are formatted to facilitate integration of additional datasets.The study focuses on English datasets or language pairs that include English and records annotator expertise and guideline availability.
- Reporting: Table 1 reports Cohen’s kappa for categorical annotations and Spearman’s correlation for graded annotations, alongside model variation and human upper-bound estimates.Datasets with both annotation types appear twice, and color distinguishes human-generated from model-generated language.
3 Model Selection and Experiment Design
The study evaluates representative open-weight and proprietary LLMs using human-collection prompts and controlled alternatives, then compares their judgments with human annotations. Invalid responses are handled explicitly, and human-agreement upper bounds contextualize model scores.
- Models: The model set includes representative proprietary and open-weight systems of varying sizes, including GPT-4o, LLaMA-3.1, Gemini-1.5, Mixtral, Command R, OLMo, Starling-7B, and Mistral.Selection drew on strong performance across tasks in the Open LLM and Chatbot Arena Leaderboards.
- Prompts: Original human-judgment instructions are reused as prompts when available, with output constrained to one permitted answer and no explanation.When instructions are unavailable, prompts are built from task descriptions and metric definitions.
- Prompt variants: Chain-of-Thought, few-shot, and paraphrased prompts are tested, but none produces systematic improvements.The alternative prompting strategies are evaluated against the study’s main prompting approach.
- Evaluation protocol: Invalid LLM responses are randomly replaced with judgments sampled from the relevant categorical or graded human annotations.This protocol equalizes the number of judgments across models for each dataset.
- Evaluation protocol: Human-agreement upper bounds are estimated with Spearman’s ρ or Cohen’s κ between bootstrapped single-rater responses and aggregated responses.The procedure uses multiple available individual human judgments, typically three.
4 Results
LLM–human agreement varies substantially by model, dataset, judged property, annotator expertise, and whether inputs are human- or machine-generated. Alignment is weakest for some safety-related tasks and differs across quality dimensions.
- Scores vary substantially across models, datasets, and the properties being judged, often remaining below the human upper bound.GPT-4o ranks first in several scenarios, while Llama-3.1-70B and Mixtral-8x22B outperform it on some assessment types.
- Toxicity and safety have among the lowest human–model alignment, with some model scores negative and valid response rates particularly low.Guardrails and models’ tendency to provide explanations instead of judgments contribute to poor performance, especially in medical safety.
- All models correlate more strongly with non-expert than expert annotations for graded judgments.The authors suggest, speculatively, that surface-level features used by non-experts may align more closely with patterns LLMs capture.
- No single model is consistently best across properties: GPT-4o and Gemini-1.5 lead acceptability and verbosity, while Mixtral models lead coherence and consistency.Correlation for engagingness remains consistently low across models.
- All models align better with human language than machine-generated text for both categorical and graded annotations.The comparison uses Cohen’s κ for categorical annotations and Spearman’s correlation for graded annotations.
5 Conclusions
The paper presents a large-scale benchmark of LLM judging across 20 datasets and concludes that evaluator reliability is task-dependent. It recommends validating and calibrating LLM judges against task-specific human judgments before deployment.
- The study measures human–LLM judgment agreement across 20 datasets, varying judged properties, human expertise, and whether data are model- or human-generated.
- Models can be reliable evaluators for some tasks, including instruction following and mathematical reasoning trace generation.
- Agreement varies widely across datasets, evaluated properties, data sources, and the expertise level of human judges.
- Chain-of-Thought prompting does not consistently improve agreement with human judgments.
- JUDGE-BENCH enables systematic evaluation across diverse tasks and can be extended with new tasks.
Limitations
The study’s validation design and benchmark scope impose important boundaries on its conclusions. The authors also leave pairwise preference evaluation and broader language coverage for future work.
- Correlation with human judges may not adequately validate LLM evaluators because humans and LLMs can share biases.The authors therefore note that human annotator reliability may also require validation, depending on the task.
- The study uses existing tasks and datasets without reassessing their quality or representativeness of downstream applications.These datasets may be less meaningful for end-users and may carry risks such as data leakage.
- The paper evaluates categorical and graded judgments but leaves pairwise preference evaluation and other recent methods for future work.
- The datasets are mostly English-language, so cross-lingual variation in LLM metaevaluation remains unresolved.Machine-translation datasets are the stated exception to the English focus.
A Datasets
JUDGE-BENCH assembles 20 datasets spanning diverse tasks, judged properties, data sources, annotation types, and annotator expertise. The datasets include human- and model-generated language, categorical and graded judgments, and multiple-annotation resources for estimating inter-rater agreement.
- Data sources: The benchmark includes both human-generated and model-generated language to test whether LLM judges behave differently across data sources.Model-generated items support evaluation of NLP systems, while human-generated items are assessed for properties such as grammaticality or toxicity.
- Annotation sources and formats: Human annotations include expert, professional, semi-professional, crowdsourced, and original-author judgments across categorical and graded formats.Examples include 0–6 translation ratings, 0–100 translation scores, binary labels, and expert grammaticality annotations.
- Scope boundary: The study focuses primarily on English datasets or language pairs containing English, leaving cross-language variation in LLM metaevaluation unresolved.The scope includes English–German and Chinese–English translation datasets.
- Human agreement: Several datasets provide multiple human annotations, enabling inter-rater agreement analysis and human upper-bound estimation.Table 3 reports inter-rater agreement for datasets with multiple public annotations.
B The JUDGE-BENCH Data Schema
JUDGE-BENCH uses a shared JSON schema to standardize dataset metadata, annotations, instances, and human judgments, making the benchmark extensible. The study also defines procedures for handling prompts, invalid outputs, graded properties, and human-agreement upper bounds.
- Extensibility: The publicly available code includes an example format and instructions for verifying that newly added datasets comply with the schema.This supports extending JUDGE-BENCH with additional datasets.
- Schema fields: The JSON schema requires dataset names and URLs, annotation descriptions, instances, aggregated judgments, and available individual annotations.The schema also records how properties are measured and prompt-like instructions when applicable.
- Human upper bounds: Human upper bounds were estimated with bootstrapped correlations between individual-rater responses and aggregated responses, although the estimates can contain errors.Model scores may therefore exceed an estimated upper bound.
- Graded properties: The study defines graded properties including acceptability, coherence, consistency, engagingness, fluency, informativeness, relevance, and verbosity.These properties are evaluated across dialogue, summarization, reasoning, and recipe-generation datasets.
F Valid Response Rates
The study reports valid response rates for each model and dataset, with the rates summarized in Table 5 and Figures 5 and 6.
- Response-rate reporting: Valid response rates are reported per model and dataset in Table 5 and summarized visually in Figures 5 and 6.The section specifically identifies Figure 5 as showing valid response rate per model.
G More Details on Toxicity and Safety Evaluation
Model response behavior differs across medical-safety, DICES, and ToxicChat evaluations. Refusals and preference for particular safety labels can lower or distort agreement scores even when response validity is high.
- Medical safety: Medical-safety scores are lowered because models often refuse, explain, copy prompt content, or respond generally helpfully instead of producing the requested judgment.When no answer is detected, the procedure randomly samples an answer, affecting task results.
- DICES: DICES scores are low despite high valid response rates because models prefer “Unsure” or “Unsafe” over labeling responses “Safe.”The available label choices therefore shape the observed evaluation behavior.
- ToxicChat: ToxicChat models performed reasonably well compared with their behavior on the Medical-safety and DICES datasets.The passage reports this outcome without specifying a numerical score.
H Additional Results
Additional experiments examine prompt variants and report per-dataset evaluation scores and response validity across models. Chain-of-Thought prompting and paraphrased or few-shot prompts produce improvements only inconsistently across datasets and models.
- Chain-of-Thought Prompts: CoT prompting appends a step-by-step reasoning instruction and requires a final labeled answer, enabling automatic extraction of model responses.The procedure uses the original human-judgment instructions and enumerates possible answer labels in the added guideline.
- Chain-of-Thought Prompts: CoT experiments exclude Mixtral-8x22B and Comm-R+; GPT-4o is evaluated on 10% of CoLa-grammar instances because of processing and rate limitations.
- Chain-of-Thought Prompts: CoT prompting improves agreement scores and correlations for some models on certain datasets, but its overall effectiveness is inconsistent against standard prompting.
- Prompt Paraphrases: Prompt paraphrases do not consistently improve results on DICES-350-expert, WMT 2023 En-De, or WMT 2023 Zh-En compared with the original prompts.
- Few-shot Prompts: Few-shot prompting does not improve DICES-350-expert correlations, while WMT improvements are higher for Llama 3.1 8B but very moderate or absent for the other two models.
- Reported Results: Tables report Cohen’s kappa for categorical annotations, Spearman’s correlation for graded annotations, and valid-response ratios across models and datasets.The tables distinguish human-generated from model-generated language and report significant-correlation counts for Spearman’s results.