Source-linked AI summary
Large Language Models are Inconsistent and Biased Evaluators
Rickard Stureborg, Dimitris Alikaniotis, Yoshi Suhara
TL;DR
The paper examines whether LLM evaluators are robust, finding systematic biases and inconsistency in summarization judgments. It analyzes these behaviors on SummEval, proposes mitigation recipes, and reports statistically significant improvements over state-of-the-art LLM evaluators on RoSE.
Problem
The robustness of reference-free LLM evaluators is understudied beyond their correlation with human expert scores, despite their use in automatic evaluation.
Method
The paper analyzes familiarity, scoring, anchoring, and consistency effects using SummEval, then evaluates mitigation recipes on the multi-domain RoSE benchmark.
Results
LLM evaluators show familiarity, score, and anchoring biases plus inconsistent judgments, while the proposed recipe significantly outperforms the current state-of-the-art method on RoSE’s CNNDM partition.
Takeaways & Limitations
LLM evaluator design should address biased and inconsistent behavior rather than optimize only agreement with human judgments.
Takeaways & Limitations
The experiments primarily use proprietary GPT-based models, so the findings’ generalization to other LLMs remains unclear.
Abstract
from arXiv · showhide
The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.
1 Introduction
The paper examines whether LLM evaluators are robust, focusing on bias and consistency through analyses of their judgments and prompt sensitivity. It identifies several systematic biases and inconsistencies, then develops recipes to address them.
- Motivation: LLM evaluator robustness remains understudied despite their growing use as reference-free automatic evaluation metrics.Their zero-shot capability allows scores to be generated directly from system outputs without reference texts.
- Contributions: The paper quantitatively analyzes evaluator bias by linking LLM behaviors with human evaluation patterns.The analyses examine familiarity, score-distribution, and anchoring effects.
- Bias: LLM evaluators show familiarity bias, assigning higher scores as summary perplexity decreases.The paper interprets this preference for lower-perplexity text as a familiarity effect.
- Bias: LLM evaluators exhibit score bias and anchoring effects when producing granular or multiple judgments.They assign some scores disproportionately often and are influenced by previously generated ratings.
- Consistency: LLM evaluators show lower inter-sample agreement than human experts and change judgments under prompt differences that should not affect quality assessments.The study evaluates these consistency problems across different prompts and generation configurations.
- Mitigation: The authors compile mitigation recipes and use them to develop a new evaluator that statistically significantly improves over two existing LLM evaluators on RoSE.The comparison is conducted for text summarization.
2 Methodology
The study analyzes LLM-based summary evaluators using SummEval and evaluates a recipe-based system on RoSE. It combines benchmark analyses, model and prompt configurations, and a generalized scoring prompt.
- Datasets: SummEval provides human evaluations of generated summaries across Coherence, Consistency, Fluency, and Relevance.Its annotations cover summaries produced by varying models on CNN/Daily Mail articles.
- Datasets: RoSE contains CNN/Daily Mail, SAMSum, and XSum datasets with annotations based on recall of Atomic Content Units.The benchmark is used to stress-test the evaluator across three domains.
- Models: The analysis uses GPT-3.5 and GPT-4, while the RoSE evaluation uses GPT-4-Turbo and text-davinci-003 for perplexity calculations.The authors report values against their own implementation of G-Eval.
- Prompts: The evaluation prompt supplies a metric, metric definition, aspects, article, and summary, then requests an integer 1-10 score followed by a rationale.The prompting strategy can evaluate one or multiple metrics of interest.
- Evaluation: Performance is primarily measured with Kendall’s τ correlation between evaluator scores and human judgments for each SummEval label.This follows the convention used in automatic text-summarization evaluation.
3 Results and Analysis
The analysis identifies several systematic weaknesses in LLM evaluators, including familiarity, score-distribution, anchoring, and consistency biases. It also examines configuration choices that can improve evaluator performance while exposing important sensitivity to prompts, temperature, and source-document inputs.
- Familiarity Bias: LLM evaluators prefer familiar summaries, assigning higher ratings to text with lower perplexity than expert annotators do.This familiarity bias also extends to source documents, whose perplexity correlates negatively with LLM ratings despite no such correlation for human experts.
- Scoring Granularity and Score Biases: 1-100 scoring produces sparse, uneven predictions, with outsized probabilities for round scores such as 60, 70, 80, 90, and 95.Most predictions fall between 70 and 100, while much of the 1-60 range is ignored.
- Scoring Granularity and Score Biases: 0.428 Kendall’s τ is the best average performance reported for GPT-4 using the 1-10 scale across SummEval labels.The 1-10 method performs best on average and relevance, while 1-100 and float scoring perform best on selected other dimensions; increasing granularity therefore requires care because score bias remains possible.
- Consistency and Sensitivity: LLM evaluators show lower inter-sample agreement than human inter-annotator agreement and can produce poor individual predictions despite useful system-level correlations.The analyses therefore treat self-consistency and prompt sensitivity as central evaluation concerns rather than optimizing correlation alone.
- Sensitivity to Temperature and CoT: CoT prompting benefits from higher temperatures, whereas non-CoT prompting performs better at lower temperatures.CoT performance rises sharply until approximately 0.5 before dropping or plateauing, motivating multiple samples with appropriately tuned temperature.
- Source-Document Dependence: Removing the source document reduces GPT-3.5-Turbo evaluation performance by 27% relative, driven heavily by relevance.The source document also substantially affects fluency ratings, even though fluency should be independent of the article text.
4 Case Study
The case study evaluates a recipe-informed LLM evaluator against prior systems on RoSE, addressing dataset-quality and replication concerns. The proposed method outperforms several baselines on key partitions, although performance varies with prompting strategies.
- Evaluation Setup: The evaluation uses RoSE to address disagreement in expert ratings, possible training-data contamination, and weak transfer from established datasets.RoSE is designed to maximize label quality and is unlikely to be included in GPT training data; CNNDM is in-domain, while SAMSum and XSum are out-of-domain.
- Evaluation Setup: The method uses 1-10 scoring, evaluation steps, an ACU definition, non-CoT prompting, temperature 0, and one generated output.These parameters were not tuned on RoSE.
- Results: Our method outperforms both G-Eval and rate-explain on the CNNDM and SAMSum partitions.It achieves Kendall’s τ = 0.220 on CNNDM and τ = 0.308 on SAMSum.
- Results: Performance differences are substantially affected by prompting strategies, leaving room for further prompt-engineering improvements.The difference from Chiang and Lee (2023) on SAMSum is not statistically significant.
- Results: The method performs significantly better than G-Eval on both CNNDM and SAMSum and than Chiang and Lee (2023) on CNNDM.The comparison is reported with bootstrap confidence intervals.
5 Related Work
Related work established LLM evaluators as flexible, high-quality metrics across NLP tasks, but prior studies focused mainly on prompt engineering and only partly examined their limitations. This paper extends that line by quantitatively analyzing additional biases in summarization evaluation.
- Automatic Evaluation: Traditional automatic metrics use token-level matching or semantic similarity against reference texts, while LLM evaluators can score outputs without references.The related work spans ROUGE, BLEU, METEOR, BERTScore, BLEURT, and MoverScore.
- LLM Evaluators: Recent studies report that LLM evaluators can perform well across summarization, machine translation, factual consistency, and other text-generation tasks.Much of this work emphasizes improvements through prompt engineering.
- Evaluator Biases: Prior analyses identify position bias, verbosity bias, self-enhancement bias, and other cognitive biases in LLM evaluators.These findings motivate deeper quantitative analysis of evaluator limitations.
- Novelty: This paper differs from Ohi et al. (2024) by using perplexity to assess bias and focusing on summarization.Ohi et al. study likelihood bias in data-to-text and grammatical error-correction tasks.
6 Conclusion
The paper finds that LLM evaluators exhibit multiple biases and inconsistencies in text summarization. It also proposes mitigation recipes that significantly improve performance against a state-of-the-art method on one RoSE partition.
- Findings: LLM evaluators favor low-perplexity summaries, disregard intended scoring scales, and suffer anchoring effects in multi-attribute judgments.These findings correspond to familiarity bias, score bias, and degradation in multi-attribute judgment.
- Findings: LLM evaluators change their judgments across settings such as whether source documents are included.The conclusion characterizes these systems as inconsistent evaluators.
- Mitigation: The paper provides recipes intended to mitigate identified evaluator problems.The recipes are derived from the preceding analyses.
- Results: The proposed approach significantly outperforms the current state-of-the-art LLM summary evaluator on the CNNDM partition of RoSE with 90% confidence.The paper concludes that further effort should address these evaluator issues.
Limitations
The paper identifies limits in generalizability and solution coverage. Its analyses rely mainly on GPT-based models and SummEval, while many evaluator issues remain unresolved.
- Reliance on GPT-based models: Results may not generalize beyond GPT-based models because experiments primarily use proprietary OpenAI models.The authors could not extend the analysis to models such as Llama-2, Vicuna, and Alpaca because repeating the experiments required substantial resources.
- Reliance on SummEval for analysis: The analysis primarily relies on SummEval, creating a risk that its findings may not generalize beyond that benchmark.RoSE includes three datasets from different domains, but the reported gains vary across domains.
- Limited solutions: The paper investigates solutions to some identified issues, while many inconsistencies and biases remain to be studied.The authors describe solution development as difficult and position further work as an opportunity for research.
Ethics Statement
Because the study focuses on text summarization and publicly available datasets, the authors report no clear ethical implications or considerations.
- The study reports no clear ethical implications because it focuses on text summarization and uses publicly available datasets.
A Our Method for Prompting LLM Evaluation
The prompting method asks an LLM to evaluate summary salience by identifying atomic content units, matching them against the summary, and returning a 1–10 score. The generation is constrained operationally by extracting the first digit from the model’s early output.
- Generation is typically stopped after 10–20 tokens, then a regular expression extracts the first digit from the output.This handles outputs that begin with metric-name restatements or other content instead of the score.
- The evaluator assesses ACU Salience, defined as how well a summary includes all and only important information from the source article.The prompt frames ACU Salience as a 1–10 quality score.
- It identifies Atomic Content Units and facts in the document before comparing the summary against those units.
- The evaluator checks what proportion of extracted ACUs the summary covers and assigns an ACU Salience score from 1 to 10.
- The user supplies both the article and summary, and the model is prompted to provide an integer 1–10 score followed by a rationale.
B Familiarity Bias
The figure compares average perplexity with automatic evaluation scores across attributes. GPT-4 favors lower-perplexity samples, whereas GPT-3.5-Turbo disfavors higher-perplexity examples.
- GPT-4 shows a preference for low-perplexity samples when assigning automatic evaluation scores.
- GPT-3.5-Turbo appears to show a dis-preference for high-perplexity examples.
C Anchoring Effect in Multiple Judgments
The section examines score distributions conditioned on previous scores and compares CoT with non-CoT prompting across temperatures. It also reports inter-annotator agreement and self-consistency using Krippendorff’s α.
- Score distributions are analyzed conditional on the values of previous scores in the same generation.
- Higher temperatures generally harm non-CoT predictions but improve CoT predictions, with diminishing returns.
- Table 7 reports Krippendorff’s α for inter-annotator agreement and self-consistency across sampling and judgment settings.For GPT-4, multiple samples are treated as judgments from different annotators; other α values compare judgments from different settings.