Source-linked AI summary
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, Dieuwke Hupkes
TL;DR
Human evaluation is expensive, and the reliability of LLMs-as-judges remains uncertain. The paper compares thirteen judges with humans and lexical baselines in a high-agreement TriviaQA setting, finding that only the largest judges align strongly while simpler methods can still rank models similarly.
Problem
Human evaluation is expensive and often impractical, creating a need to assess whether LLM judges can reliably evaluate model responses.
Method
The paper compares thirteen judge models with human judges and lexical baselines on nine exam-taker models using TriviaQA in a high-human-agreement setting.
Results
Only the largest judges achieve strong alignment, yet they remain below human alignment and can differ by up to 5 points from human scores; contains can rank models similarly.
Takeaways & Limitations
Judge scores should not be trusted solely because alignment is high; percent agreement and Scott’s π should be paired with qualitative analysis.
Takeaways & Limitations
The study uses a clean TriviaQA setting with near-perfect human agreement and plans to expand to more complex scenarios.
Abstract
from arXiv · showhide
Offering a promising solution to the scalability challenges associated with human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models (LLMs). However, there are still many open questions about the strengths and weaknesses of this paradigm, and what potential biases it may hold. In this paper, we present a comprehensive study of the performance of various LLMs acting as judges, focusing on a clean scenario in which inter-human agreement is high. Investigating thirteen judge models of different model sizes and families, judging answers of nine different 'examtaker models' - both base and instruction-tuned - we find that only the best (and largest) models achieve reasonable alignment with humans. However, they are still quite far behind inter-human agreement and their assigned scores may still differ with up to 5 points from human-assigned scores. In terms of their ranking of the nine exam-taker models, instead, also smaller models and even the lexical metric contains may provide a reasonable signal. Through error analysis and other studies, we identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency. The fact that even the best judges differ from humans in this comparatively simple setup suggest that caution may be wise when using judges in more complex setups. Lastly, our research rediscovers the importance of using alignment metrics beyond simple percent alignment, showing that judges with high percent agreement can still assign vastly different scores.
1 Introduction
Human evaluation is costly, motivating LLM-based judges, but their alignment and biases remain uncertain. This study tests judges in a high-agreement setting and finds strong judges still diverge from humans while simpler methods can preserve rankings.
- Human evaluation remains the gold standard, but its expense and impracticality motivate growing use of LLMs as judges.
- The study evaluates thirteen judge models on nine base and instruction-tuned exam-taker models in a clean, high-human-agreement setting.
- Only GPT-4 Turbo, Llama-3.1 70B, and Llama-3 70B achieve strong human alignment, yet all remain below human alignment.
- Scott’s π distinguishes judges more effectively than percent agreement, which can remain high despite 10-20-point differences from human scores.
- Even highly aligned judges can differ by up to 5 points from human scores, while smaller judges and contains can still rank exam-taker models comparably.
- Error analyses identify leniency, difficulty with under-specified answers, prompt sensitivity, and failures on verbatim matches.
2 Related work
Prior work applies LLM judges across many evaluation settings but reports persistent reliability challenges. This paper extends that work with reference-based comparison in a high-human-agreement setting.
- Recent studies use LLM judges for story generation, retrieval-augmented generation, visual QA, code comprehension, multilingual evaluation, and open-ended tasks.
- Judge models can inherit hallucinations, factual errors, and difficulty following complex instructions, while comparative assessment may outperform absolute scoring.
- Unlike prior work, this study compares outputs with reference answers in a high-human-alignment setting and evaluates more judge and exam-taker models.
3 Methodology
The methodology uses TriviaQA to compare diverse judge models with humans and lexical baselines under controlled, high-agreement conditions. Alignment is measured with both percent agreement and Scott’s π.
- The study evaluates judge models on TriviaQA answers in a comparatively controlled setup designed to isolate judge abilities from task ambiguity.
- TriviaQA supplies short reference answers; experiments use 400 randomly sampled questions, with humans establishing high agreement on the benchmark.
- The exam-taker set spans base and instruction-tuned Llama-2 and Mistral models plus GPT-4 Turbo, with few-shot prompting.
- The judge set includes instruction-tuned Llama, Mistral, GPT-4 Turbo, Gemma, and JudgeLM models across varying sizes and architectures.
- Exact match and contains provide lexical baselines, with contains marking an answer correct when a reference answer is a case-insensitive substring.
- Percent agreement and Scott’s π quantify judge-human alignment, with Scott’s π correcting for chance agreement.
- Human majority-vote alignment reaches Scott’s π of 96.2 ± 1.07 and percent agreement of 98.52% ± 0.42%.
4 Results
Judge-model alignment with humans is generally weaker than percent agreement suggests, while Scott’s π better distinguishes judges and higher values correspond to smaller score differences. Judges can nevertheless rank exam-taker models effectively despite imperfect score agreement.
- Alignment with human judgments: Scott’s π is low for most judges but reaches the high 80s for Llama-3 70B, Llama-3.1 70B, and GPT-4 Turbo.Percent alignment is high for virtually all models except Gemma 2B and EM.
- Alignment with human judgments: The best-scoring judge, Llama-3 70B, remains 8 points behind human judgment.
- Alignment metrics: Percent agreement and Scott’s π can diverge substantially, with a 26-point human-versus-EM gap for percent agreement and a 64-point gap for Scott’s π.
- Score agreement: Scott’s π above 80 corresponds to evaluation scores within up to 5 points of human scores.
- Score agreement: Even the best-aligned judges can differ by up to 5 points from human scores, limiting their ability to estimate exam-taker capabilities perfectly.
- Ranking exam-taker models: Most judges achieve rank correlations above 0.7 with human rankings, while contains and Mistral 7B reach ρ of 0.99 and 0.98.
5 Analysis
The analysis links judge quality to precision, recall, prompt robustness, and error patterns, while controlled-response tests expose failures on unrelated or even verbatim answers. Judges also show leniency, especially when evaluation criteria are not fully aligned with instructions.
- Precision, recall, and errors: Precision and recall increase moderately with judge alignment, while lower-quality judges produce more false positives.True positives remain consistent across judge quality, whereas true negatives decline slightly as quality decreases.
- Precision, recall, and errors: GPT-4 Turbo and Llama-3 70B identify incorrect entities and excessive entities well, but underspecified and incorrect answers remain more challenging.
- Prompt sensitivity: Top judges maintain high human agreement across prompt lengths and specificity, whereas other models lose alignment as instructions become more complex.The longer prompts contain 245 and 301 tokens, compared with 45 and 58 tokens for prompts without guidelines.
- Prompt sensitivity: Larger judges are less sensitive to reference order, while smaller judges generally show greater sensitivity except Mistral 7B.
- Controlled responses: Controlled tests evaluate a gold answer, two incorrect short responses, and a repetition of the question as dummy benchmark responses.
- Controlled responses: Some judges, including Llama-2 70B, incorrectly evaluate many dummy answers despite high benchmark alignment, and unrelated answers such as “Yes” and “Sure” may be marked correct.Further research is needed to clarify this behavior.
- Leniency bias: Leniency bias P+ is the probability of assigning “correct” to samples outside the correct-judgment probability Pc.For most models, P+ is significantly higher than 0.5 when evaluation criteria are not completely aligned with the instructions.
6 Conclusion
This study evaluates LLM judges in a high-human-agreement setting and finds that even strong judges remain imperfectly aligned. Ranking can be reliable with cheaper methods, but judge outputs warrant caution and multiple alignment measures.
- Conclusion: In a clean, high-agreement setting, the study identifies judge-model issues separately from task ambiguity.
- Conclusion: Larger judges align better with humans than smaller cost-efficient models, but their scores can still differ by up to 5 points.
- Conclusion: For ranking exam-taker models, contains and other cheaper approaches can perform similarly to more expensive judges despite lower alignment scores.
- Conclusion: The study finds smaller judges more likely to judge positively when uncertain, lower-alignment judges less precise, and some judges vulnerable to “Yes” and “Sure” dummy answers.
- Conclusion: The authors recommend combining percent agreement, Scott’s π, and qualitative analysis rather than blindly trusting judge judgments.
A Limitations
The study’s limitations concern task simplicity, human-annotation coverage, sample size, judge selection, and the restricted range of scenarios examined.
- Simplicity of the task: The evaluation uses a deliberately simple task with high human agreement, whereas judges are often deployed for more complex answers and preference rankings.The authors state that generalization to other scenarios remains uncertain.
- Human alignment: The human-alignment estimate initially relied on annotations for one exam-taker model, although additional Llama3.1 70B annotations produced similar agreement values.The authors therefore average results from 1,200 human-annotated samples.
- Size of the judged samples: The analysis was restricted to 400 questions because annotating all nine exam-taker models was costly, although statistical tests indicated low variance.This sample supported manual annotation and error analysis within the reported resource constraints.
- Selection of judges: The judge set covers off-the-shelf autoregressive models and one judge-trained LLM across sizes and families, but other judges were not included.The authors note that adding judges is comparatively straightforward because it requires computation rather than manual annotation.
- Future work: The authors leave evaluation across multiple samples, tasks, scenarios, and model accuracies for future work because it would require substantially more human annotations.They emphasize that more studies are needed to understand how broadly the findings generalize.
F Metrics for judge models
This section defines the agreement metrics and describes the benchmark prompts and annotation protocol used to evaluate judge models.
- Agreement metrics: Percent agreement is the fraction of annotations on which two annotators agree, ranging from 0 to 1.For binary labels, it is computed from true positives and true negatives divided by all four confusion-matrix counts.
- Agreement metrics: Scott’s Pi measures annotator alignment while correcting for agreement expected by chance.It uses observed agreement and a hypothetical chance-agreement probability.
- Prompting: The benchmark includes distinct prompt templates for base and chat exam-taker models and a single-word output instruction for judges.The judge prompt evaluates a model response against the question and reference answers.
- Judge evaluation: The evaluation classifies each response against references as correct or incorrect, treating semantic equivalence as correctness.Guidelines mark underspecified answers incorrect, while correct extra information and unnecessary verbosity can remain correct.
- Human annotation: Human evaluators used iteratively refined guidelines to improve consistency and reproducibility when judging benchmark responses.CS graduate students served as annotators.
I Statistical reliability of Evaluation sample
The authors assess whether the 400-question evaluation sample provides statistically reliable alignment estimates through repeated down-sampling.
- Evaluation sample: 400 questions were randomly sampled from TriviaQA because manual annotation and computation constrained the evaluation size.The main sample’s low variance was assessed with a bootstrapping test.
- Statistical reliability: Five random 300-question subsets produced Scott’s π means and standard deviations similar to the full evaluation set.The authors use this similarity as evidence that the reported alignment estimates are robust to down-sampling.
K Exam-taker model base vs chat analysis
The study finds substantial performance differences between base and chat exam-taker models, and its analysis attributes at least part of this gap to knowledge unlearning rather than prompt-format misunderstanding.
- K Exam-taker model base vs chat analysis: Base models outperform corresponding chat models by a large margin under exact match, with the gap smaller but still substantial under humans and GPT-4 Turbo.This suggests increased chat-model verbosity alone does not explain the difference.
- K Exam-taker model base vs chat analysis: Fourteen percent of errors are shared, nearly another 14% are correct only for base models, and 2.5% are correct only for chat models.These percentages come from the comparison of base-chat pairs shown in Figure 9b.
- K Exam-taker model base vs chat analysis: The authors compare two explanations: poorer understanding of the prompt format versus knowledge unlearning during alignment training.The analysis is designed to disentangle these factors.
- K Exam-taker model base vs chat analysis: Manual analysis of 400 Llama-2 70B and Llama-2 70B-chat questions finds substantially more semantically plausible but incorrect entities among chat-model errors.Most other error categories occur at roughly similar rates across the two models.
L Exam-taker model ranking correlation
Judge models and the contains metric can provide useful rankings of exam-taker models, but ranking stability and reliability vary across evaluators and prompt conditions.
- L Exam-taker model ranking correlation: The contains model exhibits the highest ranking stability and mean Spearman correlation across subsamples of six exam-taker models.The validation uses five samples, each randomly selecting six of the nine exam-taker models.
- L Exam-taker model ranking correlation: Most judge models achieve Spearman correlation coefficients above 0.7 when ranking exam-taker models.This indicates strong agreement with the evaluated ranking reference under the reported validation procedure.
- L Exam-taker model ranking correlation: Mistral 7B performs on par with GPT-4 Turbo in maintaining exam-taker rankings.The result highlights that smaller models can preserve rankings comparably to a much larger judge.
- L Exam-taker model ranking correlation: Reference-order tests provide the same question, response, and references while shuffling reference order across three permutations, then measure consistency across all three judgments.The consistency score is the percentage of questions receiving the same judgment in all three orderings.
- L Exam-taker model ranking correlation: Judges are more likely to mark an answer correct when its corresponding reference appears early, and smaller judges sometimes rely on their own knowledge instead of the references.The reported examples concern sensitivity to prompt ordering and difficulty capturing all prompt information.
O Leniency Bias
The leniency-bias analysis models judges as sometimes following the evaluation criteria and otherwise assigning “correct” probabilistically, then estimates these components from observed data.
- O Leniency Bias: The model assumes a judge follows the evaluation criteria with probability Pc and otherwise assigns “correct” with probability P+.When misaligned with the guidelines, the probability of “correct” is assumed independent of the actual correctness.
- O Leniency Bias: True positives combine criterion-aligned correct judgments with misaligned judgments that happen to be correct, yielding tP = s[Pc + (1 −Pc)P+].Here, s is the total ratio of correct exam-taker responses.
- O Leniency Bias: True negatives combine criterion-aligned incorrect judgments with misaligned judgments that happen to be incorrect, using probability 1 − P+ for the latter case.The incorrect-response ratio is 1 − s.
- O Leniency Bias: Estimated Pc and P+ values are obtained from observed data and reported against human evaluation in Figure 16a.The derivation provides a way to estimate the two probabilities from observed judgment outcomes.
- O Leniency Bias: Estimated Pc values correlate highly with Scott’s π, with a Pearson correlation coefficient of 0.98 across judge models.This comparison is used to validate the derived Pc estimates.