PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, Dieuwke Hupkes

arXiv:2406.12624v6cs.CLcs.AI

TL;DR

Free-form LLM responses are difficult to evaluate, and human assessment is costly. This study tests LLM judges in a controlled, high-human-agreement TriviaQA setting, finding that larger models align better than smaller ones but still fall short of humans.

  • Problem

    Free-form LLM responses are difficult to evaluate and human assessment is costly, while questions remain about LLM judges’ alignment with humans.

    Source: §1 · p. 1

  • Method

    The study compares judges of varying architectures and sizes with humans and automated methods on exam-taker answers in a controlled, high-inter-human-agreement TriviaQA setting.

    Source: §1 · p. 1, §3 · p. 3, §6 · p. 8

  • Results

    Up to 5 points separate judges’ scores from human scores; larger judges align better yet fall short of humans, while simpler methods like contains perform similarly when ranking exam-takers.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors recommend caution and combining percent agreement with Scott’s π and qualitative analysis, because percent agreement alone does not distinguish judges effectively.

    Source: §6 · p. 8

    For the remaining experiments, the study uses one human evaluator per sample, based on near-perfect human alignment to reduce annotation costs.

    Source: §3 · p. 5

Abstract

from arXiv · show

Offering a promising solution to the scalability challenges associated with human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models (LLMs). However, there are still many open questions about the strengths and weaknesses of this paradigm, and what potential biases it may hold. In this paper, we present a comprehensive study of the performance of various LLMs acting as judges, focusing on a clean scenario in which inter-human agreement is high. Investigating thirteen judge models of different model sizes and families, judging answers of nine different 'examtaker models' - both base and instruction-tuned - we find that only the best (and largest) models achieve reasonable alignment with humans. However, they are still quite far behind inter-human agreement and their assigned scores may still differ with up to 5 points from human-assigned scores. In terms of their ranking of the nine exam-taker models, instead, also smaller models and even the lexical metric contains may provide a reasonable signal. Through error analysis and other studies, we identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency. The fact that even the best judges differ from humans in this comparatively simple setup suggest that caution may be wise when using judges in more complex setups. Lastly, our research rediscovers the importance of using alignment metrics beyond simple percent alignment, showing that judges with high percent agreement can still assign vastly different scores.

1 Introduction

Because evaluating free-form LLM responses is difficult and human evaluation is costly, this study compares judge models with humans and automated methods in a high-human-agreement TriviaQA setting. It examines thirteen judges evaluating nine exam-taker models and finds that judge alignment and scoring remain imperfect, with vulnerabilities that warrant caution.

  • Only GPT-4 Turbo, Llama-3.1 70B, and Llama-3 70B achieved strong human alignment, and all fell short of human alignment.Figure 1 reports excellent Scott’s π alignment for these judges, but still below the human alignment score.Source: Figure 1 · p. 2, §1 · p. 2
  • The study uses TriviaQA to compare thirteen judges of varying architectures and sizes on nine exam-taker models in a setting chosen for high human agreement.This clean scenario is intended to separate task ambiguity and subjectivity from judge-model limitations.Source: §1 · pp. 1–2
  • High percent agreement can conceal 10–20-point score differences from humans, while Scott’s π better distinguishes judges.The introduction argues that percent alignment alone is insufficient for evaluating judge performance.Source: §1 · p. 2
  • GPT-4 Turbo and Llama-3 can differ from human scores by up to 5 points, yet rank exam-taker models comparably to cheaper Mistral 7B and contains.The cheaper alternatives have lower alignment scores but more consistent biases.Source: §1 · p. 2
  • Improved alignment appears linked to higher recall and fewer false negatives, but judges struggle with under-specified answers, show leniency, and remain sensitive to prompt length and quality.Judges sometimes fail even when asked to evaluate a verbatim match with a reference.Source: §1 · pp. 2–3

2 Related work

Prior work has applied LLMs as judges across varied tasks and found promising human alignment, while also documenting biases and other weaknesses. This study examines those weaknesses in reference-answer evaluation, expanding the range of judges and evaluated models.

2 Related work

3 Methodology

The methodology evaluates LLM judges in a controlled TriviaQA question-answering setup, comparing their judgments with human assessments and lexical baselines. It measures alignment with both chance-corrected and raw agreement metrics.

Evaluation data

  • The controlled testbed uses 400 randomly sampled questions from TriviaQA’s unfiltered validation partition, with short answers as references and training examples for few-shot prompts.The paper reports that a bootstrap test found low variance for the main result at this sample size.Source: §3 · p. 3

Exam-taker models

  • Nine exam-taker models include base and chat variants of Llama-2 and Mistral, plus GPT-4 Turbo, with five TriviaQA training examples in their prompts.Instruction-tuned exam-taker prompts also direct models to answer succinctly, following the examples.Source: §3 · pp. 3–4

Judge models

  • Thirteen instruction-tuned judges span Llama-2, Llama-3, Llama-3.1, Mistral, GPT-4 Turbo, Gemma, and JudgeLM models.Prompts follow human evaluation guidelines and require a single-word “correct” or “incorrect” response.Source: §3 · p. 4

Baselines

  • The lexical baselines are case-insensitive exact match and contains match, which checks whether a reference answer appears as a substring of the response.Exact match requires the response to match a reference answer exactly.Source: §3 · p. 4

Alignment

  • Alignment is measured with percent agreement and Scott’s π, which corrects for chance agreement between annotators.The paper describes Scott’s π as a more robust alignment measure.Source: §3 · p. 4

Human judgements

  • Human annotators reached Scott’s π of 96.2 ± 1.07 and 98.52% ± 0.42% percentage agreement with the majority vote.Three judges assessed 1,200 randomly sampled question-answer pairs; the remaining experiments used one human evaluator per sample to reduce annotation cost.Source: §3 · pp. 4–5

4 Results

Judge models fall short of human alignment in assigned scores, yet their rankings can still provide a signal for distinguishing the nine exam-taker models.

4.1 Alignment between judge models and humans

  • 8 points behind human judgment, Llama-3 70B was the best-scoring judge, while Gemma 2B had the lowest alignment and EM the greatest variance.Source: §4.1 · p. 5
  • 26-point percent-agreement gap and 64-point Scott’s π gap separated EM from human judgment; Scott’s π distinguished judges better than percent agreement.Gemma 2B had higher percent agreement than EM but the lowest Scott’s π score in the ensemble.Source: §4.1 · p. 5
  • Judges with Scott’s π above 80 differed by up to 5 points, whereas above 90% percent alignment could coincide with score differences exceeding 10 points.A judge’s deviation from human scores could also vary substantially by exam-taker model; Gemma 2B sometimes scored above humans and sometimes much lower.Source: §4.1 · p. 6

4.2 Exploring consistent patterns in judge models

  • Most judges had Spearman’s ρ above 0.7; contains (ρ of 0.99) and Mistral 7B (ρ of 0.98) ranked models like GPT-4 Turbo despite divergent scores.Even the best-aligned judges could differ from human-assigned scores by up to 5 points, while consistent biases could still yield similar rankings.Source: §4.2 · p. 6

5 Analysis

The analysis examines judge errors, prompt and reference-order sensitivity, controlled-answer evaluations, and leniency, finding vulnerabilities across these cases.

5.1 Better aligned models: Precision and recall gains with error spotlights

  • Precision and recall rose moderately with human alignment; as judge quality fell, true negatives slightly declined and false positives increased, while true positives stayed consistent.Source: §5.1 · p. 6, Figure 4 · p. 7
  • In 900 annotated Llama-7B Base outputs, GPT-4 Turbo and Llama-3;70B excelled at identifying wrong or excessive entities, while underspecified and incorrect answers were harder.GPT-4 Turbo performed better than Llama-3;70B on answers with fewer entities.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 Judge model sensitivity to prompt length and specificity

  • Top performers stayed highly human-aligned with minimal instructions and improved slightly with detail, while other models lost alignment as instructions grew more complex.Larger models also maintained consistent judgments across reference orders, whereas smaller models—except Mistral;7B—were more sensitive to order.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Evaluating controlled responses

  • Some judges mishandled dummy responses, including Llama-2;70B despite high benchmark alignment; unrelated “Yes” and “Sure” answers were especially vulnerable.Judges were robust to repeated questions, but even verbatim gold answers were not always judged correctly; the authors say further research is needed to clarify the behavior.Source: §5.3 · p. 7, Figure 5 · p. 8

5.4 Leniency bias in judge models

  • Most judges were lenient: when their criteria were misaligned with instructions, their probability of assigning “correct” (P+) was significantly above 0.5.The authors interpret this as a tendency to evaluate responses as correct when evaluation criteria are not fully aligned with instructions.Source: §5.4 · p. 8

6 Conclusion

In a clean evaluation setting with high inter-human agreement, the study finds that LLM judges can help evaluate models but retain important alignment and reliability weaknesses. It recommends caution and combining quantitative agreement metrics with qualitative analysis.

6 Conclusion

  • The study assesses LLM judges against human judges and automated methods in a clearly defined, high-human-agreement scenario to isolate issues beyond task ambiguity.It cautions against blindly trusting judgments even when they align with humans.Source: §6 · p. 8
  • Up to 5 points, even better-aligned GPT-4 Turbo, Llama-3.1;70B, and Llama-3;70B scores can differ from human scores, while Mistral;7B is less effective.The study cautions that these gaps matter especially in more complex scenarios.Source: §6 · p. 8
  • For ranking exam-taker models, contains performs similarly to GPT-4 Turbo and Llama-3 despite lower alignment, making cheaper approaches suitable when exact scores are not the goal.The conclusion says percent aligned does not distinguish judges effectively and recommends Scott’s π for better distinction.Source: §6 · p. 8
  • Smaller judges are more likely to judge positively when uncertain, lower-alignment models lack precision, and some models are fooled by dummy answers or miss partially incorrect answers.Better models are more robust across prompts but harder to steer, and judges more reliably detect completely incorrect answers than partially incorrect ones.Source: §6 · p. 8
  • The authors recommend reporting both percent agreement and Scott’s π alongside qualitative analysis, and plan to extend the work to more complex scenarios.The paper also notes its limitations are discussed in Appendix A.Source: §6 · p. 8

A Limitations … E Judge LLM Prompt templates

The sections qualify the study’s scope and generalizability, explain how Scott’s Pi differs from Cohen’s kappa, and document the models, datasets, and prompts used.

A Limitations

  • Because the study uses a simple task with high human agreement, its results may not generalize to complex judging scenarios, which require further study.The authors suggest caution in more complex setups if judges perform poorly on this simple task.Source: A Limitations · p. 13
  • Human alignment estimates drew on two exam-taker models, while cost limited the analysis to 400 samples; the authors report low variance from that sample size.For Llama2 7B, human alignment averaged Scott’s π of 96.36 ± 1.46 and percent agreement of 98.33% ± 0.76%; for Llama3.1 70B, the corresponding values were 95.78 ± 0.30 and 98.72%±0.10%. The authors averaged the two sets of human alignment results.Source: A Limitations · p. 13
  • The study’s judge selection aimed for breadth across commonly used model sizes and families, but evaluating additional judges and tasks is left to future work.The authors used off-the-shelf autoregressive judges and one judge specifically trained for judging; they note that evaluating more judges requires computation but not manual annotation.Source: A Limitations · p. 13

B A brief explanation of the theoretical issues with Cohen’s kappa

  • Scott’s Pi uses a chance baseline based on raters having the same distribution, whereas Cohen’s kappa estimates raters’ label distributions independently, adjusting for their observed differences.Scott’s Pi captures differences beyond chance agreement under an equivalent-rater baseline; when empirical distributions match, the two statistics agree, and they become closer at higher percent alignment.Source: B A brief explanation of the… · pp. 13–14

C Model and dataset details

D Model evaluation prompt templates

E Judge LLM Prompt templates

F Metrics for judge models … H Experiment costs

The sections define agreement metrics, specify human answer-judging guidelines, and report the work’s approximate total experimental costs. They distinguish simple percent agreement from chance-adjusted alignment and describe how annotators handled answer correctness and uncertainty.

F Metrics for judge models

  • Percent agreement is the fraction of annotations on which two annotators agree, and binary alignment ρ = (TP + TN)/(TP + FP + TN + FN).The metric ranges from 0 to 1.Source: F Metrics for judge models · p. 14
  • Scott’s Pi adjusts observed agreement for chance agreement, using observed agreement p_o and hypothetical chance agreement p_e.The passage notes that its values are usually above 0 in real-world situations.Source: F Metrics for judge models · p. 14

Human annotation guidelines

  • Annotators judged answers correct when semantically equivalent to a reference answer, following guidelines refined for consistency and reproducibility.CS graduate students with general English semantic knowledge served as annotators.Source: Human annotation guidelines · pp. 17–18
  • Underspecified answers were incorrect, while extra information was acceptable if it was neither incorrect nor in conflict with the rest of the answer.For example, “December” instead of “December 20” was incorrect, while adding “in Paris” to “December 20” could be correct.Source: Human annotation guidelines · p. 17
  • Unnecessary verbosity did not invalidate an otherwise correct answer, and annotators could flag uncertain judgments as “maybe correct” or “maybe incorrect” for review.The guidelines specify that extra information must not be incorrect or contrast with the rest of the answer.Source: Human annotation guidelines · p. 17

H Experiment costs

  • Approximately 200 GPU-hours, USD 125 in OpenAI credits, and 75 human annotation hours covered the work’s total experimental costs.These totals include final reported experiments plus earlier unreported trials, which cost about 120 GPU-hours, USD 100 in credits, and 50 human hours.Source: H Experiment costs · p. 18

I Statistical reliability of Evaluation sample · J Judge Scores

Across five down-sampled question sets, Scott’s π values remained similar to Figure 1b, with minimal variation for most judges; the section also presents each judge’s scores for each exam-taker model in a figure.

I Statistical reliability of Evaluation sample

J Judge Scores

  • The section presents scores assigned by each judge model to each exam-taker model.The scores are visualized in Figure 1a in Appendix K.Source: J Judge Scores · p. 18

K Exam-taker model base vs chat analysis

Across all four base–chat pairs, base models score better, though the gap narrows under human and GPT-4 Turbo judging. Error analysis suggests that knowledge unlearning, rather than prompt-format difficulty, contributes to the chat models’ weaker performance.

K Exam-taker model base vs chat analysis

  • Base models outperform chat models by a large margin under EM, and the gap remains substantial for all four pairs under human and GPT-4 Turbo judging.The smaller gap under human and GPT-4 Turbo judging suggests the difference is not merely due to chat models’ increased verbosity.Source: K Exam-taker model base vs chat analysis · p. 18
  • 14% of errors are shared between base–chat pairs; almost another 14% are judged correctly only by base models, versus 2.5% judged correctly only by chat models.These are the error comparisons reported for Figure 9b.Source: K Exam-taker model base vs chat analysis · p. 18
  • In a manual analysis of 400 questions for Llama-2 70B and Llama-2 70B-chat, chat models more often answered with a semantically plausible but incorrect entity, while unparseable or entirely incorrect errors did not differ.Source: K Exam-taker model base vs chat analysis · p. 20

L Exam-taker model ranking correlation · M Too much info confuses judges · N Judge models are sensitive to reference order

Most judge models aligned strongly with exam-taker rankings, with Mistral 7B performing on par with GPT-4 Turbo; judges were also sensitive to reference order. The prompt experiments varied how much guidance judges received, while smaller judges sometimes relied on their own knowledge instead of the references.

L Exam-taker model ranking correlation

  • Most judge models achieved Spearman ρ above 0.7, the contains model had the highest ranking stability and ρ, and Mistral 7B performed on par with GPT-4 Turbo.Rankings were assessed by randomly selecting 6 of 9 exam-taker models across 5 samples.Source: L Exam-taker model ranking correlation · p. 20
  • The ranking validation used Spearman rank correlation and measured mean ρ and standard deviation across five samples of six randomly selected exam-taker models.The rankings compared nine exam-taker models.Source: L Exam-taker model ranking correlation · p. 20

M Too much info confuses judges

N Judge models are sensitive to reference order

O Leniency Bias

This section derives estimates of judge correctness and leniency under a coin-flip model of criterion-misaligned judgments, then checks the correctness estimates against Scott’s π. The estimates are highly correlated with Scott’s π.

O Leniency Bias

  • The model treats criterion-misaligned judgments as coin flips independent of answer correctness, deriving Pc and P+ from true-positive and true-negative rates.The derivation considers correct and incorrect responses and the probability that a misaligned judgment happens to be correct.Source: O Leniency Bias · p. 26
  • Pearson correlation between estimated Pc and Scott’s π is 0.98, validating Pc estimates against the agreement measure.Figure 16b reports the correlation for judge models.Source: Figure 16 · p. 27
  • Using human evaluation as the reference, the derived expressions estimate Pc and P+ for different judge models.Figure 16a shows the estimated values.Source: Figure 16 · p. 27

Related papers

Loading 2406.12624v6…