PaperlayerView exact revision on arXiv

Source-linked AI summary

Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

Rajarshi Haldar, Julia Hockenmaier

arXiv:2510.27106v1cs.CL

TL;DR

Evaluating open-ended NLG is difficult, and LLM judges' agreement with human ratings does not reveal whether they are consistent across repeated runs. The paper measures this self-reliability across NLG tasks and benchmarks, finding that inconsistency persists across tasks and metrics, even for newer, larger models.

  • Problem

    LLM-judge evaluations commonly compare ratings with human judgments, but self-reliability across repeated runs is often unmeasured for both LLM and human judges.

    Source: §1 · p. 1

  • Method

    The study measures judge agreement with itself across repeated runs under fixed settings and examines the issue across NLG benchmarks and tasks.

    Source: §1 · p. 1

  • Results

    Low self-reliability persists across tasks and metrics; newer, larger models such as Qwen 3 are more consistent than Llama 3.1 but often remain below standard reliability thresholds.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Evaluation frameworks should incorporate self-reliability measures, and aggregating multiple runs can improve agreement with human judgments whereas eliminating variance can hurt performance.

    Source: Account For Possible Self-Reliability… · p. 8

    The study does not quantify whether summarization or multi-turn conversation subjectivity worsens self-reliability relative to more objective tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.

1 Introduction

The paper argues that evaluating open-ended NLG remains difficult and that LLM judges need to be assessed for consistency across repeated runs, not only agreement with human ratings. It studies this issue across NLG tasks and benchmarks.

Motivation

  • Traditional metrics such as BLEU and BERTScore can work for translation with multiple references but fail on open-ended tasks, while LLM-as-a-judge offers a scalable automatic alternative.LLM ratings are commonly compared with human ratings treated as the gold standard.Source: §1 · p. 1

Problem

  • Self-reliability—the agreement of a judge with itself across repeated runs under the same settings—is often unmeasured for both LLM and human judges.For LLMs, the repeated runs use the same prompt and hyperparameters.Source: §1 · p. 1

Research scope

  • Human self-reliability data is absent from existing benchmarks, and correlation or accuracy can overestimate LLM–human agreement because they do not account for chance agreement.The paper examines binary summary consistency, four 1–5 summary metrics, and preference rankings of multi-turn conversations.Source: §1 · p. 1

Contributions

  • LLM judges show low agreement across repeated runs; eliminating sampling variability also hurts agreement with human judgments, and the problem persists across tasks and benchmarks.The paper also discusses recommendations for more robust NLG evaluation.Source: §1 · p. 2

2 Background

Prior NLG evaluation work describes trade-offs among human evaluation, automatic metrics, and LLM judges. This paper situates repeated-run consistency as a reliability question that existing evaluations often omit.

Evaluating Natural Language Generation

  • Human evaluation is costly, time-intensive, and prone to unreliability and bias, while automatic metrics scale better but may miss human notions of quality on subjective tasks.The passage notes that only 18% of 135 surveyed papers included agreement analysis.Source: §2 · p. 2

LLM-as-a-judge

  • LLM judges assign scores, rankings, or labels and are commonly compared with human judgments using correlation or percentage agreement.Prior work reports advantages over conventional metrics on NLG tasks, alongside challenges including bias, uncertainty, adversarial vulnerability, and domain-specific gaps.Source: §2 · p. 2

Variability in LLM Judgments

  • Earlier work observed rating variability or sampled multiple GPT-4 scores, while this paper studies its implications, measurement, cross-task extent, and effects of setting temperature to zero.The paper frames this repeated-run variability as self-reliability or intra-rater reliability.Source: §2 · p. 2

Self-reliability

  • NLP annotation and LLM-evaluation studies frequently omit self-reliability metrics, and the authors report unstable LLM self-judgments in their experiments.Self-reliability concerns one evaluator's consistency across repeated assessments, unlike inter-rater reliability.Source: §2 · pp. 2–3

3 Experiments

The experiments examine repeated-run judging across three NLG benchmarks, covering binary factual-consistency labels, multi-metric summary ratings, and conversational preferences. They use common judges and agreement procedures adapted to each benchmark's rating scale.

3 Experiments

  • The experiments cover binary classification of machine-generated summaries, multi-metric summary ratings, and preference judgments on multi-turn conversations.MT-Bench differs from summarization by evaluating multi-turn conversational and instruction-following ability.Source: §3 · p. 3

3.1 Datasets

  • SummaC combines six datasets into a binary consistent/inconsistent classification task, while SummEval has 1,700 examples rated 1–5 on coherence, consistency, fluency, and relevance.SummEval includes ratings from three experts and five crowd-sourced annotators; reported inter-annotator kappa improved to 0.7127 in the second expert round.Source: §3.1 · p. 3
  • MT-Bench examples pair two models' responses to a question and follow-up, with a rater choosing model_a, model_b, or tie.The filtered evaluation subset contains 761 examples with at least two human ratings, drawn from 2,400 examples.Source: §3.1 · pp. 3–4

3.2 Experimental Settings

  • Three LLM judges rated the same generations independently for three runs with unchanged prompts and settings, and reliability was computed using Krippendorff’s Alpha.The authors tried up to ten runs initially and found no significant effect of run count on self-reliability.Source: §3.2 · p. 4
  • SummEval ratings were elicited independently for each of four metrics per run and analyzed with ordinal distance; MT-Bench followed its benchmark setup and added three-run consistency measurements.MT-Bench analyses also include human–human and human–LLM agreement, using GPT-4 human judgments as the LLM–human reference.Source: §3.2 · p. 4

3.3 Agreement Metric

  • The study uses Krippendorff’s Alpha because it corrects for chance agreement and supports varying annotator counts, missing data, and scale-specific distances.For SummaC it uses nominal distance and reports balanced accuracy for LLM–human comparisons; for SummEval it uses ordinal distance, and for MT-Bench it reports accuracy and ordinal Alpha.Source: §3.3 · pp. 4–5

4 Intra-Rater Reliability Results

LLM judges show low self-reliability across SummaC, SummEval, and MTBench, with reliability varying by benchmark and metric. On SummaC, reliability showed no significant change when experiments increased to five runs.

4 Intra-Rater Reliability Results

  • SummaC agreement was low across models and runs, though newer, larger models improved; Qwen 3 approached the 0.8 agreement threshold.Repeating experiments for up to five runs produced no significant difference in self-reliability.Source: §4 · p. 5
  • On MTBench, Krippendorff’s Alpha was 0.265 for Llama 3.1, 0.507 for Deepseek-R1, and 0.563 for Qwen 3; all were below 0.8.Qwen 3 gave the same judgment across all three runs in 61.3% of cases.Source: §4 · pp. 5–6
  • On SummEval, Deepseek-R1 and Qwen 3 were highly self-reliable on Coherence and Consistency but very unreliable on Fluency; Llama performed best only on Fluency.The passage says reliability depends heavily on the metric.Source: §4 · p. 5, Figure 4 · p. 6

5 Are LLM Judges a Reliable Substitute for Human Judges?

Across SummaC, SummEval, and MTBench, agreement with human judgments varies by model, dataset, metric, and judge expertise. The results show both performance trade-offs and limits to treating LLM judges as substitutes for humans.

5.1 SummaC

  • Qwen 3 significantly outperforms the baselines overall on SummaC, while Deepseek-R1 is competitive and Llama 3.1 performs poorly.The baselines were fine-tuned for the task, whereas the LLMs were prompted off-the-shelf.Source: Figure 5 · p. 6, §5.1 · p. 6
  • On SummaC, majority voting across three runs raises balanced accuracy above the expected single-run score, and for Deepseek-R1 and Qwen 3 above each model's best single run.Disabling sampling degrades performance, leaving a trade-off between self-reliability and agreement with human judgments.Source: Table 2 · p. 6, §5.1 · pp. 6–7
  • SummaC results vary by dataset: Deepseek-R1 performs best on XSumFaith, while fine-tuned SummaCCONV performs best on FactCC.The paper notes that newer and larger models generally perform better, but do not lead on every dataset.Source: §5.1 · p. 6

5.2 SummEval

  • On SummEval, experts agree most on consistency (0.798) and least on relevance (0.398), while Turker agreement is 0.48–0.51 across metrics.Expert–Turker agreement reaches a maximum of 0.247 for relevance.Source: Figure 6 · p. 7, §5.2 · p. 7
  • LLM agreement with SummEval human experts varies by metric, dataset, and expertise; even Qwen 3's best reported agreement, 0.726 for consistency, remains below the stated substitution threshold.Agreement can be modest or negative on subjective metrics such as fluency, and score distributions also differ across judge types and metrics.Source: Figure 7 · p. 7, §5.2 · p. 7

5.3 MTBench

  • On MTBench, human agreement is 0.478 by Krippendorff’s Alpha versus 0.827 accuracy; GPT-4 has higher agreement than other LLMs, but remains below human agreement.The paper says accuracy may inflate apparent agreement because it does not account for chance agreement; human agreement is below 0.8 and lower than on SummEval.Source: §5.3 · pp. 7–8, Table 3 · p. 8

6 Discussion

Low intra-rater reliability persists across metrics, scoring scales, benchmarks, and tasks, complicating evaluation and meta-evaluation. The discussion recommends measuring self-reliability, reducing variance without eliminating it, and collecting repeated human judgments.

6.1 Takeaways

  • SummaC has only one human judgment per example, while SummEval has multiple judgments that could support further reliability analysis.Source: §6.1 · p. 8
  • It remains unclear whether low reliability reflects LLM limitations or ambiguous summarization metrics; the authors call for study on other-task benchmarks.Source: §6.1 · p. 8
  • Low intra-rater reliability persists across SummEval metrics and scoring scales and across MT-Bench tasks, while accuracy inflates agreement relative to Krippendorff’s Alpha.Source: §6.1 · p. 8

Account For Possible Self-Reliability Issues

7 Conclusion

The paper analyzes intra-rater reliability in LLM-as-a-judge frameworks and finds inconsistency across runs, tasks, and metrics. It also notes risks in relying on single-run judgments and in treating human judgments as an uncomplicated standard.

  • Newer, larger Qwen 3 is more consistent than Llama 3.1, but both often fall short of standard reliability thresholds.Source: §7 · p. 9
  • Agreement with human judges can be noisy, and linguistic-style biases may affect scores; fluency also remains difficult to assess reliably for humans and machines.Source: §7 · p. 9
  • Expert and crowdsourced judges disagree markedly, raising the question of which human preferences LLM judges should model.Source: §7 · p. 9
  • Although LLM judges are scalable, reliability challenges remain relevant to frameworks intended to distinguish text quality.Source: §7 · p. 9

Limitations

The paper leaves several questions open, including how reasoning traces relate to self-reliability and whether fine-tuning improves performance. Its experiments also use only open-weight models.

  • The study does not examine relationships between reasoning traces and self-reliability or the effect of fine-tuning on performance.Source: Limitations · p. 9
  • The authors have not quantified whether subjectivity exacerbates self-reliability issues by comparing summarization or multi-turn evaluation with more objective tasks.Source: Limitations · p. 9
  • The experiments are limited to open-weight models, leaving comparison with proprietary models such as GPT-5 and Claude-4 for future work.Source: Limitations · p. 9

A More Details About Experimental Setup

The appendix describes hardware, model settings, and task-specific evaluation prompts across SummaC, SummEval, and MT-Bench.

B Agreement Metric: Krippendorff’s Alpha

Krippendorff’s alpha measures chance-corrected agreement among coders, using disagreement distances suited to the data’s measurement level.

B Agreement Metric: Krippendorff’s Alpha

  • Krippendorff’s alpha applies to two or more coders and diverse measurement levels, and corrects agreement for what would be expected by chance.It accommodates missing data and unequal sample sizes.Source: B Agreement Metric: Krippendorff’s Alpha · p. 13

B.1 Mathematical Notation

  • Observed disagreement between raters is compared with chance-expected disagreement; α=1 indicates perfect reliability, while α=0 indicates agreement only by chance.The expected disagreement uses the overall value frequencies as if assignments were randomly permuted among raters.Source: §B.1 · pp. 13, 15

B.2 Distance Functions

  • The distance function δ is chosen for the measurement level, assigns zero distance to agreement and positive distance to disagreement, and determines how disagreement is accumulated.Source: §B.2 · p. 15

B.2.1 Nominal Distance

  • For nominal labels, exact matches have zero distance and all mismatches count equally; this metric was used for SummaC’s binary labels.With nominal distance, α is a chance-corrected proportion of exact agreement.Source: §B.2.1 · pp. 15–16

B.2.2 Ordinal Distance

  • For ordered categories, ordinal distance penalizes larger rank gaps more heavily; it was used for SummEval’s Likert labels and MT-Bench’s modela/modelb/tie labels.Adjacent ratings incur less disagreement than ratings at opposite ends.Source: §B.2.2 · p. 16

B.2.3 Interval Distance

  • Interval distance squares numerical differences, but the paper does not use it because none of its datasets contain numerical labels.This distance is intended for scales where numerical differences are meaningful.Source: §B.2.3 · pp. 16–17

C.1 Dataset Statistics of SummaC Benchmark

On SummaC, few-shot and chain-of-thought prompting did not appreciably change self-reliability, and neither strategy produced a significant accuracy jump. The section also reports test-example counts across the benchmark’s six datasets.

C.1 Dataset Statistics of SummaC Benchmark

C.2 Impact of CoT and Few-shot Prompting on Intra-Rater Reliability and Accuracy on SummaC

  • Few-shot and chain-of-thought prompting did not appreciably improve SummaC self-reliability over three runs; few-shot slightly decreased consistency, nonsignificantly.Source: Table 5 · p. 17, §C.2 · p. 17
  • Few-shot prompting slightly improved Llama 3.1 and Deepseek-R1 accuracy, but neither prompting strategy produced a significant performance jump.The authors suggest few-shot prompting may not work well with some reasoning models, while chain-of-thought may not improve models that already reason implicitly.Source: Table 6 · p. 17, §C.2 · pp. 17–18
  • Few-shot prompts added five positive and five negative examples, chain-of-thought prompted step-by-step reasoning, and the Both setting combined them.Source: §C.2 · p. 17

D Additional Analysis of Agreement Between Judges in SummEval

SummEval judge categories show different score distributions across metrics, while the MT-Bench experiments use only examples with multiple human judgments.

E.1 Breakdown of Examples in MT-Bench

  • The MT-Bench experiments excluded examples with only one human judgment and retained examples with two to five judgments.Table 7 reports the number of examples in each judgment-count group.Source: §E.1 · p. 18, Table 7 · p. 18

D.1 Distributions of Scores in SummEval From Different Populations of Raters

  • Turker score distributions barely vary across SummEval metrics, while four other judge categories distinguish metrics yet retain distinct score distributions (Figure 12).The authors interpret the differing distributions as evidence that judge categories use different internal definitions of a metric, limiting agreement.Source: §D.1 · p. 18, Figure 12 · p. 19

Related papers

Loading 2510.27106v1…