PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge

Yang Ba, Michelle V. Mancenido, Rong Pan

arXiv:2610.14029v1cs.LGcs.AIstat.ML

TL;DR

Majority-vote agreement does not ensure that an individual LLM-judge panel decision is reliable, particularly when judges share biases. SCUR uses a cross-fitted latent measurement model to combine uncertainty, cross-entropy, and perturbation sensitivity into a label-free risk score. Across five benchmarks, it improves error ranking and selective acceptance, including for high-consensus decisions.

  • Problem

    Majority-vote aggregation does not establish that an individual panel decision is reliable, since judges may agree for shared biased reasons.

    Source: §1 · p. 1

  • Method

    SCUR uses a cross-fitted latent measurement model to combine panel uncertainty, measurement cross-entropy, and perturbation sensitivity into a label-free instance-level risk score.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across five benchmarks, SCUR improves error ranking and selective acceptance; mean AUROC rises from .673 to .702 versus PE.

    Source: §4.1 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    Agreement alone does not guarantee decision reliability; SCUR provides relative risk evidence for prioritizing decisions for human review.

    Source: §5 · p. 10, ETHICS STATEMENT · p. 10

    SCUR estimates relative decision risk rather than factual correctness, and low scores do not guarantee correctness, especially with shared systematic judge errors or biases.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.

1 INTRODUCTION

Majority voting can leave an individual panel decision unreliable because judges may agree for shared biased reasons. SCUR estimates instance-level risk from panel behavior and is evaluated across five real-world benchmarks.

1 INTRODUCTION

  • Shared judge biases can make a panel confidently agree for the wrong reason, so agreement alone does not establish that an individual decision is trustworthy.The paper calls this failure mode “correlation collapse.”Source: §1 · p. 1
  • SCUR combines panel uncertainty, cross-fitted measurement cross-entropy, and perturbation sensitivity into a label-free risk score for individual decisions.The score is intended to support automatic acceptance or human review.Source: §1 · p. 2, Figure 1 · p. 2
  • Across five benchmarks, SCUR consistently outperforms uncertainty-only baselines on majority-vote error ranking, selective trust, and risk stratification.The experiments cover pairwise preference, objective correctness, and instruction following; SCUR also identifies high-consensus risky items and transfers to other aggregation rules.Source: §1 · p. 2

2 RELATED WORK

Prior work improves evaluator alignment or estimates judge and annotator quality, while this paper shifts attention to the reliability of each already-produced panel decision. It adapts measurement-system analysis to LLM evaluation.

2 RELATED WORK

  • LLM-evaluation work studies evaluator artifacts and multi-judge networks, while this paper targets reliability on individual instances rather than average judge quality.The cited artifacts include position bias, verbosity bias, self-enhancement, and prompt sensitivity.Source: §2 · pp. 2–3
  • Crowdsourcing and noisy-label methods estimate latent labels or aggregate decisions; this framework instead assesses an existing panel decision without ground-truth labels.Examples discussed include Dawid–Skene, GLAD, MACE, and Bayesian annotation models.Source: §2 · p. 3
  • Pairwise preference models infer latent quality from comparisons, whereas this work models evaluator bias and stochasticity to assess the resulting panel decision’s trustworthiness.The related models include Bradley-Terry and Luce-choice models.Source: §2 · p. 3
  • The framework adapts measurement-system analysis by treating items, judges, and perturbations as sources of measurement variation.This connects the work to reliability analysis and crowdsourced annotation-quality research.Source: §2 · p. 3

3 METHODOLOGY

SCUR models judge-panel decisions under designed perturbations, derives complementary risk signals from cross-fitted predictions, and combines normalized evidence into a relative risk score. Its benchmark-average figure reports SCUR ahead of four baselines on all three displayed metrics.

3.1 LATENT MEASUREMENT MODEL

  • The latent measurement model decomposes judge verdict log-odds into item preference, judge tendency, shared and judge-specific condition effects, and residual variation.It estimates item preference jointly from panel responses; cross-fitting excludes judge j’s own response when predicting that judge’s verdict.Source: §3.1 · pp. 3–4
  • The LMM is fit on one data split, a second supplies reference distributions, and a held-out split evaluates risk without using decision-error labels for fitting or estimation.The split is 50/30/20 across fitting, reference, and evaluation data.Source: §3.1 · p. 4

3.2 EVIDENCE-BASED RISK SIGNALS

  • The three risk signals capture pooled-preference ambiguity, judge-response uncertainty or mismatch with the cross-fitted panel structure, and sensitivity to perturbation conditions.Perturbation sensitivity separates panel-wide changes from additional judge-specific changes; the paired conditions include answer-order and rubric changes.Source: §3.2 · pp. 4–5

3.3 LABEL-FREE RISK SCORE

  • Unlabeled reference ECDFs normalize available signals to percentiles, and SCUR combines their complements geometrically, accounting for whether perturbation evidence is available.A high percentile in any evidence type increases SCUR; the resulting score ranks items for automatic acceptance or human review.Source: §3.3 · p. 5
  • SCUR has the highest equally weighted average AUROC and 1 −AURC, and the lowest Err@50, among the four baselines across five benchmarks.Table 1’s caption also reports that SCUR outperforms baselines on each benchmark.Source: Figure 2 · p. 6, Table 1 · p. 6

4 EXPERIMENTS

Across five benchmarks, the experiments assess SCUR’s error ranking, high-consensus detection, and use across aggregation rules. SCUR ranks majority-vote errors more effectively than uncertainty baselines and identifies risk even among high-consensus decisions.

4.1 BENCHMARK EVALUATION

  • .702 AUROC, .841 1 −AURC, and .147 Err@50 are SCUR’s averages, compared with PE’s .673, .818, and .174, respectively.Across the five benchmarks, SCUR also has the highest mean AUROC and 1 − AURC and lowest mean Err@50 on each benchmark.Source: §4.1 · p. 8
  • Error rises across SCUR risk quintiles on four benchmarks and in the two highest-risk groups on LLMBar, while coverage curves generally show lower error than baselines.Estimates and baseline ordering are more variable on the smaller LLMBar and LLMEval2 benchmarks, but SCUR retains clearer separation between lower- and higher-risk items.Source: Figure 3 · p. 7, §4.1 · p. 8

4.2 HIGH-CONSENSUS BLIND SPOTS

  • SCUR has the highest mean AUROC on the high-consensus subset across all five benchmarks, with more variable estimates on LLMBar and LLMEval2.The subset includes items with at least 80% judge agreement; matched-vote and panel-mean-confidence analyses also find SCUR outperforms the baselines.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 RISK GENERALIZABILITY

  • Across five benchmarks and five aggregation rules, retaining the lowest-risk 80% and 50% improves accuracy by an average of 4.6 pp and 12.2 pp, respectively, versus accepting all decisions.The rules include majority vote, Dawid–Skene, MACE, GLAD, and CARE; SCUR scores are used to rank items across these rules.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4 PANEL-DECISION DIAGNOSIS

  • The instance-level diagnostics show that elevated risk can accompany strong agreement, arise without perturbation evidence, or reflect several moderately elevated signals.The risk signals provide complementary review evidence without requiring a gold label; MMLU Instance ID 19396 is shown as a low-risk contrast.Source: §4.4 · p. 9, Table 2 · p. 9

5 CONCLUSION

SCUR estimates risk for individual LLM judge-panel decisions using three interpretable signals. Across five benchmarks, it improves error ranking and selective acceptance, including for high-consensus errors.

5 CONCLUSION

  • Across five benchmarks, SCUR improves error ranking and selective acceptance, identifies errors difficult to detect under high panel consensus, and is robust to different aggregation rules.Its preference-uncertainty, measurement-cross-entropy, and perturbation-sensitivity signals also provide interpretable evidence for human review.Source: §5 · p. 10
  • SCUR is a label-free risk score for assessing individual LLM judge-panel decisions.A cross-fitted latent measurement model provides three signals: preference uncertainty, measurement cross-entropy, and perturbation sensitivity.Source: §5 · p. 10
  • The results show that agreement alone does not guarantee decision reliability, and SCUR helps prioritize decisions for automatic acceptance or human review.SCUR is intended to support interpretable decisions about which individual evaluations warrant review.Source: §5 · p. 10

AI USE STATEMENT

The authors used generative AI tools for research-methodology feedback, implementation and debugging, results interpretation, and manuscript drafting and editing. They also used LLMs as judges in two benchmark experiments, and the authors reviewed the AI-assisted work and take responsibility for the paper.

AI USE STATEMENT

  • Generative AI tools provided feedback on research methodology and experimental design and assisted with implementation and debugging.The authors state that AI-assisted code and analyses were verified by them.Source: AI USE STATEMENT · p. 10
  • Generative AI tools supported results interpretation and presentation and helped draft and edit manuscript text for clarity and readability.The authors reviewed AI-assisted research decisions and manuscript text.Source: AI USE STATEMENT · p. 10
  • LLMs served as judge models in the LLMBar and LLMEval2 experiments, and the authors take responsibility for the methodology, experiments, results, claims, and final paper.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

SCUR is intended to support rather than replace human judgment, and its score represents relative decision risk rather than factual correctness. The appendix describes the latent measurement model, fitting conventions, held-out inference, and risk baselines.

ETHICS STATEMENT

  • In high-stakes settings, SCUR should prioritize human review rather than serve as the basis for fully automated decisions.A low score does not guarantee correctness, especially when judges share systematic errors or biases.Source: ETHICS STATEMENT · p. 10

A MODEL STRUCTURE AND ESTIMATION

A.1 MEASUREMENT MODEL AND PARAMETER INTERPRETATION

  • The model is fitted to log-odds, retains judge verdicts for majority voting and vote-based baselines, and treats each non-original condition as a categorical intervention.Source: §A.1 · p. 13

A.2 SHRINKAGE AND COORDINATE CONVENTIONS

  • The estimator uses ridge least-squares updates, with penalties that shrink q, b, δ, and κ toward zero and a toward one, while enforcing positivity for a.The model separates shared item and condition components from judge-specific responses and condition deviations.Source: §A.2 · p. 14

A.3 OPTIMIZATION AND HELD-OUT ITEM INFERENCE

  • Items are split 50/30/20 into fit, reference, and evaluation sets; held-out-item margins exclude the evaluated judge’s responses.Reference items set frozen signal distributions, and neither model fitting nor reference normalization uses ground-truth labels.Source: §A.3 · p. 14

B RISK BASELINES

  • The reported risk baselines are vote entropy, predictive entropy, Dawid–Skene posterior entropy, and mean per-judge MSP uncertainty.Source: B RISK BASELINES · p. 15

B.1 COMMON INPUTS AND EVALUATION PROTOCOL

The protocol uses judge probabilities and binary verdicts, with gold labels reserved for evaluation. The appendix distinguishes four uncertainty baselines by how they represent panel or judge uncertainty.

B.1 COMMON INPUTS AND EVALUATION PROTOCOL

  • The protocol indexes each item’s available judges and their original-condition probabilities and binary verdicts; gold labels are used only for evaluation.Source: §B.1 · p. 15

B.2 PREDICTIVE ENTROPY

  • Predictive entropy applies binary entropy to the mean of judges’ LLM probabilities, without fitting the latent measurement model.It is largest when the panel-mean preference is 0.5; that can reflect individually uncertain judges or confident judges favoring opposite answers.Source: §B.2 · p. 15

B.3 VOTE ENTROPY

  • Vote entropy uses binary verdicts: it is zero for unanimous votes and largest for an evenly divided panel.Items with the same vote split receive the same score, even when their LLM probabilities differ; unanimous correct and incorrect panels both receive zero.Source: §B.3 · p. 15

B.4 DAWID–SKENE POSTERIOR ENTROPY

  • Dawid–Skene posterior entropy measures uncertainty in a posterior estimated from binary votes using a class prior and judge confusion matrices.The model assumes judges are conditionally independent given an item’s unobserved label; a confidently wrong posterior can still have low entropy.Source: §B.4 · pp. 15–16

B.5 MEAN PER-JUDGE MSP UNCERTAINTY

  • Mean per-judge MSP uncertainty averages one minus each judge’s maximum softmax probability, so larger values indicate more uncertain judges.It remains low when every judge is confident, including when confident judges disagree.Source: §B.5 · p. 16

C.1 BENCHMARK CONSTRUCTION AND VALIDATION LABELS

The evaluation uses five benchmarks spanning pairwise preference, hallucination detection, multiple-choice correctness, and instruction following. Each benchmark’s stated reference and evaluation splits support assessing whether the original-condition panel verdict matches its benchmark label.

C.1 BENCHMARK CONSTRUCTION AND VALIDATION LABELS

  • Chatbot-Arena contains 17,325 items and 17 judges; each seed uses 5,198 reference items and 3,465 evaluation items, with human pairwise preferences as the error target.The dataset contains 589,050 judge observations.Source: §C.1 · p. 16
  • HaluEval contains 20,000 items and 6 judges; each seed uses 6,000 reference items and 4,000 evaluation items, with benchmark labels determining verdict correctness.The benchmark has 120,000 observations.Source: §C.1 · p. 16
  • MMLU contains 28,084 items and 17 judges; each seed uses 8,425 reference items and 5,617 evaluation items, with the objective answer key providing validation labels.The benchmark has 477,428 judge observations.Source: §C.1 · p. 16
  • LLMBar contains 419 items and 5 judges across 6 conditions; each seed uses 126 reference items and 83 evaluation items, validated against expert preferences.The benchmark has 12,570 observations.Source: §C.1 · p. 16
  • LLMEval2 uses a 200-item pairwise-preference subset and 5 judges across 6 conditions; each seed uses 60 reference items and 40 evaluation items, validated against source preference labels.The subset yields 6,000 observations.Source: §C.1 · p. 16

C.2 PERTURBATIONS AVAILABLE FOR EACH BENCHMARK

The appendix describes perturbations that change answer order or judging criteria while keeping the task and candidate texts fixed. It also specifies the judge panel and prompt examples used to instantiate these conditions.

C.2 PERTURBATIONS AVAILABLE FOR EACH BENCHMARK

  • For LLMBar and LLMEval2, each answer order is crossed with three rubrics, producing six conditions paired against the original-order, standard-rubric condition for the same item and judge.An answer-order swap changes displayed positions while probabilities are mapped back to the original candidate.Source: §C.2 · p. 17

C.3 JUDGE PANEL

  • LLMBar and LLMEval2 use the same five judges, each evaluating every item in all six conditions for 30 judgments per item with temperature zero and a one-token A/B output.Verdict-token log-probabilities are expressed relative to the original candidate; the rubrics are standard, strict, and evidence-first.Source: §C.3 · p. 17

C.4 PROMPTS AND PERTURBATION EXAMPLES

  • The answer-order example swaps the candidate responses’ displayed positions while retaining the task, standard criterion, and A/B label semantics.Across the examples, the original condition uses the original order and standard rubric.Source: §C.4 · p. 18
  • The examples illustrate a strict rubric that changes the criterion while preserving the task, candidates, and order, and an evidence-first rubric that asks for internal factual-support and instruction-compliance checks.Both modified rubrics retain a single A/B verdict; the evidence-first example supplies no external evidence or written explanation.Source: §C.4 · pp. 19–20

D HIGH-CONSENSUS ANALYSIS

Figure 6 tests error ranking among items with identical vote patterns and among items with high panel-mean confidence. SCUR distinguishes incorrect from correct decisions in both settings, though the reported results vary by benchmark.

D HIGH-CONSENSUS ANALYSIS

  • SCUR has the highest mean AUROC among five methods on every benchmark in the high-confidence subset; on Arena it scores .676 versus .596 for the best baseline, MSP.LLMBar remains near chance at .539, and only 3–5 LLMEval2 errors per split limit estimate precision.Source: Figure 6 · p. 21, D HIGH-CONSENSUS ANALYSIS · p. 21
  • .693 AUROC for SCUR, versus .610 for PE and .621 for MSP, when ranking errors among items with the same vote pattern; VE and fitted DS entropy are .5 within each group.The same judges support each candidate in the compared items, so the vote split and judge identities are fixed.Source: D HIGH-CONSENSUS ANALYSIS · pp. 20–21, Figure 6 · p. 21
  • The high-confidence test selects items with maximum panel-mean probability at least .8, equivalent to predictive entropy no greater than H(.8).It tests whether errors remain distinguishable within a low-predictive-entropy subset.Source: D HIGH-CONSENSUS ANALYSIS · p. 21

Related papers

Loading 2610.14029v1…