PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

Jaehun Jung, Faeze Brahman, Yejin Choi

arXiv:2407.18370v1cs.LGcs.CL

TL;DR

LLM judges offer a scalable alternative to human annotation, but their reliability is uncertain because strong models can be biased and overconfident. This paper selectively trusts judge decisions, estimates confidence with Simulated Annotators, and escalates from cheaper to stronger models when needed. It reports high human agreement with substantial coverage and reduced evaluation overhead.

  • Problem

    LLM evaluation can scale beyond practical human annotation, but even strong judges exhibit biases and overconfidence, and lack a provable guarantee of human agreement.

    Source: §1 · p. 1

  • Method

    Selective evaluation trusts a judge only when its confidence is sufficient, using Simulated Annotators for confidence estimation and escalating from cheaper judges to stronger ones when needed.

    Source: §1 · p. 2

  • Results

    Over 80% human agreement in ChatArena is achieved while covering 79.1% of samples, with 88.1% evaluated by Mistral-7B or GPT-3.5 instead of GPT-4.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across summarization and real-world user-chatbot interaction, Cascaded Selective Evaluation significantly reduces evaluation overhead while guaranteeing high human agreement.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework, human agreement can be provably guaranteed -- such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce Simulated Annotators, a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Evaluation, where we use cheaper models as initial judges and escalate to stronger models only when necessary -- again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guarantees over 80% human agreement with almost 80% test coverage.

1 INTRODUCTION

The introduction argues that reliable LLM evaluation must assess whether humans would agree with a judge, not merely use its preference. It proposes selective and cascaded evaluation to guarantee human agreement while preserving coverage and reducing evaluation overhead.

1 INTRODUCTION

  • LLM evaluation is attractive at scale, but even strong judges exhibit systematic biases and overconfidence, and lack a provable reliability guarantee.The introduction frames this as the question of how to guarantee the reliability of LLM-based evaluation.Source: §1 · p. 1
  • For an evaluated unseen instance, the framework guarantees that the LLM preference agrees with humans at least 1 − α, for user-defined risk level α.This formalizes the paper's goal of a user-specified human-agreement guarantee.Source: §1 · p. 1
  • Selective evaluation estimates whether humans would agree with a judge, then trusts its result only when confidence is sufficient; fixed sequence testing on a small calibration set supports the guarantee.The guarantee is both theoretical and empirical.Source: §1 · p. 1
  • Simulated Annotators use in-context learning to model diverse annotator preferences and estimate confidence from their agreement, improving calibration and failure prediction without external supervision.This confidence estimation enables high coverage while satisfying the prescribed human agreement level.Source: §1 · p. 2
  • Cascaded Selective Evaluation starts with a substantially cheaper judge and escalates to a stronger model only when confidence is insufficient, while maintaining a guarantee of human agreement.The abstention policy is automatically determined from the user's specified risk tolerance, rather than chosen manually.Source: §1 · p. 2, Figure 1 · p. 2
  • Over 80% human agreement in ChatArena is achieved while covering 79.1% of samples, with 88.1% evaluated by Mistral-7B or GPT-3.5 instead of GPT-4.The method is tested across summarization and real-world user-chatbot preferences and significantly reduces evaluation overhead while guaranteeing high agreement.Source: §1 · p. 2

2 CASCADED SELECTIVE EVALUATION

Selective evaluation accepts an LLM judgment only when its confidence meets a calibrated threshold, enabling a user-specified guarantee of agreement with human preferences. Simulated Annotators estimates confidence from simulated annotator preferences, and cascades apply this approach from cheaper judges to stronger ones when needed.

2.1 PROVIDING HUMAN AGREEMENT GUARANTEE

  • Theorem 1 guarantees that, with probability at least 1 −δ, the calibrated selective evaluator’s human-disagreement risk is at most α.The threshold is selected using a calibration set of human preferences and fixed-sequence testing of an upper confidence bound on risk.Source: §2.1 · p. 3

2.2 SIMULATED ANNOTATORS

  • Simulated Annotators prompts the judge K times with examples from each of N annotators and ensembles predictions to estimate confidence from their agreement.When simulated annotators disagree, confidence is low; the method models individual preferences or, when those data are unavailable, majority preferences.Source: §2.2 · pp. 4–5
  • 50% lower ECE and 13% higher AUROC for GPT-4: Simulated Annotators outperforms popular confidence measures, and improves confidence reliability for weaker judges.Canonical confidence measures tend to overestimate agreement with human majorities, including for GPT-4-turbo; the Table 1 caption reports gains in calibration and failure prediction, especially for GPT-3.5-turbo and Mistral-7B.Source: §2.2 · pp. 4–5, Table 1 · p. 4

2.3 CASCADING SELECTIVE EVALUATORS

  • Cascaded Selective Evaluation starts with a cheaper, weaker judge and escalates to stronger judges only when confidence is insufficient, calibrating each threshold to retain risk control.This allows evaluation at lower cost than starting with the strongest model while maintaining a rigorous human-agreement guarantee.Source: §2.3 · p. 5

3 EXPERIMENTAL RESULTS

Across summary and chat-assistant evaluation, Cascaded Selective Evaluation meets target human-agreement levels while often using cheaper judges. Analyses show its abstentions track human-perceived subjectivity, and its guarantees persist under distribution shift and across annotator counts and judge compositions.

3.1 EVALUATING GENERATED SUMMARIES

  • At 80% human agreement, 75% of TL;DR evaluations use Mistral-7B or GPT-3.5; guarantee success exceeds 90% across runs.The approach satisfies target agreement across levels and outperforms GPT-4 without abstention; baselines do not maintain meaningful guarantee success without sacrificing coverage.Source: Figure 3 · p. 6, §3.1 · p. 6, Table 2 · p. 6

3.2 EVALUATING LLM-BASED CHAT ASSISTANTS

  • On ChatArena, the method achieves 91% guarantee success while using GPT-4 for 17.5% of evaluations, and guarantees up to 80% agreement with high probability on Auto-J.GPT-4 without abstention reaches only 63.2% agreement on Auto-J; the authors note that Auto-J includes a tie label.Source: §3.2 · p. 7, Figure 4 · p. 7, Table 3 · p. 7

3.3 UNDERSTANDING ABSTENTION POLICY

  • Abstained ChatArena samples have lower human inter-annotator agreement than evaluated samples, while length ratio and token overlap show no significant differences.Mean agreement is 0.815 for abstained samples and 0.902 for evaluated samples; the IAA difference is significant (p < 1e−8).Source: §3.3 · pp. 7–8, Table 4 · p. 7

3.4 EVALUATION UNDER DISTRIBUTION SHIFT

  • Under a model-disjoint calibration-to-test distribution shift, high human agreement is maintained in more than 90% of runs across tested α levels, with a small coverage degradation.The evaluation randomly splits ChatArena by evaluated models and repeats the procedure over 1,000 splits.Source: §3.4 · p. 8

3.5 IMPACT OF NUMBER OF SIMULATED ANNOTATORS

  • On ChatArena, larger simulated-annotator counts consistently improve coverage; even small counts retain high agreement and reduce API cost by up to 40% versus GPT-4 without abstention.The comparison tests N = 5, 3, 2, and 1.Source: §3.5 · p. 8, Table 5 · p. 8

3.6 IMPACT OF JUDGE MODEL COMPOSITION

  • At an 80% target agreement level, weaker judge cascades retain the guarantee while reducing costs by up to 87.4%; stronger cascades save up to 78.5%.The cascades balance evaluation cost and coverage rather than compromising precision; practitioners can choose stronger cascades for better coverage or weaker ones for lower costs.Source: §3.6 · p. 9, Table 6 · p. 9

4 RELATED WORKS

Prior work uses LLMs as scalable human-evaluation proxies, smaller expert judges, and multi-agent review, but often lacks provable reliability guarantees. This paper distinguishes its focus on bounded disagreement risk for LLM evaluation, unsupervised confidence estimation, and cascaded inference to reduce cost while retaining reliability.

Related work

  • LLM evaluation, smaller expert judges, and multi-agent review have been explored, but these approaches often lack a provable reliability guarantee.The cited literature also reports susceptibility of LLM judges to cognitive biases and self-preference.Source: §4 · p. 9
  • Other work applies statistical guarantees to risks such as hallucination rates and medical-decision FDR, often using conformal methods for marginal risk control.Separate work studies truthfulness fine-tuning and abstention when relevant knowledge is lacking.Source: §4 · p. 9
  • This work targets an exact upper bound on disagreement risk conditional on the calibration set, uses unsupervised confidence, and cascades judges to reduce inference cost while guaranteeing reliability.The passage contrasts its unsupervised confidence measure with supervised estimators.Source: §4 · p. 9

5 CONCLUSION

Cascaded Selective Evaluation combines a guarantee of human agreement with improved confidence estimation and dynamically selected judges to reduce evaluation overhead.

5 CONCLUSION

  • Cascaded Selective Evaluation provides LLM-based evaluation with a robust guarantee of human agreement.Source: §5 · p. 9
  • Simulated Annotators improves judge confidence estimation without external supervision.Source: §5 · p. 9
  • Dynamically selecting which judge to trust reduces evaluation overhead while maintaining reliability, often outperforming the precision of relying fully on the strongest judge.Source: §5 · p. 9

A.1 PROOF OF THEOREM 1

Theorem 1 bounds selective disagreement risk with high probability; the cascade extension calibrates judges sequentially and guarantees the same risk target across the cascade.

A.1 PROOF OF THEOREM 1

  • With probability at least 1 − δ, a selective evaluator using the calibrated threshold satisfies the stated risk guarantee.The proof reduces this to showing that the true disagreement risk at the threshold is at most α with probability at least 1 − δ.Source: §A.1 · p. 14

A.2 EXTENSION TO CASCADES OF JUDGE MODELS

  • For cascades, each judge is calibrated on instances abstained by earlier judges, with error level δ divided by the number of judges.The calibration procedure updates the set using only instances for which the current judge's confidence is below its threshold.Source: §A.2 · pp. 14–15
  • The cascade's disagreement risk is at most α with probability at least 1 − δ.The proof bounds cascade risk by the maximum individual judge risk and applies a union bound across judges.Source: §A.2 · p. 15

B ADDITIONAL RESULTS ON CONFIDENCE ESTIMATION

Additional confidence-estimation experiments compare chain-of-thought variance methods with Simulated Annotators and find the latter performs better in the reported comparisons.

B ADDITIONAL RESULTS ON CONFIDENCE ESTIMATION

C ADDITIONAL RESULTS ON CHAT ASSISTANT EVALUATION

On Auto-J, Figure 5 contrasts GPT-4 without abstention with Cascaded Selective Evaluation's guarantee at higher target human-agreement levels.

C ADDITIONAL RESULTS ON CHAT ASSISTANT EVALUATION

  • 80% is the highest target human-agreement level that Cascaded Selective Evaluation guarantees with high probability on Auto-J.Source: Figure 5 · p. 17
  • 63.2% agreement with humans is achieved by GPT-4 without abstention on Auto-J.Source: Figure 5 · p. 17
  • Figure 5 compares GPT-4 without abstention against Cascaded Selective Evaluation's guaranteed human agreement on Auto-J.Source: Figure 5 · p. 17

D EVALUATION UNDER DISTRIBUTION SHIFT

The distribution-shift evaluation samples calibration and test instances from disjoint sets with no overlap in evaluated models. Across 1,000 random splits, the paper reports that Cascaded Selective Evaluation maintains high agreement.

D EVALUATION UNDER DISTRIBUTION SHIFT

  • Cascaded Selective Evaluation guarantees high agreement under the described distribution shift.The table caption reports this conclusion without giving numerical agreement values.Source: Table 9 · p. 17
  • Calibration and test sets are drawn from disjoint instance sets with no overlap in evaluated models.Source: Table 9 · p. 17
  • The results aggregate experiments over 1,000 random splits.Source: Table 9 · p. 17

E DETAILS ON EXPERIMENTAL SETUP

The experimental prompts cast evaluation as predicting an annotator’s pairwise preference from examples of that annotator’s decisions. They provide separate formats for assistant responses and document summaries, with A/B verdict labels.

E DETAILS ON EXPERIMENTAL SETUP

  • For summarization evaluation, the judge predicts which of two summaries an annotator prefers, given a document and examples of prior decisions.The prompt format includes a document, two summaries, and a verdict slot.Source: §E.1 · p. 18
  • For assistant-response evaluation, the judge predicts an annotator’s preferred answer from examples of that annotator’s decisions.The prompt presents a question, two assistant responses, and a verdict slot.Source: §E.1 · p. 19
  • Both prompt formats encode the predicted verdict as either [[A]] or [[B]].Source: §E.1 · pp. 18–19

E.2 HUMAN EVALUATION DETAILS

The human evaluation recruited qualified Prolific annotators to compare assistant responses on sampled ChatArena examples. Each example received multiple ratings, and the paper includes the annotation interface and guidelines.

E.2 HUMAN EVALUATION DETAILS

  • The appendix provides figures of the human annotation interface and annotation guidelines.Source: Figure 6 · p. 20, §E.2 · p. 20, Figure 7 · p. 21
  • Twenty-one Prolific annotators qualified for the study after meeting screening criteria and scoring above 80% on a qualification test.The recruited annotators had at least 99% approval, English fluency, and a bachelor’s degree; they were paid $15 per hour.Source: §E.2 · p. 20
  • Annotators rated 600 randomly sampled ChatArena examples, with each example evaluated by 3–5 annotators.They judged overall response quality across helpfulness, truthfulness, and harmlessness, and could occasionally skip an instance with a reason.Source: §E.2 · p. 20

F QUALITATIVE EXAMPLES

The qualitative examples show both aligned judge decisions and abstentions across assistant-response and summary comparisons. Some examples also illustrate the kinds of response pairs and summaries being judged.

F QUALITATIVE EXAMPLES

  • In the breakup-summary example, the evaluator abstains while human annotators prefer summary B.Source: F QUALITATIVE EXAMPLES · p. 22
  • For a secured-credit-card question, Mistral-7B selects summary A, matching the human preference, with inter-annotator agreement of 0.6.Source: F QUALITATIVE EXAMPLES · p. 22
  • For the Nicola Tesla question, the illustrated pair contrasts a brief answer with a much more detailed response, and Mistral-7B and humans prefer B.Source: F QUALITATIVE EXAMPLES · p. 23
  • In three illustrated cases, Mistral-7B, GPT-4, and GPT-3.5 select the human-preferred response, with inter-annotator agreement of 1.0 in each case.Source: F QUALITATIVE EXAMPLES · pp. 23–24
  • For the invented-language query, GPT-4 and humans both prefer answer B, which gives “blux jorr orto” for “the cat likes chicken.”Source: F QUALITATIVE EXAMPLES · p. 23
  • In two cases, the evaluator abstains although human annotators prefer an answer: A for the temperature-record query and B for the programming-joke query.Both examples report inter-annotator agreement of 0.75.Source: F QUALITATIVE EXAMPLES · pp. 24–25
Loading 2407.18370v1…