SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Sher Badshah; Ali Emami; Hassan Sajjad

Source-linked AI summary

SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Sher Badshah, Ali Emami, Hassan Sajjad

arXiv:2602.13110v4Feb 13, 2026cs.CLcs.AI

TL;DR

LLM judges offer scalable pairwise evaluation but can be miscalibrated and biased, motivating reliable selective judgment. The paper introduces BPE for position-neutral uncertainty and SCOPE for conformal risk control, reporting improved uncertainty quality and target-risk control with substantial coverage.

Problem
LLM pairwise judges can introduce systematic biases and lack a principled way to determine when a judgment should be trusted with an explicit accepted-set error bound.
Method
SCOPE conformally calibrates an acceptance threshold, while BPE averages aligned preference probabilities from both response orders and converts them into entropy-based uncertainty.
Results
Across benchmarks and model scales, BPE improves calibration and discrimination, while SCOPE accepts more judgments while meeting the target risk level.
Takeaways & Limitations
Combining bias-neutral uncertainty estimation with conformal risk control provides a promising foundation for trustworthy automated evaluation at scale.
Takeaways & Limitations
The guarantees rely on exchangeability between calibration and future data, while BPE requires two forward passes and access to model probabilities or logits.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose \textsc{Scope} (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $α$. To supply \textsc{Scope} with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while \textsc{Scope} consistently satisfies the target risk bound (empirical FDR $\approx 0.097$--$0.099$ at $α=0.10$) and retains substantial coverage. Compared to vanilla baselines, \textsc{Scope} accepts up to $2.4\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.

1. Introduction

LLM judges enable scalable pairwise evaluation but can propagate position, stylistic, self-preference, and familiarity biases. SCOPE combines conformal risk control with BPE to bound accepted-set errors while improving uncertainty quality and coverage.

Motivation: LLM judging can introduce systematic errors that distort rankings, training signals, and downstream distilled preferences.The cited failure modes include verbosity bias, position-dependent ranking reversals, self-preference, and familiarity bias.
Motivation: Selective prediction addresses this problem by abstaining on uncertain cases, but ordinary confidence thresholds lack finite-sample guarantees and can be contaminated by nuisance bias.These limitations motivate combining conformal calibration with a bias-neutral uncertainty measure.
SCOPE: SCOPE calibrates an acceptance threshold so the error rate among accepted judgments is at most α under exchangeability.This provides a finite-sample validity condition rather than relying on heuristic thresholding or naive empirical tuning.
BPE: BPE scores both response orders, aligns their preference probabilities, and applies entropy to produce a position-invariant uncertainty signal.The method is designed to reflect genuine preference ambiguity rather than presentation bias.
Results: Across benchmarks and model scales, BPE improves calibration and discrimination, while SCOPE accepts more judgments while meeting the target risk level.The introduction states this as the paper’s empirical contribution without specifying a single benchmark value.

2. Methodology

The methodology defines selective pairwise judging through an uncertainty threshold, constructs BPE by averaging aligned predictions across response orders, and calibrates the largest threshold satisfying a finite-sample FDR condition. Validity depends on exchangeability, while coverage is maximized among feasible thresholds.

2.1. Problem Formulation: SCOPE accepts a deterministic judgment when s(x) ≤ λ̂ and abstains otherwise, with BPE instantiating the uncertainty score and prediction rule.The threshold separates accepted judgments from abstentions.
2.2. Bidirectional Preference Entropy: BPE averages aligned preference probabilities from forward and reversed response orders, then uses binary entropy as the uncertainty score.Order disagreement shifts the aggregate toward 0.5 and therefore increases uncertainty, while strong preference lowers it.
2.3. SCOPE Calibration: Finite-sample calibration uses a linearized loss whose calibration sum must satisfy a sufficient condition, including a −1 correction for one unseen test point.This correction ensures the test-point contribution cannot invalidate the bound under exchangeability.
2.3. SCOPE Calibration: The method selects the largest feasible threshold to maximize coverage, while abstaining on all instances if no feasible threshold exists.Feasibility need not be monotone in the threshold because newly admitted samples can be correct or incorrect.
2.3. SCOPE Calibration: BPE preserves exchangeability because it deterministically maps each labeled pair to its score and prediction using two greedy forward passes at T = 0.Thus exchangeability of the original labeled pairs transfers to the induced quantities used for calibration.
2.3. SCOPE Calibration: SCOPE calibrates λ̂ so the marginal test-time FDR is at most α, where FDR is the expected fraction of incorrect judgments among accepted predictions.The guarantee is stated for exchangeable calibration and test samples and holds across the query distribution under that assumption.

3. Experiments

The experiments test whether BPE improves uncertainty estimation and whether SCOPE controls risk while maximizing coverage across diverse benchmarks, judge models, and risk levels. Evaluation combines calibration and discrimination metrics with selective FDR and coverage measurements.

Datasets: Experiments span MT-Bench, RewardBench, Chatbot Arena, JudgeBench, and PKU-SafeRLHF, covering instruction following, reward-model-style preferences, crowdsourced comparisons, and safety evaluation.The primary benchmarks use 2,000 randomly sampled non-tied instances after filtering.
Baselines: The study compares BPE with four uncertainty estimators and SCOPE with three selective prediction strategies, including vanilla, heuristic, and naive calibration baselines.Naive calibration uses held-out empirical risk without a finite-sample correction and can violate the target constraint.
Evaluation: Uncertainty quality is assessed using Accuracy, Expected Calibration Error, AUROC, and AUPRC.ECE, AUROC, and AUPRC are evaluated after converting BPE into a confidence score when required.
Evaluation: SCOPE is evaluated for empirical FDR below the target α and for coverage, defined as the percentage of queries receiving predictions rather than abstentions.These metrics directly assess statistical validity and evaluation efficiency.
Protocol: The protocol uses 50/50 calibration-test splits, averages results over 1,000 random splits, and evaluates α values from 0.05 through 0.25.Judge models range from Qwen-2.5-7B-Instruct to Llama-3.1-70B-Instruct.

4. Results

BPE improves uncertainty estimation for pairwise LLM judging, while SCOPE controls accepted-set risk and preserves useful coverage across benchmarks and judge scales.

4.1. Uncertainty Estimation Quality: BPE achieves superior calibration and discrimination in most settings, strictly improving over Swap-and-Aggregate on ECE, AUROC, and AUPRC.
4.1. Uncertainty Estimation Quality: BPE outperforms Simulated Annotators in calibration and discrimination while requiring only two forward passes per instance.
4.2. SCOPE: Statistical Validity and Coverage: Vanilla prediction and heuristic thresholding violate the target risk constraint, while naïve calibration can exceed α under finite samples.
4.2. SCOPE: Statistical Validity and Coverage: SCOPE achieves empirical risks typically around 0.097–0.099 at α = 0.10 while delivering substantially higher coverage than naïve calibration.
4.2. SCOPE: Statistical Validity and Coverage: SCOPE coverage increases with the risk budget and model strength, reaching 0.991 for Qwen-7B and 1.000 for Llama-70B at α = 0.25 on MT-Bench.
4.2. SCOPE: Statistical Validity and Coverage: Risk variability across calibration/test splits is higher for weaker judges and at larger α, although SCOPE continues to control risk.

5. Related Work

Prior work establishes LLM judges as scalable but systematically unreliable, while uncertainty estimation and conformal methods address calibration and risk-control gaps that SCOPE combines for pairwise evaluation.

LLM-as-a-judge: LLM judges scale evaluation but exhibit position, self-preference, familiarity, and other systematic biases that can distort rankings and downstream preference signals.
Uncertainty estimation for LLM judges: Existing uncertainty approaches include token probabilities, verbalized confidence, simulated-annotator agreement, and repeated or swapped evaluations.
Conformal prediction and risk control: Conformal prediction supplies distribution-free calibration, but SCOPE adapts conformal risk control to guarantee error control specifically among accepted pairwise judgments.
Conformal prediction and risk control: SCOPE complements risk control with a bidirectional uncertainty estimator that aggregates both response positions to address known presentation bias.

6. Limitations

SCOPE’s guarantees depend on exchangeability, while BPE adds computational and access requirements and the framework remains focused on pairwise judging.

SCOPE’s selective guarantees may weaken under distribution shifts across benchmarks, prompt variations, or strategic model behaviors.
BPE requires approximately two forward passes per comparison and access to model probabilities or logits, limiting use with fully black-box evaluators.
The formulation addresses pairwise judging, leaving extensions to point-wise evaluation as future work.

7. Conclusion

The paper presents SCOPE and BPE as a statistically grounded layer for reliable pairwise LLM evaluation. Across benchmarks and model scales, the combination improves uncertainty quality, controls risk, and retains coverage without retraining judges.

SCOPE combines BPE with conformal risk control to provide user-tunable selective evaluation with certified error control and improved coverage.
Across benchmarks and model scales, BPE improves calibration and discrimination while SCOPE accepts more judgments at the desired risk level.
The framework operates as a drop-in layer over existing judges without retraining, with α trading coverage against certified error rate.

Impact Statement

SCOPE introduces statistically grounded reliability control for LLM-based pairwise evaluation, while BPE addresses position bias and supports calibrated judgments.

SCOPE provides finite-sample false-discovery-rate control for single-model pairwise LLM judging and can calibrate existing judges without retraining.The framework uses preference-token probabilities but does not require access to model internals beyond them.
BPE actively mitigates position bias, promoting fairer comparisons between models.
SCOPE aims to make scalable LLM evaluators more trustworthy by replacing heuristic confidence scores with formal, user-tunable risk control.

A.1. Proof of Theorem 2.1

The proof uses exchangeability and a calibrated threshold to establish the accepted-set error bound, while BPE supplies entropy-based uncertainty and confidence scores for downstream use.

Proof of Theorem 2.1: The accepted judgments’ conditional error is bounded by α by combining the linearized-loss condition with the selection and error indicators.When the acceptance probability is positive, rearranging the expectation inequality yields the FDR bound; zero acceptance satisfies the constraint trivially.
Proof of Theorem 2.1: Exchangeability lets the test-point loss equal the expected average loss across calibration and test points under the same calibrated threshold.The threshold is a symmetric function of the calibration set, enabling the split-conformal swap argument.
BPE scoring: The framework uses BPE both as an uncertainty score for SCOPE calibration and as a confidence score for evaluation metrics.
BPE scoring: BPE’s uncertainty score is binary entropy of the bias-neutralized preference probability, with higher values indicating greater risk and triggering abstention.Entropy is approximately 0.693 at probability 0.5 and approximately 0.20 when probability is 0.95.
BPE scoring: For benchmarking, BPE converts entropy into probability-scale confidence ranging from 0.5 for maximum entropy to 1.0 for low entropy.The transformation preserves sample rank ordering for AUROC and AUPRC comparisons.

B.2. Evaluation Metrics

The evaluation measures uncertainty quality with calibration and discrimination metrics, then compares BPE and SCOPE across diverse judging settings. BPE generally improves uncertainty quality, while SCOPE maintains risk control and coverage under challenging conditions.

Uncertainty Estimation Quality: BPE achieves the lowest ECE and highest AUROC/AUPRC in nearly every JudgeBench and PKU-SafeRLHF comparison.On PKU-SafeRLHF, BPE improves AUPRC by up to 4.1 points over the strongest baseline.
Risk Control and Coverage: SCOPE consistently satisfies the risk bound on additional benchmarks, with stronger judges and clearer contrasts yielding higher coverage.PKU-SafeRLHF admits higher coverage than JudgeBench at the same model scale and has tighter split-to-split variance.
Baseline Comparisons: BPE’s continuous entropy is better calibrated and discriminative than simulated-annotator vote agreement, while requiring two deterministic passes instead of five sampled generations.Simulated-annotator AUROC is at or below chance in five of six cells, whereas BPE reaches AUROC 0.68–0.81.
Baseline Comparisons: SCOPE achieves 1.1–3.2× higher coverage than ToE under the same risk constraint by using BPE’s continuous signal and a linear-expectation constraint.ToE’s three discrete confidence levels cap coverage at 17–23%.
Specialized Critic: On specialized critics, BPE has the lowest ECE and highest AUPRC across all three benchmarks, while verbalized confidence reaches AUROC = 0.500 throughout.The result indicates that uncertainty-signal quality depends on the judge model.
Robustness to Label Noise: Under calibration-set label noise, SCOPE preserves empirical risk below α = 0.10 by tightening its threshold and reducing coverage as noise increases.At ϵ = 20%, many configurations approach zero coverage, indicating conservative abstention when calibration data is unreliable.

G. Calibration Set Size Ablation

SCOPE preserves risk control across calibration-set sizes while coverage improves with more labeled calibration examples and largely converges by n = 200–500.

Calibration Set Size: Even with n = 50 labeled calibration instances, SCOPE keeps empirical risk below α = 0.10 across all tested settings.The calibration-set-size experiment uses four judges, three benchmarks, and 500 random splits per cell.
Calibration Set Size: Coverage increases with calibration-set size and is largely converged by n = 200–500, with n = 200 within 5–10% of n = 1,000 across cells.The marginal coverage gain beyond n = 500 is small.

H. Cross-Benchmark Calibration Transfer

SCOPE’s calibration threshold transfers across similar benchmark distributions but can violate the target risk under substantial distribution shift. The results support recalibration on the target distribution whenever possible.

Cross-Benchmark Transfer: Within-benchmark calibration satisfies α = 0.10, and transfer between similar-difficulty benchmarks such as MT-Bench and Chatbot Arena also holds.Table 12 reports coverage and risk over 500 random splits, with rows as calibration benchmarks and columns as test benchmarks.
Cross-Benchmark Transfer: Calibrating on easier RewardBench and testing on a harder benchmark violates the bound, with risk reaching 0.20, whereas the reverse direction is over-conservative at 0.035–0.067.The authors identify this as expected behavior under distribution shift and recommend recalibration on the target distribution.