Source-linked AI summary
Localize-Then-Decide Guarantees for LLM Judgments
Xinyu Li, Yi Zhou, Guanqun Cao, Zeyu Fu, Tianjin Huang, Gaojie Jin
TL;DR
The paper targets unreliable human–LLM agreement guarantees for multi-candidate judgments, where confidence monotonicity can fail as probability mass spreads across alternatives. It localizes a shortlist with conformal prediction and then selectively chooses or abstains using calibrated confidence, consistently improving guarantee validity and coverage over single-stage baselines.
Problem
Confidence-thresholding guarantees are established for pairwise judgments, but their confidence–disagreement monotonicity assumption can break down when judges select among multiple candidates.
Method
Localize-Then-Decide conformally constructs a high-probability shortlist, then applies a calibrated selective rule to choose one response or abstain.
Results
The framework consistently achieves valid guarantees while improving coverage compared with single-stage baselines across multiple datasets, judge models, and candidate sizes.
Takeaways & Limitations
Structuring LLM judgment as localized and selectively certified decision-making supports trustworthy automated evaluation beyond pairwise comparisons.
Takeaways & Limitations
The guarantees require exchangeable calibration and test instances, and deployment domain shift can make the finite-sample bounds less tight.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces confidence-thresholding methods that provide such guarantees for pairwise comparisons, relying on the assumption that higher estimated confidence implies lower disagreement risk with humans. However, this assumption can break down when the number of candidate responses increases, since distributing probability mass across many alternatives can distort confidence estimates. To address this issue, we propose a Localize-Then-Decide framework. First, conformal prediction localizes a small shortlist that contains the human-preferred response with high probability. Then, a calibrated confidence-based rule selectively chooses a single response from this shortlist or abstains. This design restores the monotonic relationship between confidence and disagreement risk and enables high-probability agreement guarantees. Experiments with multiple candidate sizes across several datasets and judge LLMs demonstrate that our framework consistently achieves higher guarantee success rates and substantially higher coverage than single-stage baselines.
1 Introduction
The paper addresses unreliable confidence-thresholding guarantees when LLM judges select among multiple candidates. Its Localize-Then-Decide framework uses localization followed by selective choice to restore confidence–agreement monotonicity and improve guarantee reliability.
- Prior guarantees address pairwise judgments, but practical multi-candidate selection requires extending reliability guarantees beyond m = 2.
- Top-3 agreement remains monotone at m = 20, while direct top-1 agreement loses monotonicity; selecting within the localized shortlist further restores monotonicity.
- As candidate count increases, probability dilution across alternatives can break the monotonic relationship between estimated confidence and human disagreement risk.
- The framework first uses conformal prediction to localize a shortlist containing the human-preferred response with high probability, then selectively chooses one response or abstains.
- Experiments across datasets, judge LLMs, and candidate counts m ∈{5, 10, 20} report improved monotonicity, higher guarantee success rates, and substantially higher coverage than single-stage baselines.
2 Preliminary
The problem setup models an LLM judge selecting a human-preferred response from m candidates. The framework seeks both a high-probability shortlist and a selectively certified single decision.
- Each input combines a query with m candidate responses, while the label y identifies the response preferred by humans.
- The judge uses scoring functions whose larger values indicate greater predicted preference for a candidate.
- Stage I outputs a small shortlist S(x) intended to contain the true human-preference label with high probability.
- Stage II optionally outputs one index from S(x) when reliability can be certified; otherwise, the system abstains or escalates.
- The formal setup assumes exchangeable calibration and test instances and a unique human-best response, with deterministic tie-breaking.
3 Two-Stage Certified Selection for Multi-Generations
The two-stage method first conformally localizes a shortlist and then calibrates selective auto-picking within it. Its guarantees combine localization coverage with conditional decision correctness under stated assumptions.
- 3.1 Stage I: Conformal Area Localization: Conformal area localization constructs S(x) from the top-k candidates under Sc(x, ·), targeting coverage of at least 1 −α for the human-best response.
- 3.2 Stage II: Selective Auto-Pick: Stage II selects a single index from S(x) with a high-probability correctness guarantee, otherwise abstaining or escalating.
- 3.2 Stage II: Selective Auto-Pick: The within-area error rate decomposes into localization misses under acceptance and selection errors when the human-best response lies in S(x).
- 3.2 Stage II: Selective Auto-Pick: Threshold calibration uses a binomial upper confidence bound and fixed-sequence testing over progressively smaller λ values.
- 3.2 Stage II: Selective Auto-Pick: The composed end-to-end guarantee additionally relies on monotonicity of top-k agreement as the within-shortlist confidence threshold changes.
- 3.2 Stage II: Selective Auto-Pick: The final selective accuracy guarantee factors into (1 −α) Stage I coverage and (1 −ε) conditional correctness on accepted instances.
4 Experiments
Experiments across diverse preference benchmarks and judge models test whether two-stage localization and selection restore monotonicity and improve selective guarantees. The framework maintains low ranking loss, exceeds the target guarantee success rate, improves coverage, and supports reliable cascades.
- Experimental Setup: Experiments span four preference benchmarks, judge LLMs from five families, and candidate counts m ∈ {5, 10, 20}.The evaluation includes summarization, dialogue, safety, and instruction-following settings, with models ranging from 7B to 120B parameters.
- Monotonicity: Single-stage ranking loss increases with candidate count, reaching 0.34 from 0.14 for Qwen2.5-72B on TL;DR.This confirms that confidence–agreement monotonicity degrades as m grows.
- Monotonicity: Both Stage I and Stage II maintain ranking loss ≤0.08 across all evaluated settings.The confidence–agreement curves likewise remain monotonic for both stages while single-stage curves degrade with m.
- Guarantees and Coverage: Every two-stage variant exceeds the 90% GSR target, whereas matched single-stage variants remain at 48–79%.At m=10 and target agreement 0.81, two-stage KL-margin coverage reaches 64–72% for GPT-OSS-120B versus 50–58% for its single-stage counterpart.
- Cascaded Architecture: Two-stage cascades achieve 91–95% GSR, while single-stage cascades reach only 64–76%.The Qwen same-family cascade covers 78% versus 64% for Qwen2.5-72B alone, and three-tier cascades cover 77–81% of instances.
- Cascaded Architecture: At a 70% target, the representative cascade covers 93.0% of instances, with tighter targets reducing coverage and routing more cases to stronger tiers.The weakest tier handles over half of instances at the 70% target.
5 Conclusion
The conclusion presents Localize-Then-Decide as a calibrated two-stage solution for reliable human–LLM agreement in multi-candidate judgments. Across experiments, it restores monotonicity, improves coverage while preserving guarantees, and supports reliable cascades.
- Framework: Localize-Then-Decide decomposes multi-candidate judgment into conformal shortlist localization followed by selective single-response choice or abstention.The first stage seeks a shortlist containing the human-preferred response with high probability; the second provides selective correctness guarantees.
- Guarantees: The decomposition restores confidence–agreement monotonicity and yields finite-sample guarantees for selective human–LLM agreement.This extends reliability guarantees beyond pairwise comparisons to multi-candidate settings.
- Empirical Findings: Experiments across datasets, judge models, and candidate sizes show valid guarantees with higher coverage than single-stage baselines.The conclusion reports this as a consistent empirical pattern across the evaluated settings.
- Cascading: The framework enables cascaded evaluation in which weaker models handle easier cases and difficult instances escalate to stronger models without violating reliability guarantees.This consequence follows from composing valid per-tier guarantees in the cascade.
Limitations
The reported guarantees depend on exchangeable calibration and test instances, use a relatively expensive confidence estimator, and are evaluated primarily for multi-candidate preference selection. Other evaluation paradigms may require task-specific adaptations.
- Statistical Assumption: Conformal guarantees require calibration and test instances to be exchangeable, and domain or prompt shift can make finite-sample bounds less tight.The authors identify relaxing this assumption as an open problem.
- Computational Cost: Simulated Annotators require five forward passes per instance with five few-shot examples per prompt, making confidence estimation more expensive than a single predictive probability.The framework itself can use lighter-weight confidence estimators if their scores remain sufficiently discriminative for calibration.
- Scope: Evaluation focuses on selecting one preferred response from multiple candidates, while Likert scoring and factuality verification may require task-specific calibration adaptations.The authors present broader applicability as a possible extension rather than an evaluated result.
Ethics Statement
The paper frames reliable LLM evaluation as requiring formal human–LLM agreement guarantees, extending prior pairwise work to multi-candidate judgments. Its two-stage framework addresses confidence monotonicity breakdown through conformal localization followed by selective auto-picking.
- Motivation: LLM evaluators offer scalable alternatives to costly human annotation, but trustworthy downstream use remains a fundamental challenge.The paper situates its contribution within automated benchmarks and preference-data collection at scale.
- Prior Work: Prior guarantee methods provide formal human-agreement bounds for pairwise judging but remain restricted to m=2.The paper contrasts these methods with broader empirical bias-mitigation approaches that lack formal bounds.
- Gap: When probability mass is diluted across many alternatives, the confidence–agreement monotonicity assumed by prior guarantees can break down.This structural challenge is distinct from methods that reduce errors empirically without formal guarantees.
- Contribution: The proposed decomposition combines conformal localization with selective auto-picking to restore monotonicity and enable guarantees for m>2.The framework can use different confidence estimators as plug-in components within either stage.
B Simulation Details
The simulation models why confidence–agreement monotonicity degrades as candidate count grows and evaluates how top-k localization and within-shortlist scoring recover it. It uses heterogeneous difficulty, candidate quality gaps, annotator strengths, and position debiasing.
- Generative Model: Competitive distractors increase from 0.2 on average at m=2 to approximately 3.8 at m=20, increasing high-confidence but incorrect predictions.Candidates are competitive with probability ρ=0.20, while annotator signal strengths are sampled from Beta(3,2).
- Panel (a): As m increases, probability dilution compresses the confidence range and progressively degrades the confidence–agreement relationship.More competitive alternatives split ensemble mean probability mass and shrink the margin between the best and second-best candidates.
- Simulation Setup: The simulation varies candidate count m∈{2,5,10,20} and evaluates agreement across confidence thresholds.Each threshold reports average agreement among instances meeting the threshold; monotonicity means agreement is non-decreasing with stricter thresholds.
- Panels (b)–(c): At m=20, top-3 localization recovers a clean monotonic agreement curve while direct top-1 agreement remains non-monotonic.Within the top-3 shortlist, margin-based confidence functions, especially KL margin, produce smoother and more monotonic reliability curves.
- Configuration: Simulations use N=25 annotators and 20,000 instances per setting, with agreement evaluated at evenly spaced confidence thresholds.The complete parameter settings are summarized in Table 4.
C Proofs
The framework’s proofs establish conformal shortlist coverage and selective within-shortlist correctness, then combine them under a monotonicity assumption. The experimental definitions distinguish localization scores, selection scores, confidence rules, and single-stage comparisons.
- Conformal Localization: Exchangeability yields a conformal shortlist containing the human-best response with probability at least 1−α.The proof uses the uniform rank of the test nonconformity score among exchangeable calibration and test scores.
- Selective Auto-Picking: Stage II calibration provides within-shortlist correctness of at least 1−ε with probability at least 1−δ over the calibration sample.The selective threshold is obtained by fixed-sequence testing over decreasing confidence thresholds.
- Composed Guarantee: Under monotonicity, shortlist coverage among accepted instances is at least 1−α, allowing the two bounds to compose.The proof derives P(y∈S(x)|A)=1−P(y∉S(x)|A)≥1−α and combines it with the Stage II bound.
- Evaluation Construction: The evaluation constructs multi-candidate instances from pairwise preference data by combining the human-preferred response with randomly sampled alternatives and shuffling their order.Approximately 3,000 instances are produced per dataset–candidate-count pair across four benchmarks.
- Scoring Pipeline: Stage I ranks all candidates with EMP, whereas Stage II and the single-stage baseline apply selection scoring within the shortlist or across all m candidates, respectively.Both selection procedures rank candidates with Sh and then use a confidence function to accept or abstain.
- Confidence Rules: Top-1 confidence uses the winner’s score, while margin confidence uses its gap over the runner-up.The margin rule corresponds to the shortlist score difference used in the selective guarantee.
G Monotonicity Evaluation Settings and Confidence Distribution Analysis
The evaluation distinguishes direct multi-candidate selection from Stage I localization and Stage II shortlist selection, then examines whether localization improves confidence discrimination and guarantee validity.
- Evaluation settings: The protocol evaluates direct m-to-1 selection, Stage I m-to-k localization, and Stage II k-to-1 selection within the localized shortlist.Stage I uses top-1 EMP confidence with a conformal shortlist; Stage II evaluates a KL-divergence margin within that shortlist.
- Monotonicity evaluation: Figure 3 uses Qwen2.5-72B on TL;DR to compare confidence–agreement relationships across candidate counts and pipeline stages.The direct setting varies m ∈{2, 5, 10, 20}; localization comparisons fix m=20 and compare k=1 with k=3.
- Confidence distributions: After Stage I localization, correct and incorrect KL-margin distributions have a median gap of 0.92, up from 0.06 in the single-stage setting.The single-stage margin is computed over all m candidates, whereas the two-stage margin is computed within shortlist S(x).
- Guarantee validation: For every tested configuration, conditional shortlist miss probability decreases monotonically with the Stage II threshold, falling below 1% at λ=0.8.At λ=0, the conditional miss probability remains below the marginal failure level α=0.10.
- Baseline comparison: Variance normalization alone does not close the gap: two-stage selection improves both GSR and coverage over strengthened single-stage alternatives.The comparison uses the same representative setting for the strengthened baseline and two-stage method.
H.3 Component-Wise Ablation of KL Margin
The ablation evaluates the KL-margin components and finds that each contributes to both guarantee success and coverage, with shortlist and variance normalization most influential.
- Component-wise ablation: Removing any one of the five KL-margin components reduces both GSR and coverage in the representative two-stage setting.The ablation removes one component at a time while retaining the other four.
I Full Per-Model and Per-Dataset Results
Across additional judges and datasets, two-stage methods preserve monotonicity and exceed the target guarantee success rate while maintaining higher coverage than single-stage baselines.
- Per-model monotonicity: Single-stage ranking loss grows substantially with m, while Stage I and Stage II maintain near-zero ranking loss across additional judges.This extends the monotonicity analysis to Mistral-7B, Qwen2.5-32B, and Llama-3-70B.
- Candidate-count effects: Two-stage GSR remains above 1−δ = 90% as m grows from 5 to 20, whereas single-stage GSR degrades sharply.Across the 70–95% target range, two-stage empirical agreement tracks above the diagonal.
- Cascaded evaluation: Three-tier two-stage cascades cover 73–81% of instances and achieve 92–94% GSR across the evaluated datasets.All two-stage cascades exceed the 1−δ = 90% threshold, while single-stage cascades fall short.