Source-linked AI summary
Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue
Ming Cheng, Yusheng Dai, Qiuhong Ke, Zhaolin Chen, Lizhen Qu
TL;DR
Pairwise LLM judges can retain model-specific biases and face strict risk requirements in open-ended dialogue evaluation. The paper combines CRC with multi-expert score or decision aggregation, then introduces MC3 for heterogeneous scoring scales. On PANEL, the framework improves accuracy and acceptance while preserving CRC validity, with MC3 achieving the highest acceptance rate across all three domains.
Problem
Single-expert LLM judging is empirically miscalibrated for pairwise comparisons, while persistent model-specific biases and unexplored multi-expert coverage effects limit reliable open-ended dialogue evaluation.
Method
The paper formulates judging as selective prediction under CRC and develops Score Averaging, Decision Voting, and MC3 with per-expert threshold ratios and a unified decision function.
Results
MC3 recovers the highest acceptance rate among CRC-valid methods across all three PANEL domains, at comparable accuracy while maintaining the formal risk guarantee.
Takeaways & Limitations
Multi-expert aggregation improves CRC-based LLM judging on homogeneous panels, while MC3 extends these gains to heterogeneous panels by accommodating distinct expert scoring scales.
Takeaways & Limitations
The framework requires logit access and its CRC guarantee relies on exchangeability, which can break under distribution shift.
Abstract
from arXiv · showhide
In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts' distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC3): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function $C_t(x)$ applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC3 extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.
1 Introduction
The paper frames pairwise LLM judging as selective prediction under CRC and introduces multi-expert aggregation to improve reliable coverage, especially when individual judges have persistent biases or incompatible scales.
- Motivation: High-stakes dialogue evaluation requires explicit risk control because automated judges can make consequential misjudgments.CRC addresses this by allowing judges to abstain when uncertain and defer cases to human review.
- Motivation: Single-expert CRC remains limited because model-specific preferences, prompt sensitivities, and domain competence gaps persist in open-ended dialogue.Prior work leaves how multi-expert panels affect CRC coverage at fixed risk largely unexplored.
- Framework: The paper introduces a formal multi-expert CRC framework that aggregates either scoring functions through Score Averaging or decisions through Decision Voting.The framework covers homogeneous panels formed from one model and heterogeneous panels formed from different models.
- Framework: MC3 estimates per-expert threshold ratios, jointly searches a global scaling parameter, and applies one unified decision function identically during calibration and testing.This design accounts for heterogeneous scoring scales while preserving exchangeability and CRC’s formal risk guarantee.
- Results: MC3 achieves the highest acceptance rate among CRC-compliant methods across all three datasets, with accuracy comparable to Decision Voting and the formal risk guarantee preserved.The result extends the gains of multi-expert aggregation to heterogeneous panels.
- Benchmark: PANEL contains 1,800 exhaustive pairwise comparisons among four open-weight LLMs across ESConv, MSC, and DREAM, with human preference labels and full logit access.The benchmark is designed to support white-box CRC evaluation for open-ended dialogue.
2 Problem Formulation
The paper formulates pairwise LLM judging as selective prediction: CRC calibrates a threshold for committing or abstaining, while multi-expert variants aggregate scores or votes before deployment.
- CRC formulation: Each evaluation instance pairs a dialogue context with two candidate responses, and calibration uses exchangeable instances with human preference labels.The system returns a prediction set over the two candidate preferences.
- CRC formulation: A singleton prediction commits to response A or B, whereas the full set {A, B} represents abstention and deferral to human evaluation.The scoring rule maps sufficiently positive or negative scores to singleton decisions and intermediate scores to abstention.
- CRC formulation: Larger λ raises the acceptance bar, producing more abstentions and lower coverage while reducing errors because the loss is non-increasing in λ.CRC therefore trades coverage for risk control through threshold selection.
- Threshold calibration: CRC selects the smallest qualifying threshold ˆλ, maximizing acceptance subject to the empirical risk constraint.With 0/1 miscoverage, abstentions incur zero loss and B = 1 is the tightest valid loss bound.
- Multi-expert strategies: Score Averaging combines expert scores, Decision Voting combines expert decisions, and MC3 proportionally scales individually calibrated thresholds before joint vote calibration.MC3 uses a unified decision function Ct(x) in both calibration and testing to preserve exchangeability.
- Guarantee: CRC’s deployment guarantee is finite-sample, distribution-free, and model-agnostic under exchangeability and loss monotonicity in λ.The calibrated predictor is applied to a new test instance drawn exchangeably with the calibration data.
3 Multi-Expert Conformal Consensus
Multi-expert CRC aggregates either scores or decisions to reduce residual judge bias, improving homogeneous-panel performance but leaving heterogeneous panels constrained by incompatible scoring scales.
- Logit-based pairwise preference probability is selected as the default per-expert conformity score because it offers fine-grained, monotone thresholding and the strongest alignment with human preferences.
- Score Averaging combines per-expert scoring functions, while Decision Voting combines per-expert decisions under a shared threshold.
- Homogeneous panels preserve a common scoring scale, enabling both strategies to maintain risk guarantees and substantially improve accuracy over a single expert.
- Heterogeneous panels remain risk-valid but recover limited coverage because one shared threshold cannot match distinct expert scoring scales.For example, a narrow-scale expert may always abstain while a wide-scale expert contributes the non-abstaining vote.
4 Marginal-Calibrated Conformal Consensus
MC3 initializes expert-specific threshold ratios, then jointly searches a single global scale through one shared decision function used in calibration and testing. This preserves exchangeability and CRC guarantees while retaining heterogeneous scoring scales.
- 4.1 Marginal Calibration Initialization: Independent calibration estimates each expert’s threshold ratio, providing a scale-aware initialization for heterogeneous judges.
- 4.1 Marginal Calibration Initialization: Using isolated per-expert thresholds as final test thresholds would mismatch calibration and collective voting, breaking the exchangeability required by CRC.
- 4.2 Joint Threshold Searching: MC3 proportionally scales all expert thresholds with one global scalar t and searches thresholds through a unified decision function C_t(x).
- 4.2 Joint Threshold Searching: As t increases, thresholds rise and abstentions increase, making the CRC loss non-increasing in t and satisfying its monotonicity requirement.
- 4.2 Joint Threshold Searching: Applying the same C_t to exchangeable calibration and test instances preserves exchangeability, so the CRC bound applies.
- 4.2 Joint Threshold Searching: MC3 combines risk guarantees, per-expert heterogeneous thresholds, and continuous one-dimensional parameterization over t.
5 Benchmark Construction
PANEL supplies a logit-accessible human preference benchmark for evaluating selective pairwise LLM judging in open-ended dialogue across three conversational domains.
- PANEL samples 100 contexts from ESConv, MSC, and DREAM, covering emotional support, multi-session social chat, and dialogue comprehension.
- PANEL was created because evaluating CRC requires pairwise preference data with candidate-model logits, which existing benchmarks lacked.
- The benchmark contains 1,800 annotated response pairs with human pairwise preference labels across five quality dimensions.
- Candidate responses and human labels were newly produced, so CRC calibration pairs remain uncontaminated by context memorization.
- The four LLMs differ in generation quality and single-judge accuracy, motivating a framework robust to individual-judge weakness.
6 Experiments
Experiments compare scoring functions and multi-expert CRC strategies across three datasets. Pairwise preference scoring and aggregation improve evaluation, while MC3 achieves the strongest risk-valid coverage for heterogeneous panels.
- Base score selection: Pref. Prob. outperforms the strongest pointwise score, Log Prob, in accuracy and AUC across all three datasets while keeping risk near α.The study therefore adopts Pref. Prob. as the per-expert base score for multi-expert CRC.
- Bias robustness: Self-judge bias changes pairwise-logit accuracy by less than 3 pp across datasets and directions, while swap correction yields only around 1 pp average gains.These surface-bias effects are smaller than the persistent gains attributed to multi-expert aggregation.
- Homogeneous panels: Decision Voting improves homogeneous-panel acceptance rate and accuracy over a single Pref. Prob. judge across all three datasets, with acceptance gains up to +15.8 pp.Score Averaging produces smaller and less consistent gains.
- Homogeneous panels: Risk remains at or below α throughout homogeneous-panel experiments, reaching at most .098, so the CRC guarantee holds without violation.Decision Voting’s advantage is attributed to applying vote-level CRC directly to the decision function.
- Heterogeneous panels: In heterogeneous panels, Score Averaging and Decision Voting remain risk-valid but recover limited coverage because one uniform threshold cannot match distinct expert scoring scales.Narrow-range experts are silenced, increasing abstention; Score Averaging additionally collapses incompatible scores into one composite.
- MC3: MC3 uses per-expert ratio initialization and joint proportional threshold scaling, achieving the highest acceptance rate among CRC-valid methods on all three datasets while keeping risk within α.Its accuracy closely tracks other voting methods because the methods share majority voting and differ mainly in threshold selection.
7 Related Work
Prior work applies conformal prediction across several LLM tasks, including pairwise LLM judging, but dialogue preference datasets remain limited in human cross-model comparisons and open-ended coverage.
- Conformal prediction: Conformal prediction has been applied to multiple-choice QA, open-ended language modeling, and API-based settings without logit access.Recent work also extends CRC to pairwise LLM-as-a-Judge evaluation.
- Preference datasets: Dialogue preference datasets often target within-model comparisons and lack human pairwise labels across different LLMs.This leaves open-ended conversational evaluation underserved relative to general instruction-following preference datasets.
8 Discussion
The framework is designed for high-stakes settings where open-weight judges provide logits and abstention can defer uncertain cases to humans. Its discussion makes logit access and free abstention explicit design choices.
- Design choices: The framework requires logit access, restricting applicability to open-weight LLMs; API-only judges require alternative conformity scores.The authors relate this choice to local deployment in privacy-sensitive clinical or enterprise pipelines.
- Design choices: The 0/1 loss treats abstention as free, while Acceptance Rate transparently tracks deferral cost; MC3’s AccR of 0.462 on MSC at α=0.1 implies about 54% deferral.The authors state that cost-sensitive extensions require changing the abstention cost in the loss.
9 Conclusion
The paper formulates pairwise LLM judging as selective prediction under CRC and systematically studies multi-expert aggregation. MC3 addresses heterogeneous expert scales while retaining CRC validity and achieves the strongest acceptance-rate result among valid methods.
- The study presents the first systematic multi-expert CRC framework for pairwise LLM-as-a-Judge evaluation in open-ended dialogue.
- Score Averaging and Decision Voting improve homogeneous panels but recover limited coverage on heterogeneous experts.
- MC3 uses per-expert threshold ratios and a unified decision function shared by calibration and test, preserving exchangeability.
- MC3 achieves the highest acceptance rate among CRC-valid methods across all three PANEL domains, with comparable accuracy and formal risk guarantees.
Limitations
The framework's main limitations concern downstream handling of abstentions and the exchangeability assumption underlying CRC guarantees. The authors propose rolling threshold-ratio monitoring and inexpensive recalibration as a mitigation for drift.
- The framework addresses selective prediction but leaves the downstream human-review workflow for abstained cases open.
- CRC guarantees rely on exchangeability, which can break under distribution shift.
- Monitoring per-expert threshold ratios on a rolling window can trigger recalibration when drift exceeds a preset threshold.The recalibration is cheap because CRC uses training-free threshold search.
- PANEL uses three in-house annotators and dialogue contexts from ESConv, MSC, and DREAM under their original licensing conditions.
- Figures 3 and 4 distinguish generator quality rankings from pairwise evaluation accuracy rankings.
B Experimental Details
The experimental scoring implementations compare pointwise and pairwise conformity scores, using model logits or additional models and sampling depending on the score.
- Pointwise scores include log probability, Self-Certainty, and DeepConf, with each requiring one forward pass per response.
- Causal scoring uses fine-tuned RoBERTa classifiers, while Consistency scoring uses five sampled responses clustered with DeBERTa-based NLI.
- Pref. Prob. extracts logits at the first generated-token position for pairwise evaluation.
C Proofs of CRC Validity for Multi-Expert Methods
The proofs establish CRC validity by verifying monotonicity, exchangeability, boundedness, right-continuity, and a suitable maximum threshold for each multi-expert method. MC3 remains valid because its jointly defined decision function is used consistently during calibration and testing.
- All three multi-expert methods satisfy the CRC theorem's conditions and inherit its finite-sample, distribution-free risk guarantee.
- Decision Voting: Decision Voting uses a shared threshold and majority vote; vote-count monotonicity yields no-flip behavior and CRC validity.
- Score Averaging: Score Averaging applies CRC to an averaged scoring function, with monotonicity from no-flip behavior and exchangeability from a deterministic mapping.
- MC3: MC3 scales per-expert thresholds proportionally with a global scalar t and applies the resulting joint decision function during calibration and testing.
- MC3: Under exchangeability, the calibrated MC3 threshold satisfies expected test loss at most α.
- MC3: Test-time Voting loses the CRC bound because its calibration-time independent decisions differ from its test-time collective vote.
- Generality: The proofs do not depend on the conformity-score choice, the number of experts, or the dataset.