Calibrating LLM Judges for Human and AI Conversations

1. INTRODUCTION

• The authors evaluate state-of-the-art LLMs as judges, finding that direct pointwise scoring achieves moderate correlation with human ground truth.Source: §1 · p. 1
• Different LLM judges use incomparable score scales, while pairwise judging faces long transcripts, context-window limits, and positional bias.These issues make cross-paper comparisons unreliable.Source: §1 · p. 1
• They propose a CANDOR anchor set and transformation function to map judges’ raw scores onto a shared, interpretable scale.Source: §1 · p. 1
• The Voice Arena Goal Dataset contains 200 human-agent or human-model conversations with human-annotated rankings, extending evaluation to human-AI settings.The paper also reports that current LLM judges struggle to discriminate human-AI conversational success.Source: §1 · p. 1

2. BACKGROUND AND RELATED WORK

• Dialogue quality accumulates across turns, motivating reference-free dialogue-level metrics and human protocols that compare complete dialogues.No evaluation protocol is uniformly resilient.Source: §2 · p. 2
• The paper compares pointwise and pairwise LLM judges using participant self-reported conversational success, rather than chatbot quality, as the reference.Source: §2 · p. 2
• LLM judges can align with human preferences on open-ended tasks, but validity varies across tasks and properties, and judges exhibit several biases.Reported biases include position, verbosity, and self-preference.Source: §2 · p. 2

3. JUDGING CONVERSATIONAL SUCCESS

• The zero-shot study compares absolute 0–10 pointwise scores with pairwise selections, using the same models and prompting strategies.Pairwise audio judgments rank sliding-window snippets; the length baseline always selects the longer transcript.Source: §3 · p. 2

3.1. Experimental Setup

• CANDOR contains 1,656 conversations; after excluding 20 without computable scores, 1,636 remain, with survey responses yielding continuous PCS scores and three success categories.Source: §3.1 · p. 2
• Pairwise evaluation includes all 3,185 HSC×LSC pairs in both orderings and a 1,000-pair general set with PCS differences above 0.1.Pointwise scores are evaluated against PCS with Spearman’s ρ and Krippendorff’s α; pairwise accuracy is also derived for comparison.Source: §3.1 · p. 2

3.2. Results of the Judges

• Pointwise-derived accuracy reaches 93.6% and 90.3% on HSC×LSC pairs, while general-test models score below 70%, near the 61.3% length baseline.Phi-4 and Qwen2.5 choose the first-presented conversation around 90% of the time; no consistent modality advantage is found.Source: §3.2 · pp. 2–3, Table 1 · p. 3

4. CALIBRATING FUTURE JUDGES

• On CANDOR, calibration makes LLM judge score distributions more aligned with human annotators’ distributions.The uncalibrated judges produce different distributions, including Qwen2.5’s consistent underestimation of the true score.Source: §4 · p. 3, Figure 1 · p. 3
• The calibration objective minimizes the Wasserstein distance between human scores and transformed judge scores over a natural conversation distribution, using an anchor set with both human ratings and judge predictions.Source: §4 · p. 3
• Wasserstein Affine and µ/σ-matching perform best; Wasserstein Affine leads at sizes 32 and 128, while µ/σ-matching leads at size 8.Optimized anchors improve over random selection most at small sizes, with the methods nearly converging by size 128.Source: Table 2 · p. 3, §4 · pp. 3–4
• The method restricts calibration to affine transformations and fits parameters by MSE or Wasserstein distance; a positive slope preserves ranking correlations.The considered matching approaches align either the judge’s range or its mean and standard deviation with the human anchor scores.Source: §4 · p. 3
• Anchor selection compares random and optimized sets of 8, 32, or 128 conversations, evaluating on conversations held out from the anchor.The optimized set is selected by random search over 1,000 candidate subsets and excludes the judge later evaluated.Source: §4 · p. 3
• The authors select a 32-item optimized anchor as a middle ground between small-anchor noise and the labeling cost of large anchors.For a new judge, they recommend closed-form µ/σ-matching using this fixed anchor.Source: §4 · p. 4

5.1. Introducing the VOICE ARENA GOAL DATASET

• VA contains 200 task-oriented voice conversations across five conditions, with human pairwise ratings of task capability and humanness.The conditions are four conversational AI systems and a human-agent baseline.Source: §5.1 · p. 4

5.2. Discrimination Accuracy

• On VA, no LLM judge closes the human-agreement gap: most score at or below the duration baseline, against human agreement of 84.2% (task) and 86.7% (humanness).The duration baseline reaches 65.0% for task capability and 60.4% for humanness.Source: Table 3 · p. 4, §5.2 · p. 4

5.3. Calibration-Transfer Test

• After CANDOR-only fitting, calibration reduces average pairwise Wasserstein distances between VA judge-score distributions by 3–10× under both methods.The calibration was fitted without observing VA; raw distances were 1.44–3.08.Source: Table 4 · p. 4, §5.3 · p. 4

• Judging conversations, particularly human-AI dialogue, remains difficult, and LLM judges do not share a calibrated score scale.Source: §6 · p. 4
• VOICE ARENA GOAL DATASET provides a benchmark for evaluating conversational judges in human-AI settings.Source: §6 · p. 4
• The proposed calibration method brings judges from different models onto a shared, comparable scale.Source: §6 · p. 4