PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging with Confidence: Calibrating Autoraters to Preference Distributions

Zhuohang Li, Xiaowei Li, Chengyu Huang, Guowang Li, Katayoon Goshvadi, Bo Dai, Dale Schuurmans, Paul Zhou, Hamid Palangi, Yiwen Song, Palash Goyal, Murat Kantarcioglu, Bradley A. Malin, Yuan Xue

arXiv:2510.00263v1cs.CL

TL;DR

Autoraters trained on discrete labels reduce human preference distributions to single verdicts, despite systematic disagreement and ambiguity. The paper calibrates verbalized probabilistic judgments with SFT for dense labels or RL for sparse binary labels. Distribution-matching finetuning improves alignment and calibration while substantially reducing positional bias, and is reported to preserve objective-task performance.

  • Problem

    Autoraters trained on discrete labels collapse subjective and sometimes systematic differences in human judgment, although the target preference is a distribution rather than one ground-truth label.

    Source: §1 · p. 1

  • Method

    The framework calibrates verbalized probability predictions to a target preference distribution using SFT for dense probabilistic labels and RL for sparse binary labels.

    Source: §1 · pp. 1–2

  • Results

    Finetuning improves alignment with target preference distributions, calibration, and positional-bias consistency while preserving performance on objective tasks.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    Modeling the spectrum of human opinion rather than a single verdict is presented as a contribution to more reliable, fair, and robust AI alignment systems.

    Source: §7 · p. 11

    The consistency guarantee assumes the policy class can realize deterministic numeric probabilities with fully parsable outputs.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

1. Introduction

Autoraters trained on single preference labels collapse human disagreement, even though preferences can vary systematically across annotators. The paper proposes distribution-matching finetuning for probabilistic judgments and reports improved alignment, calibration, and resistance to positional bias.

1. Introduction

  • Discrete-label training collapses subjective, sometimes systematic differences in human judgments into a single verdict, discarding uncertainty and minority viewpoints.The authors argue reliable autoraters should model the target population’s full preference distribution.Source: §1 · p. 1
  • The framework calibrates verbal probability predictions to a target preference distribution, using SFT for dense probabilistic labels and RL for sparse binary labels.The reported methods improve calibration and reduce positional bias while preserving performance on objective tasks.Source: §1 · pp. 1–2
  • Figure 1 contrasts single-preference and full-distribution judgments and compares finetuned autoraters with zero-shot probabilistic conversions using MSE, ECE, and agreement.The comparison uses Gemma-2-9B on the JudgeLM validation set.Source: Figure 1 · p. 2

2. A Probabilistic Framework for Calibrating Autoraters

The framework treats pairwise human preference as a population distribution and makes the conditional probability—not a single verdict—the autorater’s primary output. This probabilistic output can expose ambiguity and annotator disagreement while supporting calibration and distributional alignment.

2.1. Problem Formulation

  • For a query and response pair, the target is the population probability that a randomly chosen annotator prefers B to A.The framework represents this as p*(x), rather than a single consensus label.Source: §2.1 · pp. 2–3
  • A discrete autorater’s decision or scalar margin collapses the rater distribution, whereas a probabilistic autorater predicts the conditional preference probability and can be thresholded if a decision is needed.The probability is optimized to match population preference without relying on the Bradley-Terry assumption.Source: §2.1 · p. 3

2.2. Benefits of Probabilistic Autoraters

  • Probabilistic reporting exposes task ambiguity and minority viewpoints for cost-sensitive decisions, and matching the target distribution implies calibration under the stated conditions.Under a strictly proper scoring rule, a probabilistic prediction matching the target is better aligned than a degenerate report when preferences are non-unanimous.Source: §2.2 · p. 3

3. Finetuning Autoraters to Match the Preference Distribution

The paper presents two ways to train verbalized preference probabilities, matched to whether supervision is dense and probabilistic or sparse and binary. Its consistency analysis shows when these methods recover the target preference distribution and what the optimal RL outputs look like.

3. Finetuning Autoraters to Match the Preference Distribution

  • The framework trains verbalized probabilities, retaining natural-language reasoning while using a single decoding pass rather than sampling-based prediction.The two training paradigms are tailored to different data conditions.Source: §3 · pp. 3–4
  • With multiple annotations per prompt, SFT targets their mean preference probability through a text-to-text sequence that can include reasoning and a structured numeric output.Training maximizes the likelihood of the target sequence and encourages the predicted probability to match the multi-annotator estimate.Source: §3 · p. 4
  • With single binary labels, RL parses a numeric probability and optimizes piecewise proper rewards; the clipped Log reward penalizes overconfident incorrect predictions more heavily than Brier.Unparsable outputs receive a defined reward, and both rewards can be optimized with policy-gradient methods.Source: §3 · p. 4

3.1. Consistency Analysis

  • The multi-annotator mean is an unbiased estimate of the target preference distribution, with variance decreasing as 1/m.This supplies a high-fidelity learning target as the number of annotations increases.Source: §3.1 · p. 5
  • Under the stated realizability assumption, population-optimal Brier and clipped Log policies report the truthful probability, and stochastic or unparsable outputs strictly reduce expected reward.For the clipped Log reward, recovery is subject to clipping; the assumed policy class can produce deterministic valid probabilities and parseable outputs.Source: §3.1 · p. 5

4. Experimental Setup

The experiments compare SFT and RL under a matched annotation budget using JudgeLM-derived calibration data. They use teacher-generated preference labels and rationales, then finetune Gemma-2-9B and Qwen-2.5-7B with specified output and reward procedures.

4. Experimental Setup

  • The calibration data come from JudgeLM prompts, with 5K prompts receiving 10 annotations for SFT and 50K prompts receiving one annotation for RL under the same total annotation budget.Evaluation uses 1K validation prompts, each with 10 annotations; training pairs are also swap-augmented.Source: §4 · p. 5
  • Gemini-2.5-Flash supplies pairwise labels using temperature 1.0 and randomly sampled personas, yielding approximately 110K annotations across calibration and evaluation.For SFT with chain-of-thought, the teacher also provides reasoning traces conditioned on the estimated preference probability.Source: §4 · p. 5
  • Autoraters emit an optional reasoning trace and a tagged probability; SFT learns target sequences, while RL scores parsed probabilities with Brier or clipped Log rewards.Unparsable outputs receive a default reward, which empirically drives parsability toward 1.Source: §4 · pp. 5–6
  • The experiments finetune Gemma-2-9B and Qwen-2.5-7B with full-parameter updates, using GRPO for RL and epsilon 10^-3 for the clipped Log reward.Further hyperparameters and training details are reported in Appendix C.Source: §4 · p. 6

5. Experimental Results

Distribution-matching finetuning improves preference alignment and calibration, reduces positional bias, and retains performance on objective tasks. The experiments also compare annotation budgets and evaluate on human-annotated and objectively labeled benchmarks.

5.1. Main Evaluation

  • On Gemma-2-9B, RL-Brier achieves MSE 0.0764, agreement 0.8706, F1 0.5895, and Brier score 0.0946; on Qwen-2.5-7B, agreement is 0.8575, 12.4% above Contextual Calibration, with best ECE and Brier score.Across both models, distribution-matching methods outperform zero-shot and calibration baselines across the reported metrics.Source: §5.1 · p. 7, Table 1 · p. 7
  • With a fixed annotation budget, RL using 50K prompts with one binary label each consistently outperforms SFT using 5K prompts with 10 aggregated labels each.The authors attribute the difference to data diversity; within RL, Brier rewards outperform Log rewards, which may destabilize training through heavy penalties for tail miscalibrations.Source: §5.1 · p. 8

5.2. Evaluation of Positional Bias

  • RL-Brier on Qwen-2.5-7B reaches consistency 0.9007 and absolute symmetry deviation 0.0964, while baseline methods retain positional bias.The figure shows baseline deviation distributions skewed toward response B and finetuned distributions centered near zero.Source: Table 2 · p. 8, Figure 2 · p. 8, §5.2 · p. 9

5.3. Out-of-Distribution Evaluation on Human-Annotated Data

  • The out-of-distribution PandaLM evaluation uses 1K test examples independently annotated by three human experts, with majority vote treated as ground truth.Agreement, precision, recall, and F1 are reported, but the supplied passages do not state the evaluation outcomes.Source: §5.3 · p. 9, Table 3 · p. 9

5.4. Out-of-Distribution Evaluation on Objective Tasks

  • On JudgeBench’s objectively labeled tasks, RL-Brier Gemma-2-9B reaches 55.10% reasoning accuracy and 46.57% overall accuracy, exceeding Gemini-1.5-pro overall and all finetuned judges except Skywork-Critic-8B.Qwen-2.5-7B reaches 44.86% overall accuracy; Skywork-Critic-8B was trained on a substantially larger dataset.Source: §5.4 · p. 10, Table 4 · p. 10

6. Related Work

Prior work examines ambiguity in human annotations, calibration methods for LLMs, and LLM judges; this paper focuses on calibrating autoraters to the distribution of human preferences.

Uncertainty of Human Annotations

  • Prior work treats human-label uncertainty as important and proposes stratifying evaluation by that uncertainty; this paper also trains models to capture it.Elangovan et al. (2025) propose uncertainty-stratified metrics for evaluating LLM judges.Source: §6 · p. 10

Calibration of LLMs

  • Earlier LLM calibration work uses post-hoc methods, prompting, supervised fine-tuning, or reinforcement learning, while this paper targets preference-distribution calibration for autoraters.The cited approaches address confidence calibration in question answering, output quality, or reward models.Source: §6 · p. 11

LLM-as-a-Judge

  • LLM-as-a-Judge research spans benchmarks, bias analyses, specialized judges, uncertainty, and non-transitivity; this work addresses calibration to task subjectivity.Source: §6 · p. 11

7. Conclusion

The paper concludes that distribution-matching tuning better aligns autoraters with preference distributions, improves calibration, and reduces positional bias. Related work connects this approach to accounts of legitimate human disagreement and preference heterogeneity.

7. Conclusion

  • Distribution-matching tuning improves alignment with target preference distributions, calibration, and positional bias, supporting more reliable, fair, and robust alignment systems.The conclusion describes a shift from predicting one verdict to modeling the spectrum of human opinion.Source: §7 · p. 11

A. Extended Related Work

  • Human disagreement can reflect distinct problem definitions, cue weights, biases, or mental models, producing coherent choices under different value trade-offs or judgment thresholds.Source: A. Extended Related Work · p. 16
  • Unlike methods that infer one gold-standard label by modeling annotator reliability, this work treats reproducible disagreement on subjective tasks as a target distribution.The passage contrasts noise-based aggregation with disagreement as a valid signal of differing interpretations.Source: A. Extended Related Work · p. 16
  • Preference heterogeneity in AI alignment may reflect legitimate value differences rather than noise, and probabilistic autoraters can be used in preference-based reinforcement learning.Source: A. Extended Related Work · p. 16

B. Proofs

The appendix defines the population preference as an average over annotators and establishes that sampled annotator judgments provide an unbiased estimate whose variance decreases with sample size.

B.1. Preference Distribution and Estimation

  • The population preference is the annotator-average of each annotator’s probability of preferring B for a pairwise input.Source: §B.1 · pp. 16–17
  • When annotators hold stable preferences, the population preference is the fraction of annotators who prefer B.Source: §B.1 · p. 17
  • The multi-annotator Monte Carlo estimate is unbiased for the population preference, with variance decreasing as 1/m.The estimate samples annotators independently and draws each judgment from that annotator’s preference probability.Source: §B.1 · p. 17

B.2. Proof of Proposition 1

The proof shows that the population-optimal autorater reports the target preference probability deterministically and parsably, with clipping for the Log reward. It establishes this result through the Brier and Log reward cases.

B.2. Proof of Proposition 1

  • For the Brier reward, the optimal numeric report has zero variance and mean equal to the target preference probability.Source: §B.2 · p. 18
  • For the Log reward, strict concavity makes the clipped target probability the unique optimal report; unparsable outputs reduce expected reward.Source: §B.2 · p. 18
  • At the population optimum, the autorater gives a deterministic, parsable report of the truthful preference probability, clipped for the Log reward.Source: §B.2 · p. 18

C.1. Evaluation Details

The section describes evaluation of probabilistic autoraters against human annotations and reports that finetuned win-rate predictions better match human-voted win rates. It also outlines the evaluation metrics and GRPO finetuning details.

C.1. Evaluation Details

  • Pairwise judgments are evaluated as three-class classification, with macro-averaged metrics computed across classes.Calibration is measured using ECE and Brier score after binarizing preference distributions; Tie samples are excluded.Source: §C.1 · pp. 18–19

C.2. Finetuning Details

  • Setting 2 uses a GRPO objective with an unbiased KL estimator, and detailed finetuning settings are listed in Table 5.The objective is described using policy, reference-policy, advantage, clipping, and KL-penalty terms.Source: §C.2 · p. 19, Table 5 · p. 20

D.1. Win Rate Prediction

  • Finetuned autoraters’ predicted win rates are more aligned with human-voted win rates on the Gemma 1.0 versus Gemma 1.1 evaluation.The evaluation uses 900 prompts and Qwen-2.5-7B as the base autorater to avoid potential self-enhancement bias.Source: §D.1 · p. 20

D.2. RL with Probabilistic Labels

RL with probabilistic labels improves agreement and calibration over the zero-shot baseline, but performs below autoraters trained with a larger set of binary labels.

D.2. RL with Probabilistic Labels

  • The Brier-reward RL model uses the same 5K prompts, each with 10 annotations, as SFT..Source: §D.2 · pp. 20–21
  • RL with probabilistic labels improves agreement and ECE over zero-shot, but underperforms autoraters trained with a larger set of binary labels.The comparison includes zero-shot verbalized confidence, SFT trained on probabilistic labels, and RL trained on binary labels.Source: §D.2 · p. 21, Figure 3 · p. 21

E. Discussions

The discussion presents preference-distribution prediction as an approach to more reliable autoraters, distinguishing human-preference from epistemic uncertainty and discussing extensions alongside the alignment and out-of-distribution generalizability of point estimates. JudgeLM examples illustrate agreement on the scientifically detailed anemone answer and divergent rash judgments, attributed to differing views of practical self-care versus professional guidance.

E. Discussions

  • The framework extends beyond the empirically studied pairwise setting to pointwise evaluation with nominal or ordinal preferences, such as Likert scales.The authors leave a thorough empirical study of pointwise evaluation for future work.Source: E. Discussions · p. 21
  • For pointwise evaluation, the authors propose using multi-class strictly proper scoring rules, including the multi-class Brier score or cross-entropy, as RL rewards.They present this as a direction for extension rather than an empirically studied method.Source: E. Discussions · p. 21
  • The verbalized confidence is intended to represent aleatoric human-preference uncertainty, while epistemic uncertainty may require a more complex second-order probability distribution.The authors report that point-estimating probability is sufficient to improve alignment with the preference distribution while maintaining good out-of-distribution generalizability.Source: E. Discussions · pp. 21–22

G.1. JudgeLM Dataset

  • In the anemone example, verbalized confidence, SFT CoT, and both RL methods favor B, citing its greater scientific accuracy and detail.The explanations contrast B’s scientific classification, physical characteristics, and feeding habits with inaccuracies or missing detail in A.Source: §G.1 · pp. 30–31
  • For the rash example, verbalized CoT and RL (Log) favor A for its practical advice, directness, and multiple treatment options.The verbalized-CoT explanation says B’s allergist suggestion is accurate but lacks immediate relief; RL (Log) says it does not directly answer how to get rid of the rash.Source: §G.1 · pp. 33–34
  • In contrast, SFT CoT and RL (Brier) favor B, emphasizing professional diagnosis and a more targeted approach over A’s lengthy or potentially misleading advice.The SFT CoT explanation flags the vague “rash shot” advice and prioritizes professional guidance; RL (Brier) calls B more likely to help with the specific request.Source: §G.1 · pp. 33–34

G.2. PandaLM Dataset

The appendix illustrates PandaLM comparisons with grammar-correction and restaurant-rating examples, showing different B-better probability predictions across judging approaches.

G.2. PandaLM Dataset

  • The grammar example compares edits to “would of” and “affect,” with the displayed response texts showing B's versions “would have” and “impact.”The task input and response excerpts are shown in the appendix.Source: §G.2 · pp. 34–36
  • For grammar correction, the examples show B-better probabilities of 0.60 with SFT CoT, 0.98 with RL Brier, and 1.00 with RL Log.The accompanying reasoning favors B for its tense and word choices.Source: §G.2 · pp. 35–36
  • On the Yelp-rating example, verbalized confidence, SFT CoT, RL Brier, and RL Log predict B-better probabilities of 0.95, 1.00, 0.99, and 1.00, respectively.The shown reasoning favors B's 3-star prediction over A's 5-star response.Source: §G.2 · pp. 36–38

Related papers

Loading 2510.00263v1…