PaperlayerView exact revision on arXiv

Source-linked AI summary

CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges

Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, Yiqun Liu

arXiv:2410.15393v1cs.CL

TL;DR

LLM judges face selection bias, with judgments that can vary when option positions or ID tokens are swapped. CalibraEval uses label-free, inference-time optimization and a non-parametric order-preserving algorithm; experiments report reduced bias and improved accuracy across models and datasets.

  • Problem

    CalibraEval targets selection bias in LLM judges; evaluating subjective generated text is difficult, and traditional and semantic metrics do not comprehensively reflect LLM capabilities.

    Source: §1 · pp. 1–2

  • Method

    CalibraEval uses label-free, inference-time optimization and NOA, which preserves partial orders among prediction distributions to determine a calibration function.

    Source: §1 · p. 2

  • Results

    Across six LLMs and three representative datasets, CalibraEval reduces selection bias and enhances accuracy, outperforming strong baselines and showing robustness across prompts, option tokens, and in-context learning.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors argue that mitigating selection bias is essential for developing more reliable LLM evaluators.

    Source: §6 · p. 9

    The formulation assumes observed distributions combine prior and debiased distributions, while their relationship is difficult to estimate because interactions may be complex and observations noisy.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

The use of large language models (LLMs) as automated evaluation tools to assess the quality of generated natural language, known as LLMs-as-Judges, has demonstrated promising capabilities and is rapidly gaining widespread attention. However, when applied to pairwise comparisons of candidate responses, LLM-based evaluators often exhibit selection bias. Specifically, their judgments may become inconsistent when the option positions or ID tokens are swapped, compromising the effectiveness and fairness of the evaluation result. To address this challenge, we introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. Specifically, CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. To solve this optimization problem, we propose a non-parametric order-preserving algorithm (NOA). This algorithm leverages the partial order relationships between model prediction distributions, thereby eliminating the need for explicit labels and precise mathematical function modeling.Empirical evaluations of LLMs in multiple representative benchmarks demonstrate that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods. This work marks a step toward building more robust and unbiased automated evaluation frameworks, paving the way for improved reliability in AI-driven assessments

1 INTRODUCTION

CalibraEval targets selection bias in pairwise LLM judgments, where swapping option positions or ID tokens can make evaluations inconsistent. It calibrates prediction distributions without labels, and experiments report stronger debiasing performance than baselines across representative benchmarks.

1 INTRODUCTION

  • Evaluating generated text is difficult for subjective tasks, and traditional n-gram and semantic metrics do not fully reflect LLM capabilities.Human evaluation can measure performance accurately but is costly and time-consuming.Source: §1 · p. 1
  • LLM judgments can change when candidate positions or ID tokens are swapped, creating position and token biases that affect evaluation validity and fairness.The paper groups these related effects under selection bias because option tokens and positions are inherently linked.Source: §1 · p. 2
  • Existing responses to inconsistent judgments either discard them as ties, risking information loss, or use multi-round interactions that are costly and have uncertain debiasing effectiveness.Bias also varies with task characteristics and properties of the evaluating LLM.Source: §1 · p. 2
  • CalibraEval is a label-free, inference-time method that optimizes a projection from observed prediction distributions to unbiased distributions using consistency after position and ID swaps.NOA preserves partial-order relationships among observed prediction distributions to narrow the solution space without explicit labels or precise function modeling.Source: §1 · p. 2
  • CalibraEval outperforms strong baselines in debiasing and achieves state-of-the-art results, with robustness validated across prompts, option tokens, and in-context learning.Experiments span representative evaluation benchmarks and various LLMs.Source: §1 · p. 2

2 RELATED WORK

Related work motivates LLM judges as a scalable alternative to manual assessment, with pairwise comparisons valued for their consistency with human judgments. It then situates selection bias among reliability concerns and reviews prior mitigation efforts.

2.1 LLMs as Judges

  • LLM judges offer automated evaluation as an alternative to costly manual assessment, through either pointwise scoring or pairwise choice.Pairwise comparison has gained attention for its performance and reported alignment with human evaluation standards.Source: §2.1 · p. 2
  • Pairwise evaluation is presented as economical relative to multi-model collaboration, which can require multiple rounds and greater resource consumption.Prior findings report better consistency with human assessments than traditional score-based approaches.Source: §2.1 · pp. 2–3

2.2 Bias in LLM Judges

  • Biases in LLM judges threaten evaluation effectiveness and fairness, and selection bias is one of several identified reliability concerns.Other identified biases include position, contextual, and self-reinforcing bias.Source: §2.2 · p. 3
  • Selection bias is prevalent across tasks and model types, appearing when a judge favors an option even after its position or ID is swapped.The literature has not reached consensus on whether position or token bias is dominant.Source: §2.2 · p. 3
  • Prior mitigation work includes positional-bias studies, fine-tuning, answer splitting and merging, and estimates from partial samples, while calibration remains unresolved.The review notes that existing calibration techniques may be insufficient even with supervised data.Source: §2.2 · p. 3

3 CALIBRAEVAL

CalibraEval treats selection-bias mitigation as calibration of observed prediction distributions, using consistency under swaps as its objective. Its non-parametric order-preserving algorithm learns the calibration without explicit labels or a precise model of the distribution relationship.

3.1 Problem Statement

  • Because selection bias is consistent preference for a specific option regardless of content, the method maps observed probabilities toward an unbiased distribution rather than modeling f(·) precisely.The observed distribution is assumed to combine prior and debiased distributions, but their relationship may be complex and noise-affected.Source: §3.1 · p. 3
  • The method avoids specifying the complex relationship f(·), which may be difficult to identify when observed distributions are affected by noise.Pride [39] instead simplifies that relationship by assuming linear multiplication, an assumption described as potentially overlooking complex distribution relationships.Source: §3.1 · p. 3

3.2 Optimization Objective

  • The optimization targets consistent judgments when option IDs or positions are swapped, while regularization prevents the trivial mapping g(·) = 0.5.Its loss includes terms for token-swap and position-swap consistency.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Non-parametric Order-Preserving Algorithm (NOA)

  • CalibraEval uses a non-parametric order-preserving algorithm to learn a calibration function without explicit labels or precise mathematical modeling.The method can run at inference time with minimal computational cost, using all test samples or a subset.Source: §3.3 · pp. 4–5
  • NOA uses the order-preserving discrete mapping to obtain calibration values, then fits a continuous non-decreasing piecewise-linear function by weighted least squares.PAVA is used to derive the continuous calibration function.Source: §3.3 · pp. 4–5

4 EXPERIMENT SETUP

The experiments use three representative benchmarks, reference-free consistency metrics and reference-based bias and accuracy metrics. They compare CalibraEval with non-supervised baselines across six LLMs from three families under a shared test-data prior-estimation setup.

4.1 Datasets and Metrics

  • The evaluation uses RewardBench, MTBench, and PreferenceBench; MTBench tie options are excluded from reference-based metrics.The benchmarks contain 2,985 prompt-choice-rejection trios, 3,355 human preferences, and 2,000 response pairs with 200 criteria, respectively.Source: §4.1 · p. 5
  • Fleiss’s Kappa and ICC measure consistency after ID-token and position swaps, while RStd and accuracy provide reference-based evaluation.RStd measures recall imbalance, with greater imbalance indicating more selection bias.Source: §4.1 · p. 5

4.2 Baselines

  • The baselines are Debiasing Instruct, Contextual Calibration, Domain-context Calibration, and Pride; supervised methods are excluded because CalibraEval is label-free.The methods use instructions, calibrated priors, or a prior-bias division operation.Source: §4.2 · p. 5

4.3 Implementation Details

  • Six models from the Llama, Qwen, and GPT families are evaluated, with the full test set used for prior estimation by all baselines.The implementation uses batch gradient descent with batch size 32 and a stopping threshold of 0.001.Source: §4.3 · pp. 5–6

5 EXPERIMENT RESULT

CalibraEval improves evaluation consistency across models and tested settings, including prompt templates, ID tokens, and in-context learning. Ablations favor combining position- and token-swap objectives, while even a small estimation set supports calibration.

5.1 Main Results

  • Across three datasets, CalibraEval achieves lower RStd and higher accuracy; on RewardBench, ChatGPT’s RStd fell from 16.79 to 5.51 while accuracy rose from 65.27 to 67.13.Table 2 reports reference-based metrics; its caption identifies RStd and accuracy and marks lower RStd as better.Source: Table 2 · p. 7, §5.1 · p. 7
  • CalibraEval consistently improves across LLMs and tasks, while Debiasing Instruct is inconsistent, CC/DC priors may misfit selection bias, and Pride weakens on advanced models.Pride improves consistency for lower-capability models but its effectiveness diminishes on models such as GPT4o.Source: §5.1 · pp. 6–7

5.2.1 Different prompt templates.

  • Across prompt templates, uncorrected outputs have low consistency and high variance; Pride improves consistency but retains variance, while CalibraEval improves performance with low variance.The comparison is reported for RewardBench.Source: Figure 3 · p. 7, §5.2.1 · p. 7
  • Across A/B, a/b, Alice/Bob, and X/Y tokens, CalibraEval improves performance with low variance; on Qwen-72B, Pride’s improvement is negligible.Source: Figure 4 · p. 7, §5.2.2 · p. 7

5.2.2 Different ID tokens.

  • With 1-, 2-, and 3-shot examples, CalibraEval remains effective, although calibration effectiveness decreases as examples increase even as judgment consistency improves.The authors suggest examples may introduce new biases and see greater calibration potential in zero-shot settings.Source: Figure 5 · p. 8, §5.2.3 · p. 8

5.3 Ablation Studies

  • Increasing the estimation-set size can improve consistency, but for ChatGPT, using 10% of the data produced improvements of over 85% compared to the full dataset.The sampled test data estimates the calibration function, which is then applied to the entire test set.Source: Figure 6 · p. 8, §5.3 · p. 8
  • Both swap-consistency objectives contribute, and their combination performs best; L2 helps more under position bias, while L1 helps more under token bias.Source: §5.3 · p. 8

6 CONCLUSION

CalibraEval frames debiasing as optimization and uses NOA to determine a calibration function. Experiments report reduced selection bias and enhanced accuracy across six LLMs and three datasets.

6 CONCLUSION

  • CalibraEval reformulates debiasing as an optimization problem and uses NOA to determine its calibration function.The conclusion says the optimization objectives use characteristics of unbiased evaluators.Source: §6 · pp. 8–9
  • Table 3 presents an ablation study on RewardBench, with best results marked in bold.The supplied caption does not provide the table's cell values.Source: Table 3 · p. 9
  • Across six LLMs and three representative datasets, CalibraEval reduces selection bias while enhancing accuracy.The authors argue that mitigating selection bias is essential for more reliable LLM evaluators and plan to investigate additional biases.Source: §6 · p. 9

A SUPPLEMENTARY PROOF

The supplement gives derivative steps for the calibration objective and outlines the calibration algorithm, from swapped-ID probabilities to debiased predictions.

A SUPPLEMENTARY PROOF

  • The proof applies the chain rule and substitutes derivatives back into Equation 22.The supplied steps include a derivative for g(s_i^2).Source: A SUPPLEMENTARY PROOF · p. 11
  • For the derivative of g(z_j) with respect to d_k, the supplement applies the quotient rule and distinguishes j < k from j >= k.The passage then presents a final formula.Source: A SUPPLEMENTARY PROOF · p. 11
  • The algorithm generates probabilities after exchanging option IDs, combines sets into a global set, then sorts it and adds endpoints to form Z.The listed endpoints are z_0 = 0 and z_3K+1 = 1.Source: A SUPPLEMENTARY PROOF · p. 11
  • The algorithm obtains a discrete mapping g from Equation (12), a continuous calibration function g* from Equation (16), and returns debiased predictions.It adds the predicted answer to Y before returning Y.Source: A SUPPLEMENTARY PROOF · p. 11

B.1 Details of Datasets

The dataset statistics are described as balanced across answer lengths, label categories, and difficulty, supporting a more controlled benchmark evaluation.

B.1 Details of Datasets

  • The average answer lengths of different options within each dataset are nearly identical.The authors say this design minimizes the potential influence of other biases on evaluation results.Source: §B.1 · p. 11
  • The label-category distributions are balanced across the benchmark datasets.The passage describes the datasets as thoughtfully constructed for fair and robust evaluation.Source: §B.1 · p. 11
  • The datasets are described as having a well-balanced difficulty distribution.This is part of the authors’ characterization of the benchmark construction.Source: §B.1 · p. 11

B.2 Details of Metrics

This section describes agreement and reliability metrics, then introduces reference-based measures of recall variability and prediction correctness.

B.2 Details of Metrics

  • Fleiss’s Kappa assesses agreement reliability among multiple raters, with values from -1 to 1 and values nearer 1 indicating stronger agreement.Values around 0 indicate no agreement beyond chance; negative values indicate systematic disagreement.Source: §B.2.1 · p. 11
  • ICC(2,k) measures consistency among multiple raters under a random-effects model, whereas ICC(3,k) measures consistency for specific fixed raters under a fixed-effects model.Both assess reliability or agreement for the same subjects.Source: §B.2.1 · p. 12

B.2.2 Reference-based Metrics.

  • RStd quantifies variability in recall scores across evaluations.The section introduces it as a reference-based metric.Source: §B.2.2 · p. 12
  • Accuracy measures overall prediction correctness using true-positive, true-negative, false-positive, and false-negative counts.The section identifies accuracy as a reference-based metric.Source: §B.2.2 · p. 12

C MORE EVALUATION RESULT

The reference-based results report average improvements in RStd and accuracy. The prompt design uses a default prompt for the main experiments and four distinct prompts for robustness experiments.

D THE DESIGN OF PROMPT

C MORE EVALUATION RESULT

Related papers

Loading 2410.15393v1…