Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Yinhong Liu; Han Zhou; Zhijiang Guo; Ehsan Shareghi; Ivan Vulić; Anna Korhonen; Nigel Collier

Source-linked AI summary

Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, Nigel Collier

arXiv:2403.16950v5Mar 25, 2024cs.CLcs.AIcs.LG

TL;DR

LLM evaluators can be biased and misaligned with human judgements, while existing calibration methods do not reliably resolve this gap. The paper analyzes this limitation and introduces PAIRS, an uncertainty-guided pairwise-ranking method that improves alignment and scalability across evaluation tasks.

Problem
LLM evaluators are efficient but exhibit biases and remain misaligned with human judgements, while calibration methods are insufficient for reliable alignment.
Method
PAIRS formulates evaluation as maximum-likelihood ranking and uses uncertainty-guided search over pairwise comparisons to rank candidate texts efficiently.
Results
PAIRS substantially outperforms traditional direct-scoring evaluations and produces more aligned, robust, and transitive evaluations across representative tasks.
Takeaways & Limitations
Pairwise preference provides a scalable evaluation paradigm while enabling analysis of LLM transitivity and calibration with debiased pairwise evaluations.
Takeaways & Limitations
The calibration analysis assumes identical LLM and human evaluation likelihoods, an assumption the results find invalid.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed at mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PAIRS), an uncertainty-guided search-based rank aggregation method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PAIRS achieves state-of-the-art performance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PAIRS benefits from calibration using debiased pairwise evaluations.

1 Introduction

LLM evaluators offer efficient reference-free assessment but remain biased and misaligned with human judgements. The paper analyzes calibration limits and introduces PAIRS, which uses uncertainty-guided pairwise comparisons to construct scalable evaluation rankings.

LLM evaluators provide inexpensive reference-free assessment but are sensitive to prompts and affected by positional, verbosity, and contextual biases.
Existing calibration methods remain insufficient for aligning direct-scoring LLM evaluations with human judgements, even with supervised data.
The paper reframes evaluation as ranking through pairwise comparisons, drawing on preference data used to align reward models in RLHF.
PAIRS uses uncertainty-guided search to estimate maximum-likelihood preference rankings while avoiding exhaustive pairwise comparisons.
The paper studies PAIRS on summarization and open-ended generation, reporting stronger alignment, query efficiency, robustness, and transitivity than existing approaches.

2 On the Limitations of Calibration in Aligning LLM Evaluators

The paper models LLM and human score evaluations as posterior predictive distributions and tests whether calibration can close their gap. Results show that calibration may leave substantial misalignment because evaluation likelihoods, interpreted as standards, differ.

LLM score-based evaluation is represented as a posterior predictive distribution over ratings for an input-output pair under an instruction prompt.
The posterior distribution combines a likelihood term with a score-preference prior, motivating calibration of the LLM prior toward the human prior.
Calibration methods assume that LLM and human evaluation likelihoods are identical while adjusting score-preference priors.
MAE shifts from 1.62 to 1.16 on HANNA and from 0.78 to 0.86 on SummEval after calibration, leaving notable misalignment.
The findings attribute the remaining discrepancy primarily to different evaluation likelihoods, which represent underlying evaluation standards rather than score priors.

3 Uncertainty-Guided Pairwise-Preference Search

PAIRS formulates LLM evaluation as rank aggregation over pairwise preferences and searches for high-likelihood rankings while balancing non-transitivity, uncertainty, and computational cost. Its beam search explores alternative merge paths, uncertainty pruning reduces comparisons, and a two-stage variant scales ranking to large sets.

3.1 Evaluation as Ranking: Pairwise evaluation becomes a rank-aggregation problem because LLM preference probabilities may be non-transitive, making the optimal ranking an MLE problem.The ranking objective seeks a total order that maximizes the likelihood of observed pairwise preferences.
3.1 Evaluation as Ranking: Computing the non-transitive likelihood requires O(N^2) pairwise comparisons, while exact rank aggregation is NP-hard.Assuming transitivity reduces comparisons to O(N log N), but strict transitivity is a strong assumption.
3.2 Uncertainty-Guided Pairwise-Preference Search: PAIRS-beam improves on greedy merge search by retaining top-k trajectory candidates and considering less-preferred items at each step.This broader search is designed to address cases where local pairwise probabilities do not yield the optimal global ranking.
3.2 Uncertainty-Guided Pairwise-Preference Search: Entropy-based uncertainty pruning removes less-preferred candidates when uncertainty falls below threshold U_h, trading search breadth for efficiency.The pruning mechanism restricts the beam search space while balancing computational complexity and ranking performance.
3.3 A Scaling Variant of PAIRS: PAIRS-beam ranges from O(N log N) to O(N^2) comparisons, while a two-stage anchor-set and binary-search variant achieves O(N) complexity in its second stage.The upper bound occurs without uncertainty pruning and with a sufficiently large beam; the lower bound corresponds to PAIRS-greedy.
3.4 On the Benefits of Pairwise-Preference Search: Transitivity is proposed as a measure of LLM evaluator quality because PAIRS-direct and PAIRS-beam rely on transitivity to some extent.The paper expects more capable evaluators to exhibit more transitive pairwise behavior, supporting comparisons of evaluator suitability.

4 Related Work

Related work positions LLMs as inexpensive automatic evaluators and pairwise preference as an evaluation strategy inspired by preference-based reward modeling. These approaches address diverse generation tasks but leave pairwise-evaluation complexity and scalability as important concerns.

LLMs as Evaluators: LLM evaluators provide cheaper reference-free or reference-guided alternatives to human evaluation across diverse natural-language generation tasks.Prior applications include summarization, machine translation, open-ended story generation, and dialogue response evaluation.
Pairwise Preference: Pairwise preference has been widely used in RLHF reward modeling and has more recently been applied to evaluating LLM outputs.Prior work includes binary chosen-versus-rejected preferences and pairwise comparisons of generated responses.

5 Experiments

Experiments evaluate PAIRS across summarization and story-generation benchmarks using multiple open- and closed-source LLMs. PAIRS generally improves alignment with human judgments over direct scoring, while uncertainty-guided search, transitivity, and calibration affect efficiency and robustness.

Main results: PAIRS achieves state-of-the-art or near-state-of-the-art performance across the evaluation datasets, generally outperforming direct scoring and G-Eval baselines.The main exception is SummEval consistency, where concentrated human scores make pairwise comparisons less informative.
Model comparisons: Pairwise comparison generally benefits smaller models on News Room and SummEval, while PAIRS consistently improves performance for GPT models across all three datasets.Llama-2 7B rates summaries more effectively through comparisons than scores, whereas small models benefit less on HANNA.
Search and transitivity: PAIRS-beam generally yields higher correlations and lower Standard Error than PAIRS-greedy, although their performance difference is minimal for GPT-4-turbo.The authors attribute GPT-4-turbo’s small gap to better transitivity, which lets greedy search find near-optimal rankings.
Efficiency comparison: PAIRS-greedy typically needs about 30% of the comparisons required by ELO to reach similar performance, while calibrated PAIRS can outperform baselines using all pairwise comparisons.PAIRS exploits transitivity to identify informative comparisons, creating an efficiency advantage for ranking fixed candidate responses.
Calibration: Calibration consistently improves PAIRS for Mistral 7B and Llama-2 7B, but produces relatively small gains for GPT-3.5-turbo because its output probabilities are heavily skewed.The calibration averages probabilities from both candidate-order permutations.

6 Conclusion

The paper finds that calibration does not effectively align direct-scoring LLM evaluators with human judgments, and proposes PAIRS as an efficient, scalable ranking method. PAIRS improves alignment and performance, while the ranking formulation leaves trade-offs for future search algorithms and scaling approaches.

Calibration techniques do not effectively resolve misalignment between direct-scoring LLM evaluators and human evaluators.
PAIRS formulates evaluation as ranking and uses uncertainty-guided search to estimate maximum-likelihood rankings from pairwise comparisons.
PAIRS achieves stronger alignment with human annotations and state-of-the-art performance than direct-scoring and domain-optimized baselines.
Future Work: The ranking formulation creates a setting for future algorithms to optimize the trade-off between performance and efficiency under different transitivity assumptions.
Future Work: Alternative scaling could improve search efficiency from O(log2N) to O(logKN) by selecting the best candidate from K options rather than using pairwise ranking.

C Derivations

The derivation defines rank aggregation as finding the ranking most consistent with pairwise comparison probabilities. Under independent comparisons, the likelihood assigns each comparison its probability when consistent with the ranking and its complement otherwise.

Rank aggregation seeks a ranking of items that is most consistent with a set of pairwise comparisons.
The general ranking likelihood is defined over the set of pairwise comparisons using probabilities conditioned on a candidate ranking.
Rank aggregation assumes pairwise comparisons are independent and takes P(i ≻j) as the input probability for each comparison.
A ranking-consistent comparison contributes P(i ≻j), whereas an inconsistent comparison contributes 1 −P(i ≻j).
Using P(j ≻i) = 1 −P(i ≻j) and independence absorbs the reverse-comparison product into the likelihood expression.

C.2 Transitive Ranking Likelihood Derivation

The transitive ranking likelihood is derived from non-transitive pairwise likelihoods using compositional stochastic transitivity. This reduces comparisons to adjacent probabilities while supporting score conversion through quantile or CDF matching.

Compositional stochastic transitivity expresses P(a ≻c) as a monotone symmetric function of P(a ≻b) and P(b ≻c).
For a ranking of length N, comparisons at distance k can be represented using distance-1 comparison probabilities and k −1 coefficients.
The transitivity construction uses O(N3) coefficients, while the non-transitive likelihood can be expressed through distance-1 probabilities.
With equal μ coefficients, the formulation contains around O(N2) constraint equations expressed through λ values.
Rankings can be converted into desired score distributions by matching quantiles or cumulative density functions.
CDF matching transforms variables so that the output variable has the target variable’s cumulative distribution function.
Under a [10%, 20%, 40%, 20%, 10%] prior for scores 1 to 5, ranking positions are assigned to those score intervals.

D Datasets

The experiments evaluate summarization and story generation across datasets containing multiple candidate texts and human annotations for several quality aspects. HANNA additionally uses a scaled PAIRS variant because of computational cost.

Summarization Tasks: SummEval is annotated for coherency, fluency, consistency, and relevancy.
Summarization Tasks: NewsRoom contains 60 source texts with seven summaries per sample and annotations for coherence, fluency, relevance, and informativeness.
Story Generation Task: HANNA contains 1,056 creative stories generated from 96 prompts and evaluates coherence, surprise, and complexity.
Story Generation Task: Because of computational cost, the HANNA experiments use the scaled PAIRS variant with an anchor size of 100.

E Implementation Details

The evaluation uses both sample-level and dataset-level correlation strategies. Sample-level correlations are computed across candidates for each input and then averaged, whereas dataset-level correlation is computed across the entire dataset.

Sample-level correlation is computed across multiple candidates for the same input and then averaged across input samples.
Dataset-level correlation is computed across the entire dataset.

F.1 Scaling Variant Performance Loss

This section evaluates the scaling variant's performance–cost trade-off, anchor-set sizing, and implementation settings. PAIRS-beam-scaled maintains comparable performance to PAIRS-greedy while reducing model queries and improving computational scaling.

Scaling Variant Performance Loss: Binary search in the scaling variant is expected to reduce performance relative to PAIRS-greedy and PAIRS-beam, creating a performance–computational-cost trade-off.The ablation study uses Mistral-7B and Llama2-7B.
Scaling Variant Performance Loss: PAIRS-beam-scaled achieves comparable performance to PAIRS-greedy while requiring fewer model queries.The comparison uses Spearman correlations and model-query counts on the HANNA coherence task.
Scaling Variant Performance Loss: O(log 100) computational growth makes PAIRS-beam-scaled more efficient than PAIRS-greedy's O(N log N) growth as problem size increases.
Scaling Variant Performance Loss: Different base-model logit distributions imply different optimal uncertainty thresholds.Mistral 7B and Llama2 7B place many probabilities between 0.3 and 0.7, whereas GPT-3.5-turbo has a different distribution.
Anchor Set Size: An anchor set size of 100 yields relatively small KL-divergence error from the full score distribution across 100 random sampling repetitions.The empirical comparison uses the HANNA coherence dataset and reports means and standard deviations for the sampled-set divergences.
Anchor Set Size: Using confidence level 0.8, expected common-category proportion 0.4, 7% error margin, and design effect 1.5 gives an estimated sample size of around 120 for five categories.The anchor-size calculation treats Likert scores as a five-category classification problem and adjusts the single-proportion sample size for multiple categories.