Source-linked AI summary
Large Language Models are not Fair Evaluators
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, Zhifang Sui
TL;DR
LLM evaluators can exhibit positional bias, allowing candidate-response order to change quality rankings and weakening automated assessment. The paper introduces three calibration strategies and finds improved alignment with human judgments, including with limited human assistance.
Problem
Automated LLM evaluation is sought because human assessment is costly and time-consuming, but candidate ordering can systematically bias evaluation results.
Method
The paper proposes Multiple Evidence Calibration, Balanced Position Calibration, and Human-in-the-Loop Calibration to calibrate positional bias.
Results
MEC and BPC improve evaluation alignment accuracy by 9.8% and 14.3%, respectively, while HITLC reaches comparable or better human alignment with 20% human annotation cost.
Takeaways & Limitations
Calibrating evaluation order and selectively incorporating human assistance brings LLM-based judgments closer to human judgments.
Abstract
from arXiv · showhide
In this paper, we uncover a systematic bias in the evaluation paradigm of adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. To address this issue, we propose a calibration framework with three simple yet effective strategies: 1) Multiple Evidence Calibration, which requires the evaluator model to generate multiple evaluation evidence before assigning ratings; 2) Balanced Position Calibration, which aggregates results across various orders to determine the final score; 3) Human-in-the-Loop Calibration, which introduces a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when needed. We also manually annotate the "win/tie/lose" outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and extensive experiments demonstrate that our approach successfully mitigates evaluation bias, resulting in closer alignment with human judgments. We release our code and human annotation at \url{https://github.com/i-Eval/FairEval} to facilitate future research.
1 Introduction
LLM-based evaluation offers an efficient alternative to costly human assessment, but candidate ordering can systematically distort judgments. The paper proposes calibration strategies and reports closer alignment with human evaluations.
- Changing candidate-response order can overturn comparison results despite explicit instructions to ignore presentation order.
- Human evaluation is accurate and insightful but costly and time-consuming, motivating automated assessment methods aligned with human judgments.
- GPT-4 tends to favor the first candidate, while ChatGPT typically favors the second, revealing positional bias in LLM evaluators.
- The proposed framework combines Multiple Evidence Calibration, Balanced Position Calibration, and Human-in-the-Loop Calibration to reduce positional bias.The strategies generate multiple evidence before scoring, average evaluations across candidate positions, and use human assistance when examples appear difficult.
- 9.8% and 14.3% accuracy improvements were achieved by MEC and BPC, respectively, in evaluation alignment for GPT-4 and ChatGPT.Experiments used manually annotated win/tie/lose outcomes for ChatGPT and Vicuna-13B across 80 Vicuna Benchmark questions spanning 9 categories.
2 Positional Bias of the LLM Evaluator
LLM evaluators can change their judgments when the same responses are presented in a different order, producing substantial positional bias and conflicting results. The effect is stronger when response quality is closer, motivating calibration strategies that compare positions and incorporate evidence.
- Position sensitivity: Swapping response positions can produce different evaluation results, revealing that LLM evaluators are sensitive to the order of responses.The evaluation template queries the evaluator with the two responses assigned to different assistant slots.
- Conflict Rate: 46.3% and 5.0% conflict rates were observed for GPT-4, while ChatGPT showed 82.5% and 52.5%, respectively.Conflict Rate measures how often the evaluator gives conflicting results after the two responses exchange positions.
- Dependence on response quality: Smaller score gaps are associated with more conflicting GPT-4 evaluations, while score gaps of at least 3 produce relatively stable results.The conflict rate is negatively correlated with the score gap between responses.
- Position preference: GPT-4 tends to favor the first response, whereas ChatGPT tends to favor the second response.For Vicuna-13B versus ChatGPT, GPT-4’s Vicuna win rates were 51.3% and 23.8% across positions, while ChatGPT’s were 2.5% and 82.5%.
- Position preference: 51.3% versus 23.8% GPT-4 win rates and 2.5% versus 82.5% ChatGPT win rates show how strongly Vicuna-13B’s ranking depends on its assigned position.These values compare Vicuna-13B when positioned as Assistant 1 versus Assistant 2 against ChatGPT.
- Dependence on response quality: Positional bias varies by response pair: Vicuna-13B versus Alpaca-13B has a much lower conflict rate than Vicuna-13B versus ChatGPT.The paper suggests that a larger quality difference may make positional bias insufficient to change the evaluation.
3 Calibrating the Positional Bias
The paper introduces a calibration framework to mitigate positional bias in LLM evaluation through multiple evidence, balanced position, and human-in-the-loop strategies.
- Positional bias can significantly affect LLM evaluation results, making evaluators unfair.
- 3.1 Multiple Evidence Calibration: Multiple Evidence Calibration prompts the model to generate evaluation evidence before assigning scores and ensembles multiple results.The evidence calibration template addresses the mismatch between conclusions and later explanations in autoregressive models.
- 3.2 Balanced Position Calibration: Balanced Position Calibration swaps the two responses and averages the resulting 2k scores to obtain calibrated response scores.Each candidate is evaluated in both positions across the original and swapped prompts.
- 3.3 Human-in-the-Loop Calibration: Human-in-the-Loop Calibration uses Balanced Position Diversity Entropy to select examples likely to require manual correction.Examples with higher BPDE scores are selected up to a top-β threshold and then integrated with majority human annotations.
4 Experiments
Experiments compare LLM evaluators and calibration methods against human annotations on 80 Vicuna Benchmark questions, finding improved alignment and lower evaluation cost.
- 4.1 Human Annotation: 80 Vicuna Benchmark questions were independently annotated by three authors using win/tie/lose judgments across four response-quality perspectives.The perspectives were helpfulness, relevance, accuracy, and level of detail; each example took an average of three minutes.
- 4.2 Experimental Settings: Accuracy and kappa correlation coefficient were measured against the final majority of human annotations.The experiments used OpenAI API models, with deterministic generation for methods not requiring multiple samples and k = 3 for multiple evidence.
- 4.3 Main Results: 14.3% accuracy improvement raised ChatGPT’s kappa correlation coefficient from 0.06 to 0.31 with MEC and BPC.The proposed calibration strategies improved alignment for both GPT-4 and ChatGPT compared with VANILLA evaluation.
- 4.3 Main Results: 20% human assistance reduced annotation cost from $30 to $18.3, a 39% reduction, while ChatGPT attained comparable Human Average accuracy.
- 4.3 Main Results: The calibration methods improved evaluation performance while maintaining low costs.
5 Analysis
Analysis examines evidence count, sampling temperature, human-selection criteria, evaluation templates, and task categories to characterize calibration performance and robustness.
- 5.1 Ablation on Evidence Number k and Temperature t: k = 3 yields optimal performance while keeping API cost relatively low; larger k values plateau or decrease slightly.ChatGPT performance increases and then tends to remain constant or decline slightly as the number of evidence samples grows.
- 5.1 Ablation on Evidence Number k and Temperature t: Sampling temperatures of 0.2 and 1.4 produce suboptimal evaluation alignment, whereas 0.6 or 1.0 are appropriate choices in the experiments.Low temperature weakens MEC’s sampling randomness, while high temperature compromises generation quality.
- 5.2 Effectiveness of the BPDE: BPDE and Vanilla Diversity Entropy outperform Random for selecting examples requiring human annotation, while BPDE also outperforms Vanilla DE.The comparison uses six evaluation results for both entropy methods and shows that BPC results improve HITLC over MEC alone.
- 5.4 Fine-grained Analysis: MEC and BPC improve ChatGPT and GPT-4 evaluation performance across nearly all categories, especially common sense, coding, and math for ChatGPT.
- 5.3 Different Evaluation Templates: Calibration narrows the nearly 6% accuracy gap between VANILLA scoring and comparing templates and reduces their conflict rate.
6 Related Work
LLM-based evaluation has emerged because traditional metrics inadequately capture alignment with human intent, while human evaluation is accurate but costly. However, the reliability of using LLMs as evaluators remains insufficiently explored, motivating examination of positional bias.
- Traditional metrics such as BLEU, ROUGE, BERTScore, and BARTScore do not adequately measure whether generated responses align with human intent.
- Human evaluation is considered the most accurate measure of model performance but is costly and time-consuming at scale.
- LLMs have therefore been adopted to evaluate generative-model proficiency because of their powerful general generation capabilities.
- Despite widespread use as proxies for human evaluators, the reliability of the LLMs-as-evaluator paradigm is not well explored.
- This paper uncovers significant positional bias and proposes three methods to calibrate it for more reliable and fair evaluation results.
7 Conclusion
The paper shows that changing candidate-response order can significantly influence LLM evaluation rankings. It introduces three calibration strategies that reduce this bias and improve alignment with human judgments, while releasing code and human annotations for future research.
- Manipulating candidate-response order can significantly influence quality rankings produced by ChatGPT and GPT-4 evaluators.
- The framework combines Multiple Evidence Calibration, Balanced Position Calibration, and Human-in-the-Loop Calibration to address positional bias.
- MEC elicits multiple evaluation evidence before ratings, BPC aggregates results across response orders, and HITLC selects examples for human annotation using balanced position diversity entropy.
- The strategies reduce evaluation bias and improve alignment with human judgments.
- The authors provide code and human annotations to support future studies of generative-model evaluation.