Source-linked AI summary
Self-Preference Bias in LLM-as-a-Judge
Koki Wataoka, Tsubasa Takahashi, Ryokan Ri
TL;DR
LLM-as-a-judge systems face self-preference bias that can promote evaluator-specific styles or policies, while reliable quantitative measurement and causal understanding remain limited. The paper introduces a fairness-based metric, evaluates eight LLMs, and examines perplexity as a possible mechanism. GPT-4 shows significant self-preference bias, and LLMs rate lower-perplexity texts more highly than humans regardless of whether they generated them, suggesting a role for text familiarity.
Problem
Reliable quantitative metrics for self-preference bias are lacking, and its fundamental causes remain unclear despite risks of promoting evaluator-intrinsic styles or policies.
Method
The paper proposes an Equal Opportunity-based metric, measures self-preference bias across eight LLMs in pairwise evaluation, and analyzes evaluations against text perplexity.
Results
GPT-4 exhibits significant self-preference bias, while LLMs assign higher evaluations than humans to lower-perplexity texts regardless of whether they are self-generated.
Takeaways & Limitations
The findings suggest that self-preference bias may reflect LLM preference for texts more familiar to them, as indicated by lower perplexity.
Takeaways & Limitations
Perplexity values were unavailable for GPT-4 and GPT-3.5-Turbo, and Llama analyses lacked human annotations for their responses.
Abstract
from arXiv · showhide
Automated evaluation leveraging large language models (LLMs), commonly referred to as LLM evaluators or LLM-as-a-judge, has been widely used in measuring the performance of dialogue systems. However, the self-preference bias in LLMs has posed significant risks, including promoting specific styles or policies intrinsic to the LLMs. Despite the importance of this issue, there is a lack of established methods to measure the self-preference bias quantitatively, and its underlying causes are poorly understood. In this paper, we introduce a novel quantitative metric to measure the self-preference bias. Our experimental results demonstrate that GPT-4 exhibits a significant degree of self-preference bias. To explore the causes, we hypothesize that LLMs may favor outputs that are more familiar to them, as indicated by lower perplexity. We analyze the relationship between LLM evaluations and the perplexities of outputs. Our findings reveal that LLMs assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated. This suggests that the essence of the bias lies in perplexity and that the self-preference bias exists because LLMs prefer texts more familiar to them.
1 Introduction
LLM-as-a-judge enables automated dialogue evaluation, but self-preference bias creates risks and lacks reliable measurement and clear causal explanation. This paper proposes a fairness-based metric, evaluates eight LLMs, and links the bias to preference for lower-perplexity texts.
- LLM-as-a-judge systems automate dialogue evaluation, but diverse response strategies make response quality challenging to measure.
- Self-preference bias is the tendency of an evaluator to overestimate its own outputs, potentially promoting its intrinsic ideologies or response styles.
- Reliable quantitative metrics and clear explanations of self-preference bias remain lacking, while prior approaches face assessment and real-world-diversity limitations.
- The paper proposes a fairness-based metric for pairwise evaluation and measures self-preference bias across eight LLMs.Pairwise comparisons support more consistent human judgments and are suited to analyzing discrepancies with human evaluations.
- GPT-4 exhibits significant self-preference bias, raising concern that using it as a judge may excessively influence evaluations with its own styles and policies.
- LLMs assign higher evaluations to lower-perplexity texts than human evaluators, regardless of whether the texts were self-generated.The finding suggests that text familiarity, reflected in how likely an LLM is to generate the same response, may underlie self-preference bias.
2 Related Work
Related work establishes LLM-as-a-judge as a broad evaluation and training paradigm while identifying self-preference as one among several inherent biases. Other documented biases include position and verbosity preferences, with some mitigation approaches already proposed.
- LLMs are increasingly used for flexible dialogue assessment across utility and safety perspectives.
- LLM feedback and pairwise relative-quality annotations are also used to improve LLM learning with varied training methods.
- Self-preference bias is one limitation among others in LLM-as-a-judge systems, including position bias and verbosity bias.
- Position bias can be mitigated by rearranging comparative options, and alignment-based mitigation methods have also been proposed.
3 Preliminaries: Fairness and Bias
The paper frames evaluator bias through fairness concepts, emphasizing Equal Opportunity because LLM evaluations are expected to align with human preferences. It contrasts this ground-truth-based notion with Demographic Parity, which does not rely on ground truth.
- Equal Opportunity requires equal recall across groups distinguished by sensitive attributes.
- Its condition equates the probability of a positive prediction given a positive ground-truth label across the two groups.
- Bias under Equal Opportunity is measured by the difference or ratio between group recalls, but ground-truth bias can undermine the definition's suitability.
- Demographic Parity requires consistent predictive distributions across groups and does not rely on ground truth.
- The study applies Equal Opportunity to LLM evaluators treated as classifiers because dialogue systems are assumed to align with human preferences.
4 Quantifying Self-Preference Bias
The paper quantifies self-preference bias in pairwise LLM evaluation using an Equal Opportunity-based metric and applies it to eight evaluators. GPT-4 shows the strongest measured bias, favoring its own responses even when humans prefer alternatives.
- 4.1 Self-preference Bias Metric: The metric measures deviation from human evaluations by comparing an evaluator’s favorable judgments of its own versus others’ responses.It uses conditional probabilities based on whether the human-preferred response is selected.
- 4.1 Self-preference Bias Metric: A bias value of 0 indicates no bias, values near 1 indicate strong self-preference, and −1 indicates reverse bias.Reverse bias means the evaluator tends to undervalue its own responses.
- 4.2 Experimental Setting: The experiment compares two responses in open-ended dialogues and evaluates the extent to which LLM judgments differ between self-generated and other-generated outputs.The study uses human queries and response pairs from the Chatbot Arena dataset, containing 33,000 dialogues.
- 4.2 Experimental Setting: Evaluation scores are computed from the probabilities assigned to response A and response B, with response positions swapped and scores averaged to mitigate position bias.The eight evaluators include GPT-3.5-Turbo, GPT-4, Vicuna-7b, Vicuna-13b, oasst-pythia-12b, dolly-v2-12b, Koala-13b, and stablelm-tuned-alpha-7b.
- 4.3 Result: 0.520 is GPT-4’s self-preference bias, the highest among the evaluated LLMs.GPT-4’s recall was approximately 0.945 for favorable human judgments and 0.425 for unfavorable ones.
- 4.3 Result: GPT-4 favored its own response in an example where humans and GPT-3.5-Turbo preferred the alternative response.The responses differed mainly in style: GPT-4 prefaced its answer with a limitation, while GPT-3.5-Turbo directly listed the requested items.
5 How Do LLMs Overestimate Their Own Outputs?
The paper investigates whether perplexity explains why LLM evaluators overestimate their own outputs by conditioning pairwise judgments on response perplexity. Across the analyzed models, LLMs generally favor lower-perplexity responses more strongly than humans, and this pattern is not dependent on whether the response was self-generated.
- 5 How Do LLMs Overestimate Their Own Outputs?: The analysis computes prompt-conditioned perplexity differences between response pairs, bins those differences, and compares LLM and human winning rates.GPT-4 and GPT-3.5-Turbo were excluded because their perplexity values were unavailable.
- 5 How Do LLMs Overestimate Their Own Outputs?: All analyzed models except stablelm-tuned-alpha-7b assign higher evaluations to responses with lower perplexity.The supplied Figure 3 caption also identifies dolly-v2-12b as an exception, whereas the results paragraph names only stablelm-tuned-alpha-7b.
- 5 How Do LLMs Overestimate Their Own Outputs?: Vicuna-13b, Vicuna-7b, Koala-13b, and oasst-pythia-12b show a stronger lower-perplexity preference than human evaluators.The authors interpret this as LLM evaluators changing their judgments excessively with response perplexity.
- 5 How Do LLMs Overestimate Their Own Outputs?: Winning rates are similar whether an evaluator’s own output is included or excluded, except for dolly-v2-12b and stablelm-tuned-alpha-7b.The authors therefore identify perplexity, rather than self-generation status, as the factor associated with the judgments.
- 5 How Do LLMs Overestimate Their Own Outputs?: Llama models also changed their evaluations more than humans as perplexity differences varied, extending the pattern to competitive models.These additional comparisons lacked human annotations for the original Llama responses, so the authors obtained evaluations for existing responses instead.
6 Discussion
The discussion links self-preference bias to perplexity and considers fairness-based measurement and ensemble mitigation. Results indicate that LLMs favor lower-perplexity texts more than humans, while demographic-parity scores require cautious interpretation.
- Mitigation: Ensemble evaluation using multiple models may reduce self-preference bias by avoiding reliance on a single evaluator.The authors suggest decreasing a model’s evaluation weight for samples on which it has low perplexity.
- Perplexity and bias: LLM evaluators assign higher ratings to lower-perplexity texts regardless of whether those texts are self-generated.This pattern is reported across models, with limited exceptions in the conditioned comparison.
- Perplexity and bias: High-perplexity texts may reflect domains that LLM evaluators have encountered less frequently or do not fully comprehend.The discussion connects this possibility to training processes that reduce perplexity on text corpora and dialogue data.
- Interpretation: GPT-4’s bias may reflect alignment with its predefined policies when comparing responses of comparable quality.The discussed cases often involve response-style differences rather than clear factual errors.
- Fairness metrics: The proposed metric is based on Equal Opportunity, while Demographic Parity measures how highly evaluators rate their own outputs without accounting for intrinsic quality.Demographic-parity scores therefore should not be interpreted as evidence of unjust evaluation.
- Fairness metrics: GPT-4 exhibited significant demographic-parity bias, followed by Vicuna-13b, consistent with results from the paper’s other definition.The reported ranking describes self-output ratings within the experimental setup.
7 Conclusion
The conclusion reports a metric-based measurement of self-preference bias across eight LLMs and identifies perplexity as a central correlate. GPT-4 shows high self-preference bias, while LLM evaluators favor lower-perplexity texts more than humans regardless of authorship.
- Conclusion: The study proposes a metric for self-preference bias and measures eight LLMs, finding that GPT-4 exhibits high bias.The authors warn that GPT-4 judging may reinforce its own style and policies.
- Conclusion: Compared with human evaluators, LLM evaluators assign higher evaluations to lower-perplexity texts regardless of whether they generated them.The conclusion states that this supports perplexity as the essence of the bias and familiarity as its basis.