PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge

Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, Soroush Vosoughi

arXiv:2406.07791v9cs.CLcs.AI

TL;DR

LLM-as-a-Judge can scale evaluation, but position bias threatens its accuracy, fairness, and reliability. This study systematically measures the bias across pairwise and list-wise comparisons using three metrics and analyzes judge-, candidate-, and task-level factors. The findings show that position bias is consistent rather than random, varies across judges and tasks, and is strongly associated with answer-quality gaps.

  • Problem

    Position bias in LLM judges favors solution positions over content, undermining the accuracy, fairness, and reliability of automated evaluation.

  • Method

    The study evaluates position bias in pairwise and list-wise LLM judgments using Position Consistency, Preference Fairness, and Repetition Stability, then analyzes Judge-, Candidate-, and Task-level factors.

  • Results

    Position bias is not due to random variation, varies across judges and tasks, and is strongly influenced by differences in candidate answer quality.

  • Takeaways & Limitations

    Agreement patterns can identify easier and harder-to-judge instances, while dataset modifications may help study the distribution of judging difficulty.

  • Takeaways & Limitations

    The study covers limited model, task, prompting, and list-wise candidate settings, constraining the generalizability of its findings.

Abstract

from arXiv · show

LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.

1 Introduction

LLM-as-a-Judge offers scalable evaluation but position bias undermines evaluator accuracy, fairness, and reliability. This study systematically examines that bias across pairwise and list-wise comparisons, finding that it is structured, judge- and task-dependent, and linked to solution quality differences.

  • Human evaluation is costly and difficult to scale, motivating LLM-as-a-Judge as an automated alternative.
  • Position bias makes LLM judges favor prompt positions rather than evaluating solution content objectively.
  • The study introduces Preference Fairness and Repetition Stability alongside Position Consistency to measure position bias more comprehensively.Preference Fairness examines answer-direction preferences, while Repetition Stability tests whether judgments reflect consistent patterns rather than random variation.
  • Position bias is not random, varies in preference direction across tasks within a judge, and is significantly influenced by candidate answer-quality differences.
  • Agreement analysis indicates that instances with judge disagreement are harder to evaluate and more prone to position bias.The analysis suggests that datasets could be modified by adding instances that are easier or harder to judge.

2 Evaluation Settings & Definitions

The study evaluates position bias through repeated, reordered comparisons in pairwise and list-wise settings. It defines three metrics and analyzes Judge-, Candidate-, and Task-level factors, including answer-quality gaps and preference fairness.

  • Pairwise Comparison: Pairwise evaluation compares two solutions in a double-blind prompt and repeats the judgment after swapping their positions.The resulting judgment pair reveals whether the same solution is preferred independently of its prompt position.
  • List-wise Comparison: List-wise evaluation selects the best of three or more candidates across order permutations, with each candidate appearing in every position once.For p candidates, the design uses p permutations and also permits ties.
  • Evaluation Metrics: Repetition Stability measures the most frequent selection across repeated identical queries, ranging from near-random behavior to 1.0 for perfect stability.It establishes whether observed position bias reflects consistent evaluation patterns rather than random variations.
  • Evaluation Metrics: Position Consistency measures how often a judge selects the same winning solution after solution-order permutations.It is the ratio of consistent evaluation series to valid evaluations.
  • Evaluation Metrics: Preference Fairness measures directional preference for solution positions and combines prior counting approaches into a granular, scalable score.The metric treats position-consistent judgments as preference fair and extends to list-wise comparisons through a one-vs-all approach.
  • Factors Affecting Position Bias: The analysis groups potential causes of position bias into Judge-level, Candidate-level, and Task-level factors, including answer-quality gaps.The answer quality gap compares solution quality between one candidate and the others for the same question, while overall win rate also captures comparable-quality cases.

3 Experiment

The experiments evaluate 15 LLM judges on MTBench and DevBench using pairwise and list-wise settings, repeated trials, and regression analyses. Results show substantial variation in position bias across judges and tasks, with judge series, candidate identities, task categories, and output length contributing to the metrics.

  • Experimental Setup: 15 LLM judges from GPT, Claude, Gemini, and Llama series were evaluated on modified MTBench and DevBench benchmarks.
  • Experimental Setup: MTBench used two-option comparisons, DevBench used three-option comparisons, and list-wise experiments evaluated three-candidate lists with four representative judges.
  • Experimental Setup: Repetition-stability evaluation used repeated trials, while position-consistency and preference-fairness evaluations covered larger sets of instances.Each judge was evaluated on 576 MTBench repetition-stability instances and 432 DevBench instances.
  • Analysis: Bidirectional stepwise regression tested prompt lengths, answer-quality gaps, judge series, candidate identities, and task categories as predictors of Position Consistency and Preference Fairness.
  • Empirical Results: Position bias varies significantly across judges and tasks, and high Repetition Stability is required for meaningful Position Consistency and Preference Fairness measurements.
  • Empirical Results: Judge series, candidate identities, and task categories significantly affect Position Consistency and Preference Fairness, while average output length significantly predicts Preference Fairness.

4 Main Findings

Position bias varies significantly across judges and tasks, persists across pairwise and list-wise settings, and is not attributable to random variation. It is strongly associated with answer-quality gaps, while prompt-length effects are weak; agreement patterns further reveal uneven judging difficulty.

  • RS values above 0.85 for capable judges, with Claude-3.5-Sonnet, GPT-4, and Llama-3.3-70B exceeding 0.95 on both benchmarks.These results support treating their position-bias patterns as stable rather than random.
  • More capable judges maintain high consistency from pairwise to list-wise evaluation, whereas less capable judges are more sensitive to additional candidates.List-wise evaluation was conducted on four representative judges to assess scalability.
  • Position bias varies significantly by judge and task, with judge identities and task categories statistically significant predictors of PC and PF.GPT-4o is more position-consistent on coding tasks than on other tasks, while preference directions can also differ across datasets.
  • As the answer quality gap δq increases, judges generally become more position consistent and preference fair, although exceptions show that other factors also matter.The quality gap is quantified from overall win rates relative to the candidate-list baseline; PC follows significant parabolic patterns, while PF changes less strongly.
  • Prompt-length effects are weak: average task output length is significant only for PF and produces a minimal AIC change.The study examines question length, candidate-solution length, and total prompt length using stepwise regression.
  • More than half of MTBench instances have disagreement ≤3 among 15 judges, while fewer than 2% have disagreement ≥8.Consensus is generally easier on the former instances, whereas high-disagreement instances are likely especially difficult to judge.

6 Limitations

The study identifies limitations in model, data, prompting, and mitigation scope that constrain the generalizability and practical reach of its findings.

  • Scope and Generalizability: The evaluation covered 12 commercial judges for pairwise settings and 4 for list-wise settings across two benchmarks, with list-wise comparisons limited to three candidates.Only three Llama 3.1+ open-source models were assessed because earlier or smaller models lacked sufficient context windows.
  • Future Extensions: Alternative prompting techniques, more models and tasks, and larger list-wise candidate pools could improve the findings’ generalizability.The experiments used original benchmark prompt templates and relatively small model and candidate pools.
  • Data Accessibility: Closed-source architecture and parameter-size effects could not be analyzed directly, while evidence from only three Llama models is insufficient for broad open-source conclusions.The study instead grouped models using family properties as an approximation.
  • Analysis Design: The exploratory analyses were conducted post hoc after judgments were collected, leaving pre-judgment estimation or control of influencing factors for future work.Such methods could reduce computational costs and support proactive mitigation strategies.
  • Mitigation: The study evaluated and understood position bias rather than directly mitigating it.Further work is needed to maintain consistency and fairness when answer-quality gaps are minimal, where position bias is most pronounced.

A Reproducibility

The paper situates LLM-as-a-Judge as a scalable alternative to human evaluation but emphasizes that position bias remains difficult to eliminate, especially for inaccessible closed-source models.

  • Reproducibility: The paper’s reproducibility materials include API-based experiments costing approximately 3,000 USD and a code repository for reproducing the results.The repository URL is provided in the paper passage.
  • Reproducibility: LLM-as-a-Judge offers scalability and reproducibility advantages over human assessments and has been applied across diverse evaluative tasks.Applications include story generation, adversarial attacks, summarization, machine translation, and instruction following.
  • Reproducibility: Position bias favors prompt positions irrespective of content and is pervasive across domains and tasks, undermining evaluator reliability.Excluding inconsistent judgments or treating them as ties preserves practical evaluation but does not mitigate the underlying bias.
  • Reproducibility: Bootstrapping, split-and-merge methods, and multi-agent discussions can be costly, time-consuming, or insufficient to eliminate position bias.Mechanistic approaches modify model internals but are generally inapplicable to closed-source models whose internals are inaccessible.

B.3 Evaluating Position Bias Evaluation

Prior position-bias evaluation is more established for pairwise comparisons than list-wise settings, while systematic validation and factor analysis remain necessary for reliable interpretation.

  • Evaluation Gaps: Pairwise metrics assess judgment consistency under candidate-order changes and preference direction, but rigorous measurement in list-wise settings remains less explored.List-wise ranking can introduce computational costs and ranking-performance degradation.
  • Evaluation Gaps: Prior list-wise methods may become computationally expensive because the number of candidate pairs grows exponentially, or may suffer performance degradation from ranking complexity.These limitations motivate simpler best-of-n selection settings and more unified measurement.
  • Evaluation Gaps: A unified metric for preference direction is needed because traditional position-preference measurement becomes more complicated as list size increases.The paper frames this need around choosing the best item from an n-item list rather than exhaustively ranking the list.
  • Influencing Factors: Position bias may depend on judge attributes, task types, candidate lengths, and candidate qualities, but the effect of answer-quality gaps requires further investigation.Prior work reported a negative correlation between judgment conflict rate and score gap.
  • Reliability: Repeated judgments and mode aggregation can reduce concerns that observed position bias reflects random variation, but they increase computational expense.The paper presents this validation need as part of a broader framework for consistency, fairness, and reliability.
  • Instance-Level Agreement: Judges with similar average position-consistency and preference-fairness scores may still disagree substantially on individual instances.The analysis therefore examines pairwise judge agreement and variation in all judges’ choices per instance.
  • Figure 4: Figure 4 compares judge agreement with ties included and excluded, using brighter colors for higher mutual agreement, while panel (d) shows DevBench disagreement distributions.Panel (b) specifically excludes the tie option {C}.

C.1 Mutual Agreement & Familial Property

Mutual-agreement heatmaps reveal capability- and family-linked patterns in LLM judges’ choices, while cross-family and size differences produce weaker agreement.

  • Mutual Agreement: Mutual agreement is defined as the proportion of benchmark instances on which two LLM judges make identical choices.For MTBench, agreement is also computed without the tie option because tie-versus-solution choices represent less disagreement.
  • GPT Family: The GPT-4, GPT-4-Turbo, and GPT-4o series exceed 70% agreement with ties included and 85% without ties on MTBench.GPT-3.5-Turbo disagrees with the GPT-4 series and o1-mini on approximately 40% of instances.
  • Claude Family: Claude-3-Opus agrees highly with Claude-3.5-Sonnet and Claude-3-Sonnet, whereas Claude-3.5-Sonnet and Claude-3-Sonnet lack significantly high agreement.The passage associates these patterns with similar capabilities, model structure, and the upgrade from series 3 to 3.5.
  • Gemini Family: Gemini judges show relatively low mutual agreement, although Gemini-1.5-Pro aligns more closely with capable models such as the GPT-4 series and Claude-3-Opus.The passage characterizes familial structure as minimal among Gemini models.
  • Llama Family: Llama-3.3-70B and Llama-405B agree highly with capable family members and GPT models, while Llama-3.1-8B does not agree strongly with them.The pattern distinguishes larger, more capable Llama judges from the smaller model.
  • Familial Property: Judges are primarily grouped by judging capability, with comparable models within the same family sharing more agreement than models across families.The paper suggests analogous model sizes, training data, and strategies may contribute to these familial patterns.
  • Interpretation: Comparisons between judges from different groups can reveal distinct positional biases and enrich understanding of the phenomenon.The analysis treats mutual-agreement groupings as informative beyond average position-consistency and preference-fairness scores.

C.2 Disagreement & Benchmark Design Insight

Disagreement analysis distinguishes easier evaluation instances, where judges often agree, from harder instances, where disagreement and position bias are more likely. These patterns support benchmark difficulty control and selective use of majority voting.

  • Disagreement Analysis: At least 75% of judges reached consensus on more than half of the instances in both MTBench and DevBench.Such instances are likely easier to evaluate, and majority voting enhances judgment reliability.
  • Disagreement Analysis: Instances with the highest disagreement are likely difficult to evaluate and more prone to position bias.They occupy less than 5% of instances on each benchmark, so majority voting could be useful for over 95% of evaluations.
  • Benchmark Design Insight: Disagreement values provide a rough proxy for judging difficulty, with the benchmarks showing a smoothly increasing difficulty curve.Except for the highest-disagreement instances, counts across disagreement levels do not vary substantially.
  • Benchmark Design Insight: Disagreement analysis can help benchmark designers control evaluation difficulty by managing the number of difficult or trivial instances.It can also help identify cases where majority voting is likely to provide reliable judgments without direct human comparison.

D More Results of Position Bias and Answer Quality Gap Measurement

Additional results show that position consistency and preference fairness vary by judge and task, while their relationship with answer quality gaps differs across judges. The study also uses AIC-based regression to identify relevant predictors.

  • More Evaluation Results: Position bias varies across judges and tasks, while high repetition stability indicates that these biases are systematic rather than random.Additional open-source-model results reinforce these observations across MTBench and DevBench.
  • MTBench: A larger answer quality gap generally leads to better position consistency across all individual judges.The relationship appears as a parabolic shape in Figure 5(a).
  • MTBench: Preference fairness is more judge-dependent than position consistency, with some judges preserving fairness regardless of the answer quality gap.Claude-3-opus and Claude-3-sonnet show conspicuous parabolic patterns, whereas GPT-4 and GPT-3.5 show nearly linear curves.
  • DevBench: Position consistency and preference fairness vary significantly across judges and tasks on DevBench.GPT-4 is generally strong in position consistency, but other judges surpass it on selected architecture-design and UML-class evaluations.
  • AIC-Based Regression: Bidirectional stepwise regression iteratively adds or removes predictors according to AIC changes until neither operation lowers AIC.The procedure balances model fit and complexity.

E.2 Test results

Stepwise regression identifies judge series, candidate identities, and task categories as significant contributors to both position consistency and preference fairness. Output length contributes to preference fairness, whereas task-output-length dependence is minimal overall.

  • Test Results: The final stepwise models are reported separately for Position Consistency and Preference Fairness across MTBench, DevBench, and their integrated set.Tables 4–9 record the corresponding model-selection results.
  • Test Results: LLM judge series, candidate identities, and task categories significantly impact position consistency and preference fairness across the benchmark analyses.The regression evaluates input, output, and prompt lengths, answer quality gap, judges, candidates, and task categories as predictors.
  • Test Results: The dependence of position bias on task output length is minimal despite output length being considered among the tested predictors.Removing this variable produces only a minimal AIC change consistently across both benchmarks individually and combined.

F Experiment Settings

The study evaluates position bias using MTBench and DevBench, selected for public information, broad model and task coverage, and validated judging models. It uses multiple LLM judges, reference answers, benchmark tasks, and standardized pairwise prompts.

  • Benchmark Selection: MTBench and DevBench were selected because their benchmark information is public, their models and tasks are diverse, and prior human evaluations support validated judges.These properties also make benchmark modifications convenient and support comprehensive evaluation.
  • Judges: The study uses seven GPT, four Claude, and three Gemini models as LLM judges.The paper specifies the API versions for the principal judge models.
  • Answer-Generating Models: Vicuna-13b-v1.3 is the MTBench reference model, while human is the DevBench reference answer-generating model.The references provide baseline response quality and a broad expected quality gap.
  • Tasks: The experiments cover eight MTBench tasks and detailed DevBench task categories, including UML class and UML sequence evaluations.The task set follows the original benchmarks, with DevBench general metrics separated into detailed tasks.
  • Prompt and Evaluation Procedure: Pairwise prompts present two candidate solutions to a blinded judge, then repeat the evaluation after swapping their positions.The prompts include judging instructions, question-and-answer placeholders, an output format, and chain-of-thought reasoning requirements.
Loading 2406.07791v9…