PaperlayerView exact revision on arXiv

Source-linked AI summary

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, Xiangliang Zhang

arXiv:2410.02736v2cs.CLcs.AI

TL;DR

LLM trustworthiness matters because models can make erroneous judgments based on statistical patterns. This paper presents CALM, an automated framework for quantifying 12 biases in LLM-as-a-Judge, and finds that models can perform reliably on specific tasks while broader judge use still needs improvement.

  • Problem

    Ensuring LLM trustworthiness is important because LLMs may make erroneous judgments based on specific statistical patterns.

    Source: §A.2 · p. 15

  • Method

    CALM is an automated framework that evaluates 12 bias types using automated bias injection and metrics including Robustness Rate and Consistency Rate.

    Source: §2 · p. 2, §2.3 · p. 6, §6 · p. 10

  • Results

    Models may reliably judge specific tasks, but significant room for improvement remains in the broader use of LLMs as judges, with bias effects differing across models.

    Source: §4.1 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    CALM can evaluate future LLM-based judge solutions against higher standards of bias mitigation, while application should account for task-specific bias and model differences.

    Source: §4.1 · p. 7, §6 · p. 10, §5 · p. 10

    Some question sets and bias-related responses may contain NSFW content, requiring cautious and responsible research use.

    Source: ETHICAL CONSIDERATION · p. 11

Abstract

from arXiv · show

LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework-CALM-which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

2 PROPOSED FRAMEWORK: CALM

CALM assesses LLM-as-a-Judge bias by applying principle-guided perturbations to judged content or instructions and comparing resulting judgments. Its experiments show that vulnerabilities vary across models and tasks, including dataset-dependent, positional, length, self-enhancement, irrelevant-content, authority, emotional, identity, and CoT effects.

Framework components

  • CALM combines 12 bias categories, diverse datasets, and task-specific metrics to evaluate reliability in LLM-as-a-Judge systems.The framework supports pairwise comparison and scoring judgments across evaluation aspects.Source: §2 · pp. 2–3

Automated perturbation

  • Principle-guided perturbations modify responses or instructions while preserving correctness and meaning, enabling bias detection through judgment consistency.The judge compares original and modified evaluations; differing outputs indicate bias, while matching outputs indicate robustness.Source: §2.1 · p. 3, §2.3 · p. 6

Metrics

  • Robustness and consistency rates quantify whether judgments remain stable after bias injection or across repeated unperturbed evaluations.CALM also introduces an accuracy metric for CoT bias to measure changes in correct judgments after step-by-step reasoning.Source: §2.3 · p. 6

Main results

  • Bias effects differ across models and tasks: Claude-3.5 is generally most resilient, yet GPT-4-Turbo can be inconsistent on emotional responses while ChatGPT is more stable.The results caution against assuming that greater model capability guarantees greater reliability for every bias or task.Source: §4.1 · p. 7
  • Position bias intensifies with more answer candidates, with most models falling below 0.5 robustness when evaluating three or four options.The authors suggest selecting models with stronger robustness rates or randomizing answer order.Source: §4.2 · p. 8

5 DISCUSSION

The discussion distinguishes explicit from implicit bias and proposes practical measures for detecting and mitigating bias in LLM-as-a-Judge. CALM’s findings support prompt safeguards and automated detection, while broader reliability improvements remain necessary.

  • Explicit biases are openly stated preferences, whereas implicit biases influence judgments without being acknowledged in the reasoning.Authority bias is given as an example of explicit bias.Source: §5 · p. 10
  • The authors recommend carefully constructed prompts and advanced reasoning strategies to reduce bias interference.Protective phrases can instruct models to disregard identity information, while step-by-step reasoning can guide judgments.Source: §5 · p. 10
  • Prompt-injection safeguards are recommended to prevent biased information embedded in prompts from influencing judgments.The proposed safeguards target external attempts to introduce bias into the judging process.Source: §5 · p. 10
  • A prompt-based detection mechanism can identify potential biases in judging templates before evaluation begins.Its effectiveness varies by bias type, but the authors report promise in uncovering a majority of biases.Source: §5 · p. 10
  • CALM evaluates 12 bias types through automated bias injection and qualification, providing an objective and scalable assessment approach.The conclusion states that models may reliably judge specific tasks, while broader use still has significant room for improvement.Source: §6 · p. 10

ETHICAL CONSIDERATION

The paper cautions that some question sets and bias-related responses contain NSFW content. Although the data were manually reviewed and curated, applications should follow ethical guidelines and consider societal impacts.

  • Some question sets and bias-related responses may contain NSFW content.The authors state that the data were manually reviewed and curated for research appropriateness.Source: ETHICAL CONSIDERATION · p. 11
  • Applications or extensions should be conducted responsibly with consideration for ethical guidelines and potential societal impacts.Source: ETHICAL CONSIDERATION · p. 11

A.1 LLM-AS-A-JUDGE

LLM-as-a-Judge uses language models to evaluate responses without requiring reference texts. This approach has shown performance on open-ended questions that highly matches human preference, while fairness remains an active research focus.

  • LLM-as-a-Judge evaluates responses without requiring reference texts.Source: §A.1 · p. 15
  • The method has demonstrated performance on open-ended questions that highly matches human preference.Source: §A.1 · p. 15
  • Recent research has examined the fairness of LLM-as-a-Judge.Source: §A.1 · p. 15

A.2 FAIRNESS IN TRUSTWORTHY LLMS

Fairness is an ethical principle requiring LLMs to avoid biased or discriminatory outcomes and treat users and groups equitably. Because training-data imbalance can produce demographic biases, fairness affects the trustworthiness of LLM-as-a-Judge.

  • Stereotypes and erroneous judgments based on statistical patterns highlight the importance of fairness in evaluating LLMs.Source: §A.2 · pp. 15–16
  • Fairness requires LLMs to avoid biased or discriminatory outcomes and treat users and groups equitably.Source: §A.2 · p. 16
  • Pre-training data imbalance can create training imbalances and biases against demographic groups.The paper names gender, age, and language as examples of affected attributes.Source: §A.2 · p. 16
  • Fairness in LLMs significantly affects the trustworthiness of LLM-as-a-Judge.Source: §A.2 · p. 16

A.3 BIASES IN LLM-AS-A-JUDGE APPLICATION

Prior research identifies multiple cognitive biases that can influence LLM-as-a-Judge evaluations, including positional, verbosity, self-enhancement, order, compassion-fade, egocentric, salience, bandwagon-effect, and attentional biases.

  • Prior studies identify position, verbosity, and self-enhancement biases in LLM-as-a-Judge evaluations.Source: §A.3 · p. 16
  • Other reported biases include order, compassion-fade, egocentric, salience, bandwagon-effect, and attentional biases.Source: §A.3 · p. 16

B DETAILS OF BIAS TYPES

The paper defines twelve bias types affecting LLM-as-a-Judge decisions and specifies how each can alter evaluations across responses, prompts, identities, references, and reasoning content.

  • Position bias favors responses based on their input position, and its evaluation extends to settings with more than two responses.Source: B DETAILS OF BIAS TYPES · p. 16
  • Verbosity bias reflects potential preferences for longer responses, while compassion-fade bias concerns model-name anonymity in judgments.Source: B DETAILS OF BIAS TYPES · p. 16
  • Bandwagon-effect bias concerns majority opinions, distraction bias concerns irrelevant content, and fallacy-oversight bias concerns recognizing logical errors.Source: B DETAILS OF BIAS TYPES · p. 16
  • Authority bias captures influence from authoritative references, while sentiment bias captures preferences for emotional tones.Source: B DETAILS OF BIAS TYPES · p. 16
  • Diversity bias concerns identity markers; CoT bias concerns explicit reasoning steps; self-enhancement bias concerns favoring a model’s own outputs.Source: B DETAILS OF BIAS TYPES · p. 16
  • Refinement-aware bias concerns different scores for original, refined, and history-exposed answers.Source: B DETAILS OF BIAS TYPES · p. 16

C DETAILS OF BIAS EVALUATION

The evaluation systematically perturbs answers or system instructions to test whether judgments change under controlled bias-related modifications across multiple evaluation settings.

  • Position bias is tested by rotationally permuting answer order for evaluations containing two, three, or four answers.Source: C DETAILS OF BIAS EVALUATION · p. 17
  • Verbosity bias is tested by lengthening worse answers while preserving essential content, then comparing judgments before and after expansion.Source: C DETAILS OF BIAS EVALUATION · p. 17
  • Self-enhancement bias compares each model’s scores for its own answers with scores for answers generated by other models without disclosed authorship.Source: C DETAILS OF BIAS EVALUATION · p. 17
  • Compassion-fade bias compares judgments when model identities are disclosed against judgments under anonymized conditions.Source: C DETAILS OF BIAS EVALUATION · p. 17
  • Bandwagon and distraction tests insert majority-opinion claims or meaningless system statements and measure resulting judgment robustness.Source: C DETAILS OF BIAS EVALUATION · p. 17
  • Fallacy, authority, sentiment, diversity, CoT, and refinement-aware tests modify logic, references, emotions, identities, reasoning prompts, or answer histories before re-evaluation.Source: C DETAILS OF BIAS EVALUATION · p. 17

D DETAILED RESULTS

The detailed results report bias-specific robustness, accuracy, score, and error-rate analyses across judge models, with findings varying by bias and model.

  • Robustness rates are reported for position bias in pairwise and multiple-answer comparisons, and for verbosity across answer-length ratios.Source: D DETAILED RESULTS · p. 18
  • Self-enhancement results use Z-score-normalized score heat maps and the ErrorRateSE metric for each judge model.Source: D DETAILED RESULTS · p. 18
  • Bandwagon results examine varying public-opinion percentages, whose influence differs across models without a statistical pattern.Source: D DETAILED RESULTS · p. 18
  • Distraction results report robustness after irrelevant content is added to both high-quality and low-quality answers.Source: D DETAILED RESULTS · p. 18
  • Authority results show that quote and book-type fake references strongly influence most models.Source: D DETAILED RESULTS · p. 18
  • Most models do not favor emotionally charged expressions, while CoT and refinement-aware analyses report accuracy and ErrorRateRA metrics.Source: D DETAILED RESULTS · p. 18

E CASE STUDY

The case-study section enumerates concrete manifestations of bias using Figures 9–12 and analyzes them in detail.

  • Figures 9–12 present case studies of sentiment, refinement-aware, authority, and bandwagon-effect bias.Source: E CASE STUDY · p. 18
  • The case studies illustrate actual manifestations of bias in LLM-as-a-Judge evaluations.Source: E CASE STUDY · p. 18
  • The section follows the case studies with detailed analysis of these bias manifestations.Source: E CASE STUDY · p. 18

F PROMPT TEMPLATE

This section documents the experimental materials and templates used to evaluate bias across models, datasets, comparison formats, and perturbation types.

Related papers

Loading 2410.02736v2…