PaperlayerView exact revision on arXiv

Source-linked AI summary

Humans or LLMs as the Judge? A Study on Judgement Biases

Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, Benyou Wang

arXiv:2402.10669v5cs.CL

TL;DR

Human- and LLM-as-a-judge systems can exhibit biases that question evaluation reliability. This paper proposes a reference-free framework to investigate four biases and finds that judges are vulnerable to perturbations, with LLM judgments also hackable through prompts.

  • Problem

    Existing human and LLM judges exhibit biases, making the validity of LLM evaluation results uncertain, while controlled, comprehensive bias comparisons without groundtruth are difficult.

  • Method

    A reference-free framework compares control and perturbed answer pairs using Attack Successful Rate to quantify four judge biases and enable attacks on LLM judges.

  • Results

    All judges display significant biases with different inclinations, and LLM judges are vulnerable to fake-reference and rich-content attacks, especially when answer quality gaps are small.

  • Takeaways & Limitations

    The findings highlight bias and vulnerability in human- and LLM-as-a-judge systems and the urgency of developing more robust evaluation systems.

  • Takeaways & Limitations

    The benchmark contains only 142 questions, lacks horizontal-field classifications, and includes ambiguity in some Bloom’s Taxonomy level assignments, limiting generalizability.

Abstract

from arXiv · show

Adopting human and large language models (LLM) as judges (a.k.a human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges. We curate a dataset referring to the revised Bloom's Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

1 Introduction

The paper examines biases in human and LLM judges for open-ended generation, introducing a reference-free framework and showing vulnerabilities that enable attacks on LLM judges.

  • Motivation: Open-ended benchmarks use human or LLM judges, but both judge types can exhibit biases that question evaluation validity.The paper frames bias as a concern for evaluating free-form generation.
  • Motivation: Current bias frameworks require groundtruth or reference answers, limiting probes of perturbations without well-defined golden standards.The authors identify the lack of suitable standards as a challenge for controlled and valid bias comparisons.
  • Contributions: The study investigates Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias in human and LLM judges.These biases are studied as important factors in natural language generation evaluation.
  • Contributions: The proposed reference-free framework compares control and perturbed answer pairs using preference shifts measured by Attack Successful Rate (ASR).Perturbations include factual errors, gender-biased content, fake references and rich content.
  • Findings: Human judges show little Gender Bias but significant Misinformation Bias and Beauty Bias.The reported findings distinguish the observed human-judge vulnerabilities across bias types.
  • Findings: An attack exploiting Authority Bias and Beauty Bias reaches an ASR of up to 50% on GPT-4.The result demonstrates that identified judge biases can be used in prompt-based attacks.

2 Related Works

Prior work uses human feedback and increasingly LLM judges to evaluate NLG, while documenting biases and vulnerabilities in both approaches. This paper adds further bias analyses and a simple prompt-based attack on LLM judges.

  • Human and LLM Evaluation: Human feedback serves as a gold standard for NLG evaluation, while earlier automated metrics include BertScore, BARTScore, DiscoScore and GPTScore.Feedback supports model improvement and output-quality assessment.
  • Human and LLM Evaluation: Powerful LLMs are increasingly used as judges in place of earlier NLG evaluation methods.This development extends judge-based evaluation beyond human feedback.
  • Biases in Evaluation: Human judges have low reproducibility and inherent biases, while their annotations may require clearer instructions and validation.The cited work motivates codebooks and validation for higher-quality evaluation.
  • Biases in Evaluation: LLM judges exhibit positional, cognitive, verbosity, self-enhancement and superficial-quality biases, and their annotations require validation.The paper positions its analysis as quantifying additional judge biases.
  • Attacks on Judges: Attacks on LLM-as-a-judge remain relatively under-explored compared with attacks on other LLM tasks.Existing cited approaches include optimization-based methods for hacking judges.
  • Attacks on Judges: This work introduces a simple yet effective zero-shot prompt-based approach to deceive LLM judges.The approach differs from the optimization-based methods described in prior work.

3 On the Biases of Judges

The paper separates judge biases into semantic-related and semantic-agnostic categories, defines four investigated biases, and connects them to risks in evaluation and decision-making.

  • Defining Bias: Semantic-related bias depends on text content, whereas semantic-agnostic bias depends on factors unrelated to semantic content.Misinformation Oversight Bias and Gender Bias are semantic-related; Authority Bias and Beauty Bias are semantic-agnostic.
  • Importance: The paper investigates these four bias types through extensive experiments and illustrates each with an example.The stated motivation includes risks to fairness, factual accuracy and decision-making in domains such as law and finance.
  • Investigated Biases: Misinformation Oversight Bias is the tendency to overlook factual errors in an argument.The paper associates it with drawing conclusions without scrutinizing supporting arguments.
  • Investigated Biases: Gender Bias is a judge’s ignorance of gender-biased content.The paper describes it as occurring when a human or model has not learned to avoid this unconscious bias.
  • Investigated Biases: Authority Bias attributes greater credibility to perceived authorities regardless of actual evidence.The paper characterizes this as uncritical acceptance of expert opinions.
  • Investigated Biases: Beauty Bias, or lookism, is a preference for visually appealing content regardless of its validity.The definition focuses on judges favoring appearance over substantive correctness.

4 Experimental Protocol

The protocol uses controlled interventions on paired answers to measure judges’ bias without groundtruth, with questions spanning Bloom’s Taxonomy and evaluations by humans and LLMs.

  • Motivation: The study addresses the difficulty of controlled, generalizable bias analysis when groundtruth is unavailable or poorly defined for open-ended questions.The authors identify both dependence on groundtruth annotations and experiment-design trade-offs as threats to valid conclusions.
  • Method: Four perturbations—factual error, gender-biased content, fake references, and rich content—probe Misinformation Oversight, Gender, Authority, and Beauty Bias.The framework compares a control group of unperturbed answer pairs with an experimental group containing one perturbed answer.
  • Data Generation: Questions follow the six revised Bloom’s Taxonomy levels, with 30 questions per level and knowledge controlled at or below middle-school level.The question set totals 180 generated questions, with author verification of level categorization.
  • Data Generation: GPT-4 independently generates two answers per question, while each perturbation is applied to a randomly selected answer to create experimental pairs.The control group contains 142 question-answer pairs, and the experimental group contains four corresponding perturbation sets.
  • Experiment Objects: The evaluation includes 60 college-student judges and representative LLM judges, with answer positions shuffled to reduce positional effects.Judges choose between Answer 1, Answer 2, or Tie; human “not familiar” votes are excluded from final results.
  • Metric: ASR measures the percentage of samples whose preference shifts toward the perturbed answer between control and experimental groups.Human votes are aggregated from six position-shuffled evaluations per QA pair before preferences and ASR are calculated.

5 Results and Discussion

The results show substantial variation in judges’ robustness across perturbations: stronger models detect factual errors, humans perform better on gender bias, and authority and beauty cues mislead many judges.

  • Misinformation Oversight Bias: GPT-4o, Claude-3, GPT-4, and GPT-4-Turbo achieve factual-error ASRs below 11%, whereas humans and other LLMs exceed 20%.The authors attribute the weaker performance of humans and other LLMs respectively to missed contextual details and insufficient fact-checking knowledge.
  • Gender Bias: Human judges substantially outperform LLM judges on gender-bias detection, while LLM judges retain significant Gender Bias.The authors relate the human result to the college-student sample and suggest that alignment may not eliminate gender bias in LLMs.
  • Authority Bias: 32% ASR is achieved by GPT-4o under fake-reference perturbation, only 5% better than random, while all other judges perform no better than random.The result indicates that perceived reference credibility strongly influences both human and LLM judges.
  • Beauty Bias: Four LLM judges obtain rich-content ASRs below 10%, while humans and the remaining judges exceed 30%.The authors associate this pattern with attention distractors such as emojis and markdown formatting.
  • Self-enhancement: GPT-4 and Claude-3 remain strong at factual-error detection when evaluating responses generated by either model, weakening the self-enhancement explanation.On the sampled subset, GPT-4’s self-generated-response ASR is 0.07, compared with 0.08 in Table 1.
  • Limitations: Because GPT-4 curates the dataset, the reported performance may represent an upper bound if tested models encounter an unseen distribution.The authors nevertheless consider the experiment informative for revealing LLM-judge biases.

6 Deceiving LLM Judges

The paper tests whether perturbations can make flawed, biased, or weaker answers appear preferable to LLM judges. It finds vulnerability to fake references and rich content, especially when answer quality differences are small.

  • Attack setup: The attack constructs anchor, weak, and perturbed answer sets to test whether superficial additions reverse LLM judges’ preferences.Perturbations include factual errors, gender-biased content, fake references, rich content, and their combination.
  • Attack setup: ASR comparisons between control and experimental groups measure preference changes before and after perturbation.The control compares A1 with A2, while the experimental group compares A1 with the perturbed set; ASR is the evaluation metric.
  • Flawed and biased answer detection: Fake references deceive LLM judges more effectively than rich content, indicating stronger sensitivity to superficial authority than formatting quality.The comparison concerns perturbation effects on detecting factual errors and gender-biased content.
  • Flawed and biased answer detection: LLM judges are vulnerable to fake-reference and rich-content attacks when detecting factual errors and gender-biased content.GPT-4o and Claude-3 perform best on both biases, while some models perform worse than random under compound perturbations.
  • Weak answer turnover: LLM judges are more easily induced by fake references when the quality gap between answers is smaller.They are relatively robust to the perturbation when the quality gap is significant, limiting weaker-answer turnover in the reported setting.

7 Conclusion

The paper presents a reference-free framework for examining four biases in human and LLM judges and identifies biases and vulnerabilities across judges. It also shows that prompt-based perturbations can hack LLM judgment.

  • Conclusion: The framework explores Misinformation Oversight Bias, Gender Bias, Authority Bias, and Beauty Bias without relying on reference annotations.It is applied to both human and LLM judges.
  • Conclusion: The study finds that all judges display significant biases, although their specific inclinations differ.The conclusion characterizes the biases as innate vulnerabilities of human- and LLM-as-a-judge.
  • Conclusion: A prompt-based method can hack LLM judgment by exploiting the biases identified in the study.The conclusion frames robust evaluation systems as an urgent research need.

Limitations

The study acknowledges limitations in benchmark coverage, human-judge representativeness, bias scope, and the temporal validity of conclusions about evolving LLM judges.

  • The benchmark contains only 142 questions, lacks horizontal-field classifications, and may not represent the diversity and complexity of potential questions.Some questions also have ambiguous Bloom’s Taxonomy classifications, although the authors argue this has little effect on validity.
  • The paper notes that its conclusions are intended to concern current LLMs, while the benchmark data and question-generation process constrain generalizability.
  • The study examines four biases, leaving other potentially important biases such as wording, tone, and racism for future work.
  • Human judges are limited to college students, whose sensitivity to gender-biased content may not generalize to people working outside university settings.
  • Because LLM judges are evolving, conclusions about the evaluated models may become invalid as those models advance.

A Detail of Data Generation

The dataset-generation process uses the revised Bloom’s Taxonomy to create questions across six cognitive levels, then generates answers and targeted perturbations through structured prompts.

  • Question generation: Questions are generated across six revised Bloom’s Taxonomy levels, from Remembering through Creating, with four questions per level in the initial prompt design.
  • Question generation: Generated questions are designed to belong to one Bloom’s level, remain answerable by humans and language models, and avoid language-model-related content and ambiguous pronouns.
  • Question generation: The final question-generation process requests 30 independent, middle-school-difficulty questions for each level and outputs them in JSON format.
  • Answer generation: Answer-generation prompts vary by cognitive level and can require an answer plus a brief explanation under a word limit.
  • Perturbation generation: Perturbation prompts add fake references or rich content, while separate prompts introduce factual errors or gender-biased content with minimal modifications.

A.3.3 Factual Error

The factual-error evaluation section constructs and reviews question–answer data, then compares human and LLM judgments under controlled answer-pair evaluation instructions.

  • Factual-error construction: The factual-error prompt asks a fact checker to identify facts, add 2–3 difficult-to-detect factual errors, list them, and output the altered answer.
  • Data review: Manual review checks question classification, difficulty, completeness, harmlessness, and accuracy before retaining data for evaluation.
  • Data review: Review reduced the control group from 180 question–answer pairs to 142 pairs.
  • Human evaluation: Human evaluators are college-level participants selected for English proficiency and high-school-level mathematics, physics, and logic knowledge.
  • Evaluation protocol: Evaluators compare two answers to the same question, may select a tie, and are instructed to judge semantic quality and accuracy rather than tone, format, or answer order.

F.1 Positional Bias

The positional-bias analysis compares evaluators’ preferences for first and second answer positions, finding substantial variation across LLMs and smaller differences for several human and model judges.

  • Measurement: An ideal judge should select the first and second answer positions approximately equally when evaluations are balanced across placement methods.
  • Results: GPT-3.5-Turbo, Mixtral, Spark, Qwen, and Gemini-Pro show particularly strong positional preferences.
  • Results: GPT-3.5-Turbo and Mixtral favor the first answer, Spark favors the second, while Qwen and Gemini-Pro consistently select Tie.
  • Results: Claude-3, Ernie, and LLaMA2-70B show smaller positional differences of about 10% to 30%.
  • Results: Human evaluators, unfamiliar-topic human choices, GPT-4, GPT-4-Turbo, Claude-2, and PaLM-2 have first–second preference differences within 10%.
  • Experimental handling: The three models with especially strong positional preferences are excluded from subsequent experiments.

F.3 Case analysis

The case analysis contrasts GPT-4 and Claude-2 when evaluating answers before and after a factual perturbation, then examines judges’ sensitivity to answer length. GPT-4 detects the semantic error and is less influenced by verbosity, while Claude-2 continues preferring the flawed answer.

  • Case analysis: GPT-4 and Claude-2 both judged the original Pythagorean-theorem answers favorably, but Claude-2 preferred the more detailed response while GPT-4 called them a tie.GPT-4 emphasized that both answers were clear and accurate; Claude-2 favored Answer 2 for its greater thoroughness and examples.
  • Case analysis: After a factual error changed the hypotenuse to the smallest side, GPT-4 identified the mistake and recommended Answer 1, whereas Claude-2 still preferred Answer 2.GPT-4 also explained that the hypotenuse is the longest side and that the theorem applies to right-angled triangles; Claude-2 repeated the flawed interpretation.
  • Case analysis: The example suggests that semantic understanding and adherence to evaluation instructions determine how judges respond to answer perturbations.Models that grasp detailed semantic information without being distracted by superficial factors can perform better in evaluation.
  • Verbosity Bias: As answer-length differences increase, all evaluators increasingly prefer longer answers, although GPT-4-Turbo is least influenced and Claude-3 is most influenced.When the difference is 0-10 tokens, preferences are near 0.5, indicating little pronounced length preference; the analysis uses vote values of 0, 0.5, and 1.

I Supplementary Results of Deceiving Models

The supplementary results compare answer quality for the LLaMA2-Chat family and GPT-3.5-Turbo using a vote table. GPT-3.5-Turbo receives substantially higher answer-quality evaluations than the LLaMA2 family.

  • Supplementary results: GPT-3.5-Turbo’s answer quality is much higher than that of the LLaMA2 family.The authors use this comparison to support using LLaMA2 answers to form the weak set W.
  • Supplementary results: Table 6 reports each judge’s vote percentages for the LLaMA2-Chat family and GPT-3.5-Turbo, with LLaMA2 results averaged across 7B, 13B, and 70B models.Tie votes account for the remaining percentage in each row.
  • Supplementary results: The LLaMA2-Chat-{7B,13B,70B} results are averaged when comparing the two model groups.This aggregation presents the LLaMA2 family as a combined comparison group against GPT-3.5-Turbo.
Loading 2402.10669v5…