Benchmarking Cognitive Biases in Large Language Models as Evaluators

Ryan Koo; Minhwa Lee; Vipul Raheja; Jong Inn Park; Zae Myung Kim; Dongyeop Kang

Source-linked AI summary

Benchmarking Cognitive Biases in Large Language Models as Evaluators

Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, Dongyeop Kang

arXiv:2309.17012v3Sep 29, 2023cs.CLcs.AIcs.LG

TL;DR

LLMs are increasingly used as automatic evaluators, but their cognitive biases raise questions about whether their judgments are reliable and aligned with humans. COBBLER benchmarks six such biases across LLM evaluators and compares their rankings with human preferences. Most models show substantial bias, and the reported average human–machine agreement is 44%, leading the authors to conclude that they remain unsuitable as fair, reliable automatic evaluators.

Problem
Existing task-specific benchmarks do not adequately measure the quality of generated text in the wild, while LLMs’ use as evaluators raises questions about unbiased judgment.
Method
COBBLER generates responses to 50 question-answering instructions from 16 LLMs, performs pairwise evaluations, and tests six implicit and induced cognitive biases.
Results
Most LLMs exhibit cognitive biases, and LLM evaluations achieve 44% average agreement with human preferences.
Takeaways & Limitations
The results indicate that most LLMs are still unsuitable as fair and reliable automatic evaluators aligned with human preferences.
Takeaways & Limitations
Some models have very low valid response rates, and the human judgment study has subpar inter-annotator agreement.

Abstract

from arXiv · show

Large Language Models are cognitively biased judges. Large Language Models (LLMs) have recently been shown to be effective as automatic evaluators with simple prompting and in-context learning. In this work, we assemble 15 LLMs of four different size ranges and evaluate their output responses by preference ranking from the other LLMs as evaluators, such as System Star is better than System Square. We then evaluate the quality of ranking outputs introducing the Cognitive Bias Benchmark for LLMs as Evaluators (CoBBLEr), a benchmark to measure six different cognitive biases in LLM evaluation outputs, such as the Egocentric bias where a model prefers to rank its own outputs highly in evaluation. We find that LLMs are biased text quality evaluators, exhibiting strong indications on our bias benchmark (average of 40% of comparisons across all models) within each of their evaluations that question their robustness as evaluators. Furthermore, we examine the correlation between human and machine preferences and calculate the average Rank-Biased Overlap (RBO) score to be 49.6%, indicating that machine preferences are misaligned with humans. According to our findings, LLMs may still be unable to be utilized for automatic annotation aligned with human preferences. Our project page is at: https://minnesotanlp.github.io/cobbler.

1 Introduction

COBBLER benchmarks whether LLMs can act as unbiased automatic evaluators by testing cognitive biases and comparing machine preferences with human judgment. The study finds substantial evaluator bias and low agreement with human preferences.

COBBLER evaluates LLMs’ suitability as unbiased automatic evaluators in question-answering settings.
The benchmark examines six cognitive evaluation biases, including biases that can be extracted implicitly and those induced through prompt modifications.
Most evaluated models exhibit cognitive biases that may compromise the credibility of their evaluator role and negatively affect evaluation quality.
44% average agreement was found between LLM evaluations and human preferences.
The benchmark is intended to estimate gaps between model and human judgment and support future efforts to mitigate evaluator biases.

2 Related Work

Prior work uses LLMs for many downstream tasks and as automatic evaluators, while existing benchmarks primarily measure general model performance. COBBLER extends this literature with a broader evaluation of cognitive biases across models and question-answering settings.

LLMs have been applied to machine translation, summarization, code generation, writing assistance, and factual consistency evaluation.
Existing benchmarks such as LM-EVAL-HARNESS, MMLU, HELM, and BIG-BENCH focus on general LLM performance rather than evaluator capabilities.
COBBLER differs from related evaluator benchmarks by covering more popular language models, focusing on question answering, and comprehensively analyzing cognitive biases.

3 COBBLER: Cognitive Bias Benchmark for LLMs as Evaluators

COBBLER selects broadly applicable, impartial, memoryless bias tests and divides them into implicit and induced categories. The benchmark operationalizes six evaluator biases through paired-response prompts, reordered responses, recognizable names, length differences, false statistics, and irrelevant information.

Biases are selected for general applicability, impartiality, and memorylessness across evaluation settings.
Implicit biases are measured from ordinary pairwise evaluation prompts, whereas induced biases require added information or prompt modifications.
3.1 Implicit Biases: Order Bias is detected when an evaluator selects the first or last response across both orderings of the same pair.
3.1 Implicit Biases: Compassion Fade (Naming) measures whether recognizable model names change evaluations relative to anonymous aliases.
3.1 Implicit Biases: Egocentric Bias (Self-Preference) is measured when an evaluator prefers its own response over other models’ responses, despite possible genuine quality differences.
3.1 Implicit Biases: Salience Bias tests whether evaluators systematically favor shorter or longer responses rather than splitting evenly by length.
Induced tests measure susceptibility to a false majority statistic in Bandwagon Effect and irrelevant model information in Attentional Bias.

4 Experiment Setup

The study benchmarks cognitive biases in LLM evaluators using QA responses generated by 16 models and pairwise comparisons across model outputs. It also compares machine rankings with human preferences and tests bias patterns against random-selection thresholds.

Evaluation Framework: 16 models are grouped across four parameter-size ranges, spanning models larger than 100B to smaller than 10B parameters.The lineup includes both open- and closed-source systems selected from leaderboard, API-based, and recent open-source models.
Pairwise Evaluation: Each evaluator ranks anonymized response pairs for every instruction, with both pair orderings evaluated to test order-related effects.The pairwise comparisons assess coherence, correctness, and alignment with the instruction or reference.
Bias Benchmarking: Bias proportions are compared with empirically calculated RANDOM thresholds, with above-threshold patterns treated as evidence of bias and tested statistically against the baseline.The threshold is based on randomly selecting a response in each pair and is used as a rough litmus test.
Human Preference Study: 750 sampled pairs from 25 instructions were used to collect human preferences from six Amazon Mechanical Turk workers for comparison with model evaluations.Human annotations were analyzed using Rank-Biased Overlap and average bias proportions across annotators.
Human Preference Study: Rank-Biased Overlap measures agreement between human and model rankings by weighting higher-ranked list positions more heavily.The metric ranges from 0 for non-conjoint rankings to 1 for identical rankings, with 86% of weights concentrated on the top five positions.

5 Results and Discussion

Across the bias benchmarks, LLM evaluators showed strong sensitivity to ordering, model names, response length, external preferences, and irrelevant information. Their rankings also showed limited agreement with human preferences, while human annotators exhibited less bias and higher mutual agreement.

Implicit Biases: 11/15 models favored the first- or last-ordered response, and the first-ordered system was favored in over 50% of comparisons within the >40B group.These results indicate substantial order sensitivity in pairwise evaluation.
Implicit Biases: All models were dramatically influenced by recognizable model names, with disparities between ORDER BIAS and COMPASSION FADE supporting name-driven evaluation effects.OLMO also showed a drastic performance decrease under this comparison.
Implicit Biases: Larger models in the first and second size groups were more affected by longer responses, whereas smaller models excluding MPT were less influenced by response length.Table 3 reports only small deviations in preferences for generations from large versus small models with respect to saliency.
Induced Biases: Around half of the models were influenced by irrelevant information, with >10B models distracted in over 80% of evaluations while ChatGPT and Cohere remained robust.The distracting information was introduced in the attentional-bias benchmark.
Agreement with Human Preferences: 0.44 was the average RBO between human and model preferences, while human annotators had pairwise RBO variance of 0.004 and generally higher mutual agreement.Model-human agreement was lower than agreement among human annotators.

6 Conclusion

The paper introduces COBBLER to assess implicit and induced biases in LLM evaluators and compares their judgments with human preferences. Most models exhibit substantial cognitive bias, with only 44% average agreement with human preferences, suggesting they remain unsuitable as fair and reliable automatic evaluators.

COBBLER assesses LLM evaluation performance against implicit and induced cognitive biases in question-answering settings.
44% average agreement was found between LLM evaluations and human preferences.
Most LLMs exhibit cognitive biases to a greater extent than humans.
The findings suggest that LLMs remain unsuitable as fair and reliable automatic evaluators.

Limitations

The study acknowledges limitations involving response validity, human annotation agreement, and the durability of benchmark findings as models evolve. Prompting and task complexity may affect results, while rapid capability changes can make prior measurements outdated.

Some models produce very low valid response rates, possibly because of the prompting format.
Model-specific prompts may extract clearer results for individual bias benchmarks.
Human inter-annotator agreement is fairly subpar, possibly because ranking 15 models increases task complexity for annotators.
Benchmark findings may quickly become outdated as language-model capabilities scale and new developments appear frequently.

A Experimental Setup

The experimental setup standardizes generation parameters across models and uses controlled generation tools where supported. Responses are generated from ELI5 and BIG-bench StrategyQA, which provide long-form and multi-step question-answering tasks for evaluator benchmarking.

Generation settings: The study uses identical hyperparameters across models for response and evaluation generation.
Generation settings: Generation uses a 128-token limit and temperature 1.0, with repetition penalty 1.2 and three beams for Huggingface models.
Generation tools: Microsoft Guidance controls generations for ChatGPT, InstructGPT, GPT-4, and Vicuna, while Huggingface pipelines support other models.
Datasets: ELI5 provides long-form questions and highest-rated answers from 270k Reddit discussion threads.
Datasets: StrategyQA supplies questions and short justifications designed to require multi-step reasoning when comparing explanations.

B.1 Correlation between BANDWAGON and Percentage

The experiments test whether bandwagon preferences change when the stated percentage of people preferring a response is modified. Most models remain biased, although sensitivity varies across models and prompting with ties produces many invalid responses for several mid-range systems.

Modified bandwagon statistic: The additional bandwagon experiment tests a fake statistic claiming that 0% of people prefer the selected response.
Modified bandwagon statistic: Bandwagon preferences change substantially for most models when the statistic changes, supporting a relationship between the statistic and biased tendency.
Model-specific behavior: Vicuna is comparatively insensitive to the statistics and may focus on the phrase claiming that people believe one model is better.
Modified bandwagon statistic: Most models continue to show biased tendencies when the stated preference percentage is randomly selected between 50% and 85%.
Prompting with ties: Including a tie option leaves the strongest and smallest models largely unchanged, while Alpaca, Vicuna, and InstructGPT assign ties at rates of at least approximately 90% without valid results.

B.4 Decoupling Confounding Factors

The study separates egocentric preference from salience-related effects by examining response length, model strength, and evaluation order. It also uses hierarchical rules to reduce confounding between bias measurements.

Egocentric and Salience Bias: The authors reason that repeated self-preference across models may indicate egocentric qualities regardless of objective model strength.The evaluated pairwise pool is shared across evaluators, supporting comparison of self-preference tendencies.
Confound Control: A hierarchical rubric gives order bias priority, excluding evaluations identified as order-biased from subsequent salience or egocentric analysis.This strategy is intended to isolate each bias as much as possible.
Egocentric and Salience Bias: Self-preference appears in both stronger and smaller models when their own generations are longer, complicating attribution to salience alone.GPT4, ChatGPT, Cohere, and Koala are cited as examples of this pattern.
Order Bias: Order-bias comparisons report the highest-proportion bias ratio and distinguish first-ordered from last-ordered preferences, including analyses with ties.The tables use subscripts xF and xL to mark first- and last-ordered bias.

B.5 Significance of Results

The authors test whether observed evaluator biases differ from random choice and examine agreement, validity, and performance across model scales and list-wise ranking settings. Results include widespread order-bias significance and difficulty producing valid rankings for many models.

Significance Testing: The null hypothesis tests whether an evaluator is as likely as random selection to flip preference according to response order.A two-sample Z test of proportions compares each evaluator with the random baseline.
Significance Testing: Almost all ORDER benchmarks show biased-evaluation proportions statistically significant from the random score.The authors caution that both first-order and last-order tests are needed to characterize evaluator tendencies.
Agreement and Validity: Models within the same size group, especially those above 100B parameters, show greater agreement with one another than with models from other size groups.Figure 5 visualizes average Rank-Biased Overlap between LLMs.
Model Scale: Scaling affects validity and bias differently: LLAMAV2 validity decreases with size, while bias impact generally increases as model size decreases, except for SALIENCE BIAS.The comparison covers LLAMAV2 and VICUNA across model scales.
N-Rankwise Setting: Most models other than closed-source API models struggle to produce a proper ranking even for N = 4 responses.The authors associate this difficulty with the increased complexity of ranking multiple generations.
Evaluation Prompts: COBBLER evaluates six biases using bias-specific prompts, including modifications derived from the original ORDER prompt.The supplied materials include prompt templates for compassion, bandwagon, attentional, and order-related evaluations.

D.3 Details on Pairwise Human Preference Experiments

The human preference experiments use sampled pairwise model outputs, reversed or modified to test three biases, and collect annotations through Amazon Mechanical Turk. Rankings and bias proportions are aggregated with RBO-based agreement calculations.

Experimental Scope: Humans do not test COMPASSION FADE or EGOCENTRIC BIAS because the setup is not applicable to human cases.The authors cite anonymity of model names and the non-human origin of the texts as reasons.
Pairwise Sampling: 750 model pairs across 25 instructions are created by sampling 15 pairs per instruction and adding reversed or bias-switched counterparts.The resulting design contains 30 pairs per instruction and covers ORDER, BANDWAGON EFFECT, and ATTENTIONAL BIAS.
Aggregation and Agreement: Human annotations are converted into ranked model lists, normalized by each model’s wins divided by appearances, and compared using mean RBO.The same RBO approach is used to compute inter-annotator agreement for each bias.
AMT Procedure: The AMT interface asks workers to rank AI-generated answers by fluency, reasonableness, and alignment with a reference sentence.Workers drag systems from highest to lowest quality and may reorder previously placed answers.
AMT Procedure: The pairwise human-bias interfaces separately support ORDER, BANDWAGON EFFECT, and ATTENTIONAL BIAS experiments.The supplied figure captions identify the interface designs for each pairwise setup.