PaperlayerView exact revision on arXiv

Source-linked AI summary

Style Over Substance: Evaluation Biases for Large Language Models

Minghao Wu, Alham Fikri Aji

arXiv:2307.03025v3cs.CL

TL;DR

Whether human and LLM judges can reliably compare open-ended LLM outputs remains uncertain. The paper proposes MERS, which evaluates text across separate dimensions, and reports improved GPT-4 evaluation quality, particularly for factual accuracy, while crowd judges remain indecisive.

  • Problem

    For open-ended LLM outputs, whether human and LLM judges can reliably identify the better answer remains unresolved.

    Source: §1 · p. 1

  • Method

    The paper proposes MERS, evaluating machine-generated text across multiple dimensions independently rather than collapsing quality into one score.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    MERS significantly improves GPT-4 evaluation quality, particularly factual accuracy, while human judges continue to show indecisiveness.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors recommend multidimensional evaluation and caution when using crowd annotators, citing their indecisiveness, length bias, and limited fact-checking.

    Source: §7 · p. 9

    The study uses only 40 questions, which may not represent the full range of question types and variations.

    Source: §8 · p. 9

Abstract

from arXiv · show

As large language models (LLMs) continue to advance, accurately and comprehensively evaluating their performance becomes increasingly challenging. Ranking the relative performance of LLMs based on Elo ratings, according to human judgment, is gaining more popularity. However, the extent to which humans and LLMs are capable evaluators remains uncertain. This study investigates the behavior of crowd-sourced and expert annotators, as well as LLMs, when comparing outputs from different models. To achieve this, we curate a dataset of intentionally flawed machine-generated answers. Our findings reveal a concerning bias in the evaluation process, as answers with factual errors are rated more favorably than answers that are too short or contained grammatical errors. To address this issue, we propose independently evaluating machine-generated text across multiple dimensions, rather than merging all the evaluation aspects into a single score. We instantiate this idea with the Elo rating system, resulting in the Multi-Elo Rating System (MERS). Empirical results from our study reveal that this proposed approach significantly enhances the quality of LLM-based evaluations, particularly in terms of factual accuracy. However, there is no significant improvement in crowd-sourced-based evaluations, indicating the need for further investigation.

1 Introduction

Elo-based judgments of open-ended LLM answers raise questions about whether human and LLM judges can reliably identify better outputs. The study probes their biases and introduces multi-dimensional evaluation as an alternative to a single score.

1 Introduction

  • The study uses GPT-4 to generate answer variants with language, factuality, and length differences for 40 general-purpose questions, then collects judgments from crowds, experts, GPT-4, and Claude-1.Researchers manually reviewed and post-edited the generated responses before annotation.Source: §1 · pp. 1–2
  • Human judges hesitate and generally fact-check only glaring errors, whereas LLMs fact-check to some extent but imperfectly.Humans include both crowd-sourced and expert annotators in this finding.Source: §1 · p. 2
  • Both humans and LLMs favor longer answers, while crowd annotators are nearly order-neutral, LLMs favor the first answer, and experts favor the second.The ordering preferences differ by evaluator group.Source: §1 · p. 2
  • Judges favor answers with factual errors over answers that are short or grammatically incorrect.The study identifies this preference as a problem with evaluation based on a unified measure.Source: §1 · p. 2
  • MERS evaluates machine-generated text independently across Accuracy, Helpfulness, and Language, rather than combining dimensions into one score.The authors recommend evaluating text from multiple perspectives and exercising caution with crowd-sourced LLM assessments.Source: §1 · p. 2
  • MERS significantly improves GPT-4 evaluation quality, particularly factual accuracy, while human judges remain indecisive.The introduction reports this improvement alongside persistent indecisiveness among human evaluators.Source: §1 · p. 2

2 Evaluation Method

The evaluation method uses Elo ratings to compare intentionally varied answers, then gathers judgments from crowd-sourced annotators, experts, and two LLM judges. The study varies answer correctness, language quality, and length across a selected set of general-purpose questions.

2.1 Elo Rating System

  • Elo ratings compare players’ relative skill through expected and actual scores, with judgments encoded as wins, losses, or ties and ratings calculated across 10,000 randomized game orderings.The study sets K = 32 and uses the same Elo system to assess its answer settings.Source: §2.1 · p. 3

2.2 Answer Generation

  • The study selects 40 general-purpose questions and generates 12 answer settings varying factual errors, language proficiency, and response length.Questions exclude categories requiring specialized expertise or subjective creative assessment; generated answers are manually reviewed and edited for the intended error types.Source: §2.2 · pp. 3–4

2.3 Crowd-Sourced Evaluation

  • Crowd-sourced judgments come from 377 U.S. annotators screened for prior task volume and success rate, who compare anonymized responses after a mandatory 20-second delay.Annotators contribute an average of 14 judgments each and can choose the better response or a tie.Source: §2.3 · p. 4

2.4 Expert Evaluation

  • A team of 20 expert annotators evaluates a 200-game sample using the same instructions as the crowd-sourced participants.The experts each annotate 10 games; the paper notes that expert annotation costs more than crowd-sourcing.Source: §2.4 · p. 4

2.5 LLM Evaluation

  • GPT-4 and Claude-1 serve as LLM judges, using a prompt that evaluates helpfulness, relevance, accuracy, and detail while aiming to avoid answer-order bias.Claude-1 is added because the answers were refined by humans after GPT-4 generated them, raising concern about GPT-4’s potential bias toward its own outputs.Source: §2.5 · pp. 4–5

3 Analysis

The analysis finds that judges can favor factually flawed answers over short or grammatically flawed ones, while human judgments are comparatively indecisive. Judges also show answer-length and answer-order biases, and agreement is generally slight except between the two LLM judges.

Factual accuracy

  • 1206 vs. 1096 Elo: GPT-4 rates “Several Minor Factual Errors” above “Correct + Short” and sometimes favors answers with major factual errors despite detecting them.The authors report that GPT-4 does not consistently detect inaccuracies.Source: §3 · p. 5

Answer length

  • All judges favor longer answers; Figure 2 shows GPT-4 has the strongest length bias and expert annotators the weakest.The authors note that the prompt’s “level of detail” criterion may correlate with response length.Source: §3 · p. 5

Human judgments

  • Human Elo scores remain relatively close to 1000, ranging from 926 to 1086 for crowd-sourced annotators and 832 to 1162 for experts.Figure 3 illustrates that human judges assign ties more often than LLM judges.Source: §3 · p. 5, Figure 3 · p. 5

Answer order

  • Crowd-sourced annotators, GPT-4, and Claude-1 favor Assistant 1, whereas experts favor Assistant 2 and more often choose ties.The authors recommend evaluating both answer orders.Source: §3 · p. 5, Figure 3 · p. 5

Inter-annotator agreement

  • Only GPT-4 and Claude-1 achieve moderate inter-annotator agreement; the other comparisons show slight agreement when ties and inconsistent annotations are retained.The reported agreement uses Cohen’s κ; expert comparisons cover 200 games.Source: §3 · p. 6, Table 3 · p. 6

4 Multi-Elo Rating System

MERS responds to the limits of single-score evaluation by assessing LLM outputs across Accuracy, Helpfulness, and Language. The reported comparisons show that experts and GPT-4 differ in how they judge factual accuracy, helpfulness, and language quality.

4 Multi-Elo Rating System

  • MERS addresses the subjectivity of collapsing accuracy, detail, relevance, language, and helpfulness into one score by providing a multidimensional evaluation approach.The approach draws inspiration from multidimensional evaluation in machine translation.Source: §4 · p. 6

4.1 Methodology

  • MERS evaluates Accuracy, Helpfulness, and Language separately, with factual correctness and logical consistency, relevance and depth, and clarity and grammar defining those dimensions.Answer quality depends on context, and the reported GPT-4 results use three independent queries for the dimensions.Source: §4.1 · pp. 6–7

4.2 Stop Using Crowd-Sourced Annotators!

  • Crowd annotators nearly equated Several Major Factual Errors + Short (955 Elo) with Correct + Short (963 Elo), while experts were more proficient at identifying factual errors.Both crowd and expert annotators struggled to recognize spelling or grammatical errors.Source: §4.2 · p. 7

4.3 Experts versus GPT-4

  • GPT-4 effectively ranks factual-error severity and distinguishes language quality, while experts prioritize factual accuracy but struggle to recognize spelling or grammatical errors.Both experts and GPT-4 favor longer answers as more helpful.Source: §4.3 · pp. 7–8

5 Discussion

The discussion argues for evaluating quality factors individually rather than combining them into a single score. It also cautions that human judgments have limits, including fact-checking and scaling constraints.

5 Discussion

  • Accuracy, fluency, and other factors should be assessed individually because a single score obscures their relative importance and leaves their prioritization subjective.The proposed categories may not capture every aspect of an ideal answer, so further research is needed.Source: §5 · p. 8
  • Crowd feedback can indicate audience perceptions but may lack rigorous fact-checking, while expert evaluation is better in this respect but harder to scale.Both human groups also show biases such as preferences for answer length and order, motivating supplementation with other evaluation methods.Source: §5 · p. 8

6 Related Work

Related work describes efforts to evaluate LLMs through benchmarks, human judgments, and LLM judges, including Elo-based comparisons. It also notes that prior work has treated human evaluation differently from this study’s broader focus on factual accuracy, language quality, and text length.

6 Related Work

7 Conclusion

The study examines evaluator behavior using machine-generated answers with deliberately introduced factual and grammatical errors. It proposes multidimensional Elo evaluation, reporting improved GPT-4 evaluation quality, particularly for factual accuracy, while crowd judges remain indecisive.

7 Conclusion

  • The study deliberately introduces factual and grammatical errors into machine-generated answers and examines judgments from crowd-sourced, expert, and LLM evaluators.The stated goal is to investigate limitations and biases in human and LLM evaluation.Source: §7 · p. 9
  • The authors report that independent multidimensional evaluation with MERS improves GPT-4 evaluation quality, particularly factual accuracy, while crowd judges remain indecisive.They recommend caution with crowd annotation due to indecisiveness, length bias, and limited fact-checking.Source: §7 · p. 9

8 Limitations

The study’s limitations include narrow question coverage and a three-dimension evaluation scheme whose definitions may not reflect every stakeholder’s perspective. It also documents the prompts used to generate answers, including a roughly 50-word target for short answers.

8 Limitations

  • The study uses only 40 questions, which may not capture the full range of question types or sufficiently represent the research questions.The authors identify this limited selection as a potential gap in coverage.Source: §8 · p. 9
  • MERS covers Accuracy, Helpfulness, and Language, but these dimensions may not span text evaluation and their definitions may vary across stakeholders.The authors describe their definitions as not infallible and leave addressing these limitations to future work.Source: §8 · p. 9

A Answer Generation Prompts

  • For prompts that generate short answers, GPT-4 is instructed to produce responses of roughly 50 words.The section presents prompts for several answer conditions, including correct, factual-error, and learner-style answers.Source: A Answer Generation Prompts · p. 13

B LLM Evaluation Prompt

The LLM evaluation prompt is sourced from Dettmers et al. (2023) and is presented in Figure 10.

B LLM Evaluation Prompt

C Human Evaluation Interface

The paper shows the interface used for crowd-sourced evaluation in Figures 11 and 12.

C Human Evaluation Interface

D Separate vs. Compound

The study compares separate and compound prompts for GPT-4’s Accuracy, Helpfulness, and Language evaluations. Separate assessment performs better for factual accuracy, while compound assessment produces different rankings on Accuracy and Helpfulness; Language rankings remain consistent.

D Separate vs. Compound

  • The section presents the two evaluation strategies and reports their GPT-4 results in Table 5.The table covers Accuracy, Helpfulness, and Language ratings.Source: D Separate vs. Compound · p. 13, Table 5 · p. 14
  • GPT-4 assesses factual accuracy better with separate prompts; Table 5 reports ratings across Accuracy, Helpfulness, and Language for separate and compound strategies.The authors choose separate prompts for this work because they yield better factual-accuracy results.Source: D Separate vs. Compound · p. 14, Table 5 · p. 14
  • For Accuracy, compound prompting ranks “One Minor Factual Error” and “Advanced Learner” above “Correct + Short,” unlike separate assessment.Source: D Separate vs. Compound · p. 14
  • For Helpfulness, separate prompting ranks “Several Minor Factual Errors” above “Advanced Learner,” whereas compound prompting reverses that ranking.Source: D Separate vs. Compound · p. 14
  • Language rankings are the same with separate and compound prompts, unlike the reported Accuracy and Helpfulness rankings.The authors hypothesize that evaluation dimensions can influence one another when assessed concurrently, even with explicit instructions to assess each independently.Source: D Separate vs. Compound · p. 14

Related papers

Loading 2307.03025v3…