PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbosity Bias in Preference Labeling by Large Language Models

Keita Saito, Akifumi Wachi, Koki Wataoka, Youhei Akimoto

arXiv:2310.10076v1cs.CLcs.AI

TL;DR

LLMs evaluating open-ended responses can show verbosity bias, while RLAIF replaces costly human feedback with judgments from other LLMs. This paper compares LLM preferences with human labels and finds that GPT-4 favors longer answers in creative writing, with lower human alignment when people prefer shorter answers.

  • Problem

    LLM judgments of open-ended responses can be arbitrary and favor longer answers despite similar quality, while prior verbosity studies did not compare these preferences with humans.

    Source: §1 · p. 1, §2.3 · p. 4

  • Method

    The paper tests LLM verbosity preferences against human labels and formulates an accuracy-parity measure for comparing models.

    Source: §1 · p. 2, §6 · p. 8, §7.1 · p. 8

  • Results

    GPT-4 tends to favor longer answers in creative writing, and its alignment with humans is lower when humans prefer the shorter answer; the paper reports that verbosity bias still exists for GPT-4.

    Source: §4 · p. 5, §5 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The proposed measure can be used to compare models on their degree of verbosity bias.

    Source: §1 · p. 2

    The metric can miss bias within its two length-based groups, so the authors recommend showing the human-alignment plot alongside it.

    Source: §7.3 · p. 9

Abstract

from arXiv · show

In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.

1 Introduction

LLM-based feedback offers an alternative to costly human labeling, but LLM judgments of open-ended responses can favor verbosity. This paper compares LLM and human preferences and proposes an accuracy-parity measure of verbosity bias.

1 Introduction

  • Human feedback can be costly, motivating RLAIF, which replaces it with feedback from other LLMs.This paper investigates biases that may accompany evaluation by LLMs.Source: §1 · p. 1
  • When prompts have no clear-cut correct answer, LLM evaluations can become arbitrary and introduce biases such as favoring verbose responses.Unaccounted-for verbosity bias may lead LLM agents to generate unnecessarily long responses, including for simple questions.Source: §1 · pp. 1–2
  • Prior verbosity-bias studies focused on specific settings and did not compare LLM preferences with human preferences.The authors identify such a comparison as important for examining whether longer answers are inherently better.Source: §1 · p. 2
  • The paper finds that LLMs favor longer answers in creative writing and that their verbosity preferences can diverge from humans; it also proposes an accuracy-parity measure.The measure is intended to compare models on their degree of verbosity bias.Source: §1 · p. 2

2 Preliminaries

The section introduces supervised fine-tuning and RLHF, then describes RLAIF as a lower-cost substitute for human feedback. It also reviews biases that can affect LLM evaluation and prompting approaches proposed to mitigate them.

2 Preliminaries

  • Supervised fine-tuning uses expert examples, while RLHF trains a reward model on human preferences and uses it to guide language-model training.The reward model is trained to match human feedback before providing reward signals in the subsequent RL phase.Source: §2 · p. 2, Figure 2 · p. 3

2.1 RLHF

  • RLHF trains a reward model on preferred and rejected response pairs, then uses its score to reward generated text during reinforcement learning.The reward is given after the full text is generated; an optional KL term can penalize deviation from the original policy.Source: §2.1 · p. 3

2.2 RLAIF

  • RLAIF replaces human feedback with feedback from other LLMs, reducing evaluation costs relative to human labeling.The paper reports an evaluation cost of around $0.05 each, or 1/15th the cost cited for human feedback in Wang et al. (2023).Source: §2.2 · p. 3

2.3 Biases in Automated LLM Evaluation

  • Position bias favors particular answer positions, while self-enhancement bias favors an LLM’s own outputs; swapping answer order can reveal position bias.The authors describe contradictory judgments across permutations as a draw.Source: §2.3 · p. 4
  • Verbosity bias favors longer answers despite similar quality, and may yield unnecessarily long RLAIF outputs even when concise responses suffice.The section notes that verbosity can also be important to usefulness in tasks such as question answering.Source: §2.3 · p. 4
  • Chain-of-thought and one-shot or few-shot prompting are presented as ways to encourage more grounded or intended LLM evaluations.Chain-of-thought prompting asks for a thought process before the evaluation; example-based prompting supplies examples and corresponding answers.Source: §2.3 · p. 4

3 Related Works

Prior work explores RLAIF and LLM-based evaluation, including studies of verbosity bias in narrow task settings. This paper broadens that setting and compares LLM preferences with human judgments.

3.1 RLAIF Advancements

  • RLAIF studies report near-human summarization performance without human feedback and investigate balancing helpfulness with harmlessness.Zheng et al. (2023) also report over 80% agreement between GPT-4 and humans when evaluating LLMs.Source: §3.1 · p. 4

3.2 On Verbosity Bias in Evaluations by LLMs

  • In repetitive-list attacks, Zheng et al. (2023) report success rates below 10% for GPT-4 but over 90% for GPT-3.5 and Claude-v1.Their experiment concerns listing tasks with answers made verbose by repeating items.Source: §3.2 · pp. 4–5
  • For summarization, Huang et al. (2023) found GPT-4 preferred shorter responses on faithfulness and coverage, especially in single-answer rather than comparison grading.The cited result suggests verbosity preferences can differ across tasks.Source: §3.2 · p. 5
  • Unlike prior studies that artificially lengthened texts while assuming unchanged content preserves true quality, this work uses human feedback as the oracle.The authors describe their problem setting as more general and compare verbosity preferences between humans and LLMs.Source: §3.2 · p. 5

4 Verbosity Preference of LLMs

Across three creative prompts, GPT-4 tended to prefer longer answers, with preference patterns varying by question. The experiment compares same-model responses and cannot establish whether longer answers were actually more helpful.

4 Verbosity Preference of LLMs

  • The study tested GPT-4’s general preference by comparing responses to open-ended prompts rather than restricting evaluation to a specific answer format.It generated 100 answers for each of three prompts from Vicuna-7b-v1.5 at temperature 0.7.Source: §4 · p. 5
  • GPT-4 evaluated response pairs with the positions swapped, and a result was counted only when both permutations produced the same choice.Otherwise, the pair was treated as a draw.Source: §4 · p. 5
  • GPT-4 increasingly preferred the longer response as the word-count gap widened across three creative prompts, though preference curves differed by question.For the smartphone prompt, evaluations had high variance when the word-count difference was small; the authors say post-evaluation adjustment would require knowing each prompt’s preference shape.Source: §4 · p. 5, Figure 3 · p. 6
  • Because the experiment lacked ground-truth judgments, it could not determine whether GPT-4’s preference for longer responses reflected verbosity bias or genuinely higher quality.The authors next use human evaluations as a baseline.Source: §4 · p. 5

5 Is There A Difference in Verbosity Preference Between LLMs and Humans?

Using human feedback as a baseline, the study examines whether LLMs’ verbosity preferences match human judgments. LLMs aligned well when humans chose longer answers, but alignment fell when humans chose shorter ones.

5 Is There A Difference in Verbosity Preference Between LLMs and Humans?

  • Human judgments also tended to favor longer answers, and the study treats matching those judgments—not eliminating verbosity preference—as the RLAIF objective.The authors suggest, without establishing, that LLMs may have learned a heuristic favoring longer answers by mimicking human behavior.Source: §5 · pp. 6–7
  • The experiment used HH-RLHF human feedback to relate response-pair word-count differences to how often LLM judgments matched human judgments.The dataset provides one human feedback label per prompt, so the authors examine the relationship across questions rather than plotting human preference as in the previous experiment.Source: §5 · p. 6
  • Figure 5 relates the chosen-versus-rejected answer length gap to the rate at which LLM decisions agree with human judgments.Bracketed numbers indicate sample sizes.Source: Figure 5 · p. 7
  • GPT-4 judged which assistant was more helpful by evaluating pairs of whole conversations, rather than answers to a single question.The study used the previous experiment’s prompt template.Source: §5 · p. 7
  • When humans preferred the shorter answer, LLM-human alignment was low because the LLMs still chose longer answers regardless of the shorter answer’s helpfulness.When humans preferred the longer answer, alignment was high because LLMs also preferred the longer response.Source: §5 · p. 7

6 Formulation of Verbosity Bias

The paper formulates verbosity bias as a directional difference in inaccuracy across answer-length groups, using accuracy parity to compare models. Applying the measure, it reports that GPT-4 improved over GPT-3.5 but remained susceptible to verbosity bias.

6 Formulation of Verbosity Bias

  • The formulation labels the more helpful human-selected option Y and the LLM decision Y′, while S indicates which response is longer.S is 0 when y0 has more words and 1 when y1 has more words.Source: §6 · p. 8
  • Accuracy parity requires equal prediction accuracy across the two response-length groups, extending equal opportunity beyond cases where human feedback prefers y0.The equal-opportunity condition shown first covers only cases where human feedback prefers y0.Source: §6 · p. 8
  • The paper’s directional measure uses the difference in inaccuracy between response-length groups, with positive values indicating a preference for verbose answers and negative values for shorter answers.The authors preserve the direction of the bias because some tasks, such as summarization, may show a negative bias.Source: §6 · p. 8
  • GPT-4 improved in verbosity bias relative to GPT-3.5, but verbosity bias remained; the authors call for experiments with additional LLMs.The reported values were calculated using data from Section 5.Source: §6 · p. 8

7 Discussion

The discussion explains why verbosity is treated differently from conventional sensitive attributes, describes how experiment design limits the specificity of results, and identifies a blind spot in the proposed bias metric.

7.1 Other Metrics of Equality

  • Verbosity is a relevant evaluation attribute, so demographic parity is unsuitable; the authors instead base measurement on equal opportunity and accuracy parity.Unlike gender or race in other bias settings, verbosity should be considered when evaluating responses.Source: §7.1 · p. 8

7.2 Limitations of Our Experiments

  • Mixing responses across questions in Section 5 yielded aggregate results across question types, unlike Section 4’s same-question samples, and the authors say which result is preferable is debatable.Section 5 therefore does not provide results for any specific question.Source: §7.2 · p. 9

7.3 Limitation of Our Metric of Verbosity Bias

  • The metric groups pairs only by whether y0 is longer than y1, so it misses bias differences within either group.A concave human-alignment pattern could produce high alignment at large length differences while the metric suggests near-zero bias; the authors recommend showing the alignment plot alongside the metric.Source: §7.3 · p. 9

8 Conclusion

The conclusion reports that LLMs favor longer answers in creative writing and that human alignment varies with verbosity preference. It also presents an accuracy-parity-based metric for comparing models’ verbosity bias.

8 Conclusion

  • LLMs favor longer answers in creative writing, while human alignment is lower when humans prefer shorter answers.The paper formulates verbosity bias based on accuracy parity to compare models quantitatively.Source: §8 · p. 9
  • Previous studies had limited settings and did not compare LLM verbosity preferences with human preferences.The paper’s experiments make that comparison and formulate an accuracy-parity-based measure of verbosity bias.Source: §8 · p. 9

Related papers

Loading 2310.10076v1…