PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Feedback is not Gold Standard

Tom Hosking, Phil Blunsom, Max Bartolo

arXiv:2309.16349v2cs.CL

TL;DR

This paper asks whether overall preference scores capture important output qualities well enough to evaluate or train LLMs. It analyzes preference ratings alongside error annotations and varies output assertiveness and complexity. Preference scores under-represent important errors, assertiveness skews factuality judgments, and training evidence suggests preference-based training may disproportionately increase assertiveness.

  • Problem

    It is unclear whether preference scores capture desired output qualities, while task-specific error criteria are difficult to apply across LLMs.

    Source: §1 · p. 1

  • Method

    The authors compare overall preference ratings with task-agnostic error annotations and examine annotations for outputs varying in assertiveness and complexity.

    Source: §1 · p. 1, §2 · p. 2, §3.2 · p. 6

  • Results

    Preference scores under-represent factuality and inconsistency, assertiveness confounds factuality and inconsistency judgments, and preliminary evidence links preference-based training with increased assertiveness.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Human feedback is a useful but imperfect proxy for the desired objective, and a pleasing response is not necessarily useful.

    Source: §6 · p. 9

    The analysis is limited to annotation; reward models and learning algorithms may introduce additional biases.

    Source: §6 · p. 9

Abstract

from arXiv · show

Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of a generated output this single `preference' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and error annotation may be affected by confounders, and leverage instruction-tuned models to generate outputs that vary along two possible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicating that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model outputs. We encourage future work to carefully consider whether preference scores are well aligned with the desired objective.

1 INTRODUCTION

Single overall preference scores are widely used for open-ended LLM evaluation, but obscure annotators’ trade-offs and may favor superficial qualities over errors that are harder to check. This paper tests whether preference scores cover key error criteria and whether annotation is affected by bias.

1 INTRODUCTION

  • Human evaluation using overall scores has become the de facto standard for open-ended generation, despite the difficulty of designing an all-purpose metric for flexible LLMs.Source: §1 · p. 1
  • Annotators may rely on superficial properties such as fluency and linguistic complexity rather than effortful-to-check aspects such as factuality.Source: §1 · p. 1
  • A single overall score conceals annotators’ decision processes, including the trade-offs behind judging one response better than another.Source: §1 · p. 1
  • The study compares ratings of task-independent error types with annotators’ overall quality judgments to test what preference scores capture.Earlier evaluation criteria were often task-specific, limiting their applicability to LLMs.Source: §1 · p. 1
  • Preference scores under-represent factuality and faithfulness, while assertiveness biases factuality judgments and may be disproportionately increased by preference-based training.The paper also reports under-representation of inconsistency and preliminary evidence about increased assertiveness after RLHF.Source: §1 · p. 2

2 ARE PREFERENCE SCORES RELIABLE?

The section establishes task-agnostic output-error criteria and tests whether overall preferences capture them. It finds that preference scores under-represent factuality and inconsistency, while distractor checks reveal errors in granular annotations.

2 ARE PREFERENCE SCORES RELIABLE?

  • The study defined task-agnostic error criteria, including harmfulness, fluency, scope, repetition, refusal, formatting, relevance, factuality, inconsistency, and contradiction.The criteria were selected to be broadly applicable and sufficiently specified for annotation.Source: §2 · p. 2

2.1 EXPERIMENTAL SETUP

  • Annotators gave binary error labels and separate 1–5 overall-quality ratings for outputs drawn from summarization, product-description, and how-to tasks.Crowdworkers produced four annotations per output; agreement ranged from 0.64 for factuality to 0.94 for refusal.Source: §2.1 · pp. 2–3

2.2 RESULTS

  • Six of ten error types contributed to preference scores; refusal weighed most, while factuality and inconsistency weighed much less.The Lasso weights estimate the expected reduction in overall score when each error is present.Source: §2.2 · p. 3, Figure 1 · p. 3
  • Errors that did not contribute to overall scores occurred in fewer than 1% of outputs, while the models and tasks made harmfulness and fluency errors infrequent.The authors also suspect first impressions can influence error judgments, though precise instructions and annotator training may partially mitigate this ambiguity.Source: §2.2 · pp. 3–4
  • Distractors were appropriately penalized for relevance and inconsistency, but factuality and contradiction were also incorrectly penalized.The latter criteria should be independent of the input prompt, suggesting annotators struggled to separate them from overall quality.Source: Figure 2 · p. 4, §2.2 · p. 4

3 ARE ANNOTATIONS AFFECTED BY CONFOUNDERS?

The section tests whether assertiveness and complexity bias human judgments of model errors. Preambles successfully vary these dimensions, and assertiveness is associated with missed factuality and inconsistency errors.

3.1 EXPERIMENTAL SETUP

  • The study varies output tone with preambles and compares crowd annotations with author ‘expert’ annotations on a subset of 300 examples per error type.The authors treat the subset as a useful estimate of true error rates, while noting it is not strictly unbiased; scope, fluency, and harmfulness are excluded because they are rare.Source: §3.1 · pp. 4–6

3.2 RESULTS

  • Preambles induce the intended assertiveness and complexity changes, but the two dimensions are entangled in perceived ratings.The experiment used separate groups to score quality, annotate errors, and rate assertiveness and complexity across 1,500 outputs.Source: §3.2 · p. 6, §3.1 · p. 6
  • Low-assertiveness prompts raise refusal errors from 3.5% at baseline to 24%, so the reported error analyses exclude refusals.The authors say conclusions do not significantly change in the full, unfiltered results.Source: §3.2 · p. 6
  • Crowdworkers underestimate factuality and inconsistency errors more for highly assertive outputs, while assertiveness and complexity have differing or no significant effects on other error judgments.More complex or assertive responses are also perceived as less repetitive; complexity does not significantly change crowdsourced factuality estimates, although expert annotations find fewer factual errors in more complex responses.Source: §3.2 · p. 6
  • Crowdsourced factuality estimates for low-assertiveness outputs exceed baseline, whereas expert estimates are lower, consistent with those outputs being shorter and making fewer factual assertions.Error-rate patterns track perceived assertiveness whether it is manipulated by preamble or measured directly.Source: §3.2 · p. 6

4 ARE HUMAN PREFERENCES A GOOD TRAINING OBJECTIVE?

The paper examines whether human preference training disproportionately favors assertiveness. Its model comparisons suggest that quality can rise without assertiveness rising, but the evidence about RLHF is preliminary because training details are not comparable.

4 ARE HUMAN PREFERENCES A GOOD TRAINING OBJECTIVE?

  • Assertiveness correlates more strongly with overall quality scores (Pearson r = 0.68) than complexity does (Pearson r = 0.53), but the causal direction is unclear.The authors note that preference training could inadvertently increase both complexity and assertiveness as a side effect.Source: §4 · pp. 6–7
  • The model comparison offers preliminary evidence that RLHF may increase Llama 2’s assertiveness more than its quality, but the models’ training details are not directly comparable.The authors call for a more targeted comparison using equivalent base models.Source: §4 · p. 7
  • Output extracts show more dramatic style changes for the RLHF-trained Llama 2 model than for Command 52B.The authors suggest RLHF may prioritize perceived compliance over genuine utility, while human preference scores remain proxies for unknown utility.Source: §4 · p. 7, Table 1 · p. 8
  • Command 52B responses show lower assertiveness than Llama 2 13B responses at equivalent quality, suggesting quality can increase without increasing assertiveness.The authors argue that high-quality, low-assertiveness models may be preferable to “confidently wrong” counterparts.Source: Figure 6 · p. 7, §4 · p. 8

5 RELATED WORK

Prior work has explored task-specific evaluation criteria, granular LLM assessment, and biases in human judgments. The paper situates its focus on preference scores and assertiveness within those lines of research.

5 RELATED WORK

  • Earlier NLG evaluations used task-specific criteria, while surveys found wide variation in criteria and annotation protocols.Wang et al. (2023) interpreted variation in semantic-similarity judgments as a possible sign of example complexity.Source: §5 · pp. 8–9
  • Granular LLM evaluation has included detailed criteria, span-level RLHF, skill decompositions, and learned aggregation of interpretable features (Wu et al., 2023).The paper draws on Xu et al. (2023c) for error criteria, while noting that introspection may not fully reveal annotator biases.Source: §5 · p. 9
  • Prior studies report confounds in summary evaluation and preferences for error-containing, verbose chatbot responses; other work links preference optimization to reduced truthfulness or sycophancy (Liu et al., 2023; Perez et al., 2023).Si et al. (2023) found users can over-rely on convincing but incorrect LLM explanations.Source: §5 · p. 9
  • Language-ideology research connects speech style with perceived intelligence and credibility, and studies have found people may trust machines despite contrary evidence.The paper notes that much of this research concerns speaker demographics, particularly accent.Source: §5 · p. 9

6 CONCLUSION

Human preference scores cover many error types but under-represent factuality and inconsistency, while assertiveness can bias error judgments. The analysis is limited to annotation, and the authors suggest possible mitigations through trained annotators or careful aggregation.

6 CONCLUSION

  • Preference scores capture a wide range of errors but under-represent factuality and inconsistency; assertive outputs are more preferred and may be disproportionately favored by RLHF.The authors describe the RLHF finding as preliminary evidence.Source: §6 · p. 9
  • The analysis covers annotation, not potential additional biases from reward models or learning algorithms.The authors note that these other pipeline stages may introduce further biases.Source: §6 · p. 9
  • Trained, incentivized annotators or multiple annotators with careful aggregation may partially mitigate the identified issues.The authors give jury learning as an example of careful aggregation.Source: §6 · p. 9

A ANNOTATION DETAILS

The appendix describes participant recruitment and payment, then gives separate instructions for overall-quality scoring and confound ratings. Overall scores use a five-point scale, while confound ratings cover complexity and persuasiveness.

A ANNOTATION DETAILS

  • Participants were recruited through Prolific’s Balanced Sample option, could participate up to 200 times, and were blocked from doing both error and overall-score annotation.Payment was £0.30 per pair for overall-score or assertiveness/complexity ratings and £0.60 per pair for error annotation; median pay exceeded £13 per hour.Source: A ANNOTATION DETAILS · p. 14

B.2 OVERALL SCORE ANNOTATION INSTRUCTIONS

  • Annotators rated each of two responses from 1 (very bad) to 5 (very good) according to their own judgment of how happy they would be receiving it.Source: §B.2 · p. 15

B.3 CONFOUND ANNOTATION INSTRUCTIONS

  • Confound ratings asked annotators to assess each response’s language complexity and its assertiveness or persuasiveness separately where possible.The instructions defined complexity by the words and language used, and persuasiveness by how assertive and persuasive the response was.Source: §B.3 · pp. 15–16

B.1 ERROR ANNOTATION INSTRUCTIONS

The instructions ask annotators to assess chatbot responses for specified errors and to judge each criterion separately where possible. They also collect an overall quality score based on the annotator’s imagined experience as the user.

B.1 ERROR ANNOTATION INSTRUCTIONS

  • The form presents a prompt and two responses, asks annotators to read them before assessing errors, and includes two attention checks.Source: §B.1 · p. 14
  • The error checklist covers inconsistency with the request, self-contradiction, factuality, relevance, formatting, unreasonable refusal, and repetition.Source: §B.1 · pp. 14–15
  • Annotators are asked to judge each error criterion separately from overall quality, so a correct but irrelevant answer can pass factuality while failing relevance.Source: §B.1 · p. 15

C TASK TEMPLATES

The appendix converts examples from three datasets into prompts using dataset-specific templates. WikiHow and Curation use fixed templates, while Amazon product prompts use a range of templates.

C TASK TEMPLATES

  • WikiHow article titles were converted into prompts using “Can you tell me {title}?”.Source: C TASK TEMPLATES · p. 16
  • Amazon Product Specifications were converted into prompts using a wide range of templates.The passage directs readers to the dataset files for template details.Source: C TASK TEMPLATES · p. 16
  • Curation articles were converted into prompts using “{article}\n\nGenerate a summary:\n”.Source: C TASK TEMPLATES · p. 16

D INFERENCE HYPERPARAMETERS

Inference settings included temperature 0.7 and model-specific prompting: Llama 2 and MPT used templates, Falcon was prompted directly, and Cohere used an undisclosed internal template. The accompanying figures report priming and response-length findings.

D INFERENCE HYPERPARAMETERS

E ADDITIONAL FINDINGS

Additional analyses find priming in overall quality ratings, a preference for longer responses until repetition becomes problematic, and stable regression weights. Reference outputs scored lowest, while low harmfulness and out-of-scope error rates limited follow-up analyses.

E ADDITIONAL FINDINGS

  • Error-checking annotators showed higher correlations between overall scores and each error criterion than annotators who rated quality alone, indicating a priming effect.Source: E ADDITIONAL FINDINGS · p. 17
  • Annotators preferred longer responses until repetition became an issue, and repetition errors increased with response length.Factuality error rates also increased for both short and long responses.Source: E ADDITIONAL FINDINGS · p. 17
  • Leaving out each error type caused only minor changes to Lasso regression weights, and the selected features remained unchanged.Source: Figure 9 · p. 18
  • Reference outputs received the lowest overall quality scores among the five evaluated systems.Source: E ADDITIONAL FINDINGS · p. 18, Table 2 · p. 18
  • Harmfulness and out-of-scope errors were below 1%, so the authors excluded them from follow-up experiments while noting their importance in the wild.Source: E ADDITIONAL FINDINGS · p. 18

F SUPPLEMENTARY RESULTS

The supplementary results describe successful shifts in output complexity and sentiment, alongside a weaker complexity association with perceived errors than the assertiveness association. More assertive outputs were less likely to be judged factually or inconsistently erroneous.

F SUPPLEMENTARY RESULTS

  • The preambles shifted the output distributions for reading-grade complexity and automatic sentiment scores.The sentiment shift could also reflect a change in assertiveness.Source: Figure 10 · p. 19, Figure 11 · p. 19
  • More complex outputs were somewhat less likely to be judged erroneous, but this effect was weaker than the assertiveness effect.Source: Figure 12 · p. 19
  • More assertive outputs were less likely to be judged as containing factuality or inconsistency errors, and the crowd–expert difference widened with assertiveness.Source: Table 3 · p. 20
  • Figure 13 compares crowdsourced and author-expert error annotations, including samples marked as refusals; Table 4 reports Section 3 error rates and quality scores by model.Source: Figure 13 · p. 20, Table 4 · p. 20

Related papers

Loading 2309.16349v2…