PaperlayerView exact revision on arXiv

Source-linked AI summary

Methodological reflections for AI alignment research using human feedback

Thilo Hagendorff, Sarah Fabi

arXiv:2301.06859v1cs.HCcs.AIcs.CLcs.LG

TL;DR

Human-feedback alignment for LLM summarization faces methodological challenges, including possible errors and bias in judgments. The paper analyzes this setup and recommends a staged design that establishes expert baselines and checks trainer agreement before scaling. It presents these methods as relevant to human feedback for alignment beyond summarization.

  • Problem

    Human feedback for LLM summarization can face methodological challenges that may thwart alignment, including overlooked errors and potential bias in judgments.

    Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 3

  • Method

    The paper analyzes a summarization setup in which human ratings train a reward model and proposes changes to the experimental design and rater process.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The authors recommend establishing expert-expert agreement, testing a small trainer subgroup, and adding sandwiching techniques if expert-trainer agreement is unsatisfactory.

    Source: §4 · p. 6

  • Takeaways & Limitations

    The authors present more reliable human feedback as important for scaling alignment research beyond summarization tasks.

    Source: §1 · p. 2, §4 · p. 6

    The assumed setup requires the fine-tuned model to receive higher ratings than initial summaries from a new evaluator group, with models and evaluators also seeing the original texts.

    Source: §1 · p. 2

Abstract

from arXiv · show

The field of artificial intelligence (AI) alignment aims to investigate whether AI technologies align with human interests and values and function in a safe and ethical manner. AI alignment is particularly relevant for large language models (LLMs), which have the potential to exhibit unintended behavior due to their ability to learn and adapt in ways that are difficult to predict. In this paper, we discuss methodological challenges for the alignment problem specifically in the context of LLMs trained to summarize texts. In particular, we focus on methods for collecting reliable human feedback on summaries to train a reward model which in turn improves the summarization model. We conclude by suggesting specific improvements in the experimental design of alignment studies for LLMs' summarization capabilities.

1 Introduction

AI alignment asks whether AI systems serve human interests and values safely, while scalable alignment faces uncertainty about what those values require. The paper uses LLM summarization to examine how human feedback can support alignment and proposes methodological improvements.

1 Introduction

  • For summarization, scalable alignment means models should meet good-summary criteria that may be partly unknown, making the task a prototype for broader LLM alignment problems.The paper notes that values can be vague or unknown even though LLMs should not violate them.Source: §1 · p. 2
  • The assumed setup trains a reward model on human ratings of existing summaries, then uses it to improve a summarization model whose outputs are rated higher by new evaluators.Models and evaluators are assumed to see the original texts as well.Source: §1 · p. 2
  • The paper first analyzes alignment challenges in this setup, then introduces techniques and possible amendments to summarization alignment methods.Source: §1 · p. 2

2 Problem analysis

Human-feedback summarization can encounter errors, bias, verbosity or copying, and disagreement among raters. The paper describes checks for diagnosing these problems and assessing whether human judgments provide a sound basis for alignment.

2 Problem analysis

  • Human ratings train a reward model that updates the LLM’s summarization through reinforcement learning, but methodological problems can thwart alignment.The paper frames error, bias, and other shortcomings as possible problems in researcher assessments of generated summaries.Source: §2 · p. 2

2.1 Error-proneness

  • Non-experts may overlook summary errors, especially when technical source material makes oversimplified or false summaries hard to judge.The paper proposes sandwiching techniques to help close the expert–non-expert feedback gap.Source: §2.1 · pp. 2–3

2.2 Biases

  • Trainer demographics may shape perceived relevance and summary representation, so demographic data can help researchers assess potential sources of bias.The paper distinguishes biases already present in source texts from those more likely to arise from trainers when texts contain factual information.Source: §2.2 · p. 3

2.3 Verbosity and copying

  • Word-count correlations can reveal preferences for longer summaries, while longest-common-subsequence checks can detect extractive copying from source text.The former may indicate that trainers prioritize completeness over compression.Source: §2.3 · p. 3

2.4 Agreement

  • Researchers agreed in 60% of cases in Ziegler et al. (2020) and 73% in Stiennon et al. (2020), motivating researcher-, trainer-, and reward-model agreement checks.Post-discussion agreement may reflect conformity, and trainer subgroups may differ in how closely their judgments match researchers.Source: §2.4 · pp. 3–4

3 Improvements in the experimental design

The proposed design clarifies summary criteria and instructions, trains and checks raters, and selects suitable trainers and expert baselines. When expert and non-expert judgments still differ, sandwiching techniques may help.

3.1 Evaluation criteria

  • Use accuracy, coverage, condensation, and fluency criteria, with a six-point rating scale and normalized labels; condensation addresses length as a quality confounder.A question framework can guide judgments, while controlling for length is an alternative to adding condensation.Source: §3.1 · p. 4

3.2 Instructions

  • Give trainers precise, understandable instructions to read the full text and attend to their own biases, including primacy and recency effects.The paper also suggests making the task criteria accessible and allowing trainers to ask researchers questions.Source: §3.2 · pp. 4–5

3.3 Practice trials

  • Use practice trials to familiarize trainers with summary-rating dimensions and tricky cases, requiring at least 10 adequate trials before the experiment.Feedback follows each trial; a one-point difference from expert ratings on the six-point scale could be tolerated.Source: §3.3 · p. 5

3.4 Quality checks

  • Check labeling quality with pre-rated summaries and monitored response times, excluding data from trainers whose check ratings deviate too much from expert ratings.The paper also suggests collecting confidence ratings, while noting that removing low-confidence trials did not improve results in Stiennon et al. (2020).Source: §3.4 · p. 5

3.5 AI trainers

  • Prefer a small, well-selected group of language-proficient trainers who label many summaries, and consider screening and motivation alongside demographic data for bias assessment.The paper suggests that intrinsic motivation may help, while higher pay does not necessarily improve motivation.Source: §3.5 · p. 5

3.6 Expert baseline

  • Use handpicked experts as baseline labelers, choosing subject-matter researchers when factual expertise is needed and language experts otherwise.Experts should receive training on the selected criteria and biases; the paper argues that representation matters because summaries may influence views of the world.Source: §3.6 · pp. 5–6

3.7 Sandwiching techniques

  • For persistent expert–non-expert gaps, sandwiching options include decomposing long texts, explaining technical terms, or using LLM critique and debate, though debate may be costly.The paper describes LLM debate as potentially excessive and time-inefficient for summarization, but potentially worthwhile for insights into more complex tasks.Source: §3.7 · p. 6

4 Conclusion

The conclusion recommends validating expert judgments and trainer agreement before scaling human feedback. It proposes progressively adding assistance and monitoring as the process expands.

4 Conclusion

  • Establish an expert baseline only after expert-expert agreement reaches a chosen threshold, then test a small trainer subgroup and use sandwiching if agreement is unsatisfactory.The paper presents improved human feedback for tasks beyond human capabilities as important for scaling alignment research.Source: §4 · p. 6
  • For texts that are not especially long, first use a model to explain technical terms; if agreement remains low, consider LLM critique before recruiting more trainers.The paper proposes monitoring exclusions, training the reward and summarization models semi-online, and recruiting and training new evaluators for final summaries.Source: §4 · p. 7

Publication bibliography

The bibliography brings together work on AI safety and alignment alongside research on summarization, human feedback, language models, and crowdsourced evaluation.

Loading 2301.06859v1…