PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to summarize from human feedback

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano

arXiv:2009.01325v3cs.CLcs.AIcs.LG

TL;DR

Summarization training and evaluation rely on objectives and metrics that only imperfectly reflect human judgments of quality. This paper trains a reward model from human comparisons and uses reinforcement learning, finding better summaries than strong supervised baselines and strong transfer across domains.

  • Problem

    Summarization objectives and automatic metrics such as ROUGE imperfectly capture human judgments of summary quality.

  • Method

    The authors collect human summary comparisons, train a preference-based reward model, and use reinforcement learning to fine-tune a summarization policy.

  • Results

    Human-feedback models outperform strong supervised baselines on Reddit summaries, nearly match CNN/DM reference quality without news-specific fine-tuning, and beat ROUGE optimization according to humans.

  • Takeaways & Limitations

    Optimizing human-preference rewards produces summaries that humans prefer and transfers better to a new domain than supervised training.

  • Takeaways & Limitations

    The approach requires substantial resources: reinforcement-learning fine-tuning took approximately 320 GPU-days, while data collection required thousands of labeler hours.

Abstract

from arXiv · show

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

1 Introduction

The paper addresses the mismatch between supervised fine-tuning objectives and human judgments of summary quality by training summarization models from human preferences. On Reddit TL;DR, human-feedback models outperform strong supervised baselines, generalize nearly to human-reference quality on CNN/DM without news-specific fine-tuning, and support analyses of reward optimization and future research.

  • Motivation: Supervised fine-tuning maximizes the likelihood of human-written text, which can misweight factual errors, low-quality demonstrations, and sampling-related degradation relative to human-perceived quality.The paper argues that optimizing for quality may provide a principled way to address these problems.
  • Method: The method collects pairwise human preferences, trains a reward model to predict preferred summaries, and uses PPO-based reinforcement learning to train a policy maximizing that reward.This directly targets the human judgments that are difficult to capture with automatic summarization metrics.
  • Results: Human-feedback policies on Reddit TL;DR produce better summaries than much larger supervised-learning policies, and labelers prefer their summaries to the dataset’s original human demonstrations.These results constitute the paper’s primary evidence that human feedback improves English summarization.
  • Results: Reddit-trained human-feedback models generate CNN/DM summaries without news-specific fine-tuning that almost match the quality of the dataset’s reference summaries.The authors perform checks including monitoring agreement rates among labelers to assess whether preferences reflect real quality differences.
  • Contributions: The paper analyzes model and data scale, reward-model optimization, and summary perturbations, finds the reward model predicts human preferences better than ROUGE, and releases 64,832 TL;DR summary comparisons plus evaluation data.The released evaluation data covers TL;DR comparisons and Likert scores and CNN/DM Likert scores.

2 Related work

Prior work has used human feedback and reinforcement learning for summarization and several other language tasks. This work builds on reward modeling developed in learning-to-rank research.

  • Human feedback for summarization: Prior summarization work learned rewards from human ratings of 2.5k CNN/DM summaries and trained policies preferred over a ROUGE-optimizing policy.The cited work used human ratings to train a reward function before reinforcement-learning policy training.
  • Human feedback for summarization: Related research trained Transformer models to optimize human feedback across tasks including summarization on Reddit TL;DR and CNN/DM.The paper identifies this line of work as most similar to its own approach.
  • Human feedback beyond summarization: Human feedback has also served as a reward in dialogue, translation, semantic parsing, story generation, review generation, and evidence extraction.These applications span multiple language-generation and language-understanding domains.
  • Learning to rank: The reward-modeling approach was developed in prior learning-to-rank research applied to search-result ranking with explicit or click-through feedback.The passage distinguishes explicit feedback from implicit feedback represented by click-through data.

3 Method and experiment details

The method iteratively collects human comparisons, learns a reward model, and optimizes a summarization policy with PPO. Experiments use a filtered Reddit TL;DR task, strong supervised baselines, and procedures achieving high labeler–researcher agreement.

  • Method: The procedure begins with a supervised-learning policy and iterates through collecting comparisons, learning a reward model, and optimizing the policy with PPO.Human evaluators compare summaries sampled from the current and initial policies, references, and baselines; the reward model predicts human-preferred log odds, whose logit is optimized with PPO.
  • Datasets and task: The filtered TL;DR task requires summaries fewer than 48 tokens that faithfully convey the original Reddit post according to human judgments.The dataset contains approximately 3 million Reddit posts and filters human-written summaries to 24–48 tokens, alongside a whitelist of generally understandable subreddits.
  • Human feedback quality: 77% ± 2% labeler–researcher agreement exceeded 73% ± 4% researcher–researcher agreement on a subset of comparison tasks.The authors attribute the high agreement to offline batches of comparison data, cumulative retraining, detailed labeler instructions, ongoing support, and regular performance feedback.
  • Models and baselines: The experiments use Transformer decoders with 1.3B and 6.7B parameters, with supervised fine-tuning providing initial policies, reward models, and evaluation baselines.Final human evaluations sample all models with T=0; the 6.7B supervised model achieves slightly better ROUGE scores than mid-2019 state-of-the-art CNN/DM models.

4 Results

Human-feedback policies outperform supervised baselines on TL;DR, across multiple quality dimensions, and transfer strongly to CNN/DM without news-specific fine-tuning. Reward models improve alignment with human preferences but can overfit under excessive optimization, while scaling and reward-based evaluation outperform standard metrics.

  • TL;DR results: Human-feedback policies significantly outperform supervised baselines on TL;DR according to human preference over dataset reference summaries.Policy quality is measured as the percentage of generated summaries humans prefer over the reference summaries.
  • TL;DR results: 6.7B PPO summaries receive perfect 7/7 overall-quality scores 45% of the time, versus 20% for the supervised baseline and 23% for references.Human-feedback models outperform supervised baselines across coverage, accuracy, coherence, and overall quality, particularly coverage.
  • CNN/DM transfer: The 6.7B human-feedback model nearly matches a CNN/DM-finetuned 6.7B model despite never training on news summarization and producing much shorter summaries.CNN/DM evaluations use four quality dimensions because transferred and CNN/DM-trained models have substantially different length distributions.
  • Reward optimization: Further reward-model optimization initially improves summaries, but excessive optimization makes predicted rewards diverge from true preferences and eventually become anti-correlated with humans.The same over-optimization pattern also occurs when optimizing ROUGE.
  • Reward-model scaling: Doubling training data increases reward-model validation accuracy by ~1.1%, while doubling model size increases it by ~1.8%.The ablation spans 160M–13B-parameter reward models trained on 8k–64k human comparisons.
  • Reward-model evaluation: Reward models generalize to CNN/DM, agreeing with labeler preferences 62.4% and 66.5% of the time for 1.3B and 6.7B models, respectively.The 6.7B reward model nearly matches the 66.9% inter-labeler agreement value and outperforms ROUGE and log probability on human-preference comparisons.

5 Discussion · Appendix

The paper identifies substantial costs and limitations of learning from human feedback, while outlining applications across tasks and emphasizing both alignment benefits and risks of misuse, bias, disagreement, and automation.

  • 5 Discussion: RL fine-tuning the 6.7B model required approximately 320 GPU-days, while data collection took thousands of labeler hours and significant researcher time.These costs prevented collecting an equivalent amount of high-quality human demonstrations for supervised baselines.
  • 5 Discussion: The method could apply wherever humans can compare samples, including dialogue, translation, question answering, speech synthesis, and music generation.The authors expect particular importance for long samples and suggest predicting feedback across many tasks to improve sample efficiency.
  • 5 Discussion: Scaling human feedback to outputs that humans cannot easily evaluate remains challenging, especially for determining whether systems match designers’ intentions.One proposed approach is training systems to help humans evaluate outputs quickly and accurately.
  • 5 Discussion: Beyond binary comparisons, demonstrations, edits, and explanations could provide signals for training more capable reward models and policies.These alternatives represent a broader landscape of human feedback methods.
  • 5 Discussion: Human-feedback techniques have broad potential because they can support machine learning applications wherever humans can evaluate output quality.The research is motivated by aligning algorithms with designer preferences rather than optimizing rough proxy metrics, which can produce problems such as clickbait recommendations.
  • 5 Discussion: As systems become more capable, safety may become harder to ensure because mistakes will be more difficult to detect and consequences more severe.The paper contrasts easily noticed, low-consequence inaccurate news summaries with substantially less safe human-driving imitation.
  • 5 Discussion: Human feedback could also help malicious actors make models persuasive, induce dependence, or generate toxic content, with few obvious solutions.The authors identify avoiding these societal harms as a significant challenge.
  • 5 Discussion: Careful definition of desirable behavior is essential because labelers’ reinforcement can affect many groups, especially when humans disagree about complex objectives.Reddit TL;DR’s minimally moderated content may also lead models to generate biased or offensive summaries, and automation could contribute to significant job loss and societal harm.

A TL;DR dataset details

The filtered TL;DR dataset removes duplicates, unsuitable posts, and overly long bodies before providing 287,790 posts for reinforcement learning. Reference summaries receive additional filtering, while the dataset’s concentration in relationship-related posts raises generality concerns despite strong CNN/DM transfer.

  • Post preprocessing: Nearly 20,000 exact duplicate posts were removed, alongside comments, non-whitelisted subreddits, edited or updated posts, sensitive topics, and bodies exceeding 512 tokens.The preprocessing re-parsed posts with heuristics and retained only top-level posts.
  • Post preprocessing: 287,790 posts remained after body filtering, with approximately 5% held out for validation and the resulting set used for RL.The count excludes filtering by summary.
  • Summary preprocessing: Reference summaries were filtered to remove edit, update, or P.S. openings, profanity, and lengths outside 24–48 tokens.The length range creates overlap with RL summaries for length-controlled analysis, while summaries shorter than 16 tokens were usually low quality.
  • Dataset limitations: About two thirds of the dataset concerns relationships or relationship advice, raising generality concerns despite strong transfer performance on CNN/DM news articles.This concentration represents a fairly specific domain, but the reported transfer suggests the models are not unreasonably specialized to relationship advice.

B Further model training details … C.1 Process for ensuring high-quality human data

The paper specifies Transformer-based pretraining and fine-tuning procedures, input-format conventions, and a staged process for collecting and calibrating high-quality human comparison data. It also notes a small precision-related discrepancy between supervised and reinforcement-learning runs.

  • B.1 Hyperparameters: All models use the standard Transformer architecture, 2048 learned position embeddings, fp16 activations, Adam, and the same byte-pair encoding.Most supervised baselines, reward models, and reinforcement-learning models use fp32 weights; TL;DR supervised baselines use fp16 weights.
  • B.1 Hyperparameters: Pretraining uses Commoncrawl, Webtext, books, and Wikipedia for 200-300 billion tokens, with 2048-token inputs and cosine learning-rate decay.Training lasts 1-3 epochs per corpus, with a short warmup and decay to 10% of the maximum learning rate.
  • B.1 Hyperparameters: Supervised baselines initialize from pretrained models, use cosine learning-rate decay, batch size 128, and train for a single epoch.Initial learning rates are selected from a log-linear sweep of at least 7 values; the reported rates vary by model size and dataset.
  • B.1 Hyperparameters: Reward models initialize from supervised baselines with a randomly initialized reward head, train for one epoch, sweep 3-10 seeds, and select the best development-set model.The 1.3B and 6.7B reward models use learning rates of 1.5e-5 and 5e-6, respectively, with batch size 64.
  • B.1 Hyperparameters: PPO uses separate policy and value networks, initializes them from the supervised baseline and reward model, and applies γ = 1 and λ = 0.95 for advantage estimation.It performs 4 optimization epochs per rollout batch and uses linear learning-rate decay.
  • B.2 Input format: Inputs have a fixed byte-pair-encoded size: short inputs are left-padded, while long post/article fields are truncated at newlines.For pretrained-only models, the context is instead left-padded with examples of posts/articles and high-quality summaries when space permits.
  • C.1 Process for ensuring high-quality human data: The human-data process has four stages: understanding the task, onboarding labelers, collecting comparisons, providing feedback, and calibrating against researchers.Labelers receive paid training and calibration, write naive interpretations before direct comparisons, and are assessed using agreement statistics and researcher comparisons.
  • C.1 Process for ensuring high-quality human data: The project retains good labelers throughout its lifetime while firing the lowest-performing workers.This worker-management rule complements the project’s onboarding, feedback, and calibration procedures.

C.2 Assessing human feedback quality … C.5 Instructions for labelers

Human feedback was fairly reliable despite subjective comparisons, with agreement varying by worker and difficulty; aggregating labels helped, but filtering weaker labels did not. The authors also documented diverse labelers, built a standardized interface, and tailored detailed instructions and quality control to Reddit and CNN/DM.

  • C.2 Assessing human feedback quality: 77% ± 2% labeler-researcher agreement exceeded 73% ± 4% researcher-researcher agreement on comparisons from the 1.3B supervised baseline.The authors attribute substantial remaining noise to difficult and subjective comparisons.
  • C.2 Assessing human feedback quality: Agreement rates ranged from about 65% for least proficient labelers and hardest comparisons to about 85% for most proficient labelers and easiest comparisons.The easiest comparisons contrasted supervised-baseline samples with reference summaries, while the hardest compared high-temperature samples from one RL policy.
  • C.2 Assessing human feedback quality: 72% labeler-labeler agreement rose to 77% with modal labels from three workers, although the study usually collected one label per comparison for throughput.Across the reward-model training corpus, the estimated agreement rate was 73% ± 3%.
  • C.2 Assessing human feedback quality: Researcher agreement ranged from about 65% on hardest comparisons to about 80% on easiest comparisons and about 95% after researchers discussed comparisons.The authors conclude that feedback quality is fairly high, and report that filtering low-confidence labels or poorer-agreement workers reduced reward-model accuracy.
  • C.3 Labeler demographics: Labelers spanned ethnicities, nationalities, ages, genders, and educational backgrounds, but were more likely to be White and American.The demographic results came from an optional anonymous survey of labelers.
  • C.4 Labeler website: Because the authors hired and trained their own labelers, they built a standardized customizable website with separate profiles, task-specific renderers, concern fields, and centralized data export.The interface supported naive interpretations, summary comparisons, and Likert evaluations.
  • C.5 Instructions for labelers: Labeler instructions covered summary comparisons and 7-point Likert evaluations across coherence, accuracy, coverage, and overall score, with examples, rubrics, and FAQs.Reddit instructions also explained Reddit-specific terminology and included example labeled comparisons and annotated summaries.
  • C.5 Instructions for labelers: CNN/DM instructions deemphasized sentence fluidity and matching article intent, while quality control used author labels, revised instructions, trial labeling, and larger-scale onboarding.These changes reflected bullet-point reference summaries and differences from Reddit evaluation.

C.6 Composition of the labeled dataset · C.7 Example comparison tasks · D Choice of baselines

The labeled dataset combined summaries sampled from evolving policies, while comparisons illustrated both ordinary and difficult accuracy–coverage trade-offs. Baselines were chosen under labeler-budget constraints, favoring the paper’s own supervised and zero-shot models over an additional demonstration-trained system.

  • C.6 Composition of the labeled dataset: Labels were collected from summaries sampled across several policies, without a systematic sampling plan, as part of exploratory research.Each reward model used all labels collected up to that point, with later models also benefiting from improved hyperparameters and dataset cleaning.
  • C.6 Composition of the labeled dataset: Labeler drift could gradually change evaluation criteria and policy preferences, so most batches included supervised-baseline and reference-summary comparisons.These comparisons were intended to help guard against shifts in labeling behavior over time.
  • C.6 Composition of the labeled dataset: The study also evaluated best-of-N policies by sampling N summaries at temperature 0.7, scoring them with a reward model, and selecting the highest-scoring summary.This procedure required no training and contributed samples to the training data.
  • C.7 Example comparison tasks: Example comparison tasks included a random TL;DR validation comparison and a difficult cherry-picked comparison designed to show an accuracy–coverage trade-off.The difficult example was selected from comparisons where labelers disagreed.
  • D Choice of baselines: The authors did not provide a demonstration-trained supervised baseline because collecting a similarly large, high-quality demonstration dataset would have been prohibitively expensive.The comparison was motivated by fairness against supervision-based techniques and their labeler-time requirements.
  • D Choice of baselines: Because prior PEGASUS work used smaller 500M-parameter models whose outputs were worse than human references, the study used its own supervised and zero-shot models as baselines.The authors also sanity-checked the supervised models’ ROUGE performance.
  • D Choice of baselines: T5 outputs served as an additional comparison using beam-search decoding, with tokenization differences carefully accounted for.CNN/Daily Mail outputs were lower-cased and heuristically re-capitalized to normalize capitalization and punctuation across models.

E CNN/DM lead-3 vs reference summaries

On CNN/DM, labelers preferred the simple lead-3 extractive summaries to the dataset’s reference summaries, even after controlling for length. Examination of disputed cases attributed much of the reference summaries’ underperformance to omitted key points and unsupported information, raising concerns about treating them as references.

  • E CNN/DM lead-3 vs reference summaries: Labelers significantly preferred lead-3 summaries to CNN/DM reference summaries, with lead-3 being 50% longer and therefore receiving higher coverage scores.Lead-3 consists of the article’s first three sentences.
  • E CNN/DM lead-3 vs reference summaries: 5.68 was the length-controlled lead-3 quality at 314 characters, modestly higher than the reference summaries.The estimate used linear regression to predict lead-3 performance at the reference summaries’ average length.
  • E CNN/DM lead-3 vs reference summaries: 20/143 cases had labelers preferring lead-3 by 3 points or more, while the reference summaries were preferred by a similar margin in only 7/143 cases.Excluding the 20 cases would raise the reference summaries’ relative score by about 0.5 points.
  • E CNN/DM lead-3 vs reference summaries: 13 of the 20 disputed reference summaries omitted a key article point, while 10 introduced information absent from the original article.The omitted points were linked to highlights written for readers who had already seen article titles, which were not included in the dataset.
  • E CNN/DM lead-3 vs reference summaries: The authors judged the labeler comparisons reasonable and warned that CNN/DM highlights may be problematic reference summaries.Their concern follows from the observed omissions and unsupported additions in the reference summaries.

F Controlling for summary length … G.6 Reward model validation sets

The additional analyses examine length control, RL architecture, quality dimensions, optimization behavior, ROUGE, copying, and reward-model validation. Together, they show where human-feedback models succeed, how evaluation metrics differ, and which failure modes remain.

  • F Controlling for summary length: Human-feedback models generate longer summaries because longer outputs received higher labeler preference under the task’s 24–48-token limit.The authors train a logistic regression using policy identity and log summary-length ratio, then set the ratio to zero for length-controlled comparisons.
  • G.1 Value function ablation: Separate value-function and policy networks outperform a shared network, although they increase RL fine-tuning memory requirements.Separate networks also permit initializing the value function with the learned reward model being optimized.
  • G.2 Evaluating policies along axes of quality: Coverage correlates strongly with overall score across TL;DR and CNN/DM models, while all models achieve high coherence scores.Evaluations use a 7-point Likert scale across multiple quality axes.
  • G.3 Studying best-of-N optimization: Best-of-N and PPO policies achieve higher average reward as N increases or KL coefficient β decreases, with similar human-rated quality at matched reward.PPO remains farther from the supervised baseline than best-of-N according to KL divergence.
  • G.4 ROUGE scores: On TL;DR, human-feedback models obtain slightly lower ROUGE than supervised models at T = 0, indicating poor correlation between ROUGE and human preferences.At higher temperatures, feedback models outperform supervised counterparts, while supervised models benefit more from lowering temperature than increasing model size.
  • G.4 ROUGE scores: On CNN/DM, ROUGE agrees that human-feedback models transfer better, but supervised models achieve much higher ROUGE; the 6.7B baseline exceeds some literature models but remains below T5.The reported comparison uses the 6.7B supervised baseline and models from the literature.
  • G.5 Bigram overlap statistics: CNN/DM-evaluated models generally copy more than TL;DR-evaluated models, while supervised and human-feedback models copy less than pretrained models.Bigram overlap is computed using the longest common subsequence of bigrams divided by summary bigram count.
  • G.6 Reward model validation sets: The 6.7B reward model prefers human-improved summaries at a rate similar to humans and detects sentence shuffling, role switches, and small semantic changes.It nevertheless sometimes prefers poor artificial summaries, including duplicated titles or summaries ending with advice requests.

G.7 Measuring agreement between different evaluation metrics · H Samples · H.1 Random samples

The paper compares human and automated summary-evaluation metrics under controlled policy conditions, finding that the reward model tracks labelers better than ROUGE and related baselines. It also documents qualitative reward changes and provides uncurated samples with human evaluations across TL;DR and CNN/DM.

  • G.7 Measuring agreement between different evaluation metrics: Agreement is measured across automatic metrics and human evaluators using summaries generated by the same policy and temperature to control for policy quality.The analysis uses 1.3B supervised-model samples at T=0.7 on TL;DR.
  • G.7 Measuring agreement between different evaluation metrics: The 6.7B reward model generally agrees with labelers about as often as individual labelers agree, while labeler ensembles perform better.This comparison concerns agreement rates between human judgments and automated metrics.
  • G.7 Measuring agreement between different evaluation metrics: ROUGE and supervised-baseline log probability generally show poor agreement with humans, while copying and length heuristics perform comparably.The result indicates that several simple heuristics are no worse than these commonly used automated metrics for agreement.
  • G.7 Measuring agreement between different evaluation metrics: Qualitative examples show the reward model responding to small but semantically meaningful edits that improve TL;DR summaries.Examples are randomly selected for edit distances below 5 and reward changes greater than 0.5.
  • G.7 Measuring agreement between different evaluation metrics: Tables 20–22 report agreement rates between humans and automated metrics for 1.3B supervised, 6.7B supervised, and 6.7B human-feedback-optimized models.The tables use TL;DR samples at T=0.7, bootstrap standard errors, and labeler ensembles with at least 3 workers.
  • H Samples: The samples section presents non-cherry-picked model outputs together with human evaluations.Tables 25–26 cover TL;DR, while Tables 27–28 cover CNN/DM with articles truncated for brevity.
  • H.1 Random samples: Random-sample tables cover outputs from various models on both TL;DR and CNN/DM datasets.The paper directs readers to its website for additional uncurated policy samples.

H.2 Overoptimized samples

Overoptimizing the reward model produces summaries that are clearly long, low quality, and idiosyncratic, although they still capture the rough gist of the original post.

  • Overoptimized samples: Overoptimization to rm3 yields summaries that are clearly long, low quality, and full of idiosyncrasies.These samples remain recognizably related to the source despite their degraded quality.
  • Overoptimized samples: The overoptimized summaries still reflect the rough gist of the post.
  • Overoptimized samples: The examples illustrate a tradeoff in which reward optimization preserves broad topical content while producing visibly poor summaries.The samples are described as long, low quality, and idiosyncratic, yet still gist-preserving.
Loading 2009.01325v3…