Source-linked AI summary
Fine-Tuning Language Models from Human Preferences
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
TL;DR
The paper asks whether human preference-based reward learning can extend reinforcement learning to natural-language tasks, where rewards are difficult to specify programmatically. It combines pretrained language models, learned human-feedback rewards, PPO, and KL regularization across stylistic continuation and summarization. Stylistic continuation performs well with few comparisons, while summarization produces extractive smart copiers whose evaluation may reflect labeling heuristics.
Problem
The paper addresses how to apply reinforcement learning to complex natural-language tasks whose rewards are defined by human judgment rather than explicit programs.
Method
The paper fine-tunes pretrained language models with reinforcement learning using reward models trained from human preferences, while applying KL regularization and testing stylistic continuation and summarization.
Results
5,000 comparisons yielded strong human preferences for stylistic continuation, while 60,000-comparison summarization models copied input sentences, skipped irrelevant preamble, and were preferred to several baselines but not lead-3.
Takeaways & Limitations
Direct human preferences can support reward-learning experiments for language, but the summarization policies remained extractive and their apparent quality may depend on simple evaluation heuristics.
Takeaways & Limitations
Summarization evaluation is difficult because labelers trade off accuracy, grammar, redundancy, and coverage, making consistent labeling and interpretation challenging.
Abstract
from arXiv · showhide
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.
1. Introduction
The paper applies human preference-based reward learning to natural-language reinforcement learning, combining pretrained language models with RL across stylistic continuation and summarization tasks. It reports strong human preferences for stylistic continuation with few comparisons, while summarization policies behave as extractive “smart copiers” and show sensitivity to data-collection strategy.
- Motivation: The paper targets reinforcement learning for complex tasks whose rewards are defined by human judgment, especially where natural language expresses value-laden concepts.Prior reward-learning work had mainly used relatively simple simulated environments, while natural language also supports communication with human supervisors.
- Approach: The method combines pretrained generative language models with human preference learning, KL-constrained reward optimization, and applications to stylistic continuation and summarization.The tasks include positive-sentiment and physically descriptive continuation, plus CNN/Daily Mail and TL;DR summarization.
- Results: 5,000 human comparisons led stylistic-continuation outputs to be preferred 86% of the time over zero-shot outputs and 77% over supervised sentiment fine-tuning.Each comparison asked humans to choose the best of four continuations.
- Results: 60,000 human samples produced summarization models that copy whole input sentences while skipping irrelevant preamble, and human labelers preferred them to supervised and human-written summaries but not lead-3.The paper notes that this behavior emerged without an explicit copying mechanism and may reflect labeler heuristics.
- Data collection: For TL;DR summarization, human labelers preferred online-data policies 71% of the time, whereas offline models often produced inaccurate summaries; offline collection worked similarly well for stylistic continuation.Online training repeatedly collects data and retrains the reward model as the policy improves.
2. Methods
The method learns a reward model from four-way human comparisons, then fine-tunes a language-model policy with PPO under KL regularization. It can use offline or online preference collection, with online retraining addressing distribution shift as the policy changes.
- Policy optimization: The policy is initialized from the reference language model and optimized with PPO using the learned reward model rather than a directly specified task reward.This makes human judgments the source of the optimization signal.
- KL regularization: KL regularization keeps the fine-tuned policy near the reference model, while also acting as an entropy bonus, limiting reward-model extrapolation, and encouraging coherence and topicality on style tasks.The coefficient β can be fixed or dynamically adjusted to target a particular KL value.
- Reward modeling: The pipeline samples four candidate continuations from a reference language model, asks humans to select the best, and trains a reward model on those comparisons.The reward model is initialized from the reference model and fitted to the human-selected options.
- Implementation: The experiments use a 774M-parameter GPT-2 model, supervised pretraining or fine-tuning where specified, Adam for reward modeling, and PPO2 for policy training.Reward-model training uses a single epoch, while policy training uses 2M episodes.
- Online data collection: Online collection samples from the improving policy and periodically retrains the reward model, reducing distributional shift between reward-model training and policy evaluation.The paper states that online collection mattered for summarization but not for the simpler style tasks.
- Human labeling: Human comparisons lack unambiguous ground truth, and authors agreed only about 60% of the time on four-way sentiment and TL;DR comparisons versus 25% for random guessing.This disagreement complicates quality control and makes consistent labeling difficult.
3. Experiments
The experiments apply human-preference reward learning to stylistic continuation and summarization, finding data-efficient human gains for continuation but extractive, potentially shortcut-driven behavior for summarization.
- Experimental scope: RL fine-tuning with human preferences is evaluated on mock sentiment, stylistic continuation, and CNN/Daily Mail and TL;DR summarization.The experiments also compare online and offline reward-model data collection and examine copying, accuracy, and human-evaluation behavior.
- Mock sentiment: 20k to 60k queries optimize the mock sentiment reward nearly as well as direct RL access to the known reward.The mock task simulates human judgments by selecting the continuation with higher classifier reward.
- Human evaluations of continuations: For stylistic continuation, offline and online data collection perform similarly, while 5k, 10k, and 20k reward-model samples perform similarly and degrade below 5k.Human-preference optimization is preferred to the review sentiment classifier in 77% of contexts.
- Summarization: Online data collection improves summarization relative to offline collection, with a 3 point R-AVG gain on CNN/DM at 60k labels, although training-run variation may make the gap noisy.ROUGE gains continue with data volume up to 60k labels, while human-evaluation trends are less clear.
- Summarization: Human labelers strongly preferred the 60k online model, including over human references, revealing a mismatch between intended quality and the heuristics actually evaluated.Copying improves factual accuracy: the most-copying models were 90% and 95% accurate on TL;DR and CNN/Daily Mail, whereas more abstractive models reached at most 70%.
4. Challenges
The paper identifies practical challenges in fully online human-feedback training, including software complexity, debugging difficulty, data-quality regressions, and ambiguous evaluation. It proposes batched data collection as a simpler middle ground and highlights imbalance-related difficulties in jointly training reward models and policies.
- Online data collection is hard: Fully online data collection interleaves data gathering, reward-model training, and RL fine-tuning, making the distributed system significantly more complicated.Bugs in any component can break the entire training process.
- Online data collection is hard: Online experiments were difficult to debug because individual machine-learning components could not be iterated on independently.The authors often switched temporarily to offline runs for debugging before returning online.
- Online data collection is hard: Maintaining high data quality over long online runs was difficult, with regressions sometimes detected only after training completed.Low-latency quality-control mechanisms required substantial work, and poor worker performance could affect already-collected data.
- Online data collection is hard: Batched data collection is proposed as a middle ground that alternates policy sampling, reward-model training, and frozen-policy fine-tuning.Longer batch latency can simplify quality control while retaining data from updated policies.
- Other challenges: Jointly training the reward model and policy was not made to work because the policy has 2M episodes versus at most 60k reward-model samples.This massive data imbalance makes it difficult to maintain both tasks without reward-model overfitting.
- Ambiguous tasks make labeling hard: Summary evaluation is subjective and multidimensional, requiring labelers to trade off accuracy, grammar, nonredundancy, and coverage across imperfect candidates.The authors note that this makes consistent labeling and rapid detection of problematic labelers difficult.
- Other challenges: A reward-sign bug produced negative-sentiment outputs while preserving natural-language regularization, and the problem was noticed only after training finished.The authors suggest an Andon-cord-like mechanism could allow labelers to stop problematic training earlier.
5. Conclusion
The paper applies reward learning with pretrained language models to four NLP tasks, achieving strong continuation results with few human comparisons but mixed summarization outcomes. It frames direct human preferences as a step toward scalable reward learning for language and AI safety.
- The approach fine-tunes pretrained language models with reinforcement learning and KL regularization across stylistic continuation and two summarization datasets.The tasks include positive-sentiment and physically descriptive continuation, plus TL;DR and CNN/Daily Mail summarization.
- 2.5k for sentiment and 5k for descriptiveness comparisons produced good continuation results versus the zero-shot baseline under human evaluation.
- Summarization policies became “smart copiers” that extract sentences from inputs while skipping irrelevant preamble.The authors contrast this extractive behavior with natural, plausible-looking summaries from zero-shot and supervised models that were often untruthful.
- The authors identify data quality, exacerbated by online data collection, as the limiting factor in their summarization experiments.They plan to investigate batched data collection as a future improvement.
- Direct human preferences for language are presented as a step toward scalable reward learning, including amplification, debate, and recursive reward modeling.Natural language supports interactive communication between humans and models when conveying complex ideas.
A. Instructions for labelers
The labeling instructions define human judgments for sentiment, descriptiveness, and two summarization tasks using task-specific criteria for quality, accuracy, relevance, and self-containment.
- A.1. Sentiment: Sentiment labelers choose the continuation with the most positive and happy mood across characters, narrator, events, and descriptions.
- A.1. Sentiment: Offensive or sexually explicit sentiment continuations are treated as strongly negative even when their attitude is positive.
- A.2. Descriptiveness: Descriptiveness labelers judge both the quantity and vividness of physical details, especially sensory details about how a scene looks, sounds, or smells.
- A.2. Descriptiveness: Descriptiveness judgments count physical details but exclude details about abstract facts.
- A.3. Summarization: TL;DR: TL;DR summaries should be useful, accurately state important points, and make sense independently.
- A.3. Summarization: TL;DR: TL;DR summaries are bad when they include information absent from the source text.
- A.4. Summarization: CNN/DM: CNN/Daily Mail summaries use the same usefulness, accuracy, and self-containment criteria as the TL;DR task.
- A.4. Summarization: CNN/DM: CNN/Daily Mail summaries are bad when they add information absent from the article or use quotations not appearing verbatim.
B. Human labeling details
Human-labeling procedures combined trusted-worker training, benchmark questions, quality checks, and rapid collection, but the resulting data was noisier than author-labeled data.
- The quality-assurance process introduced major changes to Scale AI’s usual systems to handle subjective tasks and provide very fast turnaround.
- Scale AI trained a small group of trusted labelers before collecting benchmark examples and mixing them with real queries for freelance workers.
- 5% of queries received answers from 5 distinct labelers, and authors later evaluated 100 sampled queries generated from ρ.
- Table 9 reports agreement probabilities for two tasks based on whether two individuals selected the same best sample out of 4.
- The collected data contained significant signal but did not match the quality of data hand-labeled by the authors.
- 1-10 ratings were difficult to normalize, vulnerable to simple filtering strategies, and prone to absolute-score drift as sample distributions changed.
- Reported label counts exclude quality-control costs, which can dominate short training runs but be amortized across multiple tasks.
C. Samples
The appendix catalogs generated samples and their associated tables for sentiment, descriptiveness, TL;DR, and CNN/Daily Mail summarization, including examples with and without KL regularization.
- The samples section presents model outputs in tables covering both continuation and summarization tasks.
- The continuation samples include mock sentiment without a KL penalty, positive sentiment, and descriptiveness.
- TL;DR sample tables use randomly sampled contexts from the TL;DR validation set.
- CNN/Daily Mail sample tables use randomly sampled, truncated contexts from the CNN/DM test set with temperatures T = 0.5.
- Table 10 shows that sentiment models without regularization produce gibberish despite an entropy bonus, while both reach about +8.0 in mock-model score.
- The positive-sentiment and descriptiveness tables use contexts randomly sampled from the BookCorpus test set with temperature T = 0.7.