PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining Language Models with Human Preferences

Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, Ethan Perez

arXiv:2302.08582v2cs.CLcs.LG

TL;DR

Language models learn from internet text that may conflict with human preferences, raising the question of how to align them without sacrificing capabilities. The paper benchmarks five human-feedback pretraining objectives across three tasks and finds conditional training to be Pareto-optimal, reducing undesirable content while maintaining downstream performance. It also reports better preference satisfaction than pretraining with MLE followed by feedback finetuning.

  • Problem

    Pretraining language models to imitate internet text can reproduce undesirable content, while post-training alignment methods may not adequately address this problem.

  • Method

    The paper benchmarks five pretraining-with-human-feedback objectives across three alignment and capability tasks, comparing them with maximum likelihood estimation.

  • Results

    Conditional training is Pareto-optimal across all three tasks, reducing undesirable content by up to an order of magnitude while maintaining comparable downstream performance to MLE.

  • Takeaways & Limitations

    The findings support incorporating human preferences during pretraining rather than relying only on finetuning to align language models.

  • Takeaways & Limitations

    All PHF objectives remain vulnerable to adversarial red-teaming, and the resulting models are not completely aligned or safe in all deployment scenarios.

Abstract

from arXiv · show

Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences. We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs. We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model. Conditional training reduces the rate of undesirable content by up to an order of magnitude, both when generating without a prompt and with an adversarially-chosen prompt. Moreover, conditional training maintains the downstream task performance of standard LM pretraining, both before and after task-specific finetuning. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback, i.e., learning and then unlearning undesirable behavior. Our results suggest that we should move beyond imitation learning when pretraining LMs and incorporate human preferences from the start of training.

1. Introduction

The paper argues that aligning language models during pretraining can preserve broad capabilities while reducing undesirable generation, rather than relying mainly on post hoc finetuning or filtering. It evaluates several pretraining-with-human-feedback objectives and identifies conditional training as the strongest overall approach.

  • Language-model pretraining imitates internet content that can include falsehoods, offensive comments, PII, and low-quality code.
  • Filtering undesirable content can handicap capabilities and reduce robustness or amplify social biases, motivating preference-aware pretraining that retains diverse data.
  • PHF uses reward functions to estimate human preferences while allowing models to learn from undesirable content without imitating it at inference time.
  • The study benchmarks five PHF objectives across nontoxic generation, PII avoidance, and PEP8-compliant Python, comparing alignment with downstream capabilities.
  • Conditional training lies on the Pareto frontier across all three tasks and reduces undesirable content by up to an order of magnitude while preserving comparable downstream performance to MLE.
  • PHF provides equal or sometimes dramatically better alignment than MLE pretraining followed by feedback finetuning, with improved adversarial robustness across the tasks.

2. Methods

The methods define pretraining objectives over document segments scored by a reward function, including standard MLE and five alternatives that incorporate human-preference signals. These alternatives range from filtering and conditional control tokens to loss reweighting and unlikelihood training.

  • PHF assumes a segment-level reward function R that assigns each document segment a scalar preference score.
  • MLE is the standard objective, maximizing the log likelihood of training documents and serving as the baseline for the PHF objectives.
  • Dataset filtering keeps the MLE objective but sets it to zero for documents whose average reward falls below threshold t.
  • Conditional training prepends reward-derived control tokens to segments, then samples at inference using the good token.
  • Unlikelihood training maximizes likelihood for segments above threshold t and applies token-level unlikelihood to lower-reward segments.
  • RWR reweights segments by exponentiated reward, while AWR additionally subtracts token-level value estimates and jointly trains a shared value head.

3. Experimental Setup

The experiments compare MLE and human-feedback pretraining objectives across toxicity, PII, and PEP8-compliant code tasks, measuring alignment alongside capabilities.

  • Tasks and evaluation: PHF objectives are compared with MLE on alignment and downstream capabilities across three tasks.The setup covers offensive-content avoidance, PII leakage avoidance, and Python code following PEP8.
  • Tasks and evaluation: Toxicity is measured with Detoxify, using sentence-level training scores and sample-level evaluation scores.The reward is the negative toxicity probability, while the misalignment score is the toxicity probability.
  • Tasks and evaluation: PII is detected with pattern matching and named-entity recognition, with reward defined as negative detected PII instances per character.Training documents are scored at sentence level.
  • Tasks and evaluation: PEP8 alignment is evaluated with pycodestyle, using the negative number of violations per character as the reward.Rewards are assigned to individual lines, although violations can depend on previous lines.
  • Model and training setup: All models use the 124M-parameter gpt2-small architecture, with learning rate and batch size tuned for each task-objective pair.Objective-specific hyperparameters are also tuned, and configurations are selected using misalignment and KL divergence from GPT-3.
  • Model and training setup: Training uses 3.32B tokens, drawn from the Pile for toxicity and PII and from cleaned GitHub Python files for code generation.The data comprise 1.95M Pile documents or 1.5M Python files, depending on the task.

4. Pretraining Experiments

The experiments evaluate PHF objectives by measuring alignment, capabilities, adversarial robustness, downstream performance, and diversity. Conditional training provides the strongest overall trade-off, substantially reducing misalignment while largely preserving capabilities, though adversarial vulnerabilities remain.

  • Capabilities-Alignment Trade-offs: PHF reduces undesirable content, with conditional pretraining lowering toxicity misalignment from 0.0141 for MLE to 0.0011.The reduction persists for right-tail misalignment metrics, and conditional training shifts the right tail furthest left.
  • Capabilities-Alignment Trade-offs: Conditional training offers the best alignment–capabilities trade-off, remaining on the Pareto frontier across toxicity, PII, and PEP8.It is strictly Pareto-optimal for toxicity and on the frontier for PII and PEP8; filtering performs better on PEP8 but incurs larger capability penalties on PII and PEP8.
  • Robustness to Red-Teaming: After ten red-teaming rounds, conditional training outperforms MLE by up to an order of magnitude on toxicity and PII.All PHF models except unlikelihood-trained PII models are significantly more robust than MLE, although adversarial pressure continues increasing misalignment for every objective.
  • Downstream Benchmarks: Conditional training generally matches MLE on GLUE and zero-shot performance, but filtering, AWR, and RWR can outperform it on HumanEval pass@100.Unlikelihood performs particularly poorly on toxicity and consistently obtains the lowest scores on HumanEval.
  • Diversity: Unlikelihood, AWR, and RWR retain MLE-like diversity but show slightly more degeneration, whereas conditional training reduces diversity while better matching MLE’s distinct-unigram fraction.No PHF objective causes significant degeneration or entropy collapse in absolute terms.

5. Finetuning with Human Feedback

The paper compares pretraining with human feedback against applying feedback only after MLE pretraining. Across the reported evaluations, pretraining with feedback achieves stronger alignment, especially when feedback is available throughout training.

  • PHF achieves scores that are always better, typically dramatically better, than finetuning with feedback.
  • In PII, conditional pretraining reaches misalignment score 0.0013 compared to 0.0018 for finetuning on 300M tokens.The paper describes pretraining as two to three times more effective in this comparison.
  • The comparison includes unusually large finetuning allocations of 50% or 10% of the training budget, enabling interpolation between pretraining and finetuning.
  • Pretraining with feedback for the whole time is always better than using feedback with only the final 330M tokens and tends to be better than using it for the final 1.6B tokens.
  • The gap between PHF and finetuning with feedback widens as fewer tokens are available for finetuning.
  • PHF-trained LMs are significantly more robust to red-teaming than LMs only finetuned with conditional training on toxicity and PII.On PII, ten red-teaming rounds are required for PHF-trained LMs to reach the misalignment score of a finetuned LM after one iteration.

6. Related Work

Related work primarily aligns language models after MLE pretraining, while this paper’s setting connects pretraining with human feedback to offline reinforcement learning. The broader literature also includes filtering, supervised finetuning, RLHF, and divergence-based objectives.

  • Most prior alignment work adjusts pretrained language models through safety filters, supervised finetuning, or reinforcement learning from human feedback.
  • Offline RL: The paper’s training setting is closely related to offline reinforcement learning, which learns policies from demonstrations annotated with rewards.
  • LM alignment during finetuning: Language-model alignment research has also explored divergence from a target distribution and supervised finetuning on generated or highly curated instruction data.

7. Conclusion

The conclusion advocates incorporating human feedback during pretraining rather than relying on finetuning alone. Conditional training generally offers the strongest combined alignment and capability profile, while retaining practical overhead from reward-model annotation.

  • Conditional training consistently outperforms the other PHF objectives in both capabilities and alignment, with two task-specific exceptions.Unlikelihood is more robust to toxicity red-teaming, while filtering achieves better HumanEval results.
  • Conditional training tends to match MLE’s capabilities while providing much better alignment.
  • The paper advocates using human feedback during pretraining itself instead of aligning language models only during finetuning.
  • PHF requires annotating training data with a reward model, although reward-model inference has low computational cost relative to total pretraining.

A. Hyperparameters and Implementation Details

The appendix specifies how conditional training, filtering, red-teaming, and evaluation were implemented, including threshold ablations and adversarial-prompt procedures. It also documents a vocabulary-extension issue during finetuning.

  • Implementation: Conditional training prepends <|good|> when R(xi) ≥ t and <|bad|> otherwise, with 1% of sentences receiving no control token.The intervention slightly improved capabilities measured by KL from GPT-3, with a negligible alignment penalty.
  • Implementation: During unconditional generation, models are prompted with <|endoftext|><|good|>, while special-token blocking differs across toxicity, PII, and PEP8.For PEP8, <|good|> tokens may be generated between lines and are removed afterward.
  • Finetuning: Conditional-training finetuning requires extending the pretrained vocabulary with <|good|> and <|bad|> embeddings initialized from the mean embedding plus ϵ = 0.01 Gaussian noise.A notable drop in alignment and capabilities remains visible during the first 100m finetuning tokens after introducing the new tokens.
  • Threshold ablation: The threshold t is set to a percentile of document-level rewards; filtering discards documents below t and trains for multiple epochs at a fixed token budget.Figure 8 varies t, with higher thresholds filtering more data or prepending <|good|> to fewer segments.
  • Red-teaming: Red-teaming evaluates robustness over ten rounds using adversarial prompts generated by a few-shot red LM and target-LM responses sampled at T = 0.7 and top-p = 0.9.The tracked measures include average and maximum misalignment scores for prompts found in the pool or generated in each round.

C. Details on GLUE evaluation

The GLUE evaluation covers eight tasks, uses shared transferred hyperparameter settings with early stopping and three random restarts, and reports aggregated test performance. Toxicity models using conditional training achieve an average of 71.4 versus 72.1 for MLE.

  • Evaluation setup: The evaluation selects eight GLUE tasks: CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, and RTE.WNLI is omitted following prior work.
  • Evaluation setup: Each GLUE experiment trains for up to 6 epochs with early stopping, using three random restarts and hyperparameters transferred from toxicity MLE-pretrained models.Learning rates are swept over {5e-4,1e-4,5e-5,2e-5}, and batch sizes over {32,64,128}.
  • Scoring: Test scores average task metrics, MNLI matched and mismatched results, and three finetuning seeds, with standard deviations reported.STS-B predictions are clipped to [0,5] for GLUE submission formatting.
  • Results: Toxicity models pretrained with conditional training score 71.4±0.6 average GLUE performance versus 72.1±0.74 for MLE.Conditional training is close to MLE overall, although individual task scores differ.
  • Results: Tables 8 and 9 report selected GLUE test-set results for toxicity and PII models pretrained with six objectives.The appendix also identifies OpenAI-pretrained and randomly initialized GPT-2 as baseline models.

D. Additional results on scores of LM samples

Additional analyses characterize misalignment beyond average scores by examining empirical score distributions, expected maxima, and threshold exceedance rates across LM samples.

  • Sample-level scores: The appendix reports empirical distributions of misalignment scores over 10240 samples.This complements aggregate alignment metrics with the spread of sample-level outcomes.
  • Sample-level scores: Expected maximum misalignment is defined as the maximum score expected among 25 LM samples, with lower values preferred.The metric emphasizes the likelihood of encountering an especially misaligned sample.
  • Threshold exceedance: The analysis measures the fraction of LM samples exceeding toxicity and PEP thresholds and the average toxicity misalignment score under challenging RealToxicityPrompts.These evaluations condition toxicity generation on adversarially challenging prompts.

E. Additional results for diversity evaluation

The diversity appendix compares PHF-pretrained models with MLE using entropy, Self-BLEU, and distinct-n-gram measures, alongside degeneration metrics.

  • Diversity and degeneration: Relative to MLE, PHF models are evaluated on unigram entropy, bigram entropy, Self-BLEU-5, distinct unigrams, and distinct bigrams.Higher entropy and distinct-n-gram values are preferred, while lower Self-BLEU-5 indicates greater diversity.

F. Additional results for finetuning experiments

The finetuning experiments compare KL from GPT-3 and average misalignment across five PHF objectives and three tasks. They also track these metrics over training time and compare MLE and conditional finetuning under different pretraining conditions.

  • Evaluation setup: The experiments evaluate KL from GPT-3 and average misalignment score, with lower values preferred for both metrics.The first-column scatter plot compares the two metrics, while the second and third columns show each metric over training time.
  • Evaluation setup: The figures compare models pretrained with MLE up to 1.6B tokens and then finetuned using each of five PHF objectives across three tasks.This setup examines how the finetuning objectives affect the two reported metrics.
  • Stage comparisons: Misalignment is compared after pretraining and after finetuning with the same objective, using MLE pretraining as the reference condition.Figure 17 presents average misalignment for these stages.
  • Stage comparisons: Training-time comparisons contrast MLE finetuning after conditional pretraining with conditional finetuning after MLE pretraining, both using 1.6B-token pretrained models.Figure 18 marks these conditions with dashed and dotted lines, respectively.
Loading 2302.08582v2…