PaperlayerView exact revision on arXiv

Source-linked AI summary

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe

arXiv:2203.02155v1cs.CLcs.AIcs.LG

TL;DR

Language models often fail to follow user intent and can produce untruthful, toxic, or unhelpful outputs. The paper fine-tunes GPT-3 with demonstrations and human preference rankings, yielding InstructGPT models that are preferred by labelers and improve truthfulness while reducing toxicity.

  • Problem

    Language models often exhibit unintended behaviors, including fabrication, toxicity, bias, and failure to follow user instructions.

  • Method

    The authors fine-tune GPT-3 using supervised demonstrations and reinforcement learning from human feedback based on rankings of model outputs.

  • Results

    InstructGPT outputs are preferred over GPT-3 outputs and show improved truthfulness and reduced toxicity across the evaluated tasks.

  • Takeaways & Limitations

    Fine-tuning language models with human preferences significantly improves behavior across a wide range of tasks, although safety and reliability remain incomplete.

  • Takeaways & Limitations

    The models are not fully aligned or safe and still produce toxic or biased outputs, fabricate facts, and generate sexual or violent content without explicit prompting.

Abstract

from arXiv · show

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

1 Introduction

The paper aligns GPT-3 with user intent by fine-tuning on human demonstrations and preferences through supervised learning, reward modeling, and PPO-based RLHF. InstructGPT improves instruction following, truthfulness, toxicity, and broad task behavior, while retaining limitations and some performance costs.

  • RLHF aligns GPT-3 through human demonstrations, a reward model trained on output comparisons, and PPO optimization, producing models intended to be helpful, honest, and harmless.The procedure reflects the stated preferences of labelers and researchers rather than a broader notion of human values.
  • 1.3B InstructGPT outputs are preferred to 175B GPT-3 outputs despite over 100x fewer parameters, while 175B InstructGPT wins 85 ± 3% against GPT-3 and 71 ± 4% against few-shot GPT-3.The models share the GPT-3 architecture; their key difference is InstructGPT’s fine-tuning on human data.
  • InstructGPT generates truthful and informative TruthfulQA answers about twice as often as GPT-3 and hallucinates 21% versus 41% on closed-domain API tasks.The truthfulness improvement remains equally strong on TruthfulQA questions not adversarially selected against GPT-3.
  • InstructGPT produces about 25% fewer toxic outputs than GPT-3 when prompted to be respectful, but shows no significant improvement on Winogender or CrowSPairs bias evaluations.The toxicity finding uses both automatic and human evaluations on RealToxicityPrompts.
  • PPO-ptx greatly reduces RLHF performance regressions on SQuAD, DROP, HellaSwag, and WMT 2015 French-to-English translation without compromising labeler preference scores.The method mixes PPO updates with updates increasing the log likelihood of the pretraining distribution.
  • InstructGPT generalizes to held-out labelers and rare instruction domains, but still fails on some instructions, fabricates facts, and requires further safety and reliability improvements.On the API prompt distribution, InstructGPT is preferred to T0 and FLAN, with a 73.4 ±2% winrate versus 26.8 ±2% and 29.8 ±2%, respectively.

2 Related work

This section situates the work within reinforcement learning from human feedback, instruction-following research, and efforts to evaluate and mitigate language-model harms. It presents the work as applying RLHF to align language models across a broad distribution of language tasks.

  • Research on alignment and learning from human feedback: The work directly applies RLHF to aligning language models across a broad distribution of language tasks, building on applications in robotics, games, summarization, and other domains.Prior work used human feedback for dialogue, translation, semantic parsing, story and review generation, evidence extraction, and prompt augmentation.
  • Research on alignment and learning from human feedback: Alignment research has characterized harmful content and objective gaming as misalignment, while language-assistant studies investigate alignment testbeds, baselines, and scaling properties.These efforts address what alignment means and how misalignment manifests behaviorally.
  • Training language models to follow instructions: Instruction-tuning studies find that fine-tuning language models on diverse instructed tasks improves zero-shot and few-shot performance on held-out tasks.Related work differs in training and evaluation data, instruction formats, pretrained-model size, and other experimental details.
  • Evaluating the harms of language models: Language-model harms include biased outputs, private-data leakage, misinformation, malicious use, and domain-specific risks, motivating benchmarks for toxicity, stereotypes, and social bias.Interventions can have side effects; reducing toxicity may impair modeling of under-represented groups because of prejudicial correlations in training data.
  • Modifying the behavior of language models to mitigate harms: Prior mitigation strategies include targeted fine-tuning, data filtering, generation blocking, control tokens, human-in-the-loop collection, bias regularization, data augmentation, objective changes, and auxiliary-model steering.These approaches can reduce harmful text or bias, but some incur costs such as decreased language-modeling performance or broader behavioral tradeoffs.

3 Methods and experimental details

The method combines supervised fine-tuning, reward modeling from human comparisons, and PPO optimization, applied to diverse prompts with trained labelers. Evaluation measures helpfulness, truthfulness, and harmlessness through preference ratings and targeted proxy benchmarks.

  • 3.1 High-level methodology: The three-step pipeline fine-tunes GPT-3 on demonstrations, trains a reward model from preferred outputs, and optimizes the policy against that reward with PPO.Steps 2 and 3 can be iterated by collecting comparisons from the current best policy and retraining the reward model and policy.
  • 3.2 Dataset: The datasets combine API-submitted and labeler-written prompts, yielding about 13k SFT, 33k RM, and 31k PPO training prompts.Prompts are deduplicated, capped at 200 per user, split by user ID, and filtered for personally identifiable information in training.
  • 3.3 Tasks: Tasks span generation, question answering, dialogue, summarization, extraction, and other natural-language tasks, with labelers inferring user intent from instructions, examples, or continuations.The dataset is over 96% English, while labelers also consider truthfulness and potentially harmful outputs when judging responses.
  • 3.4 Human data collection: Training and evaluation used about 40 screened contractors, while held-out labelers provided an independent generalization check.Agreement was 72.6 ± 1.5% among training labelers and 77.3 ± 1.3% among held-out labelers.
  • 3.5 Models: Models include SFT, 6B reward models, PPO, and PPO-ptx, with PPO-ptx mixing pretraining gradients into PPO to address public-NLP performance regressions.PPO uses a per-token KL penalty from the SFT model, and InstructGPT refers to PPO-ptx unless otherwise specified.
  • 3.6 Evaluation: Evaluation uses human preference ratings on held-out API prompts plus benchmarks of truthfulness, toxicity, bias, and zero-shot traditional NLP performance.Truthfulness is measured through hallucination tendencies and TruthfulQA, while harmfulness uses specific judgments and datasets including RealToxicityPrompts and CrowS-Pairs.

4 Results

Across API prompts, InstructGPT is strongly preferred over GPT-3 and improves instruction-following, truthfulness, and control, while public-dataset regressions can be mitigated with pretraining updates. The models also generalize qualitatively beyond their fine-tuning distribution, though they retain several simple failure modes.

  • 4.1 Results on the API distribution: 175B InstructGPT outputs were preferred to GPT-3 outputs 85 ± 3% of the time and to few-shot GPT-3 outputs 71 ± 4% of the time, with gains increasing from prompting through SFT to PPO.These preferences held across model sizes, and PPO-ptx produced similar results except for slightly worse performance at larger sizes.
  • 4.1 Results on the API distribution: InstructGPT generalized to held-out labelers and was preferred over FLAN and T0 78 ± 4% and 79 ± 4% of the time, respectively.Held-out labelers retained the same preference pattern, while FLAN and T0 performed better than default GPT-3 but worse than the SFT baseline.
  • 4.2 Results on public NLP datasets: On TruthfulQA, PPO models were small but significantly more truthful and informative than GPT-3, including without explicit truth-seeking instructions, while handling uncertainty more honestly under Instruction+QA prompting.The 1.3B PPO-ptx exception performed slightly worse than same-size GPT-3, and improvements remained significant on non-adversarial prompts.
  • 4.2 Results on public NLP datasets: InstructGPT generated less toxic outputs than GPT-3 with a respectful prompt, but this advantage disappeared without one; its models were not less biased than GPT-3.When explicitly prompted to produce toxic output, InstructGPT outputs were much more toxic than GPT-3 outputs.
  • 4.2 Results on public NLP datasets: Adding pretraining updates during PPO mitigated alignment-tax regressions across public NLP datasets and surpassed GPT-3 on HellaSwag, although performance still lagged on DROP, SQuADv2, and translation.The pretraining mix reversed regressions on SQuADv2 and DROP with minimal validation-reward reductions, unlike increasing the KL coefficient.
  • 4.3 Qualitative results: InstructGPT generalized to non-English instruction following and code summarization or question-answering, but still sometimes answered in English and made simple mistakes with false premises, hedging, or multiple constraints.These behaviors were observed qualitatively outside the fine-tuning distribution; the authors suggest adversarial data collection could reduce some failures.

5 Discussion

The discussion presents RLHF as a practical, relatively low-cost alignment approach while emphasizing that it aligns only to specific, nonrepresentative preference sources. It also identifies persistent safety limitations, open methodological questions, and broader risks from misuse and deployment.

  • 5.1 Implications for alignment research: RLHF provides an empirical feedback loop for improving current systems, costs far less than pretraining GPT-3, and can improve helpfulness more than a 100x model-size increase.The authors argue that alignment investment may currently be more cost-effective than scaling models, while noting evidence of generalization beyond supervised tasks and limited performance regressions.
  • 5.2 Who are we aligning to?: The models align to preferences shaped by labelers, researchers, customers, and end users rather than to a universally representative account of human values.Labelers were mostly English-speaking people in the United States or Southeast Asia, had about 73% inter-labeler agreement, and customers were not representative of all affected groups.
  • 5.3 Limitations: The methodology is limited by a small, culturally narrow contractor pool and sparse labeling, while the models remain capable of toxic, biased, fabricated, sexual, and violent outputs.Most comparisons were labeled by one contractor, and InstructGPT can follow harmful instructions; under maximally biased prompts it can produce more toxic outputs than equivalently sized GPT-3 models.
  • 5.4 Open questions: Open questions include reducing harmful outputs, handling context-dependent refusal, improving alignment signals and algorithms, and addressing residual regressions from pretraining-data mixing.The authors also call for transparent, participatory processes that distinguish instructions, intentions, preferences, interests, and values while representing affected groups more broadly.
  • 5.5 Broader impacts: Alignment may improve helpfulness and safety but also facilitate convincing misinformation and abuse, so deployment requires broader safeguards, careful domain restrictions, and attention to centralized control.The paper highlights high-stakes domains where deployment may be inappropriate and notes that API-based safety controls trade transparency against misuse prevention; who models are aligned to affects net impact.

A Additional prompt data details … B.2 Labeling instructions

The appendix details how InstructGPT’s prompt and labeling data were constructed, organized, and governed, covering prompt sources, diversity, labeler screening, and evolving instructions. It also discusses training-data scale and unresolved issues around refusals and over-refusal.

  • A.1 Labeler-written prompts: Labelers wrote plain, few-shot, and user-based prompts to train the first instruction-following model before external users supplied prompts.Few-shot prompts converted K query-response pairs into K training examples using the other K−1 pairs as context.
  • A.1 Labeler-written prompts: Application information was anonymized into vague high-level tasks, then used to supervised-train the first InstructGPT model deployed in API beta in early 2021.A separate labeler removed information specific to individual applications before training.
  • A.2 API user prompts: API prompts came from users of the earlier InstructGPT model in the OpenAI Playground, with consent notices and personally identifiable information filtered from training data.The study used Playground prompts rather than production-customer data because informed consent was easier to obtain.
  • A.2 API user prompts: Prompts were heuristically deduplicated, capped at roughly 200 per organization, and split by organization ID so validation use cases differed from training use cases.These procedures targeted diversity while reducing overlap between splits.
  • A.2 API user prompts: API requests were grouped into ten use cases: generation, open QA, closed QA, brainstorming, chat, rewriting, summarization, classification, extraction, and other.The appendix provides fictional but realistic examples spanning these categories.
  • A.2.1 Illustrative user prompts from InstructGPT distribution: InstructGPT-distribution examples include brainstorming, classification, extraction, generation, rewriting, chat, closed QA, open QA, and summarization prompts.Examples include sentiment classification, extracting course titles, translation, navigation, conversational roles, factual questions, and audience-specific summarization.
  • A.2.2 Illustrative user prompts from GPT-3 distribution: The cited examples span classification, generation, chat with an AI assistant, open question answering, and summarization across several task categories.They include tweet sentiment classification, a story-generation prompt, an AI-assistant conversation, a Batman question with an answer, and a passage-meaning summarization prompt.
  • A.2.2 Illustrative user prompts from GPT-3 distribution: The prompt examples include requests to rephrase or summarize text in simple language alongside repetitive user statements, illustrating the broad GPT-3 prompt distribution.The excerpts include a plain-language rephrasing request, a summarization request, and repeated statements about playing Call of Duty.
  • A.3 Dataset sizes: Labeler-written and API-submitted prompts formed the datasets for SFT, reward modeling, and reinforcement learning, with dataset sizes reported by source and training stage.SFT included synthetically expanded examples from few-shot templates, while reward-model training used rankings over 4–9 outputs per prompt, producing an order of magnitude more ranked pairs than prompts.
  • A.4 Data diversity: The collected data covered diverse categories and use cases, with annotations, customer contributions, prompt lengths, and demonstration lengths summarized across multiple tables.Annotation fields changed during the project, so not every prompt was annotated for every field.
  • A.4 Data diversity: Around 96% of the dataset, corresponding to 110k datapoints, was classified as English, while a small minority spanned at least 20 other languages.The authors estimated the actual English fraction could be 99% or higher because of classifier inaccuracies.
  • B.1 Labeler selection: Labelers were screened for sensitivity detection, ranking agreement, nuanced sensitive-demonstration writing, and self-assessed coverage across topics and cultural groups.Final selection used these criteria subjectively, with soft cutoffs of 75% agreement and a 6/7 demonstration score.
  • B.2 Labeling instructions: Labeling instructions evolved with feedback, metadata changes, and improved measurement goals, and were amended when confusing or inconsistent.Training labelers prioritized helpfulness above truthfulness and harmlessness, whereas final evaluations prioritized truthfulness and harmlessness.
  • B.2 Labeling instructions: The authors considered configurable refusals for higher-risk applications but warned that models might over-generalize and refuse innocuous instructions.Evaluation instructions for the main prompt distribution and RealToxicityPrompts were excerpted in Tables 10 and 11.

B.3 Labeler demographic data … C Additional model details

The appendix characterizes the labelers, their task experience, evaluation instructions, labeling interface, and model-training configurations. It reports generally positive labeler feedback alongside demographic and implementation details.

  • B.3 Labeler demographic data: Labelers evaluated API prompts as explicit instructions, indirect task descriptions, or partial outputs indicating the desired continuation.The instructions included examples such as writing a story, predicting sentiment, and continuing a story.
  • B.3 Labeler demographic data: For toxicity evaluation, labelers judged whether models continued potentially rude, sexual, or violent prompts in a respectful, plausible, and non-toxic manner.The complete toxicity instructions covered the RealToxicityPrompts distribution.
  • B.3 Labeler demographic data: 75% of the 19 surveyed labelers were younger than 35, with fairly balanced genders and most based in the US or Southeast Asia.The demographic survey was voluntary and anonymous.
  • B.4 Labeler satisfaction survey: Labelers generally enjoyed the task, considered their compensation fair, and valued the researchers’ helpfulness and communication, although some found the work repetitive.Others reported sufficient variation to keep the task interesting and engaging.
  • B.5 Web interface: The labeling interface combines 1–7 Likert quality scores and metadata labels with rankings of all outputs for each prompt, while encouraging ties for similar-quality outputs.All labelers and researchers used this interface.
  • C Additional model details: All models use the GPT-3 architecture, fp16 weights and activations with fp32 master weights, Brown et al. (2020) byte-pair encodings, and scalar projection heads for reward models and value functions.Language models and RL policies use 2k-token contexts; prompts over 1k tokens are filtered and responses capped at 1k tokens.
  • C Additional model details: All models are trained with Adam using β1 = 0.9 and β2 = 0.95.

C.1 Details of SFT training

The SFT models were trained with fixed optimization settings tailored by model size. Final checkpoints were selected using reward-model scores because they better predicted human preferences than validation loss.

  • C.1 Details of SFT training: SFT training used 16 epochs, residual dropout of 0.2, cosine learning-rate decay to 10% without warmup, and model-size-specific learning rates and batch sizes.The 1.3B and 6B models used learning rates of 9.65e-6 and batch size 32, while the 175B model used 5.03e-6 and batch size 8.
  • C.1 Details of SFT training: Learning rates and epoch counts were tuned through geometric search over seven learning rates for 1.3B and 6B, and five for 175B.
  • C.1 Details of SFT training: Final models were selected by reward-model score, which was more predictive of human preference results than validation loss.

C.2 Details of RM training … C.5 FLAN and T0 models

The authors use a shared 6B reward model and carefully specified supervised and PPO training procedures, including pretraining-data mixing to mitigate task-performance regressions. FLAN and T0 baselines are trained on matched data budgets and selected using reward-model scores.

  • C.2 Details of RM training: A single 6B reward model was used for PPO policies of every size because it was stable, computationally cheaper, and produced equally strong PPO models.Although larger 175B reward models could achieve lower validation loss, they were less stable and substantially increased PPO compute requirements.
  • C.2 Details of RM training: The reward model was trained for one epoch at lr = 9e-6 with batch size 64; multiple epochs overfit, whereas learning-rate changes up to 50% had similar performance.It was initialized from a 6B GPT-3 model fine-tuned on public NLP datasets, though GPT-3 and SFT initialization gave similar results.
  • C.3 Details of the initialization models for RLHF: RLHF models were initialized from GPT-3 with two supervised fine-tuning epochs and 10% pretraining data mixed in, using peak learning rates of 5e-6, 1.04e-5, and 2.45e-6 for 1.3B, 6B, and 175B models.Learning rates were selected by comparing validation losses on demonstration and pretraining data.
  • C.4 Details of RLHF training: PPO training ran for 256k episodes over about 31k filtered, deduplicated prompts, with batch size 512, minibatches of 64, a 0.2 clip ratio, and rollout temperature 1.Policies used a KL reward with β = 0.02, warmup over 10 iterations, exponential moving averages with decay 0.992, and no discount for generalized advantage estimation.
  • C.4 Details of RLHF training: All PPO policies used the same 6B reward model and value function, enabling comparisons across policy sizes; value-function learning rates were 9e-6 for 1.3B and 6B and 5e-6 for 175B.The value function was initialized from the reward model.
  • C.4 Details of RLHF training: Pretraining gradients were mixed into PPO because initial RLHF caused regressions on SQuADv2 and DROP; eight times as many pretraining examples as RL episodes were used with γ = 27.8.PPO and pretraining gradients were computed consecutively for each minibatch and accumulated together.
  • C.5 FLAN and T0 models: FLAN and T0 baselines fine-tuned 175B GPT-3 on comparable one-million-example datasets, using T0++ subsampling, no repeated datapoints, cosine schedules, and batch size 64.T0 originally contained 96M datapoints versus FLAN’s 1.2M, so T0 was subsampled to 1M.
  • C.5 FLAN and T0 models: Reward-model scoring selected FLAN after 896k examples at lr = 4e-6 and T0 after 896k examples using the experiment with batch size 128, lr = 4e-6, and 1.28M examples.FLAN reward saturated after the initial 400k examples, while the selected T0 checkpoint came from the former of two experiments.

D Automatic evaluation details … E Additional results

The paper evaluates models across diverse public NLP, bias, toxicity, truthfulness, reasoning, question-answering, classification, translation, and summarization tasks using standardized prompting and scoring procedures. Additional zero-shot results show regressions for 175B PPO models, mitigated by incorporating pretraining-data updates during fine-tuning.

  • D Automatic evaluation details: Automatic evaluations cover 13 benchmarks spanning bias, toxicity, truthfulness, reasoning, question answering, classification, translation, and summarization.The datasets include Winogender, CrowS-Pairs, RealToxicityPrompts, TruthfulQA, DROP, QuAC, SQuAD v2, HellaSwag, SST, RTE, WSC, WMT15 Fr→En, CNN/Daily Mail, and Reddit TLDR.
  • D Automatic evaluation details: Tasks generally combine an optional shared instruction, query-specific context, and either sampled completions or multiple-choice outputs under standardized decoding and few-shot formatting.Sampling uses temperature T = 0 with truncation at the first newline; multiple-choice answers use the lowest average per-token log probability at T = 1.
  • D.1 Toxicity and bias evaluation details: Bias and toxicity evaluations compare basic, respectful, and biased prompt instructions to test how prompting affects model behavior.The respectful instruction requests polite, respectful, unbiased completions, whereas the biased instruction requests maximally biased and offensive completions.
  • D.1 Toxicity and bias evaluation details: Bias-task results use choice entropy, where higher entropy indicates greater uncertainty or less preference among available options.Entropy is measured in bits as H = −Σ_i∈choices P_i log2 P_i, with maximum binary-choice entropy equal to 1.
  • D.2 Prompt structure and evaluation features for each eval dataset: Evaluation details specify each dataset’s prompting structure, validation-example count, and performance metric, with task-specific examples spanning question answering, sentiment, coreference, translation, and summarization.The appendix organizes these features in Tables 14–25 and illustrates them with dataset-specific figures and prompts, including QuAC’s variable question-answer format and SST’s positive-versus-negative sentiment labels.
  • D.2 Prompt structure and evaluation features for each eval dataset: The task prompts preserve each benchmark’s intended information source and response format, including passage-based answering, common-sense completion, pronoun resolution, and French-to-English translation.Examples instruct models to answer from preceding passages, complete independent paragraphs using common-sense reasoning, identify pronoun antecedents, or translate French sentences.
  • E Additional results: 175B PPO models consistently regress on zero-shot public NLP performance, while adding pretraining-data updates during fine-tuning mitigates these regressions.The comparison covers the public NLP datasets shown in Figure 28; few-shot results are presented separately in Figure 29.

E.1 Performance on public NLP datasets · E.2 Reward model generalization across sets of labelers

Across public NLP evaluations, PPO-ptx mitigates the regressions seen with PPO without pretraining mix, especially few-shot, while reward models generalize to held-out labelers from the training pool. The evaluations cover bias, toxicity, truthfulness, and broader language capabilities, with cross-validation accuracies quantifying labeler generalization.

  • E.1 Performance on public NLP datasets: PPO-ptx mitigates the public-dataset performance regressions of PPO without pretraining mix, particularly in few-shot evaluation.The comparison includes zero-shot and few-shot performance across various public NLP datasets.
  • E.1 Performance on public NLP datasets: The public evaluations jointly measure bias, toxicity, truthfulness, and a variety of natural language capabilities.Results are reported in Table 14.
  • E.1 Performance on public NLP datasets: PPO without pretraining mix exhibits regressions on many public NLP datasets.The regressions are especially pronounced in the few-shot setting.
  • E.1 Performance on public NLP datasets: The models’ zero-shot public-dataset performance is shown separately from their few-shot performance.Figure 28 presents zero-shot results, while Figure 29 presents few-shot results.
  • E.2 Reward model generalization across sets of labelers: 72.4±0.4% inter-group and 69.6±0.9% intra-group validation accuracy show reward models generalize to held-out labelers from the training set.These accuracies predict the human-preferred output in five-fold cross-validation.
  • E.2 Reward model generalization across sets of labelers: The reward-model generalization experiment trains on four of five labeler groups and validates on the held-out group using the same predefined hyperparameters.Each labeler group contains roughly the same amount of comparison training data, and the reward model has 6B parameters.

E.3 Metadata results as a function of model size · E.4 Likert scores · E.5 Measuring bias

The paper reports metadata ratings across model types and sizes, Likert scores across models, and bias evaluations on Winogender and CrowS-Pairs. Likert results largely track preference results, while InstructGPT does not significantly improve over GPT-3 on the bias datasets.

  • E.3 Metadata results as a function of model size: Metadata results are presented as a function of model size.The results appear in Figure 30.
  • E.3 Metadata results as a function of model size: Figure 30 presents metadata ratings as a function of model type.
  • E.4 Likert scores: Likert scores are reported for each model on the prompt distribution.The results are shown in Figure 31.
  • E.4 Likert scores: The Likert results largely track the preference results reported in Section 4.1.
  • E.5 Measuring bias: InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs bias datasets.These results are shown in Figure 32.
  • E.5 Measuring bias: Figure 32 presents the paper’s Winogender and CrowS-Pairs results.

E.6 Fixing regressions on public NLP datasets … E.10 RealToxicityPrompts results as a function of input toxicity

The experiments identify pretraining-data mixing and coefficient tuning as key to recovering public-task performance while preserving alignment quality. They also characterize PPO sensitivity to training duration, initialization, learning rate, and input toxicity.

  • E.6 Fixing regressions on public NLP datasets: A pretraining loss coefficient of at least 20 recovers 1.3B public-task regressions, while 27.8 works across model sizes with limited validation-reward cost.Increasing the coefficient lowers validation reward, but human Likert scores are insensitive to its exact value.
  • E.6 Fixing regressions on public NLP datasets: Increasing the KL reward coefficient to 2.0—100 times its default—does not fix public-task regressions, showing pretraining-data distribution is critical.The experiment uses a 1.3B model with the pretraining loss coefficient set to 0; excessively large KL coefficients also sharply reduce validation reward.
  • E.6 Fixing regressions on public NLP datasets: Training for 512k rather than 256k episodes causes DROP and SquadV2 performance to fall slightly below the GPT-3 baseline after initially exceeding it.This result uses the default PPO method with a pretraining mix and three random seeds on the 1.3B model.
  • E.7 Optimal KL reward coefficient: Human Likert performance is poor at KL reward coefficients 0 and 2, with the optimum around 0.01–0.02.The result is measured with PPO using the pretraining-data mix.
  • E.8 PPO init models: Among PPO initialization variants, the only standout uses a 10% pretraining-data mix; the selected initialization trains on demonstrations for two epochs with that mix.PPO performance otherwise appears insensitive to the tested epoch and pretraining-mixture choices.
  • E.9 Learning rate optimization for PPO models: Without pretraining-data mixing, PPO runs with learning rates above 8.05e-6 diverge, whereas mixing makes performance less sensitive to learning-rate changes.The scan covers 2.55e-6 to 2.55e-5 for 1.3B and 6B models; final models use checkpoints with the highest Likert scores.
  • E.10 RealToxicityPrompts results as a function of input toxicity: RealToxicityPrompts output toxicity is highly correlated with input-prompt toxicity, so evaluation averages toxicity over 5,000 prompts approximately uniformly sampled across prompt toxicity.Toxicity is measured with the Perspective API to better capture behavior in unsafe regimes.

E.11 Additional ablations

Additional ablations selected a pretraining data ratio of 8 as a compromise between training speed and pretraining-loss performance, and selected batch and minibatch sizes using human evaluations and GPU utilization. The experiments also found limits on PPO training duration and showed toxicity depended strongly on the instruction given.

  • Pretraining data ratio: A pretraining data ratio of 8 doubled training time without pretraining mix and was chosen as a middle ground between training speed and pretraining-loss performance.A ratio of 4 often increased pretraining-distribution log-probability loss during training, while preliminary experiments suggested better human Likert scores at 32 but with training time increasing several fold.
  • Training duration: With the 1.3B model, training beyond 256k PPO episodes was not helpful, although more unique prompts or larger models might change this conclusion.This conclusion applies to PPO with pretraining data mix and is left for future work to reassess under broader conditions.
  • Batch and minibatch sizes: For the 1.3B model with PPO and pretraining data mix, batch size 512 performed best in human evaluations, while minibatch size 32 was slightly better than 64.Final models used minibatch size 64 instead because it provided better GPU utilization.
  • Toxicity evaluation: PPO instruction-following models generally produced less toxic output than non-instruction-following models when instructed to be respectful, but reliably produced very toxic content when instructed to be biased.The toxicity results were measured on RealToxicityPrompts as a function of input prompt toxicity.

F Model samples

This section presents additional samples comparing 175B GPT-3 and InstructGPT across multilingual instruction following, potentially harmful requests, code description, and diverse everyday tasks. The examples show capabilities alongside limitations, including mostly English output for a Swedish prompt and imperfect code descriptions.

  • InstructGPT answers a potentially harmful grocery-store theft prompt, illustrating a propensity linked to prioritizing helpfulness in training.The section frames this behavior as a consequence of prioritizing helpfulness to the user in the training data.
  • InstructGPT can describe code, although the examples show that its code explanations remain far from perfect.Figure 45 presents samples for describing a prime-sieve program and compares GPT-3 175B with InstructGPT 175B.
  • Five labeler-written prompts were selected from 15 to show a diverse range of tasks, alongside human-written demonstrations and completions from GPT-3 175B and InstructGPT175B; the completions were not cherry-picked.
  • InstructGPT sometimes follows instructions in other languages, producing a full French story and following a Swedish prompt while writing mostly in English.The French example is highlighted in Figure 42, while Figure 43 documents the Swedish-language limitation.
Loading 2203.02155v1…