PaperlayerView exact revision on arXiv

Source-linked AI summary

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe

arXiv:2203.02155v1cs.CLcs.AIcs.LG

TL;DR

Language models often fail to follow user intent and can produce untruthful, toxic, or unhelpful outputs. This paper fine-tunes GPT-3 with demonstrations and human preference feedback, finding that InstructGPT outputs are preferred to GPT-3 outputs and improve truthfulness and toxicity measures.

  • Problem

    Language models can make up facts, generate biased or toxic text, and fail to follow user instructions, motivating alignment with user intent.

  • Method

    The authors fine-tune GPT-3 using supervised demonstrations and reinforcement learning from human feedback based on human preferences.

  • Results

    85 ± 3% of the time, labelers prefer 175B InstructGPT outputs to 175B GPT-3 outputs, while InstructGPT also improves truthfulness and reduces toxic outputs.

  • Takeaways & Limitations

    Fine-tuning language models with human preferences significantly improves behavior across a wide range of tasks, while further safety and reliability work remains necessary.

  • Takeaways & Limitations

    The models are neither fully aligned nor fully safe, and still generate toxic or biased outputs, fabricate facts, and produce sexual or violent content without explicit prompting.

Abstract

from arXiv · show

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

1 Introduction

The paper addresses the misalignment between next-token prediction and helpful, safe instruction following by fine-tuning GPT-3 with human feedback. InstructGPT improves human-rated helpfulness, truthfulness, and toxicity outcomes, while retaining limitations and some task-performance tradeoffs.

  • The method uses supervised fine-tuning, reward-model training from ranked outputs, and PPO reinforcement learning, with human preferences serving as the reward signal.The study used 40 contractors to label data and evaluated models mainly through labeler ratings on held-out customer prompts.
  • 1.3B InstructGPT outputs were preferred to 175B GPT-3 outputs despite having over 100x fewer parameters, including against few-shot-prompted GPT-3.The models share the GPT-3 architecture and differ in their fine-tuning on human data.
  • InstructGPT generated truthful and informative TruthfulQA answers about twice as often as GPT-3 and produced about 25% fewer toxic outputs when prompted respectfully.It did not significantly improve over GPT-3 on Winogender or CrowSPairs, indicating no corresponding bias improvement.
  • RLHF fine-tuning caused regressions on some public NLP datasets, but mixing PPO with pretraining-distribution updates greatly reduced them without compromising labeler preference scores.The regressions included SQuAD, DROP, HellaSwag, and WMT 2015 French-to-English translation.
  • InstructGPT preferences generalized to held-out labelers at about the same rate as to training labelers, while broader user disagreement remains insufficiently studied.The authors also report qualitative instruction-following beyond the RLHF distribution, including code summarization, code questions, and sometimes other languages.
  • Despite these gains, InstructGPT can still fail to follow instructions, fabricate facts, hedge excessively, or miss false premises, so safety and reliability remain open problems.The paper frames next-token prediction as misaligned with the goal of following users’ instructions helpfully and safely, especially in deployed applications.

2 Related work

The paper builds on prior work in reinforcement learning from human feedback, alignment, instruction following, and methods for evaluating and mitigating language-model harms. It positions its contribution as applying RLHF to align language models across a broad distribution of language tasks.

  • Research on alignment and learning from human feedback: The work directly applies reinforcement learning from human feedback to aligning language models across a broad distribution of language tasks.RLHF has previously been used for simulated robots, Atari games, and language-model text summarization.
  • Research on alignment and learning from human feedback: Prior alignment research characterizes misalignment through harmful content and gaming misspecified objectives, while studying language assistants and alignment baselines.The broader question of what it means for language models to be aligned has also received recent attention.
  • Training language models to follow instructions: The paper relates to instruction-following research spanning cross-task generalization in language models and navigation in simulated environments.Cross-task studies fine-tune models on varied public NLP datasets with instructions and evaluate them on different tasks.
  • Evaluating the harms of language models: Prior work documents deployment risks including biased outputs and private-data leakage, motivating efforts to modify language-model behavior.These risks have been extensively documented across multiple studies.
  • Modifying the behavior of language models to mitigate harms: Existing mitigation strategies include value-targeted fine-tuning and filtering pretraining data using researcher-written trigger phrases.Value-targeted fine-tuning improves adherence to specified values on a question-answering task.

3 Methods and experimental details

The method combines supervised fine-tuning on labeler demonstrations, reward modeling from preference comparisons, and PPO optimization against the learned reward. Training uses diverse prompts from labelers and early API users, with human evaluation emphasizing truthfulness and harmlessness.

  • 3.1 Method overview: The three-stage pipeline fine-tunes a pretrained GPT-3 model on demonstrations, trains a reward model from preferred outputs, and optimizes the policy with PPO.The reward model supplies a scalar reward for PPO; comparison-data collection and policy optimization can be iterated.
  • 3.2 Prompt distribution: The datasets combine labeler-written prompts with early InstructGPT API prompts spanning generation, question answering, dialog, summarization, extraction, and other tasks.The SFT dataset contains about 13k training prompts and the RM dataset 33k training prompts; the overall dataset is over 96% English.
  • 3.2 Prompt distribution: Labelers infer user intent from direct instructions, few-shot examples, or implicit continuations, producing demonstrations and rankings for SFT and reward-model training.A separate unlabeled PPO dataset supplies prompts for RLHF fine-tuning.
  • 3.4 Human evaluation: Although training prioritizes helpfulness when alignment criteria conflict, final evaluations ask labelers to prioritize truthfulness and harmlessness.Truthfulness is measured through hallucination tendencies on closed-domain tasks and performance on TruthfulQA (Lin et al., 2021).
  • 3.3 Reinforcement learning: The RL environment samples a customer prompt, scores the response with the reward model, and adds a per-token KL penalty relative to the SFT model to mitigate reward over-optimization.The environment is a bandit setting in which each prompt-response episode ends after producing the reward.

4 Results

Across API prompts, InstructGPT outputs were preferred over GPT-3 and better followed instructions, while also improving truthfulness and conditionally reducing toxicity. These gains generalized to held-out labelers and some out-of-distribution tasks, but public-dataset performance and several behavioral limitations remained.

  • API prompt distribution: Labelers significantly preferred InstructGPT over GPT-3 across model sizes, with successive gains from prompting, supervised fine-tuning, and PPO.Held-out labelers showed similar preferences, and all InstructGPT models greatly outperformed GPT-3 baselines, indicating the models did not simply overfit training-labeler preferences.
  • API prompt distribution: InstructGPT outputs were more appropriate, followed explicit constraints more often, failed instructions less often, and hallucinated less on closed-domain tasks than GPT-3.These concrete improvements were reported alongside favorable overall ratings and support the models’ stronger alignment with user intent.
  • Public NLP datasets: 78 ±4% and 79 ± 4%: 175B InstructGPT outputs were preferred over FLAN and T0 outputs, respectively.FLAN- and T0-fine-tuned GPT-3 models outperformed GPT-3 and matched prompted GPT-3, but performed worse than the SFT baseline on the compared public datasets.
  • Truthfulness, toxicity, and bias: PPO models showed small but significant truthfulness improvements over GPT-3, including less hallucination, while toxicity gains were small and prompt-dependent rather than extending to bias.Under a respectful prompt, InstructGPT was less toxic than GPT-3; without that prompt the advantage disappeared, and explicit toxic prompting made InstructGPT more toxic.
  • Qualitative results: InstructGPT showed qualitative ability to follow non-English instructions and perform code summarization and question-answering despite little such data in fine-tuning.The 175B PPO-ptx model reliably answered code questions and followed other-language instructions, though it often responded in English and required less prompting than GPT-3.
  • Qualitative results: InstructGPT still made simple mistakes, including accepting false premises and over-hedging, behaviors the authors linked to limited training examples and reward-model incentives.The authors suggested adversarial data could reduce both behaviors; PPO training also caused an alignment tax on several public NLP datasets unless the procedure was modified.

5 Discussion

The discussion presents iterative alignment as an empirically grounded approach while emphasizing that the resulting systems reflect specific labelers, researchers, and customers rather than humanity’s preferences generally. It also highlights substantial methodological and model limitations, including residual harmful behavior and incomplete mitigation of performance regressions.

  • 5 Discussion: Iterative alignment provides empirical feedback on what works and fails in current systems, but does not directly address problems unique to superhuman systems.The approach focuses on improving existing AI systems rather than abstractly aligning systems that do not yet exist.
  • 5.1 Who is being aligned to?: The alignment process targets a specific human reference group and application, not universally correct preferences, because data reflect labelers, researchers, and API customers.Labelers were mostly English-speaking people in the United States or Southeast Asia, and labeler agreement was about 73%; customers were not representative of all potential users or affected groups.
  • 5.2 Methodology: The alignment data and methodology remain limited because value judgments depend on contractor identities and beliefs, while most comparisons were labeled by only one contractor.Multiple labels could reveal disagreement, and averaging labeler preferences may be undesirable when outputs disproportionately affect minority groups.
  • 5.3 Models: The models remain neither fully aligned nor fully safe: they can produce toxic, biased, fabricated, sexual, or violent content, and may follow harmful instructions.When instructed to be maximally biased, InstructGPT generated more toxic outputs than equivalently sized GPT-3 models.
  • 5.3 Models: Incorporating pretraining data into RLHF does not fully eliminate performance regressions and can increase undesirable behaviors when those behaviors occur in the pretraining data.The discussion suggests filtering toxic content from the pretraining mix or otherwise modifying the data as directions for further research.

A Additional prompt data details … B.2 Labeling instructions

The appendix describes how InstructGPT prompt data were bootstrapped, diversified, split, and labeled, including contractor-written prompts for the first model and consented API Playground prompts. It also documents labeler selection and evolving instructions, notably the different priorities assigned during training and final evaluation.

  • A.1 Labeler-written prompts: Labelers created plain and few-shot prompts to bootstrap the first instruction-following model, while anonymized application descriptions supported its supervised-learning training.The resulting model was deployed in the API beta in early 2021.
  • A.2 API user prompts: API prompts came from users of an earlier InstructGPT model in the OpenAI Playground, where informed-consent alerts made data collection easier than from production customers.Prompts were heuristically deduplicated, capped at roughly 200 per organization, and split by organization ID to separate use cases across training, validation, and test sets.
  • A.2.1 Illustrative user prompts from InstructGPT distribution: The API prompt distribution covered ten use cases, while GPT-3 prompts were generally less instruction-style, more explicitly prompted, and sometimes ambiguous in user intent.The categories were generation, open QA, closed QA, brainstorming, chat, rewriting, summarization, classification, extraction, and other.
  • A.2.2 Illustrative user prompts from GPT-3 distribution: Illustrative GPT-3 prompts include continuation-style generation, classification templates, conversational setups, question answering, and summarization prompts.These examples contrast with the instruction-oriented API distribution without establishing a quantitative comparison.
  • A.3 Dataset sizes: SFT contained many more labeler-written than customer prompts because few-shot instructions were reused to synthetically construct multiple datapoints, whereas RM training expanded each prompt into rankings over 4–9 outputs and many more ranked pairs.The number of ranked pairs was an order of magnitude larger than the number of prompts.
  • A.4 Data diversity: The collected data spanned diverse categories and use cases, with approximately 96% of 110k datapoints classified as English and a small minority spanning at least 20 other languages.Prompt and demonstration lengths, customer contribution counts, and category-specific lengths were reported descriptively.
  • B.1 Labeler selection: Labelers were screened for agreement on sensitive-speech flagging and rankings, quality of sensitive demonstrations, and self-assessed coverage of sensitive topics and cultural groups.Selections used soft cutoffs of 75% agreement and a 6/7 demonstration score, alongside subjective judgment for the self-assessment criterion.
  • B.2 Labeling instructions: Labeling instructions evolved with feedback and metadata changes, and training labelers prioritized helpfulness whereas final evaluators prioritized truthfulness and harmlessness.The authors identify refusals as one avenue for training models to sometimes prioritize truthfulness and harmlessness over helpfulness.

B.3 Labeler demographic data … C Additional model details

The appendix characterizes labelers, documents evaluation instructions and interfaces, and specifies core model-training implementation details. Labelers were generally young and regionally concentrated, satisfied with the work, and evaluated outputs using structured scoring and ranking procedures.

  • B.3 Labeler demographic data: API-prompt evaluations instructed labelers to assess outputs for tasks expressed either as explicit instructions or as examples of desired behavior.The supplied excerpt illustrates both direct task descriptions and example-based task specifications.
  • B.3 Labeler demographic data: Toxicity evaluations asked labelers to judge whether models continued potentially offensive or explicit sentence prefixes in a respectful, plausible, and non-toxic manner.The RealToxicityPrompts instructions explicitly framed the task as evaluating non-toxic generation even for toxic inputs.
  • B.3 Labeler demographic data: Labelers were generally young, gender-balanced, and primarily from the US or Southeast Asia, based on responses from 19 survey participants.Seventy-five percent were younger than 35.
  • B.4 Labeler satisfaction survey: The 19 surveyed labelers generally enjoyed the task, considered their pay fair, and valued researchers’ helpfulness and communication, though some found the work repetitive.Other labelers viewed the task as sufficiently varied and engaging.
  • B.5 Web interface: The labeling interface combined 1–7 overall-quality Likert scores and metadata labels with rankings of all outputs for each prompt, while encouraging ties for similar-quality outputs.The same interface was used by labelers and researchers.
  • C Additional model details: All models used the GPT-3 architecture, while reward models and value functions replaced the original unembedding layer with a scalar-output projection layer.The models used fp16 weights and activations with fp32 master weights, Brown et al. (2020) byte-pair encodings, and 2k-token context lengths.
  • C Additional model details: All models were trained with Adam using β1 = 0.9 and β2 = 0.95.

C.1 Details of SFT training

SFT models were trained with fixed optimization settings differentiated by model size, while learning rates and epochs were selected through geometric search. Final models were chosen using RM score because it better predicted human preference than validation loss.

  • C.1 Details of SFT training: SFT training used 16 epochs, residual dropout of 0.2, cosine learning-rate decay to 10% without warmup, and model-size-specific learning rates and batch sizes.The 1.3B and 6B models used learning rate 9.65e-6 and batch size 32; the 175B model used 5.03e-6 and batch size 8.
  • C.1 Details of SFT training: Learning rates were selected by geometric search over 7 candidates for the 1.3B and 6B models and 5 candidates for the 175B model.The number of training epochs was also tuned using geometric search.
  • C.1 Details of SFT training: Final SFT models were selected using RM score because it was more predictive of human preference results than validation loss.

C.2 Details of RM training … C.5 FLAN and T0 models

The training pipeline used a stable 6B reward model and value function across PPO policy sizes, with supervised initialization, pretraining-data mixing, and controlled FLAN/T0 checkpoint selection. These choices balanced compute, comparability, RLHF stability, and preservation of public NLP performance.

  • C.2 Details of RM training: A single 6B reward model powered PPO policies of every size because it was stable across learning rates, computationally cheaper, and produced equally strong PPO results.Larger 175B reward models could achieve lower validation loss but were less stable and substantially increased PPO compute requirements.
  • C.2 Details of RM training: The final reward model was initialized from a 6B GPT-3 model fine-tuned on public NLP datasets and trained for one epoch with learning rate 9e-6.Similar results were obtained when initializing from GPT-3 or supervised fine-tuning models.
  • C.3 Details of the initialization models for RLHF: RLHF models were initialized from pretrained GPT-3 with two supervised fine-tuning epochs, mixing 10% pretraining data because it helped PPO training.Cosine decay was used, with batch sizes of 32 for 1.3B and 6B models and 8 for the 175B model.
  • C.4 Details of RLHF training: All RL policies trained for 256k episodes on about 31k unique filtered prompts, using KL reward regularization with β = 0.02 and fixed PPO batch and minibatch sizes.The RL policies were initialized from the supervised models with pretraining mix, which also computed the KL reward.
  • C.4 Details of RLHF training: Using the same 6B reward model and value function for every policy size made policy-size comparisons easier, with value-function learning rates of 9e-6 or 5e-6.The 9e-6 rate applied to 1.3B and 6B policies, while 5e-6 applied to the 175B policy.
  • C.4 Details of RLHF training: Pretraining-gradient mixing mitigated RLHF regressions on SQuADv2 and DROP by using eight times more pretraining examples than RL episodes.PPO and pretraining gradients were computed in consecutive steps for each minibatch and accumulated together.
  • C.5 FLAN and T0 models: FLAN and T0 baselines were produced by fine-tuning 175B GPT-3, with T0 subsampled to 1 million datapoints to match FLAN’s training-data scale.The T0++ dataset was used, and training epochs passed through datapoints without repeats.
  • C.5 FLAN and T0 models: Reward-model scoring selected FLAN and T0 checkpoints after reward saturation and comparison of training settings, including FLAN’s 896k-example checkpoint at learning rate 4e-6.For T0, the selected checkpoint came from the 4e-6, batch-size-128 experiment after 896k examples; FLAN reward saturated after roughly 400k examples (Figure 13).

D Automatic evaluation details … E Additional results

The paper evaluates models across a broad set of bias, toxicity, truthfulness, reasoning, classification, translation, and summarization benchmarks using standardized prompting and decoding procedures. Additional zero-shot results show regressions for 175B PPO models on public NLP datasets, mitigated by updating on pretraining data during fine-tuning.

  • D Automatic evaluation details: Evaluation spans Winogender, CrowS-Pairs, RealToxicityPrompts, TruthfulQA, DROP, QuAC, SquadV2, Hellaswag, SST, RTE, WSC, WMT 15 Fr →En, and CNN/Daily Mail Summarization.These benchmarks cover bias, toxicity, truthfulness, question answering, reasoning, sentiment, natural-language inference, coreference, translation, and summarization.
  • D Automatic evaluation details: Models are evaluated with task prompts, contexts, and sampled or multiple-choice completions, using temperature T = 0 sampling truncated at the first newline and log-probability-based multiple-choice selection at T = 1.The evaluation framework standardizes how model outputs are generated and how answer choices are selected.
  • D.1 Toxicity and bias evaluation details: Bias and toxicity evaluations compare basic, respectful, and biased instructions to test how prompting affects model behavior.The respectful prompt requests polite, respectful, and unbiased completions, whereas the biased prompt requests maximally biased and offensive completions.
  • D.1 Toxicity and bias evaluation details: Bias-task performance is summarized by average choice entropy, where higher entropy indicates greater uncertainty or less preference between binary options.Entropy is measured in bits, with a maximum of 1 for binary choices, using probabilities proportional to the model’s assigned completion probabilities.
  • D.2 Prompt structure and evaluation features for each eval dataset: Dataset-specific evaluation details define prompting structures, validation-example counts, and performance metrics for each automatic benchmark, including variable-length question-answer formats for QuAC and SquadV2.The section provides task examples and feature descriptions across the benchmark suite, including instructions to answer from preceding context or abstain when information is insufficient.
  • E Additional results: 175B PPO models consistently show zero-shot performance regressions on public NLP datasets, mitigated by adding updates on pretraining data during fine-tuning.Few-shot performance is reported separately, and translation error bars are unavailable because the software package used does not report them.

E.1 Performance on public NLP datasets · E.2 Reward model generalization across sets of labelers

Public-dataset evaluations show that PPO without a pretraining mix regresses on many tasks, especially few-shot, while PPO-ptx mitigates these regressions. Reward-model overfitting is assessed through five-fold cross-validation across held-out labeler groups.

  • E.1 Performance on public NLP datasets: PPO-ptx mitigates the regressions seen with PPO without pretraining mix across many public NLP datasets, particularly in few-shot evaluation.The few-shot comparison is presented in Figure 29, while zero-shot performance is shown separately in Figure 28.
  • E.1 Performance on public NLP datasets: The evaluations collectively measure bias, toxicity, truthfulness, and diverse natural language capabilities.Results are reported in Table 14.
  • E.1 Performance on public NLP datasets: Zero-shot performance of the models is reported on various public NLP datasets.The zero-shot results are shown in Figure 28.
  • E.1 Performance on public NLP datasets: PPO without pretraining mix exhibits performance regressions on many public NLP datasets, especially in the few-shot setting.These regressions are the motivating contrast for the PPO-ptx results.
  • E.1 Performance on public NLP datasets: Few-shot performance of the models is reported on various public NLP datasets.The few-shot results are shown in Figure 29 and compared with zero-shot performance in Figure 28.
  • E.2 Reward model generalization across sets of labelers: Reward-model generalization is tested by training on subsets of labelers and evaluating on held-out labelers.The comparison data are split into five groups with roughly equal training-data amounts, followed by five-fold cross-validation of the 6B reward model using four groups for training and one for validation.

E.3 Metadata results as a function of model size · E.4 Likert scores · E.5 Measuring bias

The appendix reports metadata ratings by model type and size, Likert scores across models, and bias results on Winogender and CrowS-Pairs. Likert results largely track preference results, while InstructGPT does not significantly improve over GPT-3 on the bias datasets.

  • E.3 Metadata results as a function of model size: Figure 30 examines how metadata results vary with model size.
  • E.3 Metadata results as a function of model size: Metadata ratings are presented as a function of model type and model size in Figure 30.
  • E.4 Likert scores: Likert scores are reported for each model on the prompt distribution in Figure 31.
  • E.4 Likert scores: The Likert-score results largely track the preference results reported in Section 4.1.
  • E.5 Measuring bias: Bias results on the Winogender and CrowS-Pairs datasets are shown in Figure 32.
  • E.5 Measuring bias: InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs datasets.

E.6 Fixing regressions on public NLP datasets … E.10 RealToxicityPrompts results as a function of input toxicity

The experiments identify training choices that recover or preserve public NLP performance while balancing reward, initialization, stability, and toxicity evaluation. Pretraining-data mixing and carefully tuned coefficients are especially important, whereas longer training and excessive learning rates can cause regressions or divergence.

  • E.6 Fixing regressions on public NLP datasets: Setting the pretraining loss coefficient to at least 20 recovers regressions on public NLP tasks for the 1.3B model, with task-dependent sensitivity and reduced validation reward.Figure 33 shows improvement on DROP and SQuAD without much validation-reward regression.
  • E.6 Fixing regressions on public NLP datasets: Increasing the KL reward coefficient alone does not fully mitigate regressions on DROP and SQuAD, even when raised to 2.0, or 100 times the baseline.The experiments use a zero pretraining loss coefficient and the pretrained GPT model as the KL reward model.
  • E.6 Fixing regressions on public NLP datasets: Training for 512k rather than 256k episodes causes DROP and SQuADv2 performance to decline slightly below the GPT-3 baseline after initially exceeding it.This pattern appears on the 1.3B model under default PPO training with a pretraining mix across three random seeds.
  • E.7 Optimal KL reward coefficient: Human Likert performance is poor at KL reward coefficients of 0 and 2, with the optimum occurring around 0.01–0.02.The experiment uses PPO training with a pretraining data mix.
  • E.8 PPO init models: Among tested PPO initialization variants, the setting with a 10% pretraining mix stands out, although PPO performance is generally insensitive to the tested choices.The selected initialization trains for two epochs on human demonstrations with a 10% pretraining mix.
  • E.9 Learning rate optimization for PPO models: For PPO without a pretraining mix, learning rates above 8.05e-6 diverge, while PPO with the mix appears less sensitive to learning-rate changes.Learning rates from 2.55e-6 to 2.55e-5 were scanned for 1.3B and 6B models; 175B experiments used 2.55e-6 and 3.74e-6.
  • E.10 RealToxicityPrompts results as a function of input toxicity: RealToxicityPrompts output toxicity is highly correlated with input-prompt toxicity, so evaluation samples 5000 prompts approximately uniformly across prompt-toxicity levels.Toxicity is measured with the Perspective API to better capture behavior in unsafe regimes.

E.11 Additional ablations

Additional ablations found that pretraining-data ratio, training duration, and batch-size choices materially affected PPO with pretraining data mix, while toxicity depended strongly on the instruction condition. The final models used a minibatch size of 64 for better GPU utilization despite size 32 being slightly better.

  • E.11 Additional ablations: A pretraining data ratio of 4 often caused log probability loss on the pretraining distribution to increase during training, despite preliminary evidence of improved human Likert scores.The comparison kept the pretraining loss coefficient constant while varying the amount of pretraining data.
  • E.11 Additional ablations: For the 1.3B model, training beyond 256k episodes was not helpful for PPO with pretraining data mix.Whether more unique prompts and larger models would change this conclusion remains open.
  • E.11 Additional ablations: A batch size of 512 performed best in human evaluations, while minibatch size 32 was slightly better than 64.The final models nevertheless used minibatch size 64 because it provided better GPU utilization.
  • E.11 Additional ablations: PPO instruction-following models generally produced less toxic output than non-instruction-following models only when instructed to be respectful.When instructed to be biased, the same models reliably produced very toxic content even at low input prompt toxicity.
  • E.11 Additional ablations: The RealToxicityPrompts analyses also reported continuity, relative toxicity ratings, and win rates against 175B GPT-3 for PPO-ptx and SFT.These analyses are summarized in Figures 40 and 41.

F Model samples

Samples from 175B InstructGPT illustrate instruction following across languages and task types, while also showing limitations such as mostly English output for a Swedish prompt and imperfect code descriptions. Additional examples cover potentially harmful requests and diverse labeler-written tasks.

  • Safety and code: The samples show InstructGPT responding to potentially harmful requests and describing code, though its code explanations remain far from perfect.The harmful-prompt example reflects a training emphasis on helpfulness to the user, while the code example demonstrates an imperfect but relevant capability.
  • Diverse task samples: Five lightly selected labeler-written prompts span diverse tasks, with model samples and human-written demonstrations shown for comparison.The prompts include vocabulary use, question answering, recipe-based shopping lists, and explanatory requests; the completions are not cherry-picked.
  • Multilingual instruction following: InstructGPT sometimes follows instructions in other languages, producing a French time-travel story and following the Swedish instruction while writing mostly in English.The French example is presented in full, whereas the Swedish sample demonstrates instruction adherence with substantial English output.
Loading 2203.02155v1…