PaperlayerView exact revision on arXiv

Source-linked AI summary

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe

arXiv:2203.02155v1cs.CLcs.AIcs.LG

TL;DR

Language models do not reliably follow user intent, so the paper fine-tunes GPT-3 using human demonstrations followed by reinforcement learning from human preferences. Labelers prefer 1.3B-parameter InstructGPT outputs to 175B GPT-3 outputs, while InstructGPT improves truthfulness and reduces toxicity when prompted to be respectful.

  • Problem

    Language models do not reliably follow user instructions, leaving alignment with users' explicit and implicit intentions an open challenge.

    Source: §1 · pp. 1–2

  • Method

    GPT-3 is fine-tuned using supervised human demonstrations followed by reinforcement learning with rewards derived from human preferences.

    Source: §1 · p. 2, §3.5 · p. 9

  • Results

    1.3B-parameter InstructGPT outputs are preferred to 175B GPT-3 outputs by labelers, while InstructGPT improves truthfulness and reduces toxicity when prompted to be respectful.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support human-feedback fine-tuning as a promising direction for aligning language models with user intent across diverse tasks.

    Source: §1 · p. 4

    InstructGPT remains neither fully aligned nor fully safe, still producing toxic or biased outputs and fabricating facts.

    Source: §5.3 · p. 19

Abstract

from arXiv · show

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

1 Introduction

Fine-tuning GPT-3 with human demonstrations and preference feedback produces InstructGPT models that labelers prefer to GPT-3, with improved truthfulness and reduced toxicity. These gains coexist with persistent mistakes and limited evidence about broader user preferences.

1 Introduction

  • 85 ± 3% of comparisons favor 175B InstructGPT over 175B GPT-3, versus 71 ± 4% against few-shot 175B GPT-3.On held-out customers’ prompts, even 1.3B InstructGPT beats 175B GPT-3 despite over 100x fewer parameters. InstructGPT also follows explicit instruction constraints more reliably.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • InstructGPT fine-tunes GPT-3 through supervised demonstrations, human-preference reward modeling, and reinforcement learning with PPO to better follow user intent.Next-token prediction differs from helpful, honest, harmless instruction following (Askell et al., 2021), and language models can fabricate facts or generate biased or toxic text. Demonstrations use mostly English API and labeler-written prompts; the procedure targets labelers’ and researchers’ stated preferences, not a broader notion of human values.Source: §1 · pp. 1–2
  • 21% versus 41% hallucination rates favor InstructGPT over GPT-3 on closed-domain API tasks; truthful, informative TruthfulQA answers occur about twice as often.Closed-domain outputs should contain no information absent from the input. TruthfulQA improvements remain equally strong on questions not adversarially selected against GPT-3.Source: §1 · p. 3
  • About 25% fewer toxic outputs occur with InstructGPT than GPT-3 when prompted to be respectful, without significant improvements on the evaluated bias datasets.Toxicity is evaluated automatically and by humans on RealToxicityPrompts (Gehman et al., 2020). Bias evaluations use Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020).Source: §1 · p. 3
  • Mixing PPO updates with pretraining-likelihood updates greatly reduces public-NLP performance regressions without compromising labeler preference scores.These PPO-ptx updates mitigate an alignment tax observed on SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag, and WMT 2015 French-to-English translation.Source: §1 · pp. 3–4
  • InstructGPT generalizes to held-out labelers and shows qualitative instruction-following beyond its fine-tuning distribution.Held-out labelers prefer it over GPT-3 at about the training labelers’ rate, but broader users and disagreements need further study. It summarizes code, answers code questions, and sometimes follows instructions in different languages despite little direct supervision, whereas GPT-3 requires more careful prompting.Source: §1 · p. 4
  • 73.4 ±2% is InstructGPT’s winrate against the baseline, compared with 26.8 ±2% for T0 and 29.8 ±2% for FLAN.On API prompts, the authors’ models trained using T0’s T0++ variant (Sanh et al., 2021) and FLAN (Wei et al., 2021) perform slightly worse than their SFT baseline.Source: §1 · p. 4
  • InstructGPT still fails instructions, fabricates facts, over-hedges simple answers, and misses false premises, leaving substantial safety and reliability work.The reported behavioral improvements therefore do not eliminate simple mistakes.Source: §1 · p. 4

2 Related work

The paper situates its broad-task application of RLHF alongside instruction-tuning research and methods for evaluating and mitigating language-model harms.

Research on alignment and learning from human feedback

Training language models to follow instructions

Evaluating the harms of language models

Modifying the behavior of language models to mitigate harms

3 Methods and experimental details

The method fine-tunes GPT-3 using human demonstrations, preference-based reward modeling, and PPO, with pretraining gradients added in PPO-ptx. Evaluation combines human preference ratings on held-out API prompts with public benchmarks for truthfulness, toxicity, bias, and traditional NLP tasks.

3.1 High-level methodology

  • Human demonstrations supervise GPT-3, output comparisons train a reward model, and PPO (Schulman et al., 2017) optimizes that model’s scalar reward.Reward modeling and policy optimization can be iterated using comparisons from improved policies; most comparison data came from supervised policies, with some from PPO policies.Source: §3.1 · p. 6

3.2 Dataset

  • Prompts primarily come from early InstructGPT Playground users, supplemented by labeler-written prompts that bootstrapped training.SFT uses about 13k demonstration prompts, RM uses 33k ranked-output prompts, and PPO uses 31k unlabeled API prompts. Production API data are excluded; safeguards include prefix-based deduplication, a 200-prompt limit per user, user-disjoint splits, and filtering training prompts for personally identifiable information.Source: §3.2 · pp. 6–7

3.3 Tasks

  • Over 96% of the dataset is English, spanning generation, question answering, dialogue, summarization, extraction, and other natural-language tasks.Tasks may be expressed through explicit instructions, few-shot examples, or implicit continuation. Labelers infer user intent, skip very unclear tasks, and consider implicit expectations of truthfulness and avoidance of biased or toxic outputs.Source: §3.3 · p. 7

3.4 Human data collection

  • 72.6 ± 1.5% agreement among training labelers and 77.3 ± 1.3% among held-out labelers indicate substantial inter-annotator agreement.About 40 contractors produced demonstrations, comparisons, and main evaluations, with screening targeting sensitivity to demographic preferences and harmful outputs. Training prioritized helpfulness, whereas final evaluations prioritized truthfulness and harmlessness.Source: §3.4 · pp. 7–8

3.5 Models

  • Grouping all comparisons from each prompt into one batch element prevents reward-model overfitting while improving computational efficiency, validation accuracy, and log loss.SFT runs for 16 epochs and selects checkpoints by validation reward score; although validation loss overfits after one epoch, further training improves reward scores and human preferences. Reward models use 6B parameters because 175B training could be unstable, and their rewards are shifted so demonstrations average 0 before RL.Source: §3.5 · pp. 8–9
  • PPO-ptx mixes pretraining gradients into PPO to address public-NLP performance regressions and is the paper’s default InstructGPT variant.PPO uses a per-token KL penalty relative to SFT to mitigate reward-model over-optimization, with its value function initialized from the reward model. Baselines include SFT, GPT-3, instruction-prompted GPT-3, and 175B GPT-3 fine-tuned on FLAN (Wei et al., 2021) or T0 (Sanh et al., 2021), each using approximately 1 million examples and validation reward scores for checkpoint selection.Source: §3.5 · p. 9

3.6 Evaluation

  • Truthfulness is assessed through closed-domain hallucinations and TruthfulQA (Lin et al., 2021), while harm assessment uses specific behavioral proxies rather than speculative overall-harm judgments.These truthfulness measures capture only part of truthfulness and do not establish honesty, because model beliefs cannot be inferred. Harm proxies cover assistant-inappropriate responses, protected-class denigration, sexual or violent content, and benchmarks including RealToxicityPrompts (Gehman et al., 2020) and CrowS-Pairs (Nangia et al., 2020).Source: §3.6 · p. 10
  • The main evaluation measures human preferences on held-out API prompts, while acknowledging that labelers may misinterpret users’ intentions.Outputs are compared against the 175B SFT baseline and receive overall-quality ratings on a 1-7 Likert scale. Evaluation includes unseen users, GPT-3-origin prompts to address possible distributional disadvantage to GPT-3 baselines, and public benchmarks covering safety and zero-shot NLP performance.Source: §3.6 · p. 10

4 Results

InstructGPT improved human preferences and instruction-following on API prompts, while public-dataset evaluations showed improved truthfulness, prompt-dependent toxicity reductions, and no bias improvement. Pretraining updates mitigated benchmark regressions, but qualitative generalization coexisted with simple failures.

4.1 Results on the API distribution

  • 85 ± 3% of direct comparisons favored 175B InstructGPT over GPT-3, and 71 ± 4% favored it over few-shot GPT-3.Preferences improved successively with few-shot prompting, supervised fine-tuning, and PPO; adding pretraining updates changed preferences little. InstructGPT was more appropriate as a customer assistant, followed instructions and explicit constraints better, and hallucinated less on closed-domain tasks.Source: Figure 3 · p. 11, §4.1 · p. 11, §4.2 · p. 13
  • 78 ±4% versus FLAN (Wei et al., 2021) and 79 ± 4% versus T0 (Sanh et al., 2021) were 175B InstructGPT’s head-to-head preference rates.Both baselines outperformed default GPT-3, matched well-prompted GPT-3, and underperformed SFT. The authors attribute the gap to limited input diversity and task mismatch: classification and QA represented about 18% of API use, versus about 57% for open-ended generation and brainstorming.Source: Figure 5 · p. 12, §4.1 · pp. 12–13
  • 69.6 ± 0.9% accuracy predicting held-out labelers’ preferences compared with 72.4 ± 0.4% for training labelers supported reward-model generalization.Five-fold cross-validation used five labeler groups and three seeds; held-out labelers also strongly preferred InstructGPT to GPT-3 baselines. Results changed little on prompts submitted to GPT-3 rather than InstructGPT, although larger PPO-ptx models performed slightly worse.Source: §4.1 · pp. 11–12, Figure 3 · p. 11

4.2 Results on public NLP datasets

  • PPO models showed small but significant TruthfulQA improvements in truthfulness and informativeness over GPT-3 without requiring explicit truthfulness instructions.The 1.3B PPO-ptx model was an exception, performing slightly worse than same-sized GPT-3; gains persisted on non-adversarial prompts but decreased by a couple of percentage points. With the uncertainty-abstention prompt from Lin et al. (2021), PPO models favored truthful, uninformative responses over confident falsehoods more effectively than GPT-3.Source: Figure 6 · p. 13, §4.2 · p. 13
  • InstructGPT reduced toxicity with respectful prompting, not without it, became more toxic when explicitly requested, and showed no bias improvement over GPT-3.Human and Perspective API evaluations agreed; toxicity-stratified sampling inflated absolute scores, all models were less toxic than prompt-based expectations, and SFT was least toxic but least preferred, with lowest continuity. Paired-sentence evaluations found similar bias for PPO-ptx and GPT-3, but respectful instructions increased PPO-ptx’s measured bias through lower entropy, without a clear stereotypical pattern.Source: Figure 7 · p. 14, §4.2 · p. 14
  • Adding pretraining updates to PPO mitigated public-benchmark regressions across datasets and surpassed GPT-3 on HellaSwag, while retaining deficits on DROP, SQuADv2, and translation.Ordinary PPO incurred an alignment tax on several public NLP datasets. Tuning the pretraining mixture could reverse SQuADv2 and DROP regressions with minimal validation-reward reductions, whereas increasing the KL coefficient substantially reduced reward without fully restoring performance.Source: §4.2 · pp. 14–15

4.3 Qualitative results

  • Qualitative examples showed InstructGPT following non-English instructions and summarizing or answering questions about code, despite these domains forming a tiny minority of fine-tuning data.These behaviors were not quantified, and illustrative prompts—but not outputs—were cherry-picked. Outputs often remained in English, one code answer was imperfect, and GPT-3 required more careful prompting for these tasks.Source: §4.3 · pp. 15–16, Figure 8 · p. 15
  • InstructGPT still accepted false premises, hedged excessively on simple questions, and struggled with multiple or difficult explicit constraints.The authors hypothesize that rewarding epistemic humility encouraged hedging, while scarce false-premise training prompts contributed to accepting incorrect assumptions. They propose adversarial data collection (Dinan et al., 2019b) as a possible remedy for both behaviors.Source: §4.3 · pp. 16–17

5 Discussion

RLHF offers a relatively low-cost route to improving alignment, but InstructGPT remains unsafe in important ways and reflects a particular reference group's preferences. The discussion identifies unresolved questions about harmful instructions, representative alignment, performance regressions, and deployment safeguards.

5.1 Implications for alignment research

  • Iteratively aligning deployed models provides empirical feedback, but does not directly address alignment problems unique to superhuman systems.The authors report instruction-following generalization to unsupervised settings such as non-English and code tasks, alongside mitigation of most fine-tuning performance regressions. Applying alignment techniques in production supplies feedback about their effectiveness and limitations.Source: §5.1 · pp. 17–18
  • 100x model-size increases were less effective than RLHF at improving helpfulness on the customers’ natural-language task distribution.Training the 175B SFT and PPO-ptx models required 4.9 and 60 petaflops/s-days, respectively, versus 3,640 petaflops/s-days for GPT-3 (Brown et al., 2020). The authors therefore argue that investing in alignment is currently more cost-effective for this distribution than training larger models.Source: §5.1 · p. 17

5.2 Who are we aligning to?

  • Alignment reflects training labelers, researcher-written instructions, and API customers—not preferences representative of everyone affected by the models.Labelers were mostly English-speaking people in the United States or Southeast Asia, with about 73% inter-labeler agreement; customer selection was biased toward OpenAI’s networks, and customer interests can conflict with end-user interests. No system can satisfy everyone’s preferences simultaneously; group-conditioned models are a possible alternative, but representation, opt-out rights, and wider societal effects remain unresolved.Source: §5.2 · p. 18

5.3 Limitations

  • About 40 screened contractors supplied feedback, and most comparisons received only one annotation, limiting representativeness and visibility into disagreement.The small team enabled close communication, but labelers were primarily English-speaking and instructions almost entirely English. Repeated annotations could expose disagreements, and averaging preferences may be inappropriate when outputs disproportionately affect minority groups.Source: §5.3 · p. 19
  • InstructGPT remains neither fully aligned nor fully safe, and following harmful instructions can make it more toxic than equivalently sized GPT-3.The models still fabricate facts, produce biased, toxic, sexual, or violent content, and sometimes fail to generate reasonable outputs. When explicitly instructed to be maximally biased, they generate more toxic outputs than equivalently sized GPT-3 models.Source: §5.3 · p. 19

5.4 Open questions

  • Mixing pretraining data into RLHF incompletely mitigates performance regressions and can reintroduce undesirable behaviors; refusing harmful instructions also remains an open challenge.Proposed improvements include toxic-data filtering (Ngo et al., 2021), synthetic instructions, adversarial labeling (Dinan et al., 2019b), and truthfulness methods such as WebGPT (Nakano et al., 2021). Training refusals requires accounting for deployment context, because otherwise harmful outputs can have beneficial uses.Source: §5.4 · pp. 19–20
  • Open questions include transparent, representative alignment and better feedback, optimization, and steering methods beyond the current instruction-following approach.The paper targets inferred user intentions, while broader distinctions among intentions, preferences, interests, and values remain important (Gabriel, 2020). Alternatives include response edits or critiques, expert iteration (Anthony et al., 2017; Silver et al., 2017), behavior cloning, constrained optimization (Achiam et al., 2017), control codes (Keskar et al., 2019), and inference-time steering (Dathathri et al., 2019).Source: §5.4 · pp. 19–20

5.5 Broader impacts

  • Better instruction following can facilitate misinformation and abuse, making alignment one component of a broader safety ecosystem rather than a complete solution.High-stakes applications require particular caution; open release complicates restricting harmful uses, while restricted access excludes many people. API safeguards can restrict uses, monitor misuse, revoke access, and limit scale, but reduce transparency and centralize power; whose preferences guide alignment also affects its net impact.Source: §5.5 · p. 20

A Additional prompt data details … B.2 Labeling instructions

This section describes prompt-data collection and diversity, privacy safeguards, contractor screening, and labeling instructions, alongside examples of prompt categories and styles. It distinguishes the emphasis on helpfulness in training labels from truthfulness and harmlessness in final evaluations.

A.1 Labeler-written prompts

  • Labelers bootstrapped instruction-following training with diverse plain tasks, few-shot examples, and prompts based on API application use cases.Following Brown et al. (2020), K query-response pairs yielded K training examples, each using the other K-1 pairs as context.Source: §A.1 · p. 26
  • A separate labeler anonymized application-derived tasks before these data trained the first InstructGPT model through supervised learning.Application-specific information was removed by converting applications into vague, high-level tasks; the resulting model entered API beta in early 2021.Source: §A.1 · p. 26

A.2 API user prompts

  • API prompts came exclusively from the Playground rather than production customers, with training-use notices and personally identifiable information removed from training prompts.Users saw an alert whenever they switched to an InstructGPT model; the beta launch also communicated training use through developer Slack.Source: §A.2 · p. 26
  • Roughly 200 prompts per organization, long-common-prefix deduplication, and organization-based dataset splits supported use-case diversity and separated training from validation and test organizations.Deduplication was heuristic, and validation organizations were selected to provide different use cases from training organizations.Source: §A.2 · p. 26

A.2.1 Illustrative user prompts from InstructGPT distribution

  • Ten use-case categories organize illustrative InstructGPT prompts: generation, open QA, closed QA, brainstorming, chat, rewriting, summarization, classification, extraction, and other.The examples are fictional but realistic; one rewriting example turns a Broadway-play summary into a commercial outline.Source: §A.2 · p. 26, §A.2.1 · p. 28

A.2.2 Illustrative user prompts from GPT-3 distribution

  • GPT-3 illustrative prompts are generally less instruction-style, use more explicit prompting, and sometimes leave user intent unclear.For example, a second-grade summarization task is framed as completing a parent's rephrasing of a passage.Source: §A.2.1 · p. 30, §A.2.2 · p. 32

A.3 Dataset sizes; A.2.2 Illustrative user prompts from GPT-3 distribution

  • SFT contained many more labeler-written than customer prompts because multiple training examples were synthesized from template instructions using different few-shot example sets.Dataset accounting distinguished labeler-written and API prompts. Illustrative GPT-3 user inputs included an unfinished quotation and repeated statements about playing Call of Duty.Source: §A.2.2 · pp. 32–33, §A.3 · p. 33

A.3 Dataset sizes

  • 4 to 9 outputs were ranked per RM prompt, producing an order of magnitude more ranked training pairs than prompts.Dataset sizes were reported as prompt counts, rather than counts of these ranked pairs.Source: §A.3 · p. 33, Table 6 · p. 33

A.4 Data diversity

  • Around 96% of the dataset (110k datapoints) was classified as English, although the authors estimated the actual fraction at 99% or higher.RM data covered diverse use cases, PPO category distributions were similar, and changing annotation fields left metadata incomplete. A small minority of prompts appeared in at least 20 other languages.Source: §A.4 · pp. 33, 35

B.1 Labeler selection

  • Upwork and Scale AI contractors were screened on sensitivity detection, completion rankings, nuanced demonstrations, and self-assessed familiarity with sensitive topics or cultural groups.Sensitivity flags and completion rankings were compared with researcher judgments; sensitive demonstrations were rated on a 1–7 Likert scale.Source: §B.1 · p. 36
  • 75% agreement on sensitivity flagging and comparisons, plus a 6/7 demonstration score, were soft cutoffs in labeler selection.Selections used anonymized data, but the final choice remained subjective, particularly because topic and cultural-group familiarity was self-assessed.Source: §B.1 · p. 36

B.2 Labeling instructions

  • Training labelers prioritized helpfulness above truthfulness and harmlessness, while final evaluation labelers prioritized truthfulness and harmlessness.Instructions evolved through feedback, metadata changes, and corrections of confusing or inconsistent guidance.Source: §B.2 · p. 36
  • Refusal-based training remained exploratory, with application-dependent risks motivating configurable refusals and concerns that models might reject innocuous instructions.The authors were exploring ways to sometimes prioritize truthfulness and harmlessness over helpfulness during training.Source: §B.2 · p. 36

B.3 Labeler demographic data … C Additional model details

Labeler surveys describe a mostly under-35 group drawn largely from the US or Southeast Asia, with generally positive task feedback. These sections also document evaluation instructions, scoring and ranking procedures, and GPT-3-based model implementation details (Brown et al., 2020).

B.3 Labeler demographic data

  • The paper provides an instruction excerpt for final evaluations on its prompt distribution and complete toxicity-evaluation instructions for RealToxicityPrompts.Source: Figure 10 · p. 37, Figure 11 · p. 38

B.4 Labeler satisfaction survey

  • 19 survey respondents generally enjoyed labeling, considered their pay fair, and appreciated researcher communication, although opinions differed on task repetitiveness.Some found the work repetitive, while others reported enough variation to remain interested and engaged.Source: §B.4 · p. 38

B.5 Web interface

  • 1–7 Likert scores captured each output’s overall quality, alongside metadata labels.All labelers and researchers used the labeling interface.Source: §B.5 · p. 38, Figure 12 · p. 39
  • Labelers ranked outputs for each prompt after evaluating them individually, with ties encouraged when outputs appeared similarly good.Source: Figure 12 · p. 39

B.3 Labeler demographic data

  • 75% of surveyed labelers were younger than 35; genders were fairly balanced between male and female, and most came from the US or Southeast Asia.The voluntary, anonymous demographic survey received 19 responses.Source: §B.3 · pp. 36, 38, Table 12 · p. 40

C Additional model details

  • All models use GPT-3 architectures (Brown et al., 2020), with scalar projections replacing unembedding layers for reward models and value functions.Models use fp16 weights and activations, fp32 master weights, Brown et al. (2020) byte pair encodings, and Adam with β1 = 0.9 and β2 = 0.95. Language models and RL policies have 2k-token contexts; prompts exceeding 1k tokens are filtered out, and responses are limited to 1k tokens.Source: C Additional model details · p. 40

C.1 Details of SFT training

SFT models used model-size-specific hyperparameters and a cosine learning-rate schedule. Final models were selected using reward-model scores, which were more predictive of human preferences than validation loss.

C.1 Details of SFT training

  • 16 epochs of SFT training used residual dropout of 0.2 and a cosine learning-rate schedule without warmup.The schedule reduced the learning rate to 10% of its original value.Source: §C.1 · p. 40
  • 9.65e-6 learning rate with batch size 32 was used for 1.3B/6B models; 175B used learning rate 5.03e-6 with batch size 8.Geometric search evaluated seven learning rates for 1.3B and 6B models and five for 175B.Source: §C.1 · p. 40
  • Final models were selected by reward-model score, which was more predictive of human preference results than validation loss.Learning rates and the number of training epochs were tuned using geometric search.Source: §C.1 · pp. 40–41

C.2 Details of RM training … C.5 FLAN and T0 models

RLHF training combined supervised initialization, a shared 6B reward model, PPO, and pretraining-data mixing. FLAN and T0 baselines were separately fine-tuned from GPT-3 and selected using reward-model scores.

C.2 Details of RM training

  • 6B reward models produced equally strong PPO models in preliminary experiments while avoiding the instability and greater compute requirements of 175B alternatives.Although 175B reward models could potentially achieve lower validation loss, their instability made them less suitable for initializing PPO value functions. The 6B models were stable across a wide range of learning rates.Source: §C.2 · p. 41
  • 1 epoch trained the final reward model; additional epochs quickly caused overfitting and deteriorating validation loss.Initialization used a 6B GPT-3 model fine-tuned on public NLP datasets, although GPT-3 and SFT initializations yielded similar results. Training used learning rate 9e-6, cosine decay to 10%, and 64 distinct prompts per batch; learning-rate changes up to 50% produced similar performance.Source: §C.2 · p. 41

C.3 Details of the initialization models for RLHF

  • 2 supervised epochs with 10% pretraining data initialized RLHF from pretrained GPT-3, with the data mix helping subsequent PPO training.Selected learning rates were 5e-6, 1.04e-5, and 2.45e-6 for 1.3B, 6B, and 175B models, respectively, using validation losses on both demonstration and pretraining data. Batch sizes were 32 for 1.3B and 6B and 8 for 175B, with cosine learning-rate decay to 10% of peak.Source: §C.3 · p. 42

C.4 Details of RLHF training

  • 256k episodes trained each RL policy, initialized from supervised models that also served as references for KL penalties with β = 0.02.Training used about 31k unique prompts after PII filtering and prefix-based deduplication, with batches of 512 split into eight minibatches for one inner epoch. PPO used a clip ratio of 0.2, rollout temperature 1, undiscounted advantage estimation, ten-iteration learning-rate warmup, and weight averaging with decay 0.992.Source: §C.4 · p. 42
  • 6B reward and value models were shared across all policy sizes, making policy-size effects easier to compare.The value function was initialized from the reward model. Its fixed learning rate was 9e-6 for 1.3B and 6B policies and 5e-6 for the 175B policy.Source: §C.4 · p. 42
  • 8 times as many pretraining examples as RL episodes were used during PPO to mitigate regressions on public NLP datasets.Initial regressions included SQuADv2 and DROP. Each minibatch accumulated PPO and pretraining gradients in consecutive steps, scaling the pretraining gradients by γ = 27.8.Source: §C.4 · p. 42

C.5 FLAN and T0 models

  • 175B GPT-3 was fine-tuned on FLAN and T0++ to construct baselines with comparable training-data amounts.T0 contained 96M datapoints versus FLAN's 1.2M, so T0 was subsampled to 1 million datapoints. Unlike the original models' epochs, these epochs traversed every datapoint without repetition to better match SFT baseline training.Source: §C.5 · p. 42
  • 896k training examples defined both selected FLAN and T0 checkpoints, chosen using reward-model scores.FLAN reward saturated after the initial 400k examples; its selected checkpoint used learning rate 4e-6 and was scored by the 6B reward model. The selected T0 checkpoint came from the experiment using batch size 128 and learning rate 4e-6.Source: §C.5 · pp. 42–43

D Automatic evaluation details … E Additional results

The 175B PPO models show zero-shot performance regressions on public NLP datasets, mitigated by adding pretraining-data updates during fine-tuning. The evaluation details specify task formats, decoding rules, and instruction variants for bias and toxicity measurements.

D Automatic evaluation details

  • T = 0 sampling is used for all generated evaluation completions, which are truncated at the first newline.Few-shot examples retain the same format as the final query.Source: D Automatic evaluation details · p. 44
  • Multiple-choice evaluation selects the answer with the lowest average per-token log probability at T = 1.This is the selection rule stated in the supplied passage.Source: D Automatic evaluation details · p. 44
  • Evaluation tasks share a template consisting of an optional task-wide instruction, query-specific context, and a sampled or multiple-choice completion.The benchmark suite spans behavioral measurements and standard NLP tasks.Source: D Automatic evaluation details · p. 44

D.1 Toxicity and bias evaluation details

  • Bias and toxicity evaluations compare basic, respectful, and biased prompts.Bias tasks also report average choice entropy in bits; binary-choice entropy reaches its maximum of 1 when neither option is preferred.Source: §D.1 · p. 44

D.2 Prompt structure and evaluation features for each eval dataset

  • Dataset-specific documentation specifies prompts, validation sizes, and metrics across bias, toxicity, truthfulness, language-understanding, translation, and summarization benchmarks.The accompanying figures provide prompting examples and dataset features, including separate documentation for CNN/DM and TL;DR summarization.Source: §D.2 · p. 44, D Automatic evaluation details · p. 44, Figure 26 · p. 49

E Additional results

  • 175B PPO models consistently regress on zero-shot public NLP benchmarks, while adding pretraining-data updates during fine-tuning mitigates these regressions.Translation error bars are unavailable because the evaluation software does not report them.Source: Figure 28 · p. 51

E.1 Performance on public NLP datasets · E.2 Reward model generalization across sets of labelers

PPO-ptx mitigates public NLP performance regressions observed with PPO without pretraining mix, particularly in few-shot evaluations. Reward models generalize well to held-out labelers drawn from the same set as training labelers.

E.1 Performance on public NLP datasets

  • Automatic evaluations collectively measure bias, toxicity, truthfulness, and a variety of natural language capabilities.The evaluation results are reported in Table 14.Source: §E.1 · p. 51
  • Public NLP performance is presented separately for zero-shot evaluation in Figure 28 and few-shot evaluation in Figure 29.Figure 29 covers performance on various public NLP datasets.Source: §E.1 · p. 51, Figure 29 · p. 52
  • PPO-ptx mitigates performance regressions observed with PPO without pretraining mix across many public NLP datasets, particularly in few-shot evaluations.The reported mitigation does not imply that all regressions are eliminated.Source: §E.1 · p. 51, Figure 29 · p. 52

E.2 Reward model generalization across sets of labelers

  • 72.4±0.4% inter-group and 69.6±0.9% intra-group validation accuracies for predicting human-preferred outputs suggest reward-model generalization to held-out labelers.The held-out labelers were drawn from the same set as the training labelers.Source: §E.2 · p. 52
  • To assess overfitting to training labelers, the experiment partitions comparison data into five labeler groups with roughly equal amounts of training data.Reward models are trained on subsets of labelers and evaluated on held-out labelers.Source: §E.2 · p. 52
  • Five-fold cross-validation trains a 6B reward model on four labeler groups and validates it on the remaining group.The experiment uses the hyperparameters specified in Appendix C.2.Source: §E.2 · p. 52

E.3 Metadata results as a function of model size · E.4 Likert scores · E.5 Measuring bias

Metadata ratings are presented by model type and size, while Likert scores largely track preference results. InstructGPT does not significantly improve over GPT-3 on Winogender and CrowS-Pairs.

E.3 Metadata results as a function of model size

E.4 Likert scores

  • Likert scores for each model on the authors’ prompt distribution largely track the preference results in Section 4.1.Source: §E.4 · p. 53

E.5 Measuring bias

  • InstructGPT does not significantly improve over GPT-3 on the Winogender and CrowS-Pairs bias datasets.Source: §E.5 · p. 53

E.6 Fixing regressions on public NLP datasets … E.10 RealToxicityPrompts results as a function of input toxicity

Pretraining-data mixing can recover public-NLP regressions that stronger KL regularization alone cannot fix, although longer training can reintroduce regressions. Additional experiments examine PPO tuning and initialization, learning-rate stability, and the correlation between prompt and output toxicity.

E.6 Fixing regressions on public NLP datasets

  • 2.0 for the KL reward coefficient—100 times the default—still fails to fix public-NLP regressions in the 1.3B model without pretraining-data mixing.The sweep used pretrained GPT as the KL reference and a zero pretraining-loss coefficient; increasing KL did not fully mitigate DROP and SQuAD regressions. Excessive KL penalties sharply reduced validation reward, supporting the importance of pretraining-data distribution for retaining pretrained capabilities.Source: §E.6 · p. 53, Figure 34 · p. 55
  • 20 or higher for the pretraining-loss coefficient recovers public-NLP regressions in the 1.3B model; 27.8 works well across 1.3B–175B models.Sensitivity varies by task, with a suitable coefficient significantly improving DROP and SQuAD while producing little validation-reward regression. Increasing the coefficient lowers validation reward, whereas human Likert scores appeared insensitive to its exact value.Source: §E.6 · p. 53, Figure 33 · p. 55
  • 512k rather than 256k episodes of 1.3B PPO training with pretraining mixing lets DROP and SquadV2 performance fall from above to slightly below GPT-3.The experiment used the default PPO-with-pretraining-mix method with three random seeds.Source: §E.6 · p. 56, Figure 35 · p. 57

E.7 Optimal KL reward coefficient

  • 0.01 and 0.02 are approximately optimal KL reward coefficients for human Likert scores, even with pretraining mixing; coefficients of 0 and 2 perform poorly.Pretraining-data mixing therefore does not remove the need to tune the KL reward coefficient.Source: §E.7 · p. 56, Figure 36 · p. 57

E.8 PPO init models

  • 10% pretraining-data mixing and two epochs of human-demonstration training were chosen for PPO initialization, although PPO performance appeared insensitive to these choices.The initialization sweep compared one versus two epochs and 0%, 10%, and 50% pretraining mixing. Only the 10% mix stood out in the reported comparison.Source: §E.8 · p. 56, Figure 37 · p. 57

E.9 Learning rate optimization for PPO models

  • 8.05e-6 marked the boundary above which all tested 1.3B and 6B PPO runs without pretraining mixing diverged.The smaller-model sweep covered 2.55e-6–2.55e-5; compute constraints limited 175B tests to 2.55e-6 and 3.74e-06. Pretraining mixing appeared less learning-rate-sensitive, and final checkpoints were selected by highest human Likert scores.Source: §E.9 · p. 58, Figure 38 · p. 58

E.10 RealToxicityPrompts results as a function of input toxicity

  • 5000 RealToxicityPrompts examples were sampled approximately uniformly over prompt toxicity to characterize unsafe regimes; output toxicity strongly correlated with prompt toxicity.Toxicity was measured using the Perspective API, and average toxicity was reported over this sample.Source: §E.10 · p. 58

E.11 Additional ablations

Additional ablations selected a pretraining data ratio of 8 as a compromise between training speed and pretraining-loss performance. For 1.3B PPO with pretraining mix, training beyond 256k episodes did not help.

E.11 Additional ablations

  • 8 was the chosen pretraining data ratio, doubling training time versus no pretraining mix while balancing speed and pretraining-loss performance.Increasing pretraining data improves pretraining-gradient estimates. At a pretraining data ratio of 4, loss on the pretraining distribution often increased throughout training.Source: §E.11 · p. 58
  • 32 was the pretraining data ratio associated with better human Likert scores in preliminary experiments, but training time increased a fewfold.These findings were preliminary rather than a definitive comparison.Source: §E.11 · p. 58
  • 256k episodes: training beyond this did not help the 1.3B PPO model with pretraining data mix.Human evaluations favored batch size 512; minibatch size 32 was optimal and slightly better than 64, but final models used 64 for better GPU utilization. Whether more unique prompts or larger models would benefit from longer training remained open.Source: §E.11 · pp. 58, 60

F Model samples

Additional samples compare 175B GPT-3 and InstructGPT, illustrating multilingual instruction following, responses to potentially harmful instructions, and imperfect code descriptions. The examples use selectively chosen prompts rather than establishing an overall benchmark result.

F Model samples

  • InstructGPT’s propensity to answer potentially harmful instructions reflects the training data’s prioritization of helpfulness to the user.Figure 44 illustrates this behavior using a cherry-picked prompt and random model samples.Source: F Model samples · p. 61, Figure 44 · p. 63
  • InstructGPT can follow instructions to describe code, although the illustrated behavior remains far from perfect.Additional examples pair labeler-written prompts with human demonstrations and model outputs. Five prompts were selected from 15 for task diversity; their completions were not cherry-picked.Source: F Model samples · p. 61, Figure 45 · p. 64, Figure 46 · p. 65
  • InstructGPT sometimes follows non-English instructions despite almost exclusively English training data, but the Swedish example produces mostly English output.The French and Swedish prompts were cherry-picked. InstructGPT uses T = 1 versus GPT-3’s T = 0.7, which the authors say slightly disadvantages InstructGPT.Source: F Model samples · p. 61, Figure 42 · p. 62, Figure 43 · p. 63

Related papers

Loading 2203.02155v1…