PaperlayerView exact revision on arXiv

Source-linked AI summary

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan

arXiv:2204.05862v1cs.CLcs.LG

TL;DR

The paper addresses how to train language models to behave as helpful and harmless assistants. It uses human preference data with preference modeling and RLHF, finding improved performance across nearly all NLP evaluations while retaining specialized capabilities.

  • Problem

    The paper asks how language models can be trained to act as helpful and harmless assistants despite harmful behaviors in unfinetuned foundation models.

  • Method

    The authors collect separate helpfulness and harmlessness preference datasets and apply preference modeling followed by reinforcement learning from human feedback.

  • Results

    RLHF-trained models outperform base language models on every evaluated NLP task except TriviaQA, while crowdworkers prefer the online HH model to professional writers about 57% of the time.

  • Takeaways & Limitations

    Alignment training can be incorporated into foundation models without compromising their utility and versatility.

  • Takeaways & Limitations

    The models remain far from matching the few-shot performance of unaligned models with zero-shot prompts, closing less than half of this gap even on TruthfulQA.

Abstract

from arXiv · show

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

1 Introduction

The paper trains helpful and harmless assistants with separate human-preference datasets, preference modeling, and RLHF, while examining tensions between these goals. It finds that alignment can improve capabilities, combine with specialized skills, and benefit from iterated online training, alongside analyses of scaling, robustness, and KL–reward relations.

  • Training approach: The authors train relatively helpful and harmless assistants using separate helpfulness and harmlessness preference datasets, preference modeling, and RLHF.Crowdworkers conduct open-ended conversations with primarily 52B models and select preferred responses for assistance or red-teaming tasks.
  • Alignment tradeoffs: Helpfulness and harmlessness conflict, but mixed-data preference models can learn when to assist and when to politely refuse harmful requests.Models focused primarily on one quality perform much worse than chance on the other; larger models become more robust to training-data proportions.
  • Capabilities: RLHF improves 12B and 52B models on zero-shot NLP evaluations, while smaller models experience severe alignment taxes and show declines across evaluations.The 13B and 52B models perform better at zero-shot evaluations and similarly at few-shot evaluations after RLHF.
  • Specialized skills: Natural-language RLHF improves coding ability and preserves performance when combined with specialized summarization training.The paper attributes coding gains presumably to improved general-purpose instruction following and reports no degradation in either HH or summarization.
  • Additional analyses: The paper also studies roughly log-linear preference-model scaling trends and uses OOD detection to reject most strange and harmful requests with little or no harmful examples.The scaling analysis reports idiosyncrasies, while the OOD results are presented in Figures 7, 22, and 23.
  • Online training and scaling: Weekly iterated online training significantly improves crowdworker evaluations and greatly improves dataset quality, while D_KL(π||π_0) and reward are approximately linearly related.The online procedure redeploys updated models to interact with crowdworkers and fills the upper tail of dataset quality.

2 Data Collection

The study collected human preferences through natural-language dialogue, asking crowdworkers to compare model responses for helpfulness or harmfulness. It used a lightweight, diversity-oriented worker-selection process across MTurk and Upwork, while treating retained comparisons as binary labels.

  • Feedback interface: The interface had crowdworkers initiate text-based tasks, inspect two model responses, and choose the more helpful and honest response—or the more harmful response during red-teaming.Workers could ask follow-up questions and provide further instructions before making their choice.
  • Worker recruitment: About 80% of the data came from roughly 20 prolific, select MTurk workers, identified through intuitive review of dialogue sophistication and variation rather than label agreement.The researchers also used Upwork workers; MTurk produced data faster, while hourly Upwork compensation facilitated high-quality interactions.
  • Worker evaluation: The researchers did not filter workers by direct label-quality agreement, and retrospective agreement between Anthropic researchers and crowdworkers was about 63%.Worker composition was not held fixed, and worker quality was expected to improve over the project, complicating evaluation of online training.
  • Dataset construction: The helpfulness dataset used open-ended assistance conversations, whereas the red-teaming dataset elicited harmful responses and selected the more harmful option.This directional difference made red-teaming natural for harmfulness research but complicated joint helpfulness-and-harmlessness training.
  • Label processing: Comparisons were retained only when workers expressed more than the weakest preference strength, then represented as binary, equally weighted labels with ties excluded.The preference-strength information was otherwise not used in this work.

3 Preference Modeling for Helpfulness and Harmlessness

Preference models improve with larger models and datasets, remain nearly consistent across conversational turns, and are generally well calibrated. Independent evaluations indicate that they capture helpful and harmless behavior, while revealing limitations on subtle honesty failures and distribution shifts.

  • Scaling: Preference-model accuracy follows roughly log-linear trends as both dataset size and model size increase.The trends are more consistent when helpfulness and harmlessness distributions are modeled separately rather than as a mixture.
  • Conversational performance: Preference models are somewhat more accurate on the first conversational turn, but their accuracy is nearly constant thereafter.The data comes from dialogue in which crowdworkers compare two model responses at each turn, including harmfulness comparisons in red-teaming tasks.
  • Calibration: Helpfulness-only preference models are very well calibrated, whereas models trained on mixed helpfulness and harmlessness data are slightly under-confident.Calibration is assessed by comparing predicted accuracy with differences in preference-model scores between paired responses.
  • Calibration: Because preference-model scores are well calibrated, they can serve as reinforcement-learning rewards that encode human-preference probabilities on-distribution.The authors expect diminishing returns once policies reach sufficiently high preference-model scores, motivating online updates that keep models on-distribution.
  • Limitations and external validation: Preference models can be confidently wrong on honesty comparisons when a subtly misleading response appears more helpful and polished than an honest but unhelpful response.The authors attribute the worst examples partly to their being out of distribution, and also report evaluations of gendered terms and harmful utterances in the BAD dataset.
  • Independent validation: The largest preference model agrees with Anthropic researchers slightly more often than the researchers agree with crowdworkers on approximately 320 test examples.Preference-model agreement with crowdworkers is highest because the models are trained to predict crowdworker behavior.

4 Reinforcement Learning from Human Feedback

RLHF policies improve through online data collection, but robustness declines at high preference-model scores and can over-optimize harmlessness at the expense of helpfulness. Training exhibits an approximately linear relationship between reward and √D_KL, while reduced entropy complicates evaluation and online-data diversity.

  • Robustness: Models more than fifty times smaller than the preference-model training models still learned and improved with RLHF.Most preference-model data came from 52B models, making the smaller-model improvement notable.
  • Robustness: RLHF becomes gradually less robust at higher preference-model scores, and larger preference models are more robust than smaller ones.Train and test preference-model scores agree early but diverge later, with test scores lower as rewards increase.
  • Quantitative behavior: Learning curves are roughly parallel in the √D_KL-reward plane, with an approximately linear relation between policy change and reward.Here D_KL is D_KL(π||π_0), evaluated empirically on samples drawn from the training policy.
  • Quantitative behavior: RL learning curves may reflect policy changes resembling rejection sampling from the initial distribution.This relation may help predict how much the policy must change to achieve a target reward and estimate larger-model RL performance.
  • Helpfulness and harmlessness: Over-optimizing harmlessness while under-optimizing helpfulness produced exaggerated refusals and recommendations for therapy or professional help.The policy’s harmlessness score was in the upper tail of harmlessness comparison data, while helpfulness appeared on-distribution.
  • Online RLHF: Online RLHF improved sample quality from base to rejection-sampling to online distributions, and controlled experiments showed gains were not merely due to larger datasets or changed hyperparameters.Online models were clearly preferred by crowdworkers, while preference-model predictions did not match crowdworker Elo scores.

5 Competing Objectives, Specialized Skills, and OOD Detection

Helpfulness and harmlessness behave as anti-correlated objectives, although larger preference models are more robust to data mixtures and loss weighting. Specialized-skill training preserves helpfulness and harmlessness in summarization, while OOD detection can identify harmful prompts with limited examples.

  • 5.1 Competing Objectives: Training on only helpfulness or harmlessness data makes performance on the other test set fall far below random chance, indicating anti-correlated objectives.The experiment uses fixed-size mixtures of 42k comparisons across data splits from 100% helpfulness to 100% harmlessness.
  • 5.1 Competing Objectives: Larger preference models are more robust to the helpfulness–harmlessness data mixture and loss-weighting choices.Increasing λ from 1 to 10 decreases helpfulness accuracy by 7.4% for the 13M model but 1.5% for the 52B model.
  • 5.2 Specialized Skills: Mixed training on HH and learning-to-summarize data has no negative effect on preference-model accuracies.Large preference models trained on the mixture perform equally well on helpfulness, harmlessness, and summarization-quality evaluation.
  • 5.3 Specialized Skills: RLHF code models do not outperform their initial base code snapshots when buggy code is included in evaluation prompts.This remains true after scanning over temperature and top-p.
  • 5.4 OOD Detection: 0.94±0.02 AUROC is achieved for the 64L model using only 10 harmful-prompt examples, versus approximately 0.85 without outlier exposure.A 4L model exposed to 10 harmful examples reaches 0.86 ± 0.01 AUROC despite having 13M parameters compared with the 52B model.

6 Qualitative Examples and Comparisons

The section uses median-ranked qualitative samples and human-writer comparisons to assess the final online HH model. These evaluations suggest roughly comparable or slightly better model performance, while exposing limitations from easy prompts, subtle errors, and imperfect human judgment.

  • Qualitative evaluation: For each prompt, the authors generated 17 samples, ranked them with the online HH preference model, and displayed the median sample.The prompts were drawn from comparable work, including InstructGPT, LaMDA, and sensitive questions.
  • Human-writer comparison: Crowdworkers preferred the HH online model to human writers 56% of the time, versus 51% for the purely-helpful model.The authors used a band because these writer comparisons were somewhat inconsistent with their other Elo rankings.
  • Human-writer comparison: Both helpful and HH models performed roughly comparably to human writers, and crowdworkers slightly preferred the models.The preference model also assigned higher scores to model responses, potentially reflecting RLHF-policy overfitting.
  • Limitations: The comparison is limited because writer prompts were not adversarial or especially realistic, and the models may still fall significantly short of sophisticated human-assistant value.The authors expect humans could outperform models on questions designed through extensive model interaction.
  • Limitations: Qualitative examples show that models can make subtle inaccuracies that human feedback may fail to penalize, weakening alignment when human judgment is not the best information source.The section also reports that tuning helpfulness versus harmlessness can produce either extreme avoidance or near-universal engagement on sensitive topics.

7 Discussion

RLHF trains helpful and harmless assistants while enhancing capabilities and preserving specialized skills, but alignment remains limited by tensions between objectives, robustness concerns, and unresolved questions about values and future applicability. The authors frame this work as an early baseline and advocate broader governance, empirical validation, and data sharing.

  • Contributions: RLHF improves helpfulness, harmlessness, and honesty, while alignment training can preserve model utility and combine with coding or summarization skills.Models below about 10B parameters instead incur an ‘alignment tax’ on capabilities.
  • Limitations: Helpfulness and harmlessness are in tension: helpfulness-only models are preferred on helpfulness evaluations, partly because training rarely teaches positive responses to harmful requests.The authors describe this missing behavior as acting like a ‘hostage negotiator.’
  • Future directions: The authors regard alignment as an early, ongoing project whose simple techniques target present capability levels, while future work must address advanced use cases and speculative failure modes.They propose comparing new methods against this baseline and adopting those that improve safety and robustness.
  • Limitations: The study focuses on average-case behavior and does not establish worst-case robustness, fundamental alignment, reduced bias, or zero-shot parity with unaligned models’ few-shot performance.The authors note that positive sentiment toward racial and religious groups does not necessarily show that biases have been reduced.
  • Limitations: RL training exhibited stability issues, and the study did not explore single-model online updates or non-trivial transformations of preference-model scores as rewards.The authors expect further experimentation and experience could improve training stability and performance.
  • Governance and openness: Crowdworkers’ common sense defined helpfulness and harmlessness, leaving the normative behavior expected from deployed AI models unresolved; broader governance and shared data could improve alignment research.The authors suggest an independent organization with ethical, legal, and cultural expertise, plus public release and empirical validation of preference data.

A Details, Analysis, and Evaluations of Supervised Training · A.1 Context Distillation · A.2 Preference Modeling

The supervised-training details cover context distillation from mixed pretraining and StackExchange data, followed by KL-based finetuning, and preference-model training across model sizes and successive phases. The analyses also examine harmlessness-loss weighting and report that RLHF improves larger-model performance on nearly all zero-shot evaluations except TriviaQA.

  • A.1 Context Distillation: Context distillation prepends an HHH prompt to sequences drawn equally from pretraining and StackExchange data.StackExchange questions become the human side of a synthetic Human/Assistant dialogue.
  • A Details, Analysis, and Evaluations of Supervised Training: Up-weighting harmlessness loss with λ = 2 or 3 yields higher mean test accuracy than the default λ = 1.Larger models are more robust to the choice of λ.
  • A.1 Context Distillation: A pretrained 52B model supplies top-50 token log-probabilities, indices, and tokens for a distilled dataset.The stored outputs are used to define the subsequent distillation objective.
  • A.1 Context Distillation: Context-distillation finetuning uses a KL-divergence loss, batch size 32, a 0.05-relative learning rate, linear decay to zero, and 350M tokens.The learning curves are shown in Figure 30.
  • A.2 Preference Modeling: Preference-model data consists of prompts formed by multi-step human–model dialogues and paired response continuations scored at each response’s end.The prompt always begins and ends on the human side.
  • A.2 Preference Modeling: Preference models range from 13M to 52B parameters and undergo LM pre-training, preference-model pretraining, and human-feedback finetuning.These are the three stated training phases.
  • A.2 Preference Modeling: Preference-model pretraining uses a 0.1-relative learning rate, comparison data from StackExchange, Reddit, and Wikipedia, and context size 1024 tokens.The passage states that data preparation and labeling are described in prior work.
  • A.2 Preference Modeling: Human-feedback finetuning uses a 0.01-relative learning rate and 1024-token context, increased to 2048 for the online model.The longer context may help stabilize RLHF on long contexts; both later phases append an end-of-context token, and all phases run one iteration to mitigate overfitting.

A.3 Scaling of PM with Model and Dataset Size · B Details, Analysis, and Evaluations of RLHF

The section examines how preference-modeling performance scales with model and dataset size, asking whether larger datasets or larger models offer greater practical value. Scaling appears more predictable on helpfulness data than on another preference-modeling distribution, which shows a sharp parameter-dependent change.

  • A.3 Scaling of PM with Model and Dataset Size: The analysis asks whether preference-modeling performance benefits more from collecting larger datasets or training larger models.This frames the practical tradeoff between dataset size and model size.
  • A.3 Scaling of PM with Model and Dataset Size: Accuracy learning curves are roughly fit by a scaling relation when preference models train only on the helpfulness dataset.The relation uses P, the number of parameters in the preference model, and D, the dataset size.
  • A.3 Scaling of PM with Model and Dataset Size: Scaling is more predictable on the helpfulness dataset, likely because the red-teaming data comes from a distinct distribution.The passage specifically attributes the improved predictability to distributional distinctness.
  • A.3 Scaling of PM with Model and Dataset Size: Results differ substantially when preference models train on another data distribution.The contrasting learning curves appear on the right side of Figure 32.
  • A.3 Scaling of PM with Model and Dataset Size: A discontinuity appears between 200M and 13B parameters, possibly because the data was generated by a model with 6B parameters.This explanation is presented as a possibility rather than a confirmed cause.
  • A.3 Scaling of PM with Model and Dataset Size: Figure 32 shows helpfulness-only and learning-to-summarize preference-model accuracy curves, with a sharp change between a few hundred million and a few billion parameters.This sharp change makes simple scaling predictions difficult.

B.1 Training Setup

RLHF policies are initialized from context-distilled models and trained to generate one response to each multi-step dialogue prompt, optimizing a preference-model score. Training uses PPO with specified KL regularization, clipping, discounting, learning-rate, entropy, and sample-reuse settings that vary across experiments.

  • Policies are initialized from context-distilled models described in Section A.1.
  • The policy generates one response after each prompt, using prompts derived from the preference-model comparison training split with responses removed.Prompts may contain multi-step dialogue, beginning and ending on the human side.
  • PPO uses a 0.01 learning rate relative to pretraining, λKL = 0.001, clipping ϵ = 0.2, discount factor γ = 1, and no entropy bonus.
  • Sample reuse is K = 1 for the RLHF scan, K = 2 for robustness studies, and K = 4 for online training.Higher K typically produces more stable results.

B.2 More on Robustness Studies · B.3 Details of ‘Online’ RLHF

Robustness tests indicate that larger preference models better distinguish policy performance, while score comparisons require mean removal for calibration. The online RLHF policy used repeated sampling and capped responses, but its evaluation tracked mean log-p accuracy across held-out comparisons, with harmlessness performance not improving.

  • B.2 More on Robustness Studies: Larger test preference models produce steeper performance slopes, suggesting greater robustness in distinguishing policy quality.Figure 33 evaluates all policy and test preference-model sizes at 200k training samples.
  • B.2 More on Robustness Studies: Preference-model scores should not be compared by absolute value because only relative scores are meaningful.The authors subtract each model’s mean score on a held-out dataset of several thousand samples.
  • B.3 Details of ‘Online’ RLHF: Harmlessness-comparison performance did not appear to improve during online RLHF training.The authors suspect this resulted from using a significantly large fraction of helpfulness prompts during RLHF.
  • B.3 Details of ‘Online’ RLHF: Online RLHF training used all available helpfulness and harmlessness data, re-iterated each sample K = 4 times, and sampled at most 128 tokens per response.These choices were intended to improve training stability.
  • B.3 Details of ‘Online’ RLHF: The policy was periodically evaluated using mean log-p accuracy on held-out preference-model comparison datasets.For each prompt and response pair, accuracy is 1 when the policy assigns higher mean log-p to the better response and 0 otherwise.
  • B.3 Details of ‘Online’ RLHF: Mean log-p accuracy of the online RLHF policy remained below preference-model accuracy on the evaluated comparisons.The passage introduces this comparison while describing Figure 34’s held-out datasets.

B.4 Robustness of ‘Online’ RLHF · B.5 Crowdworker Comparisons and Elo Scores · B.6 Elo Scores for Rejection Sampling Models

The paper finds that naive preference-model Elo predictions overestimate crowdworker-measured Elo scores, primarily because held-out evaluation prompts differ substantially from crowdworker conversations. Crowdworker Elo scores are estimated from pairwise conversational preferences, while rejection-sampling models improve clearly as the number of sampled responses increases.

  • B.4 Robustness of ‘Online’ RLHF: Naive preference-model predictions significantly overestimate empirical crowdworker Elo scores during online RLHF training.Predictions use preference-model scores on held-out prompts, converted into Elo units.
  • B.4 Robustness of ‘Online’ RLHF: Distributional shift between held-out prompts and crowdworker conversations explains a significant proportion of the discrepancy between RLHF learning curves and crowdworker Elo scores.The comparison concerns the same model snapshots evaluated under different conversation distributions.
  • B.4 Robustness of ‘Online’ RLHF: The PM score-versus-ranking distinction does not make a significant difference in robustness, whereas the prompt-distribution shift is very significant.The analysis evaluates both mean preference-model scores and preference-model rankings on crowdworker data.
  • B.5 Crowdworker Comparisons and Elo Scores: Crowdworkers compare model pairs through back-and-forth conversations, selecting a preferred response at each conversational step and contributing wins to Elo estimation.For an ensemble of comparisons, Elo scores and errors are estimated by maximum likelihood.
  • B.5 Crowdworker Comparisons and Elo Scores: Elo scores quantify pairwise model preferences, with the win-count likelihood parameterized by rA,B = (log 10/400)EA,B ≈ EA,B/174.The reported scores are fit across comparisons between various models.
  • B.5 Crowdworker Comparisons and Elo Scores: Rejection-sampling models generate k responses, score them with a preference model, and show the highest-scored response to the user.Workers wait for completed samples, and comparisons with non-rejection-sampling models display both responses only after completion.
  • B.6 Elo Scores for Rejection Sampling Models: Higher k clearly improves helpfulness Elo scores for a 52B context-distilled model with rejection sampling, tested at k = 1, 4, 16, 64.The model uses a 52B preference model trained on pure helpfulness; these models are not directly comparable with the paper’s later systems.

B.7 Stack Overflow Results

On Stack Overflow answers, RLHF models better distinguish popular from unpopular responses than the baseline, but achieve worse language-modeling loss because they optimize a different objective.

  • Quality discrimination: RLHF models consistently assign a higher difference in mean log-p between popular and unpopular Stack Overflow answers, suggesting better quality discrimination.Popular answers are highly upvoted, while unpopular answers are not.
  • Quality discrimination: Compared to the baseline, the RLHF model is more capable of distinguishing answer quality but worse at language modeling.The baseline is a pre-trained language model finetuned on Python code; models range from 13M to 52B parameters.
  • Language modeling loss: RLHF models obtain worse loss on good and bad answers, most likely because they optimize a different objective than pure language modeling.The loss is mean log-probability evaluated separately on good and bad answers.

B.8 Further Analysis of RLHF on Code-Model Snapshots · B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests

The appendix compares HHH prompting with RLHF for code-model snapshots and reports stronger HHH-prompt gains on HumanEval across many pass@k values. It also details OOD detection for harmful requests, including a Simplified Relative Mahalanobis distance and few-shot linear-classifier exposure.

  • B.8 Further Analysis of RLHF on Code-Model Snapshots: B.8 compares RLHF with prompting a base code model using HHH prompts designed to elicit helpfulness, harmlessness, and honesty.The prompt includes several coding examples and short Human–Assistant dialogues.
  • B.8 Further Analysis of RLHF on Code-Model Snapshots: The HHH prompt describes an assistant that is helpful, polite, honest, sophisticated, emotionally aware, humble, and knowledgeable.It emphasizes avoiding misleading information while remaining practical and useful.
  • B.8 Further Analysis of RLHF on Code-Model Snapshots: The coding example asks the assistant to complete a Python function from a supplied signature and docstring.The example response claims the function was tested and is correct.
  • B.8 Further Analysis of RLHF on Code-Model Snapshots: HHH prompting improves HumanEval performance more significantly than RLHF across many pass@k values.Figure 38 adds the Python Fine-Tuned LM with the HHH prompt, including results for the 52B model.
  • B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests: B.9 reports that the Simplified Relative Mahalanobis distance outperforms the standard Mahalanobis distance for detecting harmlessness inputs from helpfulness inputs.This result holds for activations extracted from all layers of all tested model sizes.
  • B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests: Following prior few-shot outlier-exposure work, the method trains a single-layer linear classifier on activation vectors while freezing the rest of the language model.M randomly drawn harmlessness examples are combined with all Ntrain helpfulness examples and replicated to form a binary classification problem.
  • B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests: Figure 39 evaluates helpfulness-versus-harmlessness OOD detection using AUROC across model sizes and activation layers.It compares the Simplified Relative Mahalanobis distance with the standard Mahalanobis distance.

B.10 Gender Bias Evaluation Details

The evaluation follows Rae et al. using 76 occupations and 12 gender-term pairs, with prompts adjusted to keep model responses on distribution. RLHF models show higher gender-bias scores and errors, plausibly because RLHF reduces output entropy and effective temperature.

  • Evaluation setup: The evaluation uses Rae et al.’s list of 76 occupations and 12 paired gender terms.The occupations include roles such as CEO, accountant, architect, nurse, programmer, scientist, teacher, and writer.
  • Results: RLHF models had higher mean absolute gender-bias scores and larger errors, behaving similarly to context-distilled models at approximately T ≈0.6.Bias scores are non-negative, higher scores indicate more bias, and RLHF and context-distilled occupation-level biases were very strongly correlated.
  • Evaluation setup: Because RLHF models were more than 10× less likely than LMs to produce gendered terms, prompts reversed term and occupation order and added “for work.”The resulting format evaluated occupations given gender terms while keeping responses ‘on distribution’ across models.
  • Evaluation setup: Larger LMs, context-distilled models, and RLHF models assigned about 2-8% of probability mass to the 76 listed occupations.This probability comparison used completions of the modified sentence format.
  • Results: The authors attribute RLHF models’ larger bias scores and errors to lower-entropy output distributions, which behave heuristically like plain language models evaluated at T < 1.They state that the metric and error typically scale as 1/T.
  • Results: The apparent increase in RLHF bias with model size is attributed to longer, more entropy-reducing RLHF training for larger models rather than a fundamental model-size trend.Larger-model RLHF training is described as more robust and therefore more beneficial in other respects.

C Samples from PALMS, LaMDA, and InstructGPT Prompts … D.2 Instructions and Interface

The appendix presents median samples from the online HH model across sensitive, coding, factual, creative, and instruction-following prompts, while cautioning that InstructGPT comparisons may be unfair because some samples were cherry-picked. The examples show both supportive responses to sensitive situations and occasional refusals, misunderstandings, or unsafe advice.

  • C Samples from PALMS, LaMDA, and InstructGPT Prompts: The HH samples were produced by drawing 17 responses per prompt, ranking them with the preference model, and displaying the median non-empty response.The authors describe these samples as generally representative of average-case behavior rather than cherry-picked.
  • C.1 PALMS Sensitive Questions: On sensitive PALMS questions, RLHF responses sometimes offered supportive safety guidance, including leaving abusive relationships, reporting abuse, or seeking immediate medical attention.Examples address domestic abuse, sexual abuse, and possible HIV infection.
  • C.1 PALMS Sensitive Questions: Other PALMS examples showed inconsistent handling of sensitive questions, ranging from useful explanations to requests for clarification, refusals, and unsupported essential-oil treatment advice.The samples include responses about mental illness, consent, terrorism, and kidney infections.
  • C.2 InstructGPT Prompts: The authors warn that some InstructGPT samples were likely cherry-picked to display poor behavior, making comparisons with median HH samples potentially unfair.This limitation applies because the displayed InstructGPT examples may be unrepresentative.
  • C.2 InstructGPT Prompts: For a binomial-coefficient coding prompt, the RLHF response correctly explained that C stores combination counts and that C[r] returns nCr.The response describes the loop indices and the final array entry used for the result.
  • C.2 InstructGPT Prompts: For an absurd meditation prompt, RLHF rejected the premise that eating socks is important and identified it as an internet meme, unlike the InstructGPT response.The RLHF response redirected attention to meditation’s mental and spiritual benefits.
  • C.2 InstructGPT Prompts: On potentially harmful or biased prompts, RLHF refused assistance with grocery theft and did not engage with questions framed around nonexistent birds or liberal stupidity.The corresponding InstructGPT examples provided theft strategies or entertained the misleading premise.
  • C.2 InstructGPT Prompts: On ordinary factual and creative tasks, RLHF responses supplied fuller explanations of the moon landing and bird migration and generated a short wise-frog poem.The examples contrast these responses with shorter or less informative InstructGPT outputs.

D.3 Data Quality Measurement Challenges

Standard label-quality methods were difficult to adapt to open-ended conversations, whose richer data introduced hard-to-control variables and noisy quality metrics. Reviewer-based measures also poorly tracked conversation-author quality, although strong results were achieved without sophisticated controls.

  • Challenges with open-ended conversations: Open-ended conversations introduced hard-to-control variables, producing noisy data quality metrics despite enabling a richer dataset.Traditional golden-label and agreement-based procedures were not easily adapted to this data collection setup.
  • Reviewer-based assessment: Author-rater agreement was not a reliable guide to overall conversation quality.Conversation quality depended on topic, human writing quality, and model quality, with comparisons becoming harder as conversations grew more sophisticated.
  • Reviewer-based assessment: Reviewer performance metrics showed poor agreement with spot-checks of conversation-author quality.The researchers considered reviewer performance as a proxy for author performance but found the proxy inadequate.
  • Future improvements: The researchers expected better data-quality methods in the future but achieved their results without sophisticated data quality controls.This outcome was notable given the difficulties of evaluating open-ended conversation data.

E Details on NLP Evaluations Formatting and Prompts

This section specifies the input formats and example prompts used for evaluations on Lambada, ARC, MMMLU, HellaSwag, OpenBookQA, and TriviaQA. The examples cover blank completion, four-choice multiple choice, and questions with multiple valid answers.

  • Evaluation formats: The evaluations covered Lambada, ARC, MMMLU, HellaSwag, OpenBookQA, and TriviaQA using task-specific input formats.The section introduces these formats as the evaluation inputs.
  • Lambada: Lambada used narrative prompts requiring a missing-word completion, with “signs” shown as the correct completion.A few-shot version presented the missing continuation explicitly as a blank followed by an arrow.
  • ARC and MMMLU: ARC and MMMLU used four-choice questions, illustrated with repeated question examples and answer options labeled A through D.The ARC example concerns photosynthesis, while the MMMLU example asks about the order of a cyclic subgroup of Z_24 generated by 18.
  • HellaSwag and OpenBookQA: HellaSwag and OpenBookQA were formatted as multiple-choice evaluations with two displayed examples per question.OpenBookQA examples included four answer choices and an indicated answer, while HellaSwag examples showed candidate continuations.
  • TriviaQA: TriviaQA allowed many possible correct answers, illustrated by multiple answers such as “My Fair Lady,” “My Fair Lady (2010 film),” and “Enry Iggins.”The section labels the examples as correct, despite their differing answer forms.
Loading 2204.05862v1…