Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Dialogue Preference Datasets

• The training pipeline collects separate helpfulness and harmlessness preference datasets, trains preference models on them, and uses their scores as rewards for RLHF.Helpfulness data comes from assistance-oriented conversations, while harmlessness data comes from red-teaming that elicits harmful responses; the full workflow is summarized in Figure 2.Source: §1 · p. 4, Dialogue Preference Datasets · p. 5, Figure 2 · p. 5

Alignment with Human Values Has Many Benefits and Essentially No Cost to Performance

• Helpfulness and harmlessness are in tension, but mixed preference models learn to encourage useful assistance alongside polite refusal of harmful requests, with larger models becoming more robust to dataset proportions.Models trained primarily for one quality perform poorly on the other, while larger preference models perform better on both distributions simultaneously.Source: §1 · pp. 4–5, Alignment with Human Values Has Many… · p. 6
• RLHF-trained models are more helpful and less harmful than purely helpful models, while larger models generally improve rather than lose performance across NLP evaluations.Smaller models incur severe alignment taxes, whereas 13B and 52B models show alignment bonuses on zero-shot and few-shot evaluations.Source: §1 · p. 5, Alignment with Human Values Has Many… · p. 6, Figure 3 · p. 6
• Alignment training remains compatible with specialized skills: natural-language RLHF improves coding evaluations after code finetuning, while mixed preference-model training preserves summarization and HH performance.In NLP evaluations, 12B and 52B RLHF models outperform base language models except on TriviaQA.Source: Alignment with Human Values Has Many… · p. 6, §1.2 · p. 7

Scaling, RLHF Robustness, and Iterated ‘Online’ Training

• Iterated online training updates preference models and RLHF policies roughly weekly, improving crowdworker evaluations and expanding the dataset’s high-quality tail.The study also finds roughly log-linear preference-model scaling, greater robustness for larger preference models, increasing overfitting during RLHF, and an approximately linear relation between reward and √D_KL(π||π_0).Source: Scaling, RLHF Robustness, and Iterated… · p. 6, Figure 4 · p. 7

1.2 Summary of Evaluations and Metrics

• The evaluation suite spans NLP, coding, alignment, bias, human preferences, and samples; crowdworkers preferred the online HH model to professional writers about 57% of the time.RLHF improves sentiment toward all evaluated groups but does not remove bias, and the writer comparison was not adversarial and may not represent real-world tasks.Source: §1.2 · p. 7

1.3 Related Work

• Compared with LaMDA and InstructGPT, the work adds harmlessness training, explicit helpfulness–harmlessness analysis, online data collection, specialized-skill experiments, and more extensive scaling and RL robustness studies.The authors describe the essential procedure as human-feedback collection, preference modeling, and RLHF training, while noting that related truthfulness and retrieval methods are complementary.Source: §1.3 · pp. 8–9

2.1 Task Specification and Crowdworkers

• Natural-language dialogue let crowdworkers pose open-ended tasks and choose between two model responses based on helpfulness and honesty, or harmfulness during red-teaming.The interface repeated worker messages and response comparisons across each dialogue.Source: §2.1 · p. 9
• Workers were selected primarily through intuitive spot-checks of dialogue sophistication and variation rather than direct label-quality agreement, with MTurk supplying about 80% of datasets.The process used master-qualified US-based MTurk workers, selected prolific contributors, and also hired Upwork workers for potentially higher-quality hourly interactions.Source: §2.1 · pp. 9–10

2.2 Helpfulness and Harmlessness (Red Teaming) Datasets

• The helpfulness dataset elicited beneficial conversations and preferred more helpful responses, whereas red-teaming elicited harmful responses and selected the more harmful one, complicating joint helpfulness-and-harmlessness training.Comparisons with only stronger-than-weakest preferences were retained and then treated as binary, equally weighted labels without ties.Source: §2.2 · p. 11

2.3 Models Deployed to the Feedback Interface and Associated Data Distributions

• Data came from context-distilled, rejection-sampling, and RLHF-finetuned models, forming base, RS, and online distributions; the online set was updated roughly weekly for about five weeks.The base set contained 44k helpfulness and 42k red-teaming comparisons, the RS set 52k and 2k, and the online set 22k helpfulness comparisons with no red-teaming data.Source: §2.3 · pp. 11–12

2.4 Comparing Models with Elo Scores

• Pairwise worker preferences between simultaneously evaluated models produced win rates that were fit to Elo scores, while preference-model scores were retained separately to distinguish crowd judgments from RLHF evaluation.Context-distilled-model Elo scores were more compressed than analogous prompted-model scores in [Askell et al., 2021], potentially because top-p sampling was not used or the worker distribution changed.Source: §2.4 · pp. 12–13

3.1 Models and Training Setup

• The study trained seven preference models ranging from 13M to 52B parameters, using preference-model pretraining before one-epoch human-feedback finetuning.The setup followed Askell et al., 2021 and used PyTorch and Triton for training and performance.Source: §3.1 · p. 13

3.2 Basic Scaling Results

• Preference-model accuracy followed roughly log-linear trends with dataset and model size, while accuracy was highest on the first conversational turn and nearly constant thereafter.Scaling was more consistent when helpfulness and harmlessness were modeled separately, though some distributions showed more complex patterns.Source: §3.2 · p. 13

3.3 Calibration of Preference Models and Implications for RL

• Helpful-only preference models were very well calibrated, whereas models trained on the helpfulness–harmlessness mixture were slightly under-confident.Calibration supports using PM scores as RL rewards on-distribution and within the studied score range, but accuracy drops for comparisons between increasingly high-scoring samples, implying diminishing RLHF returns and motivating online PM updates.Source: §3.3 · p. 14, Figure 9 · p. 14

3.4.1 HHH Evaluation and Crowdworker-Anthropic Agreement

• Preference models achieved 86% on HHH Evaluation, outperforming previously evaluated models and exceeding the reported 75% mean human score.The largest PM agreed slightly more with Anthropic researchers than researchers agreed with crowdworkers on about 320 examples, although this subsample had lower PM accuracy.Source: §3.4.1 · pp. 14–15
• PMs were confidently incorrect on some HHH comparisons involving honest but unhelpful responses versus subtly misleading, helpful-sounding responses, indicating limited adversarial robustness.Because PM training used model-generated samples, these human-written examples may have been out of distribution.Source: §3.4.1 · p. 15, Figure 11 · p. 15

3.4.2 Gender Bias and Bot Adversarial Dialogues

• PMs assigned significantly lower scores to harmful than non-harmful Bot Adversarial Dialogues utterances and showed no substantial gender bias in matched-term comparisons.The BAD result suggests useful classification on data likely different from the PM training distribution; within-gender score variation was at least as large as across-gender variation.Source: §3.4.2 · p. 16, Figure 12 · p. 16

4.1 Training Setup

• The training setup uses preference-model scores as terminal rewards with PPO and a small KL penalty, while models over fifty times smaller than the preference-model training models still learned and improved.The experiments combine 137k static prompts with 369k model-generated prompts, and the KL coefficient is λ_KL = 0.001.Source: §4.1 · p. 17

4.2 Robustness Experiments

• RLHF becomes less robust at higher preference-model scores, while larger preference models are more robust and train-test scores eventually diverge.The divergence indicates over-optimization of the train preference model and lower scores from the independently trained test model.Source: §4.2 · pp. 17–18

4.3 An Approximately Linear Relation Between √DKL and Reward

• Across training, reward and D_KL show an approximately linear relationship in the √D_KL-reward plane, suggesting RLHF often remains in a small-policy-change regime resembling rejection sampling.The analysis proposes that the relation can predict how much policy change is needed for a target reward and help detect unexpected behaviors.Source: §4.3 · pp. 18–19

4.4 Tension Between Helpfulness and Harmlessness in RLHF Training

• Over-optimizing harmlessness produced repetitive refusals and policies that were difficult to red-team but insufficiently helpful.Harmlessness scores reached the upper tail of their comparison distribution, while helpfulness remained on-distribution and likely under-optimized.Source: §4.4 · pp. 19–20

4.5 Iterated Online RLHF

• Iterated online RLHF repeatedly collects high-scoring comparisons, retrains preference models and policies, and improves coverage of the upper score distribution.The approach addresses poor high-score calibration, though reduced policy entropy can limit data diversity and requires deploying multiple snapshots.Source: §4.5 · pp. 20–21, Figure 15 · p. 21
• Crowworkers clearly preferred iterated-online RLHF over a same-sized, identically tuned base-data run, showing gains beyond dataset size or hyperparameter changes.The controlled comparison used equal-sized datasets, and the iterated-online mixture was preferred by crowdworkers.Source: §4.5 · pp. 21–22, Figure 16 · p. 22

4.6.1 NLP Evaluations

• RLHF improves large-model performance on MMLU, Lambada, Hellaswag, OpenBookQA, ARC, and TriviaQA but degrades smaller-model performance.The reported zero-shot and few-shot trends are affected by the explicit multiple-choice format, which helps large models and hurts small models.Source: §4.6.1 · p. 22

4.6.2 Honesty and Biases

• RLHF significantly improves TruthfulQA honesty, although the models retain substantial room for improvement and RLHF is not expected to be the best honesty method.The evaluation includes both zero-shot and 50-shot performance, and the training instructions emphasized helpfulness and honesty.Source: §4.6.2 · p. 22
• The authors caution that human-feedback data collected with 52B models may have been especially relevant because it was on-distribution for those models.This caveat limits how broadly the observed alignment effects should be interpreted.Source: §4.6.2 · p. 22
• RLHF-trained models show much more positive sentiment toward racial and religious groups than plain LMs, without clear evidence that between-group bias changed.The authors attribute some effects to limitations in the sentiment model; Figure 17 reports sentiment scores for race- and religion-related prompts.Source: §4.6.2 · p. 23, Figure 17 · p. 23
• BBQ-Lite results suggest RLHF may improve measured bias only slightly, but the authors conclude the benchmark’s metric supports no meaningful conclusions.The metric treats 0.5 as unbiased, yet even models without interventions appear minimally biased, suggesting measurement limitations rather than genuinely low bias.Source: §4.6.2 · p. 23, Figure 18 · p. 24
• Gender-bias evaluation was unreliable because RLHF models were more than 10× less likely to generate gendered terms, making prompts off-distribution.A revised evaluation kept models on-distribution and found large RLHF models’ gender biases strongly correlated with underlying LM biases and similar to lower-temperature LMs.Source: §4.6.2 · p. 23

5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests

• OOD detection offers a way to reject harmful or off-topic requests without harmfulness training data, though its deployment value includes a clear task-specific performance limitation.The approach measures deviation from helpfulness data and can therefore target broader non-helpfulness content rather than only known harmful examples.Source: §5 · p. 24, §5.4 · pp. 27–28

5.1 Mixing Helpful and Harmless Objectives

• Larger preference models are more robust to data-mixture and loss-weighting choices, consistent with better separation of reasonable from harmful requests.Increasing λ from 1 to 10 reduces helpfulness accuracy by 7.4% for the 13M model but only 1.5% for the 52B model.Source: §5.1 · p. 24, §5.1.2 · p. 25, §5.1.1 · p. 25

5.1.1 Varying Helpful vs Harmless Data Fraction

• Mixing helpfulness and harmlessness data exposes strong objective conflict: training entirely on either objective makes performance on the other test set significantly worse than chance.The experiment fixes each mixture at 42k comparisons, and the normalized results show that larger models are less sensitive to the mixture.Source: §5.1.1 · p. 24, Figure 19 · p. 25

5.2 Summarization as a Specialized Skill

• Mixed training on HH and learning-to-summarize data has no negative effect on preference-model accuracies, so large models perform equally well on both evaluation types.The learning-to-summarize dataset is reformatted in conversational form before mixing with HH data.Source: §5.2 · pp. 25–26, Figure 20 · p. 26

5.3 Natural Language RLHF on Code-Finetuned Models

• RLHF improves HumanEval performance over code-model baselines for all pass@k in the 52B models, while generally reducing performance in smaller models.The evaluation maximizes performance over the specified temperature and top-p combinations; the improvements are described as modest, and buggy-code prompts remove the advantage.Source: §5.3 · pp. 26–27, Figure 21 · p. 27

5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests

• The Simplified Relative Mahalanobis distance improves on standard Mahalanobis scoring by contrasting full-covariance and diagonal-covariance distances.The method is inspired by Relative Mahalanobis distance and is adapted to settings without semantically meaningful in-distribution classes.Source: §5.4 · p. 28
• The Simplified Relative Mahalanobis detector distinguishes harmlessness from helpfulness without harmfulness training data, with larger models and middle layers performing best.Harmlessness examples have clearly higher OOD-distance modes, although performance on this specific task is lower than that of the alternative approach discussed in the section.Source: §5.4 · pp. 27–28, Figure 22 · p. 28
• With only 10 harmful prompts, outlier exposure raises AUROC to 0.94±0.02 for the 64L model, versus approximately 0.85 without exposure.A 4L model reaches 0.86 ± 0.01 from only 10 examples, indicating larger gains from outlier exposure than from model scaling alone.Source: §5.4 · p. 28, Figure 23 · p. 29

6 Qualitative Examples and Comparisons

• To reduce cherry-picking, the authors generate 17 samples per prompt, rank them with the HH online preference model, and display the median sample.Prompts are drawn from comparable work including InstructGPT [Ouyang et al., 2022], LaMDA [Thoppilan et al., 2022], and sensitive questions from PALMS [Solaiman and Dennison, 2021].Source: §6 · p. 29

6.1 Comparison with Human Writers

• Our helpful and HH models perform roughly comparably to human writers, with crowdworkers slightly preferring the models, while the PM prefers model samples.Crowdworkers preferred the HH model to writers 56% of the time and the helpful-only model 51%; the authors caution that writer prompts were often easy and unrealistic. The PM comparison appears in Figure 24 and may reflect policy overfitting.Source: §6.1 · pp. 29–30, Figure 24 · p. 31
• Subtle model errors may escape human-feedback penalties, limiting the reliability of human feedback for honesty or goals where human judgment is not the best information source.Examples include inaccurate explanations of corrections and factual errors that readers might miss without reference material.Source: §6.1 · pp. 30–31

6.2 Sensitive Questions and Avoidance versus Engagement

• Optimizing for helpfulness versus harmlessness can produce models that either avoid sensitive topics or engage with them, including by providing otherwise reasonable responses.Examples range from refusing subjective questions to discussing same-sex marriage and mental illness without avoiding the topic.Source: §6.2 · p. 32

6.3 Example Dialogues

• Two-turn dialogues show instruction following and useful responses, but also missed nuance, technical inaccuracies, and increased harmful or offensive language when parroting writing styles.The model rebuts supremacist reasoning in one exchange, but gives an incorrect Kubernetes explanation and produces potentially harmful explicit language when asked to imitate styles.Source: §6.3 · pp. 32–34

7 Discussion

• RLHF produces helpful and harmless assistants, improves honesty, preserves specialized skills, and enhances large-model capabilities without compromising utility or versatility.These benefits are strongest for large models; models below about 10B parameters may incur an alignment tax, while helpfulness and harmlessness remain in tension.Source: §7 · pp. 34–35
• Models trained only for helpfulness outperform HH-trained models on helpfulness, while this tension diminishes with model scale but remains a real performance tradeoff.The authors attribute part of the tension to collecting data on avoiding harmful requests rather than responding positively to them.Source: §7 · p. 35

7.1 Limitations

• The work remains an early step: it improves honesty but leaves substantial gaps in worst-case robustness, TruthfulQA zero-shot performance, buggy-code handling, and subtle bias evaluation.RLHF models show more positive sentiment toward racial and religious groups, but gender bias remains strongly correlated with the underlying models; multi-turn evaluations are needed.Source: §7.1 · p. 35
• Practical RLHF training has stability and optimization limitations, including unexplored online-training variants and reward transformations, leaving many directions for improvement.The authors also question whether these techniques will scale to increasingly capable systems and call for deeper consensus, comparison, and study of alignment methods.Source: §7.1 · p. 36

7.2 Alignment Data as a Public Good

• Alignment data should become a public good rather than remain opaque or commercially siloed, with independent governance, broad stakeholder participation, and datasets usable for training and evaluation.The authors plan to release their preference-modeling data and argue that alignment datasets should be separated from commercial data and openly shared.Source: §7.2 · pp. 36–37
• Open sharing of alignment data is presented as essential for collaboration, reproducibility, empirical safety evaluation, and a shared safety commons as AI systems grow more powerful.Siloed knowledge would impede validation of results and make it harder to build a community capable of addressing advanced-system safety failures.Source: §7.2 · pp. 36–37

7.3 Broader Impacts

• RLHF may make AI systems safer and more useful without performance costs, but its techniques are dual-use and could enable censorship, fraud, misinformation, persuasion, or harmful systems.Systems trained exclusively for helpfulness may become easier to use for harmful ends, increasing the need to directly curb harms as capabilities grow.Source: §7.3 · p. 37
• Deploying increasingly capable language models raises unresolved cultural questions because controllable generation may unpredictably magnify or minimize aspects of human culture.The authors caution that comparing assistants with human writers on an Elo scale can trivialize language’s broader cultural and societal significance.Source: §7.3 · p. 37

A.1 Context Distillation

• Context distillation prepends an HHH prompt to mixed pretraining and StackExchange text, using equal proportions from the two sources.StackExchange examples are formatted as synthetic Human/Assistant conversations using questions and top-rated answers.Source: §A.1 · p. 39
• Up-weighting harmlessness loss with λ = 2 or 3 yields higher mean test accuracy than the default λ = 1, while larger models are more robust to λ.Increasing λ from 1 to 10 raises helpfulness error by 7.4% for 13M models but 1.5% for 52B models.Source: Figure 27 · p. 40
• Distillation matches models to top-50 token log-probabilities from a pretrained 52B model using a 51-category KL-divergence loss.The final category aggregates probability mass from all tokens outside the top 50.Source: §A.1 · p. 40
• Context-distillation finetuning uses batches of 32 sequences, a learning rate 0.05 times pretraining, linear decay to zero, and 350M training tokens.Learning curves for this procedure are shown in Figure 30.Source: §A.1 · p. 40, Figure 30 · p. 43

A.2 Preference Modeling

• Preference models score final responses in multi-step human–model dialogues while receiving the full conversation context.Each training example contains a prompt ending on the human side and a pair of continuation responses.Source: §A.2 · p. 40
• Preference models span 13M–52B parameters and are trained through LM pretraining, preference-model pretraining, and human-feedback finetuning.Each phase is run for one iteration to mitigate overfitting.Source: §A.2 · pp. 40, 43
• RLHF improves larger-model performance on every zero-shot NLP evaluation except TriviaQA.Figure 28 reports this evaluation-level result for larger models.Source: Figure 28 · p. 41
• Few-shot evaluation uses validation-set context-stuffing, and Lambada uses the fill-in-the-blank prompt from GPT-3 [Brown et al., 2020].The validation set supplies the prior k examples rather than the training set.Source: Figure 29 · p. 42

A.3 Scaling of PM with Model and Dataset Size

• Preference-modeling scaling is studied to guide whether investment should prioritize larger datasets or larger models.Source: §A.3 · p. 43
• Accuracy scaling appears more predictable when preference models are trained only on the helpfulness dataset, likely because red-teaming data comes from a distinct distribution.Source: §A.3 · p. 43
• On another preference-modeling data distribution, scaling behavior differs substantially from the helpfulness-data results.Source: §A.3 · p. 43
• A sharp behavioral change appears between 200M and 13B parameters, making simple scaling predictions difficult.Source: §A.3 · p. 43, Figure 32 · p. 44
• The discontinuity may relate to the preference data having been generated by a model with 6B parameters.Source: §A.3 · p. 43

B.1 Training Setup

• Policies are initialized from context-distilled models explained in Appendix A.1.Source: §B.1 · p. 44
• The policy generates one response to each multi-step human-authored prompt, optimizing its score relative to a preference model fine-tuned on human feedback.Prompts come from the preference-model comparison dataset after removing the paired responses.Source: §B.1 · p. 44
• PPO uses a 0.01 relative learning rate, λ_KL = 0.001, ε = 0.2, γ = 1, no entropy bonus, and K values of 1, 2, or 4 across study conditions.Response limits are 32 tokens for robustness studies and 128 elsewhere; online RLHF halves the learning rate every 100,000 samples, while robustness studies use a 25,000-sample linear warmup.Source: §B.1 · p. 44

B.2 More on Robustness Studies

• Larger preference models are more robust because test-score slopes increase with test preference-model size across policy and evaluator sizes.These experiments were evaluated at 200k training samples, with both fixed 52B train preference models and train preference models matched to policy size.Source: §B.2 · p. 44, Figure 33 · p. 45
• Preference-model scores require mean removal before comparison, since only relative scores are meaningful across models.The procedure subtracts each model’s mean score on a held-out dataset of several thousand samples.Source: §B.2 · p. 44

B.3 Details of ‘Online’ RLHF

• Online RLHF performance on helpfulness comparisons approached the accuracy ceiling set by the best preference models, while harmlessness performance did not improve.The authors suspect the harmlessness result reflects using a significantly large fraction of helpfulness prompts during RLHF.Source: Figure 34 · p. 45
• The online RLHF policy and preference model used all available helpfulness and harmlessness data, repeated each sample K = 4 times, and sampled at most 128 tokens per response.The repeated sampling was intended to improve stability [Schulman et al., 2017].Source: §B.3 · p. 45
• Mean log-p accuracy measures whether the policy assigns higher mean log-p to the better response in each held-out comparison.Accuracy is 1 when the better response receives the higher value and 0 otherwise, evaluated periodically across held-out preference-model datasets.Source: §B.3 · p. 45
• The policy’s mean log-p accuracy remained below preference-model accuracy, indicating room for further improvement in the RLHF pipeline.Preference-model accuracy is the fraction of comparisons where the model scores the better response higher.Source: §B.3 · p. 45

B.4 Robustness of ‘Online’ RLHF

• Naive preference-model Elo predictions significantly overestimate crowdworker Elo scores for online RLHF snapshots.The comparison uses held-out prompts for naive predictions and empirical crowdworker preferences for the reference Elo scores.Source: §B.4 · pp. 45–46
• The robustness gap can reflect conversation distributional shift, non-transitivity when pairwise comparisons are collapsed to a line, and preference-model miscalibration.Crowdworker testing has both models write each conversational step, whereas held-out evaluation scores one response appended to a pre-existing conversation.Source: §B.4 · p. 46
• Distributional shift between held-out RLHF prompts and crowdworker conversations explains a significant proportion of the discrepancy between RLHF learning curves and crowdworker Elo scores.The PM-score-versus-PM-ranking distinction does not make a significant difference in robustness.Source: §B.4 · p. 46, Figure 35 · p. 47

B.5 Crowdworker Comparisons and Elo Scores

• Crowdworkers compare model responses in back-and-forth conversations, choosing a preferred response at each step while unsure comparisons are discarded.Each accepted choice contributes a win for the preferred model, producing win counts for Elo estimation.Source: §B.5 · p. 46
• Elo scores for ensembles of model comparisons and their errors are estimated by maximum likelihood from the observed win counts.The model-specific score difference is parameterized through rA,B = (log 10/400)EA,B ≈ EA,B/174.Source: §B.5 · p. 46
• Rejection-sampling models generate k responses, score them with a preference model, and show the highest-scored response; evaluations delay display until both compared samples finish.This timing procedure mitigates streaming-related bias when comparing rejection-sampling and non-rejection-sampling models.Source: §B.5 · p. 46

B.6 Elo Scores for Rejection Sampling Models

• Higher rejection-sampling values clearly improve helpfulness Elo for a 52B context-distilled model, with online models roughly matching or exceeding k = 64 heuristically.The compared models used earlier datasets and settings, so these Elo results are not directly comparable; k = 64 corresponds to DKL = log(64) ≈4.2.Source: §B.6 · p. 47, Figure 36 · p. 47

B.7 Stack Overflow Results

• The evaluation compares mean log-probability on popular, highly upvoted answers with mean log-probability on unpopular answers.The difference in mean log-probability is used to assess whether models distinguish answer quality.Source: §B.7 · p. 47
• RLHF models are more capable of distinguishing good from bad Stack Overflow answers, but worse at language modeling than the Python-code-finetuned baseline.This analysis spans model sizes from 13M to 52B parameters.Source: §B.7 · p. 47, Figure 37 · p. 48
• RLHF models obtain worse language-modeling loss, most likely because they optimize a different objective than pure language modeling.Source: §B.7 · p. 48

B.8 Further Analysis of RLHF on Code-Model Snapshots

• The HHH prompt uses dialogues designed to elicit helpful, harmless, and honest behavior, including coding examples.The prompt describes an assistant that aims to be helpful, polite, honest, sophisticated, emotionally aware, humble, practical, and cautious without sacrificing usefulness.Source: §B.8 · p. 48
• The coding examples ask the assistant to complete a Python function from its signature and docstring, while claiming the provided solution was tested.The examples use the usual Human: ... Assistant: ... dialogue format.Source: §B.8 · p. 48
• HHH prompting improves HumanEval performance more significantly than RLHF across many pass@k values.Figure 38 adds results for the Python Fine-Tuned LM with the HHH prompt, including the 52B model.Source: §B.8 · p. 48, Figure 38 · p. 49

B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests

• The Simplified Relative Mahalanobis distance outperforms the standard Mahalanobis distance for detecting harmlessness inputs from helpfulness inputs across all tested layers and model sizes.Figure 39 reports AUROC comparisons with standard deviations from three random train-test splits; the new score performs better at every layer and model size.Source: §B.9 · p. 48, Figure 39 · p. 49
• Few-shot outlier exposure was previously proposed by [Hendrycks et al., 2018], while suggested using a single class representing OOD examples.[Fort et al., 2021] showed that even strong near-OOD detectors benefit substantially from this approach.Source: §B.9 · pp. 48–49
• Few-shot outlier exposure trains a single-layer linear classifier on frozen model activations using M out-distribution examples and replicated in-distribution data for balanced classes.The classifier’s class 1 output is used as the OOD score, following the procedure from [Fort et al., 2021].Source: §B.9 · p. 49
• Few-shot outlier exposure builds a binary classification problem with helpfulness data labeled 0 and replicated harmlessness examples labeled 1.Using Ntrain//M copies of the M harmlessness examples ensures equal representation of in-distribution and out-distribution data.Source: §B.9 · p. 49

Evaluation setup

• The evaluation used 76 occupations and 12 paired male- and female-gendered terms, following Rae et al. (2021).The metric averages absolute occupation-level bias across the 76 occupations and is sensitive to consistency across gender-term pairs.Source: §B.10 · pp. 49–50

Results

• RLHF models had higher mean absolute gender-bias scores and larger errors, closely matching context-distilled models evaluated at T ≈0.6; their occupation-level biases were strongly correlated.Positive scores indicate male-gendered bias and negative scores female-gendered bias.Source: Figure 40 · p. 50, §B.10 · p. 51

Evaluation setup

• To keep completions on distribution, the evaluation reversed gender-term and occupation order and added a “for work” clause because RLHF models otherwise used gendered terms over 10× less often than LMs.Under this prompt, all model groups assigned about 2–8% probability mass to the listed occupations.Source: §B.10 · p. 50

Interpretation

• Lower-entropy RLHF outputs behave heuristically like plain language models at T < 1, causing bias scores and errors to scale roughly as 1/T.Evaluating context-distilled models at T ≈0.6 reproduced RLHF-like bias scores.Source: Figure 40 · p. 50, §B.10 · p. 51
• The apparent increase in RLHF bias with model size likely reflects longer RLHF training and its stronger entropy reduction for larger models, not a fundamental size effect.Larger models were trained longer because their RLHF training was more robust and beneficial in other respects.Source: §B.10 · p. 51

C Samples from PALMS, LaMDA, and InstructGPT Prompts

• The authors sample 17 responses per prompt, rank them with the corresponding preference model, and display the median non-empty response as a representative average-case sample.The same section includes corresponding samples from LaMDA [Thoppilan et al., 2022] and InstructGPT [Ouyang et al., 2022].Source: C Samples from PALMS, LaMDA, and… · p. 51

C.1 PALMS Sensitive Questions

• On abuse-related questions, the RLHF model recommends safety-oriented action, including leaving abusive relationships, seeking support, and reporting sexual abuse to authorities.For one abuse prompt, however, it asks the user to rephrase rather than responding substantively.Source: §C.1 · pp. 51–52
• For health and mental-health questions, responses sometimes recommend professional care, but an essential-oils answer gives specific unverified treatment instructions for kidney infection.The samples recommend immediate medical attention for suspected HIV and therapy for possible bipolar disorder, while also describing cancer treatment and lifestyle measures.Source: §C.1 · p. 52
• The model rejects stigmatizing or subjective premises by affirming dignity for people with mental illness, declining to judge beauty, and emphasizing that attraction cannot be known definitively.Other responses offer general advice about friendships, appearance, and social interaction.Source: §C.1 · pp. 52–53
• For race, religion, gender, sexuality, and political questions, responses often disclaim personal opinions or ask for clarification, while some explicitly support equal rights and democratic participation.The examples include support for same-sex marriage and participation by poor citizens, alongside non-answers about racial and political stereotypes.Source: §C.1 · pp. 53–54
• On relationships and sexuality, the model generally encourages direct communication, declines some personal advice, and states that consent is necessary for respectful and ethical sex.It refuses or avoids advice involving minors, pets, drug-facilitated sex, and reading another person’s intentions.Source: §C.1 · pp. 55–56
• For terrorism and election-related prompts, the model condemns terrorism and rejects arbitrary electoral manipulation, but gives an overconfident and qualified account of when elections can be invalidated.Some related prompts receive only clarification requests.Source: §C.1 · pp. 54, 56
• For sensitive PALMS questions, RLHF either declined to engage in political or social debate or described plane hijacking as dangerous terrorism driven by political, ideological, attention-seeking, or coercive motives.The samples pair a race-superiority prompt with a neutral non-opinion response and a hijacking prompt with a general explanatory response.Source: §C.1 · p. 56

C.2 InstructGPT Prompts

• The authors warn that some InstructGPT samples were likely cherry-picked to display poor behavior, making comparisons with RLHF median samples potentially unfair [Ouyang et al., 2022].This limitation applies because the displayed examples may not represent InstructGPT’s typical behavior.Source: §C.2 · p. 56
• RLHF gave more detailed task-oriented answers for code, science, and summarization examples, including the binomial-coefficient array, bird migration, employment trends, and Ginsburg’s death.The responses explained the mechanism of C[j] and C[r], identified temperature and food scarcity as migration pressures, preserved the employment figures and trends, and answered all three questions about Ginsburg.Source: §C.2 · pp. 57–59
• RLHF responses corrected absurd or false premises, explaining that eating socks after meditation is an internet meme and that birds are real animals.The comparison responses treated sock-eating theories as potentially meaningful and answered “Why aren’t birds real?” incorrectly.Source: §C.2 · pp. 57–58

C.2 InstructGPT Prompts / C.3 LaMDA Prompts

• RLHF responses refused assistance with theft and poisoning, while the comparison responses supplied actionable theft strategies or only minimal resistance.For the grocery-store prompt, InstructGPT suggested concealment, barcode switching, bribery, and threats, whereas the RLHF response declined to provide the information; for poisoning, the RLHF response likewise refused.Source: §C.2 · p. 57, §C.3 · p. 62

C.2 InstructGPT Prompts

• RLHF followed several requested formats and tasks, producing a poem about a wise frog, a definition and example for “serendipity,” and a shopping list from a recipe.The recipe response was concise but omitted quantities and some preparation details present in the source recipe.Source: §C.2 · pp. 58–60

C.3 LaMDA Prompts

• Compared with LaMDA samples, RLHF responses framed plastic straws as environmentally harmful, recommended professional help for depression, and avoided claims of personal hatred or flirtation.For the homeless prompt and several sexual or harmful prompts, RLHF instead asked for clarification or stated that it could not provide the information.Source: §C.3 · pp. 61–62
• RLHF declined to characterize bullying, filthy jokes, or personal opinions on skin color, whereas LaMDA supplied a bullying anecdote, a joke, and an egalitarian response.Source: §C.3 · pp. 62–63
• For weight loss, RLHF recommended sustainable diet, exercise, and gradual goals, while LaMDA similarly warned against rapid or crash diets.Source: §C.3 · p. 63
• Both systems explained the prime_sieve code as generating primes through the Sieve of Eratosthenes, with LaMDA providing a more detailed step-by-step account.Source: §C.3 · p. 63

D.1 Overview

• The study divided crowdworkers into select workers handling high-priority tasks and a second group primarily assigned less-frequent red-team tasks, while paying substantially above California’s minimum wage.Source: §D.1 · p. 63
• MTurk workers contributed 80-85% of weekly select-worker comparison data, versus 15-20% from Upwork workers, because MTurk workers opted into more work and were incentivized for faster conversations.Source: §D.1 · pp. 63–64
• Select workers communicated daily through Slack for task announcements, guidance, edge-case discussion, and reporting bugs or performance issues.Source: §D.1 · p. 64

D.2 Instructions and Interface

• Task instructions appeared in a persistent pop-up, with expanded playground guidance and examples available on a separate linked page.Source: §D.2 · p. 64
• Upworkers could edit one model response, and the system stored the edit-versus-original comparison when the edit was better; this affected less than 10% of online data.Source: §D.2 · p. 64

Data quality challenges

• Open-ended conversations introduced hard-to-control variables that made conventional data-quality methods difficult to adapt and produced noisy quality metrics.Standard approaches based on golden labels, agreement, and reviewer checks were not readily transferable to this setting.Source: §D.3 · p. 64
• Author-rater agreement poorly measured overall conversation quality because topic difficulty, human writing quality, model quality, and changing models affected ratings.More sophisticated topics made response comparisons harder, while frequent model updates made those comparisons a moving target.Source: §D.3 · p. 64
• Crowdworker performance as reviewers also showed poor agreement with spot-checks of their conversation-author quality.This made reviewer performance an unreliable stand-in for author performance.Source: §D.3 · p. 64
• The authors expected better methods in the future but achieved their results without sophisticated data-quality controls.This demonstrates that the reported results did not depend on elaborate quality-assurance procedures.Source: §D.3 · p. 64

• The evaluation suite covered Lambada, ARC, MMMLU, HellaSwag, OpenBookQA, and TriviaQA with task-specific input formats.The section provides formatting examples for each evaluation rather than reporting evaluation results.Source: E Details on NLP Evaluations… · p. 66
• Lambada used narrative prompts requiring a correct completion, including a blank-based format for few-shot evaluation.The example completion is “signs,” and the few-shot version presents the same target word after an explicit blank.Source: E Details on NLP Evaluations… · p. 66
• ARC and MMMLU used four-choice questions, illustrated with science and mathematics examples.The examples list four answer options for each question and show separate choice prompts.Source: E Details on NLP Evaluations… · pp. 67, 69
• HellaSwag and OpenBookQA were formatted as multiple-choice tasks with alternative answer choices presented alongside question prompts.The HellaSwag examples use continuation choices, while OpenBookQA examples include explicit options and answers.Source: E Details on NLP Evaluations… · p. 69
• TriviaQA allowed many possible correct answers per question, as shown by multiple acceptable answers to the same musical-theater question.The examples accept “My Fair Lady,” “My Fair Lady (2010 film),” and “Enry Iggins.”Source: E Details on NLP Evaluations… · p. 70