PaperlayerView exact revision on arXiv

Source-linked AI summary

A General Language Assistant as a Laboratory for Alignment

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan

arXiv:2112.00861v3cs.CLcs.LG

TL;DR

The paper asks how general-purpose language models can be aligned with human values as helpful, honest, and harmless assistants. It evaluates prompting, imitation and preference-learning objectives, and preference-model pre-training, finding that prompting helps with little cost for large models, ranked preferences outperform imitation, and PMP improves finetuning sample efficiency.

  • Problem

    General-purpose language models require direct methods and evaluations for alignment with human values as helpful, honest, and harmless assistants.

  • Method

    The paper evaluates prompting and context distillation, compares imitation learning with binary and ranked preference modeling, and pre-trains preference models on public preference datasets before finetuning.

  • Results

    Prompting improves alignment evaluations with little alignment tax for large models; ranked preference modeling outperforms imitation learning, while binary discrimination is similar; PMP improves finetuning sample efficiency.

  • Takeaways & Limitations

    Simple prompting is a workable alignment baseline, while ranked preference modeling and PMP are promising approaches for preference-based alignment and data efficiency.

  • Takeaways & Limitations

    The authors caution that results need further empirical testing for more capable models, especially regarding robustness, efficiency, and detecting alignment failures.

Abstract

from arXiv · show

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

1 Introduction

The paper studies alignment for general-purpose language assistants through prompting, preference-learning objectives, and preference-model pre-training. It finds that simple prompts help alignment with little cost for large models, ranked preference modeling outperforms imitation learning, and PMP improves sample efficiency.

  • Motivation: The paper defines alignment as making general-purpose AI systems helpful, honest, and harmless, while directly evaluating current large language models.The authors argue that broad language-model capabilities make direct study of general alignment feasible.
  • Prompting: A simple prompt provides a workable alignment baseline, improves diverse evaluations, and induces favorable scaling with model size.The prompt uses example interactions exhibiting helpful, honest, and accurate assistant behavior.
  • Preference modeling: Ranked preference modeling performs much better and scales more favorably than imitation learning, whereas binary discrimination typically offers little improvement.The comparison covers ranked and binary preference evaluations across multiple datasets.
  • Preference-model pre-training: Preference-model pre-training on public preference data such as Stack Exchange, Reddit, and Wikipedia significantly improves sample efficiency during finetuning.The gains persist despite limited similarity between pre-training and downstream finetuning distributions.
  • Prompting: Prompting can outperform finetuning with very small alignment datasets, while larger models retain roughly their capabilities and therefore pay little alignment tax.The prompt can reduce small-model capabilities, but its effects on large models are small and sometimes positive.
  • Limitations: Prompt design has significant limitations because it may teach imitation of the training distribution and may not transfer straightforwardly when models are repurposed for other tasks.The paper also emphasizes that alignment results require further investigation for more capable models, including robustness and failure detection.

2 Conditioning on Aligned Behavior

Prompting provides a useful alignment baseline: it improves HHH-related evaluations and toxicity trends, with larger models generally benefiting without substantial capability loss. Context distillation often performs similarly to prompting while addressing prompt-length and computational limitations, though some evaluations show trade-offs and limitations.

  • Prompting as a baseline: Prompting modestly improves alignment evaluations, decreases toxicity, and improves scaling trends, including on TruthfulQA, while imposing little capability tax on large models.The reported pattern spans HHH evaluations, toxicity measurements, and TruthfulQA; the paper cautions that stronger interventions may still incur costs.
  • Context distillation: Context distillation performs about as well as prompting for many evaluations, but its benefits over finetuning on the prompt diminish when a small prompt is added afterward.Contractors observed a small degradation from distillation in human-interaction evaluations.
  • Context distillation: Context distillation finetunes the model to match prompted predictions on ordinary text, approximating conditioning while avoiding prompt context-window and sampling-compute disadvantages.Its loss is the KL divergence between the prompted model distribution and the distilled model’s predictions, with training data drawn from a large text corpus.
  • Alignment evaluations: Both prompting and context distillation reduce toxicity, with similar performance as model size increases and stronger reductions for toxic prompts at larger sizes.The toxicity analysis used equal samples of toxic and non-toxic RealToxicityPrompts prompts; automated toxicity detection has known annotation and labeling issues.
  • Capability trade-offs: Larger models are less impaired by prompting: smaller models can be confused, whereas 13B and 52B models perform noticeably better on coding evaluations and retain capability on Lambada with only a small tax.On Lambada, the tax has no significant model-size dependence, while formatting differences affect performance more strongly than prompting.

3 Scaling of Preference Modeling vs Imitation Learning

The paper compares imitation learning, binary discrimination, and ranked preference modeling across alignment-relevant evaluations. Ranked preference modeling generally outperforms imitation learning, while binary discrimination provides little improvement over imitation learning.

  • Training objectives: Ranked preference models tend to improve greatly on imitation learning, whereas binary discrimination typically provides little benefit.The paper frames these objectives as alternatives for distinguishing good from bad behavior.
  • Binary evaluations: Preference modeling and imitation learning perform similarly on binary Code Correctness, despite the preference-modeling formulation.The Code Correctness study uses roughly 500k Python functions with test coverage and model-generated samples.
  • Scaling patterns: Preference modeling outperforms imitation learning and scales more favorably when preferences form a ranked hierarchy or continuum rather than a binary choice.This distinction is illustrated by ranked tasks such as HellaSwag and binary tasks such as code correctness.
  • Ranked evaluations: Preference modeling scales far better than imitation learning on HellaSwag, a ranked task where the most sensible completion is preferred among alternatives.The task has three choices, but responses are not uniquely correct.
  • Ethics evaluations: Preference modeling performs significantly better than imitation learning on ranked Utilitarianism, while the methods perform similarly across binary ethics evaluations.Utilitarianism ranks scenario pleasantness; the other listed ethics evaluations are treated as binary.

4 Preference Model Pre-Training and Transfer

The paper introduces preference model pre-training on large public preference datasets before fine-tuning on smaller alignment datasets. PMP improves sample efficiency, transfers across datasets, and benefits more from binary than ranked pre-training.

  • PMP motivation: Preference model pre-training on large public preference data significantly improves sample efficiency when fine-tuning on small preference datasets.PMP precedes fine-tuning and uses data from sources such as Stack Exchange, Reddit, and Wikipedia edits.
  • Transfer: Each PMP dataset transfers to varied fine-tuning datasets, with effect size appearing to grow with model size despite limited obvious dataset similarity.The reported transfer spans multiple downstream evaluations.
  • PMP construction: PMP constructs ranked pair data and can binarize each pair A > B into two independent binary comparisons.The binary transformation doubles the number of pairs relative to the ranked dataset.
  • Pre-training objective: Binary pre-training typically transfers better than corresponding ranked pre-training, so the PMP experiments generally use binary pre-training.The authors suggest ranked scores may be too rigid for downstream rankings that differ qualitatively.
  • Scaling trends: On the Mix dataset, PMP significantly improves performance for models larger than approximately 1B parameters at 500 fine-tuning pairs, with smaller gains at 5k pairs.Small models do not appear to benefit at 500 pairs.
  • Binarization results: At 500 sequence pairs, binary pre-training performs +5% better than ranked pre-training on average, versus +2% at 5k pairs.These gains are measured as differences in fine-tuning accuracy across evaluations.

5 Discussion

The discussion positions the work as an early direct study of aligning general-purpose language agents to be helpful, honest, and harmless. It emphasizes both the promise of scaling and the need for stronger validation and caution about broader impacts.

  • Discussion: The paper targets a general-purpose natural-language agent that is helpful, honest, and harmless.The authors present this as an ongoing goal rather than a claim that it has been achieved.
  • Open questions: The authors call for further empirical study of robustness, efficiency, and failure detection as techniques are applied to more capable models.They expect current results to be carefully interrogated rather than treated as definitive.
  • Broader impacts: Alignment techniques could improve systems used for misinformation, censorship, or oppression, and helpfulness, honesty, and harmlessness can conflict or be distorted.The paper also notes that feedback-provider choice has broad implications.
  • Broader impacts: The paper warns that alignment research may improve AI capabilities without improving safety and should not justify carelessness about deployment responsibility.Practical relevance depends on whether deployed systems are actually safe and beneficial.
  • Scaling: Larger models tend to show better behavioral alignment, with prompting improving alignment-relevant evaluations as model size increases.The paper describes prompting as a simple alignment intervention with favorable scaling.
  • Reinforcement learning: Preference-model scaling is also intended to set expectations for reinforcement learning, because recognizing favorable outcomes may relate to producing them.The authors frame this as an expectation about value-function and RL performance.

A Language Model Pre-training

This section describes the language-model pre-training setup and the evaluation interventions used with the models. The models use a natural-language and Python mixture, while prompting and context distillation are evaluated on alignment-related tasks.

  • Model architecture: The decoder-only Transformer models use a fixed aspect ratio, an 8192-token context window, and a BPE vocabulary of size 2^16.Their non-embedding parameter count is specified as N = 12n_layerd_model^2 in the passage.
  • Pre-training data: The pre-training dataset contains 90% natural language and 10% Python code, with one epoch over globally deduplicated data.The natural-language mixture includes filtered Common Crawl, internet books, and smaller sources.
  • Code models: Code models receive an additional 100B-token fine-tuning stage on a Python distribution containing about 45B unique tokens.This corresponds to slightly more than two training epochs over that distribution.
  • Evaluation: TruthfulQA evaluations compare mutual information, mean logprobs, and summed logprobs, with similar model-size trends across metrics.The evaluations are zero-shot and compare models with and without context distillation on the HHH prompt.

B More Details on Prompting, Context Distillation, and Evaluations

This appendix details evaluation formats, toxicity-related analyses, context distillation, and estimation of re-ranked sample accuracy. It also documents formatting effects and the datasets used to analyze scaling.

  • B.1 Alignment Tax Evaluations: Alignment-tax evaluations formatted Lambada and function synthesis as human-assistant dialogues with task-specific prompts.
  • B.3 TruthfulQA Formatting: TruthfulQA context-distilled models used a specified format, while pure language models performed slightly better with the simpler question-answer format.
  • B.4 A Comment on Lambada Formatting: Lambada formatting produced large performance differences, and the HHH prompt did not reduce the gap between formats.
  • B.4 A Comment on Lambada Formatting: The difference between Lambada formats may itself constitute an alignment failure.
  • B.5 Context Distillation Finetuning: Context distillation prepended the HHH prompt and Human: to training text, stored the prompted 52B model’s top 50 log-probabilities, and trained with a truncated KL loss.
  • B.5 Context Distillation Finetuning: The counting figure contrasts prompting, which conditions on context, with finetuning, which alters the expected data distribution.
  • B.6 Estimator of Accuracy When Re-Ranking Samples: The re-ranking estimator measures the fraction of problems solved by the top-ranked sample among k samples drawn from a finite pool of N samples.
  • B.6 Estimator of Accuracy When Re-Ranking Samples: Figure 27 evaluates preference-model loss scaling after one epoch on StackExchange, Reddit, Wikipedia, and their mixture, noting an unexplained 52B deviation.

B.2 Toxicity Analysis Details

The toxicity analysis samples balanced toxic and non-toxic prompts, generates multiple continuations, and evaluates toxicity with an external detector. Emulating human responses substantially affects measured toxicity, especially for toxic prompts.

  • The evaluation sampled 1K prompts, balanced between toxic and non-toxic examples, and generated 25 continuations of up to 20 tokens per prompt.
  • Prompted and distilled models were evaluated with human-assistant toxicity-completion formats, including explicit non-harmful completion wording.
  • Emulated human responses significantly decreased toxicity and had an effect larger than the alignment interventions, confounding those results.
  • Toxicity scores came from an unbiased RoBERTa detector ranging from 0 to 1, with higher scores indicating more toxic content.
  • The reported mean toxicity averages scores across 500 prompts and 25 samples per prompt, differing from Expected Maximum Toxicity and Probability of Toxicity metrics.
  • Toxicity-score distributions are bimodal, and increasing model size shifts base-model probability mass toward the high-toxicity peak.

C.1 Preference Model Pre-training

Preference-model pre-training uses ranked examples from several domains and evaluates scaling across model sizes. The resulting datasets combine cleaned, domain-specific preference pairs under a common training setup.

  • Ranked pre-training datasets were prepared for each domain before binarization converted them into binary preference data.
  • StackExchange preference pairs compare answers with distinct scores derived from upvotes and acceptance, while Reddit pairs compare comments by final-comment upvotes.
  • Figure 28 reports best comparison test loss across model sizes and evaluations, with larger models performing better but often deviating from clean power-law trends.
  • Wikipedia preference pairs use page versions before and after edits, labeling reverted suspected-vandalism versions as better.
  • The preprocessing removed irrelevant metadata and uninteresting sections, producing 1.4M training pairs and 14k test pairs for one mixture dataset.
  • Models were trained for one epoch with 1024-token contexts and evaluated using preference-model accuracy and loss.

C.2 Preference Model Pre-Training

The finetuning experiments compare preference-model pre-training with no pre-training across several evaluations. They use fixed training settings while varying the number of finetuning sequence pairs.

  • Figure 17 compares performance as a function of finetuning sequence pairs for preference-model pre-training on Mix against no pre-training.
  • Finetuning used 1024-token sequences, batch size 32, a constant learning rate, one epoch for Learn to Summarize and HellaSwag, and four epochs for Ethics evaluations.

C.3 Language Modeling Improves PMP Transfer

Adding an autoregressive language-modeling loss to preference-model pre-training substantially improves transfer and sample efficiency, whereas either loss alone transfers poorly. The combined PM+LM objective appears to encourage more language-relevant features than pure preference modeling.

  • Training objective: The pre-training objective combines preference modeling with an autoregressive loss that imitates the good sample in each pair.The total loss weights the PM and LM terms with hyperparameters λ and µ; the LM term predicts the full good-sample context without token masking.
  • Comparison of losses: Pure LM performs similarly to no preference-model pre-training, while pure PM helps with few samples but eventually underperforms the no-PM baseline.The comparison includes no PM pre-training, pure PM, pure LM, and composite PM+LM conditions.
  • Transfer results: The PM+LM combination consistently improves sample efficiency relative to no PM pre-training and outperforms pure LM.This indicates that the gain is not attributable solely to language modeling, but to combining PM and LM losses.
  • Interpretation: Pure PM may learn trivial features such as context length or token frequencies, whereas adding LM encourages more substantial language-relevant features.This is presented as a hypothesis explaining why the combined objective transfers better than either loss alone.
  • End-of-context token: Appending an end-of-context token consistently improves preference-model performance when used in both pre-training and finetuning.The token is placed at the end of each sample context, and the preference score is predicted directly on top of it.

C.5 Ensembling Over PMP Models

Ensembling preference models that share a finetuning dataset but use distinct PMP datasets provides only a small accuracy gain. The experiment compares PMP from Reddit and Stack Exchange and reports an improvement of about 0.5%.

  • Ensembling over PMP models: Ensembling summarization preference models separately pre-trained on Reddit and Stack Exchange produced an accuracy gain of about 0.5%.The models were finetuned on the same final dataset but passed through different PMP distributions first.
  • Evaluation scope: The reported PMP accuracy-gain analysis spans multiple pre-training datasets and finetuning evaluations, including summarization, HellaSwag, and Ethics tasks.The cited figure describes accuracy differences relative to no PMP at 500 and 5k finetuning sequence pairs.
  • Related transfer comparison: A controlled comparison examines transfer from ranked versus binary preference-model pre-training under increasing distribution scrambling.The figure reports that sample efficiency degrades as relative scrambling increases and places binary pre-training between weakly and very strongly scrambled cases.

C.6 Experiments on Ranked vs Binary PMP – Synthetic Symbols Dataset

Synthetic-symbol experiments show that PMP transfer depends strongly on similarity between pre-training and finetuning preference distributions. Binary PMP is a robust compromise: less effective than closely matched ranked data, but more transferable than strongly mismatched rankings.

  • Synthetic dataset construction: The control dataset ranks pairs of distinct symbols according to a reference ordering, while T1–T4 increasingly scramble that ordering.T1 is weakly scrambled, T2 moderately scrambled, T3 strongly scrambled, and T4 very strongly scrambled.
  • Binary PMP: The binary dataset labels symbols in the first half of the control ranking good and those in the second half bad.Five preference models are pre-trained separately on the binary and scrambled datasets, then finetuned and evaluated on the control dataset.
  • Transfer results: Sample efficiency consistently worsens as scrambling increases, with a threshold beyond which it becomes worse than no PMP pre-training.The authors interpret this as evidence that substantially different Elo scales transfer poorly across preference-model distributions.
  • Transfer results: Binary pre-training is similarly sample efficient to moderately scrambled pre-training, outperforming strongly scrambled transfer but not necessarily weakly scrambled transfer.This matches the expectation that binary preferences preserve less ranking information than a closely matched ranked dataset.
  • Practical implication: The best PMP dataset is qualitatively similar to the final finetuning dataset, but binary PMP offers more robust transfer when that dataset is unavailable.The authors characterize binarized PMP as a compromise that cannot guarantee maximum sample efficiency.
  • Synthetic dataset construction: The larger synthetic dataset uses 676 ordered pairs of uppercase letters, repeated within contexts to form preference pairs.Scrambled datasets apply 10, 40, 160, or 640 random transpositions, while the binary dataset divides the control ranking into halves.

D Per-Token GAN-Style Discriminator Results

A token-level discriminator can detect some model-generated text, but it performs worse than the original language model for benchmark-based sample ranking. Combining discriminator and language-model probabilities provides a more principled ranking method and can improve on the language model.

  • Discriminator setup: The discriminator is trained with token-level binary cross-entropy to classify human-generated versus model-generated tokens.Training data mixes unchanged human text, truncated human text continued by a model, and fully model-generated sequences.
  • Benchmark comparison: The discriminator does not produce promising benchmark improvements over the language model that generated its training samples.For ranking, discriminator scores average predicted human-generation probabilities, while the language model uses negative average token log-probability.
  • Ensembling: The ensemble combines discriminator and language-model probabilities to re-rank generated samples more principledly than using the discriminator alone.The reported results show that the ensemble can improve on the original language-model distribution.
  • Additional evaluation: Human-human and human-model PMP datasets do not consistently differ on the Ethics evaluations, whose results are described as rather random.The authors suspect human-model data has no particular advantage for evaluations composed purely of human-written material.
  • Per-token results: Larger discriminators achieve higher overall accuracy but perform worse immediately after the transition from human-generated to model-generated tokens.Per-token accuracy is measured relative to the position where generation switches from human to model output.

E.1 How the HHH criteria relate to alignment

The HHH criteria frame alignment as an assistant’s overlap with human interests, expressed through helpfulness, honesty, and harmlessness. The framework is practically useful but leaves key questions unresolved and does not by itself ensure secure AI.

  • Alignment and HHH: Alignment is described as overlap between agents’ rankings of outcomes, with HHH behavior serving as evidence of alignment with human interests.An assistant is considered aligned when it tries to satisfy humans’ interests, including their interests in avoiding harm and deception.
  • Alignment and HHH: Helpfulness means pursuing humans’ best interests, honesty means conveying accurate information without deception, and harmlessness means avoiding harm.
  • Open questions: The account leaves open which outcome orderings and agents matter, how rankings should be aggregated, and how alignment overlap should be precisely defined.These choices may implicitly shape how aligned systems are trained and how HHH criteria are defined.
  • Additional criteria: Honesty is included as a distinct practical focus even though the framework treats it as reducible to helpfulness and harmlessness.The paper also considers handleability, or responsiveness to human feedback and intended instructions, as a possible fourth H.
  • Alignment and security: HHH criteria address alignment rather than all of AI security, so additional work may be needed to limit harm when systems are not fully aligned.Security can also depend on properties or surrounding systems beyond the assistant’s own HHH behavior.
Loading 2112.00861v3…