PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, Jeff Wu

arXiv:2312.09390v1cs.CL

TL;DR

The paper studies whether weak model supervision can elicit capabilities from much stronger models, addressing an empirical challenge posed by future superhuman systems that humans cannot reliably supervise. Across NLP, chess, and reward modeling, naive finetuning consistently beats weak supervisors, while auxiliary confidence loss and other simple methods improve generalization, though important limitations remain.

  • Problem

    The paper asks whether weak supervision can elicit the full capabilities of much stronger models when humans cannot reliably evaluate their complex behaviors.

  • Method

    The authors study this analogy by finetuning stronger pretrained GPT-family models on weak-model labels across NLP, chess, and reward modeling tasks, then testing simple improvement methods.

  • Results

    Strong models consistently outperform their weak supervisors under naive finetuning, typically recovering about half the performance gap, while auxiliary confidence loss can recover nearly 80% on GPT-4 NLP tasks supervised by a GPT-2-level model.

  • Takeaways & Limitations

    Weak-to-strong generalization is widespread and can be substantially improved with simple methods, making empirical progress toward aligning superhuman models feasible.

  • Takeaways & Limitations

    Naive bootstrapping helped in chess but produced only small NLP improvements and none in reward modeling, so it alone is insufficient for aligning models much smarter than their supervisors.

Abstract

from arXiv · show

Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work. We find that simple methods can often significantly improve weak-to-strong generalization: for example, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.

1 INTRODUCTION

The paper studies whether weak models can supervise much stronger models as an empirical analogy to the challenge of aligning future superhuman systems. Across NLP, chess, and reward modeling, strong models outperform weak supervisors, but naive finetuning leaves substantial capability gaps that simple methods can partly reduce.

  • RLHF relies on human evaluators reliably judging whether model behavior is good or bad.
  • Superhuman models may produce behaviors too complex for humans to evaluate reliably, creating the superalignment problem of controlling models much smarter than their supervisors.
  • The paper studies this challenge by finetuning large pretrained models on labels generated by smaller models and measuring their generalization.
  • Strong pretrained models consistently outperform their weak supervisors under naive weak supervision; on NLP tasks, GPT-4 typically recovers about half the performance gap from GPT-2-level labels.
  • Naive finetuning still leaves a substantial gap from ground-truth supervision, especially for ChatGPT reward modeling, suggesting naive RLHF may scale poorly without additional work.
  • Simple improvements can substantially increase weak-to-strong generalization; an auxiliary confidence loss typically recovers nearly 80% of the weak-to-strong performance gap on NLP tasks.
  • The methods are proofs of concept rather than deployment-ready solutions because they do not work consistently, remain weak in reward modeling, and leave important setup disanalogies unresolved.

2 RELATED WORK

The paper connects weak-to-strong learning to weak supervision, student-teacher training, robust pretraining, debiasing, imitation and preference learning, and scalable oversight. It distinguishes its setting by using pretrained students that are much more capable than their weak supervisors and by targeting generalization beyond reliably supervised behavior.

  • Weakly-supervised learning: Weak-to-strong learning is a form of weakly supervised learning that uses unreliable, instance-dependent labels rather than uniform label noise.
  • Student-teacher training: Student-teacher methods commonly train students on teacher-generated pseudo-labels, whereas this paper studies pretrained students that are much more capable than their teachers.
  • Robustness of pretraining and finetuning: Pretraining can produce robust representations, while finetuning often improves in-distribution generalization but may perform poorly or degrade performance out of distribution.
  • Debiasing: The setting can be viewed as debiasing against unknown capability-related biases in weak labels, unlike methods designed for known group biases.
  • Imitation and preference learning: Imitation learning and RLHF steer already-capable pretrained models toward desired behavior using demonstrations or human preferences.
  • Scalable oversight: Scalable oversight improves human supervision through techniques such as model critiques or problem decomposition, whereas this work focuses on generalizing beyond supervision that humans cannot reliably provide.
  • Knowledge elicitation and honesty: The paper also relates to eliciting latent knowledge and honesty, which seek to elicit useful or truthful information from superhuman models.

3 METHODOLOGY

The paper studies weak-to-strong learning by training strong models with weak-model labels and comparing them with weak supervisors and ground-truth-trained strong ceilings. It defines performance-gap recovery and discusses the setup’s advantages and remaining disanalogies to superalignment.

  • Setup: Weak-to-strong learning replaces weak human supervision with weak model supervision to study how weaker supervisors can oversee much stronger models.The setup finetunes large pretrained models on labels generated by small models and measures their generalization.
  • Setup: Weak supervisors are created by finetuning small pretrained models on ground-truth labels, then generating labels from predictions on held-out examples.The weak supervisor’s performance is called weak performance.
  • Setup: Strong student models are finetuned on generated weak labels, and their resulting performance is called weak-to-strong performance.This is compared against the weak supervisor and a strong model trained on ground-truth labels.
  • Evaluation: PGR measures the fraction of the performance gap between weak and strong ceiling models recovered through weak supervision.PGR equals 1 for perfect weak-to-strong generalization and 0 when the student performs no better than the weak supervisor.
  • Advantages: The setup supports experiments across model pairs and tasks without human feedback, enabling faster feedback loops and empirical testing across varied settings.The authors also describe practical usefulness for aligning current models with weaker supervision.
  • Limitations: The analogy remains limited by imitation saliency, pretraining leakage, unknown future model inductive biases, and access to ground-truth labels in today’s experiments.These differences mean the setup removes a major prior-work disanalogy but is not perfectly analogous to future superalignment.

4 MAIN RESULTS

Across NLP, chess, and reward modeling, strong students generally outperform weak supervisors, but naive weak supervision leaves a substantial capability gap. Bootstrapping and an auxiliary confidence loss improve generalization in selected settings, with the latter producing especially large NLP gains.

  • Naive weak supervision: Strong students outperform their weak supervisors across NLP, chess, and reward modeling, demonstrating weak-to-strong generalization.The experiments evaluate GPT-4-family models across the three settings using labels generated by weak supervisors.
  • Naive weak supervision: On NLP benchmarks, performance-gap recovery often exceeds 50% for the largest students and increases with both supervisor and student size.Even very weak supervisors can yield more than 20% gap recovery, while the largest students often exceed 50%.
  • Naive weak supervision: Chess shows mixed scaling: performance-gap recovery can exceed 40% for small supervisor-student gaps but decreases as student size grows for a fixed supervisor.The corresponding accuracy curves may exhibit inverse scaling in strong student size.
  • Naive weak supervision: Reward modeling generalizes poorly, usually recovering roughly 10% of the performance gap and almost never exceeding 20% even for relatively small compute gaps.This setting shows poor generalization and poor scaling under naive finetuning.
  • Bootstrapping with intermediate model sizes: Bootstrapping with two intermediate model sizes improves chess performance-gap recovery, especially for larger students, while naive transfer curves flatten as the gap grows.With bootstrapping, accuracy continues to improve monotonically across the model sequence.
  • Auxiliary confidence loss: An auxiliary confidence loss substantially improves NLP generalization by letting strong students remain confident in predictions that disagree with weak labels.For the smallest supervisor and largest student, median performance-gap recovery rises from about 25% to nearly 80%; the loss reduces imitation of weak errors and weak-label overfitting.

5 UNDERSTANDING WEAK-TO-STRONG GENERALIZATION

The paper examines why weak supervision sometimes elicits strong-model capabilities and how training choices affect this process. Across tasks, strong students can avoid some supervisor errors, but overfitting and representation limits leave substantial gaps that several interventions reduce.

  • 5.1.1 OVERFITTING TO WEAK SUPERVISION: Large weak-strong gaps produce overfitting: ground-truth performance can rise initially, then decline as training continues.Ground-truth early stopping improves PGR by around 5 percentage points on reward modeling and by 15 percentage points over final training on NLP tasks.
  • 5.1.2 STUDENT-SUPERVISOR AGREEMENT: The confidence loss reduces imitation of supervisor mistakes by encouraging confident predictions that can contradict weak labels.It reduces the advantage of ground-truth early stopping from a 15% PGR boost to roughly 5% for large gaps on NLP tasks.
  • 5 UNDERSTANDING WEAK-TO-STRONG GENERALIZATION: Strong students agree less with supervisor errors as the student-supervisor gap grows, despite being trained to imitate weak labels.This inverse-scaling trend is strongest on examples where supervisors are wrong and persists under cross-entropy evaluation.
  • 5.2.1 ELICITING STRONG MODEL KNOWLEDGE WITH PROMPTING: Five-shot prompting becomes competitive with or better than weak-to-strong finetuning for the largest students, although confidence-loss finetuning generally performs best.The prompting comparison averages weak-to-strong performance across 7 NLP tasks.
  • 5.2.2 GENERATIVE SUPERVISION IMPROVES RM WEAK-TO-STRONG GENERALIZATION: Generative finetuning improves reward-model weak-to-strong PGR by approximately 10-20%, and combining it with ground-truth early stopping reaches approximately 30-40%.The combined result makes reward-model performance competitive with weak-to-strong generalization on NLP and chess tasks.
  • 5.2.3 FINETUNING ON WEAK SUPERVISION TO INCREASE CONCEPT SALIENCY: Weak-label finetuning can make ground-truth concepts more linearly represented: subsequent ground-truth linear probing reaches 78% accuracy and closes 60% of the relevant gap.This procedure also outperforms the naive weak-to-strong finetuning baseline.

6 DISCUSSION

The discussion frames weak-to-strong generalization as empirical progress toward superalignment while emphasizing important disanalogies, reliability requirements, and unresolved research questions.

  • 6 DISCUSSION: The setup provides an empirical analogy for studying a core superalignment challenge, but it still has important disanalogies from aligning superhuman models.The authors describe the work as significant progress rather than a complete solution.
  • 6.1 REMAINING DISANALOGIES: Superhuman models may imitate weak human errors more readily than the strong models in this setup, potentially making current results optimistic.The paper notes that its strong models are not explicitly pretrained to imitate weak models.
  • 6.1 REMAINING DISANALOGIES: Pretraining may have exposed models to some evaluated tasks, whereas future superhuman alignment capabilities could remain latent and harder to elicit.The authors identify this pretraining leakage as a disanalogy that could make results overly optimistic.
  • 6.2 FUTURE WORK: A convincing solution to superalignment will require analogous setups, scalable methods, and strong scientific understanding.The paper explicitly presents these as three necessary areas for substantial progress.
  • 6.2 FUTURE WORK: Follow-up work should improve analogy, validate disanalogies, relax task simplifications, test robustness under optimization pressure, and update methods as future models evolve.The proposed directions include complicated generative tasks, reward-model optimization, and continued setup refinement.
  • 6.2 FUTURE WORK: High benchmark performance alone is insufficient for high-stakes alignment; researchers also need to understand when and why methods work.The paper highlights questions about reward-model versus NLP performance, elicitation difficulty, label-free error estimation, and extrapolation.

A.1 NLP TASKS

The NLP appendix describes balanced binary classification experiments using GPT-4-family models, weak labels from held-out data, and accuracy-based evaluation across benchmark datasets.

  • A.1 NLP TASKS: The experiments use popular NLP classification benchmarks, with dataset names obfuscated in plots and task-specific preprocessing applied.Examples include FLAN preprocessing and grouping neutral ANLI entailments with contradictions.
  • A.1 NLP TASKS: Models are adapted to classification by replacing the unembedding layer with a two-output linear head initialized from the “0” and “1” token embeddings.The head produces binary classification outputs.
  • A.1 NLP TASKS: Training runs for 2 epochs with batch size 32, using weak-label validation accuracy for early stopping; hyperparameters are tuned only on smaller models and eight datasets.The full GPT-4 model and most datasets are held out, apart from datasets [5–12].
  • A.1 NLP TASKS: Weak labels are generated by training the supervisor on one dataset half and predicting the other, while related datapoints remain in the same split.Weak labels are also saved on the test set for agreement metrics.
  • A.1 NLP TASKS: Test accuracy is reported on balanced test sets, making random-guess performance 50% for every NLP dataset.The dataset table summarizes the benchmark sources.
  • A.1 NLP TASKS: Figure 12 reports detailed baseline and auxiliary-confidence results across all datasets, while Figure 13 reports overfitting to weak-supervisor predictions.The figures cover the full dataset suite and training dynamics.

A.2 CHESS PUZZLES

The chess experiments evaluate next-move prediction on puzzles using greedy weak labels, no early stopping, and comparisons against zero-shot performance and supervisor agreement.

  • A.2 CHESS PUZZLES: Chess preprocessing accounts for pretraining exposure to PGN game moves, while evaluation requires predicting the best move in a puzzle position.The source notes that the pretrained games contain played moves rather than necessarily best moves.
  • A.2 CHESS PUZZLES: All chess models are finetuned for 5 epochs with batch size 32 and no early stopping.This training schedule is used throughout the chess experiments.
  • A.2 CHESS PUZZLES: Weak labels are greedy-decoded completions from weak models on an additional held-out set of 50k puzzles.Temperature is set to T = 0, selecting the weak model’s highest-likelihood move.
  • A.2 CHESS PUZZLES: Chess accuracy is the fraction of held-out test positions for which greedy decoding produces the correct next move.Evaluation also uses temperature T = 0.
  • A.2 CHESS PUZZLES: Zero-shot performance improves with model size, and students supervised by much weaker models can underperform same-sized zero-shot models.Figure 15 compares baseline and bootstrapping against zero-shot performance.
  • A.2 CHESS PUZZLES: Supervisor-student agreement decreases as student models become larger on chess puzzles.The trend parallels the reported NLP agreement behavior.

A.3 CHATGPT REWARD MODELING

The reward-modeling appendix describes pairwise dialogue-completion preference experiments and examines agreement, generative finetuning, early stopping, auxiliary confidence loss, and balancing limitations.

  • A.3 CHATGPT REWARD MODELING: Each reward-model datapoint contains a user-assistant dialogue, two candidate completions, and a pairwise preference label.The label indicates which completion the human labeler preferred.
  • A.3 CHATGPT REWARD MODELING: Reward models use a single-output linear head, and pairwise predictions apply a sigmoid to the difference between completion logits.The head is initialized from an arbitrary token’s original unembedding weights.
  • A.3 CHATGPT REWARD MODELING: Reward models are trained for 1 epoch with batch size 220 and no early stopping.This is the stated reward-modeling training configuration.
  • A.3 CHATGPT REWARD MODELING: Weak comparison labels are produced from weak-model sigmoid preferences after training on half of the available comparison data.The resulting soft labels are defined by the difference between weak-model completion logits.
  • A.3 CHATGPT REWARD MODELING: Supervisor-student agreement decreases as reward-model students become larger, paralleling the NLP and chess results.Figure 16 reports this reward-modeling agreement trend.
  • A.3 CHATGPT REWARD MODELING: Generative finetuning and ground-truth early stopping stack their PGR improvements, producing reward-modeling results competitive with NLP and chess settings.Figure 18 also evaluates additional generative finetuning for both supervisors and students.
  • A.3 CHATGPT REWARD MODELING: The auxiliary confidence loss mixes weak-model predictions with thresholded strong-student predictions through a weighted cross-entropy objective.The threshold is adapted so that exactly half of examples satisfy the strong prediction condition.
  • A.3 CHATGPT REWARD MODELING: The balancing mechanism is practical only for low-n classification and usually has small or neutral effects, though it helps substantially in a few settings.Its training prior reflects the assumed label distribution.

B ADDITIONAL RESULTS ON METHODS

The experiments test several methods for improving weak-to-strong generalization, finding that confidence-based approaches can help but results are inconsistent across settings. Additional analyses examine easy-to-hard transfer and alternative training strategies.

  • Method evaluations: Many improvement methods were not consistently useful across datasets and model sizes, and the preliminary tuning was not optimal.The authors note that better tuning could change some results.
  • Confidence methods: Confidence thresholding helps mainly when weak labels are very inaccurate, but becomes ineffective once weak-label accuracy reaches roughly 70% to 80%.The method selects only the most confident examples from each class.
  • Confidence methods: The confidence auxiliary loss encourages predicted probabilities to move away from 0.5, but performs worse than the main confidence loss in finetuning.It also produces negative results with highly class-imbalanced data or without rebalancing.
  • Other methods: Weight averaging, LP-FT, weight regularization, LoRA, and data augmentation did not improve performance over the finetuning baseline in the tested settings.The experiments included exponential moving averages, stronger weight decay, dropout, low-rank updates, and consistency between original and rephrased inputs.
  • Easy-to-hard generalization: Easy-to-hard transfer improves performance outside the training difficulty range, often through Elo 1500 or higher, but underperforms on the hardest chess examples.Models trained only on easy puzzles generally outperform zero-shot performance on average, while hard-example performance can remain near the zero-shot baseline.

C.2 NLP TASKS: DIFFICULTY THRESHOLDING

This section constructs difficulty thresholds for NLP examples and evaluates training on easier subsets. Accuracy generally increases with the difficulty cutoff, while GPT-4-based difficulty filtering can show steeper but potentially confounded generalization.

  • Difficulty construction: NLP datasets lack natural difficulty labels, so the study estimates difficulty using model-size-dependent ground-truth performance and GPT-4 pairwise rankings.GPT-4 rankings are converted into Elo scores through random pairwise comparisons.
  • Difficulty construction: Examples are assigned the smallest model size that consistently predicts their labels correctly, then training sets are filtered by a difficulty cutoff.Filtered sets are subsampled to equalize training-set size, while evaluation uses the same unsampled held-out test set.
  • Model-based thresholds: Increasing the difficulty cutoff generally increases test accuracy, showing that easy-to-hard generalization remains non-trivial even without weak-label errors.The experiment varies training-example difficulty while holding label accuracy, training-set size, and test-set choice constant.
  • Model-based thresholds: Larger models often perform well when trained on easy examples, whereas smaller models eventually decline when the cutoff exceeds their capacity to fit the examples.The decline occurs when included examples become too difficult for the model itself.
  • GPT-4 difficulty thresholds: GPT-4 Elo scores align better with human difficulty and weak-model confidence on reasoning tasks such as MATH than on some natural-language tasks.The scores are evaluated against human annotations and weak-model confidence.
  • GPT-4 difficulty thresholds: GPT-4 difficulty filtering produces steeper generalization slopes, but noisy Elo measurements may confound this interpretation and the setup represents covariate shift rather than the paper’s main concept shift.The authors leave the relative relevance of the two setups unclear.

D.1 SELF-SUPERVISED VISION MODELS

Weak-to-strong generalization also appears in self-supervised vision representations using linear probes. The experiments support cross-domain transfer, while linear probing serves mainly as a faster setting for studying qualitative trends.

  • Experimental setup: The vision experiment uses AlexNet to generate weak ImageNet labels and trains linear probes on frozen DINO representations from ResNet-50 and ViT-B/8 models.The strong students use representations learned through self-supervised pretraining rather than direct ImageNet task supervision.
  • Experimental setup: The probes train on 40k validation examples and evaluate on the remaining 10k examples.Training uses Adam with learning rate 10^-3 and batch size 128.
  • Vision results: The strong vision students substantially outperform AlexNet, achieving performance recovery on the order of 50%.This mirrors the main weak-to-strong experiments.
  • Vision results: The result extends weak-to-strong generalization beyond natural language and shows transfer when task knowledge is only latent in pretrained representations.The strong students use indirect pretraining rather than explicit task training.
  • Linear probing: Linear probing freezes model parameters and trains new heads with either ground-truth or weak labels, providing a controlled and faster experimental setting.The paper reports that finetuning performs better, although qualitative trends are similar.

E.1 SYNTHETIC EXPERIMENTS ON SIMULATION DIFFICULTY

Synthetic experiments show that weak-label error structure matters beyond overall accuracy. Strong students gain little when weak mistakes are easy to simulate, but can denoise unpredictable errors.

  • Perfect simulation: The experiment constructs linear-probe models where larger feature sets can perfectly simulate models using smaller feature subsets.Representations are extracted from an intermediate GPT-4-family model, and model M_k uses the first k features.
  • Perfect simulation: When weak errors are perfectly simulatable, strong students show no substantial improvement over supervisor performance.The setup makes simulation easy by construction.
  • Unpredictable errors: Randomly flipped weak labels with the same accuracy as the original weak models produce substantially higher student accuracy because the errors are unpredictable.This setting tests weak-label accuracy separately from error structure.
  • Mixed errors: Adding noise to perfectly simulatable labels allows the strong student to recover performance close to that of the simulatable supervisor model M300.The experiment varies the amount of added label noise.
  • Interpretation: The results indicate that weak-label error structure is critical: easily imitated errors limit naive generalization, while unpredictable errors can be denoised.The implication depends on the relationship between the weak error structure and the strong model.

E.2 DIFFERENT WEAK ERROR STRUCTURE MEANS DIFFERENT GENERALIZATION

Weak error structure, not just weak-label accuracy, strongly affects weak-to-strong generalization and training dynamics. When weak-label imitation is trivial, generalization is minimal, while confidence loss can mitigate overfitting and retain some gains.

  • Weak labelers with identical accuracy produce widely varying generalization depending on their error structures.
  • Training often begins with generalization before overfitting to weak-model errors, which the confidence auxiliary loss mitigates.Easy-to-fit errors, such as longest-prompt errors, overfit faster.
  • The setup may underestimate a key future disanalogy because current strong models are not very good at imitating weak supervisors.If superhuman models imitate humans more effectively, default generalization could be less desirable.
  • The trivial-imitation setting appends the weak label in-context, allowing the strong model to simulate the weak supervisor directly.
  • 0.002 ± 0.003 average PGR for baseline and 0.046 ± 0.108 for auxiliary loss occur when weak-label emulation is trivial.These values are far below approximately 0.2 and 0.8 for the original tasks, respectively.
  • Even in the trivial-imitation setting, confidence loss improves over baseline and achieves non-trivial generalization on several tasks.

F HOW SHOULD WE EMPIRICALLY STUDY SUPERALIGNMENT, METHODOLOGICALLY?

The paper argues that superalignment should be studied with increasingly analogous, assumption-explicit, sensitivity-tested, and scalable empirical setups. Its weak-to-strong framework targets alignment of an early model capable of automating alignment research, while acknowledging important assumptions and disanalogies.

  • Superalignment is difficult to study empirically because it concerns future superhuman models rather than capabilities faced today.Prior work was either theoretical or toy-scale, or studied humans supervising current models without addressing superhuman-model challenges.
  • The paper recommends increasingly analogous empirical setups, explicit enumeration of assumptions, sensitivity analysis, scalable techniques, incidental usefulness today, and periodic updating.
  • Finetuning is emphasized over prompting because it is considered more likely to scale to superhuman models.
  • Weak-to-strong generalization is intended to solve alignment of the first model capable of automating machine-learning and alignment research.The broader plan then uses that aligned model to help align increasingly capable systems.
  • Possible elicited capabilities include safety, honesty, instruction following, and code security, potentially serving as a reward model or manually queried oracle.The paper states that future work should assess whether these capabilities can be made robust enough for safe optimization.
  • The alignment plan assumes pretrained base models are highly intelligent but not highly agentic or deceptively aligned out of the box.
Loading 2312.09390v1…