Weak-to-Strong Generalization under Distribution Shifts

Weak-to-strong generalization

• The weak-to-strong framework models human supervision with a weak model and trains a strong model to exceed weak-supervisor performance and approach ground-truth training.Prior work established that strong pretrained models can naturally generalize beyond their weak supervisors.Source: §1 · pp. 1–2

Problem

• Distribution shifts make weak models less reliable because they may be unfamiliar with the data used for fine-tuning or annotation.The paper motivates this with examples such as radiologists misinterpreting scans from unfamiliar machines or rare diseases.Source: §1 · p. 2, Figure 1 · p. 2
• Out-of-distribution weak supervision can become ineffective enough that the strong model performs worse than its weak supervisor.Weak supervision is reported to be significantly less effective in out-of-distribution than in-distribution settings.Source: §1 · p. 2

RAVEN

• RAVEN dynamically combines an ensemble of weak annotators while jointly updating their supervision weights and the strong model.The framework is designed to assign higher weights to more reliable weak models under distribution shift.Source: §1 · p. 2

Empirical results

• 55% improvement in image classification, 57% in text classification, and 33% in preference alignment over the best alternative baselines demonstrate RAVEN’s broad effectiveness.These evaluations cover image classification, text classification, and preference alignment in text generation.Source: §1 · p. 2

Weak-to-strong generalization

• Weak-to-strong generalization trains a strong model on pseudo-labels from a weak model and evaluates its performance against ground-truth-trained and weak-model references.The procedure uses source data to train the weak model, fine-tuning data for the strong model, and target data for evaluation.Source: §2 · p. 3
• PGR measures how much of the gap toward ground-truth-level performance is recovered by the strong model trained with weak supervision.A value close to 1 indicates that weakly supervised training approaches the ground-truth-trained model.Source: §2 · p. 3

Distribution shift

• The original W2S assumption is that source, fine-tuning, and target distributions come from the same underlying distribution.The paper relaxes this assumption by explicitly modeling shifts between source and fine-tuning data.Source: §2 · p. 3
• Distribution shift is characterized by the generalization gap ∆R(f) = R_tuning(f) − R_src(f), with larger gaps indicating greater misalignment between source and fine-tuning data.This setting represents weak supervision from difficult or unfamiliar domains and uses PGR to assess whether it still improves the strong model.Source: §2 · p. 3
• On 11 of 19 IMAGENET-C corruption cases, naive W2S is infeasible because the strong model performs worse than the weak model.OOD generalization rarely leverages the strong model’s capabilities as effectively as the InD setting.Source: §2 · p. 3

WeakS-to-strong generalization

• WeakS-to-strong generalization extends W2S by using multiple weak models as supervisors.A straightforward ensemble combines their predictions with predefined weights before computing the training loss.Source: §3 · pp. 3–4
• Fixed ensemble weighting combines weak-model predictions using predefined coefficients, such as averaging.The combined predictions are used with cross-entropy loss to train the strong model.Source: §3 · p. 4

Motivation and framework

• RAVEN dynamically combines weak models to handle their substantially higher OOD performance variance and domain-dependent effectiveness.The framework targets settings where weak models are trained on distributions different from the strong model’s fine-tuning data.Source: §4 · p. 4

Adaptive weighting

• Adaptive weighting trains ensemble weights jointly with the strong model so more reliable weak-model predictions receive greater influence.The weak models remain fixed while their ensembling weights are trainable and updated throughout training.Source: §4 · p. 4

Easy-sample guided initialization

• Easy-sample guided initialization warms up the strong classifier on samples where all weak models agree, reducing early shortcutting toward one similar weak model.This addresses instability while the strong classifier is still under-optimized.Source: §4 · p. 5

Optimization procedure

• RAVEN alternates updates to the weak-model weights and strong-model parameters until convergence, then returns the trained strong model.The procedure applies adaptive-weighting optimization over the full fine-tuning dataset after initialization.Source: §4 · p. 5

Weight selection behavior

• By training, RAVEN usually assigns the highest weight to the best-performing weak model for the target distribution without ground-truth target labels.This behavior is supported by a strong positive correlation between weak-model and W2S performance.Source: §4 · p. 5

Experimental setup

• RAVEN’s ensemble combines adaptive weighting with easy-sample-guided initialization, and its evaluation covers three task families and multiple shift settings.Preference alignment uses DPO-R to integrate RAVEN into alignment tasks.Source: §5 · p. 5, §5.1 · p. 6

Image classification

• 55% higher PGR and 3.7% higher accuracy than Ens show RAVEN’s image-classification advantage under distribution shift.RAVEN also improves PGR by 54% over Co-sup without using domain knowledge.Source: §5.2 · pp. 6–7

Text classification

• 57% higher PGR and 1.4% higher accuracy than the best alternative baseline establish RAVEN’s text-classification advantage in OOD settings.In-distribution PGR also improves by 27% over the best baseline.Source: §5.2 · p. 7, Table 2 · p. 7

Preference alignment

• 32.8% higher PGR and 2.7% higher WR than the strongest alternatives demonstrate RAVEN’s preference-alignment advantage in OOD settings.In-distribution improvements are 25.6% in PGR and 1.2% in WR.Source: §5.2 · p. 8, Table 3 · p. 8

Further analysis

• All RAVEN components contribute to performance, while adding more weak models improves results with diminishing gains after saturation.The ablation evaluates ensembling, easy-sample-guided initialization, and adaptive weighting.Source: §5.3 · pp. 8–9
• The strong model predominantly assigns the highest weight to the weak model with the best target performance despite lacking ground-truth performance information.This indicates that RAVEN can identify higher-quality annotations from multiple weak models.Source: §5.3 · p. 9

Weak-to-strong generalization

• Weak-to-strong generalization studies whether strong models trained on weak supervision can surpass their supervisors, extending work introduced in [10].Prior studies examined confidence loss, bootstrapping, unsupervised fine-tuning, and theoretical explanations [67] [64].Source: §6 · p. 10

Scalable oversight

• Scalable oversight uses models or task decomposition to improve supervision, whereas W2S leverages unreliable human supervision itself to train models beyond human performance.The distinction concerns the role of supervision rather than the use of models in oversight.Source: §6 · p. 10

Learning under distribution shift

• The proposed scenario is framed as an intersection of weak-to-strong generalization and learning under distribution shift.Distribution-shift research addresses poor generalization when query data differs substantially from training data.Source: §6 · p. 10

Limitations

• The weak models differ only by random seed, so their diversity may not fully represent human annotators, especially where domain expertise matters.This is identified as a limitation of the current evaluation and model construction.Source: §7 · p. 10
• Model-wise weights outperform model-sample-wise weights, but optimization challenges for the latter remain open for future work.Model-wise weighting is a subset of model-sample-wise weighting because its weights are constant across samples.Source: §7 · p. 10

Conclusion

• RAVEN addresses distribution shifts by extending weak-to-strong generalization beyond settings where source and fine-tuning distributions are aligned.The paper describes naive W2S as often intractable under these shifts.Source: §7 · p. 10, A Definition of distribution shift · p. 16

• RAVEN exceeds the weak model in most IMAGENET-C cases and outperforms naive W2S despite severe synthetic domain shifts.The experiment uses fine-tuned AlexNet weak models and DINO ResNet50 as the strong model.Source: B Motivating experiment · p. 16, Table 8 · p. 18
• Naive W2S results in Figure 2 average the no-fine-tuning and fine-tuning setups.Source: B Motivating experiment · p. 16
• The motivating experiment compares weak and strong accuracy with W2S performance reported as performance gain relative to the strong model trained with ground-truth labels.The no-fine-tuning setup averages results over five random splits.Source: B Motivating experiment · p. 16, Table 7 · p. 17

Assumptions

• RAVEN assumes the weak-model ensemble resembles the best-performing weak model, which is more accurate and confident than the ensemble prediction.Source: C Theoretical analysis of how RAVEN… · p. 19

Theoretical analysis

• The adaptation loss is optimized by alternating strong-model and adaptive-weight updates, causing the strong model to align increasingly with the best weak model.Source: C Theoretical analysis of how RAVEN… · p. 19
• RAVEN’s analysis concludes that the best weak model receives the highest weight when its predictions most closely match the strong model’s predictions.This conclusion can hold even when only the best weak model satisfies the confidence assumption, because it yields the smallest discrepancy term.Source: C Theoretical analysis of how RAVEN… · p. 19

Optimization procedure

• Adaptive weights start uniformly, while an easy-sample initialization phase trains only the strong model before minibatch-level joint optimization begins.Each minibatch combines weak-model outputs using the latest weights and updates the weights for subsequent training.Source: §D.1 · p. 20

Text classification

• For text classification, RAVEN forms pseudo-labels by applying a softmax to the weighted sum of weak-model logits.The weak-model logits represent class scores, and the resulting soft labels supervise the strong model.Source: §D.1 · p. 20

Weak-model construction

• Weak models are trained with varied seeds and task-specific optimizers, losses, schedules, and pretrained initializations to create ensemble diversity.Image, text, and reward-modeling weak supervisors use distinct training configurations.Source: §D.2 · pp. 20–21

Experimental protocol

• RAVEN and competing ensemble methods reuse pretrained weak-model groups across repeated experiments to balance statistical power and computational cost.Source: §D.2 · p. 21

Evaluation

• The evaluation reports performance gain ratios alongside weak-model and ground-truth-trained strong-model accuracy for ensemble methods.Source: §D.6 · p. 22

Preference alignment

• For preference alignment, weak models are trained as reward predictors, and their preference signals select chosen and rejected responses for strong-model DPO training.This pipeline uses source preference data for reward modeling and target data for strong-model alignment.Source: E Configurations for preference… · p. 22, §E.1 · p. 23
• DPO-R modifies DPO so adaptive weak-model weights can be learned, incorporating multiple reward models and separate weights for chosen and rejected log-likelihoods.Standard DPO cannot update weights that do not appear in its loss calculation.Source: §E.2 · p. 23

E.3 Training details for DPO

• DPO experiments use LoRA strong-model training with AdamW, cosine scheduling, and grid-searched preference-loss hyperparameters.Source: §E.3 · p. 23
• GPT-4o win rate evaluates preference alignment by comparing DPO-trained outputs with pre-DPO or baseline outputs under randomized response order.Source: §E.4 · p. 24

H.7 Best weak model vs. adaptive weighting

• RAVEN often outperforms an idealized best-weak-model baseline because it combines diverse weak signals early while later gravitating toward the best weak model.It can exceed ensemble W2S even without relying on the single best weak model, indicating favorable linear combinations matter.Source: §H.7 · p. 30

H.8 The choice of weights ΘW

• Model-wise weights outperform model-sample-wise weights, which provide finer adaptation but are harder to optimize.Source: §H.8 · p. 30
• Linear RAVEN matches the comparable nonlinear variant while using substantially fewer parameters.The nonlinear alternative uses an MLP and softmax to determine weights.Source: §H.9 · p. 30

Weight behavior

• Higher target-domain accuracy variance among weak models is associated with missed identification of the best weak model, with no misses observed for FMOW.Source: §H.10 · p. 31

• In the lethal-poison scenario, only RAVEN and the GT-aligned strong model are judged safer than the Chosen answer by ChatGPT-4o.The comparison concerns harmlessness.Source: §H.12 · p. 31
• RAVEN provides a detailed and precise warning about the dangers of using poison in this scenario.Source: §H.12 · p. 31
• The scaling experiments report average PGR using 3 weak models for Ens and RAVEN and 7 weak models for Co-sup.The best score is highlighted in red and the second-best in bold.Source: Table 20 · p. 32
• Figure 13 compares cases where ChatGPT-4o selects the Chosen sample over the generated response against cases where it prefers the generated response over the Rejected sample.These outcomes are labeled Loss (vs. Chosen) and Win (vs. Rejected), respectively.Source: Figure 13 · p. 32