PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang

arXiv:2410.18640v2cs.CL

TL;DR

The paper asks whether alignment signals from weaker models can help align stronger ones. It introduces WSPO, which learns distributional differences before and after weak-model alignment, and reports gains on key benchmarks.

  • Problem

    The paper asks whether alignment signals from weak models can be leveraged to align stronger models.

    Source: §1 · p. 2

  • Method

    WSPO transfers alignment by learning distributional differences before and after aligning the weak model.

    Source: §1 · p. 2

  • Results

    39.70 to 49.60 on Arena-Hard for Qwen2-7B-Instruct; WSPO also achieves 47.04 length-controlled win rate on AlpacaEval 2 and 7.33 on MT-bench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support WSPO as an efficient alternative to traditional alignment methods.

    Source: §7 · p. 10

    The study does not test transfer across different language-model architectures or varying weak-model alignment strengths, and does not explain the amplification effect.

    Source: §7 · p. 10

Abstract

from arXiv · show

Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible.

1 INTRODUCTION

Human-preference alignment follows pretraining and instruction fine-tuning, but conventional reward-based pipelines can be computationally costly. WSPO explores transferring alignment signals from weaker to stronger models.

1 INTRODUCTION

  • Language-model training proceeds through pretraining, supervised fine-tuning, and a final human-preference learning phase intended to better align models with human values and goals.The paper notes that models can still produce factual inaccuracies, exhibit biases, and display other undesirable behaviors.Source: §1 · p. 1
  • RLHF trains an explicit reward model and uses reinforcement learning to fine-tune the language model, while DPO aligns directly from human preferences without a separate reward model.The paper describes the RLHF pipeline as more complex and computationally costly than supervised fine-tuning.Source: §1 · p. 2
  • Weak-to-strong generalization shows that a strong model fine-tuned on labels from a weaker model can outperform its supervisor, motivating transfer of weak-model alignment signals.The paper asks whether those signals can align a stronger model.Source: §1 · p. 2
  • WSPO transfers weak-model alignment by learning distributional differences before and after that model is aligned, and the stronger model can amplify the transferred signal.The approach uses the weak model as a reward model rather than using its generated data as labels for the strong model.Source: §1 · p. 2
  • 39.70 to 49.60 on Arena-Hard for Qwen2-7B-Instruct; WSPO also achieves 47.04 length-controlled win rate on AlpacaEval 2 and 7.33 on MT-bench.The experiments also report preservation of the strong model’s knowledge on common-sense, mathematical, and other reasoning tasks.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe language-model training and the preference-based alignment approaches that WSPO builds on, from supervised fine-tuning through RLHF and DPO.

2 PRELIMINARIES

  • Autoregressive language models assign response probabilities through token-level conditional probabilities, while maximum-likelihood pretraining learns a distribution closer to the true data distribution.The response probability is decomposed using the chain rule over its tokens.Source: §2 · p. 2

2.1 SUPERVISED FINE-TUNING

  • Supervised fine-tuning adapts a pretrained model using curated expert demonstrations of high-quality responses, including instruction-following and dialogue examples.This yields the model π_sft.Source: §2.1 · p. 2

2.2 FINETUNING FROM HUMAN FEEDBACK

  • Human-feedback alignment generally maximizes expected reward from an implicit or explicit reward function while penalizing divergence from a reference policy with a KL-divergence term..Source: §2.2 · p. 2

2.2.1 RLHF

  • RLHF trains a reward model from preference comparisons, then optimizes expected reward while regularizing against a reference policy; β extremes risk under-alignment or reward hacking.Excessive divergence can compromise capabilities developed during pretraining or supervised fine-tuning.Source: §2.2.1 · p. 3

2.2.2 DPO

  • DPO bypasses reward-model training by optimizing a loss directly as a function of the policy, but may yield a less-regularized and less-robust strategy than RLHF.The passage presents this as a possible trade-off of simplifying the alignment process.Source: §2.2.2 · p. 3

3 METHOD

WSPO uses distributional differences before and after weak-model alignment to guide alignment of a stronger model. The method derives a reward representation and uses it to form an optimization objective for the strong model.

3 METHOD

  • The method uses RL theory to train a strong alignment model from distributional differences before and after weak-model alignment.This builds on weak-to-strong generalization rather than treating the strong model as simply imitating the weaker one.Source: §3 · p. 3

3.1 YOUR LANGUAGE MODEL IS SECRETLY A REWARD MODEL

  • Theorem 1 motivates representing reward as the distribution difference before and after model alignment (Rafailov et al., 2024).The theorem states that reward classes consistent with Plackett-Luce, including Bradley-Terry, can be represented using a model and a reference model.Source: §3.1 · p. 4

3.2 WEAK-TO-STRONG PREFERENCE OPTIMIZATION

  • The reward is analytically tractable, but substituting it into the KL-constrained objective is difficult because its partition function is absent.The resulting objective minimizes the difference between log-probability distributions before and after aligning the strong and weak models.Source: §3.2 · p. 4
  • WSPO uses the change in the weak model’s alignment as a supervisory signal for aligning the stronger reference model.The derivation uses the weak model before and after alignment; the reference model may be πsft or πbase.Source: §3.2 · p. 4
  • Equation 7’s gradient direction depends on its parenthetical terms, not the model’s dataset likelihood; the authors say this may mitigate DPO overfitting.The passage presents this as a possible benefit, not a demonstrated outcome.Source: §3.2 · p. 4
  • The WSPO pipeline uses offline preference or SFT datasets, requires no paired datasets, and tunes γ as its only parameter.It prepares the weak model before and after alignment, then optimizes the strong model on the specified dataset.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

Experiments test WSPO on toy summarization, single-turn dialogue, and instruction-following benchmarks. They report alignment transfer to stronger models, stability across sampling temperatures, and competitive performance against PPO and DPO.

4.1.1 EXPERIMENT SETUP

  • The toy summarization experiment rewards summaries within [Lmin, Lmax], using Qwen2 base models, XSUM data, and test-split evaluation with Lmin = 20 and Lmax = 30.The setup develops PPO-aligned weak models and uses their distributional differences for WSPO.Source: §4.1.1 · p. 5

4.1.2 RESULTS AND ANALYSIS

  • WSPO produces a length distribution different from the weak model’s and merges SFT and PPO stages, achieving comparable alignment with the PPO training-data amount.The authors say the approach works when the pre-trained model has acquired sufficient knowledge.Source: §4.1.2 · pp. 5–6
  • WSPO performs similarly to PPO, mitigating length outliers and remaining stable, with better performance at lower temperatures than the PPO-aligned strong model.Figure 2 compares generated lengths under greedy decoding and reward hits under top-p sampling at different temperatures.Source: §4.1.2 · p. 5, Figure 2 · p. 6

4.2.1 EXPERIMENTAL SETUP

  • The dialogue experiment applies DPO to preferred completions using Anthropic Helpful and Harmless training data, then evaluates single-step interactions with GPT-4o-mini as judge.Preferred completions from the test set serve as references for win-rate calculation.Source: §4.2.1 · p. 6

4.2.2 RESULTS AND ANALYSIS

  • WSPO improves alignment over Preferred-FT, outperforms DPO at γ = 0.1, and reaches a high win rate after 1k fine-tuning steps while remaining stable.Figure 3 reports GPT-4o-mini win rates across temperatures and relatively stable win rates throughout training.Source: §4.2.2 · pp. 6–7, Figure 3 · p. 7

4.3.1 EXPERIMENTAL SETUP

  • The broad evaluation uses MT-Bench, AlpacaEval 2, and Arena-Hard, plus five zero- or few-shot reasoning benchmarks to assess generalization beyond weak labels.The setup uses UltraChat-200k for Base SFT models and off-the-shelf instruction-tuned models as Instruct SFT models.Source: §4.3.1 · p. 7

4.3.2 RESULTS AND ANALYSIS

  • Instruct consistently outperforms Base; DPO shows limited improvement for Instruct and declines with Qwen2-7B-Instruct.Table 1 covers different settings and benchmarks; the comparison and decline are stated in the accompanying prose.Source: §4.3.2 · pp. 7–8, Table 1 · p. 8
  • WSPO is reported to preserve common-sense, mathematical, and reasoning capabilities while improving alignment; TruthfulQA improves for both Base and Instruct models.The cited table covers evaluation across benchmarks; the passage reports the outcomes but supplies no numerical results.Source: Table 2 · p. 8, §4.3.2 · p. 8

5 ANALYSIS

The analysis examines how weak-model size, regularization, and SFT affect WSPO transfer. A weaker model can provide a signal for stronger-model alignment, while γ controls alignment strength and WSPO’s gain from a base model is limited.

5 ANALYSIS

  • Figure 4 compares generated sequence lengths for WSPO and greedy-decoded PPO across weak-model sizes, and examines γ’s impact on WSPO in single-turn dialogue.The model-size analysis reports that even a weaker model can provide a robust alignment signal to a stronger model.Source: Figure 4 · p. 9, §5.1 · p. 9

5.1 IMPACT OF WEAK MODEL

  • 45.00 on Arena-hard was achieved by a Qwen2-7B model aligned with WSPO using a 0.5B weak model without alignment enhancement after DPO training.The authors note that weak-model alignment ability also matters and can be fine-grained before transfer.Source: §5.1 · p. 9

5.2 IMPACT OF HYPERPARAMETER

  • When γ = 1, strong-model alignment mirrors the weak model; below 1, alignment is superior, while above 1 it increasingly favors the original distribution.The authors infer that the strong model requires less regularization than the weak model in Objective 1.Source: §5.2 · p. 9

5.3 IMPACT OF SFT PHASE

  • 9.30 on Arena-Hard followed WSPO alignment of Qwen2-1.5-Base with Ultrafeedback, up from its initial score of 7.70.The authors describe the gain as modest and point to high-quality knowledge injection during SFT as significant.Source: §5.3 · p. 9

6 RELATED WORK

Related work covers preference-based training-time alignment, inference-time strategies, and weak-to-strong supervision. WSPO is distinguished as using weak-model supervision for alignment while aiming to preserve the stronger model’s original ability.

6.1 TRAINING-TIME ALIGNMENT

  • Training-time alignment includes RLHF with PPO, DPO, and newer methods including IPO, KTO, and ORPO.RLHF uses a reward model and reinforcement learning; DPO directly trains on human preferences without a reward model.Source: §6.1 · p. 10

6.2 INFERENCE-TIME ALIGNMENT

  • Inference-time approaches include Best-of-N reward selection, Emulated Fine-Tuning, DeRa, and log-probability-difference guidance for frozen large models.The cited work describes these strategies as balancing diversity and coherence or improving alignment, sometimes with inference-time computational trade-offs.Source: §6.2 · p. 10

6.3 WEAK-TO-STRONG GENERALIZATION

  • Unlike weak-supervision approaches for general capabilities or reasoning, WSPO uses weak-model supervision to improve alignment while maintaining the strong model’s original ability.Burns et al. (2023) report that strong models fine-tuned by weak supervisors outperform their weak counterparts; Yang et al. (2024b) use weak supervision to refine reasoning data.Source: §6.3 · p. 10

7 DISCUSSION

The discussion presents WSPO as transferring alignment from weaker to stronger models through pre- and post-alignment distribution differences. It reports benchmark improvements while identifying untested transfer settings and an unexplained amplification effect.

7 DISCUSSION

  • WSPO transfers alignment capabilities from a weaker model to a stronger one using distributional differences before and after weak-model alignment.The paper describes its benchmark results as offering an efficient alternative to traditional alignment methods.Source: §7 · p. 10
  • The study did not test transfer across different language-model architectures or the impact of weak-model alignment strength, and it does not explain the amplification effect.Future work could investigate weak models as reward models in reinforcement-learning frameworks or explain the amplification phenomenon.Source: §7 · p. 10

ETHICS STATEMENT

The ethics statement notes that dataset evaluation was not in-depth and that the WSPO loss did not account for safety or honesty. Reproducibility details specify the training hardware and implementation approach; an appendix lemma states an equivalence between reward functions and optimal policies.

ETHICS STATEMENT

  • The study did not deeply evaluate its open-source datasets or account for safety, honesty, and other considerations in designing the WSPO loss.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • Training experiments used 8×H100 GPUs, and WSPO implementation can modify DPO training code to calculate its loss on a selected dataset and load weaker models.The reported setup uses LLaMAFactory.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 PROOF OF THEROM1

  • Lemma 2 states that reward functions in the same equivalence class induce the same optimal policy under the constrained RL problem.Source: §A.1 · p. 14

A.2 PROOF OF PROPOSITION

The section frames fine-tuning as KL-constrained reinforcement learning and derives WSPO from the difference between a weak model before and after alignment. Experiments describe WSPO training signals for length reward and single-turn dialogue, with WSPO achieving the best reported alignment effect in the dialogue comparison.

A.2 PROOF OF PROPOSITION

  • Any fine-tuned model can be viewed as solving a KL-constrained RL problem relative to its pre-trained model.The section uses this result to define a composite reward that combines base-to-SFT and SFT-to-aligned changes.Source: §A.2 · p. 14

A.3 DERIVING THE WSPO OBJECTIVE

  • WSPO treats the aligned weak model as a hidden reward model and learns the distributional difference between the weak model before and after alignment.The derivation also notes that the strong model’s output must remain a valid distribution.Source: §A.3 · p. 15

A.4 DERIVING THE GRADIENT OF WSPO OBJECTIVE

  • The WSPO gradient derivation uses the probability chain rule and differentiates each factor in the resulting product with respect to the model parameters.Source: §A.4 · p. 15

B EXPERIMENTAL SETUPS

B.1 LENGTH REWARD

  • Qwen2-1.5B learns the PPO length-reward signal, while Qwen2-7B learns the WSPO signal from the aligned-versus-base Qwen2-1.5B probability difference.The WSPO setup uses γ = 0.5; this summarization task is described as requiring no additional model knowledge.Source: §B.1 · p. 16, Figure 5 · p. 16

B.2 SINGLE-TURN DIALOGUE

  • For single-turn dialogue, DPO win rates are higher at β = 0.5 or 1, and WSPO is trained with γ = 0.1 and reported to converge rapidly.The comparison with WSPO uses β = 0.5; GPT-4o-mini evaluates win rates, with a second judge yielding consistent results.Source: §B.2 · pp. 17–18, Figure 6 · p. 17, Figure 7 · p. 18
  • WSPO achieves the best reported alignment effect in single-turn dialogue, with GPT-4o-mini and Qwen2.5-72B-Instruct judgments consistent.The comparison is at temperature 0.7; the supplied passages do not provide the win-rate values.Source: §B.2 · p. 18, Table 3 · p. 18

B.3 A COMPLEX EVALUATION

In the complex evaluation, DPO hyperparameter changes on Qwen2-7B-Instruct did not improve Arena-Hard alignment, while WSPO results were reported as strongest across the cited evaluations. The section also describes the training data, evaluation protocols, and a Llama-family experiment.

DATA AND MODEL SCOPE

  • The evaluation uses 208k UltraChat-200k items for SFT and 64K UltraFeedback items for preference learning, and also reports experiments across Llama-family benchmarks.The Llama experiment aligns Llama3.1-8B using Llama3.2-1B as the weak model.Source: §B.3 · pp. 19–20, Table 7 · p. 20

DPO AND WSPO RESULTS

  • DPO hyperparameter tuning on Qwen2-7B-Instruct did not improve alignment and performed worse than its original Arena-Hard win rate of 39.70.The authors suggest Ultrafeedback may have harmed the model’s existing RLHF alignment or overlapped with its aligned data; the reward rose toward 1 without further alignment gains.Source: §B.3 · p. 19, Table 4 · p. 19

WSPO TRAINING

  • WSPO training uses the aligned-versus-SFT models’ logarithmic probability difference, and its loss is reported to decrease and converge gradually.The specified setup uses batch size 32, three epochs, and γ = 0.1.Source: §B.3 · p. 19

EVALUATION

  • On Arena-Hard, WSPO achieves the best reported alignment effect, while GPT-4o-mini and Qwen2.5-72B evaluation results are consistent; Table 6 reports win rates compared with the baseline.Source: §B.3 · p. 20, Table 6 · p. 20

C IMPACT OF DATASET

The experiments report that WSPO can use a rejected preference-data subset and can adjust alignment strength through γ, including when the weak model is not weak. The results support dataset flexibility within the tested settings.

C IMPACT OF DATASET

  • WSPO trained on the rejected preference-data subset is reported to remain effective, supporting use of datasets beyond preferred examples.The rejected subset may contain toxic content and is used to capture the predicted distribution difference; Table 8 reports Arena-Hard comparisons.Source: C IMPACT OF DATASET · p. 21, Table 8 · p. 21

D WHEN THE WEAK MODEL IS NOT WEAK

E VISION LANGUAGE TASK

The paper reports WSPO applications to vision-language alignment and compares its resource and alignment trade-offs with other methods. It also includes example generations and identifies PPO’s continued role in exploration beyond the dataset.

E VISION LANGUAGE TASK

  • WSPO is reported to apply to vision-language alignment, using a 2B model to align a 7B model and evaluating Qwen2-7B-VL on Arena-Hard and MMHal-Bench.The experiments use the RLHF-V image-text preference dataset; the supplied passages do not give the table values.Source: E VISION LANGUAGE TASK · p. 21, Table 10 · p. 22, Table 11 · p. 22

F EFFICIENCY ANALYSIS

  • WSPO requires loading two weak models, slightly increasing memory and computation, but does not rely on a large preference dataset.Source: F EFFICIENCY ANALYSIS · p. 22

F.1 COMPARISON TO SIMPO

  • The authors report that WSPO can adjust alignment strength from an already-aligned SimPO model, a feature they say SimPO lacks.They also characterize WSPO’s hyperparameter settings as intuitive and easy to understand.Source: §F.1 · p. 22, Table 12 · p. 22, Table 13 · p. 23

F.2.1 TRAINING A REWARD MODEL:

  • The reward-model training stage has roughly equivalent computational requirements for PPO and WSPO.The appendix describes using the trained reward model to train both methods and gives the training-time comparison in Table 14.Source: §F.2.1 · p. 23, §F.2.2 · p. 23, Table 14 · p. 23

F.2.3 TRADE-OFF ANALYSIS:

  • WSPO optimization can serve as a precursor to PPO and may accelerate training, while PPO remains useful for exploration beyond the dataset.The paper characterizes PPO as computationally intensive and potentially unstable, but robust.Source: §F.2.3 · p. 23

G EXAMPLE GENERATIONS

Related papers

Loading 2410.18640v2…