PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-to-Strong On-Policy Distillation

Fangxu Yu, Weijia Xu, Michael Xu, Tianyi Zhou, Zinan Lin

arXiv:2607.26246v2cs.LG

TL;DR

Existing on-policy distillation assumes a teacher at least as capable as the student, leaving frontier models without a stronger source. W2S-OPD constructs a proxy teacher from capability differences between weaker models and improves strong students across math and code benchmarks, including surpassing a weak teacher.

  • Problem

    On-policy distillation assumes a teacher at least as capable as the student, limiting supervision options as models approach the frontier.

  • Method

    W2S-OPD transfers capability directions from contrast pairs of weaker models by injecting their logit difference into the student’s base model for on-policy distillation.

  • Results

    Across math and code benchmarks, W2S-OPD outperforms OPD, surpasses a domain teacher, and improves the student even when all supervision sources are weaker.

  • Takeaways & Limitations

    W2S-OPD supports continued strong-student improvement from smaller or weaker models using signals from RL training, model scale, or contrastive hints.

  • Takeaways & Limitations

    How far weak-to-strong supervision can improve a stronger student before saturating, and how to elicit more informative signals, remains unresolved.

Abstract

from arXiv · show

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.

1 INTRODUCTION

W2S-OPD addresses the lack of stronger teachers by constructing a capability-focused proxy teacher from contrasts between weak models and distilling it on the strong student’s own rollouts. It transfers capabilities from reinforcement learning, model scale, and solution hints while improving beyond weaker supervision sources and the domain teacher.

  • Motivation: On-policy distillation combines dense token-level supervision with student-generated trajectories, but weak-to-strong learning is limited by teacher–student distribution mismatch and potential capability erosion.RLVR provides sparse outcome supervision, whereas off-policy distillation suffers from exposure bias; OPD combines their complementary strengths.
  • Method: W2S-OPD forms a proxy teacher by adding the positive–negative models’ logit difference to the student’s base model, isolating a capability direction while remaining distributionally adjacent to the student.The student generates on-policy rollouts and minimizes per-token reverse KL toward this proxy teacher.
  • Method: The framework instantiates contrasts using pre-RL versus post-RL models, larger versus smaller base models, and correct versus wrong hints to transfer distinct capability signals from weak models.The pre-RL/post-RL contrast isolates RL-instilled skill, while the other contrasts isolate scale- and instance-level directions.
  • Results: 11.4% and 12.0% relative gains over OPD occur on math reasoning under single- and multi-teacher distillation, while W2S-OPD also lifts the student above the domain expert.These results are reported for the pre-RL/post-RL setting.
  • Results: W2S-OPD improves the student when every supervision source is weaker and reframes weak-to-strong learning as transferring capability directions rather than imitating weak supervisors.Post-RL and hint contrasts emphasize reasoning frameworks, whereas the scale contrast emphasizes the solving procedure.

2 PRELIMINARY: ON-POLICY DISTILLATION

On-policy distillation provides dense token-level supervision on student-generated trajectories, using reverse-KL teacher feedback on the states the student actually visits. Its established uses compress stronger teachers or consolidate same-origin experts, both assuming supervision at least as capable as the student.

  • Core formulation: OPD lets the student generate on-policy rollouts while a frozen teacher scores every token, bridging reinforcement learning and imitation learning.Unlike off-policy distillation from fixed teacher-generated data, OPD supplies dense supervision on the student’s own trajectories.
  • Core formulation: Reverse KL, KL(πS ∥ πT), provides per-token divergence, while Top-k OPD improves computational efficiency.Because states come from student rollouts, teacher feedback targets states the student actually visits and avoids exposure bias.
  • Established uses: Strong-to-weak distillation compresses a larger teacher into a smaller student, whereas same-origin consolidation combines several RL-trained domain experts into one model.In same-origin consolidation, experts share the student’s base initialization so their distributions remain close.
  • Limitation: Both established OPD uses presuppose a teacher at least as capable as the student, limiting their effectiveness when training lacks such a teacher.The passage identifies this capability assumption as a shared premise of strong-to-weak distillation and same-origin consolidation.

3 WEAK-TO-STRONG ON-POLICY DISTILLATION

W2S-OPD improves a strong student from weaker models by extracting a transferable capability direction from a positive–negative contrast pair and re-anchoring it on the student’s base model. This proxy teacher addresses weak-to-strong distributional mismatch and capability degradation, while supporting several contrast sources and multiple injected directions.

  • Core idea: W2S-OPD addresses weak-to-strong distillation’s distributional mismatch and capability degradation by transferring a capability direction rather than directly imitating a weaker policy.The method isolates the direction separating stronger positive and weaker negative models, then re-anchors it onto the student’s base model.
  • Proxy teacher construction: The proxy teacher adds the positive–negative logit difference to the strong base model, using α ≥0 to trade off injected signal strength against distributional proximity.Small α keeps the proxy close but may provide too little supervision; large α strengthens the signal but risks distributional distortion.
  • Contrast-pair instantiations: Pre-RL versus post-RL contrasts isolate domain skills acquired through reinforcement learning while reducing the cost of training domain experts at the student’s scale.The positive model is an RL-trained small base model, and the negative model is its pre-RL initialization.
  • Contrast-pair instantiations: Smaller-versus-larger base-model contrasts isolate scale-emergent capability without additional training, using released-model signals to improve frontier models.An example pair is Qwen3-4B and Qwen3-0.6B.
  • Contrast-pair instantiations: Correct-versus-wrong hint contrasts cancel shared hint-induced style shifts and isolate the instance-level direction toward the correct solution using one small model and a reference solution.The positive and negative models share the same base model and hint format, differing only in whether the solution hint is correct or wrong.
  • Multiple proxy teachers: Multiple positive–negative pairs can inject distinct capability directions into one shared base model, enabling a single distillation run to merge multiple domain skills.The coefficients αk control each direction’s strength, and binary settings such as αk ∈ {0, 1} can route queries to suitable model pairs.

4 EXPERIMENTS

Experiments evaluate W2S-OPD across three contrast-pair settings, showing consistent gains over OPD and improvements from weaker supervision sources. Further analyses examine training dynamics, out-of-domain transfer, and the distinct reasoning signals emphasized by each contrast.

  • Main results: Across pre-RL/post-RL, smaller/larger, and correct/wrong-hint contrasts, W2S-OPD improves the strong student using weaker supervision sources.The pre-RL/post-RL setting outperforms OPD across benchmarks; the smaller/larger and hint settings improve the student by 6.0% and 1.4% on math, and 1.2% and 1.1% on code, respectively.
  • Training dynamics: W2S-OPD outperforms OPD throughout training and improves faster on the math and code benchmarks.Figure 3 tracks benchmark performance over training steps and reports the comparison throughout training.
  • Amplification coefficient: A moderate amplification coefficient α best balances injected capability strength against keeping the proxy teacher distributionally adjacent to the student.Small α produces weak signals, whereas excessively large α makes supervision harder to learn as the proxy teacher moves farther from the student distribution.
  • Out-of-domain generalization: On GPQA-Diamond, W2S-OPD lifts the student from 38.9 to 56.5 and outperforms OPD by an absolute 2.1%.The distilled reasoning skill transfers beyond the math training domain; the models diverge on IFBench.
  • Learning signals: Post-RL and hint contrasts emphasize reasoning-framework episodes, while the scale contrast emphasizes core solving steps.Relative to the scale contrast, post-RL and hint contrasts place more weight on Plan and Monitor, whereas scale places more weight on Analyze and Implement.

5 RELATED WORK

W2S-OPD builds on on-policy distillation, weak-to-strong generalization, and controllable text generation. It adapts these lines of work to improve stronger models using weaker supervision and decoding-time distribution composition.

  • On-Policy Distillation: On-policy distillation supervises students on their own rollouts with a superior teacher and supports strong-to-weak transfer and multi-domain expert merging.Follow-up work relaxes OPD’s requirements through self-distillation from privileged information.
  • Weak-to-Strong Generalization: Weak-to-strong generalization elicits stronger-model capabilities from weak supervision, extending to LLM reasoning and alignment when stronger models are difficult to obtain.A related learning-by-teaching line has strong models instruct weaker students.
  • Controllable Text Generation: Controllable text generation uses training-based finetuning or decoding-time steering and distribution composition; W2S-OPD adopts the latter approach.Training-based methods can be computationally costly, whereas decoding-time methods shift or compose output distributions during inference.

6 CONCLUSION

W2S-OPD improves a strong student using only smaller or weaker models by constructing a capability-directed proxy teacher from a contrast pair. The conclusion positions weak-to-strong supervision as promising but leaves its saturation limits and broader development open.

  • Conclusion: W2S-OPD improves a strong student using only smaller or weaker models by isolating a capability direction and re-anchoring it onto the student’s base model.This synthesizes a proxy teacher that couples the isolated capability while remaining distributionally adjacent to the student.
  • Discussion: Weak supervision is abundant, but its saturation limits and the elicitation of more informative signals for stronger students remain open questions.Future work may develop weak-to-strong learning into a sustained paradigm for on-policy distillation and post-training more broadly.

A KL-DIVERGENCE ESTIMATION

The paper adopts a dense top-K OPD objective that preserves teacher supervision nearly losslessly while reducing communication cost, rather than using a higher-variance single-token estimator.

  • Single-token OPD: Single-token OPD conveys one scalar per position and has higher variance than the dense top-K objective.The single-token log-ratio is treated as a token-level advantage and used in a clipped policy-gradient update.
  • Objective choice: The dense top-K objective directly differentiates the divergence at every position without advantage estimation or clipping, so it is used for both distillation arms.This choice also avoids the single-token estimator’s scalar-per-position information bottleneck.
  • Top-K OPD: Top-K OPD transmits only teacher top-K token indices and log-probabilities instead of roughly 150K logits per response position.Because the teacher distribution is highly peaked, its top-K tokens carry most probability mass at nearly all positions.
  • Top-K OPD: Top-K truncation preserves supervision nearly losslessly because its error is bounded by the tiny probability mass outside the teacher’s top-K support.The restricted teacher distribution is renormalized over Kt before computing the divergence.

B FROM LOGIT ARITHMETIC TO A KL-CONSTRAINED OBJECTIVE

The proxy teacher is an exponentially tilted version of the student’s base model, using the contrast pair’s relative capability reward while remaining within a student-centered trust region. This formulation composes multiple capability directions through product-of-experts multiplication and merges them in one distillation run.

  • Logit arithmetic: The contrast reward r(s, a) = log π+(a | s) − log π−(a | s) measures relative token-level preference, independent of either model’s absolute competence.The resulting logit construction is an exponential tilting of the student’s own base model.
  • KL-constrained objective: The proxy teacher uniquely maximizes a KL-constrained objective that injects the contrast capability while staying adjacent to the student.The trust region is centered on the student, with 1/α controlling its strength.
  • Compositional view: The proxy teacher is a product of experts: the student base anchors the distribution, π+ contributes as a conjoined expert, and π− enters as a negated expert.The exponent α controls how sharply the contrast pair is applied, while the trust region keeps the product near the anchor.
  • Multiple capability directions: With K contrast pairs, capability directions combine by multiplying experts, and the result is invariant to the order in which pairs are introduced.The combined reward is expressed as Σ_k α_k r_k, with all K directions merged by a single distillation run.
  • Design choices: Moderate α is optimal: α→0 recovers πbase and provides no supervision, whereas α→∞ removes the trust region and collapses onto arg maxa r(s, a).The reference measure remains the student’s base model, and α determines how far the proxy teacher may move.

C IMPLEMENTATION DETAILS

The experiments use verl, run distillation on two NVIDIA B200 GPUs, and share hyperparameters between W2S-OPD and OPD. The proxy teacher uses a top-K divergence estimate with K = 32, alongside specified GRPO and rollout recipes.

  • All experiments are implemented on top of the verl framework.
  • Each distillation run uses 2 NVIDIA B200 GPUs.
  • The proxy-teacher divergence is estimated on the teacher’s top-K support with K = 32.Table 7 provides shared W2S-OPD and OPD training hyperparameters, while Tables 8 and 9 specify the domain-expert GRPO recipe and OPD rollout instructions.

D ADDITIONAL EXPERIMENTAL RESULTS · D.1 RUNTIME ANALYSIS · D.2 PERFORMANCE WITH DIFFERENT SCALES OF BASE MODELS

Additional experiments show that W2S-OPD adds modest training overhead while improving performance across different base-model scales. The larger capability gap between contrast-pair models produces a larger math-reasoning gain.

  • D.1 RUNTIME ANALYSIS: W2S-OPD forwards three frozen models rather than OPD’s single teacher in the pre-RL / post-RL setting.The models are an 8B anchor and a 4B contrast pair, compared with one 4B teacher for OPD.
  • D.1 RUNTIME ANALYSIS: On-policy rollout generation dominates per-step cost, keeping W2S-OPD’s additional training overhead modest.This explains why forwarding three frozen models results in only a 20% per-step increase over OPD.
  • D.1 RUNTIME ANALYSIS: 20%: W2S-OPD increases average wall-clock time per training step over OPD in the pre-RL / post-RL setting.The comparison is reported in Table 10.
  • D.2 PERFORMANCE WITH DIFFERENT SCALES OF BASE MODELS: Both the Qwen3-4B−0.6B and Qwen3-4B−1.7B contrast pairs improve the fixed Qwen3-8B student.The experiment fixes the student and positive model while varying the negative model.
  • D.2 PERFORMANCE WITH DIFFERENT SCALES OF BASE MODELS: 6.0%: The wider Qwen3-4B−0.6B pair yields the larger improvement on math reasoning.The narrower Qwen3-4B−1.7B pair yields 4.7.
  • D.2 PERFORMANCE WITH DIFFERENT SCALES OF BASE MODELS: 4.7: The narrower Qwen3-4B−1.7B pair improves math reasoning less than the wider Qwen3-4B−0.6B pair.The reported gains increase with the capability gap within the contrast pair.

E EXAMPLE GENERATIONS

The qualitative examples compare W2S-OPD with OPD across math and code settings, showing cases where W2S-OPD succeeds while the baseline fails. Additional student examples illustrate improved mathematical reasoning, while one generated code example remains internally inconsistent.

  • Qualitative comparisons: Across Figures 6–9, W2S-OPD succeeds on the presented tasks while the baseline fails.Figures 6–7 compare W2S-OPD with OPD in the pre-RL/post-RL setting, while Figures 8–9 compare Qwen3-8B before and after W2S-OPD in the smaller/larger setting.
  • Math reasoning: In the math geometry example, the post-distillation Qwen3-8B student provides a detailed coordinate-geometry solution setup for computing FG.The setup places square ABCD in coordinates, parameterizes E on BC, and uses the circumcenter constraints; the cited passage presents the student’s reasoning rather than a final answer.
  • Math reasoning: W2S-OPD gives a parallel coordinate-geometry solution setup that explicitly uses the circumcenter’s equal-distance condition.The passage describes setting up coordinates for the two squares and states that the circumcenter is equidistant from all three vertices.
  • Code generation: The code example is internally inconsistent: its comment claims output 9, whereas strict left-to-right evaluation computes 15.The cited result passage identifies the mismatch between the claimed example output and the computation performed by the code.
Loading 2607.26246v2…