PaperlayerView exact revision on arXiv

Source-linked AI summary

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun

arXiv:2609.08798v1cs.LGcs.CL

TL;DR

Stronger models may learn from weaker supervisors yet exceed them, but conventional distillation can make the weak policy the student’s target. OPRD instead uses the teacher’s post-training policy shift to rescale verifier-driven student updates, achieving faster and stronger learning across weak-to-strong and strong-to-weak settings.

  • Problem

    Weak-to-strong generalization needs supervision that transfers weaker models’ post-training gains without imposing their final policies and capacity limits on stronger students.

  • Method

    OPRD evaluates the teacher’s policy shift relative to its reference policy on student rollouts and amplifies the aligned component of the student’s verifier-driven policy gradient.

  • Results

    33–67% fewer student updates reach weak-teacher performance than GRPO in successive transfer, while multi-teacher OPRD reaches teacher-level performance with 55% fewer updates than Mix-RL and continues beyond teacher plateaus.

  • Takeaways & Limitations

    OPRD accelerates the student’s own verifier-based optimization rather than redirecting it toward the weak teacher, and remains effective when teacher and student capacities are reversed.

  • Takeaways & Limitations

    OPRD requires a nonzero verifier-driven policy gradient; when rollout groups receive identical rewards, its teacher-guided correction vanishes.

Abstract

from arXiv · show

Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Response-style analysis shows that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the student's own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance regardless of capacity ordering.

1. Introduction

Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors while exceeding them, especially for successor transfer and multi-domain consolidation. OPRD addresses the capacity-ceiling problem by using the teacher’s post-training shift to accelerate verifier-driven student optimization rather than matching the weak policy.

  • Motivation: Standard on-policy distillation trains on student-generated responses but optimizes the student toward the teacher policy.
  • Motivation: Weak-to-strong generalization studies whether stronger models can learn from weaker supervisors and outperform them in successor transfer and multi-domain consolidation.
  • Method: OPRD extracts the weak teacher’s policy shift relative to its reference policy and amplifies the aligned component of the student’s verifier-driven policy gradient.
  • Takeaway: The paper reports that OPRD preserves policy-optimization stationary points while accelerating learning beyond weak teachers.
  • Results: 33–67% fewer student updates let OPRD reach weak-teacher performance than GRPO in successive transfer, while early performance improves by up to 22.7 percentage points.
  • Results: OPRD reaches teacher-level performance with 55% fewer updates than Mix-RL in multi-teacher distillation and ultimately outperforms all four specialists.

2. Method

OPRD transfers the teacher’s post-training policy change without making the teacher policy the student’s target. It rescales only the verifier-supported gradient component aligned with that change, preserving the student’s stationary points while enabling both teacher-aligned learning and verifier-supported departures.

  • Preliminaries: RLVR computes a verifier-based policy gradient on student-sampled responses and visited prefixes, while OPD supplies dense teacher supervision on that same student state distribution.
  • Preliminaries: OPD directly optimizes toward the teacher policy, whereas OPRD uses the teacher’s learned policy change only to rescale the student’s own gradient.
  • Teacher Policy Shift: OPRD mean-centers the teacher–reference logit difference to isolate changes in relative next-token preferences and then retains its unit direction.
  • Gradient Scaling: OPRD decomposes the student gradient into its projection onto the teacher-informed direction and an orthogonal component, amplifying only the projection by 1 + λ_t.
  • Optimization Properties: Because the transformation only rescales existing verifier gradients, OPRD preserves stationary points and adds a nonnegative first-order alignment gain.
  • Asymmetric Alignment Scaling: Positive alignment reinforces teacher-following updates, while negative alignment strengthens verifier-supported departures; a warm-up gradually activates the negative branch.

3. Experiments

Experiments evaluate OPRD on mathematical and logical reasoning in successive transfer, multi-teacher consolidation, and strong-to-weak distillation. Across these settings, OPRD combines rapid teacher-level transfer with continued improvement beyond distillation plateaus and remains effective across capacity orderings.

  • Experimental Setup: The experiments compare OPRD with verifier-only, distillation-only, and hybrid baselines under matched student checkpoints, prompts, rollout budgets, and policy updates.
  • Successive Model Transfer: In successive transfer, OPRD matches OPD’s early acceleration, surpasses the weak teacher, and reaches GRPO’s end-of-training performance substantially earlier.
  • Successive Model Transfer: 7.92 points on mathematics and 10.80 points on Reasoning Gym separate OPRD from the strongest baseline when results are averaged over five checkpoints.
  • Multi-Teacher Consolidation: 58.77 average Pass@1 exceeds Mix-RL by 11.09 points and the specialist average by 14.12 points in multi-teacher consolidation.
  • Multi-Teacher Consolidation: OPRD surpasses the corresponding specialist on all four tasks, indicating joint improvement rather than a cross-task tradeoff.
  • Strong-to-Weak Distillation: 3.79 points on AIME'24 and 29.20 points on Knights & Knaves are OPRD’s gains over OPD in strong-to-weak distillation.

4. Analysis

The analysis compares OPRD’s guidance directions, amplification strength, training dynamics, teacher quality, and failure modes. It finds that OPRD accelerates verifier-driven learning beyond weak teachers, while reference-policy choice and vanishing policy gradients define important boundaries.

  • Broader comparison: 60.81 average performance exceeds W2S-OPD by 11.59 points and Direct-OPD by 23.00 points.OPRD retains the verifier-gradient component orthogonal to the transferred policy shift, unlike methods relying only on shift information.
  • Teacher quality: Later, better-trained weak teachers generally accelerate learning more, but a step-60 teacher at 29.0% Pass@1 still guides a student to approximately 88%.Because the policy shift is normalized, the benefit is attributed to more reward-informative direction rather than shift magnitude alone.
  • Scaling direction: The weak policy delta provides the fastest and most sustained gains among the evaluated scaling directions.Comparisons include OPD teacher matching and OPSD self-distillation; the delta isolates the teacher’s post-training change relative to its reference.
  • Amplification strength: Every λ>0 improves final Pass@1 over GRPO, while values up to 0.5 accelerate early learning and larger values add little.The default directional-amplification coefficient is λ=0.5.
  • Training dynamics: Teacher guidance is strongest early, then becomes less influential as the guidance direction and verifier gradient approach orthogonality.The projected-gradient fraction is |cos θ_t|, so convergence toward 90° reduces the component available for amplification.
  • Failure modes: When verifier-driven policy gradients vanish, OPRD has no gradient component to amplify; in the extreme setting, all compared verifier-based methods remain below 20% Pass@1.OPD reaches 40.5% with dense policy-matching targets, and a short SFT or distillation warm-up is proposed as a possible bootstrap.
  • Reasoning paths and style: OPRD can preserve the stronger student’s reasoning style while using the teacher shift as an alternative route to the same correct answer.In the illustrated AIME'25 case, the student’s direct token 0 and teacher-shift token 5 both lead to 2016, with the latter taking an intermediate-sum route.

5. Related Work

Related work addresses distillation, weak-to-strong transfer, policy-shift transfer, and gradient manipulation through different optimization targets or training signals. OPRD instead preserves the student’s verifier-driven gradient and amplifies only its component aligned with the weak teacher’s policy shift.

  • Weak-to-strong generalization: Weak-to-strong methods transfer post-training gains in successive model transfer and multi-domain consolidation, but often alter training data, exploration, or feedback.OPRD is positioned as leaving those three elements unchanged while rescaling the student’s own policy gradient.
  • On-policy distillation: On-policy distillation queries teachers on student rollouts but still trains students toward teacher-defined token distributions or targets derived from them.This makes the weak policy itself, or a target derived from it, the objective optimum in the weak-to-strong setting.
  • Distillation with reinforcement learning: Distillation with reinforcement learning incorporates teacher guidance through reward terms, policy ratios, selection, calibration, or token-level interventions.These approaches differ in where and how the teacher signal enters verifier-based optimization.
  • Transferring policy shifts: Policy-shift transfer uses the difference between a post-trained policy and its reference as a decoding steer, alignment target, proxy teacher, or dense reward.OPRD instead uses that shift as a direction for modifying the student’s verifier gradient.
  • Gradient manipulation: Gradient-manipulation methods project, remove, or down-weight conflicting task-gradient components, whereas OPRD only amplifies the aligned component and removes nothing.This preserves the stationary points of the student objective in logit space.

6. Conclusion

OPRD accelerates verifier-driven optimization by amplifying teacher-aligned gradient components without making the teacher policy the student’s target. The method preserves stationary points, transfers policy shifts efficiently, and remains subject to limitations in task coverage, scale, and alignment quality.

  • Contribution: OPRD amplifies the aligned component of a stronger student’s policy gradient without making the weak teacher an optimization target.It transfers the teacher’s post-training policy shift while retaining the student’s own verifier-driven objective.
  • Empirical conclusion: OPRD reaches teacher-level performance in fewer updates, continues improving after on-policy distillation plateaus, and remains closer in response style to reward-only training than to the teacher.The reported conclusion covers successive transfer, multi-domain consolidation, and both capacity orderings.
  • Scope and future work: Broader evaluations in code generation and agentic environments are needed to test transfer under execution- or environment-based feedback.These settings involve sequential interactions in which early actions affect later observations and rewards.
  • Scope and future work: The evaluation covers Qwen3 models from 0.6B to 8B parameters, while larger-scale experiments remain necessary to test transfer efficiency and improvement beyond the teacher.The paper identifies larger models as an open setting rather than reporting results there.
  • Optimization properties: The transformed update preserves the unscaled policy objective’s stationary points while adding a nonnegative ascent term under a bounded step size.The correction amplifies the component along the teacher direction and leaves the orthogonal component unchanged.
  • Caveat: Positive-only scaling can amplify incidental or reward-irrelevant alignment because neither the student gradient nor teacher shift uniquely identifies reward-improving token changes.Shared tendencies such as response-length effects may therefore receive amplification alongside useful progress.

B.2. Isolating the Positive and Negative Alignment Branches

Branch-isolation experiments show that positive alignment provides rapid early transfer but can lengthen responses sharply, whereas negative alignment is initially less reliable. Scheduling positive and negative amplification over training combines early bootstrapping with later verifier-supported departures from the teacher.

  • Branch-specific effects: Positive-only scaling produces rapid early gains on mathematics and Knights & Knaves, accompanied by a sharp increase in response length.The branch-isolation comparison holds other training settings fixed within each task.
  • Branch-specific effects: Negative-only scaling collapses early, suggesting that teacher-opposing components are less reliable while the student’s rollouts remain weak.Response-level feedback can include locally unhelpful token choices, making early opposing components difficult to trust.
  • Branch-specific effects: Positive-only scaling reinforces a shared tendency toward longer reasoning traces that may reflect useful reasoning or length-dependent policy-gradient effects.The student and teacher can encode similar response-length tendencies during verifier-based post-training.
  • Branch scheduling: The positive and negative branches are complementary: positive alignment supports rapid early transfer, while negative alignment can amplify verifier-supported departures that help surpass the weak teacher.The latter becomes more useful as the student produces stronger and more informative on-policy rollouts.
  • Branch scheduling: The default λ− ramp-up begins with positive-branch bootstrapping and progressively increases amplification of gradient components opposing the teacher direction.This schedule is designed to avoid persistent one-sided amplification.
  • Branch scheduling: Scheduled variants achieve rapid early gains, whereas fixed symmetric scaling improves more slowly despite using λ=1.0.The comparison includes λ+ annealing, fixed symmetric scaling, and GRPO across Math and Knights & Knaves.
  • Evaluation setting: The successive-transfer evaluation uses 4B teachers and 8B students across four mathematics benchmarks and four Reasoning Gym tasks.OPRD generally retains OPD’s rapid early improvement while continuing beyond the weak-teacher plateau.

D.2. Additional Results Across Model Variants and Tasks

Across instruction-tuned and base Qwen3 models, additional reasoning tasks, and checkpoint-averaged evaluations, OPRD consistently accelerates weak-to-strong learning. Its gains remain substantial when response length decreases, though teacher-shift length bias requires reference-policy adjustment.

  • Additional model variants: OPRD achieves high performance much earlier than baselines across all four additional Qwen3-Base mathematics benchmarks.The experiments use Qwen3-4B-Base teachers and Qwen3-8B-Base students.
  • Additional tasks: OPRD improves substantially faster than baselines on three additional Reasoning Gym tasks while reducing response length.The experiments use student rollouts and compare against weak-teacher performance; later teacher checkpoints address length bias for two tasks.
  • Checkpoint-averaged results: OPRD achieves the strongest checkpoint-averaged results across the additional mathematics and Reasoning Gym settings.Table 7 summarizes Mean@16 for mathematics and Pass@1 for Reasoning Gym across multiple training checkpoints.
  • Evaluation robustness: Response-resampling evaluation holds benchmark problems and trained checkpoints fixed while estimating variability from repeated response samples.The procedure uses 1,000 replicates, sampling 16 responses per mathematics problem and one per Reasoning Gym problem.

E. Detailed Results for Multi-Teacher Weak-to-Strong Distillation

In multi-teacher distillation, one Qwen3-8B-Base student is jointly trained on domain-mixed batches paired with task-specific Qwen3-4B-Base teachers. OPRD transfers specialist capabilities across all four domains while continuing to improve without the comparable cross-task degradation observed for MOPD.

  • Multi-teacher learning: OPRD attains the highest Pass@1 across all four Reasoning Gym domains under multi-teacher distillation.The student is trained jointly on domain-mixed batches, with each example paired to its task-specific teacher.
  • Cross-task effects: OPRD rapidly transfers specialist capabilities and continues improving without comparable degradation from cross-task interference.MOPD shows the clearest interference on Quantum Lock, where it falls below the corresponding specialist teacher.

F. Detailed Results for Strong-to-Weak Distillation

OPRD remains effective when the student is weaker than the teacher, combining verifier-driven optimization with teacher guidance rather than merely matching the teacher. Across strong-to-weak and off-policy comparisons, it produces faster or stronger learning than the evaluated alternatives.

  • Strong-to-weak results: On AIME'24, OPRD raises Qwen3-1.7B Mean@16 from 10.0 to above 41 within 150 updates, while GRPO reaches about 25.GRPO reaches 35 only after 240 updates in the same comparison.
  • Strong-to-weak results: On Knights & Knaves, OPRD ultimately surpasses the Qwen3-8B-Base teacher’s Pass@1 of 64.5 with a Qwen3-0.6B student.OPD improves initially but later collapses and remains below the teacher after recovering.
  • Experimental setup: The comparisons use matched model pairs, teacher checkpoints, prompts, and evaluation protocols unless method-specific settings differ.Baseline implementations include SFT, rollout schedules, KL regularization, privileged rollouts, and restricted-support reverse KL.
  • Off-policy comparisons: OPRD exhibits the strongest and most consistent gains among methods using off-policy generations from the weak teacher.W2SR-P slightly exceeds weak-teacher performance through SFT, while S2L-PO remains competitive on two Reasoning Gym tasks but deteriorates on AIME'24 after annealing.
  • Policy-delta comparisons: OPRD improves beyond teacher-level saturation because the weak policy delta guides verifier-driven optimization instead of replacing its update direction.W2S-OPD can surpass the weak teacher but ultimately plateaus near teacher-level performance.

H. Additional Results on Guidance-Direction Construction

Guidance-direction experiments compare the weak policy delta with directions based on teacher–student mismatch and privileged-context discrepancy. The weak policy delta performs best in the main comparison and remains slightly ahead of OPD with stronger teacher checkpoints, while avoiding dependence on the evolving student or a privileged draft.

  • Guidance-direction constructions: The weak policy delta outperforms OPD and OPSD by a wide margin in the step-60 teacher comparison.The delta isolates the change acquired during teacher post-training, whereas OPD and OPSD use different discrepancy constructions.
  • Stronger teachers: With stronger teacher checkpoints, OPD performs well on mathematics and Knights & Knaves but remains slightly behind the weak policy delta overall.The step-105 Knights & Knaves teacher reaches 57.5% Pass@1, compared with 29.0% at step 60.
  • Figure setup: Figure 16 compares Delta, OPD, and OPSD guidance constructions using stronger Qwen3 teacher checkpoints on Math and Knights & Knaves.The figure marks teacher performance with gray dashed lines and follows Appendix C training configurations.
  • Default guidance direction: The weak policy delta is retained as the default because it isolates teacher post-training changes without relying on the evolving student or a privileged draft.The paper identifies OPD’s usefulness as teacher-capability-dependent and OPSD as vulnerable to restrictive reasoning paths.

I. Additional Results on Length Bias in Teacher Policy Shift

Reference-policy selection determines whether OPRD amplifies a harmful length shift or preserves useful teacher guidance. In Binary Matrix, moving the reference past the sharp shortening transition avoids continued response shortening and substantially improves Pass@1.

  • Reference-policy selection: The teacher’s mean response length falls sharply between steps 30 and 45, motivating comparisons between step-0 and step-45 references.The step-0 reference includes the full post-training shift, whereas the later reference excludes most of the shortening component.
  • Results: Both OPRD variants initially improve faster than GRPO, but their outcomes diverge after step 90.The step-0 variant continues shortening responses, while the step-45 variant avoids the same decline.
  • Results: 81.5% Pass@1 with the step-0 reference trails GRPO and KDRL as continued shortening overemphasizes length reduction.The reference includes a large shortening component in the teacher policy shift.
  • Results: 96.0% Pass@1 with the step-45 reference shows that placing the reference after the sharp length transition mitigates bias while retaining later task progress.Changing the reference modifies the entire policy shift, so isolating structured bias from task-relevant guidance remains open.

J.2. Response Style Analysis

The response-style analysis compares OPRD and OPD with a GRPO-trained student and a weak teacher across 101 standardized style features. OPRD remains closer to the GRPO student throughout training, whereas OPD shifts toward the teacher.

  • Analysis setup: The analysis represents responses with 101 standardized style features spanning connectives, modality, grammar, punctuation, and structure.The evaluated checkpoints, teacher, and GRPO student each contribute response sets for comparison across 30 AIME’24 problems.
  • Distribution comparison: Energy-distance differences use positive values for greater similarity to the GRPO student and negative values for greater similarity to the teacher.The averages cover 30 problems, with 95% confidence intervals from 2,000 bootstrap resamples.
  • Distribution comparison: OPRD remains closer to the GRPO student, with its mean difference increasing from +0.316 at update 30 to +0.447 at update 150.The same trend holds at every evaluated checkpoint.
  • Category analysis: OPRD is closer to the GRPO student in every style category at all evaluated checkpoints, while OPD shifts from student-proximal to teacher-proximal across all five categories.The category score is normalized by the separation between teacher and GRPO-student averages.

K. Computational Cost and Memory Usage

OPRD adds computational cost mainly through evaluating the teacher’s reference policy and increases memory through a higher baseline footprint. Its correction operation itself contributes little additional time and remains within the update peak.

  • Wall-clock time: 11.9% overall wall-clock overhead relative to GRPO results from OPRD evaluating both the teacher and its reference policy.Relative to OPD, the additional reference evaluation raises total time by 4.4%; the projection and scaling update adds only 2.79 seconds, or 0.25% of the full step.
  • Wall-clock time: 602 seconds of rollout generation accounts for 61.6% of total GRPO step time, making it the dominant computational component.OPD and KDRL add 7.2% and 7.9% overheads over GRPO, respectively.
  • Peak GPU memory: OPRD carries approximately 14 GiB per-GPU memory overhead relative to GRPO and 13 GiB relative to OPD and KDRL.Its higher rollout peak reflects a higher starting footprint rather than greater rollout allocation.
  • Peak GPU memory: 151.27 GiB is OPRD’s overall maximum memory use, exceeding GRPO by 13.98 GiB and OPD and KDRL by 13.21 GiB.The peak occurs during rollout under the fixed 16,384-token benchmark setup.
  • Peak GPU memory: The correction hook’s 9.43 GiB allocation is contained within OPRD’s 115.19 GiB update peak, while sparse support occupies at most 1.38 MiB.Direct gather and sparse scatter avoid an additional 9.27 GiB response-by-vocabulary copy.
Loading 2609.08798v1…