PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

arXiv:2607.05394v2cs.LGcs.AIcs.CL

TL;DR

RLVR becomes increasingly expensive to rerun on stronger models, so this paper transfers RL-induced policy shifts from cheaper weak teachers using Direct-OPD; it improves students across teacher pairs and families, raising Qwen3-1.7B from 48.3 to 58.3 on AIME 2024.

  • Problem

    Rerunning RLVR on each stronger model is costly because every update requires expensive target-model rollouts, risking a post-training bottleneck as models scale.

  • Method

    Direct-OPD treats the post-RL teacher’s log-ratio against its pre-RL reference as a dense implicit reward evaluated on the stronger student’s on-policy states.

  • Results

    Direct-OPD consistently improves stronger students across teacher pairs and model families, raising Qwen3-1.7B from 48.3 to 58.3 on AIME 2024.

  • Takeaways & Limitations

    RL outcomes can serve as reusable improvement signals across model scales rather than only as final policies to imitate.

  • Takeaways & Limitations

    Direct-OPD can fail when teacher/reference improvements are not meaningful on student-visited states, especially when late-prefix signals are off-distribution and unreliable.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

1 Introduction

Direct-OPD is a weak-to-strong post-training method that transfers a weak model’s RL-induced policy shift as an implicit reward, rather than imitating its final policy. It improves stronger students across teacher pairs and model families while reducing the cost of large-model RL.

  • Introduction: 58.3% on AIME 2024: Direct-OPD raises Qwen3-1.7B from 48.3% using 8 A100 GPUs for about 4 hours.The policy shift also improves Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B, including students initially outperforming the post-RL teacher.
  • Introduction: Direct-OPD transfers the improvement induced by weak-model RL by evaluating its policy shift on the stronger student’s own on-policy states.This treats small-model RL as a cheap generator of an implicit reward rather than copying the weak teacher’s final policy.
  • Introduction: Contrasting the weak model before and after RL removes its pre-existing preferences and retains only the behavior changed by RL.Under the KL-regularized RL objective, this log-ratio shift is mathematically equivalent to the reward that trained the weak model.
  • Introduction: Direct-OPD improves every tested student across two teacher pairs and does not require high teacher–student top-k overlap.The paper identifies response-length and KL conditions under which the implicit reward remains aligned with validation accuracy and transfers across thinking patterns.
  • Introduction: At matched RL steps, training a 1.5B model and transferring with Direct-OPD outperforms direct RL on R1-Distill-7B in accuracy and compute.This addresses the scaling bottleneck caused by slower rollouts and more expensive RL iterations on larger target models.

2 Direct On-Policy Distillation

Direct-OPD transfers the RL-induced policy shift between a weak teacher’s post-RL and pre-RL checkpoints rather than imitating the teacher’s final distribution. It applies this implicit reward on the stronger student’s own on-policy prefixes using dense token-level, top-k supervision without target-model sparse-reward RL.

  • Which signal to transfer: Direct-OPD transfers the teacher’s policy shift ∆T = log πT − log πTref, which isolates the RL-induced direction and acts as an implicit reward up to scale and a per-prompt constant.The signal is read from the post-RL/pre-RL checkpoint pair rather than from the teacher’s absolute output distribution.
  • Idealized sequence-level objective: In the idealized objective, the student is optimized as if running KL-regularized RL with the teacher’s implicit reward while remaining anchored to its own initialization πS.This reuses reward-like supervision entirely from the checkpoint pair, without querying verifiable rewards or running sparse-reward RL on the target.
  • From sequence to token level: The sequence shift decomposes into dense token rewards rt(v) = log πT(v | st) − log πTref(v | st), positive for RL-encouraged tokens and negative for suppressed tokens.Direct-OPD uses a zero-discount surrogate that credits each candidate token by its immediate shift on student-visited prefixes.
  • Analytical top-k policy gradient: Direct-OPD restricts supervision to the student’s top-k candidate actions and Rao–Blackwellizes over their full reward distribution, reducing token-sampling variance while leaving trajectory sampling on-policy.The weighted reward is stop-gradient detached so it functions as a scalar coefficient, and the student remains anchored with a standard KL penalty.
  • Adaptive KL control: Because the checkpoint-derived shift has an unobservable teacher-dependent scale, Direct-OPD adapts the student KL coefficient α using the running sign of the student-weighted dense shift.The default controller uses ϵ = 0.01 and clips α to [0.5, 2.5].

3 Experiments

Across teacher pairs and student families, Direct-OPD transfers RL-induced policy shifts to improve stronger students, including students already exceeding the post-RL teacher. Under matched RL-step budgets, small-model RL followed by Direct-OPD outperforms direct large-model RL with lower transfer cost, and independently learned shifts can be composed sequentially.

  • 3.1 Cross-family transfer: Direct-OPD improves students across teacher pairs and student families, including R1-Distill-7B and Qwen3-4B students that already exceed the post-RL JustRL teacher.The QuestA transfer provides a robustness check across a different teacher family, training pipeline, data source, and post-RL checkpoint.
  • 3.2 Matched-step comparison: A 1500-step RL run takes about 160 hours on 32 A100 GPUs for R1-Distill-1.5B versus about 320 hours for R1-Distill-7B, while Direct-OPD adds about 4 hours on 8 A100 GPUs.The transfer stage is described as negligible compared with the RL cost because it requires only a short training run.
  • 3.2 Matched-step comparison: Direct-OPD outperforms direct RL on the larger target under matched RL-step budgets while following a shorter wall-clock training path.The comparison runs RL on R1-Distill-1.5B and transfers its policy shift to R1-Distill-7B, versus running RL directly on R1-Distill-7B.
  • 3.2 Matched-step comparison: The usefulness of Direct-OPD depends on the small-teacher RL checkpoint because different checkpoints encode different policy shifts that are not equally useful for the larger student.The transferred signal is evaluated using R1-Distill-1.5B checkpoints at steps 300, 600, 900, 1200, and 1500.
  • 3.2 Matched-step comparison: A 100-step RL run on Qwen3-1.7B transfers to Qwen3-4B-nonthinking and reaches the 0.635 direct-RL level on AIME 2025.This result indicates that the small-model run can recover a useful RL direction for a stronger 4B target.
  • 3.3 Sequential composition: Direct-OPD sequentially composes two independently learned policy shifts into the same student, with different RL runs contributing different abilities.Qwen3-1.7B first receives the R1-Distill-1.5B →JustRL-1.5B signal and then the Nemotron-1.5B →QuestA-Nemotron-1.5B signal.

4 Analysis and Training Dynamics

Direct-OPD transfers the teacher’s RL-induced policy shift without requiring progressive imitation, generalizes beyond short supervised prefixes, and benefits from pair-dependent KL control. These dynamics indicate that transfer depends on informative local signals within the student’s rollout distribution.

  • 4.1 What Is Transferred?: Direct-OPD transfers RL-induced directions on student-visited tokens even when cross-pattern teacher–student overlap remains low, rather than progressively imitating either teacher checkpoint.Pattern-aligned transfer enters a higher-overlap regime, but cross-pattern gains occur without rising overlap to the post-RL teacher or reference.
  • 4.1 What Is Transferred?: Controlled actor entropy and a narrowing teacher/reference entropy gap support local policy-shift transfer within the student’s distribution rather than actor collapse or full-policy imitation.The diagnostics indicate that training changes where the student samples for evaluating the teacher’s RL-induced shift while keeping its distribution controlled.
  • 4.2 Short-Horizon Generalization: After 40 steps with 2k-token training responses, the actor shifts toward the teacher direction across much longer rollouts, while 6k training validates worse at 45.6 versus 48.8 for 2k.The 6k horizon moves the diagnostic further but may overemphasize unreliable late-prefix signals; moderate horizons capture reliable transferable signal without that instability.
  • 4.3 KL Adaptation: Fixed-KL sweeps show that the best constraint strength differs across teacher–student pairs, so the dense teacher/reference reward cannot be optimized independently of rollout distribution.A larger positive dense reward tracks best validation in one setting but coincides with worse validation in another.
  • 4.3 KL Adaptation: Adaptive KL initially corrects the dense reward and then moves it toward zero, keeping students where teacher/reference comparisons remain informative instead of encouraging unreliable or ignored shifts.Persistently large positive or negative rewards can indicate that the student has left the useful regime.

5 Related Work

Direct-OPD builds on on-policy distillation, weak-to-strong generalization, and implicit-reward methods, while differing from weight-space model reuse. Its distinctive signal is a behavioral policy/reference log-ratio applied to the stronger student’s own on-policy states.

  • On-policy distillation of reasoning models: Direct-OPD extends on-policy distillation, a modern approach for transferring reasoning-model improvements produced by reinforcement learning with verifiable rewards.This work sits within broader knowledge-distillation research spanning classic, sequence-level, and modern LLM distillation methods.
  • Weak-to-strong generalization: The method addresses weak-to-strong generalization, where weak supervisors are used to elicit capabilities from stronger models under scarce reliable supervision.This connects to scalable oversight, learning from imperfect supervision, latent-knowledge elicitation, and easy-to-hard generalization.
  • Implicit rewards and reusing trained models: Direct-OPD reverses the policy-as-reward identity by reading a dense implicit reward from a post-RL checkpoint’s log-ratio against its reference.The same identity underlies preference-optimization methods that fit policies without explicit reward models, and it connects Direct-OPD to dense and process rewards for reasoning.
  • Implicit rewards and reusing trained models: Unlike task arithmetic, model merging, and proxy tuning, Direct-OPD transfers a behavioral log-ratio rather than a weight delta on the student’s own on-policy states.This design is described as transferring across model families and scales.

6 Conclusion and Limitations

Direct-OPD transfers a small RL teacher’s learned policy shift as a dense reward, rather than imitating its post-RL policy, enabling weak-to-strong generalization. This approach improves stronger students across teacher pairs and student families while outperforming step-matched direct RL at lower compute.

  • Conclusion: Direct-OPD uses the teacher’s log-ratio against its pre-RL reference as a dense reward on student-visited tokens.The transferable object is the teacher’s policy shift, not its final post-RL policy.
  • Conclusion: A smaller, weaker RL teacher can improve stronger students because Direct-OPD transfers its policy shift rather than its model policy.
  • Conclusion: Direct-OPD transfers across teacher pairs and student families, outperforms step-matched direct RL, and requires a fraction of the compute.

A. Experimental Details

Direct-OPD uses the math subset of Skywork-OR1-RL-Data [15] with a DAPO-style prompt for training rollouts and evaluation. The experiments specify evaluation, Direct-OPD, and RL-teacher/direct-RL protocols through Tables 2–4, while transfer remains similar with DAPO-Math-17K.

  • Data and prompt: Direct-OPD training and evaluation use the math subset of Skywork-OR1-RL-Data [15] with a DAPO-style step-by-step prompt requiring a final standalone Answer line.The prompt is used for both training rollouts and evaluation prompts.
  • Data and prompt: The DAPO-style prompt differs from the boxed-answer prompt used for teacher RL and gives slightly better transfer, so the paper uses it throughout.Replacing Skywork training data with DAPO-Math-17K while retaining the prompt produces similar transfer trends.
  • Evaluation and training settings: Evaluation uses a protocol summarized in Table 2, while default Direct-OPD and RL-teacher/direct-RL training settings are specified in Tables 3 and 4.Table 4 follows the math GRPO setting of Yang et al. [64], with different teacher batch sizes for R1-Distill and Qwen3-nonthinking runs.

B. Additional Entropy Diagnostics

Additional entropy diagnostics examine student, teacher, reference, and teacher-minus-reference entropies for QuestA-Nemotron distilled into Qwen3-1.7B. Together with Figure 6, they show that the non-collapse pattern is not specific to the JustRL teacher pair.

  • B. Additional Entropy Diagnostics: The diagnostics compare student entropy, post-RL teacher entropy, teacher-reference entropy, and teacher entropy minus reference entropy.These quantities are shown across the figure’s panels for QuestA-Nemotron into Qwen3-1.7B.
  • B. Additional Entropy Diagnostics: Together with Figure 6, the diagnostics show that the non-collapse pattern is not specific to the JustRL teacher pair.
  • B. Additional Entropy Diagnostics: The analysis concerns the QuestA-Nemotron-to-Qwen3-1.7B teacher–student pairing.

C. Additional Results

This section presents QuestA transfer curves for cross-pattern transfer on AIME 2025, while noting that the corresponding AIME 2024 curves appear in the main text.

  • Additional Results: QuestA transfer curves are evaluated in the cross-pattern transfer setting.The supplied figure covers transfer behavior rather than reporting numerical results.
  • Additional Results: The evaluation uses AIME 2025.
  • Additional Results: The corresponding AIME 2024 curves are reported in the main text.
Loading 2607.05394v2…