Weak-to-Strong Generalization via Direct On-Policy Distillation

1 Introduction

• RLVR becomes costlier as target models grow because each update requires policy-generated rollouts, motivating RL on cheaper weak models instead.Direct-OPD treats the weak model as an inexpensive carrier of RL-induced improvement, not as a reasoner that must outperform the student.Source: §1 · pp. 1–2
• 48.3 to 58.3 on AIME 2024: Direct-OPD improves Qwen3-1.7B in about 4 hours on 8 A100 GPUs.Two teacher pairs—R1-Distill-1.5B → JustRL-1.5B and Nemotron-1.5B → QuestA-Nemotron-1.5B—improve all three students: Qwen3-1.7B, Qwen3-4B [3], and R1-Distill-7B, including the latter two, which already outperform the post-RL teacher. The Qwen3-1.7B result is comparable to Polaris running direct RL with 32 A100 GPUs for at least a week.Source: Figure 1 · p. 2, §1 · pp. 2–3
• 56.7 to roughly 50: vanilla OPD lowers R1-Distill-7B [1], despite its initial advantage over the post-RL JustRL-1.5B teacher [2] at 51.3.Imitating the final teacher policy entangles useful RL-induced changes with the weak model’s capacity limits and can overwrite stronger student behavior.Source: §1 · p. 2, Figure 1 · p. 2
• Direct-OPD applies the weak teacher’s pre-to-post-RL log-policy difference as an implicit reward on student-generated states, with KL anchoring to student initialization.The shift, log πT(y | x) − log πTref(y | x), removes pre-RL preferences and is positive for responses RL encouraged and negative for those it suppressed. Under KL-regularized RL, optimizing this shift recovers the original reward objective’s form, requiring neither an explicit reward model nor sparse-reward RL on the target.Source: §1 · pp. 1–3
• At matched RL steps, RL on a 1.5B model followed by Direct-OPD outperforms direct RL on R1-Distill-7B in both accuracy and compute.Source: §1 · p. 3
• Direct-OPD transfers across thinking patterns without requiring high teacher–student top-k overlap; response-length and KL conditions determine alignment between its implicit reward and validation accuracy.Source: §1 · p. 3

2.1 Preliminaries: On-Policy Distillation

• On-policy distillation supervises student-generated trajectories with teacher next-token distributions; Direct-OPD preserves its student-top-k interface while allowing a smaller or weaker teacher.The top-k KL estimator renormalizes teacher and student probabilities on the retained support [5].Source: §2.1 · p. 3

2.2 Policy Shifts as Implicit Rewards

• Direct-OPD transfers the post-RL teacher’s log-policy ratio against its pre-RL reference, rather than imitating the teacher’s absolute output distribution.Assuming the teacher is a KL-regularized optimum, this ratio recovers its implicit reward up to positive scale and a per-prompt constant. This reverses the policy–reward identity underlying Direct Preference Optimization [6], extracting a reward-like signal from existing checkpoints.Source: §2.2 · pp. 3–4

2.3 The Direct-OPD Objective

• Direct-OPD optimizes the teacher-shift signal while KL-regularizing toward the student’s own initialization, without querying verifiable rewards or running sparse-reward RL on the target.The idealized sequence-level objective therefore treats the teacher’s implicit reward as the student’s RL reward, but retains the student’s initial checkpoint as its reference.Source: §2.3 · p. 4
• The sequence shift decomposes exactly into per-token log-ratios, implemented as immediate rewards in a zero-discount surrogate restricted to renormalized student top-k actions.Positive shifts reward tokens encouraged by the teacher’s RL, while negative shifts penalize suppressed tokens. The practical surrogate credits each candidate’s immediate shift rather than future returns over subsequent shifts.Source: §2.3 · p. 4
• An analytical top-k policy gradient averages over candidate actions to remove per-step token-sampling variance while retaining sampled trajectories.The student-probability-weighted reward is detached into a scalar coefficient to avoid extra softmax Jacobian terms. The implementation anchors the student using verl’s standard KL penalty rather than differentiating the KL analytically.Source: §2.3 · p. 5

2.4 Adaptive KL Control

• The checkpoint pair fixes the transferred reward scale but cannot reveal the underlying reward or teacher KL budget, preventing advance calibration of a universal student KL coefficient.The student affects where this fixed signal is evaluated through its visited prefixes and retained top-k actions, not the token-level reward itself.Source: §2.4 · p. 5
• Adaptive KL control raises the KL coefficient for positive mean student-weighted shifts and lowers it for negative shifts, rather than targeting a KL value.Before each actor update, the coefficient is multiplied by 1 + ϵ sgn(mean shift) and clipped, with default ϵ = 0.01 and bounds [0.5, 2.5]. Zero mean shift leaves the coefficient unchanged.Source: §2.4 · pp. 5–6

3.1 A small RL teacher improves much stronger students

• Direct-OPD improves students across teacher pairs, including R1-Distill-7B and Qwen3-4B, whose starting accuracies already exceed the post-RL JustRL teacher.The results support transferring an RL-induced improvement direction rather than imitating the teacher’s full policy. QuestA provides a robustness check across training pipelines and data sources, not a strict weaker-teacher comparison because it is already strong on AIME.Source: §3.1 · p. 6, Figure 2 · p. 7

3.2 Weak-to-strong generalization beats RL

• About 160 versus 320 hours for 1500-step RL favors R1-Distill-1.5B over R1-Distill-7B, with Direct-OPD adding only about 4 hours.Small-model RL uses 32 A100 GPUs; transfer uses 8 A100 GPUs and is inexpensive because training is short and small-teacher scoring is not the bottleneck. The authors also characterize small-model RL as more stable and easier.Source: §3.2 · pp. 8–9
• Small-model RL followed by Direct-OPD improves R1-Distill-7B more than direct R1-Distill-7B RL under matched RL-step budgets.Later transfers, T600–T1500, also outperform direct RL at comparable elapsed time on AIME 2025. Transfer evaluates the small teacher/reference log-ratio on the larger student’s own on-policy states, reusing the improvement direction discovered by small-model RL.Source: §3.2 · pp. 7–8, Figure 3 · p. 8
• 0.635 AIME 2025 accuracy, matching direct RL, is reached by Qwen3-4B-nonthinking after transferring a Qwen3-1.7B RL policy shift.The small-model RL run lasts 100 steps.Source: Figure 3 · p. 8, §3.2 · p. 8
• Teacher-checkpoint choice matters: T300 transfers a weaker shift than T900–T1500, and shifts differ in usefulness on the larger student’s state distribution.The experiment uses DAPO-trained R1-Distill-1.5B checkpoints at steps 300, 600, 900, 1200, and 1500, each paired with the base model.Source: §3.2 · pp. 7–8, Figure 3 · p. 8

3.3 Sequential composition

• Direct-OPD sequentially composes independently learned policy shifts into Qwen3-1.7B, using JustRL’s signal first and QuestA’s second.The sequence transfers R1-Distill-1.5B →JustRL-1.5B, then Nemotron-1.5B →QuestA-Nemotron-1.5B, with evaluation on AIME 2024 and AIME 2025. The small stage-boundary discontinuity reflects separate evaluation samples rather than a training discontinuity.Source: §3.3 · p. 9, Figure 4 · p. 9

4.1 Cross-pattern transfer without token-overlap imitation

• Cross-pattern Direct-OPD gains occur without progressive token-overlap imitation of either teacher checkpoint.Prior OPD work emphasizes high-probability-token alignment [5]; pattern-aligned transfer reaches higher overlap, whereas cross-pattern transfers remain lower without compensating reference overlap. Direct-OPD instead evaluates teacher/reference log-ratios on student-visited candidate tokens, ranking actions within the student’s support.Source: §4.1 · pp. 9–10, Figure 5 · p. 10
• Actor entropy remains controlled rather than collapsing, while the teacher/reference entropy gap narrows during training.These diagnostics support local policy-shift transfer rather than degenerate low-entropy sharpening or imitation of a full teacher policy.Source: §4.1 · p. 10, Figure 6 · p. 11

4.2 Short-horizon training changes longer-rollout behavior

• 48.8 versus 45.6 AIME 2024/2025 validation accuracy (ave@32) makes 2k best and 6k worst among tested response horizons after 40 training steps.With fixed KL=1, the 2k-trained actor shifts toward the teacher-shift direction across approximately 16k rollout positions, beyond its supervised horizon. Longer horizons may amplify unreliable, off-distribution late-prefix signals, whereas overly short horizons capture less useful signal.Source: §4.2 · pp. 11–13, Figure 8 · p. 12

4.3 KL controls teacher-shift reward reliability

• KL controls teacher-shift reward reliability by determining which rollout states the student visits.Too little KL permits drift into regions where both teacher checkpoints assign low probability and their log-ratio becomes noisy. Too much KL prevents the student from following the teacher’s policy shift.Source: §4.3 · p. 13
• The best fixed KL coefficient is teacher–student-pair dependent, and larger dense rewards do not consistently predict better validation.The dense reward therefore should not be maximized independently of the rollout distribution; its usefulness depends on the model pair and where the student samples.Source: Figure 9 · p. 13, §4.3 · pp. 13–14
• Adaptive KL moves mean dense token rewards toward zero after an initial correction phase.This is consistent with keeping teacher/reference comparisons informative rather than exploiting unreliable disagreement. A near-zero batch mean suggests a balanced regime where local token shifts can still rank actions without global drift.Source: §4.3 · p. 14

On-policy distillation of reasoning models

• Direct-OPD transfers the teacher’s RL-induced policy shift rather than its final policy, avoiding the weak teacher’s capacity ceiling.Prior analyses link OPD success to teacher–student top-k overlap [5], identify difficulties copying much stronger reasoners [63], and reinterpret OPD as reward-rescaled, KL-constrained RL [64].Source: §5 · p. 14

Weak-to-strong generalization

• Direct-OPD uses paired small RL checkpoints to improve students already stronger than the post-RL teacher, rather than supervising with weak labels or distributions.Weak-to-strong generalization seeks to elicit stronger-model capabilities using weak supervision [65], while self-rewarding bootstraps a model with its own judgments [92]. The closest precedent reuses a weak aligned model’s implicit reward for stronger-model preference optimization [93].Source: §5 · p. 14

Implicit rewards and reusing trained models

• Under KL-regularized RL, the policy/reference log-ratio recovers reward up to a constant; Direct-OPD uses this identity in reverse to extract dense supervision.Direct Preference Optimization and variants use this identity to fit policies from preferences without explicit reward models [6] [94] [95] [96]. Direct-OPD connects to dense and process rewards [97] [98] [99] and per-token credit assignment [100] [101], while avoiding reward over-optimization from an explicit model [102].Source: §5 · p. 14
• Unlike weight-space reuse through task arithmetic, model merging, and proxy tuning [103] [104] [105], Direct-OPD applies behavioral log-ratios on student-visited states, enabling transfer across model families and scales.Its transfer signal is a behavioral log-ratio rather than a weight delta.Source: §5 · p. 15

Conclusion

• Direct-OPD transfers across teacher pairs and student families, outperforms step-matched direct RL at a fraction of the compute, and composes across teachers.These findings frame RL outcomes as reusable improvement signals rather than final models to imitate.Source: §6 · p. 15
• Direct-OPD transfers the post-RL teacher’s log-ratio against its own pre-RL reference, evaluated on student-visited tokens, rather than the teacher’s final policy.This policy shift supplies a dense reward through which a smaller, weaker teacher can improve stronger students.Source: §6 · p. 15

Limitations

• Direct-OPD can fail when the teacher/reference improvement is not meaningful on student-visited states.The best response length and KL strength also remain teacher–student dependent.Source: §6 · p. 15

Direct-OPD data and prompt

• Final Direct-OPD runs use the math subset of Skywork-OR1-RL-Data, released with Skywork-OR1 [15].Replacing this dataset with DAPO-Math-17K while retaining the prompt format produces similar transfer trends, suggesting transfer is not specific to one training dataset.Source: A. Experimental Details · p. 22
• Training rollouts and evaluation share a prompt requesting step-by-step solutions and a final answer on its own line after “Answer:”.The template includes an explicit reminder to use the requested answer format.Source: A. Experimental Details · p. 22
• The DAPO-style prompt gives slightly better transfer than teacher RL’s boxed-answer prompt and is retained throughout the experiments.Prompt-template divergence is not a focus of the study.Source: A. Experimental Details · p. 22

• The QuestA-Nemotron diagnostics, together with Figure 6, show that the non-collapse pattern is not specific to the JustRL teacher pair.Source: Figure 10 · p. 23
• Figure 10 reports student entropy, post-RL teacher entropy, and teacher-reference entropy for QuestA-Nemotron into Qwen3-1.7B.Source: Figure 10 · p. 23
• A separate panel reports teacher entropy minus reference entropy.Source: Figure 10 · p. 23

C. Additional Results

• Figure 11 shows QuestA cross-pattern transfer curves on AIME 2025, complementing the corresponding AIME 2024 curves reported in the main text.The supplied caption does not report numerical results or establish a performance comparison.Source: Figure 11 · p. 24