On-Policy Distillation with Negative-Policy Rollouts

1 Introduction

• OPD distills teacher supervision on student-generated sequences, but its objective remains mimicking the teacher despite advances in reward functions and teacher configurations.SeqKD instead uses teacher-generated sequences and is described as limited in preserving student on-policy capabilities.Source: §1 · p. 2
• NP-OPD adds negative-policy rollouts without changing OPD’s reward, concentrating suppression on negative-policy-preferred tokens while retaining teacher supervision.The negative policy is a lower-performing, lower-capability reference; the authors report that this design can combine with recent OPD advances.Source: §1 · p. 2
• NP-OPD improves OPD across student scales, thinking and non-thinking modes, three reasoning domains, and recent variants including ExOPD and OPD2.The analyses report concentrated suppression of negative-policy-preferred tokens and reduced measured overlap with the negative policy.Source: §1 · pp. 2–3
• The authors report that pre-generated negative-policy rollouts reduce rollout-generation cost, while degrading student rollouts improves OPD less than using a separate negative policy.Source: §1 · p. 3

2.1 On-Policy Distillation

• OPD trains on student-generated trajectories and uses token-level teacher–student signals to adjust the student toward the teacher distribution.The teacher remains the sole reference determining the update direction.Source: §2.1 · p. 3

2.2 Learning from Positive and Negative Signals

• DPO uses preferred and rejected responses, while GRPO reinforces positive-advantage responses and suppresses negative-advantage responses.For correctness-based reasoning, this typically means reinforcing correct responses and suppressing incorrect ones.Source: §2.2 · p. 3
• Unlike these approaches, OPD gives a direction toward the teacher but no explicit reference for what the student should move away from.The section asks whether adding such a negative signal can benefit OPD without changing its original teacher supervision.Source: §2.2 · p. 3

3 Method

• NP-OPD adds an auxiliary rollout source and applies teacher supervision to both student-generated and negative-policy trajectories.The negative policy is selected from the student’s model family with lower capacity and lower overall reasoning performance.Source: §3 · p. 3, §3.1 · p. 4

3.1 Distillation with Negative-Policy rollouts

• Each batch mixes student and negative-policy trajectories in proportions 1 − α and α, respectively, while leaving the token-level teacher–student signal unchanged.At α = 0 the method reduces to OPD; at α = 1 it uses only negative-policy trajectories.Source: §3.1 · p. 4

3.2 How Negative-Policy rollouts Provide Negative Signals

• The analysis assumes the teacher-to-negative-policy KL divergence exceeds the student-to-negative-policy divergence.The authors describe this as reasonable in practice because the negative policy is selected to have weaker performance than the student.Source: §3.2 · p. 5
• Negative-policy tokens have negative expected OPD rewards, so their increased contribution statistically pushes the student away from the negative policy.This does not mean every negative-policy token is suppressed: tokens preferred by the teacher can still be enhanced, preserving OPD’s signal direction.Source: §3.2 · p. 5

4.1 Experimental Setup

• The experiments span Qwen3 student scales and both thinking modes, with training on Math, Science, and Code and evaluation across 13 reasoning benchmarks.The setup also evaluates Gemma-4 and varies the rollout-model scale.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Main Results

• NP-OPD consistently improves non-thinking OPD across model scales and reasoning domains, with the same trend extending to Qwen3-8B and thinking-mode generation.The non-thinking results cover Math, Code, and Science averages for Qwen3-1.7B and Qwen3-4B.Source: §4.2 · p. 6, Table 1 · p. 6
• For Qwen3-1.7B, NP-OPD adds 1.98 points with OPD and 1.76 with OPD2; for Qwen3-4B, gains are 1.11, 1.86, and 2.10 with OPD, ExOPD, and OPD2.Qwen3-1.7B performance with ExOPD remains comparable.Source: §4.2 · p. 6, Table 2 · p. 7
• On Gemma-4-E4B, NP-OPD improves over OPD on Math, Science, and Code, although both methods remain below the base model on Code.The reported averages change from 63.0 to 66.4 on Math, 49.1 to 49.8 on Science, and 54.9 to 55.2 on Code; the Code base-model average is 56.5.Source: §4.2 · p. 6, Table 3 · p. 7
• For Qwen3-1.7B on one 8×H100 node, NP-OPD reduces thinking-mode training time from 9h19m to 3h34m, excluding rollout pre-generation.Negative-policy rollouts can be pre-generated and reused during training.Source: §4.2 · pp. 6–7

4.3 Understanding the Effectiveness of NP-OPD

• NP-OPD with Qwen3-0.6B yields higher NSP than OPD, concentrating suppression on tokens preferred by the negative policy over the teacher; only the lower-capability rollout improves over OPD.Higher-capability rollout models, including the teacher, yield lower NSP and accuracy.Source: §4.3 · pp. 7–8, Figure 2 · p. 8, Figure 3 · p. 8
• On fixed initial-student trajectories, mean NOR is 0.019 under NP-OPD versus 0.005 under OPD, while teacher rollouts yield −0.030.NOR measures a teacher-relative shift in top-token probabilities at positions where the teacher and negative policy disagree.Source: §4.3 · p. 8, Figure 4 · p. 9

5 Discussion

• ExOPD and OPD2 show higher NSP than standard OPD, while combining NP-OPD with either further increases NSP and average accuracy.The authors interpret this as negative-policy rollouts complementing implicit negative signals in these variants.Source: §5 · p. 9
• 0.9- and 1.0-point average-score gains from temperature scaling and persona prompting are smaller than the negative policy’s 3.4-point gain.Layer dropping also improves over standard on-policy rollouts, but the passage does not give its score gain.Source: §5 · p. 9
• Simple degradation of student rollouts can provide useful complementary signals, but does not match the separate negative policy’s move-away direction.The tested degradations were increased temperature, an elementary-school persona prompt, and dropping three layers during rollout generation.Source: §5 · p. 9

6 Conclusion

• NP-OPD adds rollouts from a lower-capability negative policy as a negative reference while preserving the original distillation reward.The negative policy is introduced at the rollout stage alongside positive teacher guidance.Source: §6 · p. 9
• Across model scales, generation modes, reasoning domains, and OPD variants, NP-OPD improves performance over on-policy baselines.Source: §6 · p. 9
• Analyses show that NP-OPD concentrates suppression on tokens the negative policy prefers over the teacher and reduces measured overlap with that policy.Source: §6 · p. 9
• Reusing pre-generated negative-policy rollouts reduces rollout-generation costs during training.Source: §6 · p. 9

A.1 Post-Training for LLMs

• Post-training includes SFT on curated prompt-response pairs, preference learning, RLHF, and DPO, which use additional supervision to adapt pretrained models.The passage frames these approaches as using complementary positive and negative signals.Source: §A.1 · p. 14

A.2 On-Policy Distillation

• OPD trains on student-generated trajectories with token-level teacher supervision, reducing train-inference mismatch compared with teacher-generated Sequence-KD [17].The passage also reports competitive performance for reasoning post-training [42] [40].Source: §A.2 · p. 14
• Recent OPD work changes reward signals or trajectories; this paper instead uses negative-policy rollouts as a negative reference under the existing reward [43] [13] [38] [41].The cited approaches include uncertainty-aware weighting, extrapolated teacher signals, teacher-to-base deltas, fixed SFT rollouts, and teacher takeover of failed prefixes [16] [43] [13] [38] [41].Source: §A.2 · p. 14

B OPD Reward Formulations

• ExOPD [43] extrapolates reward from a reference policy toward the teacher; λ = 1 recovers standard OPD, while λ > 1 extrapolates further.The experiments use the teacher’s base model as the reference policy.Source: B OPD Reward Formulations · p. 14
• OPD2 [13] uses the probability change from the teacher’s base model to its posttrained version, centers its signals under the student distribution, and conditions the delta on agreement with standard OPD.Source: B OPD Reward Formulations · p. 15

C.1 Training and Evaluation Details

• Training uses 30K reasoning prompts and, unless specified otherwise, 100 steps with global batch size 256.Rollouts have a maximum length of 16,384 tokens and sampling temperature 1.0.Source: §C.1 · p. 15
• Optimization uses AdamW at 5×10^-6 with cosine decay, 0.1 minimum learning-rate ratio, 0.1 warmup, gradient clipping at 1.0, bf16, and ZeRO-3.KL regularization β is 0 for Qwen and 0.04 for Gemma.Source: §C.1 · p. 15
• Shorter-than-20-token Qwen non-thinking rollouts are filtered, while Gemma experiments use KL regularization because OPD training is less stable than for Qwen.The Gemma KL coefficient is β = 0.04.Source: §C.1 · p. 15
• Qwen runs use one eight-H100 node with concurrent vLLM rollout generation, while Gemma runs use two nodes.Qwen3-8B uses tensor parallelism of size 4.Source: §C.1 · p. 15
• Evaluation covers math, coding, and science benchmarks, using avg@16, avg@8, or avg@3 according to benchmark.The compared methods use the same evaluation implementation and decoding configuration.Source: §C.1 · p. 15

• Across evaluated nonzero α values, NP-OPD generally improves overall average performance over the on-policy baseline in both thinking and non-thinking modes.Source: §C.2 · p. 15
• In non-thinking mode, NP-OPD improves domain-average performance in nearly all combinations of student scale, reasoning domain, and OPD reward formulation.Larger α generally helps OPD and ExOPD, while mixed rollouts are competitive or better in some OPD2 settings; gains occur at different α values.Source: §C.2 · pp. 15–16
• In thinking mode, NP-OPD broadly improves results across student scales, reasoning domains, and OPD reward formulations, with gains from mixed and fully negative-policy rollouts.Larger α often benefits Math and Science, while intermediate values sometimes yield stronger Code results.Source: §C.2 · p. 19

• Training time fell from 9h19m to 3h34m in thinking mode and from 5h15m to 1h16m in non-thinking mode with standard OPD at α = 1.The timings are for Qwen3-1.7B on one node with 8 H100 GPUs; negative-policy rollouts are generated in advance.Source: §C.3 · p. 22, Table C.15 · p. 22
• For Qwen3-4B and Qwen3-8B, NP-OPD reduces generation length across OPD, ExOPD, and OPD2, but slightly increases it for Qwen3-1.7B.The comparison covers thinking-mode AIME24 and AIME25 results.Source: §C.3 · p. 22, Table C.16 · p. 22
• Equal-weight merging improves ExOPD and OPD2 scores over their on-policy baselines across Math, Code, and Science for Qwen3-8B in thinking mode.For ExOPD, scores rise from 74.2 to 76.3 on Math, 60.3 to 60.7 on Code, and 53.6 to 54.7 on Science; OPD2 also improves in all three domains.Source: §C.3 · p. 22, Table C.17 · p. 22

C.4 Details on Section 4.3

• NSP is the fraction of selected tokens with a negative student log-probability change for which the negative policy assigns higher probability than the teacher.Selection is limited to the top 5% of token positions by absolute log-probability change within each trajectory.Source: §C.4 · pp. 22–23
• The NOR comparison uses 300 training prompts, with 100 each from Math, Code, and Science, and compares α = 1 rollout settings with the α = 0 on-policy baseline.The negative reference and teacher are fixed to Qwen3-0.6B and Qwen3-8B.Source: §C.4 · p. 23
• 0.019 with Qwen3-0.6B rollouts versus 0.005 for on-policy OPD; mean NOR falls to −0.014, −0.023, and −0.030 with Qwen3-1.7B, 4B, and 8B rollouts.Only the lower-capability negative policy has higher mean NOR than on-policy OPD.Source: §C.4 · p. 23, Figure C.1 · p. 24
• On 300 held-out prompts from nine benchmarks, most trajectories lie below the diagonal, indicating greater relative overlap reduction under NP-OPD than OPD.The held-out prompts span mathematics, science, and code and are not included in training.Source: §C.4 · p. 24, Figure C.2 · p. 24

D Details of Section 5

• The persona prompt asks the student to respond as an elementary school student, using simple ideas and avoiding advanced methods such as calculus or formal proofs.Source: D Details of Section 5 · p. 25
• The layer-drop setting removes the last three transformer layers from the student model.Source: D Details of Section 5 · p. 25