When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Is on-policy sampling always beneficial for distilling

• Pearson correlation r = −0.73 across 17 pairs links lower initial top-64 overlap with larger Semi-OPD gains over OPD.OPD is beneficial only when the teacher and student have high overlap.Source: Figure 2 · p. 2, Is on-policy sampling always… · p. 2

1. Introduction

• Prior work reports that OPD rollouts can grow longer, enter self-correction loops, and provide unhelpful or unreliable supervision on later tokens.These reported artifacts motivate examining whether on-policy sampling is always beneficial.Source: §1 · p. 2

Is on-policy sampling always beneficial for distilling

• Semi-OPD samples rollouts once from the initial student before training and reuses them, making rollout and teacher-logit generation offline.Its objective remains identical to OPD’s, with student log-probabilities recomputed using current parameters.Source: Is on-policy sampling always… · p. 2
• Semi-OPD outperforms OPD on 14 of 17 teacher–student pairs, with up to +13.6% accuracy and 11.4× training speedup.On the remaining three pairs, it achieves comparable performance, within a 0.1%–2.6% gap.Source: Is on-policy sampling always… · p. 2
• For misaligned pairs, evolving student rollouts can become increasingly off-policy with respect to the teacher as contexts grow, weakening the distillation signal.Semi-OPD is more stable by distilling over shorter contexts while covering full trajectories and exposing students to more teacher-preferred tokens.Source: Is on-policy sampling always… · pp. 2–3
• The authors propose using measured teacher–student overlap to guide the choice between OPD and Semi-OPD rather than defaulting to OPD.They also call for stronger OPD research under meaningful teacher–student pairings.Source: Is on-policy sampling always… · p. 3

2.1. Preliminaries of OPD

• The sampled-token OPD objective evaluates teacher–student log-probability ratios only on student-sampled tokens, rather than comparing the full vocabulary at every position.The section also describes the sequence-level objective as reverse KL from the student distribution to the fixed teacher distribution.Source: §2.1 · p. 3

2.2. Semi-OPD: Formulation

• Semi-OPD samples responses once from the initial student, reuses them through training, and recomputes student log-probabilities and advantages with current parameters.Unlike OPD, its rollout data stays fixed; optimization still uses the same objective.Source: §2.2 · p. 3

2.3. Benchmarking Semi-OPD vs. OPD and its Variants

• The benchmark covers 17 pairs across Qwen3, DeepSeek-R1-Distill, and Nemotron-Cascade-2, with thinking and non-thinking modes represented.Mathematical capability is measured by avg@32 across AIME 2024–2026 and HMMT February 2026.Source: §2.3 · p. 4
• Semi-OPD beats OPD on 14/17 pairs by up to ~13% at the last step, while remaining within 0.1–2.6% on three; gains are generally larger at lower overlap.Figure 3 compares avg@32 accuracy across pairs, with initial top-64 overlap in each panel.Source: Figure 3 · p. 4, §2.3 · p. 5
• On low-overlap Qwen3 non-thinking pairs, Semi-OPD exceeds Relay-OPD by +7.1% and +8.8% accuracy, with 35× and 19.5× less training time, respectively.For the high-overlap Nemotron-Cascade-2 pair, Semi-OPD is 0.7% behind the best baseline, while using 2.2× less time than FastOPD and 4.5× less than OPD.Source: §2.3 · p. 5, Table 1 · p. 5
• Semi-OPD achieves up to 11.4× speedup over standard OPD and 2.7× over FastOPD, while separating rollout generation, teacher scoring, and student optimization into phases.Once cached rollouts and teacher log-probabilities are prepared, training needs neither online rollout generation nor teacher forward passes.Source: §2.2 · p. 3, §2.3 · p. 5

3. How to Choose Between OPD and Semi-OPD?

• Across 17 pairs, Semi-OPD’s gain over OPD is strongly negatively correlated with initial overlap (Pearson’s r = −0.73); OPD is needed mainly for high-overlap pairs.The authors recommend using overlap to guide the choice rather than defaulting to OPD.Source: §3 · pp. 5–6
• For low-overlap pairs, repeatedly sampling from the evolving student can degrade online rollouts and weaken supervision; reusing initial-student rollouts can provide a more stable signal.The paper positions careful teacher–student pair selection as important when evaluating improved OPD schemes.Source: §3 · p. 6
• Teacher-rollout SFT slightly raises initial overlap, but Semi-OPD without that SFT stage can outperform Semi-OPD with it.After SFT, Semi-OPD still outperforms OPD on both tested pairs, but by a smaller margin.Source: §3 · p. 6, Figure 4 · p. 6

4. Why Does OPD Often Underperform Semi-OPD?

• Low-overlap OPD rollouts can grow longer and increasingly diverge from the teacher distribution, making later-token supervision unhelpful or harmful; truncation removes later-stage supervision.Semi-OPD uses usually shorter initial-student rollouts while retaining full reasoning trajectories.Source: §4 · pp. 6–7
• Replacing initial-student rollouts with teacher-generated rollouts makes distillation worse than Semi-OPD, indicating that student-generated trajectories also matter.The comparison keeps the other settings unchanged.Source: §4 · p. 7, Figure 5 · p. 7
• Increasing OPD rollout caps can hurt the low-overlap pair while benefiting the higher-overlap pair.The authors hypothesize that long, teacher-misaligned rollouts can reinforce superficial reasoning patterns in low-overlap pairs.Source: §4 · p. 7, Figure 6 · p. 7
• With the training-token budget fixed, restricting cached Semi-OPD traces to the first quarter slows convergence and lowers final accuracy; full trajectories achieve the best peak accuracy.The ablation compares supervision on the first 1/4, 1/2, 3/4, or all of each rollout.Source: §4 · p. 7, Figure 5 · p. 7
• Semi-OPD maintains a larger fraction of training tokens with p_T > p_S than OPD and also outperforms it under top-k reverse KL.The authors interpret the token pattern as greater exposure to teacher-preferred tokens; the advantage is not specific to the sampled-token objective.Source: §4 · p. 7, Figure 7 · p. 8, Figure 8 · p. 8
• The proposed account is that Semi-OPD’s student-anchored rollouts preserve effective teacher supervision and broad reasoning coverage, which can outweigh fully on-policy sampling under large mismatch.The authors frame effective distillation as balancing student on-policyness with high-quality teacher supervision.Source: §4 · pp. 6, 8

5. More Intriguing Observations

• On low-overlap pairs, OPD warm starts converge to the same performance level as OPD initialized from the initial student; warm starts affect only the convergence trajectory.Source: §5 · p. 8
• With 86.8% initial top-64 overlap, switching from Semi-OPD to OPD improves performance and closes most of the gap with OPD from scratch.Source: §5 · p. 8, Figure 10 · p. 9
• Figure 11 evaluates code performance for Qwen3-1.7B distilled from Qwen3-4B-Instruct-2507 on math-only data.Source: Figure 11 · p. 9

Semi-OPD requires adequately strong initial

• Semi-OPD needs meaningful initial trajectories: it underperforms OPD with a base student but regains its advantage with an instruction-tuned student.Source: Semi-OPD requires adequately strong… · pp. 8–9

• OPD uses student-policy trajectories with dense teacher supervision, mitigating exposure bias [30] and sharing an idea with DAgger [31].Related work describes reverse-KL and related objectives [1] [28].Source: §6 · p. 9
• Prior work reports longer, non-terminating or self-correcting rollouts [6] [7] [8] [9] and unreliable supervision from later tokens [11] [13] [14] [36].Source: §6 · p. 9
• Prior mitigations include truncation, teacher intervention, trajectory refinement, warm starts, and trust-region updates; this work instead reuses initial-student rollouts without teacher-SFT or refreshing.It evaluates 17 model pairs and reports that the benefit correlates with initial teacher–student top-k overlap.Source: §6 · p. 9

• Continually regenerating student rollouts is often unnecessary and can be harmful, especially when teacher and student are poorly aligned.Source: §7 · p. 9
• Effective distillation requires balancing on-policyness with respect to both student and teacher, with initial overlap serving as a practical indicator for method choice.Source: §7 · p. 9
• The findings motivate treating on-policy sampling as a design choice and developing methods that account for teacher–student alignment and rollout quality.Source: §7 · p. 9

• The 17 pairs span Qwen3, DeepSeek-R1-Distill-Qwen, and Nemotron Cascade-2, with teachers up to 235B-A22B parameters.Some students begin from checkpoints after one epoch of SFT on teacher-generated rollouts.Source: A. Training and Evaluation Setup · p. 12
• Training uses 14,116 deduplicated English DAPO-Math-17K prompts, with 128 prompts per step and one rollout per prompt.Semi-OPD samples responses from the initial student before training; OPD uses the same prompt order, and both share the sampled-token reverse-KL objective and evaluation protocol.Source: A. Training and Evaluation Setup · p. 12
• Runs use NeMo-RL for training and vLLM for rollout generation and evaluation, on one to eight nodes of eight H100 GPUs.Table 2 lists training and evaluation hyperparameters.Source: A. Training and Evaluation Setup · p. 12, Table 2 · p. 12
• Table 3 reports top-64 overlap, last-step and AUC comparisons, with gains measured as avg@32 on a four-benchmark mean.Training rollout caps are shared within each pair; evaluation uses 32,768 tokens except for one 40,960-token pair.Source: Table 3 · p. 13

Rollout length

• OPD rollouts were 7–113% longer than Semi-OPD’s in 15 of 16 pairs; the exception was R1-Distill-1.5B ←JustRL-1.5B.Growth was largest for low-overlap non-thinking Qwen3 pairs, where OPD rollouts often exceeded the teacher’s length.Source: Figure 13 · p. 14, B. Training Dynamics Across All Pairs · p. 14

Positively supervised tokens

• Semi-OPD kept a larger share of positively supervised tokens (𝑝𝑇> 𝑝𝑆) than OPD in 15 of 16 pairs.R1-Distill-1.5B ←JustRL-1.5B was the exception.Source: Figure 14 · p. 15, B. Training Dynamics Across All Pairs · p. 15

Student and teacher entropy

• For thinking pairs, OPD drove student and teacher entropy below Semi-OPD’s in 7 of 8 pairs; for non-thinking pairs, teacher entropy was usually higher under OPD.The thinking-pair exception was R1-Distill-7B ←DeepCoder-14B; student entropy stayed similar between methods on non-thinking pairs.Source: B. Training Dynamics Across All Pairs · p. 16, Figure 15 · p. 16, Figure 16 · p. 17

Teacher–student overlap

• Top-64 teacher–student overlap generally rose under both methods as students moved toward teachers, but the ordering differed across model pairs.Figures 13–17 track these properties for 16 pairs, excluding the Nemotron-Cascade-2 pair.Source: B. Training Dynamics Across All Pairs · pp. 14, 18, Figure 17 · p. 18

Training cost

• Semi-OPD generates rollouts and scores them with the teacher once before training, removing online rollout generation and teacher inference from the training loop.The cost comparison reports H100 GPU-hours for 100 training steps, excluding evaluation.Source: C. Training Cost Comparison · p. 19, Table 4 · p. 19
• With online work removed, all GPUs can be used for rollout generation, teacher log-probability scoring, and student training.Relay-OPD instead interleaves student and teacher generation throughout training, leading to higher training cost.Source: C. Training Cost Comparison · p. 19
• For a fixed student, Semi-OPD’s rollout cache can be reused across teachers; evaluating another teacher requires only its log-probabilities on cached rollouts.On two pairs, Semi-OPD uses 4K training rollouts while OPD uses 16K; both methods use 16K on the other two pairs.Source: C. Training Cost Comparison · p. 19

Measuring self-correction

• Self-correction word density counts case-insensitive whole-word hits from a fixed word list in generated completions, normalized by rollout characters.The list includes words such as “wait,” “actually,” and “however”; prompts are excluded.Source: D. Length and Self-Correction Inflation · p. 20
• Because the word list includes common discourse words such as “but” and “no,” the measure is a proxy for reflective or hedging style, not verified corrections.The teacher reference uses 128 samples at T=1.0 with the same rollout cap on the final training step’s prompts.Source: D. Length and Self-Correction Inflation · p. 20

Length and self-correction inflation

• OPD rollouts grow longer and use more self-correction words, eventually exceeding the teacher, while frozen rollouts sampled from the initial student do not.The figure tracks mean generated tokens per rollout and self-correction word density.Source: Figure 18 · p. 20, D. Length and Self-Correction Inflation · p. 20

E. Qualitative Examples: OPD versus Semi-OPD Training Rollouts

• 5,688 tokens for a Semi-OPD rollout reaches an answer, whereas the paired OPD trace reaches its 16,384-token cap without a final answer.The Semi-OPD trace reports the positive solution as 13.Source: E. Qualitative Examples: OPD versus… · pp. 21–23
• The OPD sum trace repeatedly revises its place-value reasoning, including conflicting accounts of what a hundreds-place total of 32 means.The excerpt includes incompatible explanations of whether the hundreds digit is 3 or 2.Source: E. Qualitative Examples: OPD versus… · p. 21
• 2,228 tokens for Semi-OPD versus 9,016 for OPD; both traces reach an answer, though the OPD reasoning is substantially longer.The OPD trace includes repeated checking before arriving at the positive solution.Source: E. Qualitative Examples: OPD versus… · p. 22
• 308 tokens for Semi-OPD reaches an answer, while the OPD rollout hits its 4,096-token cap.The OPD trace continues questioning the prompt and does not provide a final answer in the excerpt.Source: E. Qualitative Examples: OPD versus… · p. 23