PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillation

Youxu Shi, Yifan Sun, Dacheng Yin, Haomiao Tang, Guangting Wang, Fengyun Rao, Jing Lyu, Dong Liu

arXiv:2609.37170v1cs.LGcs.AI

TL;DR

Teacher-generated distillation traces can be difficult for students to learn from, while student-generated rollouts may contain erroneous reasoning. IPD interpolates student and teacher next-token distributions and accelerates sampling while preserving the interpolated policy. Across text and multimodal reasoning benchmarks, it outperforms endpoint, two-stage, and heuristic segment-interleaving baselines.

  • Problem

    Teacher-generated trajectories tend to be high-quality but lie far from the student’s distribution, whereas student-generated rollouts are more learnable but can contain erroneous reasoning.

    Source: §1 · p. 1

  • Method

    IPD defines each next-token distribution as a controllable linear interpolation between student and teacher policies and uses speculative sampling to preserve that distribution exactly.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Across text-only and multimodal reasoning benchmarks, IPD consistently outperforms off-policy SFT, vanilla OPD, their SFT-then-OPD combination, and recent segment-interleaving methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support explicit rollout-policy design as a promising direction for effective distillation.

    Source: §5 · p. 10

    The prototype applies one objective to all tokens, although teacher-favored tokens can make divergence signals large and destabilize optimization.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Off-policy and on-policy distillation have traditionally been formulated as separate paradigms, each favoring a different property of distillation trajectories. Teacher-generated (off-policy) traces are typically high-quality but lie far from the student's distribution, whereas student-generated (on-policy) rollouts are more learnable but often contain erroneous reasoning. We view these paradigms as the endpoints of a policy continuum and posit that a more effective rollout policy may lie in between. We introduce \textbf{Interpolated Policy Distillation (IPD)}, which defines the next-token distribution at every decoding step as an explicit linear interpolation between the student and teacher distributions. The interpolation operates at the distribution level, token by token, and its coefficient provides direct control over the balance between trajectory quality and student learnability. Naively sampling from this policy would require sequentially querying the teacher at every token and is thus expensive. To make IPD practical, we accelerate it with a new speculative-decoding rule while exactly preserving the interpolated next-token distribution.At the trajectory level, the resulting rollouts naturally interleave student- and teacher-generated segments. Unlike recent heuristic segment-interleaving methods, however, this interleaving is induced by an exactly realized token-level interpolated policy rather than by hand-designed switching rules. Across text-only and multimodal reasoning benchmarks, IPD consistently outperforms both endpoint policies (SFT and OPD), their conventional two-stage combination (SFT-then-OPD), and recent heuristic segment-interleaving methods, demonstrating that token-level policy interpolation better balances trajectory quality and student learnability.

1 INTRODUCTION

Off-policy traces tend to be high-quality but hard for students to learn from, while on-policy rollouts are learnable but can contain erroneous reasoning. IPD interpolates between teacher and student token distributions, then samples that policy efficiently while preserving it exactly; across reasoning benchmarks it outperforms endpoint, two-stage, and heuristic-interleaving baselines.

1 INTRODUCTION

  • Teacher-generated trajectories tend to be high-quality but visit prefixes students rarely encounter, whereas weak students’ own rollouts can enter erroneous or degenerate prefixes.These endpoint properties create a tension between trajectory quality and student learnability.Source: §1 · pp. 1–2
  • IPD defines each next-token distribution as a linear interpolation between student and teacher policies, with γ controlling the balance between them.At γ = 0 it recovers vanilla OPD; at γ = 1 it rolls out from the teacher.Source: §1 · p. 2
  • A speculative sampler reduces sequential teacher decoding by proposing student tokens in blocks and using teacher-guided corrections while exactly preserving the interpolated next-token distribution.Direct sampling from the interpolated policy otherwise requires a sequential teacher query at every token.Source: §1 · p. 2
  • Unlike heuristic segment-interleaving methods, IPD specifies a token-level target distribution first and derives a distribution-preserving sampling procedure from it.The resulting rollouts can interleave accepted student proposals with teacher-guided corrections.Source: §1 · p. 2
  • On a representative setting, the IPD prototype improves over OPD even with the vanilla OPD objective applied to every token, pointing to the interpolated policy as the source of the gain.The prototype shows the same improvement trend as accelerated IPD despite having no explicit interleaving.Source: Figure 2 · p. 3, §1 · p. 3
  • Across text-only and multimodal reasoning benchmarks, IPD outperforms off-policy SFT, vanilla OPD, SFT-then-OPD, and recent segment-interleaving methods.The reported scope includes both base and instruction-tuned students.Source: §1 · p. 3

2 A TRAJECTORY-POLICY CONTINUUM

This section treats off-policy and on-policy distillation as endpoints of a continuum of rollout policies. IPD defines intermediate policies by interpolating student and teacher next-token distributions, providing a per-step measure of their position while distinguishing that from distance between whole trajectories.

2.1 THE POLICY CONTINUUM

  • At each prefix, IPD defines mγ(· | st) = (1 − γ)πθ(· | st) + γπT(· | st), with γ = 0 and γ = 1 recovering the student and teacher policies.Intermediate γ values define the policies explored by IPD; per-prefix interpolation is not interpolation between complete trajectory distributions.Source: §2.1 · pp. 3–4

2.2 THE TWO ENDPOINTS AND THE INTERIOR

  • The student endpoint avoids rollout-distribution shift but offers no quality guarantee, while teacher rollouts can be high-quality yet difficult to learn from when the teacher–student gap is large.A weak student may enter erroneous prefixes, while teacher trajectories visit prefixes unlikely under the student.Source: §2.2 · p. 4
  • Intermediate γ values introduce teacher guidance without reaching the teacher endpoint; judged trajectory quality rises with γ as learnability, measured by proximity to the student, falls.This opposing movement motivates seeking a balance between the two endpoint properties.Source: §2.2 · p. 4

2.3 RELATED WORK

  • Unlike hybrid methods that use method-specific intervention rules, IPD specifies the student–teacher combination as a next-token distribution and derives a sampler that preserves it.The related methods discussed include SKD, Relay-OPD, CA-OPD, and MInTRL.Source: §2.3 · p. 4

3 INTERPOLATED POLICY DISTILLATION

The section introduces an interpolated rollout prototype, then uses speculative sampling to preserve its target distribution efficiently and distinguish accepted proposals from teacher-guided corrections for supervision.

3.1 THE IPD PROTOTYPE AND ITS LIMITATIONS

  • With the vanilla OPD objective, the prototype converges faster and reaches higher final accuracy than vanilla OPD when only the rollout policy changes.The runs share objective, data, and optimization steps.Source: §3.1 · p. 5
  • Direct sampling from mγ requires sequential teacher queries and applies one objective to both student-plausible and teacher-favored tokens.The latter can produce large divergence log ratios and destabilize optimization, while likelihood supervision is preferable for them.Source: §3.1 · p. 5

3.2 SPECULATIVE SAMPLING FROM THE INTERPOLATED POLICY

  • Speculative sampling uses student-drafted blocks and one teacher pass to accept proposals or replace the first rejection, exactly sampling from mγ.After a rejection, the remaining draft suffix is discarded and drafting resumes from the corrected prefix.Source: §3.2 · pp. 5–6, Figure 3 · p. 6
  • The coefficient γ controls rejection frequency: rejection probability scales with γ and student–teacher total variation, while corrections favor teacher-weighted tokens.For small γ, accepted runs are longer and fewer teacher forward passes are needed.Source: §3.2 · p. 6

3.3 VERIFICATION-AWARE SUPERVISION

  • Accepted proposals receive OPD supervision, while teacher-guided corrections receive direct likelihood supervision within the same rollout.Corrections are tokens the student underweights relative to the teacher, where ratio-based objectives can be unstable.Source: §3.3 · p. 7
  • The accelerated method matches and slightly exceeds the prototype while using the same rollout distribution, indicating gains from differentiated supervision.Because speculative sampling preserves mγ, the comparison changes supervision rather than trajectories.Source: §3.3 · p. 7

4 EXPERIMENTS

Across text-only and multimodal reasoning experiments, IPD outperforms OPD in the reported settings, including after student pruning. Ablations show an intermediate teacher-intervention level performs best, while IPD is less sensitive than OPD to SFT initialization.

4.1 EXPERIMENTAL SETUP

  • Text-only experiments use three student–teacher settings and math benchmarks; multimodal experiments evaluate two Qwen3-VL student settings on five reasoning benchmarks.The multimodal settings include a student with its final three decoder layers removed.Source: §4.1 · pp. 7–8

4.2.1 UNIMODAL DISTILLATION

  • IPD achieves the highest average accuracy in both base-student settings, beating vanilla OPD by 1.93 points for Qwen3-1.7B-Base and 10.64 points for Qwen3-0.6B-Base.It also performs best on eight of nine benchmark–student combinations.Source: §4.2.1 · p. 8
  • IPD improves over vanilla OPD across all five benchmarks for the post-trained student, raising average accuracy from 41.37 to 42.18.The student is DeepSeek-R1-Distill-Qwen-1.5B.Source: §4.2.1 · p. 8

4.2.2 MULTIMODAL DISTILLATION

  • 44.64 average accuracy for IPD versus 27.90 for OPD after pruning, nearing the unpruned model’s 45.87 without an extra recovery stage.Table 2 reports Avg@32 accuracy across multimodal reasoning benchmarks.Source: §4.2.2 · p. 8, Table 2 · p. 9

4.3 ABLATION STUDY

  • On GSM8K, MATH-500, and AMC, performance peaks at γ = 0.1; γ = 1 still beats vanilla OPD but falls short of γ = 0.1.The decreasing γ schedule performs better than the increasing schedule in this setting.Source: Figure 5 · p. 8, §4.3 · p. 9
  • IPD is less sensitive than OPD to SFT cold-start strength and, without a cold start, scores 46.69 versus 45.41 for the best SFT + OPD result.The authors report this cold-start observation in a single setting.Source: §4.3 · p. 9, Figure 6 · p. 9

5 CONCLUSION

IPD links off-policy and on-policy distillation through a controllable token-level policy continuum, with speculative sampling preserving its interpolated distribution exactly. The paper reports benefits across text and multimodal reasoning and notes that benchmark improvements do not establish deployment safety or reliability.

5 CONCLUSION

  • IPD uses speculative sampling to preserve the interpolated distribution exactly while unifying OPD and SFT supervision within each trajectory.Accepted proposals receive OPD supervision, while teacher-guided corrections receive direct likelihood supervision.Source: §5 · p. 10

AI USE STATEMENT

  • Authors report AI assistance with translation, methodological discussions, literature discovery, drafting, editing, formatting, and figure design, with assisted content reviewed by the authors.They state that mathematical arguments, references, and experimental descriptions were checked.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The work uses existing models, datasets, and benchmarks, and benchmark gains do not establish deployment safety or reliability.The authors note that distillation may transfer teacher biases, factual errors, and unsafe behaviors.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement identifies the method, experimental setup, and ablations as specified in the paper and appendix.It says the method section specifies the rollout policy, speculative sampling, and token-level supervision, while the setup names models, datasets, and benchmarks.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EXACT SAMPLING FROM THE INTERPOLATED POLICY

  • Speculative sampling with the student as draft model and the interpolated policy as verifier produces exact samples from the interpolated policy.The guarantee follows from acceptance and residual-correction rules; the model parameters and interpolation coefficient are held fixed during each rollout.Source: §A.1 · p. 13

A.2 PROPERTIES OF INTERPOLATED VERIFICATION

  • The rejection rate depends on both student–teacher mismatch, measured by DTV(p, q), and intervention coefficient γ, while the correction distribution remains unchanged.The positive k1 component is reduced by a factor of 1−γ, and rejected-token cross-entropy gradients are bounded.Source: §A.2 · pp. 14–15

A.3 METRIC OF TRAJECTORY QUALITY AND LEARNABILITY

The appendix evaluates trajectory quality and student learnability across interpolation coefficients using an LLM judge and a student–teacher agreement proxy.

A.3 METRIC OF TRAJECTORY QUALITY AND LEARNABILITY

  • The authors assess both metrics on 1,024 trajectories across interpolation coefficients during training and average scores over the sampled set.The judge receives the original problem, including its image when applicable, and the response without information about the generating method or coefficient.Source: §A.3 · p. 15
  • Trajectory quality is scored by an independent LLM judge assessing response coherence and completeness on a 0–4 scale.Higher scores indicate more valid reasoning and a better-supported final answer.Source: §A.3 · p. 15
  • Student learnability is estimated using exponentiated negative mean absolute sampled-token k1, interpreted as an empirical proxy rather than a formal measure.Higher scores indicate closer student–teacher agreement on sampled tokens.Source: §A.3 · p. 15

A.4 RELAY-OPD FAILURE

In the reported weak-student Relay-OPD setting, entropy collapses, responses increasingly hit the length cap, and teacher control nearly disappears.

A.4 RELAY-OPD FAILURE

  • The teacher-controlled token ratio fell from 2.65% to 0.02%, suggesting degenerate generations failed to reliably trigger the handoff mechanism.The authors caution that limited teacher intervention in this setting does not indicate healthy student rollouts.Source: §A.4 · p. 16
  • Within 30 steps, policy entropy fell from 2.659 to 0.005 while responses reaching the length limit rose from 3.1% to 95.3%.Table 3 reports the training dynamics for Qwen3-0.6B-Base with Qwen3-4B as teacher.Source: Table 3 · p. 16, §A.4 · p. 16
  • From step 20 onward, teacher-controlled tokens accounted for less than 1% of rollout tokens among the reported checkpoints, despite decreasing distillation loss.Most responses reached the 7,168-token cap as policy entropy collapsed.Source: Table 3 · p. 16

A.5 DEEPSEEK-R1-DISTILL-QWEN-1.5B: A STRONGER STUDENT SETTING

With a reasoning-trained 1.5B student, IPD improves average accuracy over vanilla OPD across five reasoning benchmarks, though the gain is smaller than for weaker base students.

A.5 DEEPSEEK-R1-DISTILL-QWEN-1.5B: A STRONGER STUDENT SETTING

  • The evaluation uses DeepSeek-R1-Distill-Qwen-1.5B as student and Skywork-OR1-7B as teacher, trained on 25,600 OpenThoughts3 math questions.The setup evaluates Avg@32 accuracy on five reasoning benchmarks.Source: §A.5 · p. 16
  • Average accuracy rose from 41.37 to 42.18 with IPD over vanilla OPD across five reasoning benchmarks.The reported metric is Avg@32 accuracy.Source: §A.5 · p. 16, Table 4 · p. 17
  • The gain is smaller than for weaker base students, but the authors report benefits for a student with substantial prior reasoning training.They state that improvements are larger when initial student capabilities are limited.Source: §A.5 · p. 16

A.6 RESULTS OF γ ANNEALING

On Qwen3-0.6B-Base, decreasing γ from 0.5 to 0.1 yields higher final accuracy across all three benchmarks than increasing it to 0.9. The results suggest that gradually shifting toward student-generated rollouts is more effective in this setting.

A.6 RESULTS OF γ ANNEALING

  • Decreasing γ yields larger gains on MATH-500 and AMC23, while increasing γ produces less consistent progress, including a temporary AMC23 decline.The comparison uses Qwen3-4B as the teacher.Source: §A.6 · p. 17
  • The results suggest that gradually shifting toward student-generated rollouts is more effective than strengthening teacher intervention in this setting.Source: §A.6 · p. 17
  • Decreasing γ from 0.5 to 0.1 achieves higher final accuracy than increasing it to 0.9 across all three benchmarks.Figure 7 reports this comparison for Qwen3-0.6B-Base.Source: Figure 7 · p. 17, §A.6 · p. 17

A.7 SFT TOKEN ABLATION

On GSM8K, IPD reaches 70.02% accuracy without an SFT cold start, exceeding SFT+OPD at every tested SFT token budget, including 60×. SFT+OPD accuracy rises with its budget, but the authors report that integrating SFT and OPD supervision within rollouts uses SFT tokens more effectively.

A.7 SFT TOKEN ABLATION

  • SFT+OPD GSM8K accuracy increases from 45.97% at 1× to 68.57% at 60× the SFT token budget.The 1× budget matches the number of tokens receiving SFT supervision within IPD.Source: §A.7 · p. 17, Figure 8 · p. 17
  • Under small γ, fewer than 1% of IPD tokens receive SFT supervision, yet IPD outperforms SFT+OPD even at a 60× SFT budget.The authors conclude that integrating SFT and OPD supervision within rollouts uses SFT tokens more effectively than applying the objectives successively.Source: §A.7 · p. 17, Figure 8 · p. 17
  • 70.02%: IPD accuracy on GSM8K without an SFT cold start, exceeding SFT+OPD at all tested budgets, including 60×.Figure 8 compares final accuracy after SFT+OPD with IPD.Source: Figure 8 · p. 17, §A.7 · p. 17

B TRAINING AND EVALUATION SETTING

The training-setting materials include configurations for vanilla OPD and IPD with Qwen3 base students, and for original and pruned Qwen3-VL-2B-Instruct students. Evaluations use a specified vLLM generation setup.

B TRAINING AND EVALUATION SETTING

  • Table 5 covers vanilla OPD and IPD training configurations for Qwen3-0.6B-Base and Qwen3-1.7B-Base with Qwen3-4B.Source: Table 5 · p. 18
  • Table 6 covers configurations for original and pruned Qwen3-VL-2B-Instruct students; the pruned student removes the final three decoder layers.Both settings share the listed configurations.Source: Table 6 · p. 18
  • Evaluations use vLLM with temperature 0.6, maximum generation length 8,192 tokens, top-p 0.95, and top-k 20.Source: B TRAINING AND EVALUATION SETTING · p. 18

Related papers

Loading 2609.37170v1…