PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang

arXiv:2609.36546v1cs.LGcs.AI

TL;DR

On-policy distillation trains students on their own trajectories, but teacher feedback may become unreliable when those trajectories drift far from the teacher’s likely states. IPD intervenes adaptively with a bidirectional propose-and-verify process and source-specific supervision; across teacher–student pairs it reports stronger distillation performance and data efficiency, with a 30.44 mean@8 result for Qwen3-1.7B-Base.

  • Problem

    On-policy distillation can query teachers on student-generated states they would hardly visit, where feedback may be unreliable.

    Source: §1 · p. 2

  • Method

    IPD uses a bidirectional propose-and-verify state machine to intervene in student rollouts and applies reverse KL to student-generated tokens and forward KL to teacher-generated tokens.

    Source: §1 · p. 2

  • Results

    Across diverse teacher–student pairs, IPD shows better distillation performance and higher data efficiency; Qwen3-1.7B-Base achieves 30.44 mean@8 accuracy, +2.00 over OPD and +1.64 over SKD.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    IPD demonstrates better robustness to training data and provides a unified bridge between on-policy and off-policy distillation.

    Source: §5 · p. 10

    The study considers same-tokenization distillation; extending IPD to cross-tokenization scenarios remains future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.

1 INTRODUCTION

The introduction motivates distillation as a way to transfer large models’ capabilities to more deployable students, while noting that on-policy distillation can still suffer from teacher unanchoring. It presents IPD’s adaptive teacher intervention and source-split training, alongside a fused inference engine and broad empirical gains.

1 INTRODUCTION

  • Knowledge distillation transfers capabilities from large models into compact, cheaper students suited to latency-sensitive, on-device, or high-throughput deployment.The introduction identifies high memory and computation costs as a constraint on deploying the largest models.Source: §1 · p. 1
  • IPD alternates student and teacher proposer–verifier roles to correct drifting rollouts, then applies reverse KL to student tokens and forward KL to teacher tokens.This source-split objective is described as bridging on-policy and off-policy distillation.Source: Figure 1 · p. 2, §1 · p. 2
  • Naive OPD can query the teacher on student-generated states it would hardly visit, where its feedback may be unreliable.The introduction calls this failure mode teacher unanchoring.Source: §1 · p. 2
  • Across four teacher–student pairs on mathematical reasoning, IPD achieves the highest benchmark-averaged mean@8 and best@8 accuracy and leads on most individual benchmark metrics.The introduction also reports higher data efficiency across these experiments.Source: §1 · p. 3
  • The fused inference engine co-hosts both models, maintains separate KV caches, and schedules requests by their current proposer and verifier roles.It is designed to make dual-model rollout scheduling more efficient.Source: §1 · p. 3

2 RELATED WORKS

Related work traces the shift from teacher-generated traces toward student-generated training in distillation, and reviews methods addressing unreliable teacher feedback. IPD is distinguished by dynamically swapping proposer and verifier roles and applying source-specific supervision to the resulting mixed-source trajectories.

2 RELATED WORKS

  • Off-policy distillation trains students on teacher-authored traces, while OPD uses student sequences; GKD and MiniLLM exemplify on-policy approaches, and DistiLLM variants explore skew divergences and scheduling.OPD has also entered reasoning-model post-training pipelines.Source: §2 · p. 3
  • EOPD adjusts the loss on student rollouts, TrOPD combines a teacher prefix with student continuation, and SKD dynamically corrects student tokens while keeping proposer and verifier roles fixed.The cited related-work discussion describes these as responses to unreliable feedback on student rollouts.Source: §2 · p. 3
  • IPD differs from SKD by dynamically swapping proposer and verifier roles through teacher takeover and handback, then training the mixed-source trajectory with a source-split objective.The discussion relates this intervention-and-return pattern to interactive imitation-learning approaches.Source: §2 · p. 3

3 METHODOLOGY

IPD addresses teacher unanchoring by alternating student and teacher control during rollout, then applying token-level losses according to token source. Its fused inference engine supports this mixed-source process, while top-K objectives offer a less costly alternative to full-vocabulary supervision.

3.1 PRELIMINARIES

  • Top-K OPD uses multi-token supervision over selected high-probability support, trading a biased approximation for lower computation than full-vocabulary KL and lower variance than single-token estimates.The support is formed from the K highest-probability indices; selected distributions are normalized over that support.Source: §3.1 · p. 4

3.2 INTERACTIVE-POLICY DISTILLATION

  • Incorrect student trajectories elicit high teacher uncertainty even as reverse KL decays with position, motivating adaptive intervention against teacher unanchoring.The analysis uses student responses split by final-answer correctness and reports average entropy and reverse KL over position intervals.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.2.1 ADAPTIVE TEACHER INTERVENTION WITH BIDIRECTIONAL PROPOSE-AND-VERIFY

  • Student proposals pass when covered by the teacher’s top-K support; otherwise the teacher replaces them and retains control until a teacher token falls within the student’s top-K support.That student-covered teacher token triggers handback to student control.Source: §3.2.1 · p. 5

3.2.2 SOURCE-SPLIT KL OBJECTIVE

  • IPD applies reverse KL to student-generated positions and forward KL to teacher-generated positions, evaluating both at the realized mixed-source prefix.The response trajectory carries source labels that determine which positions receive each loss.Source: §3.2.2 · p. 6
  • IPD recovers on-policy distillation when control stays with the student and off-policy distillation when the teacher takes over permanently after the first proposal.These endpoint settings pair pure student trajectories with reverse KL and pure teacher trajectories with forward KL.Source: §3.2.2 · p. 6

3.2.3 FUSED INFERENCE ENGINE FOR EFFICIENT DUAL-MODEL SCHEDULING

  • 21.4× faster initially and 71.4× faster at the final checkpoint, the fused engine outperforms a naive two-engine rollout implementation.The comparison uses identical settings and memory budget on 512 DAPO-MATH-17K prompts.Source: §3.2.3 · p. 6, Figure 3 · p. 6
  • The fused engine verifies draft chunks, updates driver models and both KV caches at accepted-prefix boundaries, then regroups requests for the next proposer stage.Each proposer extends its own cache; the counterpart catches up during the next verification prefill.Source: §3.2.3 · pp. 6–7, Figure 4 · p. 7

4 EXPERIMENTS

Across the evaluated teacher–student combinations, IPD improves on the distillation baselines and reaches higher accuracy with fewer training examples. Ablations examine loss and coverage-width choices, while a second training dataset tests robustness.

4.1 EXPERIMENTAL SETTINGS

  • The experiments train four Qwen3 teacher–student pairs on DAPO-Math-17K for one epoch and evaluate final checkpoints on six math reasoning benchmarks.Teachers use disabled thinking mode; IPD and SKD use proposal chunks of length 5, and the main experiments set K = 16.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 30.56 mean@8: for Qwen3-30B-A3B to Qwen3-1.7B-Base, IPD exceeds OPD by 3.28 and SKD by 1.74 across benchmarks.The reported comparison covers multiple teacher–student combinations and benchmarks.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • Fewer than 1/4 of training examples and 60 steps let IPD outperform OPD and match SKD trained for a whole epoch of 279 steps.During training, IPD also shows a stably increasing top-K overlap ratio; Figure 5 presents accuracy and overlap curves.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 ABLATION STUDY

  • The default source-split KL—reverse KL on student-generated positions and forward KL on teacher-generated positions—achieves the best mean@8 among the tested loss variants.Pure reverse KL is the runner-up for mean@8 and has best@8 accuracy 0.12 higher than the default.Source: §4.3 · p. 8, Figure 6 · p. 9
  • K = 4 yields 30.46 mean@8 and 45.80 best@8; default K = 16 is only 0.02 and 0.05 lower, respectively, with a much lower teacher intervention ratio.Smaller K increases takeovers and reduces handbacks, raising the intervention ratio.Source: §4.3 · p. 9, Figure 7 · p. 9

4.4 ROBUSTNESS ON DIFFERENT TRAINING DATA

  • On DeepMath-GE8-17K, intervention-based IPD and SKD resist training-data changes, while OPD drops 6.75 mean@8 in the Qwen3-30B-A3B to Qwen3-4B-Base group.The comparison uses DAPO-MATH-17K and 17,085 examples selected from DeepMath-103K at difficulty level 8 or higher.Source: Figure 8 · p. 9, §4.4 · p. 10

5 CONCLUSION AND DISCUSSION

IPD combines bidirectional takeover and handback with token-provenance-based supervision to address teacher unanchoring, and reports stronger distillation performance, data efficiency, and robustness. Its scope is limited to same-tokenization distillation and synchronous verification and proposal scheduling.

5 CONCLUSION AND DISCUSSION

  • IPD combines bidirectional takeover and handback with source-split supervision, reporting better distillation performance, data efficiency, and robustness across diverse teacher–student pairs.The paper describes the method as a bridge between on-policy and off-policy distillation and uses a fused inference engine for dual-model rollout scheduling.Source: §5 · p. 10
  • The experiments consider only same-tokenization distillation, leaving cross-tokenization scenarios for future exploration.The paper also leaves alternative top-K coverage rules and asynchronous rollout scheduling for future work; its current verification and proposal scheduling is synchronous.Source: §5 · p. 10

A DETAILED EXPERIMENTAL SETTINGS

The experiments use four Qwen3 teacher–student pairs on DAPO-Math-17K, a common training configuration, top-K distillation, and six math benchmarks. The reported evaluation uses eight sampled responses per problem and mean@8 and best@8 accuracy.

A DETAILED EXPERIMENTAL SETTINGS

  • Main experiments use DAPO-Math-17K with four Qwen3 teacher–student pairs, and disable thinking mode for all teachers.The pairs use Qwen3-1.7B-Base or Qwen3-4B-Base students and Qwen3-4B, Qwen3-8B, or Qwen3-30B-A3B teachers.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 14
  • Each model is trained for one epoch of 279 steps at learning rate 1 × 10^-6, with four rollouts per prompt and a maximum generation length of 7168.Rollouts use temperature 1.0 and top-p = 1.0; the 8192-token maximum sequence includes a maximum prompt length of 1024.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 14
  • All methods use top-K distillation with K = 16 and compute each KL term after separately renormalizing student and teacher distributions over the union support.IPD and SKD use K = 16 for takeover coverage and proposal length γ = 5; IPD also uses symmetric top-K coverage for handback.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 14
  • Final checkpoints are evaluated on six math benchmarks using eight sampled responses per problem, with mean@8 and best@8 accuracy reported for each benchmark and their averages.Evaluation sampling uses temperature 0.7 and top-p = 0.95, with a maximum generation length of 7168.Source: A DETAILED EXPERIMENTAL SETTINGS · p. 14

B.1 SAMPLED-TOKEN OPD

The appendix explains sampled-token OPD’s reverse-KL estimator and its variance under teacher–student disagreement, then records choices for distillation support, loss aggregation, and student-model settings.

B.1 SAMPLED-TOKEN OPD

  • The sampled-token loss is an unbiased estimator of full-vocabulary reverse KL in expectation, although individual realizations can be negative while the full loss is nonnegative.The appendix also relates the sampled-token OPD gradient to a policy-gradient view and gives the token loss as log q(xt | ct) − log πθ(xt | ct).Source: §B.1 · pp. 14–15
  • Strong teacher–student disagreement can make sampled-token losses and gradients fluctuate, producing noisy and potentially unstable updates.The stated source is sampling tokens where the teacher probability is much smaller than the student probability, yielding larger log-ratios.Source: §B.1 · p. 15

B.2 DISTILLATION SUPPORT

  • For different-size distillation, the authors observe slightly better top-K OPD performance using the union of the student and teacher top-K sets as support, and use it by default.They report no significant support-choice differences for same-size distillation, referring to an official repository issue discussion.Source: §B.2 · p. 15

B.3 LOSS AGGREGATION

  • Sequence-mean-token-mean aggregation weights responses equally, while global token mean weights rollouts in proportion to their lengths; the former is the default for slightly more stable optimization.The authors report comparable performance for the two aggregation choices.Source: §B.3 · p. 15

B.4 STUDENT MODELS

  • Qwen3 Base students provide the paper’s cleaner setting when teachers have thinking mode disabled; using non-base models with thinking enabled as both teacher and student would require longer responses and more compute.The latter setting is left for future exploration.Source: §B.4 · pp. 15–16

C.1 ADDITIONAL MAIN RESULTS

The additional-results figure presents training curves for mean@8 accuracy and top-K overlap ratio across four model groups.

C.1 ADDITIONAL MAIN RESULTS

  • Figure 9 plots mean@8 accuracy and top-K overlap ratio curves during training for four groups of models.The accompanying text identifies these as additional results alongside the main results in Table 1.Source: §C.1 · p. 16, Figure 9 · p. 16

C.2 ADDITIONAL ABLATION RESULTS

The additional ablations show that the default source-split KL leads benchmark-averaged mean@8, while K=4 only narrowly exceeds K=16; same-origin distillation shows no statistically significant intervention advantage over OPD.

C.2 ADDITIONAL ABLATION RESULTS

  • 30.44 benchmark-averaged mean@8 is highest with the default source-split KL, while pure reverse KL reaches a slightly higher best@8 of 45.87 versus 45.75.Source: Table 2 · p. 17, §C.2 · p. 17
  • K=4 yields the highest average mean@8 and best@8, exceeding default K=16 by only +0.02 and +0.05, respectively.Source: Table 3 · p. 17, §C.2 · p. 17

D SAME-ORIGIN DISTILLATION

E DISTILLATION ON LIMITED TRAINIG EXAMPLES

With limited training examples, IPD still outperforms OPD and SKD, while all three methods match the full-data student’s performance using about 1/17 of the training data.

E DISTILLATION ON LIMITED TRAINIG EXAMPLES

Loading 2609.36546v1…