PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Delta Distillation

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

arXiv:2607.15161v1cs.LGcs.CL

TL;DR

On-policy distillation’s loss design remains underexplored. This paper introduces OPD², which distills the difference between a reasoning-tuned teacher and its base model, and finds it consistently outperforms conventional methods across mathematics, science, and code benchmarks.

  • Problem

    The fundamental design of on-policy distillation rewards remains underexplored, despite the method’s importance in post-training research.

  • Method

    OPD² uses the delta between a reasoning-tuned teacher and its pre-post-training base model as the principal distillation reward.

  • Results

    OPD² consistently achieves the highest average performance across model sizes and reasoning domains, including mathematics, science, and code.

  • Takeaways & Limitations

    The delta signal is the primary source of OPD²’s performance gains, while agreement and centering provide smaller additional improvements.

  • Takeaways & Limitations

    OPD² adds teacher-base forward computation, increasing wall-clock training time by approximately 24–28% for Qwen3 models and 8% for Gemma4-E4B.

Abstract

from arXiv · show

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

1 Introduction

The introduction identifies a design gap in On-Policy Distillation (OPD): directly distilling a reasoning-tuned teacher may preserve pre-existing preferences rather than isolate acquired reasoning knowledge. It proposes using the teacher–base-model delta signal, with centering and joint conditioning, and evaluates the approach across math, science, and code.

  • Motivation and contribution: OPD directly trains students to follow teacher outputs, whereas OPD^2 uses the difference between the teacher and its base model to target reasoning knowledge.The delta signal is intended to capture the learning trajectory associated with reasoning tuning rather than the teacher’s retained pre-tuning preferences and style.
  • Scope: Unlike variants that extend OPD learning signals beyond sampled tokens [5] [6] [7], this paper restricts attention to the original sampled-token setting.The introduction notes that sampled-token supervision preserves the student’s original knowledge and ability throughout post-training.
  • Problem: OPD’s fundamental loss design remains underexplored because its reward is simply the log probability difference between teacher and student.The introduction focuses on the learning trace of reasoning tuning as a direction for improving OPD’s basic reward design.
  • Method: The method introduces centering and joint conditioning to use the delta signal more effectively during on-policy distillation.Centering subtracts expected rewards from the policy model’s sampling probabilities, while joint conditioning addresses difficulty identifying the learning signal’s direction and negative effects from sign bias.
  • Experiments: The verification framework mixes Math, Science, and Code data in a 1:1:1 ratio and evaluates distilled models on 7 Math, 3 Science, and 4 Code benchmarks.The training set samples equal numbers of questions from one dataset for each domain before on-policy distillation.

2 Method

OPD trains a student on on-policy responses using token-level rewards equivalent to minimizing teacher–student KL divergence. OPD² instead uses a delta signal derived from the teacher’s changes relative to its base model, targeting reasoning-specific knowledge while addressing potential convergence instability.

  • On-Policy Distillation: OPD samples responses from the student and trains next-token predictions to mimic the teacher by minimizing KL divergence.Its sampled-token reward is R_t = log π*(y_t | x, y_<t) − log π_θ(y_t | x, y_<t), applied only to sampled tokens.
  • Signal Analysis: Compared with OPD, delta emphasizes reasoning-connective words such as hence, however, and instead, while suppressing exploratory and verification-related words such as see, try, and verify.The comparison uses Qwen3-1.7B students, Qwen3-4B teachers, and 10k math questions.
  • Delta Signal: The delta signal subtracts the teacher’s base-model signal, capturing the teacher’s learned reasoning trace rather than basic next-token preferences.It is used as the primary learning signal because the distillation target is reasoning ability beyond ordinary next-token prediction.
  • Signal Analysis: Delta more consistently suppresses tokens associated with incorrect reasoning than OPD in simple reasoning examples.The token-level signals mark promoting and suppressing tokens, with values clipped to ±6 for visualization; delta is also more negative on generic reasoning expressions and erroneous intermediate steps.
  • Signal Analysis: Across math, code, and science, delta tends to enhance explicit logical-connection words while suppressing vague uncertainty expressions.The statistical analysis covers 10k questions per domain and identifies words with large changes in distillation strength.
  • Convergence Consideration: Because delta omits the student signal from reward computation, optimizing it can converge toward a one-hot maximum-reward token and potentially destabilize training.The authors note that this convergence point is not reachable in practical strong-to-weak settings but may still cause instability.

3 Experiment

Across Qwen3 and Gemma4 reasoning models, OPD2 consistently outperforms conventional on-policy distillation across mathematics, code, and science, with gains persisting throughout training. Ablations identify the delta signal as the primary contributor, while its additional computation remains limited relative to ExOPD.

  • Main Results: OPD2 achieves the highest average performance across model sizes and domains, including Math scores of 62.7, 74.8, and 75.9 for Qwen3-1.7B, 4B, and 8B.On Qwen3-1.7B, 4B, and 8B, OPD2 improves the original Math models by 3.5, 1.5, and 2.2 points, respectively; on HMMT25 for Qwen3-8B, it improves 44.3 to 52.3.
  • Main Results: The advantage transfers to Gemma4: OPD2 improves Math average performance from 60.6 to 67.8 and retains more Code capability than OPD and ExOPD, despite not surpassing the original Code model.On Gemma4 Code, OPD2 scores 49.5 versus 36.9 for OPD and 45.1 for ExOPD, and improves RGAlgo from 62.2 to 64.0.
  • Training Dynamics: OPD2 maintains a higher performance trajectory than OPD and ExOPD after the early training stage, whereas the competing methods often plateau or degrade.The persistent separation is especially clear on CodeContests, Math, and Science; main-table results use the final training step rather than the best checkpoint.
  • Ablation and Cost: Replacing the delta signal with the standard OPD signal causes the largest ablation degradation, while removing agreement or centering has smaller effects.OPD2 requires an additional teacher-base forward pass, increasing wall-clock training time by approximately 24-28% for Qwen3 and 8% for Gemma4-E4B, while remaining comparable to ExOPD.

4 Related Work

Post-training commonly uses SFT and RL for reasoning alignment, while on-policy distillation transfers outputs from a high-performing teacher to a student [37]. SFT includes teacher-generated sequence training and later large-model-generated sequence training [38–40], and subsequent work modifies or analyzes OPD in several ways.

  • 4 Related Work: On-policy distillation transfers a high-performing teacher’s outputs to a student as an alternative post-training approach based on knowledge distillation [37].The passage situates OPD alongside SFT and RL for aligning LLMs with reasoning tasks.
  • 4 Related Work: SFT is itself a form of knowledge distillation, including Sequence-KD from teacher-generated sequences and later training on large-model-generated sequences [38–40].
  • 4 Related Work: Follow-up studies modify OPD’s learning signal, permit controlled off-policy data for efficiency, or remove the external teacher through privileged verified solutions [46] [47] [48] [49].These variants include uncertainty-based reweighting [46], controlled off-policy data [47], and single-LLM teacher-student setups [48] [49].

5 Conclusion

The paper examines the delta signal—the difference between teacher and base models—as a training signal for on-policy distillation, proposing that it can transfer reasoning knowledge from reasoning-tuned teachers. Analyses indicate that the delta signal contains meaningful reasoning information and offers benefits over the original approach.

  • 5 Conclusion: The study evaluates the delta signal, defined as the difference between the teacher and base models, for on-policy distillation.
  • 5 Conclusion: The authors conjecture that the delta signal can transfer reasoning knowledge acquired through teacher reasoning tuning from base models.
  • 5 Conclusion: Analyses show that the delta signal contains meaningful reasoning signals and provides benefits compared with the original approach.

A Appendix

The appendix specifies the model, optimization, hardware, and evaluation configurations used in the experiments. Evaluation reports pass@1 averaged over repeated stochastic generations, with 16 repetitions for AIME24 and AIME25 and fewer for larger benchmarks.

  • Model and hardware configuration: Experiments use NVIDIA H100 GPUs, with Qwen3-1.7B trained on one 8-GPU node and larger models trained across four nodes.Multi-node training allocates 24 GPUs to the student and 8 GPUs to the vLLM rollout server.
  • Model and hardware configuration: Qwen3 students use larger same-family teachers, while Gemma4-E4B-it uses Gemma4-31B-it in the thinking setting.The corresponding pretrained checkpoints without instruction tuning serve as teacher-base models for ExOPD.
  • Common training parameters: All methods are trained for 100 optimization steps with effective global batch sizes of 256 or 288 and AdamW at a 5 × 10−6 learning rate.Training uses cosine decay, 0.1 warmup and minimum learning-rate ratios, gradient clipping at 1.0, reward scaling by 0.1, maximum generation length 8192, temperature 0.7, and β = 0 for the KL coefficient.
  • Evaluation configuration: Evaluation reports pass@1 as single-sample accuracy averaged over independent repetitions, without using repetitions for best-of-k selection.AIME24 and AIME25 use 16 repetitions because they contain relatively few problems, whereas larger benchmarks use fewer repetitions.
Loading 2607.15161v1…