PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Rong Wu, Daocheng Fu, Yu Yang, Xuemeng Yang, Siqi Li, Shu Zou, Yao Zhu, Botian Shi, Bo Zhang, Cong Zhang, Pinlong Cai, Jianbiao Mei

arXiv:2610.13205v1cs.LG

TL;DR

Conventional on-policy distillation typically depends on a substantially stronger teacher, and it remains unclear which relative policy differences provide useful supervision and which targets can benefit. This paper derives an implicit-reward formulation for transferring proxy expert–reference differences to targets. It finds that task-directed differences transfer more consistently than scale-only differences, while success also depends on the reference and target compatibility.

  • Problem

    Relative policy transfer’s useful source differences and compatible targets remain unclear, limiting understanding of when it succeeds.

    Source: §2.2 · p. 3

  • Method

    The paper derives an any-to-any OPD objective that treats a proxy expert–reference log-probability difference as an implicit reward and recovers conventional OPD as a special case.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Task-directed differences from RL and SFT transfer consistently, whereas scale-only differences transfer weakly or unreliably; expert, reference, and target compatibility also shape transfer.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Effective transfer requires a difference that moves the target toward the expert and generalizes beyond sampled rollouts; an appropriate reference can rescue otherwise failed transfer.

    Source: §1 · p. 2

Abstract

from arXiv · show

On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.

1 Introduction

Conventional OPD depends on a teacher that substantially outperforms the student, while relative policy transfer instead transfers the improvement between an expert and a reference. This work derives a unified transfer objective and studies which differences and targets support effective transfer.

1 Introduction

  • Conventional OPD relies on a substantially stronger teacher, limiting its applicability and reuse of existing post-training outcomes.When teacher and student capabilities are comparable, or the teacher is weaker, distillation can yield limited gains or impair existing capabilities.Source: §1 · p. 1
  • Relative policy transfer compares a teacher before and after post-training, so the source need only outperform its corresponding base model, not the student.This broadens possible transfer across models of different scales and architectures and offers a way to reuse post-training outcomes.Source: §1 · pp. 1–2
  • The study separates whether a source pair encodes a useful improvement from whether a target can realize it.It examines differences induced by RL, SFT, and model scaling across targets with different initial policies.Source: §1 · p. 2
  • Task-directed improvements from RL or SFT transfer consistently, while differences from model scaling alone transfer weakly or unreliably.A stronger model is not necessarily a better source of supervision.Source: §1 · p. 2
  • The proxy expert supplies the capability, while the proxy reference filters shared behavior to expose the residual improvement for transfer.An appropriate reference can recover an otherwise failed transfer.Source: §1 · p. 2
  • The paper formulates an any-to-any OPD objective that distinguishes the proxy expert, reference, and target.Its evaluation varies policy differences and targets to examine how these models jointly shape transfer.Source: §1 · p. 2
  • The paper identifies task-directed training, reference filtering, and target compatibility as key factors governing transfer.These factors frame its characterization of when policy differences are useful and transferable.Source: §1 · p. 2

2 Related Works

On-policy distillation uses supervision on student-generated trajectories, but its reliance on teacher log-probabilities limits usefulness when teacher and student policies are too similar. Relative policy transfer instead applies the expert–reference log-ratio as a signal on student rollouts, while the conditions for successful transfer remain incompletely understood.

2.1 On-Policy Distillation

  • OPD combines student-generated trajectories with token-level teacher supervision, typically minimizing reverse KL to avoid forcing a lower-capacity student to model the teacher’s long tail.Its on-policy signal improves sample efficiency, reduces distribution shift, and mitigates catastrophic forgetting associated with off-policy SFT.Source: §2.1 · p. 2
  • OPD becomes uninformative when teacher and student policies are too similar because its reward comes from teacher log-probabilities on student trajectories.Production examples include Qwen3’s use of OPD for logit alignment and MiMo-V2-Flash’s use of multiple domain-specialized RL teachers.Source: §2.1 · p. 2

2.2 Relative Policy Differences in Post-Training

  • Relative policy transfer uses the expert–reference log-ratio on student rollouts, transferring a direction of improvement rather than matching an absolute distribution.The source need not outperform the student; it must outperform its corresponding reference.Source: §2.2 · p. 3
  • Prior work mostly evaluates specific source pairs on students from the same generation and similar scale, leaving transfer across different priors and capability levels unclear.This study varies both the source pair and target model to characterize effective differences and transfer boundaries.Source: §2.2 · p. 3

3 A Unified Policy Optimization View of OPD

The paper recasts conventional OPD as KL-regularized optimization and treats a proxy expert–reference difference as an implicit reward. Its any-to-any formulation applies that signal to a separately chosen target while retaining a target reference.

3.1 Conventional OPD as KL-Regularized Optimization

  • Conventional OPD can be viewed as KL-regularized optimization in which the teacher log-probability supplies reward and the target is optimized on its own visited states.The negative reverse KL is rewritten relative to the target’s initial policy, making the log-probability difference reward-like while the KL term anchors the policy to that reference.Source: §3.1 · p. 3

3.2 From Absolute to Relative Matching: An Implicit Reward Perspective

  • In conventional OPD, the reference in the reward-like term and KL regularizer cancels, reducing the objective to direct matching of the teacher distribution.This absolute matching can be restrictive when teacher and target output distributions are poorly aligned.Source: §3.2 · p. 3
  • A proxy expert is obtained from a proxy reference through KL-regularized optimization with token-level task reward, and their log-probability difference acts as an implicit reward.The difference preserves action-wise reward differences up to a positive scale set by the KL-regularization strength.Source: §3.2 · pp. 3–4
  • The proxy pair defines the transferred difference independently of the target reference, separating the source of the learning signal from the policy receiving it.The target can therefore use a reference distinct from the proxy reference.Source: §3.2 · p. 4

3.3 Any-to-Any Policy Transfer

  • A2-OPD transfers a proxy pair’s relative difference to a target with its own reference, and conventional OPD is recovered when the proxy expert and reference are the teacher and teacher base.The framework permits independent source–target combinations in principle, though not all transfer effectively; prior methods differ in proxy pair, target reference, and transfer strength.Source: §3.3 · p. 4, Table 1 · p. 4
  • The target update uses on-policy samples and combines the transferred difference with regularization toward its reference.Centering the action weight removes a state-only normalizer without changing the expected gradient and reduces variance; for language models the signal is evaluated token-wise on target trajectories.Source: §3.3 · pp. 4–5

4 When Do Relative Policy Differences Transfer?

Across math and code, transfer patterns persist, but useful policy differences do not benefit every target. Transfer depends on task-relevant updates, the expert–reference residual, and whether the target can move toward the expert and generalize.

4.1 Experiment Setup and Metrics

  • Experiments use Qwen3 and Qwen2.5 models, math and code tasks, and a shared transfer objective; reported diagnostics track performance gains and policy-space differences.Training and evaluation use temperature 1.0 and top-p 1.0; experiments use top-16 token distributions rather than the full vocabulary.Source: §4.1 · p. 5

4.2 Which Policy Differences Provide Useful Supervision?

  • Task-optimized differences from RL or SFT improve the target, while scale-only differences yield much smaller gains despite positive proxy capability gaps.An alternative reference also works; comparable induced-update magnitudes can produce sharply different target gains, indicating that task-relevant direction matters beyond magnitude.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3 Which Part of the Expert’s Capability Gets Transferred?

  • The proxy expert supplies capability, while the reference filters shared behavior and determines the residual presented for transfer.With Qwen3-4B-RL fixed as expert, target gains rise from 9.62 to 26.84 and 29.09 as expert–reference overlap falls from 0.916 to 0.839 and 0.625.Source: Table 3 · p. 7, §4.3 · pp. 7–8

4.4 Which Targets Can Realize a Useful Difference?

  • A useful difference benefits some targets but not others, and raw proxy–target distance does not predict the outcome.Qwen3 targets gain 9.78–29.09, while Qwen2.5 targets under a Qwen3 reference gain −2.46 to 2.72; changing to a Qwen2.5 reference yields positive gains of 6.74, 2.25, and 5.76. Successful updates move toward the expert and generalize beyond training examples.Source: Table 4 · p. 8, §4.4 · pp. 8–9, Figure 4 · p. 9

4.5 Do the Transfer Patterns Generalize across Domains?

  • Across math and code, tested configurations retain the same qualitative success or failure patterns.Qwen3-to-Qwen3 transfer is positive in both domains, while Qwen3-derived differences fail on Qwen2.5 targets in both.Source: Table 5 · p. 10, §4.5 · p. 10

5 Conclusion

A2-OPD formulates any-to-any relative policy difference transfer, with conventional OPD as a special case. Experiments identify which differences and targets support effective transfer.

5 Conclusion

  • A2-OPD decouples the proxy pair defining a relative difference from the target receiving it, and recovers conventional OPD as a special case.Source: §5 · p. 10
  • Differences induced by task-directed training, including RL and SFT, transfer consistently, while differences due to model scaling alone are often ineffective.Source: §5 · p. 10
  • Transfer depends on target compatibility: the induced update must move the target toward the expert and generalize beyond sampled rollouts; an appropriate reference can rescue some transfers.Source: §5 · p. 10

A.1 Implicit Reward Encoded by the Proxy Pair

The appendix derives the proxy-pair implicit reward and the policy induced by the any-to-any objective. The proxy pair sets relative reweighting, while the reference sets the distribution receiving it.

A.1 Implicit Reward Encoded by the Proxy Pair

  • Solving the state-wise optimization condition and imposing normalization establishes Eq. 6 for the implicit reward encoded by the proxy pair.Source: §A.1 · p. 13

A.2 Closed-Form Policy Induced by the Any-to-Any Objective

  • Solving the state-wise any-to-any objective and enforcing normalization yields the closed-form policy in Eq. 19.Source: §A.2 · pp. 13–14
  • The proxy pair determines relative reweighting, whereas πref determines the distribution to which it is applied; for β > 0, the state-wise maximizer is unique on πref's support.Source: §A.2 · p. 14

A.3 Gradient of the KL-Regularized Objective

The appendix derives the conditional gradient for the state-wise on-policy surrogate. It treats rollout states as fixed samples and distinguishes this gradient from the full trajectory gradient.

A.3 Gradient of the KL-Regularized Objective

  • The on-policy surrogate treats states collected by the current rollout policy as fixed samples, with no gradient propagated through dπθ.Source: §A.3 · p. 14
  • Because πref is fixed, the second term in Eq. 24 vanishes, and empirical expectation over states sampled from dπθ yields the Eq. 8 gradient estimator.Source: §A.3 · p. 14
  • For a full trajectory objective retaining gradients through the policy-induced state distribution, the exact score-function weight is the return-to-go or action value induced by uθ, not the single-step uθ(s, a).Source: §A.3 · p. 14

B Experimental and Implementation Details

The appendix specifies expert and SFT configurations, on-policy training settings, and performance and distributional diagnostics. Evaluation uses math and code benchmarks with domain-specific reporting metrics.

B Experimental and Implementation Details

D Complete Experimental Results and Analyses

The appendix reports validation performance and distributional evaluations across the experimental configurations, with tables covering policy differences and variations in the proxy expert, proxy reference, and target.

D Complete Experimental Results and Analyses

  • The appendix reports complete validation performance and distributional evaluations for the experimental configurations presented in the main text.It also includes validation performance for the base and expert checkpoints.Source: D Complete Experimental Results and… · p. 16
  • Table 7 covers configurations, validation results, and distributional and efficiency metrics for policy-difference experiments.Source: Table 7 · p. 17
  • Tables 8 and 9 separately vary the proxy reference and proxy expert while holding the other fixed.Both tables include validation, distributional, and efficiency metrics.Source: Table 8 · p. 18, Table 9 · p. 19
  • Table 10 varies the target while holding useful differences fixed and reports validation, distributional, and efficiency metrics.Distributional and efficiency metrics are not reported for ID 3 because it uses standard OPD.Source: D Complete Experimental Results and… · p. 17, Table 10 · p. 20
Loading 2610.13205v1…