PaperlayerView exact revision on arXiv

Source-linked AI summary

CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts

Naibin Gu, Qingyi Si, Chenxu Yang, Chuanyu Qin, Junhao Zhou, Peng Fu, Zheng Lin, Weiping Wang

arXiv:2609.10154v1cs.LG

TL;DR

Cross-family OPD can fail to transfer stronger teacher capabilities even after tokenizer alignment, motivating a method that isolates capability-sensitive supervision. CompassOPD removes the cross-family offset, transfers within-family likelihood changes, and anchors updates to the student’s initial policy; it consistently outperforms standard cross-family OPD, with gains up to 5.50 points in average reasoning accuracy.

  • Problem

    Despite tokenizer alignment, cross-family OPD does not reliably preserve same-family benefits, and stronger external teachers may yield nearly identical student performance.

  • Method

    CompassOPD removes the cross-family offset, transfers within-family likelihood changes, and anchors updates to a frozen copy of the student’s initial policy.

  • Results

    CompassOPD consistently outperforms standard cross-family OPD across three student families and multiple teacher families, improving average reasoning accuracy by up to 5.50 points.

  • Takeaways & Limitations

    Relative likelihood changes within the teacher family provide a basis for transferring teacher-capability improvements across model families, including with an MoE-derived reference.

  • Takeaways & Limitations

    Due to computational constraints, CompassOPD was not evaluated on very large models.

Abstract

from arXiv · show

On-policy distillation (OPD) provides dense token-level supervision on student-generated trajectories. Although OPD performs strongly when teacher and student belong to the same model family, we find that its effectiveness degrades in cross-family settings even after tokenizer alignment, with substantially stronger external teachers offering little additional improvement. To understand this disconnect, we decompose the cross-family OPD signal into two components: an offset between a low-capability teacher-family reference and the student, and the within-family log-likelihood shift from that reference to the strong teacher. Standard OPD transfers both components together, allowing the offset to dominate the update direction and obscure the changes associated with teacher capability improvements. We propose CompassOPD, which removes this offset and transfers the within-family shift, while a frozen student reference anchors updates to the student's initial policy. Thus, both teacher-side and student-side changes are measured within their respective model families. Experiments across three student families and multiple teacher families show that CompassOPD consistently outperforms standard cross-family OPD, improving average reasoning accuracy by up to 5.50 points. For an MoE teacher, we further construct the reference directly from the teacher checkpoint by reducing expert activation, eliminating the need for a separate reference checkpoint while retaining a 3.43-point gain over OPD.

1 Introduction

Cross-family OPD remains ineffective despite tokenizer alignment: stronger external teachers do not reliably improve student performance. CompassOPD separates cross-family offsets from within-family teacher changes and consistently improves cross-family distillation.

  • Motivation: Despite vocabulary alignment, cross-family OPD plateaus quickly, and substantially stronger external teachers produce nearly identical student performance.Same-family OPD instead produces sustained improvements, showing that tokenizer alignment alone does not preserve OPD benefits across families.
  • Motivation: Model families are release series such as Qwen3, Qwen3.5, or Ministral-3, which can differ in vocabulary and training recipes.These differences motivate using stronger teachers from other families and aligning likelihoods over shared textual units.
  • Problem analysis: Standard cross-family OPD mixes a pre-existing cross-family offset with the within-family likelihood shift associated with stronger teacher capability.When the offset determines the signal’s sign, the strong teacher changes only the update magnitude rather than its direction.
  • Proposed method: CompassOPD removes the cross-family offset, transfers within-family likelihood changes, and anchors updates to a frozen copy of the student’s initial policy.This measures teacher-side and student-side variation within their respective model families.
  • Main results: CompassOPD consistently outperforms standard cross-family OPD across three student families and multiple teacher families, improving average reasoning accuracy by up to 5.50 points.The method also supports teacher-scale improvements under a fixed teacher-family reference and can use an MoE-derived reference without a separate checkpoint.

2 Preliminaries

OPD evaluates student-generated actions with teacher likelihoods to provide dense, on-policy token-level supervision. Cross-tokenizer alignment converts student and teacher representations into comparable textual units before applying the same comparison to OPD variants.

  • On-Policy Distillation: Given a student-generated trajectory, OPD scores each sampled action with the teacher under the same student-visited context and uses the resulting signal for policy updates.Positive signals encourage sampled actions, while negative signals discourage them.
  • On-Policy Distillation: The token-level OPD signal is t = log T(y_t | c_t) − log π_θ(y_t | c_t), comparing teacher and student likelihoods for the sampled action.The signal is detached and used as an advantage in the policy update.
  • Cross-Tokenizer Likelihood Alignment: Cross-tokenizer alignment decodes the student response, re-encodes it with the teacher tokenizer, and aligns sequences at shared textual boundaries.The resulting units may represent one-to-one token matches or spans containing different numbers of teacher and student tokens.
  • Cross-Tokenizer Likelihood Alignment: The same text-space alignment procedure is applied to standard cross-family OPD and CompassOPD so alignment differences do not confound their comparison.Each aligned unit receives a model-specific log-likelihood score under its preceding textual context.

3 Understanding Cross-Family OPD

Cross-family OPD can plateau despite teacher capability differences, because standard supervision combines a shared cross-family offset with the teacher-family capability shift. Decomposing these signals reveals why the offset can obscure capability-related updates.

  • 3.1 Same-Family and Cross-Family OPD: Cross-family OPD plateaus after limited early gains, whereas same-family OPD sustains reasoning improvements despite cross-family teacher-trajectory SFT.
  • 3.1 Same-Family and Cross-Family OPD: 16.7 points separate the Qwen3.5-35B-A3B and Qwen3.5-4B teachers, but their distilled Qwen3-4B students differ by only 0.1 points on average.
  • 3.2 Dissecting cross-family OPD signals: Qwen3.5-0.8B agrees in update direction with Qwen3.5-35B-A3B on 70.0% of aligned units and with Qwen3.5-4B on 71.8%.The shared directions persist despite substantial capability differences within the teacher family.
  • 3.2 Dissecting cross-family OPD signals: The cross-family signal decomposes into an offset O induced by reference R and a within-family shift ∆T from R to strong teacher T.O captures the baseline discrepancy between the low-capability reference and student, while ∆T captures capability-related likelihood changes.
  • 3.2 Dissecting cross-family OPD signals: Standard OPD transfers O + ∆T, so a larger offset can determine the update direction when it conflicts with the teacher-family shift.CompassOPD therefore removes O and transfers ∆T as the capability-sensitive signal.

4 CompassOPD

CompassOPD scores the same student-generated actions with a strong teacher and a within-family reference, removing the cross-family offset. It combines the resulting teacher-family shift with a frozen student reference to anchor updates to the initial policy.

  • 4 CompassOPD: CompassOPD removes the shared cross-family offset and uses the within-family log-likelihood shift to supervise student-generated trajectories.A frozen student reference further anchors updates according to displacement from the initial policy.
  • 4.1 A Capability Signal from the Teacher Family: The student likelihood cancels when strong-teacher and reference scores are subtracted, leaving a teacher-family-only capability signal.Positive ∆T means the strong teacher assigns higher likelihood to the same action than the reference; negative ∆T means lower likelihood.
  • 4.2 Anchoring the Student Update: CompassOPD combines the aligned teacher-family shift ∆T,t with the student’s displacement from its frozen reference policy.The stop-gradient operator is used, and α controls the strength of the student reference anchor.
  • 4.2 Anchoring the Student Update: At initialization, ∆S = 0, so updates are determined entirely by the teacher-family shift; subsequently, the anchor opposes further movement in the favored direction.This provides feedback that keeps the student update relative to its starting policy.
  • 4.3 MoE Self-Reference: An MoE teacher can supply the reference without a separate checkpoint by reducing expert activation while keeping frozen weights shared.The standard activation configuration serves as T, and the reduced-activation configuration serves as R.

5 Experiments

Across three student and teacher families, CompassOPD consistently improves cross-family distillation over standard OPD, while component analyses support removing the offset, anchoring the student, and using within-family scale differences.

  • 5.2 Main Results: CompassOPD consistently outperforms standard OPD across all five cross-family configurations spanning three teacher and three student families.
  • 5.2 Main Results: CompassOPD improves 20 of 21 student–benchmark combinations with Qwen3.5-35B-A3B, including gains for Granite4.1-3B, Qwen3-4B, and OLMo-3-7B.
  • 5.2 Main Results: CompassOPD improves 13 of 14 teacher–benchmark combinations for Qwen3 and Mistral teachers, with one tie, extending the gains beyond Qwen3.5.
  • 5.3 Component Analysis: Performance peaks at α = 0.5, while stronger anchoring reduces performance; restoring more of the cross-family offset monotonically decreases accuracy toward OPD.The anchor accounts for the student’s displacement from its initial policy, whereas offset restoration removes most of CompassOPD’s gain.
  • 5.4 Teacher and Reference Configurations: With a fixed Qwen3.5-0.8B reference, CompassOPD performance decreases as teacher scale approaches the reference, yet a 2B teacher still outperforms standard OPD with a 35B-A3B teacher.The results associate wider within-family scale separation with more informative likelihood shifts.
  • 5.4 Teacher and Reference Configurations: MoE self-reference improves average accuracy over OPD by 3.43 points without a separate reference checkpoint, though it trails the dedicated 0.8B reference by 2.07 points.The self-reference reduces expert activation to approximately 2B active parameters.

6 Related Work

Related work addresses cross-family distillation through sequence-level transfer, representation or distribution alignment, and alternative OPD objectives, whereas CompassOPD targets the cross-family offset directly.

  • Sequence-level distillation avoids tokenizer mismatch through teacher-generated text, while DSKD and ULD align representations or distributions across models.
  • On-policy distillation provides teacher feedback on student-generated trajectories, with prior work varying divergence objectives and studying transfer compatibility.
  • RLCSD and W2S-OPD use contrastive or within-family differences for distinct goals, while CompassOPD removes the cross-family offset for capability transfer.

7 Conclusion

The paper identifies the cross-family offset as a central limitation of standard OPD and proposes relative within-family likelihood changes with student-side anchoring as a more effective interface for capability transfer.

  • Standard cross-family OPD mixes the cross-family offset with within-family likelihood changes, masking capability-sensitive supervision.
  • CompassOPD removes the offset, transfers relative teacher-family likelihood changes, and anchors updates to the student’s initial policy.
  • Experiments across three student families and multiple teacher families consistently favor CompassOPD, while MoE self-reference removes the need for an independent small-scale checkpoint.

Limitations

The evaluation does not cover very large models because of computational constraints, although experiments span multiple teacher and student sizes.

  • Very large models were not evaluated because of computational constraints.The authors identify validation at substantially larger scales as future work.

A Implementation Details

The implementation aligns teacher and reference likelihoods on student trajectories, while using matched initialization and fixed teacher-side references for OPD comparisons.

  • The main experiments pair Qwen3.5-35B-A3B with Qwen3.5-0.8B as teacher and reference models for Granite4.1-3B, Qwen3-4B, and OLMo-3-7B-Instruct-SFT.Additional experiments vary teacher families and teacher-reference scale separation for Granite4.1-3B.
  • Both standard OPD and CompassOPD start from the same cold-start SFT checkpoint in every comparison.The checkpoint is initialized using teacher-generated reasoning trajectories to reduce cross-family differences in response style, format, and reasoning length.
  • Training uses teacher-generated datasets, full-parameter cold-start SFT, and distillation without task-correctness rewards.The datasets include OpenThoughts-114k for teacher-generated responses, while DAPO-Math or DeepScaleR-Preview supplies later rollouts according to student capability.
  • Teacher and reference scores are aligned over shared text boundaries, with span averaging used when tokenizations differ.This reduces scale differences caused by tokenizer granularity before distillation signals are assigned.
  • CompassOPD assigns the aligned teacher-family likelihood shift to student tokens and computes student-reference displacement at each position.An alignment mask excludes positions where consistent text boundaries cannot be formed, and both methods use the same alignment procedure.
  • Self-reference experiments reduce MoE expert activation while keeping model weights, tokenizer, and context fixed.The strong configuration activates eight fine-grained experts, while the reference activates one and retains the shared expert; their aligned log-likelihood difference defines ∆T.

A.6 Evaluation Protocol

Evaluation samples multiple responses independently for each benchmark problem and reports response-averaged accuracy, including an unweighted mean across seven benchmarks.

  • Evaluation uses a maximum generation length of 24,576 tokens with temperature 0.7, top-p 1.0, top-k 40, and presence penalty 2.0.Models use their native chat templates during evaluation.
  • Each model independently samples 16 responses per benchmark problem, and accuracy averages over all sampled responses.Scores are produced using EvalScope’s automatic scoring pipeline.
  • The Avg. column is the unweighted mean of the seven benchmark accuracies.

B Analysis of the CompassOPD Target Policy

The analysis studies the CompassOPD objective at a fixed context with teacher-side and reference quantities held fixed, then establishes its upper bound and optimizer.

  • The objective is analyzed at a fixed context while ∆T and πref remain fixed over a shared aligned textual action space.The analysis assumes α > 0, positive πref, and finite quantities.
  • The equality condition establishes the optimality of the policy specified in Equation 11.The derivation proceeds by substituting the identity associated with Equation 11 into the objective.
  • The objective is bounded above by α log Z(c), with equality exactly when π(· | c) = π∗(· | c).The bound follows because Z(c) is independent of π and the KL divergence is nonnegative.
Loading 2609.10154v1…