PaperlayerView exact revision on arXiv

Source-linked AI summary

Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts

Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard

arXiv:2610.10460v1cs.LGcs.AI

TL;DR

MOPD transfers teacher endpoints that mix post-training changes with preferences inherited from each teacher’s base, and whether teacher-relative shifts are better transfer targets in either MOPD setting had not been tested. The paper compares endpoint supervision with Δ-MOPD, which re-anchors each teacher-minus-base shift at the student’s initialization while holding teacher selection fixed. Shifts show their clearest benefits when signals are combined, with supporting gains under phased routing but comparable results under interleaved routing.

  • Problem

    MOPD endpoint supervision can transfer preferences inherited from a teacher’s base, and the value of teacher-relative differences in either MOPD setting had not been tested.

    Source: §1 · p. 1

  • Method

    Δ-MOPD transfers each teacher’s endpoint-minus-base logit shift re-anchored at the student’s initialization, with teacher selection held fixed against endpoint supervision.

    Source: §1 · pp. 1–2

  • Results

    With three composed teachers, Δ-MOPD exceeds endpoint composition by 4.11 Math and 1.95 five-benchmark points; phased routing improves both orders, while interleaved routing is comparable.

    Source: §1 · p. 2, §6 · p. 6

  • Takeaways & Limitations

    Target construction is an MOPD design axis independent of teacher selection, and shift targets are most useful when teacher signals are combined at a state.

    Source: §1 · p. 2

    Δ-MOPD requires each teacher’s precursor and a compatible tokenizer or an explicit projection rule, unlike black-box OPD.

    Source: §9 · p. 7

Abstract

from arXiv · show

Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher's endpoint policy, which mixes what post-training changed with preferences inherited from the teacher's base. We introduce $\Delta$-MOPD, which transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio and target--student KL. Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, $\Delta$-MOPD exceeds endpoint composition by $4.11$ Math and $1.95$ five-benchmark points; with two, it matches endpoint accuracy. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from $10.50$ to $6.42$ points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection.

1 Introduction

The paper asks whether MOPD should transfer teachers’ endpoints or their post-training changes, since endpoints can include preferences inherited from a different base. It introduces Δ-MOPD as a target-construction alternative and finds its clearest gains when teacher signals are combined.

1 Introduction

  • Endpoint supervision transfers inherited base differences when a specialist’s base differs from the student’s initialization, a common situation for public specialists.Whether teacher-relative differences improve transfer in either MOPD setting had not been tested.Source: §1 · p. 1
  • Holding teacher selection fixed, Δ-MOPD measures each teacher against its own base and re-anchors that shift at the student’s initialization.Composition selects all teachers, while routing selects one; when a teacher shares the student’s base, both targets give the same term.Source: §1 · pp. 1–2
  • The paper frames target construction as a design axis independent of teacher selection, with Δ-MOPD reducing exactly to endpoint OPD for same-origin teachers.The proposed shift target is hypothesized to be closer to the student and, under composition, more balanced across teachers.Source: §1 · p. 2
  • Removing inherited base pull lowers the teacher-term norm ratio from 5.2:1 to 1.44:1 and reduces target–student KL fivefold.The inherited base pull can exceed the post-training shift.Source: §1 · p. 2
  • With three composed teachers, Δ-MOPD exceeds endpoint composition by 4.11 Math and 1.95 five-benchmark points; phased routing improves both orders, while interleaved routing is comparable.The observed phased-routing order gap falls from 10.50 to 6.42 points.Source: §1 · p. 2

2 Preliminaries

OPD trains on student-generated rollouts, using teacher scores at the states the student visits. MOPD varies which teachers contribute at each state, while endpoint and shift constructions distinguish inherited base differences from post-training changes.

2 Preliminaries

  • OPD minimizes token-level reverse KL between the student policy and the target distribution at student-generated states.The sampled-token reward is log target probability minus log student probability, used with a centered score-function estimator.Source: §2 · p. 2
  • A teacher’s shift is its endpoint logits minus its precursor logits; its inherited base difference is the precursor’s logits minus the anchor’s.The shift includes all post-training effects, including capability, style, and calibration.Source: §2 · p. 2
  • Common-domain composition selects all teachers at each state, whereas routed-domain distillation selects the teacher assigned to the prompt’s domain.Routed domains may be interleaved during training or assigned to contiguous phases.Source: §2 · pp. 2–3
  • Figure 1 contrasts endpoint terms, which include inherited base differences, with shift terms re-anchored at the student’s initialization.Teacher 1 is drawn as sharing the anchor base.Source: Figure 1 · p. 3, §2 · p. 3

3 ∆-MOPD

Δ-MOPD builds a target by re-anchoring teacher-relative logit shifts at the student’s initialization. Its reference-term interpretation distinguishes the shared initialization pull from endpoint supervision’s teacher-specific base pulls.

3.1 Construction

  • The shared-anchor target combines selected teachers’ relative probabilities with the anchor distribution and normalizes over the vocabulary.Because each model’s log-partition term is constant across tokens, the target can be formed by summing logits and normalizing once.Source: §3.1 · p. 3
  • The endpoint control replaces each teacher shift with its endpoint-minus-anchor logits, so the two targets differ by the inherited base differences.The comparison changes only the representation of cross-origin teacher terms.Source: §3.1 · p. 3
  • For a same-origin teacher, Δ-MOPD’s shift target equals its endpoint, so the constructions coincide when all selected teachers are same-origin.Only cross-origin terms differ between the two constructions.Source: §3.1 · p. 3

3.2 Reference-term interpretation

  • Endpoint supervision pulls toward each selected teacher’s base, while Δ-MOPD replaces those heterogeneous pulls with one pull toward the student’s initialization.At the start of training, that shared initialization pull is zero; under composition, removing inherited differences may keep a cross-origin term from setting its scale relative to other teachers.Source: §3.2 · p. 4

4 Experimental Setup

The experiments compare paired endpoint and shift targets while holding the teacher selection and run protocol fixed. The diagnostic metrics track composition geometry and target distance from the student and anchor.

4 Experimental Setup

  • Common-domain runs compose Nemotron and Polaris, then add JustRL for the three-teacher comparison; routing assigns Math to Polaris and Science/IF to Nemotron.Routing is tested with interleaved batches and two phased orders, alongside a Polaris-only acquisition run.Source: §4 · p. 4
  • The paired comparisons share initialization, prompts, optimizer, update budget, and evaluation, although their on-policy trajectories diverge.Frozen models score the same prefixes within each arm.Source: §4 · p. 4

4.1 Diagnostic metrics

  • Composition diagnostics include teacher-term norm ratio, centered-shift cosine, cancellation, and top-16 sign conflict; target–student and target–anchor KL use the full effective vocabulary.The reported forward target–student KL measures target tracking, not the training objective.Source: §4.1 · p. 4

5 Results

The results examine how inherited base differences affect composition geometry, then test shift targets in cross-origin acquisition, common-domain composition, and routed distillation. Benefits are clearest when teachers are composed at a state, with phased routing offering supporting evidence of gains across training phases.

5.1 Inherited base differences distort composition geometry

  • The endpoint teacher-term norm ratio falls from 5.2 to 1.44 under shift composition, while target–student KL falls from 0.152 to 0.031.Polaris’s base-reference pull has roughly twice the gradient norm of its shift; the inherited component has greatest relative influence while specialization is being acquired.Source: §5.1 · p. 4, Figure 2 · p. 5

5.2 A closer target accelerates cross-origin acquisition

  • At step 100, ∆-MOPD leads the English-Math macro by 2.87 pp; it scores 47.90% at 28.0 allocated H100-hours versus Endpoint’s best evaluated 46.54% at 42.9 hours.At initialization, the endpoint target is 4.94× farther from the student in target–student KL across 16,384 shared prefix states.Source: §5.2 · p. 5

5.3 Balanced shifts improve common-domain composition

  • With two teachers, ∆-MOPD and endpoint composition reach similar aggregate accuracy: 44.91% versus 45.14% under Avg@K and 37.73% versus 36.25% under greedy decoding.∆-MOPD is higher on Science/IF under both decodings, while endpoint composition is 3.4 pp higher on MATH-500.Source: §5.3 · p. 5
  • At M = 3, ∆-MOPD exceeds endpoint composition by 4.11 pp on the Math macro and 1.95 pp on the five-suite macro.Adding same-origin JustRL raises shift composition by 4.40 Math and 1.78 five-suite pp, versus 0.43 and 0.58 for endpoint composition.Source: §5.3 · p. 5, Table 2 · p. 6

5.4 Shift targets show consistent gains under phased routing

  • Under phased routing, ∆-MOPD finishes above its same-order endpoint control in both orders, while the phase-order gap falls from 10.50 pp to 6.42 pp.Under interleaved routing, the five-suite macro differs by only 0.62 pp in favor of ∆-MOPD, and the targets are described as comparable.Source: §5.4 · p. 6, Table 3 · p. 6

6 Discussion

Shift targets show their clearest benefits when several teachers are composed at a state; phased routing offers supporting evidence of benefits when signals accumulate across phases. The discussion also notes that base subtraction does not calibrate differences in logit temperature or confidence.

Where shifts help

  • The clearest benefits arise when several teachers are composed at a state; phased routing is supporting evidence, while interleaved routing with one teacher per update yields comparable targets.The paired design holds same-origin terms fixed and changes only the representation of Polaris.Source: §6 · p. 6

Geometry and scale

  • Shift norms reflect both post-training change and checkpoint scale because the policy-ratio form does not calibrate logit-temperature or confidence differences.Calibration-aware composition is complementary to base subtraction.Source: §6 · p. 6

7 Related Work

Related work spans language-model distillation, multi-teacher on-policy distillation, and teacher-relative differences as transfer objects. This paper studies what each selected teacher contributes in both MOPD settings.

Distillation for language models

  • OPD scores student-generated rollouts, while prior work broadening the on-policy signal continues to match the teacher’s endpoint.Examples include rubric judgments, sequence- and token-level views, and context rather than weights.Source: §7 · p. 7

Multi-teacher on-policy distillation

  • Prior MOPD work acts on teacher selection, scheduling, or rehearsal; this work varies what each selected teacher contributes.Routing, counteraction, and cascaded overwrite motivate the paired controls.Source: §7 · p. 7

Differences as transfer objects

  • Prior delta-oriented methods use teacher–base differences for single-teacher transfer, while task arithmetic and proxy tuning compose differences offline; this work tests them as MOPD transfer objects on student prefixes.The differences are normalized under one softmax.Source: §7 · p. 7

8 Conclusion

The conclusion presents teacher-relative shifts re-anchored at the student’s initialization as an alternative to endpoint supervision. It reports their clearest gains in within-state composition and supporting evidence under phased routing, with target construction complementary to teacher selection.

8 Conclusion

  • Inherited base differences can dominate a teacher’s post-training shift; removing them yields a more balanced target closer to the student.The construction reduces exactly to endpoint OPD for same-origin teachers and acts only on inherited base differences.Source: §8 · p. 7
  • Shift targets match endpoint accuracy with two composed teachers and raise Math and five-benchmark accuracy with three.An unpaired cross-tokenizer extension adds a projected fourth shift and yields further gains on both macros.Source: §8 · p. 7

9 Limitations •

The experiments compare endpoint supervision with teacher-relative shifts across composition, routed distillation, and acquisition. They report benchmark outcomes, diagnostic trends, evaluation scope, and implementation requirements.

9 Limitations •

  • The evaluation studies public reasoning specialists and common-domain composition on Math prompts, with Science/IF assessed through held-out evaluation; broader specialist families and domain-mixed composition remain extensions.Δ-MOPD also requires each teacher’s precursor and a compatible tokenizer or explicit projection rule, unlike black-box OPD.Source: §9 · p. 7

A Protocols and Reproducibility

  • The reported evaluation uses 1,309 frozen held-out items, with five-seed means and standard deviations where error bars are shown; paired arms share prompts, updates, and evaluation settings.The protocol specifies distinct response-length caps for IF-Eval and other evaluations, and the scaling run and teacher references use a 10,000-token greedy protocol.Source: A Protocols and Reproducibility · p. 9

C.1 Mechanism run: capability

  • 37.73% versus 36.25% group-balanced greedy totals favor Δ-MOPD over endpoint composition; greedy English-Math and Science/IF macros also favor Δ-MOPD, while Avg@K is effectively tied.The respective group macros are 50.91% versus 48.16% for English-Math and 24.54% versus 24.34% for Science/IF.Source: §C.1 · p. 11

C.2 Mechanism run: diagnostics

  • The two mechanism-run arms separate by step 20, and their ordering remains stable through the reported checkpoints.Tables 7 and 8 cover late-training and early checkpoints, respectively.Source: §C.2 · p. 11

C.3 Scaling run

  • At M = 3, shift composition has a final target-to-anchor KL of 0.483 versus 0.695 for endpoint composition, with lower target displacement, reward variance, and gradient magnitude across recorded scaling diagnostics.The shift arm’s per-update time is 11.0% higher because it also scores the Polaris precursor.Source: §C.3 · p. 11

D Routed-Domain Distillation Details

  • Interleaved routing assigns Math prompts to Polaris and Science and instruction-following prompts to Nemotron, with one teacher per prompt and domains mixed from the first update.The training mixture is 50% Math, 25% Science, and 25% instruction-following.Source: D Routed-Domain Distillation Details · p. 11

D.1 Phased routing

  • In phased routing, the Science/IF phase is identical across methods, so the contrast lies in the Polaris Math phase; only one of ten benchmark-by-order comparisons is significant before correction.The sole comparison below p = 0.05 is MATH-500 in Math → Science/IF (56 versus 36 discordant items, p = 0.047), which does not survive correction.Source: §D.1 · p. 11

E Cross-Origin Acquisition and Compute

  • At step 100, Δ-MOPD leads Endpoint by 0.91 pp on the GPQA-Diamond and IF-Eval Avg@4 macro, while greedy decoding favors Endpoint by 1.07 pp.The English-Math macro values across steps 100/150/195 are 47.90/45.65/47.68 for Δ-MOPD and 45.03/46.18/46.54 for Endpoint.Source: E Cross-Origin Acquisition and Compute · p. 11

E.1 Compute accounting

Compute accounting charges allocated H100 device time, including rollout generation and teacher, precursor, and anchor scoring. Δ-MOPD reaches comparable or higher reported accuracy with fewer updates, despite slower per-update training.

E.1 Compute accounting

  • One hour of training on four allocated H100s counts as four H100-hours, and timed compute includes rollouts, optimization, batching, and teacher, precursor, and anchor forward passes.The measure is allocated device time, not FLOPs, and does not adjust for utilization.Source: §E.1 · p. 12
  • 35% fewer allocated H100-hours are needed for Δ-MOPD to exceed Endpoint’s best evaluated English-Math macro: 28.0 versus 42.9 H100-hours.The reduction is reported as 34.7% before rounding and comes from requiring fewer updates, not cheaper updates.Source: §E.1 · p. 12
  • Δ-MOPD stabilizes near step 75 versus step 125 for Endpoint, at 21.0 versus 27.5 H100-hours; at comparable hours, its block-mean rollout accuracy is 53.38% versus 51.12%.The plateau is defined by later 25-step block means varying by at most 2.5 pp.Source: §E.1 · p. 12

F Cross-Tokenizer Extension

The cross-tokenizer extension projects a Qwen3 teacher shift onto the anchor vocabulary using token-string overlap. Adding that projected shift improves both reported macros, though no paired endpoint control is defined.

F Cross-Tokenizer Extension

  • The Qwen3 shift is computed within the Qwen3 family and transferred only on overlapping action coordinates in the anchor vocabulary; only <think> and </think> require remapping.Five anchor-only special tokens receive zero Qwen3 shift, and Qwen3-only special tokens are dropped.Source: F Cross-Tokenizer Extension · p. 13
  • No paired endpoint control is defined because cross-tokenizer endpoint logits are not placed on the anchor vocabulary.The projection is narrower than text-span likelihood alignment for general cross-tokenizer OPD.Source: F Cross-Tokenizer Extension · p. 13
  • 36.82 versus 33.50 Math macro and 28.04 versus 26.19 five-suite macro result from adding the projected fourth shift to the M = 3 shift composite.All three Math benchmarks improve, with the largest gain on AIME 2025, while Science/IF does not improve.Source: F Cross-Tokenizer Extension · p. 13, Table 13 · p. 14

Related papers

Loading 2610.10460v1…