PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Xin Li, Hao Jiang, Xin Gao, Annan Wang, Yuchen Xie, Jinghao Guo, Xingwei Qu, Yichi Zhang, Chau Yuen

arXiv:2609.35347v1cs.LG

TL;DR

RL-trained specialists must be integrated into one deployable model, but routing prompts to domain experts alone does not calibrate how their feedback updates the shared student. DN-MOPD rescales each domain’s feedback by measured spread; across three Qwen3.5 sizes, it improves the six-task average over MOPD at both answer-length caps.

  • Problem

    RL yields domain experts that excel in separate domains, but a deployable model must integrate them.

    Source: §1 · p. 1

  • Method

    DN-MOPD retains expert routing and rescales each domain’s distillation advantages using its batch-measured teacher–student log-ratio spread.

    Source: §1 · p. 2

  • Results

    Across Qwen3.5 9B, 4B, and 2B, DN-MOPD raises the six-task average over MOPD by 1.17–2.36 percentage points at 16K and 2.47–3.08 at 8K.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Controls show most gains come from limiting instruction-following feedback; fixed weights near DN-MOPD’s measured multipliers perform comparably at 9B and 4B.

    Source: §1 · p. 2

    Evidence covers one expert pool per size in one model family; student seeds do not cover expert retraining, and benefits depend on the teacher–student configuration.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

1. Introduction

RL-trained specialists need to be integrated into one model, but MOPD routing selects a teacher without controlling how much its feedback counts. DN-MOPD calibrates feedback by domain spread and improves the six-task average over MOPD across Qwen3.5 sizes and evaluation budgets.

1. Introduction

  • 1.17–2.36 percentage points at 16K and 2.47–3.08 at 8K: DN-MOPD improves the six-task average over MOPD across Qwen3.5 9B, 4B, and 2B.It exceeds the strongest single-teacher student at every size and recovers most of the lost mathematics gain; the advantage holds across three student seeds.Source: Figure 1 · p. 1, §1 · pp. 2–3
  • MOPD with label routing does not outperform the strongest single-teacher student at any tested size and transfers little of the mathematics expert’s gain.At 8K, the student retains only 14–32% of that gain; at 16K, it shows no gain over its initialization.Source: §1 · p. 2
  • Instruction-following feedback is 2.3–4.4 times as dispersed as the pooled first-batch signal, while mathematics feedback is about half as dispersed.For the initial 4B student, instruction-following loss supplies 94% of the combined gradient under equal weights.Source: §1 · p. 2
  • DN-MOPD measures each domain’s log-ratio spread and uses a bounded multiplier to bring its distillation advantages toward the pooled scale.It retains label routing and fresh student answers, adding no teacher model, teacher call, or learned router; MOPD is the all-multipliers-equal-one case.Source: §1 · p. 2
  • Fixed-weight controls attribute most gains to limiting instruction-following feedback rather than amplifying mathematics alone.Fixed weights near DN-MOPD’s measured multipliers perform comparably at 9B and 4B, while DN-MOPD calibrates from batch statistics without a per-size weight search.Source: §1 · p. 2

2. DN-MOPD

DN-MOPD retains prompt-domain teacher assignment but calibrates feedback magnitudes before updating the shared student. It measures token-level teacher–student log-ratio spread per domain against a pooled batch reference and rescales each domain’s advantages.

2. DN-MOPD

  • The student samples answers and the domain-matched specialist scores each token; DN-MOPD retains this routing and adds per-domain rescaling before the shared update.The setup uses one student and separately trained mathematics, code, and instruction-following specialists.Source: §2 · p. 3
  • In initial Qwen3.5 batches, instruction-following log-ratios are 2.3–4.4 times the pooled spread and mathematics log-ratios about half as dispersed; labels do not calibrate scale.All domains update the same student parameters, while resulting domain gradients also depend on their directions and the loss reduction.Source: §2 · pp. 3–4
  • Each batch measures population standard deviation over valid tokens within each domain and across domains, then rescales domain advantages relative to the pooled spread.This targets dispersion, not teacher quality or the remaining capability gap.Source: §2 · pp. 3–4
  • Positive multipliers preserve advantage signs, while bounds of [0.25, 4] limit adjustments and can prevent full equalization; degenerate spreads use a multiplier of 1.Spread matching does not equalize domain gradient norms or fix the batch loss scale.Source: §2 · p. 4
  • In Qwen3.5 runs, mathematics feedback is amplified about 1.5–2.1 times at every size, code stays near 1 at 4B and 2B, and instruction-following often hits the 0.25 floor.The operation reuses rollout log-ratios and adds no teacher call; it changes neither teacher assignment nor domain prompt counts.Source: §2 · p. 4

3. Experiments

The experiments compare DN-MOPD with label-routed OPD across Qwen3.5 sizes, then examine capability transfer, feedback scales, and evaluation and training budgets. DN-MOPD improves the main comparison across sizes and evaluation budgets, while controls and extensions probe why and when the advantage holds.

3. Experiments

  • The experiments use independently trained Qwen3.5 specialist pools at 9B, 4B, and 2B, with matched experts, initialization, prompts, and 80 updates for main OPD comparisons.Evaluation covers mathematics, code, and instruction following; the main tables use a 16,384-token cap.Source: §3 · pp. 5–6

3.1. Main results

  • DN-MOPD improves on Label at every model size and both evaluation budgets, with paired intervals above zero and gains across three student seeds.The main comparison holds experts, initialization, prompts, and update budget fixed.Source: §3.1 · p. 6
  • DN-MOPD improves mathematics where Label shows no gain over the initial student, while code and instruction-following gains are smaller and less consistent.At 9B, code does not improve at 16K; SeqKD-SFT and task arithmetic retain higher Totals under their own recipes.Source: §3.1 · pp. 6–7

3.2. Understanding the improvement

  • Each specialist improves its own domain, but Label retains little mathematics gain; mathematics-only OPD retains more, indicating difficulty when domain feedback is combined.The expert effects on other domains vary.Source: §3.2 · p. 7
  • Fixed-weight controls show that reducing instruction-following feedback alone recovers most of DN-MOPD’s improvement, while amplifying mathematics alone recovers less.Weights near DN-MOPD’s measured multipliers perform comparably at 9B and 4B; per-batch estimation outperforms first-batch weights by 1.10 points at 2B.Source: §3.2 · p. 7
  • Instruction-following log-ratios are more dispersed than the pooled signal, while mathematics log-ratios are less dispersed; DN-MOPD amplifies mathematics and downweights instruction following during training.After 80 updates, instruction following dominates the gradient under either weighting, while mathematics and code log-ratios are about three times less dispersed for DN-MOPD than Label.Source: §3.2 · pp. 7–8, Figure 4 · p. 8

3.3. Training budget and scope

  • Longer evaluation responses help Label and narrow the gap, but DN-MOPD remains ahead across sizes with positive paired intervals.The comparison varies evaluation response room and training duration.Source: §3.3 · p. 8
  • After longer training, DN-MOPD remains ahead at every size under both evaluation caps, although the eventual ordering at convergence remains unresolved.The longer runs narrow gaps at the two smaller sizes.Source: §3.3 · p. 9, Table 5 · p. 9

4. Related work

Related work situates DN-MOPD among on-policy distillation, multi-teacher integration, and methods for balancing learning signals. The paper distinguishes its approach by rescaling domain feedback according to measured spread while retaining the assignment and data mix.

On-Policy Distillation

Multi-Teacher On-Policy Distillation

  • Multi-teacher OPD integrates domain specialists, while related work studies token allocation, domain budgets, reward refresh, and specialized multi-teacher applications.DN-MOPD focuses on relative feedback strength under fixed assignment and prompt schedule.Source: §4 · p. 10

Balancing learning signals across objectives and domains

  • Prior balancing methods adjust multi-task losses, value targets, response or token normalization, task-distillation losses, or rewards by domain rarity and difficulty.Examples include gradient norms (Chen et al., 2018), learned uncertainty (Kendall et al., 2018), PopArt, DTO-KD, and DRPO (Dai et al., 2025).Source: §4 · p. 10
  • DN-MOPD rescales distillation advantages by measured teacher–student log-ratio spread, without learned weights or per-domain gradients, while retaining the loss, teacher assignment, and data mix.This targets imbalance across teachers rather than token share or mean reward magnitude.Source: §4 · p. 10

5. Conclusion and limitations

The conclusion reports that routing specialists alone did not outperform the strongest single-teacher student in the tested Qwen3.5 settings, while DN-MOPD improved the multi-teacher results. The authors qualify the scope by noting dependence on configuration and limits to the expert pools and scale estimates.

5. Conclusion and limitations

  • Across Qwen3.5-9B, 4B, and 2B and three student seeds, DN-MOPD improves the six-task average over MOPD at both evaluation caps and recovers most lost mathematics gain.Controls attribute most of the gain to limiting instruction-following feedback that otherwise dominates the initial student’s shared gradient.Source: §5 · p. 10
  • The evidence uses one expert pool per size in one model family, and three student seeds do not cover retraining experts.The authors also note configuration dependence, scale-estimate sensitivity to domain composition and response length, and that the IF multiplier usually sits at its lower bound.Source: §5 · p. 10

AI use statement

The authors report using LLM agents in experiments, writing, and checking, while retaining responsibility for research decisions and conclusions.

AI use statement

  • LLM agents drafted and revised text, formatted tables and figures, wrote experiment scripts, and launched and monitored runs.They also searched and verified references, re-derived reported numbers from stored results, and reviewed protocols and analysis code.Source: AI use statement · p. 11
  • The authors treated agent outputs as drafts, approved the final text, and state that they chose research questions, designs, decisions, and conclusions.They state responsibility for everything in the paper.Source: AI use statement · p. 11

Reproducibility statement

The paper documents its method, experiments, and evaluation materials and releases records and code to support score recomputation without rerunning models.

Reproducibility statement

  • Sections 2 and appendices A–D document the method, experimental setup, baselines, evaluation, results, controls, and feedback diagnostics.Section 2, Algorithm 1, and Appendix D.1 define DN-MOPD; Appendix A covers experimental details, B–C report results and controls, and D defines diagnostics.Source: Reproducibility statement · p. 11
  • Released records link model scores to checkpoints, configurations, question manifests, and grader revisions, enabling score recomputation without rerunning models.The release also includes a reference implementation, unit tests, training code, recipes, and multiplier traces; checkpoints are planned for later release.Source: Reproducibility statement · p. 11

Ethics statement

The work uses public open-weight models and data without human subjects or personal data; appendices provide supporting methodological and result details.

Ethics statement

  • The study distils public open-weight models using mathematics, code, and instruction-following data, with no human subjects or personal data.The code judge ran generated programs in a resource-limited sandbox on completions from the authors’ own rollouts.Source: Ethics statement · p. 11

Appendix overview

  • Appendices A–E cover recipes, baselines, evaluation, complete results, controls, diagnostics, mathematics transfer, and an earlier construction.Appendix E reports that the earlier Qwen3 construction yielded no clear improvement over Label.Source: Appendix overview · p. 15

A. Experimental setup and reproducibility

The experiments compare domain specialists and student methods under shared Qwen3.5 training setups, with specified evaluation and uncertainty procedures.

Models, data and training

  • Separate Qwen3.5 specialist pools at 9B, 4B, and 2B teach same-size students using a fixed mixture of 2,700 prompts, 900 per domain.Teacher data come from DeepMath-103K, Eurus, and generated instruction-following prompts.Source: A. Experimental setup and… · p. 16
  • Experts use GRPO and student comparisons share each size’s specialist roster; expert update counts vary with the pool’s wall-clock training budget.Expert training uses seed 42 and runs for up to 400 updates or until its fixed wall-clock budget ends; the last complete checkpoint is used.Source: A. Experimental setup and… · p. 16, Table A.1 · p. 16
  • Main OPD students train for 80 updates with seed 42, 512-response rollout batches, and a sampled-token reverse-KL advantage in the clipped policy-gradient surrogate.DN-MOPD changes that learning signal; its implementation is specified in Appendix D.Source: A. Experimental setup and… · p. 16

Baselines

  • Baselines include single-teacher, label-routed, uniform-pooling, and dynamic-teacher OPD; the strongest single-teacher reference is selected on the same public Total.SeqKD-SFT, parameter averaging, and task arithmetic use different token and optimization budgets, so their scores compare capability under the stated recipes.Source: A. Experimental setup and… · p. 16

Evaluation and uncertainty

  • Evaluation covers two mathematics, two code, and two instruction-following suites at response caps of 8,192 and 16,384 tokens.Scores average correctness within questions and then across questions; Total equally averages all six tasks, and displayed differences are percentage points.Source: A. Experimental setup and… · p. 17
  • Paired 95% intervals bootstrap questions within suites, but condition on one student seed and one expert pool per size.They do not quantify variation from retraining students or experts.Source: A. Experimental setup and… · p. 17

Method selection and reporting

  • Scaling and clipping bounds were selected on an earlier Qwen3 development instrument and transferred unchanged to all three Qwen3.5 sizes.The 16K main-text presentation followed inspection of both caps; the original public protocol was 8K, and complete 8K results remain in Appendix B.Source: A. Experimental setup and… · p. 17

Reproduction materials

  • Released reproduction materials associate scores with models, configurations, question manifests, and grader revisions, with machine-readable task scores and paired contrasts.The records derive scores, domain means, paired contrasts, and plotting data from stored model outputs.Source: A. Experimental setup and… · p. 17

B. Complete six-task results

Across the smaller Qwen3.5 sizes, DN-MOPD leads the multi-teacher OPD block at both evaluation caps, with mathematics gains over Label.

Complete six-task results

  • The expanded tables report the smaller sizes at 16K and all three sizes at the originally specified 8K cap, using the same checkpoint at both caps.All methods use 80 updates; Total equally averages the six tasks.Source: B. Complete six-task results · p. 18

B.1 Full six-task results for the smaller scales at 16K

  • At 4B and 2B, DN-MOPD has the highest multi-teacher OPD Total and best score on every task, tying Dynamic on LCB v6 at 4B.Relative to Label, AIME25 and AIME26 improve by 2.9 to 4.5 points, while both LCB versions improve by 1.7 to 3.1 points; SeqKD-SFT and task arithmetic retain higher Totals.Source: B. Complete six-task results · p. 18, Table B.1 · p. 18, Table B.2 · p. 19

B.2 Complete results at the declared 8K evaluation cap

  • At 8K, DN-MOPD has the highest multi-teacher OPD Total at every size and the best score on five or six tasks.Label leads IFEval by 0.06 points at 9B and 0.10 points at 2B; AIME gains over Label range from 3.6 to 9.0 points, while SeqKD-SFT and task arithmetic again have higher Totals.Source: B. Complete six-task results · p. 18, Table B.3 · p. 19, Table B.4 · p. 20

C. Additional controls and training budgets

Across training budgets, evaluation caps, and student seeds, DN-MOPD outperforms Label, while the controls test longer training, alternative supervision, fixed weights, and mathematics transfer. These comparisons show when its gains persist and which tested changes account for them.

Annealed injection

  • Annealed teacher-answer injection trails DN-MOPD at every size, with 16K Total changes versus Label of −0.24, +1.11, and +0.30 points at 9B, 4B, and 2B.The intervention replaces one of eight response slots for prompts with a usable verified teacher answer and stops at step 40; coverage varies across pools.Source: C. Additional controls and training… · p. 21, Table C.1 · p. 21

Longer training

  • At 8K, the 80-update DN-MOPD student leads Label after 160 updates by 1.93, 1.95, and 1.71 points at 9B, 4B, and 2B, respectively.At 16K, the corresponding differences are +0.18 to +0.94, with intervals including or touching zero; DN-MOPD remains highest within every continued-training block.Source: C. Additional controls and training… · pp. 21–22

Single-teacher comparison

Domain results

Generation length and truncation

  • Mathematics improvements accompany shorter answers at every size under both evaluation caps, weakening the explanation that gains simply reflect more output tokens.Average length alone does not measure inference latency or training cost, and the length comparison leaves open whether shorter answers cause or follow better solutions.Source: C. Additional controls and training… · p. 25, Table C.9 · p. 25

Additional student seeds

Fixed domain weights

  • Fixed controls that change both mathematics and instruction-following weights improve over Label; reducing instruction-following alone raises mathematics scores more than amplifying mathematics alone.Reducing instruction-following alone raises mathematics by 2.76 and 3.07 points at 16K for 4B and 2B; these one-seed controls do not exceed DN-MOPD.Source: C. Additional controls and training… · pp. 25–26, Table C.11 · p. 26

MATH-500

D. Implementation and diagnostic details

DN-MOPD changes the advantage computation while preserving domain-based teacher routing and the Label training setup. Diagnostics describe unequal feedback scales, their relationship to pooled variance and gradients, and limits on what those measurements establish.

Domain scaling and policy optimization

  • DN-MOPD changes only the advantage computation: each domain’s batch multiplier scales its advantages, while label routing, student-generated responses, masking, and response-level loss reduction remain as in Label.The factor is clipped to [0.25, 4], with a multiplier of one for degenerate estimates; the same factor applies to each sample’s advantages and is recomputed each batch.Source: D. Implementation and diagnostic details · p. 28
  • For an unclipped, nondegenerate domain, scaling gives the rollout signal the pooled standard deviation without subtracting domain means or guaranteeing equal actor-side gradients.The common target scale also depends on batch domain composition, not only each domain’s individual spread.Source: D. Implementation and diagnostic details · pp. 28–29

Feedback measurements

  • At the first update, mathematics spread ratios are 0.48–0.57, code ratios 1.13–1.23, and instruction-following ratios 2.32–4.41 across the three sizes.These measurements start from the same initial student as Label; instruction-following spread grows during training while mathematics remains below the pooled spread.Source: D. Implementation and diagnostic details · p. 29

Scale imbalance, clipping and token composition

  • Instruction-following’s large spread often drives its multiplier to the lower clipping bound, limiting its feedback without fully equalizing domain spreads.The diagnostic table reports raw and applied multipliers, clipping, and each domain’s share of valid response tokens.Source: D. Implementation and diagnostic details · p. 29, Table D.1 · p. 30

What the controls establish

  • The controls support feedback scaling as a useful change to the tested Label recipe, but do not separately vary clipping bounds, domain proportions, or the scale statistic.The implementation and configurations are released with the training code.Source: D. Implementation and diagnostic details · pp. 29–30

Sources of the pooled spread and of the update

  • Instruction-following contributes about 1% of valid response tokens but 51%, 21%, and 19% of pooled log-ratio variance at 9B, 4B, and 2B.In Label runs, recorded but unapplied multipliers remain near 1.5–1.9 for mathematics and 0.25–0.34 for instruction-following across seeds.Source: D. Implementation and diagnostic details · p. 30, Table D.2 · p. 31
  • At the initial 4B student, instruction-following supplies 94% of the combined gradient under equal weights, falling to 64% under DN-MOPD’s first-batch weights.Mathematics and code shares rise from 1% and 5% to 16% and 20%; these measurements use 32 prompts per domain and four responses each.Source: D. Implementation and diagnostic details · p. 30, Table D.3 · p. 31
  • Post-training probe log-ratio spreads for mathematics and code are lower under DN-MOPD than Label, consistent with greater teacher-feedback absorption but not establishing its cause.These descriptive measurements use one probe set and gradients at policy ratio one without clipping or optimizer state.Source: D. Implementation and diagnostic details · pp. 30–31

E. Capability transfer and a boundary case

The paper evaluates how well a shared student retains specialist mathematics capability and reports an earlier construction where the measured benefit differs. The comparisons indicate that transfer and scaling benefits depend on the tested construction.

Capability transfer in Qwen3.5

  • Every specialist improves its own domain, but Label retains little mathematics-specialist gain; mathematics-only OPD preserves more under the same update budget.This comparison indicates the student can learn from the mathematics teacher, while the loss of transfer appears when feedback is combined across domains.Source: E. Capability transfer and a boundary… · p. 32
  • At 8K, DN-MOPD retains 61–90% of the mathematics expert’s gain across sizes, versus 14–32% for Label.At 16K, Label shows no mathematics gain over initialization at any size and falls below it at 4B, whereas DN-MOPD improves mathematics at every size; retained fractions depend on the expert’s gain and are a within-construction diagnostic.Source: E. Capability transfer and a boundary… · p. 32, Table E.1 · p. 32

Earlier Qwen3-4B comparison

  • In the earlier Qwen3-4B construction, domain results show no clear mathematics or code improvement, and the small instruction-following increase is uncertain.Label already transfers substantial mathematics capability there, with a median mathematics multiplier of 1.00; differences in model and protocol prevent a controlled comparison with Qwen3.5.Source: E. Capability transfer and a boundary… · pp. 32–33, Table E.2 · p. 32

Related papers

Loading 2609.35347v1…