PaperlayerView exact revision on arXiv

Source-linked AI summary

MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

arXiv:2609.30837v1cs.LGcs.AI

TL;DR

Prompt-level domain-label routing in multi-teacher OPD limits the use of unlabeled mixtures and leaves other teachers’ token-level signals unused. MOPD-Router routes supervision across teachers at each token without domain labels or a separate routing model; ExpertAlign performs best overall across the evaluated settings.

  • Problem

    Existing MOPD assigns one domain-matched teacher to a prompt’s entire rollout, while mixed datasets may lack reliable labels and other teachers may offer useful token-level signals.

    Source: §1 · p. 2

  • Method

    MOPD-Router uses plug-in metrics to select and weight supervision from the full teacher pool at each token without domain labels or a separate routing model.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    ExpertAlign achieves the strongest overall performance across unlabeled and domain-labeled data in both strong-to-weak and same-size distillation scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings demonstrate the value of dynamically selecting and combining complementary teacher signals during multi-teacher distillation.

    Source: §5 · p. 10

    Generalization to larger teacher pools remains to be tested.

    Source: A Limitations and Open Directions · p. 13

Abstract

from arXiv · show

Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher-student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment. Code is available at: https://github.com/TURLEing/MOPD-Router.

1. Introduction

Existing MOPD assigns a domain-matched teacher to each prompt for the full rollout, relying on labels and leaving other teachers’ token-level contributions unused. MOPD-Router replaces that assignment with token-level routing, and ExpertAlign achieves the strongest overall performance across the evaluated settings.

1. Introduction

  • Domain-label hard routing assigns one domain-matched teacher to a prompt’s entire rollout, although mixed datasets may lack reliable labels and other teachers may help at individual tokens.The usefulness of cross-domain signals can vary across generation stages and token positions.Source: §1 · p. 2
  • ExpertAlign achieves the strongest overall performance across unlabeled and domain-labeled data in both strong-to-weak and same-size distillation scenarios.The framework also outperforms naive mean aggregation on unlabeled data and matches or exceeds domain-label hard routing on labeled data without using those labels.Source: §1 · p. 2
  • MOPD-Router selects and weights supervision from the full teacher pool at every token without domain labels or a separate routing model.Its metrics include teacher entropy, teacher–student distributional difference, and ExpertAlign’s alignment between acquired specialization and teaching direction.Source: §1 · p. 2
  • The paper’s analyses find that cross-domain teacher signals provide complementary supervision across capabilities.The authors open-source MOPD-Router to support further research on token-level teacher routing.Source: §1 · p. 2

2. Related Work

Related work covers adaptations to single-teacher on-policy distillation and approaches to integrating capabilities from multiple specialized teachers. Existing MOPD commonly uses prompt-level domain-label routing, while label-free routing metric design remains underexplored.

2.1 On-Policy Distillation

  • On-policy distillation research includes strong-to-weak, self-, and adaptive-distillation directions, which primarily modify supervision within single-teacher OPD rather than allocate it among specialized teachers.These directions address stronger-teacher transfer, different teacher and student contexts or feedback, and non-uniform supervision, respectively.Source: §2.1 · p. 3

2.2 Multi-Teacher Capability Integration

  • Prior capability-integration approaches include joint training, sequential training, and weight-space composition, while MOPD integrates domain teachers through policy-space supervision.Examples include Mix-RL’s pooled prompts and domain-specific rewards, Cascade RL’s sequential stages, and parameter merging of specialized checkpoints.Source: §2.2 · p. 3
  • Most existing MOPD implementations rely on prompt-level hard routing by domain label, leaving label-free routing metric design underexplored.Open-MOPD and D3-MOPD improve recovery of domain-teacher gains through optimization-budget balancing and adaptive domain scheduling; H-OPD uses confidence-based token-level arbitration for multimodal reasoning.Source: §2.2 · p. 3

3 MOPD-Router

MOPD-Router replaces fixed prompt-to-teacher assignment with token-level weighting over the full teacher pool. It formalizes sampled-token supervision and supplies a plug-in interface for routing metrics.

3 MOPD-Router

  • MOPD-Router replaces fixed prompt-to-teacher assignment with token-level weighting over the full teacher set, using metrics based on teacher distributions, teacher–student relations, or specialization.The specialization comparison is relative to the shared pre-RL base model.Source: §3 · p. 3

3.1 Multi-Teacher On-Policy Distillation

  • For each student-generated response, teachers evaluate visited contexts and provide sampled-token signals that are aggregated with non-negative weights into the advantage slot without a learned critic.The per-teacher signal is the log-probability difference between the teacher and rollout student for the sampled token.Source: §3.1 · p. 3

3.2 Token-Level Metric Routing

  • At every student-rollout token, a plug-in metric scores teachers to select and weight their contributions over the full pool, without domain labels or a separate routing model.The resulting selection and weighting are context-dependent.Source: §3.2 · p. 4

3.3 Routing Metrics

  • Entropy routes by negative predictive entropy, but cannot tell whether a low-entropy teacher usefully corrects the student and may systematically favor teachers with different entropy scales.Novelty scores teacher–student differences within shared high-probability candidates where both models assign sufficient probability mass; ExpertAlign assesses specialization in the teacher’s teaching signal.Source: §3 · p. 3, §3.3 · p. 4

4. Experiments

MOPD-Router evaluates token-level routing over the teacher pool across unlabeled and domain-labeled mixtures. ExpertAlign retains teachers whose teaching directions align with their acquired specialization, then weights their signals by alignment.

4.1 Experimental Setup

  • Evaluation spans a 60K unlabeled chat-and-math mixture and a 66K labeled math, code, and instruction-following mixture, with 1.7B and 4B students.The labeled-data routing methods ignore the available domain labels.Source: §4.1 · pp. 5–6

4.2 Main Results

  • ExpertAlign leads overall in all four settings: 38.58 for Qwen3-1.7B and 53.76 for Qwen3-4B on unlabeled data, and 40.19 and 54.58 on labeled data.It significantly outperforms Mean Aggregation and Standard MOPD, and outperforms or matches Open-MOPD.Source: Table 1 · p. 6, Table 2 · p. 7, §4.2 · p. 7

4.3.1 Teacher confidence is not a reliable routing signal.

  • Low entropy does not reliably identify useful supervision: high-entropy positions can still be decision points with multiple plausible continuations and important learning signals.Across both distillation settings, the math teacher also has substantially lower entropy than the other teachers, favoring that teacher rather than adapting to student context.Source: §4.3.1 · p. 7, §4.3 · p. 8

4.3.3 Separating teacher selection from teacher weighting.

  • Uniform selection improves Overall over Mean by 1.22–4.75 points, and Cosine weighting adds a further 1.62–3.34 points across all four settings.The reported gains concentrate in math and code for Uniform, while instruction-following is slightly lower on unlabeled data.Source: Table 3 · p. 8, §4.3.3 · p. 8

4.3.4 Cross-domain teachers provide complementary supervision.

  • Using the full teacher pool improves Overall by 2.40 and 2.75 points over domain restriction, while token-level routing adds 3.62 and 1.97 points over response-level weighting.Figure 2 shows off-domain teachers receive more than 50% of conditional routing mass; in the example, teacher weights track generated content stages.Source: Table 4 · p. 8, §4.3.4 · p. 9, Figure 2 · p. 9

4.3.2 Accessibility matters when student–teacher support mismatch.

  • Accessibility raises Novelty Overall from 34.12 to 36.70 (+7.5%) in strong-to-weak distillation, but NoveltyOnly exceeds full Novelty by 0.55 points in same-size distillation.The reported pattern suggests accessibility helps more when student and teacher have mismatched local token support.Source: §4.3.2 · p. 8

4.4 Training Cost Analysis

  • ExpertAlign costs 716.9 GPU-hours under the common step budget, a 3.5% increase over Standard MOPD, and 1.18× token-normalized cost.The authors attribute the modest end-to-end overhead to the shared-base forward pass and top-k routing computation.Source: Table 5 · p. 9, §4.4 · p. 9

5. Conclusion

The conclusion presents MOPD-Router as token-level routing over the full teacher pool, replacing fixed prompt-to-teacher assignment without domain labels or a separate routing model.

5. Conclusion

  • Across unlabeled and domain-labeled mixtures in strong-to-weak and same-size settings, ExpertAlign performs best overall, supporting dynamic selection and combination of complementary teacher signals.The framework is presented as enabling multi-teacher distillation on unlabeled mixtures without a separate teacher-assignment stage.Source: §5 · p. 10

A Limitations and Open Directions

The evaluation uses three same-family Qwen3 teachers and a shared pre-RL base, so generalization to larger teacher pools remains untested. The authors identify systematic metric design and different-base teachers as open directions.

A Limitations and Open Directions

  • Experiments use three same-family Qwen3 teachers, leaving generalization to larger teacher pools untested.The controlled setup isolates routing from architecture and pretraining differences.Source: A Limitations and Open Directions · p. 13
  • Routing metrics are guided by empirical intuition rather than a unified design framework.Source: A Limitations and Open Directions · p. 13
  • Future work could test more diverse pools and scenarios, develop systematic metric-design principles, and extend ExpertAlign to teachers with different bases.The proposed extension uses each teacher’s own pre-RL checkpoint.Source: A Limitations and Open Directions · p. 13

B Training and Evaluation Hyperparameters

MOPD methods share the experiment settings and differ only in teacher-signal routing; evaluation uses specified sampling settings. The appendix also describes single-label annotation of the unlabeled mixture and highlights ambiguity in assigning one domain to mixed-skill prompts.

B Training and Evaluation Hyperparameters

  • MOPD routing methods share the experiment settings and differ only in how teacher signals are routed.Table 6 separates MOPD experiment hyperparameters from training settings for the instruction-following teacher.Source: B Training and Evaluation… · p. 13, Table 6 · p. 13
  • Evaluation uses temperature 0.7, top-p 0.8, and top-k 20, with maximum generation lengths of 16,384 tokens for math and code and 4,096 for instruction-following benchmarks.Source: B Training and Evaluation… · p. 13

C.1 Annotation Procedure and Label Distribution

  • The AI annotation assigns one teacher domain per example using both the conversation and reference answer, with math, code, and instruction-following categories.The rubric treats code-assisted mathematical solving as math when mathematics is the objective, but assigns code when programming is central.Source: §C.1 · p. 14
  • The code label accounts for 3.30% of the mixture, illustrating how a single domain label compresses intertwined mathematics and code use into one rollout-wide teacher choice.NuminaMath-TIR problems often use executable code as a tool for mathematical problem solving.Source: §C.1 · p. 14

C.2 Human Validation

Human validation found 87.33% macro agreement with AI-generated labels across a category-balanced sample, with most disagreement at the boundary between code-assisted math and programming-centered tasks.

C.2 Human Validation

  • Five annotators labeled 100 examples from each AI-generated category, and majority vote supplied the human labels.Inter-annotator agreement was Fleiss’ κ = 0.84.Source: §C.2 · pp. 14–15
  • 87.33% macro agreement: AI-generated labels matched human-majority labels on 262 of 300 sampled examples.Agreement was 99% for math, 76% for code, and 87% for instruction following; most disagreements involved AI-labeled code judged as mathematics.Source: §C.2 · p. 15, Table 8 · p. 15

D.1 Evaluation-Set Uncertainty

Paired bootstrap analysis finds ExpertAlign consistently outperforms Mean Aggregation and Standard MOPD across four settings, and improves over or matches the label-informed Open-MOPD baseline. The analysis reports 95% confidence intervals and Holm-adjusted tests for Overall comparisons.

D.1 Evaluation-Set Uncertainty

  • The analysis uses 10,000 paired bootstrap samples per benchmark and percentile 95% confidence intervals for the unweighted macro-average of nine benchmark scores.Holm’s method adjusts two-sided p-values jointly over twelve Overall comparisons.Source: §D.1 · p. 15
  • 4.09–6.42 points: ExpertAlign exceeds Mean Aggregation across four settings, with every comparison significant after Holm correction; gains over Standard MOPD are 2.25–5.00 points and significant in every setting.Against Open-MOPD, ExpertAlign has significant gains in three settings and statistically matches it in the labeled 1.7B setting.Source: Table 9 · p. 15, §D.1 · p. 15

D.2 Training-Seed Robustness

Across the additional analyses, ExpertAlign remains robust across training seeds, while routing behavior varies with teacher confidence, support accessibility, and response content.

D.2 Training-Seed Robustness

  • ExpertAlign has the highest mean in each domain and exceeds both baselines under every matched seed, averaging +3.76 ± 0.32 Overall over Standard MOPD and +1.88 ± 0.31 over Open-MOPD.The comparison uses the domain-labeled setting with a Qwen3-4B student and three training seeds.Source: Table 10 · p. 16, §D.2 · p. 16

E.1 Why Teacher Confidence Fails as a Routing Signal

  • After entropy calibration, Overall rises only from 34.65 to 34.83, while confidence still favors mathematics at about 70% top-1 positions versus code’s 9%.Calibration makes average routing weights nearly uniform, but the small Overall change indicates confidence alone is insufficient to measure supervision value.Source: §E.1 · p. 16, Table 12 · p. 16, Table 11 · p. 16

E.2 Accessibility and Shared-Support Statistics

  • Accessibility raises unlabeled strong-to-weak Overall from 34.12 to 36.70, while same-size Overall is 50.30 with accessibility and 50.85 without.On shared top-k support, the 1.7B student has lower teacher overlap than the 4B student for every teacher; instruction following has the lowest overlap in both settings.Source: §E.2 · pp. 16–17, Table 13 · p. 17, Table 14 · p. 17

E.3 Cross-Domain Routing Example

  • Routing centers align with semantic roles: mathematics around derivation, code around Python calculation, and instruction following around structural markers and JSON output.The word-span view normalizes color intensity independently for each teacher, showing each teacher’s relative concentration across response words.Source: §E.3 · p. 17, Figure 3 · p. 18

E.4 Sensitivity to the Student Top-k Support Size

ExpertAlign’s performance is relatively stable across support sizes from k = 1 to k = 16, but declines at k = 32; the best support size varies by domain.

E.4 Sensitivity to the Student Top-k Support Size

  • k = 1 to k = 16 gives relatively stable Overall performance, followed by a clear drop at k = 32.The support-size comparison uses the domain-labeled set with the Qwen3-4B student; Overall for k = 1 to k = 16 lies within the error bars.Source: §E.4 · p. 18
  • Mathematics is the only domain that peaks at k = 1, where single-token alignment fits its sharpened distribution but misses competing continuations common in code and instruction following.In this case, alignment uses the product of scalar specialization and teaching quantities; the mathematics teacher has the lowest predictive entropy.Source: §E.4 · p. 18
  • At k = 32, all three domains degrade as low-probability tail candidates increasingly dominate alignment and make teacher selection and weighting noisy.The shared top-16 candidates already cover about 99% of student probability mass, while log-ratios can amplify differences between small probabilities.Source: §E.4 · p. 18

E.5 ExpertAlign Selection Coverage and Training Dynamics

ExpertAlign selectively changes teacher composition on many tokens while preserving broad supervision coverage; fully filtered tokens remain rare but increase mildly during training.

E.5 ExpertAlign Selection Coverage and Training Dynamics

  • Across four settings, 37.02–51.24% of tokens retain all three teachers, while 97.52–99.48% receive a signal from at least one teacher.ExpertAlign selects a strict subset on 48.76–62.98% of tokens; only 0.52–2.48% have no retained teacher, so the gate mostly changes teacher composition rather than removing tokens from OPD.Source: §E.5 · p. 18, Figure 5 · p. 19
  • The K_t = 0 fraction has a positive fitted slope in every setting, increasing by 0.071–0.161 percentage points per 100 steps.The authors describe this as a mild increasing tendency, with the slopes measured over the displayed training range.Source: §E.5 · p. 19, Figure 5 · p. 19
  • As the student approaches a teacher along its specialization direction, the teaching vector contracts and its alignment can fall below the margin, yielding little or no positively aligned correction.The gate can also become non-positive when the student’s projection reaches or exceeds the teacher’s base-relative displacement along that direction.Source: §E.5 · p. 19

Related papers

Loading 2609.30837v1…