Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

• The controlled testbed starts from SmolLM3-3B-Base, uses three-domain mixed-SFT initialization and mathematics, coding, and instruction-following RL teachers, and applies ground-truth oracle routing.The pipeline and ablation ladder are reproducible on an 8×A100-80GB academic setup.Source: §1 · p. 2
• Naive M-OPD averaged 28.05 versus 31.55 for domain-specific distillation; instruction following fell 6.16 points below RouteOPD and plateaued earliest.The IF gap was 3.3× the degradation observed in mathematics, and Figure 1 shows IF had the lowest recovered headroom.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The gap stems from token-budget distortion, not dominant gradient conflict: IF received 20.3% of prompts but only 0.99% of gradient tokens, with further drift from uneven convergence and stale rewards.Equalizing raw token budgets initially does not prevent drift within tens of steps; multi-step rollout reuse also leaves student-dependent rewards stale.Source: §1 · pp. 2–3
• Open-MOPD balances token shares and allocates budget by student–teacher gap; reward refresh recomputes student log-probabilities before each gradient step while caching teacher states.Gap-aware allocation avoids the collapse seen with naive reward normalization at step 74, while reward refresh eliminates sample staleness with negligible overhead.Source: §1 · pp. 2–3
• 83.4% headroom recovery, up from 35.6%, was achieved by Open-MOPD in a single student model.The framework combines token-share balancing, gap-aware dynamic budget allocation, and student reward refresh.Source: §1 · p. 3
• The authors open-source the end-to-end post-training pipeline, model checkpoints, and evaluation suites on a reproducible academic compute budget.Source: §1 · p. 3

2 Open-MOPD: An Open Recipe for Multi-Teacher Capability Integration

• The benchmark targets transferring math, code, and instruction-following expertise from domain-specific RL teachers into one unified student.Training samples carry domain labels that select the corresponding expert supervision.Source: §2 · p. 3

2.1 The Multi-Teacher On-Policy Distillation Objective

• The student generates each response from its current policy; a routed teacher supplies top-k token-level rewards that are used directly as PPO advantages.The default top-k is 16, and on-policy rollouts keep the supervision distribution aligned with the student rather than fixed offline teacher trajectories.Source: §2.1 · pp. 3–4

2.2 Base Model Selection

• SmolLM3-3B-Base is selected to balance repeatable experiments on one 8×A100-80GB node with capacity to learn from long math and code responses.The recipe uses a 32,768-token SFT limit; the authors report that smaller Qwen3-1.7B-Base often truncated responses even with a 31K generation budget.Source: §2.2 · p. 4

2.3 The End-to-End Training Recipe

• The three-stage pipeline uses mixed-domain SFT, separate domain-specific RL teachers, then multi-teacher OPD from the shared SFT checkpoint.Stage I uses math, code, and instruction-following data; Stage II trains one teacher per domain; Stage III routes each sampled prompt to its domain teacher. Student response limits are 16K for math and code and 2K for instruction following.Source: §2.3 · pp. 4–5, Table 1 · p. 5

2.4 Evaluation Setup

• Integration gap is measured against RouteOPD, the combination of best single-teacher students selected by domain label at evaluation time.The protocol averages dataset scores within each domain and then equally averages the three domain scores, using six benchmarks across math, code, and instruction following.Source: §2.4 · p. 5, Table 2 · p. 6

3.1 A baseline reveals the integration gap

• 28.05 is Naive M-OPD’s score, up from πmixsft’s 25.67 but 0.91 below πmixrl and 3.50 below RouteOPD (31.55).The results show that multi-teacher supervision helps, but combining teacher signals in one student still leaves a gap to the domain-routed reference.Source: §3.1 · p. 6
• IF falls 6.16 points below its RouteOPD reference, a gap 3.3 times math’s 1.89-point gap, and is the main source of the total integration gap.None of the three domains reaches its RouteOPD reference within 300 steps.Source: §3.1 · p. 6
• Over steps 200–300, math closes 64% of its gap, code 40%, and IF 26%; IF alone declines mid-training, falling 11% over steps 100–200.Source: §3.1 · p. 6

3.2 Testing teacher conflict

• Teacher disagreement averages 0.126 nat, stays below 0.27 nat, and exceeds 1 nat for only 0.62% of tokens overall (3.9% on IF; 0.31% on math).Although the maximum reaches 30.58 nat, the findings do not support widespread teacher conflict as the dominant bottleneck.Source: §3.2 · p. 7
• −0.52, −0.73, and −0.75 points were the baseline-relative drops from masking the top 1%, 5%, and 20% of tokens by disagreement; consensus targeting dropped 0.83 points.The interventions suggest token-level disagreement alone cannot distinguish irrelevant noise from domain-specific information.Source: §3.2 · p. 7

3.3 Measuring training imbalance across domains

• With token shares fixed at one-third, IF’s budget share falls from 48.7% to 11.4%, while code’s rises from 39.6% to 63.8%.Early per-token reward magnitudes are 0.019 for math, 0.063 for code, and 0.091 for IF; their decline also differs across domains.Source: §3.3 · p. 8
• 49.7%/49.3%/0.99% of gradient tokens go to math/code/IF despite prompt shares of 39.8%/39.8%/20.3%, because math and code responses average about 10,500 tokens versus IF’s 409.Balancing IF’s token share through prompt oversampling would require about 33.6 times as many prompts, squeezing math and code long-chain supervision.Source: §3.3 · p. 8, Figure 3 · p. 9
• At K=32, rollout-to-current-policy KL reaches 0.216 and PPO clipping reaches 0.86, while dense rewards still use rollout-time probabilities.The policy shift rises monotonically with inner updates, making later updates increasingly stale.Source: §3.3 · p. 9

4 From Diagnosis to Method: Open-MOPD

• Open-MOPD keeps the existing routing and distillation objective, with three mechanisms acting on different timescales and validated independently or combined.Figure 6 shows the data flow of one training iteration and where each mechanism applies.Source: §4 · p. 9

4.1 Token-Share Balancing

• 33.33% is the weighted token share guaranteed for each domain by the equal-share target, directly controlling loss token ratios without relying on response lengths or oversampling.The recipe amplifies each IF token about 48 times to offset its 25-fold length disadvantage, while preserving math and code prompt diversity.Source: §4.1 · p. 10

4.2 Gap-Following Allocation

• Gap-following allocation preserves the direction of the remaining teacher–student gap, giving more updates to domains with larger gaps; as a domain approaches its teacher, its md and budget fall together.Source: §4.2 · p. 11

4.3 Reward Refresh

• Reward refresh recomputes the student-dependent reward before each inner update, correcting mismatch between the rollout policy and the changing student without another teacher forward pass.At K = 1 the objective reduces to the original one; at K > 1, sampled student states still come from rollout time, so refresh removes only the reward staleness that can be corrected without another prefill.Source: §4.3 · pp. 11–12

4.4 The Complete Algorithm

• The complete algorithm combines token-share balancing, gap-following allocation, and reward refresh to adjust token weights, domain update shares, and reward feedback.Teacher log-probabilities are computed once per rollout, while the student-dependent reward term is recomputed before each inner update.Source: §4.4 · p. 12

5.1 Token-Share Balancing

• Token-share balancing yields a +1.17-point gain, almost entirely from IF, while math and code barely move.The weighted token share stays at 33.33% across 300 steps; without balancing, 99% of gradient tokens go to math and code and about 1% to IF.Source: §5.1 · p. 13, Table 4 · p. 14

5.3 Reward Refresh

• At K = 4, reward refresh adds +0.81 points atop token-share balancing and gap-following allocation, with no measurable extra runtime cost.The dense-reward computation takes 27.3 s with refresh and 27.8 s without; the full step takes 1298 s and 1313 s, respectively, with the difference within step-to-step variation.Source: Table 4 · p. 14, §5.3 · p. 15, Figure 8 · p. 15

5.2 Gap-Following Allocation

• Gap-following allocation improves the score by +0.72 points at K = 1, directing more budget to domains with larger teacher–student gaps.Across steps 200–300, math share grows from 15.1% to 32.6%, IF share drops from 34.5% to 17.1%, and code averages 55.4% of the budget, peaking at 87.6%.Source: §5.2 · pp. 13–14, Figure 7 · p. 14, Table 4 · p. 14
• Reversing the gap factor creates positive feedback: the IF gap shrinks by 32.5×, its weight rises from 24.4 to 80.9, and training stops at step 74.The reversed formulation allocates more budget to domains with smaller gaps, favoring domains the student already handles well.Source: §5.2 · p. 14, Figure 7 · p. 14

6.1 On-Policy Distillation and Multi-Teacher Extensions

• Fixed-corpus distillation can create exposure bias because students train on teacher-generated prefixes, whereas on-policy distillation uses student rollouts and teacher feedback on visited prefixes.Prior work also identifies large teacher–student capacity differences as a challenge.Source: §6.1 · p. 15
• OPD research varies divergences, sampling, and token-level signals, while prior work examines stability and staleness; Open-MOPD refreshes student-dependent rewards before each inner update using PPO’s actor forward.In its synchronous pipeline, each rollout batch is partitioned into K minibatches for sequential student updates, which can make rewards stale as the student changes; refresh adds no extra student forward.Source: §6.1 · p. 15
• Multi-teacher OPD routes each student rollout to a domain specialist and trains from token-level feedback; public systems apply this recipe at different model scales.Open-MOPD instead studies how to balance the training received by domains during integration.Source: §6.1 · pp. 15–16

6.2 Integrating Domain Experts

• Domain-specific RL produces specialists in areas including mathematics, software engineering, search, and instruction following, while deployment often calls for one model across domains.Open-MOPD studies integrating these specialists into one student through multi-teacher OPD.Source: §6.2 · p. 16
• Data-space integration mixes domains jointly or trains them sequentially, but mixture choices and unequal gradient magnitudes can affect balance; Open-MOPD targets token counts, teacher–student gaps, and reward staleness.Source: §6.2 · p. 16
• Weight-space methods merge task-specific parameter changes, while module-space methods average, route, or connect experts; unlike these post-training approaches, multi-teacher OPD trains one shared student from expert outputs.Open-MOPD studies balancing domain updates during this shared training.Source: §6.2 · p. 16

7 Conclusion

• Unequal token counts, unequal reward magnitudes, and outdated rewards during repeated inner updates create three separable sources of optimization-budget imbalance.Token-share balancing, gap-following allocation, and reward refresh correspond to these sources and are validated separately in ablation experiments.Source: §7 · p. 16
• Open-MOPD reduces the integration gap from 3.50 points to 0.31 points and raises recovery relative to RouteRL from 35.6% to 83.4%.The authors release the complete recipe and mechanism implementations to support reproducible follow-up work.Source: §7 · p. 16

A Constructing the Open-MOPD Recipe

• The team screened base models on math SFT for stable reasoning trajectories before selecting one for recipe development.This screening was the first step in constructing the final Open-MOPD recipe.Source: A Constructing the Open-MOPD Recipe · p. 20
• The selected base model was then extended with mixed-domain SFT on math, code, and instruction following.Source: A Constructing the Open-MOPD Recipe · p. 20

A.1 Base Model Choice

• On Qwen3-1.7B-Base, 69.17–80.42% of samples exhausted the 31K budget, only 20.00–31.25% closed </think>, and AIME24 stayed below 7.1%.Adding epochs did not produce a monotone improvement; many subsequent RL or OPD trajectories would therefore contain reasoning prefixes without final answers.Source: §A.1 · p. 20
• On Qwen2.5-7B-Base, AIME24 rose from 16.25% to 31.25% while the truncation rate fell from 42.92% to 12.92%.This shows that OpenR1 trajectories do not inherently force truncation across base models, but the 7B scale was too costly for the repeated full-recipe experiments.Source: §A.1 · p. 20
• Four mixed-domain SFT epochs gave OpenR1 Math, OCR-50k, and Instruction-Nemotron roughly 37.3%, 28.1%, and 34.6% of response tokens, respectively.Balancing samples by response-token count prevents the 820K short instruction-following answers from drowning out the fewer, longer math and code trajectories.Source: §A.1 · p. 20
• SmolLM3-3B-Base was selected as the operating point between 1.7B trajectory failures and the resource cost of running the full recipe at 7B scale.The final Open-MOPD training fits on a single 8×A100-80GB node.Source: §A.1 · p. 20

Stage hyperparameters

• Tables 7 and 8 report the hyperparameters for the recipe’s SFT, domain RL teachers, single-domain OPD, and multi-teacher OPD runs.The values for token-share balancing, gap-following allocation, and reward refresh in Table 8 are those used in the final recipe.Source: §A.2 · p. 20, Table 7 · p. 22, Table 8 · p. 22

Mixed-domain SFT and domain RL teachers

• The three domain RL teachers share a GRPO configuration but differ in data, response length, rollout group size, and training steps; all disable KL and filter non-discriminative groups.IF uses shorter responses, smaller groups, and more steps; math and code allow 30,000-token responses, and groups may be resampled up to 8 times.Source: §A.2 · pp. 20–21, Table 7 · p. 22

Single-domain OPD and multi-teacher OPD

• The RouteOPD oracle and multi-teacher OPD share optimiser and distillation settings, so their training-configuration difference is the number of teachers and domain mixture.Both use student top-k (k=16) dense rewards and off-policy depth K=4; multi-teacher OPD additionally enables token-share balancing, gap-following allocation, and reward refresh.Source: §A.2 · p. 21, Table 8 · p. 22