A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

1 Introduction

• RLVR provides outcome-level rewards but sparse intermediate supervision, while OPD offers dense token-level feedback but requires a strong compatible teacher; OPSD instead depends on privileged information.Such signals may require unavailable ground-truth solutions and can encourage shortcut memorization that hurts generalization.Source: §1 · p. 1
• Increasing activation from k to k′ exposes additional expert capacity that could form a compatible internal teacher, but naive expansion can cause mode collapse because co-activated experts were not trained to cooperate.The original low-k model's dense token-level supervision can repair the expanded model's behavior.Source: §1 · pp. 1–2
• MoEvolve first restores and strengthens the activation-expanded model, then reverses distillation to transfer its capabilities to the original low-k model without increasing inference-time computation.The framework avoids dependence on a stronger external teacher or hand-designed privileged information.Source: §1 · p. 2
• Across four MoE LLMs and twelve benchmarks, MoEvolve improves initial models by up to 22.2 domain-average points and same-budget RLVR baselines by up to 6.5 points.The evaluations span mathematics, code, and scientific knowledge, with gains at unchanged inference cost.Source: §1 · p. 2

2 Related Work

• Prior MoE work studies expert specialization and load balancing, while simply raising activation can hurt performance because newly co-activated experts were not optimized to cooperate.Distillation has also been used to compress MoE teachers and reduce expert activation through self-distillation.Source: §2 · p. 2
• OPD supplies dense token-level supervision on student-sampled trajectories but relies on a strong teacher with a compatible tokenizer and output space.Such a teacher may not be available or affordable in practice.Source: §2 · p. 2
• OPSD avoids an external teacher by using privileged signals such as hints or reference answers, but designing them can be task-specific and may hurt generalization.The signals may require unavailable ground-truth solutions or encourage imitation of reference trajectories.Source: §2 · p. 2

Mixture-of-Experts

• An MoE router selects top-k experts per token; the paper denotes the original model Mk and its activation-expanded copy Mk′, which differ only in activation budget.Because only k experts are active per token, computation depends on k rather than the total expert count N.Source: §3 · p. 3

On-Policy Distillation

• OPD trains on responses sampled from the student and applies token-level teacher supervision; G-OPD adds a reference policy and reward scaling, with ExOPD using λ > 1.The paper uses ExOPD for reverse distillation between paired MoE models with different activation budgets.Source: §3 · p. 3

Soft Adaptive Policy Optimization (SAPO)

• SAPO replaces GRPO's hard clipping with a smooth temperature-controlled gate, and the paper uses it to mitigate instability from training-rollout mismatch.Its asymmetric temperatures downweight high-variance negative updates.Source: §3 · pp. 2–4

4 MoEvolve: Bidirectional On-Policy Optimization for MoE Self-Distillation

• MoEvolve has two phases: forward distillation applies ECRD to the expanded model, then backward distillation uses RL and reverse distillation to return capability to the original budget.The final model retains the original k-expert inference cost.Source: §4 · p. 4

4.1 Forward Distillation

• Raising activation from k to k′ adds representational capacity but can cause mode collapse because the newly co-activated experts were not trained to work together.ECRD uses on-policy OPD, with Mk as teacher and Mk′ as student, to restore cooperation and recover performance to at least match Mk.Source: §4.1 · p. 4

4.2.1 Capability Unlocking via Reinforcement Learning (CU)

• After ECRD restores cooperation, RL with SAPO exploits the expanded capacity and can strengthen Mk′ beyond Mk, temporarily activating k′ experts at inference.The recovered coordination makes RL optimization stable and effective.Source: §4.2.1 · p. 4

4.2.2 Reverse Distillation for Capability Transfer (RD)

• Reverse distillation uses ExOPD to transfer the strengthened teacher's gains into Mk while keeping the original k-expert activation budget.The paired models share tokenizer, vocabulary, and architecture; the teacher-reference log-ratio supplies a dense signal while anchoring the student to Mk.Source: §4.2.2 · p. 5

5.1 Experimental Setup

• Experiments evaluate four MoE backbones across mathematics, code generation, and ScienceQA, with training data matched to those domains.The baselines include SFT, RFT, RL, and OPSD, all starting from the same checkpoint; SFT is limited to mathematics because only that domain has shipped reference labels.Source: §5.1 · p. 5

5.2 Main Results

• MoEvolve achieves the best domain average on every backbone and domain, gaining up to +22.2 on mathematics over the original model and up to +6.5 over RL on mathematics.It is significantly better than RL on every backbone and domain (p < 0.05); the main results cover OLMoE-1B-7B and DeepSeek-V2-Lite in Table 1 and GPT-OSS-20B and Qwen3-30B-A3B in Table 2.Source: Table 1 · p. 6, Table 2 · p. 6, §5.2 · p. 6

5.3 Expert Cooperation Restoration Methods for the Activation-Expanded Model

• ECRD restores the activation-expanded model more faithfully than off-policy SFT and RFT by supervising trajectories sampled from the expanded model itself.SFT and RFT train on fixed target trajectories, whereas ECRD uses the original model as teacher and token-level logits on the student’s own trajectories.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Capability Unlocked by the CU Stage

• CU improves the ECRD-recovered model on all four backbones, with gains of +3.2 to +22.1 points on mathematics and +2.2 to +7.4 on code.The effect is especially large on Qwen3-30B-A3B, where the recovered model leaves the most headroom after expansion.Source: §5.4 · p. 7, Figure 3 · p. 7

5.5 How Best to Transfer Capability in RD

• ExOPD ranks above OPD, which ranks above RFT, for transferring capability from the expanded model back to the original model across backbones.Reward extrapolation contributes the largest gain among the compared transfer objectives.Source: §5.5 · p. 7

5.6 Effect of the Intermediate Activation Budget

• Using the largest feasible intermediate activation budget generally gives the best final result, so MoEvolve does not require careful tuning of k′.Source: §5.6 · p. 7

6.1 Analysis of Expert Cooperation after Activation Expansion

• During expansion, Qwen3-30B-A3B’s default k experts retain 26.2% of gate mass, about 4.2× the uniform reference, while ECRD shifts weight toward lower-ranked experts.Updating experts alone recovers the model; updating both experts and router recovers 90.4% of full-parameter ECRD, while router-only training fails.Source: §6.1 · pp. 7–8, Figure 4 · p. 8, Table 3 · p. 8

6.2 Why the Teacher Must Be Activation-Expanded

• Under the same training budget, CU on recovered activation-expanded models reaches higher reward and entropy and outperforms RL on the original model across domains and backbones.The activation-expanded model also consistently serves as the better reverse-distillation teacher, transferring more of its unlocked capability to the original activation budget.Source: §6.2 · p. 8

6.3 How Much Capability Survives the Transfer

• The final k-activation student can surpass its activation-expanded teacher while activating only k experts.Source: §6.3 · p. 8

• MoEvolve builds a stronger teacher from the MoE model itself, transfers its gains back to the original activation budget, and improves over initial models and strong RL baselines without additional inference cost.The reverse-distilled model can surpass its activation-expanded teacher.Source: §7 · p. 8

• MoEvolve has not been validated on substantially larger MoE models or broader tasks such as open-ended generation, agentic tasks, and long-context settings.The approach also requires training procedures and has no training-free way to unlock latent expert capacity without additional supervision or optimization.Source: Limitations · p. 9

Ethical Considerations

• The work uses public datasets and open-weight models, involves no human subjects or personal data, and notes that its framework may inherit backbone risks such as misuse, bias, or hallucination.Source: Ethical Considerations · p. 9

A Model Details; C.2 Training Configuration; Evaluation

• The experiments use four open-weight MoE backbones and domain-matched training and evaluation across mathematics, code, and scientific knowledge.Methods share the same backbone, tokenizer, and vocabulary; each domain uses a separate checkpoint and equal wall-clock training time per backbone.Source: A Model Details · p. 12, §C.2 · p. 13

B.2 Decoding and Scoring

• Benchmark evaluation is zero-shot at temperature 1.0 and top-p 1.0, with correctness determined by exact-answer verification, matching multiple-choice options, or passing all code tests.Ten generations are sampled per problem, and average pass@1 is reported for each benchmark.Source: §B.2 · p. 13

C.1 Training Datasets

• Training data cover the three evaluation domains, and the authors report no near-duplicates in checked held-out benchmarks under a 5-gram Jaccard threshold of 0.8.The same training corpus is used for all methods, and the MMLU auxiliary training set is disjoint from the MMLU-STEM test split.Source: §C.1 · p. 13

C.3 EMA-Based Stopping Criterion for ECRD

• ECRD stops when the relative drop in its EMA-smoothed distillation objective stays below a threshold for P consecutive windows.The procedure checks every W steps and rolls the window baseline forward after each check.Source: §C.3 · p. 14

D Intermediate Activation Width under Larger Activation Budgets

• Increasing the activation budget widens each layer’s active intermediate representation and exposes pretrained capacity without adding parameters.The added active features per token number (k′ − k)de; ECRD and CU are designed to exploit this already encoded capacity.Source: D Intermediate Activation Width under… · p. 15

E Training Instability without Expert Cooperation Restoration

• Direct RL on activation-expanded DeepSeek-V2-Lite without ECRD quickly destabilizes: policy entropy collapses early and training reward fails to improve consistently.This supports restoring expert cooperation before applying RL in the CU stage.Source: E Training Instability without Expert… · p. 15, Figure 5 · p. 15

F Computational Cost

• MoEvolve reaches a higher final mathematics score than same-time RL on Qwen3-30B-A3B, despite spending its first 12 hours building an internal teacher.Figure 6 separates Forward Distillation, when Mk is unchanged, from Backward Distillation, when capability is transferred back to Mk.Source: F Computational Cost · p. 16, Figure 6 · p. 16

G Analysis of Training Dynamics and Expert Load

• MoEvolve improves for longer, finishes with the highest reward, and retains the highest entropy; RL raises reward fastest but saturates early, while RFT and OPSD converge to lower reward.The comparison tracks mean reward, policy entropy, and response length during training.Source: G Analysis of Training Dynamics and… · p. 16, Figure 8 · p. 17
• MoEvolve remains closest to base routing, with H/log N of 0.837 vs. 0.848 and 0.72% vs. 0.44% of 6144 expert slots never selected, while fixed-budget baselines skew more strongly.Figure 8 reports utilization, load balance, and never-selected slots for the final k-activation models.Source: G Analysis of Training Dynamics and… · p. 16, Figure 8 · p. 17
• The base and final MoEvolve per-expert load maps are almost indistinguishable; training adds 17 dead slots out of 6144 and lowers the hottest slot from 11.8× to 10.7× the uniform-load reference.Figure 7 compares per-layer, per-expert load before and after training on the same mathematics benchmark traces.Source: G Analysis of Training Dynamics and… · p. 16, Figure 7 · p. 16
• The reported gains are obtained without reorganizing which experts carry the traffic.Source: G Analysis of Training Dynamics and… · p. 17

H Ablation on the Reverse-Distillation Objective

• RFT imitates the teacher’s correct trajectories off-policy, OPD supervises the student’s own trajectories with token-level KL, and ExOPD adds reward extrapolation.Source: H Ablation on the Reverse-Distillation… · p. 17
• ExOPD > OPD > RFT across backbones; RFT→OPD gains about +0.1 to +0.8 average points, and reward extrapolation adds a further +1.1 to +1.5 points.Table 6 summarizes the reverse-distillation objective ablation.Source: H Ablation on the Reverse-Distillation… · p. 17
• The authors attribute the gains to the stronger expanded teacher leaving the student room to improve, with reward extrapolation providing a cleaner token-level signal than plain OPD or RFT targets.Source: H Ablation on the Reverse-Distillation… · p. 17

I Experiments on the Effect of the Number of Activated Experts

• OLMoE-1B-7B and GPT-OSS-20B peak at maximum k′ values of 64 and 32, while DeepSeek-V2-Lite remains near its peak at k′=64.The authors use the maximum feasible activation budget as a simple default rather than selecting k′ on the test set.Source: I Experiments on the Effect of the… · pp. 17–18, Figure 9 · p. 18

J Why CU unlocks a higher capability ceiling

• On Qwen3-30B-A3B, CU lets the recovered expanded model overtake RL on Mk and converge about 0.03 reward higher, with higher policy entropy throughout CU.Held-out evaluation reports M⋆ k′ beating RL on Mk in every domain of every backbone; Figure 11 summarizes three domain averages.Source: J Why CU unlocks a higher capability… · p. 18, Figure 11 · p. 18, Figure 10 · p. 18

K Analysis of Teacher Choice

• Replacing RL on Mk with M⋆ k′ as the reverse-distillation teacher yields a clearly stronger student on every backbone.The comparison holds the student and ExOPD objective fixed; Table 7 covers the mathematics average.Source: K Analysis of Teacher Choice · pp. 18–19, Table 7 · p. 19

L Analysis of Student Surpassing the Teacher

• 36.2 vs. 35.3 on mathematics for DeepSeek-V2-Lite and 59.3 vs. 57.2 on code for GPT-OSS-20B show the student surpasses its teacher in most settings.Qwen3-30B-A3B is the exception: student and teacher are on par on mathematics, while the teacher remains slightly ahead on code.Source: L Analysis of Student Surpassing the… · p. 19, Table 8 · p. 19

M Prompt Templates

• Figures 12, 13, and 14 provide the ScienceQA, mathematics, and code prompt templates used throughout training and evaluation.Source: M Prompt Templates · p. 19, Figure 12 · p. 19, Figure 13 · p. 19

N AI Use Statement

• ChatGPT and Claude assisted with wording, phrasing, and grammar; authors reviewed the content and retained responsibility for the manuscript.The tools did not independently contribute to research ideation, experimental design, data collection, analysis, or interpretation.Source: N AI Use Statement · p. 19