Post-Trained MoE Can Skip Half Experts via Self-Distillation

Xingtai Lv; Li Sheng; Kaiyan Zhang; Yichen You; Siyan Gao; Xueheng Luo; Yuxin Zuo; Yuchen Fan; Junlin Yang; Ganqu Cui; Bingning Wang; Fan Yang; Youbang Sun; Ning Ding; Bowen Zhou

Source-linked AI summary

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Xingtai Lv, Li Sheng, Kaiyan Zhang, Yichen You, Siyan Gao, Xueheng Luo, Yuxin Zuo, Yuchen Fan, Junlin Yang, Ganqu Cui, Bingning Wang, Fan Yang, Youbang Sun, Ning Ding, Bowen Zhou

arXiv:2605.18643v2May 18, 2026cs.LGcs.AIcs.CL

TL;DR

Fully trained static MoE models are difficult to convert to dynamic ones because existing methods focus on from-scratch pre-training or task-specific adaptation, while direct conversion risks disrupting learned routing and capabilities. ZEDA injects parameter-free zero-output experts and applies two-stage self-distillation with the original MoE as teacher plus group-level balancing. Across two MoE models and 11 benchmarks, it removes over half of expert computation with marginal accuracy loss, outperforms the strongest baseline by 6.1 and 4.0 points, and provides roughly 20% inference speedup.

Problem
Existing dynamic MoE methods largely target from-scratch pre-training or task-specific adaptation, leaving migration of fully post-trained MoE models underexplored and risking disruption of learned routing and capabilities.
Method
ZEDA injects parameter-free zero-output experts, then adapts the augmented model through SFT and OPD self-distillation using the original MoE as a frozen teacher and group-level balancing.
Results
Over 50% of expert computation is eliminated across 11 benchmarks with marginal accuracy loss; ZEDA also beats the strongest baseline by 6.1 and 4.0 points on Qwen and GLM, respectively.
Takeaways & Limitations
Post-trained MoE models can be adapted into efficient dynamic models through zero-expert injection and self-distillation while retaining performance across diverse domains.
Takeaways & Limitations
The study evaluates 30B-scale MoE models but does not yet test substantially larger deployments because of computational resource constraints.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup.

1. Introduction

Dynamic MoE can allocate fewer experts to easy tokens, but converting fully post-trained static MoE models remains largely unexplored and risks disrupting learned routing and capabilities. ZEDA addresses this gap by adding zero experts and self-distilling the augmented model, reducing computation with marginal accuracy loss.

Motivation: Dynamic MoE adjusts activated experts per token, enabling input-dependent computation budgets and fewer experts for easy tokens.This makes dynamic routing a route to inference-time efficiency.
Research gap: Existing approaches mainly pre-train dynamic MoE models from scratch or adapt pre-trained models for specific tasks, leaving fully trained MoE migration largely unexplored.Post-trained MoE models commonly include pre-training followed by SFT, RL, and on-policy distillation.
Research gap: Directly applying existing dynamic MoE methods risks disrupting routing and capability distributions established during the full training pipeline.The paper therefore asks whether post-trained MoE models can be migrated without sacrificing established capabilities.
Approach: ZEDA transforms post-trained MoE models into dynamic ones by injecting parameterless zero-output experts and adapting the expanded model through two-stage self-distillation.The original MoE serves as the teacher, while group-level balancing regulates zero-expert utilization.
Results: Over 50% of expert computation is eliminated across 11 math, code, and instruction-following benchmarks, with marginal accuracy loss and approximately 20% inference speedup.On Qwen3-30B-A3B and GLM-4.7-Flash, ZEDA outperforms the strongest baseline by 6.1 and 4.0 points, respectively.

2. Method

ZEDA expands each post-trained MoE with zero-output experts, then uses two-stage self-distillation and group-level balancing to reduce normal-expert computation while preserving learned routing structure.

Adaptation framework: ZEDA augments each MoE module with zero experts and adapts the expanded model through self-distillation using the original MoE as a fixed teacher.The method is designed to convert a static post-trained MoE into a dynamic model with faster inference.
Zero-expert injection: Zero experts always output zero, expand the router candidate pool without changing the top-K budget, and reduce active normal experts token by token.Normal expert parameters remain unchanged, while zero-expert selection creates token-dependent computation.
Zero-expert injection: New zero-expert router parameters are initialized from a Gaussian matching the original router parameters’ mean and variance.This preserves the post-trained scale of router logits while adding routing options.
Two-stage self-distillation: The SFT stage trains the student on teacher-sampled responses, and the subsequent OPD stage aligns it with the teacher under the student’s own rollout distribution.SFT stabilizes the initial static-to-dynamic transition, while OPD performs on-policy alignment.
Group auxiliary loss: The Group Auxiliary Loss regulates zero-versus-normal expert utilization while preserving relative routing structure within the normal-expert group.Unlike vanilla expert-level balancing, it applies constraints only between the normal and zero groups because post-trained routing is non-uniform and input-dependent.

3. Experiments

Experiments evaluate ZEDA on two post-trained MoE backbones across 11 benchmarks, measuring performance, adaptation cost, expert-computation reduction, and inference efficiency. ZEDA preserves competitive performance while roughly halving expert computation and improving inference speed.

Evaluation Setup: ZEDA is evaluated on Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning mathematical reasoning, code generation, and instruction following.The experiments use two different backbone architectures and expert configurations.
Performance: ZEDA achieves the highest average evaluation scores among all baselines on both Qwen3-30B-A3B and GLM-4.7-Flash while incurring only marginal average accuracy loss versus the original post-trained MoE.It also surpasses the original model on several individual benchmarks and preserves competitive performance across domains.
Adaptation Time: Less than 31 hours for Qwen3-30B-A3B and 62 hours for GLM-4.7-Flash on 8 NVIDIA H200 GPUs are required for ZEDA adaptation.The reported times are described as negligible compared with prior MoE pre-training and post-training costs.
Expert Computation: 51.2% and 53.0% average zero-expert activation ratios on Qwen and GLM, respectively, effectively halve expert-level computation.These ratios exceed or match the baselines, indicating comparable or lower computation at stronger overall performance.
Inference Efficiency: Approximately 20% speedup is achieved during both prefill and decode phases for ZEDA relative to the original model at 8192 sequence length.Inference efficiency is measured with matched input and output token counts across compared models.

4. Analysis

ZEDA’s analysis shows that zero-expert activation varies primarily with token-level distributional gap and uncertainty, while remaining largely independent of overall task difficulty and layer position. Adaptation reaches a stable performance-routing regime early, and the full SFT→OPD sequence best balances efficiency and accuracy.

Zero Expert Activation Dynamics: Larger teacher-student logp differences or higher entropy correspond to lower r_ZE, so the model activates more normal experts for uncertain tokens.This relationship was observed across tokens from 110 sampled rollouts.
Zero Expert Activation Dynamics: Code fragments and mathematical expressions exhibit higher r_ZE than natural text, while instruction-following responses show more uniform activation patterns.Math and code responses therefore tend to show increasing average r_ZE toward the response end.
Zero Expert Activation Dynamics: r_ZE remains comparable across MATH-500 difficulty levels and AIME24, indicating allocation follows token-level characteristics rather than overall task difficulty.The analysis reports comparable performance and r_ZE across all five MATH-500 levels.
Zero Expert Activation Dynamics: Layer-wise r_ZE differences are minor and show no systematic pattern across the dynamic model’s 48 MoE layers.The analysis uses per-response distributions across layers and response positions.
Effect of Adaptation Cost: Average benchmark score and r_ZE rise rapidly with SFT data and GPU hours before converging at approximately 60k prompts.Additional supervised adaptation after this point mainly provides incremental refinements.
Ablations: Increasing the zero-expert group weight monotonically raises r_ZE while gradually lowering benchmark scores, making w a quality-efficiency control knob.The reported empirical setting identifies w=2 as the optimal balance.
Ablations: α=0.1 produces r_ZE closest to the 50% target while keeping average accuracy comparable to the original model.The coefficient controls the influence of the Group Auxiliary Loss.
Ablations: The full SFT→OPD pipeline consistently outperforms SFT-only and OPD-only alternatives under matched or greater single-stage compute.SFT stabilizes routing before OPD closes the remaining distribution gap on on-policy rollouts.

5. Related Work

Related work improves MoE efficiency through expert reduction or dynamic activation, but most dynamic approaches require pre-training or substantial adaptation. ZEDA instead targets low-cost post-training migration using zero-computation experts and self-distillation.

Efficiency-oriented MoE methods: MoE efficiency work includes expert pruning, merging, compression, and dynamic expert activation.Dynamic activation replaces fixed top-k routing with input-dependent computation.
Dynamic expert activation: Prior dynamic methods include jointly tuning expert counts and activation budgets, threshold routing, and zero-computation experts that let tokens bypass expensive FFN computation.These approaches differ in how they determine variable-size expert sets or computation paths.
ZEDA’s positioning: ZEDA operates entirely after training, avoiding expensive re-pretraining or substantial router redesign while following the zero-computation expert paradigm.Its design augments post-trained MoE modules and adapts them through self-distillation.
Knowledge distillation: Knowledge distillation has expanded from softened teacher outputs to sequence-level, supervised, and on-policy language-model distillation.The paper situates ZEDA within this broader distillation lineage.
Self-distillation for architecture adaptation: Recent architecture-adaptation methods use self-distillation to convert full-attention layers into more computationally efficient alternatives while maintaining performance.ZEDA extends this architecture-adaptation use of self-distillation to post-trained MoE models.

6. Conclusion

ZEDA migrates post-trained static MoE models to dynamic ones by combining zero-expert injection, group-level balancing, and two-stage self-distillation. Across architectures and benchmarks, it removes over half of expert computation with negligible performance impact and significant inference speedups.

Conclusion: ZEDA combines zero-expert injection and two-stage self-distillation to migrate post-trained static MoE models into dynamic ones.The group auxiliary loss regulates computation allocation while preserving the original routing distributions.
Conclusion: ZEDA eliminates over half of expert computation and provides significant inference speedups with negligible impact on model performance.The conclusion reports this pattern across multiple architectures and benchmarks.

A. Limitations and Future Work

The evaluation is bounded by model scale, workload type, and sequence length. Larger MoE deployments and long-horizon agentic tasks remain untested, while speedups diminish as sequences become longer.

Limitations: The study evaluates 30B-scale MoE models but does not test substantially larger deployments because of computational resource constraints.This limits evidence about scaling ZEDA beyond the evaluated model size.
Limitations: The evaluation excludes long-horizon agentic workloads because mature open-source infrastructure and training recipes are limited.Experiments are confined to standard post-training tasks.
Limitations: ZEDA’s speedup gradually diminishes as sequence length increases, although it remains approximately 20% at 8k tokens.The paper identifies integration with advanced communication frameworks as future work.

B. Zero Experts versus Copy Experts

Zero experts are preferable to copy experts for converting post-trained MoE models to dynamic computation because they preserve the original output more closely and avoid severe accuracy degradation.

Design comparison: Zero experts provide true expert omission, whereas copy experts add an input-carrying component despite both having negligible computational cost.The copy-expert output includes an additional component beyond the normal expert contribution.
Benchmark comparison: 53.2% vs. 52.7% activation ratios accompanied substantially worse performance for copy experts across all five mathematical reasoning benchmarks.The comparison holds training data, SFT, and routing regularization constant.
Benchmark comparison: 1.0, 2.9, and 0.8 were the copy-expert scores on AIME 24, AIME 25, and AIME 26, respectively.The gap was especially pronounced on these challenging AIME tasks.
Mechanistic analysis: Zero-expert outputs remain better aligned with original MoE outputs in both scale and direction across layers.Copy components create the main scale and directional mismatches, increasingly pulling normal expert outputs away from the original direction at deeper layers.

C. Auxiliary-Loss Comparison

Group-level balancing substantially improves adaptation quality over standard expert-level balancing while retaining zero-expert activation, with the group weight controlling a quality-efficiency trade-off.

Experimental setup: The comparison uses identical architecture and SFT adaptation recipes while varying only the balancing objective and group weight.The evaluated variants include standard expert-level loss and two group auxiliary-loss settings.
Results: 82.2 average accuracy at w=1.0 nearly recovered the original model’s 82.8 while maintaining a similar zero-expert activation ratio.Replacing the original auxiliary loss with the group auxiliary loss raised average accuracy from 59.5 to 82.2.
Results: 81.0 average accuracy at w=2.0 coincided with r_ZE=52.7, indicating a quality-efficiency trade-off.Increasing the group weight raised zero-expert activation while preserving strong accuracy.
Mechanism: Group-level balancing regulates competition between normal and zero-expert groups while preserving relative routing structure among normal experts.Expert-level uniformity can disrupt the non-uniform, input-dependent routing patterns learned by the post-trained model.

D. Theoretical FLOPs Analysis

The FLOP analysis compares original and ZEDA-adapted MoE models per Transformer layer, counting dominant matrix-multiplication costs while omitting lower-order operations.

Counting convention: 2mnp FLOPs are counted for a [m,n] × [n,p] matrix multiplication.The analysis focuses on dominant matrix multiplication terms.
Scope: Per-layer FLOP expressions omit normalization, residual, activation, routing top-k selection, and softmax overhead.Multiplying by the number of layers does not change ZEDA/original FLOP ratios when configurations are shared.
Notation: The notation used throughout the theoretical FLOP analysis is summarized in Table 11.The table provides the symbols referenced by the subsequent prefill and decode expressions.

D.1. Shared MoE Cost Decomposition

Original and ZEDA models share attention and router-cost structures across prefill and decode, while ZEDA reduces expert FFN computation according to zero-expert activation and expands router scoring.

Shared MoE Cost Decomposition: n denotes the token count processed in the current forward pass, and original MoE tokens activate K normal experts.The MoE FFN and router costs have the same form in prefill and decode; only the processed token count differs.
Shared MoE Cost Decomposition: For ZEDA, only a (1 − r_ZE) fraction of activated experts performs FFN computation, while the router scores normal and zero-computation experts.The reduced FFN term and expanded router candidate pool determine the adapted-model cost.
D.2. Prefill Stage: In prefill, n=l because all l tokens are processed in parallel.Substituting n=l into the MoE costs and adding attention yields the total prefill FLOPs.
D.2. Prefill Stage: The prefill ZEDA expression leaves attention unchanged, scales expert FFN cost by (1 − r_ZE), and increases router cost from N to N+N_Z scored experts.The first two terms represent attention, the third MoE FFN cost, and the last router cost.
D.3. Decode Stage: During decode, each step processes one new token while attention also uses cached tokens, and total cost sums across l decode steps.The decode analysis assumes standard KV caching.
D.3. Decode Stage: Decode preserves the original attention term while applying the same ZEDA MoE changes: reduced FFN computation and router expansion to N+N_Z outputs.The resulting decode FLOP ratio is obtained from the original and ZEDA decode expressions.

D.4. Numerical Results

The FLOP analysis compares theoretical and empirical prefill and decode speedups for Qwen3-30B-A3B across sequence lengths. Speedups decline as sequences lengthen, with decode consistently benefiting more than prefill.

Table 13 reports theoretical and empirically measured prefill and decode speedups for sequence lengths from 1024 through 8192 with r_ZE = 0.5.The theoretical values are obtained by taking reciprocals of the FLOP ratios.
1.403× to 1.178×: prefill theoretical speedup declines as sequence length increases from 1024 to 8192.Decode theoretical speedup similarly declines from 1.443× to 1.261× over the same range.
Empirical speedups broadly follow the theoretical predictions but remain lower because of implementation overheads and unmodeled computational costs.
Decode speedup is consistently higher than prefill speedup at fixed sequence lengths.The unchanged attention cost is smaller during decode, so reduced MoE computation represents a larger share of total FLOPs.