Source-linked AI summary
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
Xing Chen, Hengshuai Yao
TL;DR
Reducing activated experts in fine-grained MoE models is confounded by renormalization, which changes expert-branch gain as well as expert identity. The paper decouples these effects by activating top k1 experts while normalizing with top-k2 probability mass, and finds that this training-free knob can halve routed-expert compute with small downstream losses across two model scales.
Problem
Inference-time reduction of activated experts is difficult to evaluate because standard renormalization changes the expert-branch gain in addition to removing experts.
Method
The method activates the top k1 experts and normalizes their outputs by the probability mass of the top k2 experts, using one integer without training or measurable compute overhead.
Results
4 of 8 experts reduce MMLU by 4.65 points under standard renormalization but 0.35 points at k2=16, while 5 of 10 experts lose 0.55 points with an appropriate reference set on the larger model.
Takeaways & Limitations
Reference-mass selection should be validated on downstream tasks because perplexity and downstream accuracy favor different k2 values, while expert identity matters more than expert weighting.
Takeaways & Limitations
Both evaluated models come from the same series, so cross-architecture generalization remains a hypothesis, and long-context and long-form generation are untested.
Abstract
from arXiv · showhide
Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with $k_2=16$, while halving routed-expert compute. The result replicates on the $11\times$ larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different $k_2$, cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.
Introduction
The paper identifies renormalization as a hidden source of degradation when reducing activated experts at inference, then decouples activation count from normalization mass to preserve expert-branch gain. This training-free change substantially improves half-expert performance while adding no measurable compute.
- In fine-grained routing, standard renormalization amplifies surviving experts because the top-8 probabilities sum to only 0.182, conflating expert loss with gain miscalibration.The measured degradation therefore reflects both removing experts and increasing the expert branch beyond its training-calibrated scale.
- The method activates the top k1 experts while normalizing by the probability mass of the top k2 experts, separating compute control from gain control.Setting k2=k1 recovers standard renormalization, while k2>k can reduce the expert-branch gain relative to its trained value.
- 4 of 8 activated experts cost 4.65 MMLU points under standard renormalization but only 0.35 points at k2=16 on Qwen3.6-35B-A3B.The paired test could not distinguish the k2=16 result from the top-8 baseline, while routed-expert compute was halved.
- 5 of 10 activated experts cost 2.10 points at k2=k1=5 but only 0.55 points at k2=10 on the 11× larger Qwen3.5-397B-A17B.
- Removing renormalization entirely costs 27.5 MMLU points and 356% perplexity, showing that the reference mass must remain appropriately calibrated.With activated experts held fixed, both shrinking and enlarging the reference set away from its optimum degrade performance.
Related Work
Prior work addresses MoE memory through pruning, merging, and quantization, while changing the number of activated experts has generally required training-based remedies. Related routing studies examine specialization and balance, but this paper focuses on inference-time gain calibration.
- Sparse MoE research established conditional computation and later scaled it to fine-grained models with many small experts and token-choice routing.
- Studies of learned routing report token-identity-driven behavior and balanced routers, while this work finds a stronger domain effect than within-domain resampling noise.
- Training-free compression methods primarily target memory through expert pruning, merging, and quantization rather than reducing activated-expert compute.
- Prior approaches to changing k after training include retraining, training expert combinations, null experts, and self-distillation.The closest prior work also argues that renormalizing survivors inflates the expert residual branch relative to its pretrained scale.
Background and Setup
The models use sparse routing with a trained top-k, but inference can decouple the number of activated experts from the probability mass used to scale their outputs. This preserves compute savings while making the gain independently adjustable without modifying weights.
- Models: Qwen3.6-35B-A3B activates 8 of 256 routed experts per token, while Qwen3.5-397B-A17B activates 10 of 512.The 35B model activates only 3.1% of expert parameters per token, creating a 32:1 memory-to-compute asymmetry.
- Router and gain: The router selects top-k experts from softmax probabilities and normally renormalizes their retained mass into weights summing to one.Renormalization makes the expert-branch gain depend on the selected expert count.
- Reference-set method: The method activates the top k1 experts but computes their weights using the retained mass of the top k2 experts.This keeps selection and compute under k1 while using k2 to control output gain.
- Reference-set method: Setting k2=k1 gives standard renormalization, k2=k anchors gain to training, and k2>k1 produces adaptive per-token down-scaling.When k2 exceeds k1, weights sum to a ratio below one rather than one.
- Implementation: Only k1 expert FFNs are evaluated, while selecting a larger k2 adds negligible cost because router probabilities for all experts are already computed.The method is implemented at inference time without changing weights.
- Evaluation: Perplexity and downstream evaluation use fixed token chunks and question sets, with paired comparisons against the native baseline.Perplexity uses WikiText-103 and CodeParrot; downstream tasks include MMLU, C-Eval, and GSM8K.
How the Router Actually Behaves
The routers are balanced but highly specialized: few experts are obviously redundant, and rankings transfer poorly across domains. Attention type also organizes routing behavior across layers.
- Load balance: Expert usage is well balanced, with almost no dead experts and steep routing-mass loss when retaining only half the experts.Top-half retention is 88.8% on WikiText and 81.3% on code, while the worst layer retains 64.2%.
- Domain specialization: Cross-domain expert distributions differ substantially, with mean Jensen–Shannon divergence of 0.258 nats and top-64 overlap of 0.164 versus a 0.250 random baseline.The observed divergence is roughly 8,200 times the estimated within-domain noise floor.
- Domain specialization: Domain-specific pruning rankings transfer poorly: code-derived top-128 experts retain 81.3% of code mass, whereas WikiText-derived rankings retain only 38.0% on the same domain.The two domains’ top-128 union covers 198 of 256 experts.
- Attention type: Full-attention layers share expert preferences, while linear-attention layers are near-independent and cross-type overlap is near chance.Full-attention top-64 overlap is 0.309 versus the 0.250 random baseline; linear-attention overlap is 0.258.
Weighting
Fixed-compute ablations indicate that expert identity dominates expert weighting, supporting corrections that adjust systematic gain while preserving the selected expert set.
- Fixed-compute ablations: +49.7% WikiText perplexity follows from discarding router weights while retaining the selected experts, compared with +310% from sampling and +1667% from uniform selection.Each comparison activates exactly 8 experts, so the compute budget is held constant.
- Implication: The results rule out approximate or stochastic selection schemes on this model while supporting weight-magnitude correction when expert choices remain intact.The paper describes exact argmax selection as effectively required and weight distortion as substantially more tolerable.
The Reference Set Controls the Damage
The reference set determines how much the reduced expert set is amplified: both removing renormalization and shrinking the denominator with compute are harmful, while intermediate choices recover performance. Perplexity consistently favors the trained k as the reference size.
- Routing mass: The top-8 experts carry only 0.182 of router mass, so standard renormalization applies a 5.50× gain tied to k=8.Keeping half the experts also loses 11–19% of routing mass across layers, with worse loss in the worst layer.
- Reference-set extremes: +356% WikiText perplexity and −27.45 MMLU points result from removing renormalization entirely, while shrinking the reference set with k1=6 costs +4.08% perplexity and 2.60 MMLU points.Intermediate k2 values in [k, 2k] recover nearly all of the loss, producing U-shaped perplexity and inverted-U MMLU curves.
- Perplexity: Perplexity is minimized at k2 equal to the trained k across both models and all six corpus/model combinations.The held-out WikiText split reproduces the ranking, indicating the choice is not an artifact of the tuning split.
Downstream Validation
Downstream evaluations show that preserving an appropriate reference mass largely prevents capability loss when activated experts are halved, but the best reference size differs across metrics and models.
- MMLU: 4.65 MMLU points are lost under standard renormalization when Qwen3.6-35B-A3B is reduced from 8 to 4 experts, versus 0.35 points at k2=16.The paired test gives p=0.66 for k2=16; the same compute is used in both settings.
- Metric disagreement: Perplexity is minimized at k2=8, whereas MMLU is optimized at k2=16 on Qwen3.6-35B-A3B.The perplexity-optimal setting is significantly worse on MMLU by 1.10 points, while the MMLU-optimal setting is 1.81% worse on perplexity.
- Metric disagreement: The optimal k2 is model-dependent and lies in [k, 2k], so the authors recommend scanning {k1, k, 2k} rather than using a universal value.The default k2=k1 is the worst choice in the reported grid once the cut is deep.
- Generation: 6.60 and 4.20 GSM8K points are lost under standard renormalization on the 35B and 397B models, versus 0.60 and 0.80 points with native-k reference sets.The corrected settings have p=0.68 and p=0.34, respectively, and neither loss is detectable.
- Cross-benchmark validation: C-Eval sensitivity depends on the model: the 35B default reduction is nonsignificant, while the 397B model loses 2.92 points at k1=k2=8.Anchoring to k2=10 recovers most of the 397B loss.
- Compute: Halving k1 from 8 to 4 halves routed-expert compute, cuts the total expert term by 44%, and reduces end-to-end compute by 17%.All experts remain resident, so memory is unchanged.
Discussion
The method is presented as a compute-side control for top-k reductions, while the paper cautions that pruning and broader generalization remain architecture- and setting-dependent.
- Practical recommendation: Scanning k2 over {k1, k, 2k} with paired testing provides a one-integer control for top-k reductions when compute or decode bandwidth binds.Without decoupled k2, published reductions can overstate the cost of lowering k.
- Pruning: The authors do not recommend expert pruning for this architecture because routing is balanced, retention falls quickly, and specialization is domain-bound.The two-domain union retains 198 of 256 experts, leaving little room for general-purpose pruning.
- Pruning: Quantization is presented as better matched to the memory problem because routed experts are homogeneous and account for 89.6% of parameters.It composes with the paper’s compute-only reference-set method.
- Scope: Cross-architecture generalization remains a hypothesis because both evaluated models come from the same series.The mechanism is expected to benefit coarse-grained MoEs little, and the preliminary third-architecture check is not directly comparable.
- Scope: The evaluation covers English and Python corpora, short-form generation, and multiple-choice benchmarks, but not long-form generation or long-context settings.The authors also note that the perplexity-optimal k2 is domain-dependent.
Conclusion
Decoupling expert activation from renormalization substantially reduces the accuracy cost of halving activated experts, but the best reference set depends on the evaluation metric.
- Conclusion: A one-integer k1/k2 parameterization turns a 4.65-point MMLU regression at half the activated experts into 0.35 points on Qwen3.6-35B-A3B.The method uses no training, parameters, or measurable compute.
- Conclusion: The same approach reduces the regression from 2.10 points to 0.55 points on the 11× larger Qwen3.5-397B-A17B.
- Conclusion: Perplexity and downstream accuracy favor different reference sets, so MoE compression settings should be validated on downstream tasks with paired testing rather than perplexity alone.
Appendix A: A Third Architecture
Preliminary Gemma-4-26B-A4B experiments provide directional evidence that scalar gain correction helps but does not fully recover reduced-expert performance. The results also replicate the paper’s flatness and selection-over-weighting patterns, while highlighting substantial protocol dependence.
- Scalar-gain correction: A scalar gain recovered up to 1.00 MMLU point after reducing from top-8 to top-6, but the best α=0.80 setting remained 1.30 points below baseline.The native α=1 setting cost 2.30 points from an 80.40% baseline; α=0.80 was the only non-significant setting.
- Router amplitude: Gemma’s router temperature was strongly trained, whereas all 3840 post-selection expert-amplitude parameters remained essentially at initialization.The temperature reached 32.06 ± 0.02 across layers, while expert scales stayed at 1.000 ± 0.011.
- Flatness and renormalization: Gemma’s flatter router makes renormalization less load-bearing: removing it at k1=6 increased WikiText perplexity by only 0.7%, unlike the much larger Qwen effect.At k2=E, perplexity rose from 9.158 to 9.224 and remained below standard renormalization’s 9.318.
- Selection versus weighting: With eight activated experts fixed, uniform weighting increased WikiText perplexity by 34.4%, while sampling and random selection increased it by 218% and 2436%.These results replicate the finding that expert identity matters more than expert weighting.
- Evaluation protocol: Gemma evaluation is protocol-dependent: perplexity uses the base variant, whereas MMLU uses the instruction-tuned variant with a chat template.Plain-completion perplexity is 573 for the instruction-tuned variant versus 9.6 for the base variant because bare completion triggers degenerate repetition.
- Implementation: The experiments implement the method as a router-forward replacement without modifying weights or adding measurable compute overhead.The router computation was numerically identical across Transformers versions 5.3–5.14, reproducing the WikiText baseline exactly.