Source-linked AI summary
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Yushuai Sun, Zikun Zhou, Lin Gao, Jun Yu, Wenjie Pei
TL;DR
MoE deployment remains memory-intensive because the full expert pool must be loaded, while existing pruning criteria may not align with predictive performance and subset search can be costly. MoRA learns router biases that balance predictive importance and routing diversity, then approximates pruned experts with retained experts. It outperforms other pruning baselines on nine zero-shot benchmarks at 50% routed-expert pruning.
Problem
MoE deployment requires loading the full expert pool, while existing pruning rankings may not align with predictive performance, subset search can be costly, and methods may overlook routing redundancy.
Source: §1 · pp. 1–2Method
MoRA learns per-expert router biases using language-modeling loss and a routing-diversity regularizer, then uses retained experts with affine transformations to approximate pruned-expert outputs.
Source: §1 · p. 2Results
At 50% routed-expert pruning, MoRA outperforms other pruning baselines on nine zero-shot benchmarks while better preserving predictive performance.
Source: §1 · p. 3Takeaways & Limitations
Across three MoE architectures, MoRA preserves predictive performance with minimal additional parameter overhead.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still requires loading the complete expert pool into memory. Structured expert pruning can effectively reduce the memory usage by removing experts. However, existing pruning methods either use expert ranking criteria that are not well aligned with model performance or rely on effective expert subset searching that is computationally expensive. Moreover, these methods typically overlook the routing-behavior redundancy among the retained experts. In this paper, we propose MoE Pruning via Router Bias Learning and Expert Approximation (MoRA), a framework for structured MoE expert pruning. We introduce a learnable router bias for each expert and optimize these biases by minimizing the language-modeling loss and a routing-diversity regularizer. The learned router biases sharpen the routing probability distributions to identify experts critical to model performance while encouraging the selection of experts with diverse routing preferences. In addition, we introduce an expert approximation mechanism as a post-pruning enhancement. It leverages the remaining experts to approximate the outputs of pruned experts by affine transformation, further improving the performance of the pruned model. We evaluate MoRA on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B, removing 25\% and 50\% of the routed experts in each MoE layer. Extensive experiments on nine zero-shot benchmarks show that MoRA outperforms state-of-the-art pruning algorithms. Our code will be released.
1 Introduction
MoE pruning can reduce the memory cost of deployment, but selecting experts that preserve predictive behavior remains challenging. MoRA learns expert-selection biases and adds an approximation step for pruned experts; it outperforms other pruning baselines on nine zero-shot benchmarks at 50% routed-expert pruning.
1 Introduction
- Existing pruning ranks experts by aggregate routing statistics that may not align with predictive performance, while subset reconstruction search can be costly and neglect routing diversity.Frequency-based pruning may favor experts processing common function words; reconstruction methods evaluate candidate subsets against layer-output error.Source: §1 · pp. 1–2
- MoRA’s learned router biases are optimized to sharpen routing probabilities and establish an expert ranking for subsequent selection.Figure 1 contrasts this with fixed routing-statistic scores and reconstruction-based candidate-subset selection.Source: Figure 1 · p. 2
- MoRA learns a router bias for each expert using language-modeling loss and a routing-diversity regularizer to prioritize predictive importance and diverse routing preferences.The regularizer discourages experts with similar routing preferences from receiving high biases simultaneously.Source: §1 · p. 2
- As a post-pruning enhancement, expert approximation uses lightweight affine transformations to approximate pruned-expert outputs with retained experts.The paper reports that this mitigates performance degradation caused by pruning.Source: §1 · p. 2
- At 50% routed-expert pruning, MoRA outperforms other pruning baselines on nine zero-shot benchmarks while better preserving predictive performance.The evaluation includes Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B, with 25% and 50% routed-expert pruning.Source: §1 · pp. 2–3
2 Related Work
Related work reduces MoE storage through weight-level or expert-level pruning, while other approaches merge or replace experts. The section contrasts their selection costs and approximation trade-offs.
2 Related Work
- Weight-level methods such as SparseGPT, Wanda, and MoE-Pruner prune individual expert weights, but typically need specialized hardware for inference speedup.Expert-level pruning removes entire experts and directly reduces storage requirements.Source: §2 · p. 3
- Early expert-level methods rank experts by activation frequency and gating scores [Koishekenov et al., 2023], while NAEE minimizes layer-output reconstruction loss through subset selection [Lu et al., 2024].Enumerating subsets becomes expensive as expert counts and pruning budgets increase.Source: §2 · p. 3
- STUN uses expert clustering and parameter-space approximations to avoid exhaustive subset evaluation, while merging or replacing experts can introduce approximation errors.The section also identifies expert merging [Li et al., 2024; Chen et al., 2025] and replacement with lightweight modules [Zhang et al., 2026; Hao et al., 2026].Source: §2 · p. 3
3 MoRA
MoRA learns router biases to rank experts by predictive importance while accounting for routing diversity, then uses affine approximation as a post-pruning enhancement.
3 MoRA
- MoRA optimizes a learnable router bias for each expert with language-modeling loss and a routing-diversity regularizer, using the resulting biases as selection scores.The scores balance predictive importance and routing diversity.Source: §3 · p. 3
- For each removed expert, MoRA selects the retained expert with the highest proxy fidelity, measured using a fitted channel-wise affine transformation.The transformation is fitted from each retained candidate's output to the removed expert's output.Source: §3 · p. 3
- During inference, MoRA reuses the fitted affine map to approximate each pruned expert's output with its selected retained-expert proxy.This approximation is applied after pruning.Source: §3 · p. 3
3.1 Preliminary
The preliminary setup describes a routed MoE layer in which a token receives outputs from its selected top-k experts, combined using their mixing weights.
3.1 Preliminary
- For a token representation, the router selects a top-k set from the layer’s routed experts.The setup denotes the layer’s number of routed experts and its token-dependent selected set.Source: §3.1 · p. 3
- Each selected expert implements a function, and its output contribution is weighted by the router’s mixing weight.Source: §3.1 · p. 3
3.2 MoE pruning: router bias learning
MoRA seeks to preserve predictive performance while retaining experts with diverse routing preferences. It learns per-expert router biases and uses them to select experts after calibration.
3.2 MoE pruning: router bias learning
- MoRA learns per-expert router biases with the language-modeling loss and a routing-diversity regularizer, using the resulting scores to balance predictive utility and routing diversity.The target is a retained set that preserves next-token prediction while covering complementary routing preferences.Source: §3.2 · p. 4
3.2.1 Learnable Router Biases
- Each learnable bias is added to a frozen pretrained router to sharpen routing probabilities and express a global expert preference across tokens.The original logit remains token-dependent; the bias strength and calibration temperature control the adjustment.Source: §3.2.1 · p. 4
3.2.2 Optimizing the Biases for Predictive Utility and Routing Diversity
- The LM loss aggregates experts’ effects on predictive performance across calibration tokens, while the diversity regularizer addresses overlap in routing preferences.For similar expert pairs, the penalty suppresses the lower-bias expert to encourage priority separation.Source: §3.2.2 · pp. 4–5
3.2.3 Expert Selection from Learned Router Biases
- After calibration, MoRA ranks routed experts by learned bias in each layer and physically removes those outside the retention budget.The learned biases are discarded afterward, so they add no inference overhead.Source: §3.2.3 · p. 5
3.3 Post-pruning Enhancement: Expert Approximation
MoRA mitigates pruning-related performance degradation by using retained experts to approximate removed experts’ outputs. It selects proxies by directed approximation fidelity and applies their fitted affine maps during inference.
3.3 Post-pruning Enhancement: Expert Approximation
- MoRA fits a channel-wise affine map from each retained candidate to each removed expert, selects the candidate with the highest approximation fidelity, and reuses that map after pruning.This post-pruning enhancement is intended to mitigate performance degradation from removing experts.Source: §3.3 · p. 5
3.3.1 Directed Proxy Fidelity Estimation
- Proxy fidelity evaluates each affine predictor on a scoring split using target-conditioned token weights, and the highest-fidelity retained candidate is assigned to each removed expert.The relation is directional: the map predicts a removed target from a retained candidate, with both predictor and weights conditioned on the target.Source: §3.3.1 · p. 6
3.3.2 Expert Approximation based on Affine Transformation
- During inference, MoRA substitutes the removed expert’s output with its assigned proxy’s affine-transformed output only when proxy fidelity exceeds a prescribed threshold.The exported model stores retained experts, mappings, and fitted affine parameters; each removed expert needs 2d_o affine parameters.Source: §3.3.2 · p. 6
4 Experiments
Across three MoE models and nine zero-shot benchmarks, MoRA leads pruning baselines at both tested pruning budgets; ablations and routing analyses probe its expert-selection strategy.
4.1 Experimental Setup
- Experiments prune 25% or 50% of routed experts across three MoE models, calibrate on C4, and evaluate zero-shot performance on nine benchmarks.Shared experts and non-expert parameters are preserved, and the original routing top-k remains unchanged.Source: §4.1 · pp. 6–7
4.2 Main Results
- MoRA achieves the highest average on all three models at both pruning ratios, beating the strongest baselines by 1.14/0.43/1.01 points at 25% and 6.43/0.56/0.97 points at 50% on Qwen3/DeepSeek/Moonlight.The model order for both sets of differences is Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
- At 25% pruning, MoRA's average score drops only 0.60, 2.37, and 2.59 percentage points on Qwen3, DeepSeek, and Moonlight, respectively.Affine transformations add less than 0.08% of retained parameters and at most 0.0005% of original-model inference FLOPs.Source: §4.2 · p. 7, Table 4 · p. 9
4.3 Ablation Study
- At 25%/50% pruning, joint LM-and-diversity scores are 72.17/65.30; removing diversity costs 4.84/7.22 points, while removing LM costs 4.69/10.73 points.Source: Table 5 · p. 9, §4.3 · p. 9
4.4.1 Learned Router Biases and Expert Deletion Sensitivity
- Bias-adjusted gate mass improves over pretrained gate mass in all six settings, but trails direct bias ranking by 6.49 and 2.64 points at 50% pruning on Qwen3 and DeepSeek.Figure 2 compares score correlations with deletion-induced loss increases for the top 50% of experts per layer.Source: §4.3 · p. 9, Table 6 · p. 10, Figure 2 · p. 10
4.4.2 Token-category routing analysis
- Content words tend to route to higher-bias experts, whereas numbers and punctuation tend to route to experts with higher activation frequencies.The Figure 3 heatmap shows token categories by row, with darker cells representing higher routing probabilities and experts ordered by bias or frequency.Source: §4.4.2 · p. 10, Figure 3 · p. 11
5 Conclusion
MoRA learns router biases to prioritize experts critical to predictive performance, then uses retained experts to approximate pruned experts’ outputs. Experiments across three MoE architectures show it outperforms evaluated baselines with minimal additional parameter overhead.
5 Conclusion
- Across three MoE architectures, MoRA outperforms evaluated baselines while preserving predictive performance with minimal additional parameter overhead.Source: §5 · p. 11
- Learned router biases sharpen routing distributions and prioritize experts critical to predictive performance for retention.Source: §5 · p. 11
- Retained experts approximate pruned experts’ outputs through channel-wise affine transformations.Source: §5 · p. 11