PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Yifan Yang, Hanling Yi, Dongquan Yang, Feng Lin, Rong Xiao, Aijun Li, Xiaotian Yu

arXiv:2610.29662v1cs.LGcs.CL

TL;DR

MoEs reduce computation through conditional activation but retain a large memory footprint, while expert pruning methods commonly focus on which experts to remove rather than how to distribute pruning across layers. SPARE uses token-level routing entropy to allocate layer budgets, and improves average performance over uniform allocation in evaluated settings, with larger gains at higher pruning ratios.

  • Problem

    MoEs reduce FLOPs through conditional computation but still require loading all experts, and expert-pruning methods commonly focus on expert selection rather than cross-layer budget allocation.

    Source: §1 · p. 1

  • Method

    SPARE is a training-free method that uses token-level routing entropy to allocate non-uniform pruning budgets across layers while allowing existing selectors to choose which experts to remove.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    At 50% pruning, SPARE raises Qwen3-30B-A3B’s eight-task average from 57.2% to 63.8% and DeepSeek-R1’s math, code, and science average from 80.1 to 82.8, with larger gains at high pruning ratios.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Under the same expert selector and global budget, entropy-guided allocation improves average performance over uniform allocation in the evaluated settings.

    Source: §7 · p. 8

    SPARE is studied only for expert pruning; combining it with quantization or expert merging is left for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.

1 Introduction

SPARE separates inter-layer budget allocation from intra-layer expert selection, using routing entropy to allocate pruning budgets without retraining. With the same selector and budget, it improves average accuracy over uniform allocation, particularly at high pruning ratios.

1 Introduction

  • MoE conditional computation reduces FLOPs but still requires loading all experts, creating memory overhead and deployment bottlenecks.Distributing experts across devices also complicates communication topology.Source: §1 · p. 1
  • Expert-importance scores can be unreliable across layers because their scales and calibration-domain token distributions vary, motivating separate inter-layer allocation and intra-layer selection.SPARE addresses allocation and can be paired with criteria such as REAP or EASY-EP for selection.Source: §1 · p. 1
  • High-entropy layers spread gate weight across active experts, so removing one discards a smaller share of each token’s weight than in concentrated-routing layers.SPARE uses this layer-level signal rather than assuming particular experts are indispensable.Source: §1 · pp. 1–2
  • SPARE assigns non-uniform pruning budgets across layers using token routing entropy, while existing criteria select which experts to remove within each layer.The method is training-free and does not modify the router.Source: §1 · p. 2
  • With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek, especially at high pruning ratios.At 50% pruning, the Qwen3-30B-A3B eight-task average rises from 57.2% to 63.8%, and the DeepSeek-R1 math, code, and science average rises from 80.1 to 82.8.Source: §1 · p. 2

2 Related Work

Prior work studies routing heterogeneity, expert selection and compression, and non-uniform sparsity allocation. SPARE differs by using per-token gate-weight entropy to allocate pruning budgets across MoE layers.

Routing Heterogeneity

  • Expert behavior can remain non-uniform despite load-balancing objectives, motivating analysis of routing heterogeneity across layers.Prior work examines super-experts and relationships between routing weights and expert outputs.Source: §2 · p. 2

Expert Pruning and Compression

  • MoE pruning work scores experts using routing statistics, activation saliency, reconstruction loss, or evolutionary search, while other approaches target different compression granularities.SPARE focuses on allocating budgets across layers rather than expert merging, skipping, dynamic Top-K routing, or intra-expert pruning.Source: §2 · p. 2

Layer-wise Sparsity Allocation

  • SPARE averages each token’s normalized Top-K gate-weight entropy and prunes more experts in higher-entropy layers, unlike usage-entropy allocation, which retains more experts there.Usage entropy measures how broadly expert identities are used across inputs; gate entropy measures how evenly weight is distributed among one token’s active experts.Source: §2 · p. 2

Routing Heterogeneity

  • Figure 1 contrasts diffuse and concentrated token gate distributions with expert-selection frequencies on C4 and CodeAlpaca, where selected identities can differ substantially across domains.SPARE uses layer-wise routing entropy rather than domain-specific expert frequencies to guide allocation.Source: Figure 1 · p. 3

3 Routing Entropy and Heterogeneity of MoE Layers

SPARE formulates expert pruning as a layer-wise budget allocation problem and uses routing entropy to distinguish layers with more diffuse or concentrated gating. In a Qwen3-30B-A3B intervention, pruning high-entropy layers was better tolerated than pruning low-entropy layers.

3.1 Problem Formulation

  • SPARE represents the pruning profile as the number of experts removed from each layer under a global target, leaving within-layer expert choice to a separate ranking rule.EASY-EP and REAP are examples of such selectors.Source: §3.1 · p. 3

3.2 Routing Entropy Gap

  • Routing entropy is the layer-average Shannon entropy of each token’s normalized Top-K gate weights; its gap from uniform routing indicates concentration.A larger gap motivates a more conservative pruning budget.Source: §3.2 · p. 3

3.3 Layer-wise Routing Heterogeneity

  • Layer-level entropy avoids relying on expert identities that vary across domains, but it does not establish whether remaining experts can compensate after pruning.Figure 1 motivates using layer-wise entropy rather than domain-specific expert frequencies.Source: §3.3 · pp. 3–4
  • In Qwen3-30B-A3B, pruning 50% of experts within high-, medium-, and low-entropy terciles yielded eight-task averages of 68.2%, 67.2%, and 66.4%, respectively.Each intervention pruned about 16.7% of all routed experts; the unpruned average was 72.1%.Source: §3.3 · p. 4

4 Theoretical Analysis: Routing-Entropy Perturbation Bounds

The analysis bounds pruning-induced output perturbations using the routing mass assigned to removed experts, then connects layer-level perturbations to network-level error. The resulting entropy signal motivates allocation, but the proposed score is a heuristic rather than a minimizer of the summed bound.

4 Theoretical Analysis: Routing-Entropy Perturbation Bounds

  • The analysis bounds pruning perturbations for fixed masks and derives an entropy-gap form for uniform random pruning under routing and bounded-output assumptions.It also gives a sufficient condition for a deterministic selector to inherit the entropy-dependent bound, with an additive term when that condition holds approximately.Source: §4 · p. 4

4.1 Single-Layer Pruning Perturbation

  • The single-layer analysis relates output perturbation to the routing mass assigned to removed experts and applies the bound to mask distributions, including uniform random pruning.The deterministic-selector result requires a second-moment condition; that condition is not established for REAP or EASY-EP.Source: §4.1 · p. 4

4.2 End-to-End Propagation as a Weighting Factor

  • Network-level error is decomposed into layer perturbations weighted by propagation factors under uniform Lipschitz assumptions on a common domain.The decomposition motivates a propagation regularizer but does not identify an exact estimator of the propagation factor.Source: §4.2 · p. 5

4.3 Implications for Pruning Strategy

  • For fixed conditions, the bound increases with the routing entropy gap, but SPARE uses that gap only to rank layers and ignores the layer-dependent scale Bℓ.The resulting allocation score is a heuristic; a cosine-based proxy serves as an empirical propagation regularizer because exact Lipschitz products are intractable for large MoEs.Source: §4.3 · p. 5

5 Entropy-Aware Sparsity Allocation

SPARE turns layer-wise routing entropy into an integer pruning profile under a global budget. An optional cosine-based propagation regularizer adjusts the entropy-driven allocation before discrete budget projection.

5 Entropy-Aware Sparsity Allocation

  • SPARE converts layer-wise routing entropy into an integer pruning profile under the global budget.Entropy is the primary allocation signal; a propagation-aware term can act as a lightweight regularizer.Source: §5 · p. 5

5.1 Entropy-Based Allocation Score

  • Layers with larger routing entropy receive larger pruning budgets, with a contrast coefficient controlling allocation sharpness.The method expresses this through a normalized allocation score based on the entropy gap.Source: §5.1 · p. 5

5.2 Propagation-Aware Regularization

  • Because exact Lipschitz products are intractable for large MoEs, SPARE uses cosine similarity as a data-dependent propagation proxy.The proxy is the average cosine similarity between a layer’s input and output activations; its term mildly reduces pruning weight for layers whose perturbations pass through more transformative downstream blocks, and β = 0 recovers entropy-only allocation.Source: §5.2 · p. 5

5.3 Discrete Budget Projection

  • SPARE projects continuous allocation weights onto a feasible integer pruning profile by solving for a global scaling factor λ∗.It rounds to the nearest feasible integer under 0 ≤ mℓ ≤ N − K and uses bisection because the constraint’s left-hand side is monotone in λ∗.Source: §5.3 · p. 5

6 Experiments

Experiments test whether entropy-guided allocation improves accuracy at matched pruning budgets, whether gains persist across selectors, and how pruning risk and calibration choices relate to routing entropy.

6 Experiments

  • The experiments ask whether entropy-guided allocation improves benchmark performance at the same pruning budget, generalizes across selectors, and remains useful across calibration domains.Source: §6 · p. 6

6.1 Experimental Setup

  • The evaluation spans four MoEs, ten tasks across four domains, calibration samples from C4 or reasoning benchmarks, and comparisons that separate expert ranking from layer-budget allocation.The allocation rules compared are Uniform, OWL, and SPARE; the expert selectors are Random, EASY-EP, and REAP.Source: §6.1 · p. 6

6.2 Main Results

  • SPARE improves macro-average accuracy in all ten evaluated model, pruning-ratio, and selector combinations; on Qwen3-30B-A3B at 50% pruning with EASY-EP, the mean rises from 57.2% to 63.8%.The gain is 6.6 percentage points; at the same pruning ratio, gains with Random and REAP are 0.9 and 2.3 points.Source: §6.2 · p. 6, Table 1 · p. 7

6.3 Controlled Analysis of Pruning Risk

  • With REAP fixed, low-entropy layers show higher perturbation than high-entropy layers in all three Mixtral domains; single-layer tests find moderate positive concentration–KL correlations of 0.26–0.45.The correlation explains only part of layer-to-layer pruning-risk variation, so concentration is used to rank layers for budget allocation rather than predict individual-layer perturbations.Source: §6.3 · p. 6, Table 3 · p. 7, Table 4 · p. 8

6.4 Scalability to Large-scale Reasoning MoEs

  • At 50% pruning with EASY-EP, SPARE raises DeepSeek-R1’s three-score mean from 80.1 to 82.8 and Qwen3-235B-A22B’s from 86.4 to 87.9.On DeepSeek-R1, GPQA-Diamond improves by 5.5 points and LiveCodeBench by 3.0, while MATH-500 declines by 0.4 points.Source: §6.4 · p. 7, Table 2 · p. 7

6.5 Comparison with Layer-wise Allocation Baselines

  • Against OWL with EASY-EP fixed, SPARE’s average accuracy is comparable at 12.5% pruning and higher at 25%, 37.5%, and 50%; its allocation profiles vary with p but remain similar across the tested β values.Direct comparisons with other MoE-specific allocation pipelines are outside the evaluation.Source: §6.5 · p. 7, Figure 2 · p. 8

6.6 Calibration Robustness

  • On DeepSeek-R1, layer-average entropy rankings are similar across C4, math, code, and science calibration domains, with Spearman correlations of 0.958–0.974 relative to C4.This measures entropy-profile stability, not identical integer budgets or unchanged downstream performance for every calibration choice.Source: §6.6 · p. 8, Table 5 · p. 8

6.7 Ablation on Hyper-parameters

  • In separate Qwen3-30B-A3B sweeps at 50% pruning, mean accuracy peaks at p = 2.0 (63.2%) and at β = 0.6 (63.8%); these are not a joint parameter grid.The p sweep fixes β = 0, while the β sweep fixes p = 2.0; larger p increases allocation-profile contrast, whereas profiles remain close across β ∈ [0, 0.6].Source: §6.7 · p. 8, Table 6 · p. 8

7 Conclusion and Future Work

SPARE uses token-level routing entropy to allocate expert-pruning budgets across MoE layers and can pair with existing intra-layer selectors. Its analysis and controlled interventions associate concentrated routing with greater pruning risk, while evaluations report gains over uniform allocation.

7 Conclusion and Future Work

  • SPARE is training-free and uses token-level routing entropy to allocate pruning budgets across layers while remaining compatible with existing intra-layer selectors.It does not require retraining or modifying the router.Source: §7 · p. 8
  • At 50% expert pruning, Qwen3-30B-A3B’s eight-task average rises from 57.2% to 63.8%, and DeepSeek-R1’s three-benchmark average rises from 80.1 to 82.8.The DeepSeek-R1 result is close to the unpruned 83.4.Source: §7 · p. 8
  • A perturbation analysis and controlled interventions with a fixed selector associate concentrated routing with greater pruning risk.Future work includes studying calibration shift and combining entropy-guided allocation with quantization and expert merging.Source: §7 · p. 8

Limitations

SPARE’s gains over uniform allocation are largest at aggressive pruning ratios, while its improvements are small at moderate ratios such as 25%. The study covers expert pruning, leaving combinations with quantization or expert merging for future work.

Limitations

  • At moderate pruning ratios such as 25%, SPARE’s improvements over uniform allocation are small.Its largest gains occur at aggressive pruning ratios.Source: Limitations · p. 9
  • The study evaluates SPARE only for expert pruning.Source: Limitations · p. 9
  • Combining SPARE with quantization or expert merging is left for future work.Source: Limitations · p. 9

A Additional results

On GLM-5, SPARE improves average accuracy over uniform allocation at 50% global expert pruning. The additional evaluation covers a fine-grained MoE architecture, while measured throughput remains similar across tested batch sizes.

A Additional results

  • GLM-5 has 256 routed experts, top-8 routing, one shared expert, and 78 MoE layers, compared with 61 layers in DeepSeek-R1.The models differ in routed expert count, routing top-K, shared expert design, and layer count.Source: A Additional results · p. 11, Table 7 · p. 11
  • At 50% global expert pruning with EASY-EP, GLM-5 average accuracy rises from 87.5% to 89.8%, a +2.3 point gain over uniform allocation.This extends the observed average improvement to another fine-grained MoE architecture.Source: A Additional results · p. 11
  • At 50% pruning, measured throughput differences from uniform allocation range from a 1.16% decrease to a 0.56% increase across tested batch sizes.Throughput is similar in these measured configurations.Source: A Additional results · p. 11, Table 8 · p. 11
  • Non-uniform allocation could allow layers retaining more experts to co-locate frequently activated experts, potentially reducing HBM transfer beyond uniform pruning.Source: A Additional results · p. 11

B Comparison with Layer-wise Allocation Methods

This section compares SPARE with adapted OWL on Qwen3-30B-A3B at several pruning rates and develops theoretical bounds on pruning perturbations and their amplification across layers. It also formalizes how routing entropy relates to perturbations under random pruning and specified assumptions.

B Comparison with Layer-wise Allocation Methods

  • SPARE has higher average accuracy than adapted OWL at 25%, 37.5%, and 50% pruning, while the methods are comparable at 12.5%.The comparison uses Qwen3-30B-A3B and holds EASY-EP fixed; OWL uses activation-aware weight outlier statistics, whereas SPARE uses token-level routing entropy.Source: B Comparison with Layer-wise… · p. 11

C.1.1 Data and Tokens

  • SPARE estimates layer routing entropy from a finite calibration set of M tokens, using normalized gate weights over each token’s Top-K support.The appendix defines Top-K gating as a distribution with K nonzero entries and entropy for that sparse distribution.Source: §C.1.1 · p. 11, §C.1.3 · p. 12, §C.1.6 · p. 12

C.1.5 Pruning Operation and Top-K Rerouting

  • After pruning, router logits remain unchanged and Top-K routing is recomputed over the remaining experts, so removed active experts’ original gate mass is redistributed.The recomputed support remains well defined when all originally active experts are removed, provided at least K experts remain.Source: §C.1.5 · p. 12

C.3 Single-Layer Pruning Perturbation

  • A selector minimizing expected squared removed gate mass satisfies the stated condition, but calibration residuals do not certify it on the evaluation distribution.The appendix also cautions that this condition is not established for selectors optimizing a different saliency criterion.Source: §C.3 · pp. 13, 15
  • Under bounded expert-output scale, random pruning’s expected perturbation bound increases with routing entropy gap, but does not order actual perturbations across layers.The proof links gate-distribution change to removed active gate mass, then bounds random-pruning collision mass using the entropy gap; the result assumes uniform random pruning and fixed layer scale and pruning parameters.Source: §C.3 · pp. 13, 15

C.4.1 Model: Residual Network with MoE Blocks

  • The residual-network analysis assumes locally Lipschitz pruned blocks and derives a one-step layer-error decomposition, with later layers governing upstream error amplification.The propagation factor depends on subsequent layers’ Lipschitz constants.Source: §C.4.2 · p. 16, §C.4.1 · p. 16

C.4.2 Key Lemmas: One-step recursion and discrete Gronwall bound

  • With zero initial error and fixed pruning masks, Lemma C.10 gives a global bound on final-layer error.The supplied passage introduces the bound but does not include its formula.Source: §C.4.2 · p. 17
  • Each layer’s injected error is scaled by later-layer expansion factors, whose product has an exponential upper bound.The passage states that perturbations are amplified only by later layers and that the product is bounded using 1 + x ≤ e^x.Source: §C.4.2 · p. 17

C.4.3 Main Theorem: End-to-end Expected Perturbation Bound

The theorem bounds expected final-layer ℓ1 perturbation under layer-wise random pruning, with layer errors weighted by downstream propagation. Under depth-scaled initialization, the propagation factor is bounded independently of depth, while SPARE’s allocation rule remains a ranking heuristic rather than an optimizer of the full bound.

C.4.3 Main Theorem: End-to-end Expected Perturbation Bound

  • The end-to-end theorem bounds expected final-layer ℓ1 perturbation under layer-wise random pruning and stated assumptions.The bound uses layer-specific downstream propagation factors.Source: §C.4.3 · p. 17
  • The proof obtains explicit and exponential forms by substituting a single-layer error bound into the generic bound and applying an exponential upper bound.The derivation starts from an unrolled discrete Gronwall bound and takes expectations over inputs and independent pruning masks.Source: §C.4.3 · pp. 17–18

C.5 Corollaries and Design Implications

  • Under depth-scaled initialization α = 1/L and λk ≤ λmax, the propagation factor is bounded independently of network depth.The stated upper bound is e^λmax.Source: §C.5 · p. 18
  • SPARE uses routing concentration only to rank layers; its allocation rule omits the layer-dependent output scale Bℓ and does not minimize the summed perturbation bound.The low-removal approximation does not apply to the main 50% pruning setting, where both coefficients approach 1/4 as Nℓ grows.Source: §C.5 · p. 18

D Communication cost analysis

With two expert-parallel nodes, pruning that enables consolidation to one node eliminates cross-node all-to-all traffic. The estimated savings scale with routed tokens per step.

D Communication cost analysis

  • With balanced expert ownership across two nodes, the expected cross-node fraction of all-to-all traffic is approximately one half.Each MoE layer has dispatch and combine exchanges; N is the number of routed tokens per step.Source: D Communication cost analysis · p. 18
  • Consolidating expert parallelism from two nodes to one makes cross-node traffic vanish, saving approximately 6.34 MiB · N per step.For decode with batch B, the saving is approximately 6.34 MiB × B per step.Source: D Communication cost analysis · p. 18
Loading 2610.29662v1…