PaperlayerView exact revision on arXiv

Source-linked AI summary

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, Vithursan Thangarasa

arXiv:2510.13999v3cs.LGcs.AI

TL;DR

SMoEs reduce pre-training cost and latency but their large parameter counts create memory overhead, and prior comparisons favoring merging had focused on discriminative benchmarks. The paper proposes REAP, a pruning criterion based on router gates and expert activation norms. Across generative evaluations, pruning performs strongly, including near-lossless code-generation compression at 50% for two large models.

  • Problem

    SMoEs have substantial memory overhead, and although prior comparisons favored merging on discriminative benchmarks, generative-benchmark comparisons had not been conducted.

    Source: §1 · p. 1

  • Method

    REAP prunes experts using router gate-values and expert activation norms to target minimal layer-output contribution and the reconstruction-error bound.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    REAP consistently outperforms merging and other pruning methods on generative benchmarks, and at 50% pruning achieves near-lossless code-generation accuracy (∆acc ≤2%) on Qwen3-Coder-480B and Kimi-K2.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results highlight the importance of preserving coordination between the router and experts when compressing SMoEs for generative tasks.

    Source: §6 · p. 10

    Discriminative metrics such as perplexity and log-likelihood-based multiple-choice benchmarks are not necessarily good proxies for generative model quality.

    Source: §6 · p. 10

Abstract

from arXiv · show

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.

1 INTRODUCTION

SMoEs reduce pre-training cost and latency but carry substantial memory overhead, motivating expert compression. The paper argues that pruning, with a suitable saliency criterion, outperforms merging on generative tasks and introduces REAP as its proposed criterion.

Motivation

  • SMoEs offer lower latency and more efficient pre-training than dense models, but require more parameters for similar accuracy, creating substantial memory overhead.Expert-usage imbalance can also impair accelerator utilization and increase latency or lead to dropped tokens.Source: §1 · p. 1

Research gap

  • Earlier comparisons found merging superior to pruning on perplexity and multiple-choice benchmarks, while generative-benchmark comparisons had not yet been conducted.This work highlights code generation, creative writing, and mathematical reasoning as generative tasks where pruning can outperform merging.Source: §1 · p. 1

Contributions

  • Existing expert merging loses the router’s independent, input-dependent modulation of expert outputs, introducing irreducible error and functional subspace collapse in high-granularity SMoEs.The paper also reports that merging distorts functional-manifold topology, whereas pruning preserves it.Source: §1 · p. 1
  • REAP combines router gate-values and expert activation norms to prune experts with minimal layer-output contribution, targeting the reconstruction-error bound.The paper reports consistent advantages across generative evaluations, especially at 50% compression.Source: §1 · p. 2
  • At 50% compression, REAP achieves near-lossless code-generation accuracy (∆acc ≤2%) on Qwen3-Coder-480B and Kimi-K2.The paper evaluates SMoEs ranging from 20B to 1T parameters.Source: §1 · p. 2

2 RELATED WORK

Related work covers SMoE architecture, expert pruning and merging, other compression methods, and LLM evaluation. Prior merging results favor merging over pruning on multiple-choice tasks, while generative evaluation measures model-generated responses.

Sparsely activated SMoE architecture

  • SMoE layers replace transformer feed-forward layers with experts and a router that selects a subset per input, enabling large parameter counts without a matching rise in computation.The architecture combines specialized subnetworks with router-produced gates that modulate their outputs.Source: §2 · p. 2

Expert pruning

  • Pruning research ranges from router-weight and frequency criteria to reconstruction-loss searches, evolutionary algorithms, activation norms, and gate-weighted output norms.REAP distinguishes itself by computing conditional means only over tokens where an expert is active, avoiding frequency-dominated global averages.Source: §2 · p. 2

Expert merging

  • M-SMoE and HC-SMoE use clustering and frequency-weighted parameter averaging; reported comparisons found them stronger than pruning methods on multiple-choice benchmarks.M-SMoE clusters using router-logit similarity, while HC-SMoE uses representative-vector similarity.Source: §2 · p. 2

Other compression techniques

  • Experts can also be compressed through quantization, low-rank decomposition, or weight sparsity; these approaches are described as orthogonal to pruning and merging.Merging, unlike pruning, requires re-quantization for block formats with shared scaling coefficients.Source: §2 · pp. 2–3

Model merging

  • Model-merging research includes parameter averaging and task-vector methods, with non-local merging more closely related to expert merging than merging fine-tunes from a shared checkpoint.Sharma et al. (2024) found activation re-scaling necessary for high-quality non-local merging.Source: §2 · p. 3

LLM evaluation

  • Perplexity and multiple-choice accuracy are discriminative metrics based on evaluating responses without generation, whereas generative benchmarks require models to produce responses.Code generation, structured mathematical reasoning, and creative writing are examples of generative tasks.Source: §2 · p. 3

3 MOTIVATION

The analysis contrasts summed-gate merging, which replaces dynamic router control with a static expert, with pruning, which retains independent control over surviving experts. Empirical manifold analyses support this distinction, with effects varying by routing granularity.

3.1 MERGING INDUCES AN INPUT-DEPENDENT TARGET A SINGLE EXPERT CANNOT REALIZE

  • After merging, the summed gate weights a constant convex combination, so a static expert must approximate the original input-dependent target.The original pair’s contribution depends on the router’s input-dependent mixing ratio.Source: §3.1 · pp. 3–4
  • Merging with summed gates incurs irreducible error when routing policy varies across inputs and the two experts are functionally distinct.The minimum error is proportional to Var[r(x)] under the stated simplifying assumptions about correlations.Source: §3.1 · p. 4

3.2 PRUNING PRESERVES INDEPENDENT CONTROL

  • Pruning incurs error only when the removed expert is selected, and its substitution error scales with that expert’s gate-value rather than routing-policy variability.Merging ties gates and loses independent modulation; pruning retains independent control over surviving experts.Source: §3.2 · p. 4
  • In high-granularity SMoEs, variable routing combines many small expert contributions, a setting where the analysis expects summed-gate merging to be disadvantaged.The pruning analysis identifies substitution error as dominant over renormalization error.Source: §3.2 · p. 4

3.3 EMPIRICAL EVIDENCE FOR LOSS OF INDEPENDENT CONTROL

  • In high-granularity Qwen3 and ERNIE-4.5, merging contracts expert activation manifolds toward their centers, while pruning preserves surviving experts’ positions on the original manifold.The projection analysis used mean activations from 32 samples of 2048 C4 tokens across four architectures.Source: §3.3 · p. 5
  • Across models and layers, pruning preserves functional-manifold topology, whereas merging alters it through loss of independent router control and introduction of novel functions.Merged outputs also have consistently higher transport cost from the original manifold.Source: §3.3 · p. 5
  • In low-granularity Llama-4-Scout and Mixtral, merging-related subspace collapse is less apparent, but merging still introduces greater manifold distortion than pruning’s substitution error.These architectures have fewer experts per layer and active experts per token, less variable routing, and higher gate-values.Source: §3.3 · p. 5

4 ROUTER-WEIGHTED EXPERT ACTIVATION PRUNING (REAP)

REAP ranks experts by router-weighted activation magnitude to minimize the substitution error bound. Unlike frequency-based pruning, it accounts for both router gates and expert function size, and averages only over tokens where each expert is active.

4 ROUTER-WEIGHTED EXPERT ACTIVATION PRUNING (REAP)

  • In Qwen3-30B functional subspaces, pruning preserves manifold geometry while merging contracts it; the merging distortion is also reflected in higher Wasserstein distance.The late-layer contraction is more pronounced, reaching up to 100× reduction in spread for models with many experts.Source: Figure 1 · p. 6, Figure 2 · p. 6
  • Frequency-based pruning ignores router gates and expert norms, assumes active experts contribute equally, and therefore fails to minimize the derived error bound.The bound depends on both the router-expert coordination and the pruned expert’s functional properties.Source: §4 · pp. 5–6
  • REAP targets the pruned expert’s impact, g_j(x)∥f_j(x)∥, to minimize known terms in the substitution error bound while shrinking the scaling coefficient of the unknown promoted-expert term.The promoted expert varies across tokens, so the criterion focuses on the pruned expert’s router gate and activation norm.Source: §4 · pp. 5–6
  • REAP’s saliency score averages each expert’s weighted magnitude over tokens where it is active, separating functional impact from activation frequency.Pruning experts with the lowest score targets those making a weak contribution even when selected by the router.Source: §4 · p. 6

5 EXPERIMENTS

Experiments compare one-shot pruning and merging across SMoE sizes and discriminative and generative tasks. REAP is strongest on generative evaluations, with architecture- and task-specific exceptions and strong large-scale coding results.

5 EXPERIMENTS

  • All models are evaluated one-shot, with no additional fine-tuning after compression; models ≤110B use randomly selected samples packed to 2,048 tokens, while models ≥110B use 12,228 samples of at most 16,384 tokens without truncation or packing.Evaluation spans six SMoE architectures from 21B to 1T and includes MC question answering, code generation, mathematical reasoning, creative writing, and tool calling. C4 calibration caused accuracy collapse, with several compressed model instances at 0% accuracy.Source: §5 · p. 7, §5.1 · p. 10

5.1 RESULTS

  • For MC question answering, HC-SMoE and REAP have mean accuracy decreases of approximately 4% and 13% at 25% and 50% compression, respectively, excluding large-scale SMoEs.REAP ranks first or second across the reported methods, models, and compression ratios.Source: §5.1 · p. 7
  • At 50% compression, REAP’s mean decrease in coding accuracy is 6.9%, versus >20% for HC-SMoE and M-SMoE; REAP also outperforms other pruning methods.Figure 3 compares accuracy by task type for GLM-4.5-Air and Qwen3-30B and notes merging’s relative strength on MC versus its drop on generative tasks.Source: §5.1 · pp. 7–8, Figure 3 · p. 9
  • On creative writing, REAP and EAN are near-lossless at 25% compression, while REAP improves quality at 50%; REAP’s mean decrease on math reasoning is 0.1% at 25%.Merging results vary by architecture: M-SMoE is best on Qwen3-30B but worst on GLM-4.5-Air for creative writing at 50%.Source: §5.1 · p. 8
  • On large-scale models, REAP’s mean decrease in non-agentic coding accuracy is 0.16% at 25% compression and 1.2% at 50%, with high accuracy also reported on SWE-Bench.For tool use at 50% compression, REAP’s mean decrease is 5.9% versus 6.2% for EAN; frequency-based pruning degrades sharply at 50%.Source: §5.1 · p. 8, Table 3 · p. 8
  • At 50% compression, merged Qwen3-30B generations have lower n-gram diversity, higher and more variable cross-perplexity, and faster baseline-logit divergence than REAP generations.The analysis prompts compressed models with 100 randomly sampled evol-codealpaca questions; REAP remains similar to the base model in diversity, though slightly less diverse.Source: §5.1 · p. 9, Figure 4 · p. 9
  • In expert merging, within-layer clustered experts are farther apart in weight space than pretrained-to-instruct fine-tuned experts, and M-SMoE clusters remain poorly aligned after weight matching.Figure 4d reports mean relative L2-distance and singular-vector alignment; the authors identify poorly aligned experts as a challenge for coherent parameter merging.Source: §5.1 · pp. 9–10, Figure 4 · p. 9
  • HC-SMoE often leaves singleton clusters unmerged while grouping remaining experts into mega-clusters; limiting clusters to 32 causes large coding-quality decreases.Singleton clusters are analogous to experts retained by pruning, and mega-clusters are especially prevalent in high-granularity models.Source: §5.1 · p. 10

6 DISCUSSION

Expert pruning is more consistent and accurate than merging on generative tasks, while merging can perform reasonably on multiple-choice benchmarks. The authors caution that discriminative metrics are not necessarily reliable proxies for generative quality.

6 DISCUSSION

  • Merging is less consistent on generative tasks, particularly at 50% compression and in high-granularity architectures, and produces more repetitive outputs less aligned with the base model.The authors connect these observations to functional manifold distortion and loss of fine-grained router control.Source: §6 · p. 10
  • Pruned models offer consistently higher accuracy than merged models on generative tasks, while REAP generalizes across architectures, compression ratios, and generative tasks.The paper reports near-lossless compression on coding tasks with Qwen3-Coder-480B and Kimi-K2.Source: §6 · p. 10
  • Perplexity and log-likelihood-based multiple-choice metrics are not necessarily good proxies for generative model quality.The discussion stresses comprehensive downstream evaluation of LLMs.Source: §6 · p. 10

7 CONCLUSION

The conclusion attributes irreducible error in expert merging to lost router control and presents REAP as pruning experts that contribute least to layer output. For hierarchical merging, error increases with gate-value and mixture variance, making large or diverse clusters especially problematic.

7 CONCLUSION

  • REAP prunes experts that contribute least to layer output, minimizing the reconstruction error bound, and retains high accuracy across generative tasks and model architectures.The conclusion frames REAP as a novel expert-pruning method.Source: §7 · p. 10

A EXTENSION TO HIERARCHICAL CLUSTERING

  • After hierarchical merging, the router applies the summed gate to one static cluster representative, inducing irreducible error.The representative is typically a weighted average of cluster members based on calibration data.Source: A EXTENSION TO HIERARCHICAL CLUSTERING · p. 20
  • Hierarchical error grows with both a cluster’s total gate-value and the variance of the dynamic mixture approximated by its static representative.The tension makes large clusters and clusters of functionally diverse experts especially error-prone.Source: A EXTENSION TO HIERARCHICAL CLUSTERING · p. 20
  • Magnitude-based distances such as Euclidean distance can group experts with similar norms but different specializations, creating mega-clusters that force one control signal across formerly independent experts.The text links this to functional collapse in late layers, where Var[wj(x)] is highest.Source: A EXTENSION TO HIERARCHICAL CLUSTERING · p. 20

B.1 FUNCTIONAL SUBSPACE PCA ANALYSIS

The PCA analysis reports that pruning preserves more of the original expert-output manifold’s geometry and complexity than merging, whose outputs can contract or introduce novel functions. The Wasserstein distance quantifies differences between original and compressed manifolds.

B.1 FUNCTIONAL SUBSPACE PCA ANALYSIS

  • Merging contracts expert-output manifolds while pruning often preserves outlier experts and the original span; Qwen3 layer 0’s PC1 range contracts from −0.4–1.0 to approximately −0.2–0.3 under merging.With 64 pruned experts, the full original range is maintained from 128 experts.Source: §B.1 · p. 20
  • Merged outputs have consistently higher PC1+PC2 explained variance than baseline, while pruned outputs have lower explained variance, suggesting merging loses high-dimensional complexity and pruning preserves it.The passage reports this pattern across models, including high-granularity Qwen3 and ERNIE.Source: §B.1 · pp. 20–21, Table A4 · p. 22
  • Functional-subspace collapse under merging is more pronounced in high-granularity Qwen3-30B-A3B and ERNIE-4.5-21B than in low-granularity Mixtral-8x7B and Llama-4-Scout.The high-granularity models have 128 and 64 routed experts per layer, respectively, versus fewer experts and fewer experts per token in the low-granularity models.Source: §B.1 · p. 21

B.2 WASSERSTEIN DISTANCE

  • The 1-Wasserstein distance compares original and compressed expert-output manifolds, and higher merged-model distances indicate less similarity and loss of fidelity that cannot be restored in one-shot compression.The analysis also describes merging as introducing novel expert functions.Source: §B.1 · p. 21, §B.2 · pp. 21–22

C EVALUATION DETAILS

The evaluation details describe zero-shot multiple-choice testing and generative benchmarks spanning coding, agentic coding, tool use, math, and creative writing. Protocols vary by model size and task, with specific sampling and output-length settings.

C EVALUATION DETAILS

  • Multiple-choice evaluation uses zero-shot log-likelihood on ARC, BoolQ, HellaSwag, MMLU, OpenBookQA, RTE, and WinoGrande, reporting byte-length normalized accuracy for selected benchmarks.The reported normalized-accuracy subset is ARC-c, ARC-e, HellaSwag, and OBQA.Source: C EVALUATION DETAILS · p. 22
  • Coding evaluation covers EvalPlus and 182 LiveCodeBench questions; Qwen3-Coder-480B and Kimi-K2-Instruct-W4A16 also receive SWE-Bench and BFCLv3 evaluation.SWE-Bench uses the Verified test set and mini-SWE-agent scaffolding, with model-specific output-token limits and a 65,536-token input context.Source: C EVALUATION DETAILS · p. 22
  • Math reasoning is evaluated on GSM8K and MATH-500, while creative writing uses 146 WildBench prompts judged with gpt-4o-2024-05-13.Creative-writing scores use the WildBench rubric.Source: C EVALUATION DETAILS · p. 23
  • For models with at most 110B parameters, code and math use greedy sampling; larger models use task-specific settings, including default sampling for Qwen3-Coder-480B on LiveCodeBench.Output-token limits and other sampling settings vary across tasks and model sizes.Source: C EVALUATION DETAILS · p. 23

D BASELINE METHODS

The section defines saliency-based pruning and describes frequency and activation-norm criteria, then outlines the clustering, alignment, and parameter-averaging steps used by merging baselines.

D BASELINE METHODS

  • Pruning removes experts with the lowest saliency, while merging clusters experts by pairwise distance and merges parameters within each cluster.The notation defines calibration data, router gates, expert usage frequency, and the SMoE layer setup.Source: D BASELINE METHODS · p. 23
  • Frequency-based pruning scores each expert by its token usage frequency across the calibration dataset.EAN pruning instead accumulates each expert’s activation norm over tokens where it is activated.Source: D BASELINE METHODS · p. 23
  • M-SMoE aligns experts with a permutation matrix before clustering them using router gate-value cosine distance.Its expert-output invariance to the intermediate-dimension permutation is described in the method.Source: D BASELINE METHODS · p. 24
  • The HC-SMoE and M-SMoE baselines merge each resulting cluster using frequency-weighted averages of permuted expert parameters.The clusters are merged with Equation (15).Source: D BASELINE METHODS · p. 24
  • HC-SMoE represents experts by average calibration-token activations and clusters them by cosine distance using hierarchical agglomerative clustering with average linkage.It initializes singleton clusters and repeatedly joins the closest pair.Source: D BASELINE METHODS · p. 24

E QUANTIZATION AND EXPERT PRUNING

The section examines combining expert pruning with quantization: although quantization alone is stronger through 4-bit weights, combining methods reaches smaller model sizes with better accuracy than more aggressive quantization.

E QUANTIZATION AND EXPERT PRUNING

  • 87.5% total size reduction combines 4-bit weights with 50% expert pruning on Kimi-K2.Pruning can be combined with quantization without the parameter-group scale reconciliation required for merging.Source: E QUANTIZATION AND EXPERT PRUNING · p. 24
  • Quantizing Qwen3-30B-A3B to 4 bits with AWQ outperforms a 50%-expert-pruned REAP model at 16 bits, despite half the checkpoint size.The comparison is reported as an example of quantization’s accuracy at a given compression ratio down to 4 bits.Source: E QUANTIZATION AND EXPERT PRUNING · p. 24
  • A footprint matching the Kimi-K2 combination through quantization alone would require 2-bit weights, which are not natively supported on most current accelerators and typically degrade accuracy significantly.The Qwen3-30B-A3B preliminary analysis suggests REAP combined with 4-bit weights outperforms more aggressive quantization; broader low-bit and pruning combinations remain future work.Source: E QUANTIZATION AND EXPERT PRUNING · p. 25
  • Beyond 4-bit weights, combining quantization with expert pruning yields significantly better EvalPlus accuracy than more aggressive quantization on Qwen3-30B-A3B.Table A5 compares EvalPlus accuracy against relative checkpoint size for AWQ, REAP, and their combination.Source: Table A5 · p. 25

F ADDITIONAL RESULTS

Additional evaluations cover multiple-choice, coding, agentic, and long-context tasks, as well as calibration-data and cluster-size effects. They report near-lossless REAP performance for large SMoEs at 50% compression and show that calibration domain materially affects compressed-model quality.

F ADDITIONAL RESULTS

  • With C4 calibration, multiple-choice performance is generally slightly higher than with evol-codealpaca, but code generation quality is abysmal and most models fail to produce coherent output.Table A8 reports C4-calibrated coding and multiple-choice results.Source: F ADDITIONAL RESULTS · p. 25, Table A8 · p. 28
  • REAP evaluations include τ2-bench across Retail, Airline, and Telecom, plus BFCLv3 for additional non-reasoning and reasoning models.The reported calibration mix for non-reasoning models includes 12,228 samples; reasoning models receive an additional 12,228 Open-R1 samples.Source: F ADDITIONAL RESULTS · p. 25, Table A10 · p. 28, Table A9 · p. 28
  • The Kimi-Linear-48B-A3B-Instruct evaluation applies 30% REAP compression to coding, math, and long-context tasks; performance is retained beyond the 16,384-token maximum calibration length.The long-context benchmarks are LongBench v2 and FRAMES, with overlength prompts truncated in the middle and FRAMES accuracy estimated using an LLM judge.Source: F ADDITIONAL RESULTS · pp. 25–26, Table A12 · p. 29
  • At 50% compression, REAP is near-lossless on large-scale SMoEs, while frequency-based pruning’s shortcomings become apparent.Figure A6 plots non-agentic coding and multiple-choice accuracy against compressed model size.Source: F ADDITIONAL RESULTS · p. 25, Figure A6 · p. 29
  • Merging baselines often form many singleton clusters and compensate with oversized clusters; restricting maximum cluster size to four preserves multiple-choice accuracy but severely diminishes generative coding capability.Figure A7 compares singleton-cluster proportions and Qwen3-30B accuracy after M-SMoE compresses 50% of experts.Source: Figure A7 · p. 29
  • Domain-specific calibration substantially improves compressed-model quality in the target domain, while REAP best preserves accuracy across tasks when calibrated on a larger general dataset.The general calibration mixture contains three times as many samples as the domain-specific datasets; fine-grained models show greater degradation, with some methods producing no coherent output on C4.Source: Figure A8 · p. 30, Figure A9 · p. 30

Related papers

Loading 2510.13999v3…