PaperlayerView exact revision on arXiv

Source-linked AI summary

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu, Bei Liu, Sirui Han, Yike Guo

arXiv:2609.31631v2cs.LG

TL;DR

MoE models are difficult to deploy at scale, and existing pruning can require costly search or overlook expert interdependencies. OMP-MoE combines greedy sparse-reconstruction pruning, cross-layer allocation, and energy-based adaptive inference. Across tested models, it reports strong performance retention alongside faster search and inference.

  • Problem

    MoE deployment is challenged by massive model memory requirements, while pruning methods can incur prohibitive search costs or neglect expert interdependencies.

    Source: §1 · p. 2

  • Method

    OMP-MoE uses OMP to select expert contribution atoms, water-filling to allocate experts across layers, and energy prediction to adapt inference.

    Source: “The shortest distance between two… · p. 2, §3.4 · p. 7

  • Results

    At 50% pruning, OMP-MoE retains 93.3% of original Qwen3-30B-A3B accuracy and consistently outperforms state-of-the-art merging and pruning baselines across the evaluated MoE models.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    OMP-MoE is reported to be orthogonally compatible with quantization and unstructured pruning.

    Source: “The shortest distance between two… · p. 3

    OMP-MoE relies on calibration dataset selection, and the authors identify advanced calibration procedures as future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE{\dag}, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33$\times$ faster search and 1.55$\times$ inference speedup. Codes will be available after acceptance.

1 Introduction

MoE deployment is constrained by model scale, while existing pruning can demand prohibitive search or training costs. OMP-MoE uses greedy expert selection, cross-layer allocation, and adaptive inference; experiments report strong performance retention and efficiency gains.

1 Introduction

  • Models exceeding hundreds of billions of parameters pose deployment challenges in resource-limited environments, while expert pruning methods face an efficiency-performance trade-off.Combinatorial selection can involve ≈2.4 × 10^37 combinations for 128 experts and a target of 64; Hessian-based pruning also incurs high memory and compute costs, while differentiable methods require fine-tuning.Source: §1 · p. 2

“The shortest distance between two points is a straight line.”

  • Residual curves decay at different rates across layers, and expert importance changes as each selection updates the residual, making uniform budgets and static rankings suboptimal.Source: “The shortest distance between two… · p. 2
  • OMP-MoE treats expert contributions as dictionary atoms and greedily selects experts to reduce reconstruction residuals, avoiding exhaustive subset search.The framework also uses water-filling to allocate layer budgets by reconstruction error and routing stability, and OMP-MoE† skips low-energy expert executions at inference.Source: “The shortest distance between two… · p. 2
  • At 50% pruning, OMP-MoE retains 93.3% of Qwen3-30B-A3B’s original accuracy, versus 53.0% for MC-SMoE and 80.5% for NAEE.Search takes 641 seconds, a 33× speedup over NAEE; OMP-MoE† achieves 1.46× end-to-end acceleration on DeepSeek-V2-Lite at 50% pruning.Source: “The shortest distance between two… · p. 2
  • OMP-MoE is reported to be orthogonally compatible with quantization and unstructured pruning.Source: “The shortest distance between two… · p. 3

2 Related Work

Prior MoE compression methods merge experts or prune them, with trade-offs involving specialization, optimization cost, or training overhead. OMP-MoE applies iterative OMP selection to experts with linear computational complexity.

Mixture-of-Experts Compression

  • OMP-MoE uses a training-free iterative mechanism with linear computational complexity to select experts, addressing prior methods’ optimization costs and neglect of inter-expert correlations.Source: §2 · p. 3
  • Merging methods reduce footprints but can blend distinct expert functions, while pruning methods include costly combinatorial search, fine-tuning, Shapley estimation, or Hessian-based overhead.The paper applies Orthogonal Matching Pursuit, a greedy sparse-reconstruction algorithm, to expert selection by treating individual experts as dictionary atoms.Source: §2 · p. 3

3 Methodology

OMP-MoE frames expert pruning as sparse reconstruction, then combines iterative expert selection with layer-wise budget allocation and input-adaptive inference. Its allocation accounts for reconstruction error and routing stability, while its inference rule skips low-energy experts.

3.1 Preliminary

  • OMP models sparse reconstruction by iteratively selecting dictionary atoms most correlated with the residual and updating coefficients through orthogonal projection.OMP-MoE treats expert contribution atoms as a dictionary and constrains retention coefficients to binary values, turning expert selection into reconstruction from a selected subset.Source: §3.1 · p. 4

3.2 Iterative Expert Selection via OMP

  • Each OMP selection maximizes the reconstruction gain among remaining experts, which is locally optimal for minimizing the next residual norm under binary expert retention.A one-time statistics pass caches layer outputs and expert contributions; the greedy search has complexity O(L · n · Ne · Bd).Source: §3.2 · pp. 4–5

3.3 Cross-Layer Budget Allocation with Routing Stability

  • Water-filling allocates each remaining expert to the layer with the largest immediate decrease in combined reconstruction error and routing risk.The allocation considers layer-specific reconstruction difficulty and preserved routing mass, accounting for routing renormalization when experts are removed.Source: §3.3 · pp. 5–6

3.4 OMP-MoE† : Adaptive Inference via Energy Prediction

  • OMP-MoE† predicts token-level expert contribution energy and executes the minimum number of experts needed to meet a relative residual threshold.The mechanism skips the remaining experts for that token, reducing computational overhead while preserving reconstructed-signal fidelity.Source: §3.4 · pp. 6–7

4 Experiments

Evaluations span four MoE architectures and zero-shot, reasoning, long-context, and compression settings, using 64 WikiText-2 calibration samples. Results cover expert pruning, search and inference efficiency, compatibility with other compression methods, and ablations of allocation and calibration choices.

4.2 Main Results

  • At 50% pruning, OMP-MoE retains 93.3% of Qwen3-30B-A3B’s original 0.640 average-accuracy baseline, with 0.597 average accuracy and superior accuracy to state-of-the-art baselines.The results also report resilience across other MoE models.Source: §4.2 · p. 8
  • On Qwen3-30B-A3B, search takes 641 seconds, 33× faster than NAEE and 14× faster than DiEP, while using less peak memory than both; OMP-MoE† reaches 1.46× inference speedup on DeepSeek-V2-Lite at 50% pruning.The inference speedup comes from skipping low-energy expert executions.Source: §4.2 · p. 8
  • At 50% pruning, Qwen3-30B-A3B scores 0.531 on GSM8K, 0.523 on LongBench, and 0.323 on BBH; its GSM8K score rises from NAEE’s 0.236 to 0.531.Table 4 covers mathematical reasoning, long-context modeling, and compositional reasoning on Qwen3-30B-A3B at this pruning ratio.Source: §4.2 · p. 8, Table 4 · p. 8
  • With 25% expert pruning on DeepSeek-V2-Lite, combining OMP-MoE with GPTQ, SparseGPT, or Wanda yields average accuracies of 0.541, 0.525, and 0.534, respectively.These results indicate compatibility with quantization and unstructured pruning.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 Ablation Studies

  • At 50% pruning, cross-layer allocation reaches 0.643 on Qwen3-30B-A3B, versus 0.511 for DiEP and 0.480 for HEAPr; it also improves over uniform allocation on Qwen3 and DeepSeek-V2-Lite.Against uniform allocation, scores rise from 0.618 to 0.643 on Qwen3 and from 0.468 to 0.489 on DeepSeek-V2-Lite.Source: §4.3 · pp. 8–9, Table 6 · p. 8
  • At 50% pruning, adding the routing-risk penalty raises Qwen3-30B-A3B accuracy from 0.618 with reconstruction error alone to 0.643.Accuracy improves monotonically as λ increases, suggesting reconstruction fidelity and routing stability provide complementary allocation signals.Source: §4.3 · p. 9, Table 7 · p. 9
  • On DeepSeek-V2-Lite, ϵ = 0.1 gives 0.581 accuracy while skipping about 35% of expert executions; larger thresholds reduce accuracy.The threshold controls residual expert energy skipped at inference.Source: §4.3 · p. 9, Figure 3 · p. 9
  • On 50%-pruned Qwen3-30B-A3B, WikiText-2 peaks at 0.643 with 64 calibration samples, while C4 reaches 0.630 with 128; additional samples provide little benefit.Calibration domain can affect attainable accuracy.Source: §4.3 · p. 9, Figure 3 · p. 9

5 Conclusion

OMP-MoE combines greedy expert pruning, cross-layer budget allocation, and adaptive inference in a training-free framework. Its theoretical analysis covers binary expert selection, the difference from standard OMP projection, and conditions for optimal greedy allocation.

5 Conclusion

  • OMP-MoE uses training-free sparse reconstruction, redundancy-aware cross-layer budgets, and adaptive inference to reduce search and inference costs.The conclusion presents it as effective across architectures and practical for deploying large-scale MoE LLMs.Source: §5 · p. 9

A Formal Theoretical Analysis of OMP-MoE

  • The appendix frames expert selection in Hilbert space and proves the greedy criterion’s optimality under the linear decomposition property.It provides a formal mathematical treatment of the selection process.Source: A Formal Theoretical Analysis of OMP-MoE · p. 14

A.1 Mathematical Framework

  • The framework treats expert retention as a Boolean subset-selection problem because an expert must be either executed or pruned.The analysis formulates the objective over binary coefficients and support-set indices.Source: §A.1 · p. 14

A.3 Comparison with Standard OMP Projection

  • Under a binary retention constraint, direct residual subtraction replaces orthogonalization, avoiding matrix inversion and keeping expert contributions invariant to future selections.The stated implications are lower computational cost and preservation of the router-based interpretation of expert contributions.Source: §A.3 · p. 15

A.4 Budget Allocation via Marginal Gain

  • Greedy cross-layer allocation is globally optimal when each layer’s marginal gains are non-increasing as more experts are allocated.The proof relates this condition to diminishing marginal returns and discrete convexity.Source: §A.4 · p. 15

A.2 Derivation of the Greedy Selection Rule

The derivation establishes a locally optimal greedy rule for binary expert selection and relates reconstruction residuals to downstream loss perturbation. The appendices also describe the pruning and adaptive-inference procedures and their computational scaling.

A.2 Derivation of the Greedy Selection Rule

  • Greedy selection by reconstruction gain is locally optimal under the binary constraint because choosing an expert minimizes the next residual norm.With selection fixed at coefficient 1, adding an expert directly subtracts its contribution from the current residual.Source: §A.2 · p. 14

A.5 Reconstruction Error and Loss Perturbation

  • Reducing layer-wise reconstruction residual reduces a local upper bound on downstream loss perturbation.For OMP-MoE†, predicted expert energy is used to skip low-energy directions while keeping the induced perturbation controlled under the stated assumptions.Source: §A.5 · pp. 15–16

B Extended Algorithmic Procedures and Computational Analysis

  • The computational analysis separates offline statistics acquisition from online adaptive inference to account for the framework’s efficiency gains.The appendix presents this as the transition between its two temporal phases.Source: B Extended Algorithmic Procedures and… · p. 16

B.1 Comprehensive Pruning and Selection Pipeline

  • The pruning pipeline collects layer signals and expert contribution atoms, performs layer-wise greedy selection, then allocates quotas with marginal-gain-guided water-filling.The selected indices are the highest-ranked experts up to each layer’s quota.Source: §A.5 · p. 16, §B.1 · p. 17

B.2 Adaptive Online Inference via OMP-MoE†

  • OMP-MoE† uses precomputed energy coefficients to predict reconstruction sufficiency and dynamically adjust the number of experts activated for each token.The mechanism makes its activation decision before full execution.Source: §B.2 · p. 17

B.3 Computational Complexity and Asymptotic Bounds

  • Search uses one calibration forward pass followed by cached-atom operations, scaling polynomially rather than with NAEE’s combinatorial subset enumeration.Statistics acquisition costs O(L · B · K · dffn) in computation and O(L · Ne · B · d) in cache memory; the acquisition cost is incurred once per architecture.Source: §B.3 · p. 17

B.4 Theoretical FLOPs Reduction in OMP-MoE†

OMP-MoE† estimates how many routed experts are needed to meet an energy threshold, reducing routed-expert FLOPs in the reported analysis. The implementation section also provides configuration details for reproducibility.

B.4 Theoretical FLOPs Reduction in OMP-MoE†

  • 35.0% to 39.0%: OMP-MoE† predicts routed-expert FLOPs reductions across three architectures at ϵ = 0.1.The normalized baseline cost for each architecture is 1.000.Source: §B.4 · p. 18, Table 9 · p. 18

C Implementation and Configuration

  • The paper provides implementation details, model specifications, and hyperparameter configurations to support reproducibility.The supplied passage describes the scope of the implementation section.Source: C Implementation and Configuration · p. 18

C.1 Model Architecture Specifications

The appendices document the evaluated architectures, calibration and benchmark protocols, router treatment, baseline configurations, and experimental environment. They also map the scope of the extended experimental results.

C.1 Model Architecture Specifications

  • The evaluated models span expert dictionaries of 8 to 128 experts and native activation counts of 2 to 8; DeepSeek-V2-Lite’s shared experts remain unchanged during pruning.The architecture and routing tables describe the model specifications and gating configurations.Source: §C.1 · pp. 18–19, Table 10 · p. 19, Table 11 · p. 19

C.2 Dataset Processing and Calibration

  • C4 and WikiText-2 calibration uses raw text split into 64 non-overlapping 4,096-token blocks, totaling 262,144 tokens per corpus.The same preprocessing protocol is used for both corpora.Source: §C.2 · p. 19, Table 12 · p. 20

C.3 Benchmark Evaluation Details

  • Evaluation covers twelve benchmarks, including eight general benchmarks and specialized reasoning, long-context, and code-generation tasks.Reported metrics include zero-shot accuracy, exact match, LongBench comprehensive score, and HumanEval pass@1.Source: §C.3 · p. 20, Table 13 · p. 20

C.4 Standardization of Router Renormalization

  • Pruning preserves each checkpoint’s native router transformation, top-k selection, normalization, and any architecture-specific bias correction.The same architecture-native treatment is applied to pruning and merging baselines.Source: §C.4 · p. 20

C.5 Hyperparameter Display

  • The appendix lists hyperparameters for pruning and adaptive inference and configurations for search and statistics collection.The corresponding settings are provided in Tables 14 and 15.Source: §C.5 · p. 20, Table 14 · p. 21, Table 15 · p. 21

C.6 Baseline Configuration Details

  • NAEE search is capped at 200 iterations per layer except on Mixtral-8×7B, where the full search space is tractable.Additional REAP and EASY-EP comparisons use the same Qwen3 checkpoint, pruning ratios, evaluation harness, router treatment, and calibration sequences as OMP-MoE.Source: §C.6 · p. 21

C.7 Hardware and Software Environment

  • The appendix reports the hardware and software stack used for pruning-time and inference-time measurements.It notes that accelerator count, memory capacity, and library versions contextualize absolute search and latency values.Source: §C.7 · pp. 21–22, Table 16 · p. 21

D Extended Experimental Results

  • Extended experiments cover task-level comparisons, compatibility and sensitivity studies, inference measurements, stronger compression, scaling, and specialized generative evaluations.Each subsection states its evaluation question and discusses the corresponding tables.Source: D Extended Experimental Results · p. 22

D.1 Detailed Results for Zero-shot Tasks

Across the evaluated architectures and pruning ratios, OMP-MoE leads the compared methods in average accuracy. Additional tests examine one-shot baselines, compatibility with other compression methods, calibration sensitivity, and adaptive inference.

D.1 Detailed Results for Zero-shot Tasks

  • OMP-MoE has the highest average accuracy among compared methods on every evaluated architecture at both pruning ratios.The zero-shot evaluation covers eight tasks across Qwen3, DeepSeek-V2-Lite, GPT-OSS-20B, and Mixtral.Source: §D.1 · p. 22, Table 17 · p. 22, Table 18 · p. 23

D.2 Comparison with One-Shot Expert Pruning Baselines

  • At 50% pruning, OMP-MoE exceeds REAP by 0.052 and EASY-EP by 0.016 in average accuracy, and leads on seven of eight tasks.The comparison uses the same checkpoint, pruning ratio, router treatment, evaluation harness, and 64-sequence WikiText-2 calibration protocol.Source: §D.2 · p. 23, Table 19 · p. 23

D.3 Compatibility with Orthogonal Compression Methods

  • OMP-MoE remains functional when combined with GPTQ, SparseGPT, or Wanda, supporting multi-level compression without catastrophic performance degradation.The compatibility evaluation is on DeepSeek-V2-Lite at a 25% pruning ratio.Source: §D.3 · pp. 23–24, Table 20 · p. 24

D.4 Sensitivity Studies on Calibration Set Size

  • On Qwen3-30B-A3B at 50% pruning, accuracy remains stable from 16 to 256 calibration samples, with 64 samples sufficient for near-optimal selection across C4 and WikiText-2.The sensitivity study varies both calibration sample size and dataset source.Source: §D.4 · p. 24, Table 21 · p. 24

D.5 Ablation of Adaptive Inference (OMP-MoE† )

  • For DeepSeek-V2-Lite, a threshold ϵ of 0.1 balances FLOPs and accuracy under adaptive inference.The ablation reports expert activation counts, skip ratios, perplexity, and eight-benchmark average accuracy across thresholds.Source: §D.5 · p. 24, Table 22 · p. 24

D.6 Empirical Inference Efficiency across Diverse Architectures

Adaptive OMP-MoE† execution adds latency reductions after static expert pruning across four MoE architectures, with an architecture-dependent accuracy-efficiency trade-off.

D.6 Empirical Inference Efficiency across Diverse Architectures

  • At 50% pruning, combined speedups range from 1.28× on Mixtral to 2.00× on GPT-OSS-20B; accuracy changes are small on GPT-OSS-20B, Mixtral, and Qwen3, but DeepSeek-V2-Lite drops from 0.489 to 0.476.The comparison evaluates static pruning against adaptive execution under global budget allocation across four architectures.Source: Table 23 · p. 25, Table 24 · p. 25, Table 25 · p. 25

D.7 Robustness and Scaling Results

The robustness and analysis results span stronger pruning, larger models, domain-specific calibration, reconstruction fidelity, and expert-selection visualizations.

D.7 Robustness and Scaling Results

  • With 40% and 25% of experts retained, OMP-MoE reaches average accuracies of 0.546 and 0.429, exceeding DiEP by 0.064 and 0.048 and NAEE by 0.115 and 0.102, respectively.Performance declines when three quarters of experts are removed, while the relative advantage of residual-conditioned selection remains.Source: §D.7 · pp. 25–26, Table 27 · p. 26
  • On Qwen3-235B-A22B at 50% pruning, OMP-MoE reaches 0.685 average accuracy versus 0.660 for DiEP and 0.640 for NAEE, with an 846-second search.The search is more than 12× faster than DiEP and 24× faster than NAEE.Source: §D.7 · p. 26, Table 28 · p. 26
  • The 50%-pruned Qwen3-30B-A3B reaches 0.643 average accuracy, exceeding dense Qwen3-4B and Qwen3-1.7B by 0.036 and 0.082, respectively.The comparison covers eight tasks.Source: §D.7 · p. 26, Table 29 · p. 26

D.8 Domain-Specific Calibration on Generative Tasks

  • With mathematical and code calibration, OMP-MoE outperforms REAP and EASY-EP at both pruning ratios on GSM8K and HumanEval, with results varying by calibration distribution.GSM8K exact match is 0.897 and 0.898 at 25% and 50% pruning versus 0.891 for the full model; HumanEval pass@1 is 0.939 and 0.896 versus 0.933 for the full model.Source: §D.8 · p. 26, Table 30 · p. 27

D.9 Reconstruction Fidelity and Downstream Accuracy

  • Across five pruning ratios, higher reconstruction fidelity is associated with higher downstream average accuracy, though local deviations can occur in low-sensitivity directions.The study defines total reconstruction error as the sum of normalized residual rates across MoE layers.Source: §D.9 · p. 27, Table 31 · p. 27

E.1 Reconstruction Gain Heatmaps

E.2 Normalized Residual Curves

Residual and risk curves characterize layer-specific reconstruction and routing trade-offs, while the discussion scopes the greedy guarantee and contrasts dynamic selection with static heuristics.

E.2 Normalized Residual Curves

  • Residual curves typically decay monotonically as experts are selected; rapid descent indicates concentrated signal and supports smaller budgets, while slower decay indicates a wider expert contribution distribution.Shallow layers often show high signal concentration, whereas deeper layers can require larger budgets to meet the same reconstruction threshold.Source: §E.2 · p. 31, Figure 6 · p. 32, Figure 7 · p. 32
  • Residual and routing-risk trajectories expose a fidelity-stability trade-off, and their layer-specific gain-to-risk ratios motivate discrete water-filling allocation.Decay rates vary across layers and architectures, supplying marginal-gain curves rather than assuming uniform redundancy.Source: §E.2 · p. 31

E.3 Identification of High-Contribution Experts

  • A high-contribution expert is a calibration-conditioned first-step OMP selection that reconstructs more than 90% of a layer’s routed-output energy, distinct from Su et al.’s [42] activation-outlier Super Experts.Five representative cases have first-step residuals from 0.0001 to 0.0407, each corresponding to more than 95% reconstruction of layer calibration output.Source: §E.3 · p. 33, Table 32 · p. 33

F.1 Scope of Greedy Optimality

  • With binary coefficients fixed at one, OMP-MoE’s optimality guarantee covers only the immediate residual reduction conditioned on the current subset, not a global optimum.This formulation avoids matrix-inversion overhead while accounting for inter-expert dependencies.Source: §F.1 · p. 33

F.2 Superiority over Static Heuristics

  • Residual updates re-evaluate candidate experts against uncaptured information, addressing the dynamic interactions that one-shot routing-frequency or norm rankings neglect.The paper states that this can identify complementary experts that static magnitude-based methods would discard.Source: §F.2 · p. 33

F.3 Broader Impact

  • The authors suggest that reducing MoE memory and inference costs may lower hardware barriers and energy use, broadening access for users with limited compute.Source: §F.3 · p. 34

Related papers

Loading 2609.31631v2…