PaperlayerView exact revision on arXiv

Source-linked AI summary

PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression

Shanfeng Huang, Rongqian Zhang, Zhou Fang, Song Xiao, Hai Du

arXiv:2610.13982v1cs.LGstat.ML

TL;DR

MoE models face deployment memory constraints, while existing compression approaches often separate pruning and merging and neglect expert-frequency effects. PRIME jointly optimizes prune-merge assignments with frequency-weighted costs. Under compression, it retains stronger benchmark results than the cited baselines in reported comparisons, and its merged-model serving benchmarks show benefits under memory-constrained conditions.

  • Problem

    MoE expert memory footprints hinder deployment, while common compression approaches separate pruning and merging and ignore frequency-dependent distortion costs.

    Source: §1 · p. 1

  • Method

    PRIME jointly selects pruning and merging assignments with an ILP that minimizes a frequency-aware surrogate for compression-induced error.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    At K = 72, PRIME maintains 60.42 MC average, 72.13 coding average, 39.13 LongBench, and 87.49 GSM8K, while REAP collapses across dimensions and HC-SMoE reaches 54.74 on GSM8K.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    PRIME provides a unified frequency-aware approach that coordinates prune-merge decisions and helps maintain routing stability under parameter reduction.

    Source: §5 · p. 8

    PRIME optimizes layers independently, and its English-centric benchmark evaluation may not capture multilingual, safety-critical, or interactive settings, so specialized deployments require re-evaluation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and throughput benefits of expert compression in vLLM.

1. Introduction

MoE models use sparse activation to scale capacity, but their full expert sets create memory barriers to deployment. PRIME jointly optimizes pruning and merging with frequency-aware costs, and its serving benchmarks report memory and throughput benefits.

1. Introduction

  • MoE sparse activation decouples model capacity from inference cost, but the full expert set's memory footprint hinders deployment on commodity hardware and edge devices.MoEs activate only a sparse subset of experts per token while achieving performance comparable to dense models via conditional computation.Source: §1 · p. 1
  • Existing compression methods often separate pruning and merging and ignore expert frequencies, although distortions to high-frequency experts accumulate over tokens and cause larger inference errors.This leaves out a unified joint decision space and treats experts equally during similarity-based clustering.Source: §1 · p. 1
  • PRIME formulates pruning and merging jointly as an ILP over a frequency-weighted assignment objective, optimizing the stated surrogate rather than relying on greedy local decisions.The paper also motivates frequency weighting with error-bound analysis and derives a sufficient condition favoring high-frequency experts as centroids.Source: §1 · p. 1
  • A 25% expert reduction lowers weight memory by 23.7%, expands KV cache capacity by 2.20×, and improves output throughput by 1.74× in a representative memory-constrained workload.The PRIME overview depicts profiling, unified ILP decisions, and frequency-weighted aggregation.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Works

Prior MoE compression work includes pruning based on expert importance and merging experts into representatives. PRIME addresses the limits of binary pruning, greedy merging, and frequency-agnostic clustering by jointly selecting prune-merge assignments.

Expert Pruning

  • Pruning remains a binary retain-or-discard operation, limiting optimization and preventing consolidation of information from useful but low-frequency experts.Earlier methods use static metrics or learned pruning decisions; the cited examples include MoE-Pruner, DiEP, and REAP.Source: §2 · p. 2

Expert Merging

  • HC-SMoE uses hierarchical clustering to merge functionally similar experts, but its greedy stepwise decisions may miss lower-cost assignments under a joint objective.Earlier merging work includes EEP and MC-SMoE, which consolidate experts without retraining.Source: §2 · p. 2

The Gap

  • Current paradigms often separate pruning from merging, while geometric clustering ignores activation frequencies and can disproportionately inflate aggregate inference error.The gap motivates a unified, frequency-aware framework for prune-merge assignments.Source: §2 · pp. 2–3

3. Methodology

PRIME jointly assigns MoE experts to be retained, merged, or pruned through a frequency-aware optimization framework. The method uses frequency-weighted assignment and aggregation, then reconstructs the router to approximate grouped routing.

3. Methodology

  • PRIME formulates expert pruning and merging as a frequency-aware ILP that minimizes a surrogate for compression-induced error.Expert distances use an output-based metric, and the surrogate optimum is intended to preserve high-traffic routing paths.Source: §3 · p. 3

3.1. Problem Definition

  • Each expert is assigned as a centroid, merged into a centroid, or pruned as an isolated outlier or when merging cost exceeds the pruning penalty.The target is a smaller expert set without retraining.Source: §3.1 · p. 3

3.2. Frequency-Aware Surrogate Optimization

  • The ILP weights assignment costs by activation frequency, encouraging high-frequency experts to anchor clusters and promoting routing stability.Its distance is an empirical output-space proxy, motivated by but not identical to the parameter-space metric; the solver is optimal for the surrogate, not necessarily for downstream loss.Source: §3.2 · p. 3

3.3. Experts Aggregation

  • After assignment, each centroid aggregates its cluster through frequency-weighted averaging, so high-frequency experts dominate the resulting parameters.This aggregation is a post-assignment construction step, not part of the ILP objective.Source: §3.3 · p. 4

3.4. Router Reconstruction

  • Router reconstruction removes pruned experts and fits a compressed router to grouped-logit targets, approximating grouped probabilities before top-k selection.Merging requires summing member probabilities, which averaging router rows generally cannot reproduce; the reconstruction does not guarantee identical sparse routing or original MoE outputs.Source: §3.4 · p. 4

4. Experiments

Experiments evaluate PRIME against pruning and merging baselines on Qwen3-30B-A3B and Qwen3.5-122B-A10B across knowledge, coding, math, and long-context benchmarks. They also examine ablations, compression mechanisms, scalability, and vLLM serving under stated hardware and workload conditions.

4.2. Main Results and Performance Comparisons

  • At K = 72, PRIME retains 60.42 MC average, 72.13 coding average, 39.13 LongBench, and 87.49 GSM8K, while both baselines collapse in at least one dimension.Across compression levels, PRIME achieves the best aggregate performance, although individual metrics sometimes favor single-paradigm baselines.Source: §4.2 · p. 5, Table 1 · p. 6

4.3. Ablation Study

  • At K = 72, Prune-Only collapses on coding and long-context, Merge-Only degrades on multiple dimensions, and removing frequency weighting lowers PRIME’s MC and coding averages.The ablations support the complementarity of adaptive prune-merge decisions and frequency weighting, with frequency-awareness effects increasing under stricter compression.Source: §4.3 · p. 5, Table 2 · p. 6

4.4. Mechanistic Analysis

  • PRIME reduces layer-wise WIV by 67.21% on average over HC-SMoE, using frequency-weighted deviation from cluster centroids as a proxy for consolidation-induced distortion.The analysis attributes the gap to uniform geometric objectives overlooking the disproportionate effect of altering high-frequency experts.Source: Figure 2 · p. 6, §4.4 · p. 6
  • The t-SNE comparison shows PRIME maintaining a spatially balanced expert manifold, while HC-SMoE forms dense aggregations of heterogeneous experts; PRIME prunes geometrically isolated outliers.The visualization uses Qwen1.5-MoE for clarity, with Qwen3 results in Appendix E.Source: Figure 3 · p. 6, §4.4 · p. 7
  • Centroid selection acts as a frequency-dependent soft gate: experts above activation frequency 10^-2 are prioritized, and tighter compression shifts selection toward higher-frequency experts.The centroid budget is reallocated to the highest-frequency nodes as the target tightens from K = 96 to K = 72.Source: Figure 4 · p. 7, §4.4 · p. 7
  • At K = 72, PRIME’s routing heatmap closely aligns with the original model, whereas HC-SMoE shows traffic concentrated on a limited subset of centroids.The paper interprets this contrast as preserved routing stability and distributed expert utilization for PRIME, avoiding the mode collapse observed in HC-SMoE.Source: Figure 5 · p. 7, §4.4 · p. 7

4.5. Scalability Analysis

  • On the 256-expert Qwen3.5-122B-A10B, PRIME leads on MC and coding at K = 192, and on MC, coding, and LongBench at K = 160 and K = 144.At K = 144, it records 67.05 MC average, 83.00 coding average, and 46.63 LongBench; profiling overhead is 15.5 minutes versus 3.5 minutes for the 128-expert setting.Source: §4.5 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

4.6. Deployment Metrics on vLLM

  • Merging from 128 to 96 experts frees 13.50 GiB for vLLM’s KV cache, raises capacity from 121,968 to 268,400 tokens, and improves output throughput by 1.74×.The gains apply to the merged checkpoints under the stated hardware and workload conditions; single-request and prefill-dominated workloads show little or no speedup, and task accuracy was not evaluated in these runs.Source: Table 5 · p. 8, §4.6 · p. 8

5. Conclusion

PRIME unifies expert pruning and merging as a frequency-aware surrogate optimization over assignments. The framework aims to preserve model capacity under parameter reduction while reducing compression-induced drift and helping maintain routing stability.

5. Conclusion

  • PRIME jointly formulates expert pruning and merging as a frequency-aware surrogate optimization over assignments.This departs from prior disjoint pipelines.Source: §5 · p. 8
  • Coordinated pruning and merging decisions better preserve model capacity under parameter reduction.Source: §5 · p. 8
  • Incorporating expert usage frequency reduces compression-induced drift and helps maintain routing stability.Source: §5 · p. 8

Limitations

The authors identify limitations in PRIME’s layerwise optimization, calibration mixture, evaluation scope, and one-shot post-training design. They recommend re-evaluation before deployment in specialized applications.

Limitations

  • PRIME optimizes each layer independently and does not model cross-layer dependencies in expert utilization.The authors note that a globally coordinated multi-layer formulation could potentially yield better compression decisions.Source: Limitations · p. 9
  • The best calibration-data mixture may depend on the target deployment domain, and systematic study of mixing ratios remains future work.The experiments use a mixed C4, Evol-CodeAlpaca, and NuminaMath corpus.Source: Limitations · p. 9
  • Evaluation on standard English-centric benchmarks may not capture performance variation in multilingual, safety-critical, or interactive settings.The authors advise re-evaluating compressed models for specialized applications.Source: Limitations · p. 9
  • As a one-shot post-training method without fine-tuning recovery, PRIME may achieve lower compression ratios than methods that allow retraining.Source: Limitations · p. 9

Ethical Considerations

The work concerns structural compression of pretrained MoE models and uses publicly available calibration and evaluation datasets. The authors note that compressed models inherit source-model biases and limitations and call for standard safety evaluations before deployment.

Ethical Considerations

  • The study focuses on structural compression of pretrained MoE models using mathematical optimization.Source: Ethical Considerations · p. 9
  • Users should follow source model and dataset licenses, terms, attribution, use restrictions, and redistribution requirements when releasing compressed checkpoints.Source: Ethical Considerations · p. 9
  • Compressed models inherit the biases and limitations of their original pretrained models, so practitioners should apply standard safety evaluations before deployment.Source: Ethical Considerations · p. 9

A. Theoretical Analysis

The analysis motivates PRIME’s frequency weighting with an inference-error bound and derives a sufficient condition for high-frequency experts to be centroids. It also distinguishes the parameter-space bound from the output-space distance used in the practical surrogate.

Notations

  • The analysis assumes expert functions are uniformly L-Lipschitz in their parameters, and defines activation frequency as expected gating probability over the data distribution.The compressed output assigns experts to clusters whose merged parameters replace their original parameters.Source: Notations · p. 11

A.1. Motivating Frequency Weighting via an Error Bound

  • Under a parameter-space Lipschitz assumption, expected inference error is bounded by frequency-weighted parameter distance, and this bound is tighter than the frequency-agnostic bound when activation frequencies are non-uniform.The analysis motivates weighting assignment costs by expert activation frequency.Source: Notations · pp. 11–12, §A.1 · p. 11

Notations

  • The theorem justifies the frequency-weighted structure, but PRIME’s ILP uses normalized Euclidean distances between mean expert outputs on calibration data rather than the theorem’s parameter-space distance.The paper describes this practical objective as a related but distinct surrogate.Source: Notations · p. 12

A.2. Sufficient Condition for Centroid Selection

  • A sufficient condition guarantees an expert is a centroid when its activation frequency is high relative to the minimum cluster-dissolution cost.The authors say this gives a mechanism for preserving high-traffic routing paths, not a formal load-balancing guarantee.Source: §A.2 · pp. 12–13

B. Serving Performance Benefits

Serving gains from expert compression depend on memory pressure and workload shape: they are largest for concurrent, long-output TP1 workloads and small or absent in less constrained settings.

Scope and controls

  • Serving ratios compare each compressed variant with the 128-expert baseline in the same deployment arm, not across GPU counts.The arms are TP1 with automatic KV allocation, TP1 with KV fixed at 121,968 tokens, and TP2 with ample KV capacity.Source: B. Serving Performance Benefits · p. 14

Expert-weight bandwidth

  • With eight active experts per token, compression leaves expert computation approximately unchanged but lets a concurrent decode batch touch fewer distinct experts.Under independent uniform routing, the expected number touched depends on actual decode batch size; at 128/128 and C = 128, free- and fixed-KV gains both equal 1.42× because the workload fits baseline KV capacity.Source: B. Serving Performance Benefits · p. 14

KV-cache capacity

  • At 512/4096 and concurrency 128, the 64-expert model's TP1 gain rises from 1.32× with fixed KV to 2.22× with free KV.At concurrency 256, throughput reaches 2.23× and preemptions fall from 693 to 264; the gain ratio does not establish independent multiplicative causes.Source: B. Serving Performance Benefits · p. 14

Deployment implications and limits

  • At 512/4096 and concurrency 128, the 64-expert model gains 2.22× on TP1 with free KV, versus 1.32× with fixed KV and 1.26× on TP2.The 96-expert model gains 1.74× under the same TP1 setting; single-request decoding shows no meaningful 64-expert improvement, and prefill-dominated serving remains near unity.Source: B. Serving Performance Benefits · p. 15, Figure 7 · p. 15

C. Empirical Analysis of Expert Utilization

Expert utilization on the Qwen3-30B-A3B calibration data is strongly long-tailed, and PRIME's frequency-weighted formulation can treat near-zero-use experts with little penalty.

Expert activation frequency

Compression dynamics

  • When an expert's activation frequency is near zero, its contribution to a frequency-weighted centroid is mathematically insignificant, so merging it into a high-frequency centroid nearly reproduces that centroid.In this regime, merging effectively degenerates into pruning as the low-frequency expert's information is diluted.Source: C. Empirical Analysis of Expert… · p. 16
  • PRIME's frequency-aware objective assigns minimal weight to near-zero-use experts, allowing the ILP to prune or merge them with minimal penalty.The stated design preserves the model's primary inference pathways.Source: C. Empirical Analysis of Expert… · p. 16

D. Impact of Penalty Threshold

The penalty threshold balances pruning against merging, with moderate settings performing best in the tested zero-shot proxy; a Qwen3 visualization also reports balanced expert geometry under PRIME.

D. Impact of Penalty Threshold

  • At λ = 0, the method becomes prune-only and drops sharply on zero-shot tasks, while λ near 1 forces merge-only behavior and degrades accuracy through feature interference.The zero-shot evaluation uses selected multiple-choice tasks as a lightweight hyperparameter-selection proxy across K ∈ {96, 80, 72}.Source: D. Impact of Penalty Threshold · p. 16
  • Across tested sparsity levels, zero-shot performance peaks at λ = 0.25, which the authors adopt as the default.The reported rationale is a balance between filtering structural outliers and retaining information while maintaining representational coherence.Source: D. Impact of Penalty Threshold · p. 16

E. t-SNE visualization of Qwen3-30B-A3B

F. Routing Stability on Downstream Reasoning Tasks

The supplied results report PRIME's downstream routing resemblance to the original model, strong Mixtral comparisons, layer-adaptive pruning and merging, and lower compression overhead than HC-SMoE.

F. Routing Stability on Downstream Reasoning Tasks

  • On OpenBookQA, PRIME's activation heatmap closely aligns with the original model, whereas HC-SMoE shows static dark banding associated with routing collapse and load imbalance.The reported pattern holds on this downstream task, beyond the calibration dataset used for the initial routing analysis.Source: F. Routing Stability on Downstream… · p. 17, Figure 11 · p. 17

G. Effectiveness on Mixtral

  • At K = 6 on Mixtral-8×7B, PRIME records the highest MC average (63.52), Coding Avg (50.45), and GSM8K (69.67) among the compared compressed methods.The reported comparisons are MC average 63.52 versus REAP 62.26 and HC-SMoE 61.38; Coding Avg 50.45 versus 49.13 and 47.77; GSM8K 69.67 versus 65.88 and 68.69, respectively.Source: G. Effectiveness on Mixtral · p. 18, Table 6 · p. 18

H. Per-Layer Pruned Expert Profiling

  • At K = 96, 73% of removed experts are merged and 27% are pruned, with the ILP selecting between them according to layerwise expert distances.The analysis describes merging similar experts and pruning when pairwise distances are large; it notes that merge-only and prune-only strategies can respectively force harmful averaging or discard recoverable information in different layer groups.Source: H. Per-Layer Pruned Expert Profiling · p. 18, Table 7 · p. 18

I. Compression Overhead Comparison

  • PRIME's compression overhead is 3.5 minutes for Qwen3-30B-A3B and 15.5 minutes for Qwen3.5-122B-A10B, versus 142.7 minutes for HC-SMoE overall.The table compares end-to-end stages including collection, distance computation, optimization or clustering, and merging; the paper frames compression as a one-time offline procedure.Source: Table 8 · p. 18, §I · p. 19

Related papers

Loading 2610.13982v1…