Source-linked AI summary
REAM: Merging Improves Pruning of Experts in LLMs
Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand, Ali Parviz, Min-Joong Lee, Boris Knyazev
TL;DR
MoE models are difficult to deploy because their total expert count creates substantial memory and overhead, motivating compression that preserves useful knowledge. The paper proposes REAM, which groups and merges experts rather than removing them, and finds that calibration mixtures create an MC–GEN trade-off while REAM often outperforms baselines and can approach uncompressed performance.
Problem
MoE deployment retains a large total expert footprint, while pruning may discard useful knowledge and existing merging comparisons rely on contestable assumptions.
Method
REAM uses saliency-aware expert grouping and weighted merging to preserve all experts’ knowledge while achieving pruning-like compression.
Results
REAM often outperforms baselines; its performance depends on calibration composition, producing an MC–GEN trade-off, while 25% reduction is often comparable to the original model.
Takeaways & Limitations
Calibration mixtures should be chosen with task type in mind because REAM can occupy favorable MC–GEN trade-offs and match uncompressed GEN performance in some settings.
Takeaways & Limitations
No single compression method dominates across all setups and tasks, and expert-compression performance varies with calibration composition.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
1 Introduction
MoE models reduce per-token computation through sparse expert routing but retain a large total parameter footprint, motivating compression that removes redundancy without sacrificing performance. REAM addresses the pruning–merging trade-off by preserving expert knowledge while achieving pruning-like compression, and performs strongly across MC and GEN evaluations.
- MoE layers activate only a small subset of experts per token, enabling very large parameter counts with comparatively small active compute.
- Redundant experts motivate two main compression directions: pruning removes experts, while merging combines similar experts.
- REAP can outperform simple merging, but removing experts may discard useful knowledge and leave the pruning–merging trade-off unresolved.
- REAM preserves all experts’ knowledge while using grouping and weighting to behave similarly to pruning.
- REAM combines similarity modeling, pseudo-pruning, informed weight alignment, and sequential merging into one expert-compression framework.
- Across Qwen3 and GLM4.5, REAM is evaluated at 25% and 50% expert reduction on eight MC and six GEN benchmarks, often outperforming baselines.At 25% reduction, it performs comparably to or only slightly below the original uncompressed models.
2 Related Work
MoE compression reduces deployment costs through system-level efficiency or model-level reduction, with expert pruning and merging as the main model-level paradigms. The paper builds on REAP while challenging assumptions about discarded knowledge and merging error.
- MoE deployment cost depends on the total number of experts despite sparse inference, motivating both serving-system and model-compression approaches.
- Expert reduction mainly uses pruning, which removes experts by saliency, or merging, which combines experts in weight–activation space or shared-subspace representations.
- REAP achieves state-of-the-art large-scale results at 25% and 50% compression, but may discard knowledge outside calibration domains and relies on assumptions about merging error.
3 Background
MoE layers use routers to select a few experts and weight their outputs, while compression methods estimate expert saliency, similarity, and group structure. REAM combines these signals with saliency-aware pseudo-pruning and modified gate handling.
- An MoE layer replaces a Transformer feed-forward block with N experts and a learned router that produces input-dependent scores.
- Router scores are converted to probabilities, and masked top-k routing activates only the highest-scoring experts for each token.In Qwen3, an example is N = 128 and top-k = 8.
- Expert saliency estimates importance; routing frequency counts top-k selections, while REAP weights selections by estimated contribution magnitude.
- REAP’s contribution-weighted saliency better preserves MoE layer outputs and is used by REAM.
- Expert similarity can use output similarity or gate-logit similarity, with gate-logit similarity comparing experts’ router-score vectors across calibration tokens.
- Standard grouping selects salient centroids and assigns other experts by similarity before merging aligned expert weights within each group.
- Figure 1 contrasts pruning, similarity-based merging, and REAM’s saliency-weighted pseudo-pruning, which protects salient centroids and leaves many singleton groups unchanged.REAM also recomputes activations after merging each layer before processing the next layer.
- REAM removes gate-weight rows for non-centroid experts, following pruning rather than summing gate logits as conventional merging methods do.
4 Router-weighted Expert Activation Merging
REAM measures expert similarity using both routing behavior and gated outputs, then uses that signal to form groups, align weights, and merge experts sequentially. Its pseudo-pruning structure preserves many experts unchanged while concentrating merging into a few larger groups.
- Aggregated expert similarity: REAM aggregates gate-logit similarity with gated expert-output similarity to capture both routing-level and representation-level expert specialization.Gated outputs are modulated by the router, making the metric aware of how experts are used.
- Pseudo-pruning: Pseudo-pruning selects high-saliency experts as centroids and greedily assigns similar non-centroid experts to them.The procedure uses REAP saliency scores computed over calibration data and assigns up to C experts per centroid.
- Pseudo-pruning: Because non-centroid experts are typically fewer than centroid absorption capacity, pseudo-pruning creates a few large groups while many singleton experts remain unchanged.The paper gives the example that N′ is 25% smaller than N.
- Activation and weight permutation alignment: REAM aligns expert weights with a combined activation-and-weight cost matrix before computing weighted averages.Activation costs compare normalized calibration-token neuron activations, while weight costs compare corresponding weight distances.
- Activation and weight permutation alignment: Combining activation and weight costs makes matched neurons consistent across both data-driven behavior and weight space, avoiding the limitations of either cost alone.Data-only matching can be noisy, whereas weight-only matching ignores how neurons are used in practice.
- Sequential merging: Sequential merging updates model outputs after each compressed layer so statistics used for subsequent layers reflect the currently merged model.This avoids relying on stale statistics collected from the original unmodified model.
5 Experiments
REAM is evaluated against pruning and merging baselines across MoE models, calibration mixtures, compression levels, and MC/GEN benchmark suites. Its results show that calibration composition creates a substantial MC–GEN trade-off, while REAM can achieve strong Pareto performance and preserve GEN quality at larger-model scale.
- Experimental setup: The evaluation covers Qwen3 and GLM4.5 MoE models, 25% and 50% expert reductions, eight MC tasks, and six GEN reasoning and coding tasks.Compressed models are evaluated without fine-tuning after compression, using calibration mixtures of general text, mathematics, and code.
- Main results: Calibration composition strongly affects Freq, REAP, and REAM, whereas HC-SMoE is comparatively insensitive to the mixture.For Freq and REAP, omitting code causes HumanEval and LiveCodeBench scores to collapse near zero despite strong math performance, with a gap exceeding 40 points versus the best configuration.
- Main results: REAM reaches a GEN average of 69.8 with a 0:0.5:0.5 mixture, within 1.1 points of the uncompressed 128-expert baseline at 70.9, but falls to 47.7 with a 0.5:0.5:0 mixture.With well-chosen mixtures, REAM consistently outperforms the baselines; REAP ranks second, while HC-SMoE and Freq are roughly tied.
- Calibration and Pareto analysis: C4 proportion correlates positively with MC scores and negatively with GEN scores, while code proportion correlates positively with GEN scores for Freq, REAP, and REAM.The reported C4 correlations are r ≥0.95 for MC and r ≤−0.82 for GEN in the 96-expert setting.
- Calibration and Pareto analysis: At 25% compression, REAM has the highest Pareto hypervolume, 920.3, and the highest frontier fraction, 7/10, exceeding REAP’s 878.0 hypervolume and 5/10 frontier fraction.These metrics indicate broad performance coverage across calibration mixtures rather than reliance on a single configuration.
- Larger models: REAM matches the GEN score of uncompressed Qwen3-Coder-Next at 25% compression and consistently outperforms REAP on GEN across the three additional models.On IFEval, AIME25, and GSM8K, REAM often recovers the original score; GPQA remains sensitive for both methods.
- Additional experiments: REAM’s pre-logit embedding rank is an excellent predictor of downstream performance across calibration mixtures, with the steepest and tightest regression curve among the methods.The analysis supports rank as a cheap, task-agnostic proxy for estimating favorable calibration mixtures.
6 Conclusion
REAM achieves strong generative performance at 25% and 50% compression, but no compression method dominates across all setups and tasks. Calibration mixtures expose an MC–GEN trade-off, while small evaluation sets can make gaps from uncompressed models uncertain.
- HC-SMoE balances MC and GEN performance, whereas REAP and REAM can each dominate one of the two task types.
- MC performance can deteriorate under some calibration mixtures, suggesting MC and GEN rely on different expert subsets.
- Small-sample benchmarks such as AIME25, with 30 problems, introduce considerable variance in estimating gaps from uncompressed models.
A.1 Hyperparameters
REAM uses a single pseudo-pruning group-size hyperparameter, set according to the model’s original or target expert count rather than heavily tuned.
- REAM’s only hyperparameter is pseudo-pruning group size C, fixed separately for each model and compression target.
- C is set to 16 or 32 across the evaluated Qwen3 and GLM-4.5 models, with larger values used when models have more experts to merge.
A.2 MC and GEN Tasks
The evaluation covers eight multiple-choice tasks and six generative reasoning and coding tasks, using standard harnesses and task-specific evaluation procedures.
- MC tasks: The MC suite contains eight tasks, including WinoGrande, ARC, BoolQ, HellaSwag, MMLU, OpenBookQA, and RTE.
- GEN tasks: The GEN suite contains six generative reasoning and coding tasks.
- Evaluation procedure: Evaluation uses EleutherAI Language Model Evaluation Harness with HuggingFace or vLLM backends and default task settings.
- Evaluation procedure: GPQA-Diamond uses five shots without chain-of-thought, while LiveCodeBench-v6 uses its official evaluation code.
A.3 Why Evaluate on Different Mixtures of the Calibration Dataset?
The paper varies calibration mixtures because merging decisions depend on activation data, making downstream performance sensitive to which capabilities the calibration distribution represents. The experiments span general text, math, and code mixtures and analyze their MC–GEN effects.
- Calibration sensitivity: Merging decisions are computed from calibration-set activations, so experts underrepresented in that distribution may receive low saliency and be absorbed despite target-task importance.
- Calibration mixtures: The experiments use mixtures of C4, NuminaMath, and The-Stack-Smol, with ten ratios spanning math-heavy to code-heavy configurations.
- MC–GEN trade-off: Figure 6 compares MC and GEN scores for Freq, REAP, HC-SMoE, and REAM across ten mixtures at 64 and 96 experts, with marker size showing dataset share.
- Calibration sensitivity: C4 proportion correlates positively with MC and negatively with GEN, whereas code shows the opposite pattern; math has weak correlations with both.Across Freq, REAP, and REAM, C4 correlations are r ≈+0.95–+0.96 for MC and r ≈−0.82–−0.85 for GEN; code correlations are r ≈+0.59–+0.71 for GEN and r ≈−0.40–−0.57 for MC.
A.4 Additional Ablations on Qwen3-Coder-Next
Additional ablations show that calibration composition strongly affects generative performance, while sequential merging is the most damaging component to remove. Code-biased calibration can produce task-specific gains above the uncompressed model, but some gains do not generalize across ratios.
- Calibration sensitivity: AIME25 ranges from 53.3 to 83.3 across methods and calibration mixtures, demonstrating roughly a 30-point sensitivity swing.The endpoints are REAP with a code-heavy 0.1:0.1:0.8 mixture and REAM without logit-profile similarity at 0.0:0.3:0.7.
- Calibration sensitivity: The code-biased 0.0:0.3:0.7 mixture achieves the best overall GEN average across REAM variants.
- Task-specific gains: Code-heavy calibration raises GSM8K above the original model for REAP and REAM, reaching 89.7 and 89.0 versus 85.4.The same calibration setting pushes HumanEval to 95.1, also exceeding the uncompressed model.
- Component ablations: Removing sequential merging causes the largest ablation damage, reducing REAM GEN from 72.9 to 69.0 at the 0.0:0.3:0.7 ratio.
- Component ablations: The 83.3 AIME25 score without logit-profile similarity does not transfer across ratios, indicating a calibration interaction rather than a consistent gain.