PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture Compressor for Mixture-of-Experts LLMs Gains More

Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, Xiaojuan Qi

arXiv:2410.06270v2cs.LGcs.CL

TL;DR

MoE-LLMs still face memory and computation burdens because all experts must be loaded and activated experts can be redundant. MC combines expert-aware mixed-precision quantization with token-aware dynamic pruning. At 2.54-bit, compressed Mixtral activates around 2b parameters and beats 16-bit LLaMA2-13b by around 8% on MMLU (5-shot).

  • Problem

    MoE-LLMs require all experts to be loaded, and typically activate at least two experts, creating memory and computation overhead despite sparse routing.

    Source: §1 · p. 1

  • Method

    MC combines expert-weighted pre-loading mixed-precision quantization with online pruning of low-confidence experts while protecting important tokens.

    Source: §1 · p. 2

  • Results

    At around 8b (2.54-bit), Mixtral activates only 2b parameters and outperforms 16-bit LLaMA2-13b by around 8% on MMLU (5-shot).

    Source: §1 · p. 3

  • Takeaways & Limitations

    Mixture compression exploits differences among MoE experts to surpass smaller FP models of equivalent size under extreme compression without training.

    Source: §1 · p. 3

Abstract

from arXiv · show

Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, as many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequencies, highlighting their differing importance, and b) not all tokens are equally important -- only a small subset is critical. Building on these insights, we propose MC, a training-free Mixture-Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixed-Precision Quantization, which formulates the adaptive bit-width allocation as a Linear Programming problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning, which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency. Extensive experiments confirm the effectiveness of our approach. For instance, at 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%.

1 INTRODUCTION

MoE-LLMs reduce computation through sparse expert activation, but storing all experts still strains deployment resources. MC combines expert-aware quantization with token-aware dynamic pruning; at 2.54-bit, compressed Mixtral activates 2b parameters and outperforms 16-bit LLaMA2-13b on MMLU.

1 INTRODUCTION

  • MoE deployment remains costly because all experts must be loaded and typically at least two are activated, despite sparse routing.The paper notes that an NVIDIA A100-80GB cannot accommodate Mixtral 8×7b.Source: §1 · p. 1
  • PMQ allocates expert-specific bit-widths by minimizing a weighted objective that combines activation frequency, routing scores, and quantization loss.The allocation is formulated as a Linear Programming model and uses training-free GPTQ.Source: §1 · p. 2
  • ODP prunes low-confidence experts per token using routing weights, while protecting important tokens identified through attention scores and feature magnitudes.Protecting 2% of important tokens mitigates pruning loss while retaining nearly the same compression ratio.Source: §1 · p. 2
  • MC outperforms uniform quantization and other mixed-precision strategies, even surpassing FP models with the same number of activated parameters.The approach combines static expert quantization and dynamic expert pruning.Source: §1 · p. 2
  • At around 8b (2.54-bit), compressed Mixtral activates only 2b parameters and beats 16-bit LLaMA2-13b by around 8% on MMLU (5-shot).The authors attribute the result to exploiting disparities among experts without training.Source: §1 · p. 3

2 RELATED WORKS

Related work situates MoE compression among quantization and pruning approaches for LLMs. It notes that expert-wise quantization remains challenging and that pruning offers dynamic selection during inference.

Quantization for LLMs

  • Prior MoE quantization work introduced expert-guided, block-wise mixed precision, but more effective expert-wise quantization remains a challenge.Codebook methods can improve post-quantization performance through fine-tuning, while QAT requires significant resources.Source: §2 · p. 3
  • PTQ avoids additional training, while mixed-precision methods allocate different bit-widths according to weight salience.QAT or PTQ combined with fine-tuning can better maintain lightweight-model performance.Source: §2 · p. 3

Parameter pruning for LLMs

  • MoE pruning can select experts using activation frequencies or gating statistics, and dynamic pruning can select activation parameters during inference.Prior work explored dynamically activating top-k experts based on gating weights to improve inference efficiency.Source: §2 · p. 3

3 METHOD

MC combines pre-loading mixed-precision expert quantization with online dynamic pruning that retains important tokens. It allocates precision according to expert importance and protects critical tokens during inference.

3.1 PRELIMINARIES

  • Mixtral routes each token to the top 2 of 8 experts, and GPTQ provides the weight-quantization foundation for the compression strategy.The method focuses on expert weights because they dominate MoE-LLM storage.Source: §3.1 · pp. 3–4

3.2 PRE-LOADING MIXED-PRECISION QUANTIZATION

  • PMQ targets expert weights before loading, using mixed precision because uniform bit-width quantization does not meet extreme-compression accuracy requirements.The paper identifies expert weights as the primary storage overhead.Source: §3.2 · p. 4

3.2.1 EXPERTS SIGNIFICANCE ANALYSIS

  • Experts vary in reconstruction impact, routing scores, and activation frequency, while expert routing is sparser on Math than on C4.The method measures expert significance using access frequency and summed routing weights.Source: §3.2.1 · p. 5

3.2.2 OPTIMAL EXPERTS BIT-WIDTH ALLOCATION WITH WEIGHTED IMPORTANCE FACTORS

  • PMQ assigns bit-widths according to expert importance while optimizing expert-wise output reconstruction error under a target average bit-width.The available expert bit-widths are 1, 2, and 3 bits, with constraints retaining at least one 2-bit and one 3-bit expert.Source: §3.2.2 · pp. 5–6

3.3 ONLINE DYNAMIC PRUNING

  • ODP retains important tokens and dynamically selects activated experts for other tokens to improve inference efficiency while maintaining performance.This stage addresses the inference cost that remains after pre-loading quantization.Source: §3.3 · p. 6

3.3.1 ATTENTION DECAY UNDER WEIGHT-ONLY PRUNING

  • Weight-only pruning reduces computation by 15% but decreases performance by approximately 10%, with pruning at token 67 followed by attention decay in block 15.The cited example shows the attention change after pruning an expert for token 67 in block 14.Source: Figure 4 · p. 6, §3.3.1 · p. 7

3.3.2 SIGNIFICANCE-AWARE TOKEN PROTECTION

  • Protecting 2% of important tokens significantly reduces performance losses while retaining an approximately 15% computational-efficiency improvement.The token protection mechanism preserves all experts for protected tokens.Source: §3.3.2 · p. 7

4 EXPERIMENT

Experiments evaluate PMQ's expert-wise quantization and ODP's token-aware pruning, then report their combined memory, inference, and accuracy trade-offs. The results compare bit-width allocation strategies, token-protection settings, and compressed Mixtral models.

4 EXPERIMENT

  • The evaluation calibrates expert factors on 128 C4 sequences of 2,048 tokens, tests Mixtral 8 × 7b and 8 × 22b, and uses PPL plus eight zero-shot benchmarks.Other non-MoE layers use 4-bit quantization; the models are compressed on two and four A100-80GB GPUs, respectively.Source: §4 · p. 7, Table 1 · p. 7

4.1 EXPERIMENT ON PRE-LOADING MIXED-PRECISION QUANTIZATION

  • PMQ outperforms random bit-width allocation; F-norm and PMQ outperform Hessian, with PMQ’s lead over F-norm widening below 2-bit.Activation frequencies perform better than weight-based allocation, while routing-score allocation improves on random allocation but retains high PPL.Source: §4.1 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
  • At 2.54-bit, PMQ reaches 67.50% average accuracy versus BSP’s 49.07%, 3.8% below 16-bit Mixtral; at 1.57-bit, PMQ reaches 54.49%.The comparison covers eight zero-shot benchmarks; at 2.54-bit, uniform quantization loses approximately 28.6% average accuracy.Source: Table 2 · p. 8, §4.1 · p. 9

4.2 EXPERIMENT ON ONLINE DYNAMIC PRUNING

  • Protecting 2% of crucial tokens lowers PPL from 6.46 to 6.24, while activated experts’ parameters change only from 15.1% to 14.8%.Masking all experts for 2% of tokens reaches 15.8% compression and PPL 6.35, outperforming weight-only pruning on both efficiency and accuracy; masking all experts also produces exponential performance-curve growth.Source: §4.2 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
  • At 2.05-bit, MC averages 3.23 GB of activated parameters per token, runs 1.80× faster, and scores 62.68%; ODP reduces activated parameters by about 15% with less than 1% average accuracy loss.At 2.54-bit, the 16.65 GB Mixtral 8 × 7b outperforms 16-bit LLaMA2-13b, whose model size is 26.03 GB.Source: §4.2 · p. 10, Table 4 · p. 10

5 CONCLUSION

The conclusion presents MC as a co-designed quantization-and-pruning strategy for compressing MoE models. It reports high compression with performance retained and identifies multimodal and hardware-specific adaptation as future work.

  • MC co-designs Pre-Loading Mixed-Precision Quantization and Online Dynamic Pruning around differing expert significance.Source: §5 · p. 10
  • Experiments show highly compressed MoE-LLMs can outperform equal-size full-precision dense LLMs without training the original model.Source: §5 · p. 10
  • Future work will adapt the strategy for multimodal applications and optimize it for specific hardware platforms.Source: §5 · p. 10

A.1 MORE QUANTIZED RESULTS OF PMQ

Additional comparisons report PMQ's zero-shot, few-shot, and language-modeling performance against Hessian, BSP, and uniform quantization across bit widths.

  • Across 1.57–2.54-bit settings, PMQ outperforms Hessian by 0.3%–8.6% on eight zero-shot benchmarks, scoring 54.49% versus 45.91% at 1.57-bit.Source: §A.1 · p. 15, Table 5 · p. 15
  • At 1.69-bit, PMQ scores 38.35% on MMLU five-shot, with 16% smaller model size and 4.5% higher accuracy than 2-bit uniform quantization.Source: §A.1 · p. 15
  • At 2.54-bit, PMQ compresses model size by 84% versus 16-bit while few-shot performance is 9.4% lower; at 1.57-bit, WikiText2 PPL is 8.50.Source: §A.1 · p. 15

A.2 ONE-BIT WEIGHT SAVING AND DEQUANTIZATION

The one-bit path binarizes weights, converts them to a storable representation, and reduces the cost of binary matrix operations.

A.2 ONE-BIT WEIGHT SAVING AND DEQUANTIZATION

  • 1-bit weights are binarized and transformed from ±1 values to {0,1}, allowing each element to be stored in one-bit memory.The paper says ±1 values cannot be directly stored in compact memory.Source: §A.2 · p. 16
  • At inference, the binary weights are dequantized for matrix multiplication with an input token vector x and a per-matrix scaling factor s.The passage identifies x as one input vector and s as the scaling factor for each binary matrix.Source: §A.2 · p. 16
  • Binary matrix operations reduce MACs from dm to m and computational complexity from O(m^2) to O(m).Source: §A.2 · p. 16

A.3 RESULTS OF DIFFERENT QUANTIZATION TECHNIQUES

The section evaluates PMQ with GPTQ and Omniquant on Mixtral 8 × 7b, reporting Omniquant’s benchmark performance and quantization cost.

A.3 RESULTS OF DIFFERENT QUANTIZATION TECHNIQUES

  • GPTQ is used as PMQ’s quantization backend and quantizes Mixtral 8 × 7b in 90 minutes.Source: §A.3 · p. 16
  • Omniquant outperforms GPTQ across eight zero-shot benchmarks, demonstrating PMQ’s flexibility to use another quantization method.Source: Table 8 · p. 17, §A.3 · p. 17
  • Omniquant uses learnable weight clipping for calibration and takes approximately 480 minutes to quantize Mixtral 8 × 7b.Its calibration uses 256 C4 sequences for gradient optimization.Source: §A.3 · p. 17

A.4 QUANTIZATION RESULTS ON CHALLENGING BENCHMARKS

On challenging benchmarks, PMQ maintains state-of-the-art performance against BSP and HAWQ, although compression losses are more pronounced on reasoning and long-context tasks.

A.4 QUANTIZATION RESULTS ON CHALLENGING BENCHMARKS

  • PMQ maintains state-of-the-art performance against BSP and HAWQ on challenging benchmarks.Source: §A.4 · p. 17
  • Compression-related performance drops are more pronounced on GSM8K, HumanEval, and long-context Needle-in-a-haystack, a pattern also seen in other MoE-LLM compression methods.Source: §A.4 · p. 17
  • ARC-C and GSM8K are categorized as chain-of-thought tasks while MMLU is categorized as in-context learning, and the paper anticipates larger losses on reasoning-heavy tasks at ultra-low bit-widths.Source: §A.4 · p. 17

A.5 DETAILED RESULTS ON BIT-WIDTH ALLOCATION

PMQ allocates bit-widths according to expert-position importance, protecting a small subset at low bit-widths and retaining those priorities as bit-width increases.

A.5 DETAILED RESULTS ON BIT-WIDTH ALLOCATION

  • Expert importance varies by position in the Mixtral 8 × 7b bit-width allocations.Source: §A.5 · p. 17
  • At lower bit-widths, PMQ protects only a small subset of positions, which the paper says improves calculation efficiency.Source: §A.5 · p. 17
  • As bit-width increases, positions prioritized at lower bit-widths remain protected, supporting consistency in the allocation pattern.Source: §A.5 · p. 17

A.6 ABLATION ANALYSIS ON HYPER-PARAMETERS OF EXPERT SIGNIFICANCE WEIGHT

The ablations examine expert-significance weights, token-dependent pruning metrics, and the ODP threshold. The computation analysis reports deployment speedups and weighs token-importance calculation against pruning efficiency.

A.6 ABLATION ANALYSIS ON HYPER-PARAMETERS OF EXPERT SIGNIFICANCE WEIGHT

  • Overall accuracy remains stable across expert-significance weight settings, with a slight decline when α and β combined exceed the quantization-error weight γ fixed at 2.The experiment tests α and β values of 1, 1.5, and 2 on Mixtral 8 × 7B (2 bit); Table 10 is captioned as using WikiText2.Source: §A.6 · p. 18, Table 10 · p. 18

A.7 COMPARISON OF DIFFERENT TOKEN-DEPENDENT PRUNING METRIC

  • The token-dependent pruning comparison evaluates token kurtosis, token variance, and token mean, pruning 30% of tokens from top-2 to top-1.The section describes these as metrics for token-dependent expert pruning.Source: §A.7 · p. 18

A.8 ABLATION OF DYNAMIC EXPERT PRUNING THRESHOLD

  • PPL is 6.22 with 14.88% of experts pruned for ODP, versus PPL 6.29 with 12.00% pruned under manual threshold 0.4.The authors report that ODP has both superior accuracy and enhanced efficiency in this comparison.Source: Table 12 · p. 19, §A.8 · p. 19

A.9 COMPUTATION ANALYSIS OF ONLINE DYNAMIC PRUNING

  • Token-importance calculation is described as usually low-cost, and the reported efficiency gain from dynamic expert pruning outweighs that calculation cost.The calculation comprises summing attention weights, computing the ℓ1 norm, and top-k calculations.Source: §A.9 · p. 19
  • 40% to 60% speed enhancements are reported for MC, and MC-MoE averages 52 token/s on an RTX 3090 GPU under extreme compression.The reported speedups come from PMQ, CUDA-kernel adaptations, and ODP.Source: Table 13 · p. 19, §A.9 · pp. 19–20, Table 14 · p. 20
Loading 2410.06270v2…