PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Afsara Benazir, Chen Chen, Rongxiao Qu, Jiabo Huang, Jingtao Li, Lingjuan Lyu

arXiv:2608.21693v1cs.LG

TL;DR

MoE compression lacks systematic evidence on how pruning, quantization, and KV-cache compression interact in realistic deployment pipelines. MoE-XBench evaluates these techniques across MoE models, architectures, workloads, and hardware, finding non-additive effects with expert pruning as the dominant degradation source. The benchmark supports practical accuracy-memory-latency comparisons for deployment choices.

  • Problem

    Prior MoE compression studies largely evaluate pruning, quantization, and KV-cache compression in isolation, leaving their composable interactions and deployment tradeoffs insufficiently characterized.

  • Method

    MoE-XBench uses an end-to-end benchmark and eight-module evaluation suite to compare standalone and combined compression pipelines across MoE models, workloads, architectures, and hardware.

  • Results

    Composable compression produces non-additive quality and efficiency effects, with expert pruning dominating degradation and hardware efficiency often decoupled from memory reduction.

  • Takeaways & Limitations

    Deployment choices should target an accuracy-memory-latency point and validate the complete compressed pipeline on the target backend rather than optimizing size reduction alone.

  • Takeaways & Limitations

    The study uses representative methods for each compression axis, scopes efficiency claims to the evaluated execution path, and covers CUDA and Metal rather than all methods, serving layers, models, or devices.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache growth make deployment difficult on commodity hardware. Practical deployment often requires stacking multiple compression techniques: expert pruning removes redundant experts, weight quantization lowers model memory footprint, and KV-cache compression reduces long-context memory pressure. However, these techniques are typically evaluated in isolation, leaving open how they interact when applied together in realistic deployment pipelines. In this work, we present MoEXBench, a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow. MoEXBench studies 10 MoE models ranging from 30B to 235B total parameters across standard-attention, hybrid linear-attention, and sliding window attention architectures. It evaluates 20%-50% expert pruning rates, 1 to 16 bit weight-quantization schemes, and multiple KV-cache precision settings, applied both individually and in combination. MoEXBench introduces an eight-module evaluation suite that jointly measures composable-compression quality, workload and architecture robustness, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware. Our results reveal non-trivial interactions among compression methods: composable compression cannot be predicted from standalone techniques, compression rate alone does not reliably predict quality loss or runtime gain, expert pruning is the dominant degradation source, and average quality can hide workload and architecture-specific failures. By releasing normalized module scores, compressed artifacts, and reproducible scripts, MoEXBench enables practical accuracy-memory-latency comparison across MoE families and hardware backends.

1 Introduction

MoE models reduce active computation through sparse routing but remain difficult to deploy because of expert storage, routing, imbalance, and long-context KV-cache costs. MoE-XBench addresses gaps in systematic and composable evaluation by benchmarking compression quality, robustness, sensitivity, and deployment efficiency.

  • Motivation: MoE routing scales total capacity while keeping active computation relatively small, but deployment still faces expert storage, routing, imbalance, and KV-cache challenges.These costs make commodity-hardware deployment expensive in memory, latency, and energy.
  • Research gaps: Prior MoE compression research lacks systematic coverage across architectures, where sparse routing, imbalance, and redundancy create distinct accuracy-efficiency tradeoffs.The relevant families include standard-attention, hybrid linear-attention, and sliding-window-attention MoEs.
  • Research gaps: Pruning, quantization, and KV-cache compression have largely been evaluated separately, leaving their combined effects unclear.The interactions may be additive, redundant, or harmful because the techniques alter expert utilization, numerical error, and attention behavior.
  • Research gaps: Proxy metrics such as parameter count and FLOPs do not fully capture runtime behavior on target hardware.Evaluation therefore needs peak memory, prefill/decode throughput, and hardware-acceleration measurements.
  • Benchmark: MoE-XBench converts MoE checkpoints into deployable artifacts through calibration, expert pruning, weight quantization, and KV-cache compression.It evaluates 10 MoE LLMs on seven workloads under varying pruning, quantization, and KV-cache settings.
  • Benchmark: The benchmark uses eight modules to separate model quality from deployment efficiency while testing interactions across accuracy, workloads, architectures, efficiency, and hardware backends.This suite targets both composable-compression behavior and practical deployment tradeoffs.
  • Findings: Composable compression is non-additive, with expert pruning dominating degradation and hardware efficiency often decoupled from memory reduction.The authors argue that compression should be evaluated as a deployment pipeline rather than as isolated accuracy-preserving steps.

2 Background

MoE compression combines offline expert and weight reduction with runtime KV-cache compression, but prior work has not systematically evaluated their interactions across MoE families. MoE-XBench fills this gap with unified coverage of compression combinations, architectures, quality, robustness, sensitivity, and efficiency.

  • MoE foundations: MoE models route tokens to a small subset of experts, increasing total capacity without proportionally increasing active computation.Routing designs differ across standard sparse top-k, shared-expert, and other MoE architectures.
  • Compression axes: The benchmark studies offline expert pruning and weight quantization alongside runtime KV-cache compression as post-training deployment techniques.Retraining-based and serving-layer methods are treated as complementary or orthogonal.
  • Compression axes: Expert pruning removes low-utility experts, quantization lowers weight precision, and KV-cache compression reduces long-context memory growth.These techniques act on different parts of the deployed model and inference-time state.
  • Prior work: MoE compression remains less systematically characterized than dense-model compression because studies often use few models, one axis, and narrow evaluation settings.This limits separation of architecture-specific observations from broader deployment trends.
  • Research gap: Interactions among pruning, quantization, and KV-cache compression remain unclear, including whether theoretical savings translate into real memory and throughput gains.The techniques affect different parts of the inference stack, making composability a distinct evaluation problem.
  • Research gap: Existing benchmarks and method-specific studies do not jointly cover all three compression axes, composable pipelines, and multiple MoE architecture families.MoE-XBench addresses this with an eight-module suite spanning quality retention, robustness, sensitivity, and runtime efficiency.

3 MoE-XBench

MoE-XBench evaluates pruning, quantization, and KV-cache compression as standalone or composable deployment primitives, separating model quality from deployment efficiency. Its eight-module suite measures retention, robustness, sensitivity, and practical cost while exposing interactions across compression choices.

  • Overview: MoE-XBench treats expert pruning, weight quantization, and KV-cache compression as composable deployment primitives for standalone or combined evaluation.The workflow supports controlled comparisons between individual compression stages and combined pipelines.
  • Design goals: The benchmark separates model quality from deployment efficiency by reporting accuracy alongside practical efficiency metrics.This prevents gains in one dimension from obscuring losses in another.
  • Benchmark modules: Eight modules quantify stacked performance, workload robustness, architecture reliability, quantization sensitivity, KV-cache sensitivity, expert-pruning sensitivity, and inference efficiency.The modules use raw measurements and normalized scores for detailed analysis, compact comparison, and Pareto analysis.
  • Scoring: Retention-stability scores reward high average retention while penalizing uneven degradation across evaluated settings.Perplexity-based modules use inverse perplexity so higher transformed scores indicate better quality.
  • Inference efficiency: Inference-efficiency scoring compares runtime speedup with effective memory compression rather than measuring absolute speedup alone.A score below 100 indicates that compression does not deliver proportional speedup.

4 Experimental Setup

The experimental setup spans 10 MoE models, seven workload categories, multiple compression configurations, and two commodity-hardware platforms. Measurements cover accuracy, perplexity, memory, throughput, and normalized cross-configuration tradeoffs.

  • Models: 10 MoE LLMs span standard-attention, hybrid linear-attention, and sliding-window-attention families across diverse scales and deployment use cases.Controlled compression sweeps use primary Qwen3 models, while architecture-reliability evaluation uses the full model set.
  • Compression settings: Six compression configurations cover empty, single-axis, and composable subsets of expert pruning, weight quantization, and KV-cache compression.REAP, GGUF, uniform/turboquant, and an alternate REAM pruning evaluation provide representative methods across the axes.
  • Tasks and datasets: Seven workload categories cover knowledge, math reasoning, generative reasoning, instruction following, code generation, tool use, and long-context understanding.Modules 3–6 additionally report perplexity on Wikitext-2.
  • Hardware setup: Efficiency is evaluated on an Apple M1 Max with 64 GB unified memory and an NVIDIA H100 with 80 GB RAM.Inference uses llama.cpp, with the framework designed to remain compatible with expanding GGUF backends.
  • Measurement metrics: Reported metrics include task accuracy, perplexity, compression rate, peak memory, and prefill and decode throughput.Raw results are indexed by model, dataset, hardware, context length, batch size, and KV-cache setting; normalized scores summarize tradeoffs.

5 Accuracy Evaluation

Accuracy depends more on the compression axis and model architecture than nominal compression rate. Expert pruning dominates degradation, while KV-cache compression is comparatively robust at 8-bit but sensitive at lower precision.

  • 19.35% expert pruning increases PPL by 28.5%, whereas 69.5% Q4_K_M weight quantization increases PPL by only 0.7%.
  • Prune+Quant retains Scoreperf = 96.35 and Scorerobust = 88.91, while adding KV-Q8 changes scores only marginally to 96.31/88.28.
  • Qwen3-235B-A22B suffers a 54% PPL increase under Prune+Quant, compared with 31% for the 6.71x smaller Qwen3.6-35B-A3B.
  • Prune+Quant retains high average performance but drops to Scorerobust = 88.91, with larger degradation on MMLU and HumanEval than on GSM8K, MuSR, and RULER.
  • KV-q8 preserves short-context accuracy, whereas q4 and TurboQuant variants cause larger losses, especially in pruned models.
  • At ∼75–80% compression, Q4-based prune+quant remains near quantization-only PPL, while IQ1_M sharply increases PPL; adding KV-q8 changes PPL only marginally.

6 Efficiency Evaluation

Memory reduction does not reliably translate into throughput gains. Runtime outcomes depend on compression type, context length, inference phase, and hardware, with expert pruning often accelerating more effectively than aggressive low-bit quantization.

  • Q4_K_M Prune+Quant reduces peak memory from 60.3 GB to 17.1 GB while improving prefill by 1.37× and decode by 1.30× over BF16.
  • Adding KV compression reduces memory to 15.7 GB but lowers prefill to 0.96× and decode to 0.63× of Prune+Quant.
  • The efficiency relationship between compression rate and throughput is non-monotonic across context lengths and compression configurations.
  • At 32K context, Q4 Quant+KV reduces memory from 21.2 to 19.9 GB while decode drops to 0.61×; at 2K, decode remains 0.97×.
  • Q4_K_M reduces H100 peak memory from 60.3 to 21.2 GB and improves prefill by 1.36×, while M1 Max prefill remains near parity at 0.97×.
  • Prune-only achieves ∼1.3x compression with 0.94-1.03 speedup, whereas Q4_K_M compresses 2.8x-3.1x but achieves only 1.38x-1.51x effective speedup.

7 Discussion

Composable compression should be assessed as a complete deployment pipeline because its effects vary across accuracy, architecture reliability, memory, latency, and hardware. The benchmark therefore recommends target-specific evaluation rather than maximizing size reduction alone.

  • Composable compression mainly shifts sensitivity and hardware axes rather than average accuracy, with pruning driving cross-architecture reliability loss and KV compression further reducing Shw.
  • Practitioners should optimize for the target accuracy-memory-latency point because smaller artifacts do not necessarily improve throughput.
  • Expert pruning dominates quality loss and interacts with quantization and KV-cache compression, making results dependent on pruning ratio and MoE model.
  • Compression order has limited sensitivity but is not order invariant because the techniques act on largely independent axes.
  • The benchmark’s efficiency claims are scoped to the evaluated execution path, while serving-layer techniques and additional models or devices remain future work.

8 Conclusion

MoE-XBench’s calibration and quantization-aware pruning analyses show that deployment choices materially affect retained capabilities, while structural expert removal remains the main concern.

  • 8 Conclusion: The benchmark’s end-to-end tool, configuration, and raw measurements are intended for open-source release.
  • 8 Conclusion: Calibration materially changes the retained capability profile after REAP pruning.C4 performs best on GSM8K and MMLU-R but is weakest on MuSR, HumanEval, Math-Hard, and IFEval; math-only calibration leads on several of those tasks.
  • 8 Conclusion: The official calibration mixture remains the safest general-purpose default because it avoids the largest single-domain failures.
  • 8 Conclusion: Packed-MoE-aware scoring improves over module-level quantization-aware scoring but not over standard BF16 REAP.The comparison uses Qwen3.6-35B-A3B at r = 0.30 and evaluates the pruned artifact after GGUF Q4_K_M export.
  • 8 Conclusion: Structural expert removal explains the remaining degradation more strongly than BF16-to-GGUF mismatch, so BF16 REAP remains the default pruning path.

A.3 Alternate expert compression methods

Comparisons with expert merging show that expert compression remains the dominant degradation source, while the effects of quantization and KV-cache compression vary by method and are not separable from pruning.

  • A.3 Alternate expert compression methods: At a matched 25% expert-count reduction, expert-only compression loses 5.31 points under REAP and 6.05 under REAM from score_perf=100.
  • A.3 Alternate expert compression methods: Adding Q4_K_M weights and Q8 KV contributes only 1.23 and 0.31 additional points, respectively, after expert compression.
  • A.3 Alternate expert compression methods: The fully composable configurations retain nearly identical average performance: 93.46 under REAP versus 93.64 under REAM.
  • A.3 Alternate expert compression methods: Adding Q4_K_M changes score_perf by -1.18 under REAP but -0.06 under REAM, while adding Q8 KV changes it by -0.05 and -0.25, respectively.These differences show that exact degradation magnitudes can vary across expert-compression methods.
  • A.3 Alternate expert compression methods: The central findings are not artifacts of REAP, although the study does not claim method independence across all possible algorithms.
  • A.3 Alternate expert compression methods: Across 0%–50% pruning, quantization and KV-cache gaps remain small relative to the pruning-induced PPL increase.The bf16/f16 row serves as the same-prune baseline in the sensitivity sweep.

C Inference Efficiency on Apple M1 Max

On Apple M1 Max, long-context memory is dominated by KV cache and runtime buffers rather than weights, making KV-cache compression more impactful for memory but costly for decode throughput.

  • C Inference Efficiency on Apple M1 Max: Long-context footprint is dominated by KV cache and runtime buffers rather than weights in the unified-memory regime.Figure 7 evaluates Qwen3-30B-A3B-Instruct at 2K, 8K, and 32K contexts.
  • C Inference Efficiency on Apple M1 Max: Weight-only compression yields limited memory benefit on Apple M1 Max.
  • C Inference Efficiency on Apple M1 Max: KV-cache compression lowers peak memory substantially but hurts decode throughput.On-the-fly dequantization remains on the critical path.
  • C Inference Efficiency on Apple M1 Max: Figure 7 compares peak memory, prefill throughput, and decode throughput across six configurations, two weight formats, and Q8 KV-cache.Q8_0 is the baseline because BF16 caused memory thrashing for this model.

D Reproducibility

The experiments use checkpoints listed in Table 11, including original and REAP-pruned Hugging Face checkpoints.

  • D Reproducibility: Table 11 lists all original and REAP-pruned Hugging Face checkpoints used in the experiments.
Loading 2608.21693v1…