Delta Decompression for MoE-based LLMs Compression

1. Introduction

• MoE models’ large parameter footprints and multiple expert weights create storage and memory barriers, while pruning or merging can compromise expert knowledge or diversity.The section frames the challenge as balancing compression with preservation of expert diversity.Source: §1 · pp. 1–2

"Diversity is not about how we differ. Diversity is about embracing one another’s uniqueness."

• Expert-weight CKA similarities of 0.3 to 0.5 indicate moderate overlap, while delta weights concentrate energy in larger singular values, suggesting low-rank structure.The authors use these observations to motivate retaining expert-specific differences while compressing them.Source: "Diversity is not about how we differ.… · p. 2, Figure 1 · p. 2, Figure 2 · p. 2
• D2-MoE decomposes expert weights into a shared base weight and expert-specific delta weights, using Fisher-weighted merging, truncation-aware SVD, and semi-dynamical pruning.The base captures common structure; the deltas retain expert-specific variations. The pruning approach combines static and dynamic phases.Source: "Diversity is not about how we differ.… · p. 2
• 0.52 average accuracy at 60% compression for Mixtral-8×7B versus 0.36 for NAEE; D2-MoE also reports strong results across other models and tasks.The reported pattern includes lowest perplexity on language-modeling datasets and highest average accuracy on reasoning benchmarks for Mixtral-8×7B and DeepSeekMoE-16B-Base.Source: "Diversity is not about how we differ.… · pp. 2–3

Related approaches

• Pruning methods can lose expert knowledge and require fine-tuning, while merging can reduce diversity and trade off compression against model capacity.The related-work discussion contrasts direct removal and merging with retaining expert-specific variations.Source: §2 · p. 3

Delta compression

• Earlier delta-compression work targets differences between fine-tuned models and base models through quantization or serving systems.The cited examples include GPT-Zip, BitDelta, Delta-CoMe, and DeltaZip.Source: §2 · p. 3

D2-MoE distinction

• D2-MoE applies delta compression to MoE compression and adds truncation-aware SVD and semi-dynamic pruning.The authors contrast these techniques with prior quantization- and system-level delta-compression work.Source: §2 · p. 3

3.1. Delta Compression in MoE LLMs

• Expert weights split into a shared base that captures common structure and expert-specific deltas that preserve unique characteristics, reducing redundancy and minimizing performance loss.Source: §3.1 · p. 3

3.2. Experts Fisher Merging

• Fisher-weighted merging gives greater influence to more important experts, producing more compact, lower-rank deltas for later compression.Fisher importance measures the sensitivity of expert parameters to the data log-likelihood.Source: §3.2 · p. 4

3.3. Truncation-aware Singular Value Decomposition

• Truncation-aware SVD decomposes activation-weighted deltas and retains top singular components, mitigating reconstruction loss from activation outliers.Source: §3.3 · p. 4

3.4. Semi-dynamical Structured Pruning

• Base-weight pruning first removes consistently low-contribution columns, then reprioritizes retained columns by the current batch’s activations to reach the remaining target sparsity.Each phase removes half of the target sparsity.Source: §3.4 · p. 5

3.5. Overall Algorithm Procedure

• The forward pass combines the shared base with selected experts’ delta factors; after compression, storage is (n·p%+s%/2)m and activation parameters are (k·p%+s%)m.Source: §3.5 · p. 5

4.1. Experimental Setups

• Evaluation covers four MoE models and ten datasets in a zero-shot setting, using 512 WikiText-2 samples for calibration without retraining full model parameters.The models range from Mixtral-8×7B and Phi-3.5-MoE with 8 and 16 experts to DeepSeek-MoE-16B-Base and Qwen2-57B-A14B with 64 experts.Source: §4.1 · pp. 5–6

4.2. Compression Performance and Comparisons

• At 60% compression, Mixtral’s average score is 0.52 versus NAEE’s 0.36; DeepSeek-MoE-16B-Base reaches an average score of 0.41, with similar superiority on Phi-3.5-MoE and Qwen2-57B-A14B.The reported Mixtral average score is 0.60 at 20% compression, or 95.2% of the original 0.63.Source: Table 2 · p. 6, §4.2 · p. 6
• At 20% compression on Mixtral, D2-MoE achieves the best overall average score, 0.60 (95.2% of original 0.63), across pruning-, SVD-, and hybrid-method comparisons.The reported WikiText-2, PTB, and C4 perplexities are 4.65, 16.32, and 8.59; ARC-e and WinoG accuracies are 0.80 and 0.75.Source: §4.2 · pp. 6–7, Table 3 · p. 7
• At 80% compression, D2-MoE reaches 313.29 tokens/sec with 290 TFLOPs, outperforming NAEE by 12.5% and MoE-I2 by 6.5% in throughput.On WikiText-2, perplexities are 6.35, 8.15, and 12.95 at 60%, 70%, and 80% compression, respectively.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Ablation study

• At 40% compression on Mixtral, Fisher merging yields the lowest reported perplexities: WikiText-2 5.28, PTB 20.54, and C4 10.10.The method selectively extracts important weights from different experts into the base weights.Source: §4.3 · p. 7
• Dynamic pruning performs best among tested base-weight compressors, while truncation-aware SVD with a scale matrix performs best for delta weights.The selected D2-MoE combination uses semi-dynamic base-weight pruning and truncation-aware delta-weight SVD.Source: §4.3 · pp. 7–8
• At 40% total compression, compressing base weights less generally preserves more expert-critical information, while more base compression typically speeds inference but may degrade performance.The authors select a balanced base/delta ratio for performance and inference efficiency.Source: Table 7 · p. 8, §4.3 · p. 8
• Trimming seven delta weights gives 75% compression with WikiText-2 perplexity 14.71; calibration-data choice has minimal influence on overall task performance.With one delta weight trimmed, 43% compression yields WikiText-2 perplexity 6.43, comparable to the untrimmed model.Source: §4.3 · p. 8

5. Conclusion

• D2-MoE integrates delta decomposition, Fisher-weighted merging, truncation-aware SVD, and semi-dynamical structured pruning to reduce parameters while maintaining model performance.It separates shared base weights from expert-specific delta weights to address weight redundancy.Source: §5 · p. 8
• The decomposition and pruning steps involve complexity that the authors aim to simplify in future work.Source: §5 · p. 8

Impact Statement

• The work focuses on technical approaches to improve storage and reasoning efficiency in MoE LLMs, with evaluations on public benchmarks for transparency and reproducibility.Source: Impact Statement · p. 9

Appendix

• The appendix provides extended results and calibration-dataset analyses, experimental configurations, and algorithms or pseudo-code for key framework components.Source: Appendix · p. 11

A.1. Detail Analysis of Main Results

• At 20% compression, D2-MoE scores 0.60 on Mixtral-8×7B versus the original 0.63; at 60%, its average score is 0.52 versus NAEE's 0.36.At 20%, it maintains 95.2% of the original model's performance.Source: §A.1 · p. 11
• DeepSeek-MoE-16B-Base average scores are 0.54, 0.49, and 0.41 at 20%, 40%, and 60% compression, respectively.The passage also reports similar higher-performance patterns for Phi-3.5-MoE and Qwen2-57B-A14B.Source: §A.1 · p. 11
• At 20% compression, D2-MoE preserves close to 90% of original performance on ARC-e, WinoG, and PIQA across tested models.Source: §A.1 · p. 11

A.2. Comparison with Other Methods

• At 20% compression, D2-MoE scores 0.60 versus the original model's 0.63 and outperforms pruning-, SVD-, and hybrid-based methods across reported metrics.It maintains 95.2% of the original model's capabilities.Source: §A.2 · p. 11
• D2-MoE reports WikiText-2 4.65, PTB 16.32, and C4 8.59, matching or outperforming pure pruning methods on these perplexity evaluations.Source: §A.2 · p. 11
• On downstream reasoning tasks, D2-MoE scores 0.80 on ARC-e and 0.75 on WinoG, surpassing all baseline methods.Source: §A.2 · p. 11
• LoSparse and MC-SMoE show significant degradation, especially on perplexity metrics, while D2-MoE remains stable across evaluation dimensions.Source: §A.2 · p. 11

A.3. Analysis of Runtime SpeedUp and Memory Usage

• At 60% compression, D2-MoE reaches 277.72 tokens/sec with 481 TFLOPs, 2.1% higher throughput than NAEE at matching computational efficiency.MoE-I2 and LoSparse require 838 and 1150 TFLOPs, respectively, while delivering lower throughput.Source: §A.3 · p. 11
• At 80% compression, D2-MoE reaches 313.29 tokens/sec, beating NAEE by 12.5% and MoE-I2 by 6.5% while using the lowest TFLOPs (290).At this setting, it uses 9.33B parameters and 17.3G memory.Source: §A.3 · pp. 11–12

A.4. Computational Cost Discussion of D2-MoE

• On Mixtral-8x7B, Fisher merging takes 11 minutes, scale-matrix collection 19 minutes, and delta-weight SVD 25 minutes.Metric calculation uses calibration data and gradients; Fisher metrics weight parameters during expert merging.Source: §A.4 · p. 12, Table 10 · p. 12

A.5. Additional Results on Compression Ratios between Base Weights and Delta Weights

• At 40% compression, D2-MoE records WikiText-2 perplexity 5.28; the authors choose 10% Base-weight pruning, or 60% when prioritizing throughput.The corresponding SVD decomposition ratio for Delta weights is selected to preserve performance.Source: §A.5 · p. 12

A.6. Pushing the Limit of Compressing Delta Weight

• Delta-weight V matrices have CKA similarity around 0.9 and U matrices around 0.3, motivating shared V matrices and merging U matrices.The authors suggest Fisher merging as one way to merge U matrices.Source: §A.6 · p. 12, Figure 5 · p. 13

A.7. Adaptive Compression Ratio for Delta Weight

• Layer 2 is most sensitive and receives the largest Delta-weight budget, while least-sensitive layer 1 receives the smallest.The strategy reallocates parameters across layers under the same total budget.Source: §A.7 · p. 13, Figure 6 · p. 13

B.1. Details of Delta Weights Trimming

• Trimming one Delta weight gives 43% compression with WikiText-2 perplexity 6.43; trimming seven gives 75% compression and perplexity 14.71.Perplexity generally rises as more Delta weights are trimmed; trimming selects the expert with the lowest router sampling frequency.Source: §B.1 · p. 13

B.2. Motivation of CKA Similarity between experts and Single Value energy Retention

• CKA similarity between experts reflects redundancy motivating Base-weight merging, while energy retention measures information preserved by SVD truncation.Energy retention is evaluated using the number of largest singular values retained, k.Source: §B.2 · pp. 13–14

• Mixtral-8x7B has eight 7-billion-parameter experts and activates two experts per token through top-2 routing.Source: §B.3 · p. 14
• DeepSeekMoE-16B-Base has 16 one-billion-parameter experts plus a shared expert, with routing that selects experts per token.Source: §B.3 · p. 14
• Phi-3.5-MoE has 16 experts, each containing 3.5 billion parameters, and uses routing to distribute computational load.Source: §B.3 · p. 14
• Qwen2-57B-A14B comprises 64 experts and a shared expert, with 57 billion total parameters and 14B active.Its sparse gating activates only a subset of experts per token.Source: §B.3 · p. 14
• The models vary in expert counts, parameter distribution, and routing; evaluation covers 10 datasets: three language-modeling and seven reasoning tasks.The datasets include WikiText-2, PTB, C4, OpenbookQA, WinoGrande, HellaSwag, PIQA, MathQA, ARC-easy, and ARC-challenge.Source: §B.3 · p. 14

B.4. Calibration Dataset

• WikiText-2 calibration samples consistently outperform C4, and 512 samples are selected as the performance–efficiency trade-off.Source: §B.4 · p. 14
• Increasing calibration samples generally improves results, but 2048 samples cause slight performance degradation.Source: §B.4 · p. 14

B.5. Hyperparameters and Experimental Configurations

• The experiment settings include target compression ratios, SVD truncation thresholds, and static and dynamic pruning sparsity levels.Source: §B.5 · p. 15