Source-linked AI summary
Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
Tianxiao Cao, Jiahe Shao, Yuning Qiu, Kyohei Atarashi, Hisashi Kashima, Qibin Zhao
TL;DR
MoE models face storage and serving costs because all expert weights must be stored despite sparse routing. The paper analyzes structural costs across pruning, merging, and reconstruction, then proposes data-free SLBF with shared rank-k bases. SLBF consistently outperforms representative methods across five MoE architectures spanning 16B to 122B parameters.
Problem
Sparse routing reduces compute but does not remove the memory burden of storing every expert, making MoE storage and serving costly.
Source: §1 · p. 1Method
The paper analyzes structural error in pruning, merging, and weight reconstruction, and proposes data-free SLBF using shared rank-k bases for weight reconstruction.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Preserving expert structure and routing avoids pruning- and merging-induced structural costs, leaving reconstruction error tied to parameter reconstruction quality.
Source: §1 · p. 1, §6 · p. 9The study focuses on expert FFN weights with fixed decomposition choices and does not explore calibration-guided budget allocation, recovery fine-tuning, efficient factorized inference, quantization, or attention compression.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity. In contrast, weight reconstruction avoids these structural costs by preserving expert structure and routing. Motivated by the analysis, we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that uses rank-$k$ bases shared among experts, enabling richer cross-expert sharing, faster convergence, and lower reconstruction error. A post-hoc gauge fixing removes redundant parameters at no representational cost. Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms methods from all three compression families.
1 Introduction
Sparse routing lets MoE models scale capacity without proportional compute, but all expert weights still occupy memory, making storage and serving costly. The paper analyzes structural costs across three compression families and proposes data-free SLBF, which uses shared low-rank bases; it reports consistent improvements across five MoE architectures.
1 Introduction
- Every expert must reside in accelerator memory even when rarely activated, so sparse routing does not resolve MoE storage and serving costs.The paper cites Qwen3-235B-A22B, where each token routes to 8 of 128 expert FFNs, leaving over 93% of expert parameters idle per forward pass.Source: §1 · p. 1
- Pruning and merging can incur non-vanishing structural errors tied to routing and expert heterogeneity, whereas weight reconstruction preserves expert structure and routing.The paper attributes remaining reconstruction error to imperfect parameter reconstruction.Source: §1 · pp. 1–2
- SLBF replaces MoBE’s shared full-rank bases with shared rank-k factorizations, allowing more bases under the same compression budget.The paper links this shift to richer cross-expert mixing, faster convergence, and lower reconstruction error; post-hoc gauge fixing removes mk2 redundant parameters at no representational cost.Source: §1 · pp. 1–2
- SLBF consistently outperforms representative methods from all three compression families across five MoE architectures spanning 16B to 122B parameters.The evaluation covers five architectures; the introduction does not provide per-model result values.Source: §1 · p. 2
2 Related Work
The paper groups MoE compression work into expert pruning, expert merging, and weight reconstruction, while also noting techniques outside these families. Prior methods differ in which experts or weights they change and how they select or combine them.
2 Related Work
- Merging reduces distinct expert parameter sets by grouping experts and combining their weights into shared representatives.Prior approaches use router-logit similarity, hierarchical clustering on expert outputs, or activation-based saliency for grouping.Source: §2 · p. 2
- Pruning removes entire experts using saliency criteria, with methods ranging from task-specific fine-tuning to retraining-free selection based on routing, reconstruction, gradients, relaxation, or activation norms.REAP combines router values with activation norms.Source: §2 · p. 2
- Weight reconstruction compresses expert matrices while preserving all experts and routing, using shared latent matrices, shared-and-delta decompositions, or mixtures of shared bases.The cited examples are MoLAE, D2-MoE, and MoBE, respectively.Source: §2 · p. 2
- Quantization and weight pruning are additional MoE compression techniques, while post-pruning conversions to bias vectors or subnetworks can compensate for compression loss.These methods operate on a different axis from the three families in the paper’s structural analysis.Source: §2 · p. 2
3 Structural Motivation
The section compares pairwise pruning, merging, and weight reconstruction, then examines structural costs and realized compression error in two pretrained MoEs. The analysis identifies intervention-specific residuals, and the measurements show nonzero structural costs and lower output discrepancy for SLBF across the tested layers.
3.1 Three interventions, three structural costs
- Merging replaces input-dependent mixing with a static mixture, leaving residual cost that grows with expert functional gap, router variability, and pair routing mass.Under the stated approximation, the cost vanishes when routing is static on the pair or the experts are functionally identical.Source: §3.1 · p. 3
- Pruning removes one expert and compensates through the surviving expert; its structural cost is positive when the removed expert is active and its output is not parallel to the substitute’s.The section derives a pairwise pruning bound for this intervention.Source: §3.1 · pp. 3–4
- Weight reconstruction preserves expert identities and routing, so its output error is controlled by parameter reconstruction quality rather than an additional removal or routing-tying term.The stated bound is for simplified linear experts, with an analogous bound for SwiGLU experts whose prefactor depends on weight norms and activation Lipschitz constant.Source: §3.1 · p. 4
3.2 Empirical evidence for three interventions
- Both pruning and merging structural costs are strictly above zero in every layer of the two tested architectures and grow about 3–4 orders of magnitude with depth.The two cost profiles also trace similar layerwise shapes within each model.Source: Figure 1 · p. 3, §3.2 · p. 4
- The empirical comparison uses REAP, HC-SMoE, and SLBF at 25% MoE compression on Qwen3 and Mixtral, spanning high- and low-granularity expert settings.Activations are collected from 128 sequences × 2048 tokens from C4.Source: §3 · p. 3, §3.2 · p. 4
- SLBF has the lowest per-layer output discrepancy across both Qwen3 and Mixtral, while the relative ordering of pruning and merging changes with expert granularity.HC-SMoE has larger drift on Qwen3, whereas REAP has larger drift on Mixtral.Source: Figure 1 · p. 3, §3.2 · p. 4
4 Method
SLBF addresses MoBE’s basis-cost bottleneck with shared rank-k bases that support richer expert mixing and lower reconstruction error. Bilinear gauge fixing removes redundant parameters without changing the represented weights.
4.1 Improved Weight Reconstruction with Shared Low-rank Basis Factorization
- On Qwen3-30B-A3B layer 7’s gate, SLBF reaches MoBE’s reconstruction loss in ∼2k iterations and improves by >20% by 20k iterations.Under the same compression budget, SLBF permits m = n and richer cross-expert mixing, while MoBE is restricted to m < n.Source: Figure 2 · p. 5
4 Method
- MoBE’s full-rank shared bases cost r·d2 parameters each, limiting the number of bases under tight compression budgets.Weight reconstruction preserves expert structure and routing, leaving compression error controlled by parameter reconstruction quality.Source: §4 · pp. 4–5
4.1 Improved Weight Reconstruction with Shared Low-rank Basis Factorization
- SLBF replaces each full basis with a rank-k factorization, lowering per-basis cost and allowing more shared bases under the same budget.The mixture can generically reach rank min(mk, r, d2), despite each individual basis having rank at most k.Source: §4.1 · p. 5
- SLBF achieves lower reconstruction MSE across all 32 Mixtral-8x7B MoE layers under matched compression.Figure 3 reports uniformly lower per-layer MSE.Source: §4.1 · p. 5, Figure 3 · p. 6
4.2 Parameter Reduction via Bilinear Symmetry Gauge Fixing
- Bilinear gauge fixing removes mk2 redundant parameters while preserving each basis product, provided the relevant U submatrix is invertible.The paper reports that this full-column-rank condition held for all trained factors in its experiments.Source: §4.2 · p. 6
5 Experiments
The experiments compare SLBF with representative compression methods across five MoE models and multiple benchmarks. SLBF leads across architectures and compression families, with ablations and serving tests examining its design and runtime trade-offs.
5.1 Setup
- The evaluation spans five models from 16B to 122B parameters, using each model’s conventional benchmark suite and one-shot compression without fine-tuning.SLBF uses m = n shared bases, with rank k selected for the target compression ratio.Source: §5.1 · pp. 6–7, §5.2 · p. 7
5.2 Multi-architecture Results
- SLBF’s advantage reaches a 14.3-point gap over MoBE on Gemma4-26B and a 5.2-point gap on Qwen3.5-122B, where it remains comparable to the original model.At 24% compression on Qwen3-30B, SLBF preserves near-original accuracy while MoBE drops 2.1 points and pruning/merging methods drop over 8 points.Source: Table 2 · p. 7, §5.2 · p. 8
- At 15% compression on Moonlight, the average score drops 1.8 points for SLBF, versus 9.9 for MoBE and over 30 for MoLAE.Moonlight was trained end-to-end with the Muon optimizer, unlike the other evaluated models.Source: §5.2 · p. 8
5.3 Cross-family Comparison on Mixtral
- On Mixtral-8x7B-v0.1, in one-shot runs without fine-tuning, SLBF achieves WikiText-2 perplexity 6.92, versus EAN’s 8.28 and HC-SMoE’s 8.54 (accuracies 70.2 and 69.1, respectively).At 30% compression, SLBF has perplexity 7.78 versus MoBE’s 8.59.Source: §5.3 · p. 8, Table 3 · p. 8
5.4 Ablation study and runtime analysis
- Gauge fixing adds a secondary gain, compressing all three projections scores 62.5 versus 58.2 on Moonlight, and Mixtral projection compression takes 40.9 minutes versus MoBE’s 94.3.On 2×H100, runtime-decompressed serving reduces model-weight memory by 30% and raises KV-cache capacity and concurrency by 53%, at 0.88× throughput.Source: §5.4 · pp. 8–9
6 Conclusion
The conclusion presents SLBF as data-free weight reconstruction using shared rank-k bases, and reports consistent gains across five MoE architectures. Its structural analysis distinguishes reconstruction from pruning and merging by the intervention costs those methods can incur.
6 Conclusion
- Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms representative methods.The comparison includes methods from all three compression families.Source: §6 · p. 9
- SLBF is a data-free weight reconstruction method that uses shared rank-k bases to improve parameter efficiency under fixed compression budgets.It preserves expert identities and routing, while pruning and merging can incur intervention-induced costs.Source: §6 · p. 9
- The structural analysis attributes pruning and merging costs to expert removal and tied routing, while reconstruction error remains controlled by parameter reconstruction quality.Source: §6 · p. 9
Limitations
The study’s scope is limited to expert FFN weight compression and fixed decomposition choices. It also does not assess calibration-guided allocation, post-compression fine-tuning, or broad deployment configurations.
Limitations
- The study focuses on expert FFN weight compression and does not explore interactions with quantization or attention compression.Source: Limitations · p. 9
- The decomposition uses fixed r = d1 and m = n across models and layers; adaptive per-layer or per-projection choices may improve results.Source: Limitations · p. 9
- Deployment evaluation covers limited hardware and serving configurations, and the memory–throughput trade-off may differ with optimized MoE kernels or other systems.Direct loading of the factorized representation can also incur substantial reconstruction overhead for large MoEs in the current implementation.Source: Limitations · p. 9
A.1 Missing Derivations in Sec. 3.1
The appendix derives pairwise error behavior for merging and pruning, then bounds reconstruction error when expert weights are compressed while routing and expert identities are preserved.
A.1 Missing Derivations in Sec. 3.1
- Under a weak-correlation approximation, static merging leaves residual error because it replaces the input-dependent mixing ratio with a constant.The defined irreducible structural cost is the minimum residual for this pairwise static-merging intervention.Source: §A.1 · p. 12
- The pruning derivation considers an oracle router that minimizes error, but it depends on the removed expert’s output and is therefore structurally unrealizable.The appendix frames pruning as removing one expert and compensating through the surviving expert’s modified router.Source: §A.1 · pp. 12–13
- For linear experts, reconstruction error is bounded using Jensen’s and Cauchy–Schwarz inequalities, with the derivation extended to RMSNorm and SwiGLU-style FFNs.The RMSNorm extension uses a finite layer-wise bound on squared gain magnitude; the SwiGLU extension invokes the SiLU Lipschitz property.Source: §A.1 · p. 13
A.2 SLBF training details
SLBF uses weight-side training and SVD-derived initialization, with projection scope adjusted by model. The ablations find gains from gauge fixing and an intermediate number of shared bases.
A.2 SLBF training details
- SVD-derived matrix-sign initialization seeds each expert’s basis factors in a rank-k singular subspace while discarding the singular spectrum.The initialized factors carry no magnitude, which the optimizer recovers during training.Source: §A.2 · p. 14
- Each layer/projection pair is trained independently without calibration data, with separate learning rates for {A, w} and {U, V}; projection scope varies by model.Mixtral leaves down_proj uncompressed, whereas Qwen3, Qwen3.5, Moonlight, and Gemma-4 compress all three FFN projections.Source: §A.2 · p. 14
B.1 Effect of Gauge Fixing on Compression Frontier
- SLBF without gauge fixing outperforms MoBE across the evaluated compression sweep, while gauge fixing adds a gain that widens at higher compression.The freed parameters are reinvested as higher per-basis rank at matched budget.Source: Figure 4 · p. 15, §B.1 · p. 15
B.2 Effect of Number of Bases at Matched Budget
- At 37.5% compression, the reconstruction-loss optimum lies at m = 160 to 224; the experimental choice m = n = 128 remains in a strong operating region.Too few bases converge more slowly and to higher MSE, while too many degrade slightly as per-basis rank falls.Source: §B.2 · p. 15, Figure 5 · p. 16
B.3 Ablation on Downstream Benchmarks
Downstream ablations compare SLBF and MoBE at matched compression and projection scope, and also measure SLBF’s training-time advantage on Mixtral.
B.3 Ablation on Downstream Benchmarks
- At 24% compression on Qwen3-30B-A3B, SLBF reaches 83.7 average accuracy versus MoBE’s 82.8 at matched gate+up scope, and 84.3 with gate+up+down.The comparison separates an SLBF advantage at matched scope from the further gain associated with compressing all three projections.Source: §B.3 · p. 16
- At 15% compression on Moonlight, SLBF improves average accuracy from 54.5 to 58.2 and reduces WikiText-2 perplexity from 8.22 to 7.77 under matched gate+up scope.Extending SLBF to all three projections raises the average score to 62.2, with an additional gain from gauge fixing.Source: Table 5 · p. 16, §B.3 · p. 16
B.4 Compression Runtime
- On Mixtral-8x7B gate_proj, SLBF takes 40.9 minutes per layer/projection pair versus MoBE’s 94.3 minutes, using 40% of the iterations.The methods have comparable per-iteration cost; the reported time difference is enabled by SLBF’s faster convergence.Source: §B.4 · p. 16
B.5 Inference Analysis
Gauge-fixed basis storage reduces basis-materialization work, while runtime decompression trades lower model-weight memory for some throughput overhead. The appendix also records implementation and source details.
B.5 Inference Analysis
- Gauge fixing saves k^2d multiply-adds per basis; for Mixtral-8x7B, the reported saving is about 20% of basis reconstruction cost per projection per layer.The fixed identity block lets the top k rows copy V without multiplication, leaving only the remaining rows for an explicit product.Source: §B.5 · pp. 16–17
- The implementation stores pivot indices and assembles each basis by scattering V into pivot rows and multiplying only the remaining rows.The runtime comparison shares a stock attention backend and uses a naive per-expert loop to isolate basis-reconstruction cost.Source: §B.5 · p. 17
- With runtime decompression on Mixtral-8x7B, SLBF cuts on-device weights by 30%, raises KV-cache budget by 53%, and supports 115.5 versus 75.4 concurrent requests; throughput is 0.88× at 96 prompts.The comparison uses the configured 4416-token sequence length and the uncompressed baseline’s KV-cache saturation operating point.Source: Table 6 · p. 17, §B.5 · p. 17
C Source and licenses
- The five evaluated base models come from official HuggingFace repositories and are used under their respective public licenses; baseline calibration uses 1024 C4 sequences of 2048 tokens.Downstream evaluations use EleutherAI’s lm-evaluation-harness, and code and evaluation scripts are publicly available.Source: C Source and licenses · p. 17
C.1 Evaluation Details
The evaluation specifies the inference backend, task and prompting protocols, and model-specific decoding settings.
C.1 Evaluation Details
- Evaluations use vLLM with lm-evaluation-harness v0.4.11, and original and compressed checkpoints share identical settings.Qwen3, Moonlight, and Mixtral use vLLM v0.17.1; Gemma-4 and Qwen3.5 use v0.19.1.Source: §C.1 · p. 17
- Instruction-tuned models are evaluated on nine zero-shot or few-shot tasks, with task-specific datasets and chat formatting.Tasks are mmlu_flan_cot_zeroshot, gpqa_diamond_cot_zeroshot, C-Eval, CMMLU, gsm8k, arc_challenge_chat, leaderboard_ifeval, mbpp, and HumanEval. gsm8k uses 5-shot and mbpp 3-shot prompting; C-Eval and CMMLU datasets vary by model, and chat formatting is disabled for log-likelihood tasks.Source: §C.1 · p. 17
- Decoding settings vary by model, including greedy decoding for Qwen3 and Moonlight and sampling configurations for Gemma-4 and Qwen3.5.Maximum output lengths are 4,096 tokens for Qwen3, task-dependent 1,024–6,144 for Moonlight and 32–32,768 for Gemma-4, and 32,768 for Qwen3.5.Source: §C.1 · p. 17