GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

• MoE compression must address both shared structure across experts and redundancy within each expert, which shared-subspace methods and width pruning alone leave partly unaddressed.A fixed shared subspace restricts expert-specific variation, while width reduction alone does not encode common structure through shared parameters.Source: §1 · p. 1
• GS-MoE combines group-shared projections for common structure with expert-specific projections and intermediate-width compression for individual transformations.Ridge-leverage neuron selection reduces width before up-projection factorization, while Nyström reconstruction maps retained hidden activations to shared down-projection coordinates.Source: §1 · pp. 1–2
• At 60% compression on DeepSeek-MoE-16B-Base, GS-MoE retains 90% of average reasoning performance versus 73% for D2-MoE and achieves 20% lower out-of-domain perplexity.Source: §1 · p. 2
• For fixed neuron selections and shared projections, per-expert reconstruction-error reductions guide how experts are grouped.The fitted expert-specific projections express the reconstruction objective as a constant minus those error reductions.Source: §1 · p. 2
• A gradient-free layer-importance score guides rank and width budgets across layers, while eigenvalue-based scores and neuron leverage scores allocate capacity across groups and experts.The allocation operates under a parameter budget and adapts the representation at layer, group, and expert levels.Source: §1 · p. 2

2.1 PRUNING AND MERGING OF MIXTURE-OF-EXPERTS

• Expert merging consolidates similar experts, while pruning removes experts; neuron-width methods preserve finer-grained compression but leave shared inter-expert structure unexploited.Source: §2.1 · p. 2

2.2 LOW-RANK DECOMPOSITION AND PARAMETER SHARING

• MoE-SVD, D2-MoE, and TD-MoE use shared components spanning every expert, whereas GS-MoE groups experts to avoid forcing dissimilar subspaces onto one basis.GS-MoE forms groups to maximize each expert’s reconstruction-error reduction.Source: §2.2 · p. 3
• Unlike MoLAE’s arbitrary groups and fixed dimension-ratio compression, GS-MoE compresses every expert, including down projections, with rank and width separately budgeted from calibration statistics.Source: §2.2 · p. 3

3.1 NOTATION

• The notation sets E as the number of experts and d and dff as the model dimension and intermediate width, respectively.Source: §3.1 · p. 3

3.2 COMPRESSION ALONG RANK AND WIDTH

• Group-shared projections use a latent rank while expert-specific projections retain selected intermediate neurons, with ranks and widths allocated non-uniformly.Source: §3.2 · pp. 3–4

3.3 GROUPWISE SUBSPACE SHARING AND PRUNING

• Ridge-leverage scores select each expert’s retained neurons; the up projection uses a group-level surrogate, while the down projection has a closed-form solution before width compression and Nyström reconstruction afterward.Source: §3.3 · pp. 4–5

3.4 ERROR-GUIDED EXPERT GROUPING

• Experts are grouped using per-expert reconstruction-error reductions for candidate shared projections, followed by balanced medoid clustering with fixed group sizes.The normalized dissimilarity is not guaranteed to lie in [0, 1] because the up subspace is fitted approximately.Source: §3.4 · pp. 5–6

3.5 CAPACITY ALLOCATION

• A gradient-free, gate-weighted layer-importance score guides rank and width budgets across layers; within layers, leverage scores select widths and eigenvalue-based scores allocate group ranks.Source: §3.5 · p. 6

4.1 SETTINGS

• The evaluation covers Mixtral-8x7B, DeepSeek-MoE-16B, and Qwen2-57B-A14B, with rank and width compression tested separately and jointly.The baselines are MoE-SVD, D2-MoE, TD-MoE, NAEE, and MoE-I2.Source: §4.1 · pp. 7–8

4.2 RESULTS

• Grouped assignment lowers WikiText perplexity from 29.49 to 25.23; adding group-rank allocation reduces perplexity by 0.57 on average across three datasets.Without grouped assignment, rank allocation provides no consistent improvement.Source: Table 2 · p. 7, §4.2 · p. 9

4.1 SETTINGS

• Calibration uses 512 Alpaca samples of 2048 tokens each, and evaluation spans nine datasets, including language-modeling and zero-shot reasoning tasks.Compression and evaluation use one to two NVIDIA B200 GPUs; compression requires one GPU, and GS-MoE uses no post-compression training.Source: §4.1 · pp. 7–8

4.2 RESULTS

• At 60% compression, Qwen2 attains lossless accuracy and DeepSeek retains 90% reasoning accuracy; GS-MoE significantly outperforms D2-MoE across compression ratios on both models.GS-MoE also compresses Mixtral-8x7B up to 40%, matching or beating other leading methods.Source: §4.2 · p. 8
• Two clustering iterations with two restarts recover 98% of the largest tested gain; lowest perplexity occurs with four iterations and eight restarts.Further iterations slightly increase perplexity.Source: Figure 2 · p. 8, §4.2 · p. 9
• At 60% compression, Mixtral GS-MoE reaches 374.77 tokens/sec with G = 1, 3.32× D2-MoE and 3.66× TD-MoE.On DeepSeek-MoE-16B-Base, increasing G from 1 to 8 reduces throughput by 9.6%; with G = 8, tokens activate an average of 3.68 subspaces, about 19% fewer than with random grouping.Source: Table 3 · p. 9, §4.2 · p. 9, Figure 3 · p. 9

5 CONCLUSION

• For fine-grained architectures, combining group-wise shared subspaces and expert intermediate-width reduction retains state-of-the-art fidelity under aggressive parameter budgets.The framework also uses eigenvalue-based group-rank allocation and leverage-score-guided expert-width allocation.Source: §5 · p. 9

A.1 UP PROJECTION SOLUTION

• For a fixed shared up projection, least-squares fitting minimizes each expert’s reconstruction error, with a pseudoinverse providing a minimizer when projected statistics are singular.The shared projection is selected from the leading eigenspace of the surrogate.Source: §A.1 · p. 13
• For heterogeneous expert input statistics, the up-projection objective does not reduce to one fixed-matrix eigenproblem, so a group-level surrogate estimates the shared projection.The expert-specific projections are then fitted using each expert’s original input statistics.Source: §A.1 · p. 13

A.2 DOWN PROJECTION SOLUTION

• For a fixed shared down projection, orthogonal decomposition yields minimizing temporary expert projections, and an eigendecomposition jointly solves for the shared and temporary projections.The weighted derivation need not minimize the displayed unweighted objective; the paper uses the unweighted sum for H↓ to match Obj. 4.Source: §A.2 · pp. 13–14

A.3 DOWN PROJECTIONS AFTER INTERMEDIATE-WIDTH COMPRESSION

• After width compression, the final down projection is a least-squares minimizer for fixed temporary projections and selected neurons, not a globally optimal compressed expert.It does not jointly optimize neuron selection or account for hidden-activation changes caused by compressed up and gate projections.Source: §A.3 · p. 14

A.4 DERIVATION OF THE GROUPING OBJECTIVE

• For fixed candidate projections, ranks, and neuron selections, assignment minimizes summed up- and down-projection reconstruction errors after fitting expert-specific projections for each candidate group.Assignment-independent terms can be removed, leaving a per-expert score evaluated from calibration second moments.Source: §A.4 · pp. 14–15
• The minimized summed error is a constant minus the sum of per-expert assignment scores, so minimizing error is equivalent to maximizing those scores over assignments.For gated experts, the score includes gate reconstruction error reduction.Source: §A.4 · p. 15
• The equivalence holds for fixed candidate projections and selections, but does not guarantee improvement after surrogate refitting.Scaling the down error by fixed βscale preserves the assignment equivalence for the weighted objective.Source: §A.4 · p. 15

A.5 DISCRETE GROUP-RANK ALLOCATION

• For each group, multiplier search selects the integer rank maximizing eigenvalue-derived value minus a rank-cost penalty; greedy completion assigns affordable residual budget by gain per parameter.Minimum ranks must fit the budget, and the procedure does not generally guarantee a global optimum.Source: §A.5 · pp. 15–16

A.6 LAYER ALLOCATION AND BUDGET CORRECTION

• The global routed-expert retention target is split between width and rank using exponent ν, while pure width pruning is defined as a separate configuration.The reference layer budget includes shared-projection overhead, and discrete allocation can leave unused slack.Source: §A.6 · p. 16
• Layer importance distributes width and rank removal across layers, with width retentions kept fixed as ranks are rescaled to match the continuous total budget.The stated equality holds when preliminary allocations and rescaled ranks are feasible; rounding and discrete allocation can change achieved compression.Source: §A.6 · p. 16

B ADDITIONAL EXPERIMENTAL SETTINGS

• The appendix points to separate procedures for neuron selection, expert grouping, and capacity allocation, while Table 4 lists default compression hyperparameters for main comparisons.Ablation settings are specified separately.Source: B ADDITIONAL EXPERIMENTAL SETTINGS · p. 16, Table 4 · p. 17
• Neuron selection retains at least 5% of each expert’s intermediate width; η varies by model and calibration size, with the main DeepSeek and Qwen comparisons using 512 calibration samples.For DeepSeek and Qwen with 256 samples, η = 0.01 was more effective; the main 512-sample comparisons use η = 0.1.Source: B ADDITIONAL EXPERIMENTAL SETTINGS · pp. 16–17
• Experts with too few routed calibration inputs receive randomly sampled hidden states or activations to reach a sample threshold, while experts with too few routed tokens are excluded from the shared update.Qwen2-57B-A14B includes experts with zero routed calibration tokens; the other evaluated models have sufficiently routed experts.Source: B ADDITIONAL EXPERIMENTAL SETTINGS · pp. 16–17
• Layer removals are clipped and redistributed, group ranks are bounded relative to the layer reference rank, and achieved compression is reported after rounding and discrete allocation.The corresponding removal cap rises to 0.95 when aggregate removal exceeds 0.40.Source: B ADDITIONAL EXPERIMENTAL SETTINGS · p. 17

C ADDITIONAL EXPERIMENTS

• WikiText perplexity falls from 33.5 at G = 1 to 20.75 at G = 8, with diminishing returns as group count doubles.Perplexity is lowest at ν = 0.45; both smaller and larger values perform worse.Source: C ADDITIONAL EXPERIMENTS · p. 17, Figure 4 · p. 18

C.1 NON-UNIFORM LAYER AND INTERMEDIATE WIDTH ALLOCATION

• At 60% compression on Qwen models, non-uniform expert widths improve Wikitext perplexity by 0.23, while non-uniform layer allocation reduces it by 0.26.Table 5 compares uniform and non-uniform allocation; its width setting uses leverage-based allocation and ν controls the width–rank split.Source: §C.1 · p. 17, Table 5 · p. 18

C.2 EVALUATIONS OF GS-MOE’S SUBSETS

• On Qwen and DeepSeek, the full pipeline recovers performance losses seen when factorization or width reduction alone reaches 60% compression.Table 6 compares factorization, Nyström width reduction, and GS-MoE’s full pipeline.Source: §C.2 · p. 18, Table 6 · p. 19
• For Mixtral-8x7B, width reduction compresses up to 40% and outperforms factorization at the same ratios, while the full pipeline does not exceed its reasoning performance.The authors attribute this to Mixtral’s larger intermediate dimension and reduced expert count, which make subspace rank reduction more damaging.Source: §C.2 · p. 18, Table 6 · p. 19