PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning and Distilling Mixture-of-Experts into Dense Language Models

Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho

arXiv:2605.28207v2cs.CLcs.AIcs.LG

TL;DR

MoE models are costly for memory-constrained deployment because all expert parameters must be loaded, while existing compression retains an MoE architecture. The paper converts trained MoEs into dense FFNs through expert scoring, selection, grouping, concatenation, scaling, and knowledge distillation. Diversity-aware scoring is the strongest design choice, and MoE-to-dense pruning exceeds matched-parameter dense-to-dense pruning by +6.3 pp at 1.6× faster training after ∼4B-token distillation.

  • Problem

    MoE models require all expert parameters in memory despite sparse activation, and existing compression methods still produce MoE models rather than fully dense outputs.

  • Method

    The framework scores and selects experts, groups and merges them when needed, concatenates them into a dense FFN with magnitude scaling, and distills from the MoE teacher.

  • Results

    +6.3 pp average downstream accuracy over matched-parameter dense-to-dense pruning after ∼4B-token distillation, with 1.6× faster training wall-clock time; diversity-aware scoring generalizes across three models.

  • Takeaways & Limitations

    Expert scoring is the dominant design axis, and diversity-aware scoring supports pure pruning that retains exactly the router’s top-k experts without weight averaging.

  • Takeaways & Limitations

    The benefit over random FFN initialization is smaller on GPT-OSS than on Qwen3, suggesting effectiveness depends on expert pool size.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ~4B-token distillation at 1.6x faster training wall-clock speed.

1. Introduction

MoE models scale parameter counts but remain memory-intensive because all experts must be loaded, while prior compression methods preserve this limitation. The paper introduces and evaluates a systematic MoE-to-dense conversion framework using expert selection, grouping, concatenation, scaling, and distillation.

  • Motivation: MoE models require loading all expert parameters despite activating only a fraction per token, limiting their suitability for memory-constrained deployment.The motivation includes on-device deployment and single-GPU serving.
  • Motivation: Existing MoE compression methods reduce expert counts but still produce MoE models, so they retain the fundamental memory inefficiency.The paper identifies MoE-to-dense conversion as an insufficiently studied alternative.
  • Method: The framework scores and selects experts, groups and merges them, concatenates them into a dense FFN, applies magnitude scaling, and distills from the MoE teacher.Distillation is used to recover quality lost during compression.
  • Evaluation: 350 configurations combine 7 scoring methods, 5 grouping methods, 2 scaling options, and varied selected-expert counts on Qwen3-30B-A3B.The findings are further validated on DeepSeek-V2-Lite and GPT-OSS-20B.
  • Findings: 5.7 pp separates the best and worst scoring methods in average downstream accuracy, whereas grouping contributes only ∼1 pp.Diversity-aware scoring achieves the best accuracy across all 35 scoring-grouping combinations.
  • Findings: +6.3 pp is the MoE-to-dense advantage over matched-parameter dense-to-dense pruning after ∼4B-token distillation, with 1.6× faster training wall-clock time.The controlled comparison uses an approximately 30B-parameter teacher and a 3B-parameter student.

2. Related work

Prior work compresses MoE models by pruning experts, merging experts, or compressing dense models, while D-optimal subset selection supplies the paper’s diversity-aware foundation. The paper applies this criterion in expert output space to select jointly informative experts.

  • MoE expert pruning: MoE-pruning methods reduce active experts or prune neurons and weights within experts, generally retaining sparse MoE routing.Examples include REAP, SlimMoE, and MoE-Pruner.
  • MoE expert merging: MoE-merging methods combine experts through router-weighted averaging, similarity-based grouping, or output-based clustering.The cited approaches differ in how they group experts and compute merge weights.
  • Dense compression and distillation: Dense compression methods use activation-based importance scoring, structural pruning, and knowledge distillation to produce compact language models.The paper situates its pipeline alongside Minitron and related pruning or continued-pretraining methods.
  • D-optimal design: D-optimal subset selection maximizes an information matrix’s log-determinant and is monotone submodular, admitting a greedy (1−1/e)-approximation.The criterion comes from classical experimental design.
  • D-optimal design: The importance-weighted Gram matrix captures expert quality and pairwise redundancy, selecting jointly informative experts rather than individually top-ranked ones.The paper describes this as the first application of D-optimal subset selection to MoE compression.

3. Method

The method converts an MoE layer into a dense FFN by selecting and optionally merging experts, concatenating their weights, and distilling from the MoE teacher. Its scoring framework combines expert importance with output-space diversity to avoid redundant selections.

  • 3.1. MoE-to-dense conversion: The conversion selects the top-K experts, assigns them to k groups, optionally merges groups, and concatenates the resulting weights into a dense FFN.When K=k, experts are copied without merging; when K>k, experts are merged within groups before concatenation.
  • 3.1. MoE-to-dense conversion: The constructed dense FFN uses intermediate dimension d_dense = k × d_expert, matching the teacher’s active FFN width per token.Across the evaluated models, d_expert × k falls in the typical dense range of approximately 3–5 × d.
  • 3.1. MoE-to-dense conversion: The dense construction preserves group-representative intermediate activations, but static aggregation cannot reproduce token-dependent router weights.Attention layers, embeddings, and layer norms are copied unchanged.
  • 3.2. Scoring: Seven scoring methods span frequency-based, conditional-probability, activation-weighted, and D-Optimal approaches computed from calibration-pass statistics.Frequency-based methods use selection frequency, pre-selection probability, or post-selection probability; CP isolates routing confidence for selected tokens.
  • 3.2. Scoring: D-Optimal selection raises effective rank for K=8 from 6.07 to 7.37 with CP and from 6.31 to 6.93 with ACP.The effective-rank measure ranges from 1 for near-duplicates to K for maximally diverse selections.
  • 3.2. Scoring: D-Optimal selection jointly maximizes importance and diversity through an importance-weighted expert-output Gram matrix and greedily selects experts by log-determinant gain.The greedy procedure has total time complexity O(K^3E), and applying it to CP and ACP yields DO-CP and DO-ACP.

4. Experiments

Across 350 configurations and three MoE models, diversity-aware expert selection consistently mattered more than grouping, while pure pruning and MoE-to-dense distillation produced strong accuracy and efficiency results.

  • Experimental setup: 350 configurations sweep seven scoring methods, five grouping strategies, two scaling options, and selected expert counts across Qwen3-30B-A3B.The workflow evaluates initialization with pre-distillation WikiText-2 perplexity, then distills selected configurations for 0.3B tokens and tests five downstream benchmarks.
  • Scoring and grouping: 32 of 35 scoring×grouping configurations select K=8 or K=16, indicating that retaining a small number of experts usually beats merging many.The best-performing configurations use small K, which also limits grouping’s effect.
  • Scoring and grouping: 5.7 pp separates the best and worst scoring families, roughly 5× the 1.2 pp grouping spread.Frequency-based methods favor generalists, conditional-probability methods favor specialists, and D-optimal scoring adds diversity.
  • Scoring and grouping: 43.41% is achieved by DO-ACP at K=8, the highest average accuracy across the 35 configurations.The top configurations use pure pruning, with one expert per group and no weight averaging; ACP K=8 reaches 42.52% versus 40.50% for ACP×OC K=16.
  • Distillation method exploration: Expanded teacher routing improves quality modestly at k′=16 but costs approximately 2× teacher FLOPs per MoE layer.Beyond k′=32, performance degrades monotonically as lower-ranked experts add noise.
  • Extended training: +6.3 pp separates DO-ACP from dense-to-dense pruning after extended distillation, reaching 58.10% versus 51.84% average accuracy.DO-ACP also trains at 73 s/step versus 116 s/step for D2D, a 1.6× speedup on identical hardware.
  • Qualitative analysis: DO-ACP’s advantage includes lower catastrophic failure and knowledge-error rates, with 54.5% catastrophic failures and 4.2% knowledge errors.Its MMLU advantage is especially large on knowledge-intensive subjects, including 49.2% on humanities versus 25.4% for SF.

5. Conclusion

The paper presents a systematic MoE-to-dense conversion framework using expert selection, grouping, concatenation, and distillation. Diversity-aware scoring is the central design choice, and the resulting method is validated across three MoE models.

  • The framework converts trained MoE models into fully dense architectures by scoring, selecting, grouping, and concatenating experts before distillation.The evaluation covers Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B.
  • 5.7 pp separates the best and worst scoring methods, compared with ∼1 pp for grouping, making scoring the dominant design axis.DO-ACP achieves the best accuracy across all configurations and all three evaluated models.
  • +6.3 pp average downstream accuracy distinguishes MoE-to-dense from dense-to-dense pruning after ∼4B-token distillation.The comparison also reports 1.6× faster training wall-clock speed for MoE-to-dense pruning.
  • The reported recipe uses DO-ACP scoring, exactly the top-k experts per layer, and forward-KL distillation.This corresponds to pure pruning without weight averaging.

6. Limitations

The appendix details the conversion and selection procedures, while identifying practical boundaries around merging, training scale, and expert-pool dependence. Several formal components rely on assumptions or proxy interpretations.

  • Scope limitations: ∼4B-token extended training is insufficient to establish the quality ceiling, and method gains vary with expert-pool size.The gain over random FFN initialization is +3.3 pp on GPT-OSS with 32 experts versus +10.7 pp on Qwen3 with 128 experts.
  • Approximation boundary: Block concatenation preserves representative intermediate activations, but the dense model replaces token-dependent router weights with static down-projection scaling.Full conversion also fixes selected experts and may replace groups with parameter-averaged representatives; distillation compensates for these approximations.
  • Conversion procedure: The conversion selects top-K experts, partitions them into k groups, computes score-based merge weights, scales down-projections, and applies the procedure independently across MoE layers.When K=k, selected experts are copied directly; when K>k, grouped experts are averaged.
  • Expert selection: The greedy D-Optimal selector maximizes marginal log-determinant gains until K experts are chosen, with overall complexity O(K^3E).The method uses an importance-weighted Gram matrix and a regularizer.

E.1. Redundancy counterexample (Proof of Theorem 3.1)

The counterexample contrasts redundant generalist experts with orthogonal specialists: independent importance ranking can miss specialists, whereas log-determinant selection chooses a zero-error subset.

  • Construction: The constructed top-1 MoE has K identical experts and K−1 pairwise orthogonal specialist experts.The first block responds identically on shared points, while each specialist targets a distinct point.
  • Independent ranking: Independent top-K ranking selects the redundant identical experts because their base importance exceeds the specialists’ importance.This creates a selected set that lacks the specialist features.
  • Reconstruction error: The independently ranked subset incurs constant squared reconstruction error on the specialist points.The optimal scalar coefficient matches the shared points but reconstructs the remaining K−1 points as zero.
  • Log-determinant selection: Every size-K subset maximizing the log-determinant objective achieves zero reconstruction error.The maximizers contain one redundant expert and all K−1 specialists.
  • Guarantee: The greedy log-determinant objective is monotone and submodular, yielding the stated (1−1/e) approximation guarantee.The proof derives diminishing returns through Schur complements.

F.1. Finite-sample calibration guarantee

The finite-sample analysis establishes calibration-based stability for D-Optimal scoring and exact recovery for output clustering under boundedness, nondegeneracy, and separation assumptions.

  • Calibration: The empirical D-Optimal framework estimates expert scores and Gram statistics from i.i.d. calibration tokens.The population and empirical objectives use regularized log determinants over selected subsets.
  • D-Optimal scoring: Uniform stability of empirical DO-CP and DO-ACP is stated under bounded expert outputs, positive routing probabilities, and nondegenerate score conditions.The theorem provides a probability-at-least-1−δ guarantee for the resulting estimates and objective transfer.
  • Output clustering: Average-linkage output clustering exactly recovers the target partition when within-group dissimilarities are separated from between-group dissimilarities.The recovery holds up to permutation of group labels.
  • Finite-sample recovery: Finite-sample cosine-similarity concentration transfers the separation condition to empirical dissimilarities and preserves exact recovery with probability at least 1−δ.The empirical error is bounded uniformly over selected expert pairs.
  • Merging analysis: Function-space analysis identifies score-weighted averaging as the oracle representative, but parameter averaging of nonlinear experts need not realize it exactly.The implemented merge applies those weights in parameter space as a practical proxy.

I. Base vs. post-trained teacher comparison

The scoring-method hierarchy is nearly unchanged between base and post-trained teachers, and downstream results preserve the advantage of pure pruning for leading methods.

  • Scoring robustness: The top four scoring methods—ACP, DO-CP, DO-ACP, and CP—retain their exact PPL ranking across base and post-trained teacher variants.The bottom three methods swap positions but remain tightly clustered.
  • Scoring robustness: Base-teacher PPL values are 1.2–1.6× higher than post-trained values, consistent with weaker pre-instruction language modeling.
  • Downstream validation: The base-teacher sweep distills the top configuration for each scoring method at K=8 and K=16 across five benchmarks.Each configuration uses 0.3B tokens.
  • Downstream validation: ∼42.7% average accuracy is achieved by ACP and DO-ACP, ahead of CP at ∼41.1%, SF at 34–37%, and random baselines at 30–32%.
  • Downstream validation: Pure pruning at K=8 outperforms merging at K=16 for ACP by +2.68 pp and DO-ACP by +1.66 pp, while only SF benefits from merging.
  • Teacher comparison: The best base ACP configuration reaches 42.81%, versus 42.52% for its instruct counterpart, while base DO-ACP reaches 42.59% versus 43.41% for instruct.

L. Full distillation results

The full results comprise all 35 scoring–grouping configurations and report pre- and post-distillation WikiText-2 perplexity alongside downstream performance.

  • Evaluation scope: The complete evaluation covers all 35 scoring×grouping combinations on Qwen3-30B-A3B.
  • Evaluation scope: Table 13 reports 0.3B-token distillation results using pre- and post-distillation WikiText-2 perplexity for every configuration.
  • Evaluation scope: Average performance is compared across configurations, with bold values identifying the best average for each scoring method.

M. Dense-to-Dense (D2D) pruning baseline

A matched dense-to-dense baseline is constructed through architecture search and identical distillation conditions, but performs substantially worse than MoE-to-dense configurations.

  • Baseline setup: The D2D baseline prunes a 32B-parameter Qwen3-32B teacher to a comparable student and distills it with matched hyperparameters and token budget.
  • Architecture search: Five approximately 3.4B-parameter candidate architectures vary width, depth, layer count, and hidden or FFN dimensions.
  • Architecture search: The selected architecture preserves all 64 layers, uses d=2048 and ddense=6144, and has 3.44B parameters.
  • Architecture search: Removing layers substantially increases pre-distill PPL, indicating that depth preservation is critical at this compression ratio.
  • Distillation outcome: 33.28% average accuracy for D2D is only 0.6 pp above the random FFN baseline and below the weakest MoE-to-dense configuration at 36.31%.

N. Error taxonomy and examples

The qualitative analysis uses six error categories with separate rule-based and LLM-judged procedures, and reports representative examples plus cross-model results.

  • Error taxonomy: Six error categories are defined for the qualitative MMLU analysis and applied in priority order.
  • Error taxonomy: Surface-level failures are classified by rule-based heuristics, while knowledge and reasoning errors are classified by an LLM-as-a-judge.
  • Examples: Representative examples are provided for each qualitative error category.
  • Cross-model results: Cross-model configurations are reported for DeepSeek-V2-Lite and GPT-OSS-20B.
  • Cross-model results: The GPT-OSS-20B table evaluates a post-trained reasoning model in completion mode, with native-format performance expected to be substantially higher.

P. Model architecture details

The paper summarizes teacher and corresponding dense-student architectures for all three experimental models, including model-specific implementation adjustments. DeepSeek-V2-Lite additionally uses shared experts and a standard dense FFN in layer 0.

  • Table 19 summarizes the teacher and corresponding dense-student architectures for all three models used in the experiments.
  • DeepSeek-V2-Lite has 2 shared experts per MoE layer, while layer 0 uses a standard dense FFN zero-padded to match d_dense.
  • Model-specific adjustments cover DP scaling, routing renormalization, shared experts, and layer 0 handling.
Loading 2605.28207v2…