One Student Knows All Experts Know: From Sparse to Dense

Motivation

• Sparse MoE models provide large capacity but are prone to overfitting, difficult to deploy, and hardware-unfriendly, while dense models are easier to use but weaker at comparable computation.MoE deployment can require distributing experts across devices, and expert parallelism introduces expensive communication and sparse-operation overhead.Source: §1 · pp. 1–2

Proposed task

• Knowledge integration trains a dense student by gathering knowledge from multiple pre-trained MoE experts and refining the result through knowledge distillation.The student is intended to cover knowledge from many experts while retaining a standard dense architecture.Source: §1 · p. 2

Method

• The framework evaluates summation, averaging, Top-KG, and SVD-KG for gathering expert knowledge before distillation-based refinement.Top-KG and SVD-KG extract key knowledge to initialize the dense model, while distillation reduces gathering noise.Source: §1 · p. 2

Results

• 3.7× inference speedup over the MoE counterpart is achieved while OneS preserves 88.2% of MoE benefits on natural language processing benchmarks using 46% of the parameters.The framework is evaluated in both computer vision and natural language processing.Source: §1 · p. 2

2.1. Mixture-of-Experts

• MoE routes each input to a small number of expert sub-networks, enabling sparse computation and large model capacity at comparable computation cost.The gating network supplies routing weights, and top-K selection activates the sparse subset of experts.Source: §2.1 · p. 2
• Load-balance loss encourages tokens and routing probabilities to be distributed more uniformly across experts.Without regularization, a small subset of experts may receive most tokens, reducing efficiency and accuracy.Source: §2.1 · pp. 2–3

2.2. Problem Formulation

• The framework gathers knowledge from E MoE experts by approximating their expert transformations with a dense student's corresponding feed-forward network parameters.The student uses the same activation function and architecture while replacing the experts' separate trainable parameters with dense parameters.Source: §2.2 · p. 3
• Knowledge integration has two stages: knowledge gathering initializes the dense student, and knowledge distillation fine-tunes it to reduce teacher–student output differences.The distillation stage follows typical knowledge-distillation approaches.Source: §2.2 · p. 3
• MoE benefits measures how much of the teacher's performance a dense student retains relative to a dense model trained without knowledge integration.The score can be any evaluation metric, such as accuracy for image classification.Source: §2.2 · p. 3

3. Approach

• Knowledge integration first initializes a dense student from matched teacher layers and gathers knowledge from MoE experts into its single FFN, then fine-tunes it by distillation.Matched layers can copy teacher weights directly; the MoE layer requires merging multiple expert FFNs into one dense FFN.Source: §3 · p. 3

3.1.1 Summation and Averaging

• Summation and averaging provide two straightforward baselines for gathering expert weights, while the framework also evaluates Top-KG and SVD-KG.Biases in MoE layers are initialized by averaging across experts because they contain fewer parameters than weights.Source: §3.1 · pp. 3–4, §3.1.1 · p. 4, Figure 2 · p. 4

3.2. Top-K Knowledge Gathering

• Top-KG selects high-l2-norm columns from each expert and concatenates the extracted submatrices to initialize the student weight matrix.For two-layer expert FFNs, it combines column norms from the first layer with row norms from the second to avoid mismatched selections.Source: §3.2 · p. 4

3.2.1 SVD Knowledge Gathering

• SVD-KG decomposes each expert weight matrix into low-rank components, merges those components, and reconstructs a dense matrix containing selected expert knowledge.An adaptive SVD ratio controls how much rank is retained because a fixed rank may not cover key knowledge in every expert.Source: §3.2.1 · pp. 4–5
• Knowledge gathering inevitably introduces noise when conditional computation is removed, motivating a second refinement stage after SVD-KG.The paper directs detailed analysis of this induced noise to Appendix A.Source: §3.2.1 · p. 5

3.3. Knowledge Distillation

• Soft knowledge distillation minimizes KL divergence between teacher and student softmax outputs to refine the gathered dense student.The formulation uses student and teacher logits with a softmax temperature, while hard-label distillation is also evaluated separately.Source: §3.3 · p. 5

3.4. Optimization

• Optimization combines the task-specific main loss with a weighted distillation loss, while omitting MoE load balancing because the pretrained teacher remains fixed.The balancing coefficient α controls the relative contribution of the main and distillation losses.Source: §3.4 · p. 5

4.1. Computer Vision

• Computer-vision experiments use ImageNet and Cifar10, while language experiments use GLUE and SQuAD with Distill, Switch, and ALBERT as comparison baselines.The framework is evaluated across both computer vision and natural language settings.Source: §4.1 · p. 5, §4.2 · p. 7

4.1.1 Results on ImageNet

• SVD-KG performs best among the four ImageNet knowledge-gathering methods and becomes the default choice for subsequent experiments.Top-KG is comparable at large scale but slightly worse at base scale.Source: §4.1.1 · pp. 6–7
• 78.4% ImageNet top-1 accuracy with 15M parameters, while OneS-L-SVD preserves 61.7% of WideNet's improvement and improves over Switch-L by 0.6 points.OneS-L-SVD also outperforms WideNet-B by 0.9% with half the parameters and matches ViT-B using 17% of its trainable parameters.Source: §4.1.1 · pp. 6–7

4.1.2 Results on Cifar10

• OneS-L outperforms Switch-L by 0.6% on Cifar10 and reaches comparable performance to WideNet-B with 0.33× the trainable parameters.OneS-B also outperforms Switch-B, indicating that ImageNet pre-training benefits transfer to the downstream task.Source: Table 2 · p. 6, §4.1.2 · p. 7

4.2.1 Results on NLU benchmarks

• 88.2% MoE benefits across four natural-language tasks, with OneS outperforming ALBERT, Distill, and Switch baselines on all tasks.OneS surpasses Switch by 0.42 on average and achieves 51.7% MoE benefits over Distill.Source: §4.2.1 · p. 7
• 3.7× inference speedup over the MoE counterpart reflects both lower computation and the greater hardware friendliness of dense operators.WideNet uses 2.4× the FLOPs at MoE layers, while other layers have the same computation cost as OneS or ALBERT.Source: Table 3 · p. 7, §4.2.1 · p. 8

4.3. Ablation study

• Removing knowledge gathering causes a significant performance drop, while knowledge distillation helps the student use noisy gathered weights and stabilizes lite-model training.Removing both components also harms the large model.Source: §4.3 · p. 8, Table 4 · p. 8
• At comparable global training epochs, OneS still consistently outperforms baselines, so its gains are not explained solely by extra training iterations.With more epochs, WideNet without MoE stops improving while OneS continues to benefit.Source: §4.3 · p. 8, Figure 3 · p. 8

5. Conclusion and Future Work

• 88.2% MoE benefits, 3.7× inference speedup, and 46% trainable parameters summarize OneS’s reported effectiveness and efficiency.The conclusion also reports 0.42× FLOPs per MoE or FFN layer.Source: §5 · p. 8
• Future work targets improved knowledge gathering and distillation, integration across experts with different architectures, and adaptation to extremely huge MoE models such as GLaM [7].Source: §5 · p. 8

A. Knowledge Gathering Noise Analysis

• SVD-KG integrates E expert linear layers into one dense student layer by approximating expert weights with selected-rank SVD components.The analysis considers expert weights and biases alongside the dense student layer’s weights and bias.Source: A. Knowledge Gathering Noise Analysis · p. 11
• The analysis simplifies the setting by treating experts and the student as single linear layers and, for one case, assumes the MoE selects the first expert.Source: A. Knowledge Gathering Noise Analysis · p. 11
• Under the load-balance assumption, selected experts have routing scores near 1.0, allowing the MoE layer to be approximated through SVD.Source: A. Knowledge Gathering Noise Analysis · p. 11
• The low-rank approximation covers most informative knowledge in selected experts, while noise decreases linearly with λ, so smaller λ is required for noise reduction.Source: A. Knowledge Gathering Noise Analysis · p. 11

B.1. Computer Vision

• Table 5 reports hyper-parameters for ImageNet pre-training and Cifar10 fine-tuning, including α and λ from the training equations.Source: Table 5 · p. 11
• Computer-vision training uses a smaller learning rate of 0.004 because the authors observe that a larger rate harms accuracy when training from MoE initialization.Source: §B.1 · p. 11

B.2. Natural Language Processing

• Table 6 reports the hyper-parameters used for NLP downstream-task fine-tuning.Source: Table 6 · p. 12
• NLP hyper-parameters follow existing works, including [6] [11] [27].Source: §B.2 · p. 12

C.1. Method

• Hard-label distillation treats the teacher’s hard decision as a true label in a classification objective.The teacher prediction and ground truth jointly supervise the student through cross-entropy.Source: §C.1 · p. 12

• Hard-label distillation achieves performance comparable to soft distillation on ImageNet.Source: §C.2 · p. 12, Table 7 · p. 12
• Soft distillation performs slightly better and is recommended as the default because it is popular across more tasks.Source: §C.2 · p. 12

• The ablation varies the SVD ratio λ, which denotes the ratio of selected k, and evaluates its effect on ImageNet top-1 accuracy.Source: D. Ablation Study on SVD Ratio · p. 12, Figure 4 · p. 12

E. Experimental Justification for Less Knowledge in Bias

• Removing bias from the MoE layer causes no obvious performance drop, suggesting that bias contains less knowledge than weights.Source: E. Experimental Justification for Less… · p. 12, Table 8 · p. 12

F.1. Mixture-of-Experts

• MoE improves effectiveness and efficiency across tasks, while this work fixes a pre-trained MoE model and targets dense-model usability.Source: §F.1 · pp. 12–13

F.2. Knowledge Integration

• Knowledge integration combines knowledge from a pre-trained large MoE, unlike knowledge inheritance from small models or approaches that sparsify dense models into MoE models.Source: §F.2 · p. 13