PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Mixture of Experts in Large Language Models

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, Jiayi Huang

arXiv:2407.06204v3cs.LGcs.CL

TL;DR

MoE research has expanded rapidly, but existing surveys do not comprehensively cover current sparse-MoE developments in LLMs. This survey addresses the gap with a taxonomy and systematic review spanning algorithms, systems, applications, evaluations, challenges, and resources.

  • Problem

    Comprehensive reviews of MoE in LLMs remain scarce, with earlier surveys focused on dense MoE or predating major post-2022 developments.

  • Method

    The survey organizes MoE research across algorithm design, system design, and practical applications, supplemented by implementations, hyperparameters, empirical assessments, and a resource repository.

  • Results

    The survey provides a systematic compendium of MoE models, designs, applications, empirical evaluations, challenges, and future research directions.

  • Takeaways & Limitations

    The taxonomy and curated resources support researchers in navigating and continuing the study of MoE technologies.

  • Takeaways & Limitations

    Sparse MoE models can overfit specific tasks or datasets, undermining generalization to unseen data and diverse input variations.

Abstract

from arXiv · show

Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs.

1 INTRODUCTION

MoE offers a way to scale LLM capacity while limiting computation by routing inputs to specialized experts. This survey addresses limited recent coverage by organizing MoE progress across algorithm, system, and application perspectives.

  • Motivation: MoE routes each input to pertinent specialized experts, increasing model capacity without a corresponding surge in computational demands.The framework uses a gating mechanism to select experts conditionally for each input.
  • Motivation: Recent MoE growth includes industrial-scale models such as Mixtral-8x7B, Grok-1, DBRX, Arctic, and DeepSeek-V2.The survey highlights particularly notable growth in 2024.
  • Research gap: Existing surveys provide either an early review of dense MoE or a pre-2022 account that misses major post-ChatGPT developments.The authors identify comprehensive reviews focused specifically on recent MoE in LLMs as scarce.
  • Scope and organization: The survey introduces a taxonomy organizing recent MoE progress into algorithm, system, and application categories.It aims to provide a clear and comprehensive overview of MoE in LLMs.
  • Scope and organization: Its algorithm coverage examines MoE layers, gating and expert-network choices, implementations, hyperparameters, evaluations, and extensions beyond standard backbone integration.Discussed extensions include soft MoE, parameter-efficient experts, dense-sparse transitions, and related derivatives.
  • Scope and organization: The survey also covers system optimizations for computation, communication, and storage, plus applications in NLP, vision, recommender systems, and multimodality.The system discussion targets sparse and dynamic computational workloads.

2 BACKGROUND ON MIXTURE OF EXPERTS

An MoE layer combines expert networks with a gating network that routes inputs to experts, either activating all experts or only a top-k subset. Sparse routing reduces computation but introduces load-balancing challenges addressed with auxiliary losses.

  • MoE layer structure: A Transformer MoE layer typically contains N expert networks and a linear-softmax gating network that directs each input to appropriate experts.The MoE layer commonly replaces the FFN after self-attention within a Transformer block.
  • Dense MoE: Dense MoE activates all expert networks for each input, combining their outputs through the gating values.Dense activation has been used in early MoE proposals and revisited by later studies.
  • Sparse MoE: Sparse MoE activates only selected experts, computing a weighted sum from the top-k experts instead of all experts.This sparse mechanism was introduced to reduce the computational overhead associated with dense activation.
  • Sparse MoE: TopK retains the k largest gating entries and sets the others to −∞, making their post-softmax contributions approximately zero.Common choices are k = 1 and k = 2; noise is often added during training to encourage exploration and stability.
  • Load balancing: Sparse gating can unevenly distribute tokens across experts, so MoE layers use an auxiliary load-balancing loss to promote equal utilization.The loss considers both token-distribution proportions and gating-probability proportions across experts.

3 TAXONOMY OF MIXTURE OF EXPERTS

MoE scales model parameters through conditional computation, using specialized experts and a gating mechanism to select experts for each input. The survey organizes MoE research across algorithm, system, and application perspectives.

  • 3 TAXONOMY OF MIXTURE OF EXPERTS: MoE uses specialized expert networks and a gating mechanism to dynamically select computational resources for each input.This conditional-computation strategy scales parameters without a corresponding increase in computational demand.
  • 3 TAXONOMY OF MIXTURE OF EXPERTS: Mixtral 8x7B contains eight FFN experts per layer and uses 13 billion active parameters while matching or surpassing larger models across benchmarks.
  • 3 TAXONOMY OF MIXTURE OF EXPERTS: The taxonomy categorizes MoE models from three perspectives: algorithm design, system design, and application.It is presented as a guide for navigating the rapidly evolving MoE literature.

4.1 Gating Function

MoE gating functions determine which experts process each input and how their outputs are combined. The survey covers sparse, dense, and soft routing, along with load-balancing objectives, capacity constraints, and alternative assignment strategies.

  • 4.1 Gating Function: Gating functions orchestrate expert computation and output combination through sparse, dense, or fully differentiable soft routing.Sparse routing activates a subset of experts, dense routing activates all experts, and soft routing includes token or expert merging.
  • 4.1.1 Sparse: Sparse gating activates a selected subset of experts per token as conditional computation, using decisions that may be binary or continuous, stochastic or deterministic.
  • 4.1.1 Sparse: Token-choice gating routes inputs to top-k experts, while hierarchical routing reduces the branching factor when the expert count reaches thousands.
  • 4.1.1 Sparse: Auxiliary losses promote equitable token distribution across experts by encouraging balanced expert importance and assigned workloads.Importance balancing alone does not guarantee an even distribution of training examples, motivating additional load-balancing objectives.
  • 4.1.1 Sparse: Expert capacity limits can cause token overflow, while random secondary routing offers an alternative when the designated expert exceeds capacity.
  • 4.1.1 Sparse: Figure 4 compares sparse top-1 gating, BASE layers, grouped domain mapping with random gating, expert-choice gating, attention routing, and soft expert merging.
  • 4.1.1 Sparse: BASE layers formulate token-to-expert allocation as a linear assignment problem that maximizes affinities while assigning each expert an equal number of tokens.S-BASE uses optimal transport, whereas DSelect-k provides a smoother top-k gating alternative for gradient-based training.

4.2 Experts

MoE experts predominantly replace Transformer FFN modules, while research also explores attention-based, shared, convolutional, and parameter-efficient expert designs. Model quality and efficiency depend on expert count, expert size, and MoE-layer placement, with hybrid dense–sparse architectures offering additional trade-offs.

  • Network Types: FFN layers remain the predominant location for MoE experts because they are hypothesized to be sparser and more domain-specific than self-attention layers.Empirical analysis reports only 20% active experts in FFN layers on downstream Wikitext tasks.
  • Network Types: Mixture of Attention Heads extends MoE to attention by routing query and output projections while sharing key and value computations.A common gating network selects matching experts for query and output projections, enabling shared pre-computation of keys and values.
  • Network Types: Researchers have also investigated CNN and LoRA-based experts, broadening expert networks beyond standard FFN architectures.These variants include convolutional experts and parameter-efficient fine-tuning methods that construct experts with LoRA.
  • Hyperparameters: The main MoE hyperparameters are expert count, expert size, and the frequency of MoE layers, whose optimal settings depend on tasks and infrastructure.These choices jointly affect model performance, computational efficiency, model size, and system overhead.
  • Hyperparameters: MoE-layer placement varies from alternating FFN replacement to every-layer integration, with studies reporting quality, speed, and overhead trade-offs.V-MoE finds fewer MoE layers can preserve performance while improving speed, whereas MoE-LLaVA reports advantages for alternating placement on multimodal tasks.
  • Hybrid Architectures: Residual-MoE combines a fixed expert with a gated expert, engaging two experts without exceeding the communication cost of top-1 gating.The gated expert functions as an error-correcting adjunct to the fixed dense FFN, and NLLB uses a conceptually similar combination.

4.3 Mixture of Parameter-Efficient Experts

Mixture of Parameter-Efficient Experts combines MoE routing and multiple experts with PEFT methods to improve resource efficiency and task or domain adaptability. The survey organizes these approaches by placement in FFN layers, attention modules, Transformer blocks, or entire layers.

  • Motivation: MoPE combines MoE’s multi-expert task versatility with PEFT’s resource efficiency to address limited trainable parameters and catastrophic forgetting.The approach targets PEFT’s difficulty generalizing across multiple tasks while retaining parameter-efficient adaptation.
  • Feed-Forward Network: FFN-based MoPE trains parameter-efficient experts and a gating mechanism while leaving the rest of the pretrained model intact.LoRAMoE inserts LoRA experts into the FFN and separates them into task-learning and instruction-alignment groups with localized balancing constraints.
  • Feed-Forward Network: Domain- and task-oriented FFN MoPE methods use staged adaptation, top-1 routing, or shared FFN weights to manage data conflicts and computational cost.MixDA separates domain-adapter and task-adapter training, LLaVA-MoLE activates one LoRA expert, and MixLoRA fuses LoRA experts with a shared FFN.
  • Attention: Attention-based MoPE augments Query, Key, Value, or Output projections with parallel parameter-efficient experts and corresponding gating networks.During fine-tuning, only activated experts and the gating network are trained; the equations define expert and gate terms for attention modules.
  • Attention: Attention-focused variants use LoRA experts, contrastive objectives, task clustering, capacity constraints, auxiliary balancing losses, and expert dropout.MoELoRA encourages diverse expert outputs, MoCLE routes clustered tasks, and SiRA addresses load imbalance and overfitting.
  • Transformer Block: Transformer-block MoPE assigns separate expert groups to attention and FFN modules, each controlled by its own gating mechanism.MoV uses lightweight (IA)3 vectors for Key, Value, and FFN activations, reducing gradient-computation burden and storage memory.

4.4 Training & Inference Scheme

MoE training and inference schemes seek to combine dense-model knowledge, sparse computation, and deployable dense forms. The survey groups them into original training, Dense-to-Sparse, Sparse-to-Dense, and expert-merging approaches, with distinct efficiency and quality trade-offs.

  • Original Training & Inference Scheme: Original MoE schemes construct the sparse model and train it from scratch, followed by inference using the same model configuration.This approach is established in seminal MoE and Transformer-based MoE studies.
  • Training & Inference Categories: Emerging schemes comprise Dense-to-Sparse transition, Sparse-to-Dense conversion, and Expert Models Merging.These categories respectively begin with dense training, convert sparse models for hardware-friendly inference, or integrate multiple pretrained expert models.
  • Dense-to-Sparse: Dense-to-Sparse methods initialize or train dense models before progressively introducing sparse expert activation to reduce training overhead or improve efficiency.RMoE fine-tunes a pretrained dense model into MoE, while EvoMoE begins with one expert and progressively diversifies experts using DTS-Gate.
  • Dense-to-Sparse: Sparse upcycling reuses a pretrained dense checkpoint to initialize a sparse MoE, but gating-score normalization can help vision models while harming language models.Skywork-MoE reports that choosing upcycling or training from scratch depends on dense-checkpoint quality and MoE-specific training resources.
  • Dense-to-Sparse: DS-MoE combines dense training with sparse inference, while SMoE-Dropout progressively increases activated capacity during training to reduce overfitting.Other methods construct experts by partitioning FFN parameters or clustering FFN components before fine-tuning.
  • Sparse-to-Dense: Sparse-to-Dense distillation and compression target hardware-friendly deployment by gathering expert knowledge, pruning or quantizing experts, and removing unimportant experts.OneS retains 61.7% of MoE benefits on ImageNet and 88.2% on NLP datasets after knowledge gathering and distillation.
  • Expert Models Merging: Expert merging independently trains domain-specialized expert language models and combines them through ensembling or parameter averaging at inference.BTM trains expert LMs without shared parameters, with examples tailored to scientific or legal text.

4.5 Derivatives

MoE derivatives adapt conditional computation or expert specialization to broader architectures and tasks. Examples span wider Transformer models, prompt tuning, and lifelong learning.

  • MoE Variants: WideNet increases model width by replacing FFN layers with MoE layers while sharing trainable parameters across Transformer layers.The design preserves shared parameters across layers except for normalization-related components described in the survey.
  • Prompt Tuning: Sparse mixture-of-prompts applies gating to multiple short soft prompts, each specialized for a different subset of data.SMoP addresses inefficiencies associated with long soft prompts during prompt tuning.
  • Lifelong Learning: Lifelong-MoE extends the MoE framework to lifelong learning, supporting continuous learning from an ongoing data stream.The survey presents lifelong learning as an application of expert specialization and MoE routing.
  • Overall Trend: Across derivatives, researchers either preserve conditional computation through gating or combine expert specialization with task-specific paradigms.The survey identifies prompt tuning and lifelong learning as examples of MoE’s adaptation across domains.

5 SYSTEM DESIGN OF MIXTURE OF EXPERTS

MoE system design addresses the computation, communication, and storage challenges created by sparse, dynamic workloads, using expert parallelism alongside complementary distributed strategies. The survey reviews these challenges, mitigation techniques, hybrid parallelism, and open-source frameworks.

  • Parallelism: Expert parallelism dispatches partitioned tokens to experts with capacity limits and load balancing, enabling efficient scaling of MoE models.It became a fundamental strategy after GShard introduced parallel gating and expert computation.
  • Parallelism: Hybrid strategies combine expert parallelism with data, tensor, pipeline, or sequence parallelism to improve scalability and efficiency.Figure 8 illustrates data + expert + tensor, data + expert + pipeline, and expert + tensor configurations.
  • Computation: Load imbalance across devices creates synchronization overhead because execution waits for the most heavily loaded expert.The survey discusses optimized gating and related algorithmic strategies as responses.
  • Computation: Gate routing, input encoding, and output decoding add overhead through redundant computation and memory movement that hardware optimizes less effectively than expert computation.Systems such as DeepSpeed-MoE and FastMoE target these operations.
  • Communication: All-to-All communication can become the primary efficiency constraint, motivating hierarchical, topology-aware, and pipelined communication strategies.These methods reduce inter-node exchanges or overlap communication with expert computation.
  • Storage: Memory capacity remains restrictive because devices may need to accommodate many experts, especially during inference on edge devices.Storage methods offload inactive experts to CPU memory or SSDs, while prefetching and forecasting mitigate transfer costs.

6 APPLICATIONS OF MIXTURE OF EXPERTS

MoE architectures are applied across natural language processing, computer vision, recommender systems, and multimodal learning. These applications use sparse or specialized experts to support diverse tasks, modalities, or objectives.

  • Natural Language Processing: MoE supports natural language understanding and generation tasks including translation, question answering, code generation, and mathematical problem-solving.The survey discusses these integrations alongside algorithmic designs for LLMs.
  • Computer Vision: Vision MoE applies sparsely activated MLP experts within selected ViT blocks for image recognition.V-MoE rivals state-of-the-art networks while requiring substantially less inference computation.
  • Recommender System: Recommender-system MoE models address settings that must balance multiple objectives, such as immediate preferences and subsequent user satisfaction.The cited discussion also describes AutoML-based control over feature extraction and fusion.
  • Multimodal Applications: MoE is used in multimodal models because expert layers can learn distinct modality partitioning across image and text data.LIMoE combines contrastive loss with entropy-based regularization to address load balancing.

7 CHALLENGES & OPPORTUNITIES

The survey identifies challenges spanning training, architecture, systems, efficiency, generalization, interpretability, and expert collaboration. It presents these as opportunities for coordinated algorithmic, hardware, and application research.

  • Training Stability and Load Balancing: Sparse gating can destabilize training and produce imbalanced expert workloads, while auxiliary losses may still overlook token importance.The survey calls for improved regularization and gating algorithms.
  • Scalability and Communication Overhead: Increasing parameter counts intensify distributed communication bottlenecks, creating a trade-off between model complexity and communication overhead.Shared-expert approaches are discussed as a way to reduce transmitted data while maintaining performance.
  • Expert Specialization and Collaboration: Sparse top-k aggregation may miss relationships across the full expert set, making collaboration among specialized experts an unresolved challenge.Shared experts are used to capture common knowledge and reduce redundancy.
  • Sparse Activation and Computational Efficiency: Non-uniform sparse operations on hardware accelerators make theoretical computational savings difficult to realize in practice.The survey highlights hardware optimization as a needed research direction.
  • Generalization and Robustness: Sparse MoE models may overfit particular tasks or datasets, limiting generalization to unseen data and diverse input variations.Explored responses include dropout, token dropping, and multi-task instruction tuning.
  • Interpretability and Transparency: Dynamic routing to specialized experts complicates interpretability when understanding model decisions is essential.
  • Optimal Expert Architecture: The number and architecture of experts require balancing computational cost against specialization and redundancy.The survey motivates automated architecture search for MoE configurations.
  • Integration with Existing Frameworks: Adapting existing LLMs to MoE without training from scratch remains important for reducing resource consumption.The survey points to parameter-efficient fine-tuning combined with MoE frameworks.

8 CONCLUSION

The survey consolidates MoE research through a taxonomy covering algorithm design, system design, and applications, while collecting implementations, configurations, evaluations, challenges, and future directions. A resource repository supports continued dissemination of MoE advances.

  • Conclusion: The survey provides a systematic review organized around algorithm design, system design, and practical applications.It also includes open-source implementations, hyperparameter configurations, empirical assessments, and future research directions.
  • Conclusion: A dedicated resource repository is established to support ongoing updates and sharing of developments in MoE research.
Loading 2407.06204v3…