PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications

Siyuan Mu, Sen Lin

arXiv:2503.07137v4cs.LGcs.AI

TL;DR

Large AI models face high computational and deployment costs and difficulty fitting heterogeneous data, while existing MoE surveys are outdated or incomplete. This paper surveys MoE designs, algorithms, theory, and applications, highlighting efficiency and performance results such as Switch Transformer’s faster pre-training and MoE model compression.

  • Problem

    Existing MoE surveys are outdated, overly focused on basic designs, or insufficiently broad across machine learning paradigms, theory, and application domains.

  • Method

    The paper provides a comprehensive survey of MoE architecture and training designs, algorithmic applications, theoretical studies, computer vision, natural language processing, and future directions.

  • Results

    MoE studies report improved efficiency and performance, including 7 times faster pre-training than T5-Base for Switch Transformer and 3.7 times smaller MoE models with DeepSpeed-MoE.

  • Takeaways & Limitations

    MoE dynamically selects relevant experts and offers a framework for scaling models while addressing diverse data and task settings across machine learning applications.

Abstract

from arXiv · show

Artificial intelligence (AI) has achieved astonishing successes in many domains, especially with the recent breakthroughs in the development of foundational large models. These large models, leveraging their extensive training data, provide versatile solutions for a wide range of downstream tasks. However, as modern datasets become increasingly diverse and complex, the development of large AI models faces two major challenges: (1) the enormous consumption of computational resources and deployment difficulties, and (2) the difficulty in fitting heterogeneous and complex data, which limits the usability of the models. Mixture of Experts (MoE) models has recently attracted much attention in addressing these challenges, by dynamically selecting and activating the most relevant sub-models to process input data. It has been shown that MoEs can significantly improve model performance and efficiency with fewer resources, particularly excelling in handling large-scale, multimodal data. Given the tremendous potential MoE has demonstrated across various domains, it is urgent to provide a comprehensive summary of recent advancements of MoEs in many important fields. Existing surveys on MoE have their limitations, e.g., being outdated or lacking discussion on certain key areas, and we aim to address these gaps. In this paper, we first introduce the basic design of MoE, including gating functions, expert networks, routing mechanisms, training strategies, and system design. We then explore the algorithm design of MoE in important machine learning paradigms such as continual learning, meta-learning, multi-task learning, and reinforcement learning. Additionally, we summarize theoretical studies aimed at understanding MoE and review its applications in computer vision and natural language processing. Finally, we discuss promising future research directions.

I. INTRODUCTION

The paper surveys Mixture-of-Experts methods as a response to the computational and modeling challenges posed by increasingly diverse, complex datasets. It covers MoE foundations, algorithms, theory, and applications across computer vision and natural language processing.

  • Motivation: Modern datasets increase both the computational cost of large models and the difficulty of integrating heterogeneous knowledge.These datasets may contain multimodal data and intricate structures, creating challenges for training and deployment.
  • MoE approach: MoE dynamically activates input-relevant parameters, enabling expert specialization and expanded model capacity without proportional computational growth.This divide-and-conquer strategy differs from dense models that activate all parameters for every input.
  • Reported progress: 7 times faster pre-training than T5-Base and 4 times faster than T5-XXL at trillion-parameter scale are reported for Switch Transformer.The model also improved performance across all 101 languages in a multilingual setting.
  • Reported progress: Mixtral 8x7B activates 13 billion parameters per token while accessing 47 billion total parameters, illustrating MoE parameter efficiency.The passage presents this architecture as a way to control computational costs while accessing broader capacity.
  • Survey scope: The survey addresses gaps in earlier reviews by covering basic designs, machine-learning paradigms, theory, and applications in computer vision and natural language processing.Earlier surveys were described as outdated, basic-design-focused, or lacking broad coverage of theory and applications.
  • Survey organization: The paper organizes its coverage into MoE design, algorithmic applications, theoretical understanding, computer vision, natural language processing, and future directions.The algorithmic discussion includes continual, meta-, multitask, reinforcement, and federated learning.

II. BASIC DESIGNS OF MOE

The paper introduces MoE as a framework built from gating, expert networks, and routing, followed by training and system strategies. These components determine how inputs are assigned and how efficiency is optimized.

  • Core components: The basic MoE framework comprises gating functions, expert networks, and routing mechanisms.Gating and expert networks form the basic framework, while routing characterizes how gating handles input data.
  • Training and systems: Training strategies use loss functions and pipeline design to support proper MoE training.System designs are then discussed to optimize MoE efficiency from multiple perspectives.
  • Architecture overview: Figure 2 presents a simple schematic of a standard MoE architecture.The figure is used to support the section’s introduction to MoE’s general workflow.

A. Gating Function

The gating function acts as MoE’s router, assigning inputs to experts according to learned relevance scores. Designs range from simple linear softmax gating to cosine, distribution-based, and soft routing mechanisms.

  • The gating function determines how each input is allocated to designated experts, making its allocation strategy central to MoE effectiveness.
  • Linear Gating: Most MoE models use linear softmax gating because of its simplicity and effectiveness.
  • Linear Gating: TopK gating retains the k highest-scoring experts, sets other scores to negative infinity, and can add noise to encourage expert exploration.
  • Linear Gating: Applying TopK before softmax filters irrelevant experts early and reduces computation, but requires subsequent normalization because the resulting scores may not form a probability distribution.
  • Non-linear Gating: Cosine gating projects inputs into a hypersphere and compares them with learnable expert embeddings, while cosine routers show advantages for cross-domain generalization.
  • Non-linear Gating: Soft MoE computes token–expert weights and forms weighted input slots, avoiding discrete allocation and potential token dropping from load imbalance.

B. Expert Networks

MoE expert networks specialize in distinct knowledge domains while remaining flexibly integrated into existing architectures. Common designs replace Transformer FFNs, augment attention, or incorporate CNN-based experts.

  • Expert networks dynamically process inputs selected by the gate, enabling specialization across knowledge domains and improving performance, generalization, and efficiency.
  • In practice, experts are often integrated into one model by replacing selected layers with MoE layers, preserving architectural flexibility.
  • Transformer FFN Experts: Replacing a Transformer FFN with an MoE layer uses selected experts’ weighted outputs, with each expert represented by a two-layer FFN.
  • Transformer FFN Experts: FFN layers are attractive MoE targets because their higher sparsity and domain specificity make them more suitable for selective expert computation than self-attention layers.
  • Mixture-of-Attention: Mixture-of-attention selects TopK attention heads and combines their outputs, reducing computation because unselected heads need not process every input.
  • CNN Experts: CNN-based MoE combines gating kernels with multiple two-layer CNN experts, leveraging local feature extraction for refined task allocation.

C. Routing Strategy

Routing strategies determine the granularity and basis of expert selection, ranging from tokens and modalities to tasks and broader context. These choices trade flexibility, specialization, communication cost, and computational efficiency.

  • Routing level determines how frequently experts specialize, with the appropriate level depending on whether tasks emphasize global features or precise local perception.
  • Token-Level Routing: Token-level routing selects experts from token representations and includes text-token and image-patch variants.
  • Token-Level Routing: MaskMoE restricts infrequent tokens to one expert while allowing frequent tokens multiple visible experts to preserve training consistency and contextual diversity.
  • Token-Level Routing: Recurring token-level routing dynamically adjusts expert selection using sequence context, improving processing of complex linguistic structures and long-range dependencies.
  • Token-Level Routing: Batch Priority Routing sorts image patches by importance and can match dense-model performance while processing only 15%-30% of patches.
  • Modality- and Task-Level Routing: Modality-level routing assigns tokens according to modality, while task-level routing uses task identifiers to reduce interference between tasks.
  • Modality- and Task-Level Routing: Task-level routing loads only task-relevant experts at inference, reducing device communication costs and memory usage.
  • Other-Level Routing: Context-level routing uses global pooling, whereas attribute-level routing incorporates modality-related binary attributes to inform routing.

D. Training Strategies

Training sparse MoE models requires specialized objectives, expert-selection policies, and pipelines because dense-model training procedures do not directly transfer. These strategies target balanced utilization, adaptive computation, efficient resource allocation, and stable convergence.

  • Sparse activations make dense-model training procedures difficult to apply directly, motivating specialized methods for convergence, efficiency, and target-task performance.
  • Auxiliary Loss Function Design: Load-balancing auxiliary losses distribute inputs across experts to improve capacity utilization and stabilize training.
  • Auxiliary Loss Function Design: Model collapse occurs when a small subset of experts receives most inputs, leaving other experts underused.
  • Auxiliary Loss Function Design: Switch Transformer simplifies balancing objectives by trading a slight reduction in accuracy for improved efficiency.
  • Auxiliary Loss Function Design: Uniform routing minimizes Switch’s auxiliary loss when each expert receives equal proportions of tokens and routing probability.
  • Auxiliary Loss Function Design: Expert capacity limits how many inputs an expert processes: insufficient capacity can skip inputs, whereas excess capacity wastes computation and memory.
  • Expert Selection: TopK selection expands model capacity without proportionally increasing computation, while Top-P adaptively activates experts until cumulative probability exceeds a threshold.
  • Pipeline Design: MoE pipelines optimize dynamic data allocation and resource use; BPR prioritizes important samples, while RMoE separates expert weights into pretrained core and residual components.

E. System Design

MoE system design must address computation, communication, and memory challenges caused by dynamic expert selection and large parameter counts. Proposed strategies include parallelization, on-demand loading, parameter migration, and hierarchical storage.

  • Computation: Dynamic expert selection creates non-uniform workloads, synchronization overhead, and idle computation in distributed MoE systems.Larger model capacity further increases computational complexity and the need for improved computation strategies.
  • Computation: Parallelization strategies include data, expert, model, and tensor parallelism, with unified layouts supporting adaptation to dynamic load changes.Different strategies impose different requirements on data and parameter distribution.
  • Communication: All-to-All routing demands substantial and fluctuating communication bandwidth because expert assignments vary across iterations.This dynamic communication pattern is difficult to predict and optimize.
  • Communication: On-demand expert loading reduces external DRAM access, while assigning experts to separate compute units enables task-level parallel processing.These strategies can reduce memory-bandwidth pressure and increase throughput.
  • Memory: MoE parameter volume can exceed a single device’s storage capacity, and dynamic expert selection complicates memory access patterns.Parameter migration distributes storage pressure across devices, while DeepSpeed-MoE uses hierarchical storage management.

III. ALGORITHMS

The survey reviews MoE algorithm designs across continual learning, meta-learning, domain adaptation, and reinforcement learning. Across these paradigms, MoE is used to specialize models, route inputs or tasks, and address forgetting, heterogeneity, and limited adaptability.

  • Continual Learning: MoE addresses catastrophic forgetting in continual learning by dynamically selecting sub-models for diverse tasks without substantial computational overhead.Continual learning adapts to new data without access to previous data, creating the forgetting challenge.
  • Continual Learning: Continual-learning methods include task-free expert mixtures, new experts for new distributions, task-specific routers, and activation-sparsity gating.Examples include CN-DPM, Lifelong-MoE, MoE-Adapters, and MoASE.
  • Continual Learning: Continual-learning research also uses expert ensembles, including one expert per task or multiple small networks trained across tasks in parallel.Some reported ensemble designs outperform a single larger model.
  • Meta-Learning: MoE-NPs model complex tasks with multiple latent variables and discrete assignment variables that select neural-process experts for each prediction point.An evidence lower bound trains the model for few-shot supervised learning and meta-reinforcement learning.
  • Meta-Learning: MixER applies Top-1 MoE to contextual meta-learning to address environments that do not follow a shared task distribution.Other meta-learning applications include multi-source domain adaptation and domain-generalizable person re-identification.
  • Meta-Learning: MoE-based approaches in meta-reinforcement learning enhance representational capacity and address differences across tasks and testing environments.The reviewed GMA protocol work uses MoE to reduce overfitting to specific tasks.

C. Multi-task Learning

MoE complements multi-task learning by routing inputs or tasks to specialized experts, reducing interference and handling complex task relationships. Applications span sequential data, recommendation, digital rock physics, and machine vision.

  • Multi-task Learning: MoE naturally decouples complex tasks by selecting suitable sub-models according to input characteristics.This differs from conventional parameter or representation sharing in multi-task learning.
  • Multi-task Learning: MMoE assigns each task a separate gating network, allowing task-specific expert selection and reducing interference between dissimilar tasks.The architecture decouples task processing through distinct gate functions.
  • Multi-task Learning: MOOR uses multiple experts to generate different representations, encouraging diversity and addressing representation collapse in multi-task reinforcement learning.Each expert handles different aspects of the task.
  • Multi-task Learning: MoE-based recommendation methods address parameter-sharing and resource-utilization trade-offs, while task-intensive experts generated by hyper-networks alleviate gradient conflicts.Some MoE recommendation systems still face unstable training or resource waste.
  • Multi-task Learning: Multi-domain multi-task recommendation decomposes learning into shared, domain-specific, and task-specific expert modules.These modules capture cross-domain knowledge, domain-specific preferences, and task-specific preferences.
  • Multi-task Learning: In machine vision, MoE aggregates similar task models to improve training and inference efficiency while preserving individual-task performance.MoE can also handle dependencies between tasks and serve as a finer-grained decoder for task-specific features.
  • Multi-task Learning: Replacing MoE experts with LSTMs improves handling of sequential user-activity data, while MMoE is applied to digital rock physics analysis.These examples extend MoE-based multi-task learning beyond standard architectures.

D. Reinforcement learning

The survey reviews MoE in reinforcement learning as a modular approach for complex environments, adaptive skills, multimodal policies, and reduced action spaces. Methods span model-based, actor-critic, value-based, offline, multi-agent, and imitation learning.

  • Reinforcement Learning: MoE in reinforcement learning addresses computational inefficiency and limited adaptability in high-dimensional, dynamic environments.RL learns policies through interaction with environments in applications including robotics, games, autonomous driving, and healthcare.
  • Modular RL: Modular RL decomposes complex problems into expert modules, each specializing in an aspect such as dynamics modeling or control.MMRL uses multiple dynamics-model and controller modules for nonlinear and nonstationary control tasks.
  • Modular RL: MACE uses multiple actor-critic pairs for terrain-adaptive locomotion, with each pair specializing in particular motion aspects.The approach is reported to learn faster than a standard single actor-critic pair.
  • Policy Design: MoE policy methods select expert actions using prototypical-state distances or represent multimodal policies with Gaussian mixture models.These designs target interpretable policy construction and optimization of nondifferentiable mixtures.
  • Deep Reinforcement Learning: Soft MoE layers can significantly improve several deep RL algorithms across online, offline, and low-interaction settings.MoE-based Transformer policies are also used for multi-task robot learning from good and sub-optimal demonstrations.
  • Applications: Hierarchical MoE language models reduce offline dialogue-management RL action spaces while supporting discourse with diverse intentions.Other reviewed applications include multi-agent RL, situation-specific modular policies, and inverse constrained RL.

IV. THEORY

The survey reviews theoretical progress on MoE, spanning approximation, estimation, deep-learning behavior, multilevel data, and continual learning. It emphasizes that theory remains less developed for deep neural networks and classification than for simpler models.

  • Theory foundations: Existing MoE theory primarily studies approximation capacity and convergence rates in hierarchical models with exponential-family or generalized-linear experts.These results characterize functional approximation and estimation behavior in relatively simple model classes.
  • Classification theory: Classification theory is less explored, with work establishing density- and parameter-estimation convergence rates for softmax-gated MoE models.Modified softmax gating addresses slower estimation when some expert parameters vanish.
  • Deep-learning theory: 87.5% is the accuracy ceiling shown for a single two-layer CNN expert on a clustered binary-classification dataset, whereas nonlinear MoE can perform better through learned cluster-based routing.Sufficient expert exploration allows the router to learn cluster structure and route inputs to suitable experts.
  • Multilevel data: MeMoE introduces random effects into the gating function to model dependencies in multilevel data and is proven dense under weak convergence for continuous mixed-effects models.For hierarchical data, the result extends to approximating dependency structures between factor levels.
  • Continual learning: In continual learning, sufficiently trained MoE experts can specialize across tasks, reducing interference and forgetting under over-parameterized linear-regression assumptions.The gating-network update must be terminated timely when learning a new task to maintain stability.
  • Applications: The survey also positions MoE research within practical applications including computer vision and natural language processing, alongside theoretical developments across machine-learning paradigms.Its application review covers tasks ranging from image classification to machine translation.

A. Computer vision

In computer vision, MoE is reviewed as a way to improve scalability, efficiency, specialization, and adaptability across classification, detection, segmentation, and generation. The surveyed methods use sparse activation, specialized experts, and redesigned gating or calibration mechanisms for task-specific challenges.

  • Image Classification: V-MoE replaces Vision Transformer MLP layers with sparse MoE layers, scaling visual models to 15 billion parameters while improving image-classification efficiency.The approach demonstrates that sparse expert layers can increase capacity without activating all parameters for every input.
  • Image Classification: Shared experts handle common classification knowledge while specialized experts process specific knowledge, but increasing expert count does not necessarily improve routing performance.The surveyed analysis links routing behavior to both the number of experts and the size of L.
  • Image Classification: Soft MoE replaces hard token assignment with weighted combinations of tokens, retaining scalable capacity while improving training stability and reducing inference time.Each expert processes a different weighted combination rather than receiving discrete token assignments.
  • Object Detection: MoCaE finds that naïvely combining object detectors can hurt performance through unfair competition, while Early and Late Calibration produce gains over single models and deep ensembles.Calibration adjusts detector confidence levels to better reflect their detection performance.
  • Semantic Segmentation: MoE alleviates hardware pressure in semantic segmentation through sparse activation and divide-and-conquer processing, supporting practical high-resolution image analysis.The survey also reports excellent segmentation performance in studies using MoE.
  • Semantic Segmentation: MoE-based segmentation for autonomous driving offers transparency by enabling analysis of individual expert outputs in addition to the overall model output.This supports inspection of how experts contribute to scenario understanding and algorithm design.
  • Image Generation: RAPHAEL separates spatial and temporal MoE layers so experts specialize in text concepts across image regions and diffusion time steps.Spatial routing focuses on region-specific visual features, while temporal routing processes concepts during generation.
  • Image Generation: MEGAN uses multiple generator experts for different modal data subsets, producing more diverse and higher-quality images than a single generator design.The method targets GAN difficulty in learning complex multimodal distributions.

B. Natural Language Processing

In NLP, MoE models use sparse, input-dependent expert selection to improve efficiency, specialization, adaptability, and performance across language, multilingual, multitask, and multimodal tasks.

  • Overview: MoE reduces computational demands by activating task- or domain-relevant experts instead of the full model.The architecture uses sparse activation and expert allocation to support multi-task, low-resource, and complex reasoning scenarios.
  • Natural Language Understanding: GLaM improves training and inference efficiency while maintaining or improving performance for natural language understanding.The reported benefits include reduced computational resources and energy consumption.
  • Natural Language Understanding: MoE-LPR expands language capability without original-language data, then uses limited review data and language-prior routing to preserve original-language performance.Its reported results indicate scalability and stability for multilingual understanding.
  • Natural Language Understanding: MoE-SLU weights and averages additional transcripts to reduce the impact of ASR errors on spoken language understanding.It achieves state-of-the-art performance on three benchmark SLU datasets, with further gains from regularized predictions.
  • Natural Language Understanding: MT-TaG routes inputs through task-aware sparse experts to address interference among NLU tasks, while MoPE-BAF improves few-shot multimodal semantic tasks through prompt experts and block-aware fusion.MoPE-BAF uses text, image, and unified prompt experts to support cross-modal interaction.
  • Natural Language Generation and Machine Translation: MoE applications in generation and translation use multiple specialized models or conditional routing to improve text quality, translation adaptability, and resource efficiency.Examples include GAN-based collaborative generation, retrieval-weighted generation, multilingual translation models, and deployment optimizations such as dynamic gating, expert caching, and load balancing.
  • Multimodal NLP: LIMoE aligns image and text representations through contrastive learning, while LLaVA-MoLE selects LoRA experts per token to mitigate mixed-domain instruction conflicts.Hunyuan is reported as a Chinese multimodal MoE model with a SuperClue-V score of 71.95.

VI. FUTURE DIRECTIONS

Future MoE research should address training stability and uneven expert utilization, which can cause imbalanced training and potential model collapse. Proposed directions include better load balancing, capacity adjustment, regularization, and theoretical analysis.

  • Training stability and load balancing: Training stability and load balancing among experts are identified as critical challenges for MoE models.Dynamic expert selection can send substantially more data to some experts than others.
  • Training stability and load balancing: Uneven expert utilization may lead to imbalanced training and potential model collapse.The proposed remedies include adaptive load balancing, dynamic expert capacity adjustment, and regularization against over-reliance on specific experts.
  • Training stability and load balancing: Theoretical studies of convergence under different load-balancing strategies could inform more stable MoE training algorithms.The passage also calls for further optimization of training strategies for sparse MoE computation.

B. Training and system efficiency

The survey identifies unresolved efficiency, architecture, interpretability, and cross-paradigm challenges for MoE models. It organizes these issues alongside opportunities for broader applications and systematic future research.

  • Efficiency: High latency remains a concern for MoE training and inference, motivating hardware-software co-design, memory management, communication reduction, and parallel processing.These measures are presented as ways to improve scalability for conditional computation.
  • Architecture design: Current methods often choose the number and specialization of experts heuristically or statically, which may not suit every task.The survey suggests meta-learning or reinforcement learning for dynamically adjusting architecture based on task complexity and data distribution.
  • Theory and interpretability: MoE routing decisions and clustering mechanisms remain insufficiently understood, requiring stronger theory to guide more robust gating functions and expert networks.The survey links this theoretical gap to interpretability and model reliability.
  • Algorithm design: Combining MoE with contrastive, transfer, self-supervised, or federated learning remains underexplored, including expert assignment for heterogeneous client data.Federated learning is presented as one example where experts could handle differing client distributions.
  • Applications: MoE applications beyond NLP remain underexplored, with opportunities in computer vision, multimodal learning, healthcare, robotics, autonomous systems, education, finance, and recommendation.The survey also systematically reviews MoE in computer vision and natural language processing.
Loading 2503.07137v4…