PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Gökdeniz Gülmez

arXiv:2603.01697v1cs.LGcs.AI

TL;DR

Standard MoE designs constrain routing and expert allocation. DynaMoE varies active experts with input complexity and schedules capacity across layers. Optimal schedules depend on task and model scale: descending schedules improve image-classification accuracy by up to 5.47%, while language-modeling schedules vary by model size.

  • Problem

    Standard MoE designs impose rigid routing and capacity allocation, with fixed Top-K routing treating tokens uniformly despite differing input complexity and computational needs.

    Source: §1 · pp. 1–2

  • Method

    DynaMoE varies the number of active experts with input complexity and uses schedules to allocate expert counts across network layers.

    Source: §3.2 · p. 3, §3.3 · p. 4

  • Results

    Optimal schedules depend on task and model scale: descending schedules improve image-classification accuracy by up to 5.47%, while scale-specific schedules outperform naive uniform allocation in language modeling.

    Source: §8 · p. 22

  • Takeaways & Limitations

    Expert distribution matters and depends on task structure, challenging uniform allocation as a general MoE design assumption.

    Source: §8 · p. 22

    Language-modeling experiments used only 1,000 training samples, so large-scale pretraining on standard corpora is needed before drawing conclusions about language-model scheduling.

    Source: §7.10 · p. 21

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.

1 Introduction

DynaMoE addresses fixed routing and uniform layer-capacity assumptions in MoE models with dynamic token-level activation and layer-wise expert scheduling. The paper also analyzes dynamic routing theoretically and evaluates the framework across image classification and language modeling.

1 Introduction

  • MoE models use conditional computation to route tokens to specialized experts, supporting large parameter counts while keeping computational costs manageable.The introduction cites Shazeer et al. and Fedus et al. as examples.Source: §1 · p. 1
  • Standard MoE fixes the number of active experts per token and assigns identical expert counts to every layer, disregarding input complexity and depth.The paper argues these constraints lack theoretical motivation and may limit expressivity and efficiency.Source: §1 · pp. 1–2

1.1 Contributions

  • DynaMoE varies the number of active experts per token based on input characteristics, with theoretical guarantees on computational efficiency.Source: §1.1 · p. 2
  • DynaMoE formalizes six strategies for distributing expert capacity across network depth and empirically validates their effectiveness.Source: §1.1 · p. 2
  • The paper derives bounds on dynamic routing’s expressivity gains and analyzes its impact on gradient variance.Source: §1.1 · p. 2
  • Experiments cover MNIST, Fashion-MNIST, and CIFAR-10 classification plus next-token prediction, identifying task-dependent capacity patterns and improvements over static baselines.Source: §1.1 · p. 2

2 Related Work

Prior work established MoE routing, dynamic computation, and load-balancing methods, while DynaMoE targets variable expert counts per token and systematic layer-wise expert scheduling.

2.1 Mixture-of-Experts Architectures

  • MoE research progressed from foundational architectures to scalable GShard, top-1 Switch routing, expert choice, and work on routing stability, balancing, and specialization.Source: §2.1 · p. 2

2.2 Dynamic Neural Networks

  • Prior dynamic-network methods use early exiting, adaptive width, dynamic depth, or vision-transformer token routing, but mainly alter the forward pass rather than fundamental routing.Source: §2.2 · p. 2

2.3 Load Balancing and Routing Control

  • Load-balancing methods include auxiliary utilization loss, capacity factors, top-2 overflow protection, and expert-choice selection, whereas DynaMoE’s percentile threshold uses neither capacity constraints nor auxiliary balancing losses.Source: §2.3 · p. 2

2.4 Dynamic and Learned Routing

  • DynaMoE uses a percentile threshold on gate values for differentiable variable-K selection without discrete optimization, while learned thresholds and per-layer learned K values remain unexplored.Related work studied stochastic and learned routing and compute allocation proportional to input complexity.Source: §2.4 · pp. 2–3

2.5 Adaptive Capacity Allocation

  • DynaMoE is the first to systematically study predefined layer-wise expert-count schedules in a unified MoE framework, linking scheduling choices to information-theoretic and optimization-theoretic principles.Prior evidence includes MoEfication’s finding that converted dense FFN layers need varying expert counts, and CNN width-scaling laws supporting increasing widths with depth.Source: §2.5 · p. 3

3 Methodology

DynaMoE combines input-dependent expert selection with layer-wise expert schedules, while its implementation uses noisy training-time routing and leaves capacity balancing as an acknowledged limitation.

3.1 Preliminaries: Standard MoE

  • A standard MoE layer has N feed-forward experts, and Top-K routing selects the K experts with the highest gate values for each input token.Source: §3.1 · p. 3

3.2 Dynamic Token-Level Routing

  • DynaMoE makes the active expert count K input-dependent for a layer with N experts; with percentile threshold τ, Kmin = 1 and Kmax = ⌈(1 −τ) · N⌉.T is a temperature parameter for stable softmax computation.Source: §3.2 · p. 3

3.2.1 Training Stability

  • Training adds Gaussian gate noise with σ = 0.1 to explore expert combinations, while temperature scaling supports stability.Source: §3.2.1 · p. 4

3.2.2 Capacity Control and Load Balancing

  • DynaMoE imposes no explicit capacity constraints or auxiliary balancing loss, processes all routed tokens, and has worst-case per-token cost O(d^2·Kmax).Unlike capacity-factor methods, auxiliary balancing losses, or expert-choice routing, its percentile thresholds create heterogeneous expert loads without explicit balancing.Source: §3.2.2 · p. 4
  • Persistent over-selection of popular experts may cause load imbalance; no catastrophic collapse appeared in small-scale experiments, but larger deployments may fare worse.Because schedule comparisons omit balancing losses, the results are not directly comparable to published MoE systems using them; balancing mechanisms are left for future work.Source: §3.2.2 · p. 4

3.3 Layer-Wise Expert Distribution

  • Layer-wise schedules map each layer to an expert count and include descending, ascending, pyramid-up, pyramid-down, and oscillating wave patterns.Wave-Up follows the inverse wave pattern; the schedules vary capacity placement across network depth.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

3.3.5 5. Wave Schedules

  • Wave schedules use complex oscillating capacity patterns, and Wave-Up follows the inverse pattern.Source: §3.3.5 · p. 5

3.4 Architecture Overview

  • The architecture applies an input projection, residual DynaMoE layers, layer normalization, and a classification head, with each layer’s expert count set by its schedule.Source: §3.4 · p. 5

4 Theoretical Analysis

The analysis characterizes dynamic routing’s potential gains in activation-pattern diversity and its compute and gradient-variance implications. It also formalizes capacity scheduling and gives a condition under which descending allocation can outperform uniform allocation.

4.1 Expressivity Analysis

  • Dynamic routing permits activation of 1 to Kmax experts rather than only fixed K, yielding strictly more routing patterns when Kmax > K.The result concerns combinatorial routing-pattern diversity, not function-space volume; the paper says the larger reachable pattern set expands implementable piecewise-linear functions for fixed expert weights.Source: §4.1 · p. 6

4.2 Computational Complexity

  • For τ = 0.7 and S(ℓ) = 8, expected active experts are approximately 2.4, versus fixed K = 2 in standard approaches.Per-token expert FLOPs depend on the selected experts evaluated, while total parameters are a storage cost; the paper’s efficiency comparisons concern active-expert FLOPs, not wall-clock throughput.Source: §4.2 · pp. 6–7

4.3 Gradient Variance Reduction

  • Under A1–A3, the gradient-variance bound is non-vacuous only when dynamic routing has greater entropy than fixed routing.The paper characterizes this as potential variance reduction under favorable routing distributions, not a tight quantitative guarantee; it does not claim numerical verification of the bound.Source: §4.3 · p. 7

4.4 Optimal Expert Distribution

  • The scheduling problem allocates layer-wise expert capacity to minimize expected loss under a model-capacity constraint; descending allocation has lower expected loss than uniform when early layers have higher loss curvature.The stated optimality is conditional on that early-layer curvature condition.Source: §4.4 · p. 7

5 Experimental Setup

The setup evaluates image classification and language modeling across four datasets and four model sizes, using specified training controls and task-specific metrics. Attention-related probes are reserved for future Transformer-based evaluation.

5.1 Datasets

  • Experiments cover MNIST, Fashion-MNIST, CIFAR-10, and Recycling-the-Web-1k for image classification and next-token prediction.Recycling-the-Web-1k uses 1,000 GPT-2-tokenized samples, a 50,257-token vocabulary, and sequence length 128; the image datasets each have 10 classes.Source: §5.1 · pp. 7–8

5.2 Model Configurations

  • The study evaluates four model sizes to examine scaling behavior.Table 1 is identified as the model-size configuration table, but its configuration values are not provided in the supplied passages.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 Training Configuration

  • Training uses AdamW, cosine-annealed learning rate, dataset-specific batch sizes and epochs, and dynamic-routing temperature and noise.AdamW uses β1 = 0.9, β2 = 0.999, and weight decay 10^-4; learning rate is 10^-3, temperature T = 0.5, and training noise σ = 0.1. Batch size is 256 for image tasks and 32 for Recycling-the-Web; training lasts 20 epochs for MNIST/Fashion-MNIST and 25 for CIFAR-10.Source: §5.3 · p. 8

5.4 Evaluation Metrics

  • Evaluation measures task accuracy, language-model perplexity, parameters per accuracy, convergence speed, and expert-usage entropy.Convergence is measured as epochs to reach 95% of final accuracy; language-model perplexity is PPL = exp(L), where L is cross-entropy loss.Source: §5.4 · p. 8
  • Because the tested DynaMoE architecture is an MLP without self-attention, attention and superposition probes were not computed and are proposed for Transformer extensions.The proposed probes include attention entropy, effective attention distance, head specialization, and a superposition pressure proxy; they would be analyzed alongside expert-usage entropy.Source: §5.4 · pp. 8–9

6 Results

Results show that descending expert schedules perform well on image-classification tasks, while language-modeling schedule performance varies with model size. The language-modeling results are limited by the small corpus and should be interpreted as a pilot feasibility study.

6.1 Expert Schedule Ablation

  • 92.68% accuracy: the descending schedule beats the MLP baseline by 3.26% and uniform MoE by 1.33% on MNIST, reaching 95% of final accuracy in 6 epochs.Figure 3 shows descending schedules concentrating activation in early layers, unlike uniform and ascending schedules.Source: Table 2 · p. 9, §6.1 · p. 9, Figure 3 · p. 9

6.2 Expert Count Sensitivity

  • E8–1 achieves the best accuracy-efficiency tradeoff; E16–1 shows diminishing returns, while E8–4 reduces flexibility and performance.The sensitivity analysis concerns MNIST with the Small model and descending schedule.Source: §6.2 · p. 9, Table 3 · p. 10

6.5 Cross-Dataset Generalization

  • +5.47% on CIFAR-10: the descending schedule shows consistent improvements across datasets, with larger gains on more complex tasks.Table 4 evaluates cross-dataset performance for the Small model over 20 epochs.Source: Table 4 · p. 10, §6.5 · p. 10

6.6 Language Modeling: Next Token Prediction

  • 1,000 training samples: the Recycling-the-Web language-modeling results are underfitted pilot evidence, not evidence of competitive language modeling.Small and Medium variants achieve near-identical next-token accuracy of ≈5.05%, close to the ≈2% unigram baseline; standard benchmarks use tens of millions to billions of tokens.Source: §6.6 · pp. 10–11

6.4 Scaling Analysis

  • +1.6% for Tiny and +3.3% for Small: descending-schedule DynaMoE outperforms baselines across model scales, with a widening gap for larger models.Figure 5 shows accuracy scaling across model sizes on MNIST.Source: §6.4 · p. 10, Figure 5 · p. 11

6.6.1 Key Findings for Language Modeling

  • 1011.80 best DynaMoE PPL: descending is best for Tiny, ascending for Small (2308.29), and uniform for Medium (2383.89).For Small, ascending marginally beats the MLP baseline (2311.02); for Medium, uniform improves PPL over MLP (2468.16) by 3.4%.Source: §6.6.1 · p. 12, Figure 6 · p. 12
  • 4940.24 and 6401.43 final PPL: Small descending and uniform schedules overfit after best validation PPL occurs in early-to-middle epochs; Medium shows the same pattern.For Medium, final PPL reaches 4719.99 for descending and 3257.70 for uniform. For Small and Medium, best validation accuracies are comparable at 0.0505, making PPL the primary differentiator.Source: §6.6.1 · p. 12, Figure 7 · p. 13

6.7 Expert Utilization Analysis

  • 3.2 experts in Layer 1 versus 1.2 in Layer 4: descending schedules concentrate computation in early layers.Figure 9 visualizes expert usage patterns across layers.Source: §6.7 · p. 12

7 Analysis and Discussion

This section analyzes explanations for the observed hierarchy of descending DynaMoE, uniform DynaMoE, and dense MLP, and for why effective expert schedules depend on task and depth-wise representational demands. It also examines dynamic routing’s effects on computation and training, offers practical recommendations, and identifies limitations.

7.1.1 Theory 1: Representational Entropy Collapse

  • As representations lose entropy with depth, early layers benefit from multiple experts for diverse inputs, while a single expert suffices for collapsed, class-discriminative features.The account explains why a descending schedule avoids maintaining excess expert capacity in deep layers.Source: §7.1.1 · pp. 13–14

7.1.2 Theory 2: Loss Landscape Curvature and Piecewise-Linear Capacity

  • Because loss curvature is highest early, descending schedules place the greatest piecewise-linear approximation capacity where the effective loss landscape is most nonlinear.Uniform schedules spend capacity in deeper, nearly linear mappings, while ascending schedules invert this alignment.Source: §7.1.2 · pp. 14–15

7.1.3 Theory 3: Kolmogorov Complexity Matching

  • The Kolmogorov-complexity account argues that required transformations simplify with depth, so expert libraries should shrink from rich input processing to a single output-stage program.This matches a large expert library to high-complexity input transformations and one expert to the final linear projection.Source: §7.1.3 · p. 15

7.1.4 Theory 4: Gradient Signal Propagation and Variance Reduction

  • A descending schedule provides many independent early-layer gradient pathways, reducing gradient-estimate noise, whereas an ascending schedule creates an early gradient bottleneck.The variance reduction is greatest where the expert count is largest.Source: §7.1.4 · p. 15

7.1.5 Theory 5: Expert Co-adaptation Prevention and Ensemble Diversity

  • Dynamic routing promotes expert specialization without co-adaptation, with the greatest diversity benefit in heterogeneous early layers; a single final-layer expert also regularizes output representations.The paper attributes the optimal Nmin = 1 to avoiding unnecessary late-stage co-adaptation pressure.Source: §7.1.5 · p. 16

7.2 The Representational Diversity-Convergence Principle: A Unified Theory

  • The Representational Diversity-Convergence principle holds that layer-wise expert counts should track representational diversity, which generally falls with depth in image classification but may grow or remain flat in language modeling.This predicts descending schedules for image classification and ascending or pyramid schedules for language modeling, and suggests matching schedules to measured or estimated diversity profiles.Source: §7.2 · p. 16

7.3 Why Uniform MoE Occupies the Second Position

  • Uniform MoE beats the dense MLP but trails descending by 1.33% to 2.67% across datasets, because it wastes capacity deep and under-provisions diverse early layers.It retains conditional computation and dynamic-routing benefits, but equal expert counts misalign capacity with the diversity profile.Source: §7.3 · pp. 16–17

7.4 Why the Dense MLP Baseline Underperforms

  • The dense MLP trails MoE by 3.26% on MNIST, 4.19% on Fashion-MNIST, and 5.47% on CIFAR-10, with the gap growing as task difficulty increases.Its shared, fixed-capacity computation lacks expert specialization and adaptive allocation, limiting performance on difficult inputs.Source: §7.4 · pp. 17–18
  • Even the worst-performing MoE schedule, ascending, consistently outperforms the best-performing dense MLP configuration in the reported experiments.The discussion attributes the MLP's underperformance to forced generalism and parameter interference, fixed computation, slow and unstable optimization, and lack of load balancing; MoE addresses these mechanisms.Source: §7.4 · pp. 18–19

7.5 Task-Dependent Schedule Selection

  • Schedule selection should match the task's representational-diversity profile: descending for image classification and ascending or pyramid for language modeling.Image diversity is described as decreasing with depth, while language-modeling diversity is flat or increasing; the profile can be estimated using pretrained activation entropy or Jacobian norms without full MoE training.Source: §7.5 · p. 19

7.6 Attention—MoE Coupling in Transformer LLMs

  • For Transformer MoE, FFN-side diversity alone cannot determine the optimal schedule because attention and expert dynamics jointly shape representational complexity.Deeper attention may increase contextual mixing, shifting the preferred allocation from descending toward ascending or pyramid; expert density should match post-attention diversity across depth.Source: §7.6 · p. 19

7.7 Superposition Theory and Expert Decoupling

  • Expert schedules can also match depth-wise superposition pressure, allocating more experts where feature overlap risks destructive interference.The discussion associates higher pressure with early image-classification layers, favoring descending schedules, and potentially deeper autoregressive language-model layers, favoring ascending or pyramid schedules.Source: §7.7 · pp. 19–20

7.8 Dynamic Routing Benefits

  • Dynamic routing increases expert-usage entropy by 0.34 bits over fixed Top-2, indicating better specialization.It activates fewer experts for simple inputs and full capacity for complex ones, while mitigating co-adaptation and expert collapse. Figure 10 reports smooth convergence, with descending schedules converging fastest initially for Tiny and Small models and ascending schedules more stable for Medium models.Source: §7.8 · p. 20, Figure 10 · p. 20

7.9 Practical Recommendations

  • The practical guidance recommends descending schedules for spatial tasks, ascending or pyramid schedules for sequential tasks, and dynamic-routing settings Nmax = 8, Nmin = 1, τ = 0.7, and T = 0.5.It also recommends monitoring both best and final validation metrics to detect overfitting.Source: §7.5 · p. 19, §7.9 · pp. 20–21

7.10 Limitations

  • The evidence is limited by pilot-scale language modeling, omitted standard MoE and load-balancing baselines, and comparisons that do not control compute consistently.Language modeling used only 1,000 training samples, with perplexities of 1,000–2,500 and about 5% per-token accuracy; the paper calls for larger-scale pretraining, standard balancing comparisons, and FLOPs-matched or accuracy-versus-active-compute evaluations.Source: §7.10 · p. 21
  • Further limitations are the narrow image-task scope, hand-designed schedules, unevaluated attention probes, and qualitative rather than formal theoretical claims.The paper calls for validation on ImageNet-scale tasks and Transformer backbones, learned or searched schedules, empirical evaluation of attention probes, and formal analysis of its explanatory hypotheses.Source: §7.10 · p. 21

8 Conclusion

DynaMoE combines input-adaptive token routing with layer-wise expert scheduling. Its conclusion emphasizes task- and scale-dependent schedules: descending allocation benefits image classification, while language-modeling schedules vary by model size.

8 Conclusion

  • DynaMoE introduces input-complexity-adaptive expert activation and six strategies for distributing expert capacity across network depth.The framework is presented as a Mixture-of-Experts design with dynamic token-level routing and layer-wise expert distribution.Source: §8 · p. 21
  • The paper’s theoretical analysis establishes expressivity gains and gradient variance reduction, alongside empirical validation on image-classification and language-modeling tasks.Source: §8 · p. 21
  • Small ascending schedules reach best overall PPL 2308.29 versus MLP 2311.02 (0.1% better), while Medium uniform achieves PPL 2383.89 versus MLP 2468.16 (3.4% lower).Appropriately scheduled DynaMoE matches or exceeds MLP baselines across model scales.Source: §8 · p. 22
  • Descending schedules improve CIFAR-10 accuracy by up to 5.47%, while optimal language schedules vary by scale: descending for Tiny, ascending for Small, and uniform for Medium.The paper reports these schedules outperforming naive uniform allocation in their respective regimes.Source: §8 · pp. 21–22
  • Future work should explore learned scheduling, application to transformer architectures, and scaling to billion-parameter models.The authors present DynaMoE as a foundation for adaptive computation in neural networks.Source: §8 · p. 22

A Additional Experimental Results · A.1 Training Curves · A.2 Detailed Expert Statistics

The appendix presents a training-convergence comparison across schedules and per-layer expert statistics for a descending schedule.

A.1 Training Curves

  • Figure 11 compares training convergence across schedules, but the supplied passages do not report which schedule converges best.The passages provide no curve values or schedule ranking.Source: Figure 11 · p. 24, §A.1 · p. 24

A.2 Detailed Expert Statistics

  • Table 6 provides per-layer expert distribution statistics for the descending schedule with Nmax = 8 and Nmin = 1.The supplied table caption gives the schedule and capacity bounds, but no per-layer values.Source: Table 6 · p. 24, §A.2 · p. 24

B Proof of Theorem 4.1

The proof compares the number of token activation patterns under dynamic and fixed Top-K routing to derive an expressivity ratio. Dynamic routing allows the active-expert count to vary with the threshold-defined capacity bounds.

B Proof of Theorem 4.1

  • Dynamic routing permits activation counts from Kmin = 1 through Kmax = ⌈(1 − τ)N⌉ for a layer with N experts.Source: B Proof of Theorem 4.1 · p. 24
  • The proof contrasts dynamic pattern counts with the fixed Top-K count, then bounds their ratio using Stirling’s approximation.Source: B Proof of Theorem 4.1 · p. 24

C Implementation Details · C.1 Pseudocode · C.2 Hyperparameter Sensitivity

C.1 outlines DynaMoE’s layer forward pass, using threshold-based expert selection with a fallback when no expert is selected. C.2 identifies a key-hyperparameter sensitivity analysis on MNIST.

C.1 Pseudocode

  • Algorithm 1 presents the DynaMoE layer computation as a forward pass.The pseudocode takes an input, expert count, threshold, and temperature as inputs.Source: §C.1 · pp. 24–25
  • The layer computes gate values with softmax and adds Gaussian noise with standard deviation 0.01 during training.Noise is added only when training.Source: §C.1 · p. 25
  • A percentile threshold selects experts whose gate values exceed the threshold.The threshold is computed as θ = percentileτ(g).Source: §C.1 · p. 25
  • If thresholding selects no experts, the layer selects the expert with the highest gate value.This fallback ensures at least one expert is selected.Source: §C.1 · p. 25
  • Selected gate values are softmaxed using temperature T to produce scores, and the selected experts’ outputs are computed and aggregated.The supplied pseudocode truncates the aggregation expression.Source: §C.1 · p. 25

C.2 Hyperparameter Sensitivity

  • Table 7 reports a key-hyperparameter sensitivity analysis on MNIST.The supplied passages identify the table’s scope but do not provide its sensitivity values or outcomes.Source: §C.2 · p. 25, Table 7 · p. 25

Related papers

Loading 2603.01697v1…