PaperlayerView exact revision on arXiv

Source-linked AI summary

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

Yuxuan Lou, Kai Yang, Geng Zhang, Yong Liu, Yang You

arXiv:2610.11317v1cs.AIcs.LG

TL;DR

Fine-grained upcycling from a single dense source can cause routing collapse and near-random accuracy. DIVMOE combines domain-specialized fine-grained initialization with structurally constrained routing, and outperforms upcycling baselines across models and benchmarks.

  • Problem

    Fine-grained experts derived from a single dense source can collapse in routing and reach near-random accuracy, limiting sparse upcycling from pre-trained models.

    Source: §1 · pp. 1–2

  • Method

    DIVMOE slices experts from domain-adapted dense models and constrains each token to select at most one expert per domain group.

    Source: §1 · p. 2

  • Results

    Across two base models and 15 benchmarks, DIVMOE consistently outperforms six baselines; its 12B model matches Moonlight-MoE (16B) at 64.5%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that its components are complementary and jointly necessary for robust fine-grained upcycling, with continual pre-training no longer regressing below the dense base.

    Source: §6 · p. 10

    Experiments use base models up to 4B parameters and four fixed broad domains; Stage 1 adds 40B tokens per base model, an 8% overhead relative to from-scratch training.

    Source: A Limitations and Future Work · p. 14

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, with recent work demonstrating the benefits of fine-grained expert designs. Training such models from scratch is expensive, and sparse upcycling from pre-trained dense models is an attractive alternative. However, we identify a structural pathology of fine-grained upcycling: when fine-grained experts are derived from a single source model, naive routing collapses and downstream accuracy drops to near-random (e.g., on Qwen3-1.7B, Drop-Upcycling-fine-grained reaches only 23.2% average accuracy across 15 benchmarks, essentially matching from-scratch training at 22.2%, while the same method's coarse-grained variant reaches 50.2%). We propose DivMoE, the first framework achieving fine-grained MoE Upcycling with structurally-balanced routing. DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training, and diversity-constrained routing, a hard structural constraint guaranteeing that each token activates experts from distinct domain groups. Across two base models and 15 benchmarks, DivMoE consistently outperforms six upcycling baselines (55.6% vs. 51.6% for the strongest baseline on Qwen3-1.7B) and strictly improves over the dense base model on every benchmark after Stage 2 continual pre-training -- closing the regression gap that has plagued prior fine-grained upcycling. After supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy while outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points.

1 Introduction

Fine-grained upcycling from a single dense source can collapse routing and reduce accuracy to near-random levels. DIVMOE combines domain-specialized experts with a structural routing constraint, and its experiments show strong performance across models and benchmarks.

1 Introduction

  • Single-source fine-grained upcycling can collapse to 23.2% average accuracy across 15 benchmarks, near the 22.2% from-scratch result and below its coarse-grained variant at 50.2%.NVIDIA Upcycling reaches 51.1%, but its homogeneous expert initial scores suppress routing diversity.Source: §1 · pp. 1–2
  • The paper identifies domain-specialized initialization, fine-grained partitioning, and prevention of same-source co-activation as jointly necessary design axes.Prior methods address only subsets of these axes, while DIVMOE combines them.Source: §1 · p. 2
  • DIVMOE combines domain-specialized fine-grained expert initialization with a hard constraint that each token selects at most one expert per domain group.Experts are sliced from domain-adapted dense models, preserving pre-trained knowledge while introducing structured routing diversity.Source: §1 · p. 2
  • Across two base models and 15 benchmarks, DIVMOE consistently outperforms six baselines; its 12B model matches Moonlight-MoE (16B) at 64.5%.After supervised fine-tuning on a public reasoning mixture, it also exceeds a controlled NVIDIA-Upcycling baseline by 6.1 percentage points.Source: §1 · p. 2
  • The authors identify the fine-grained, single-source collapse pathology as a cause of routing collapse and severe performance degradation.The introduction frames DIVMOE as addressing this pathology.Source: §1 · p. 2

2 Related Work

Prior work develops MoE scaling, expert specialization, dense-to-MoE conversion, routing stabilization, and continual pre-training. DIVMOE is positioned as combining fine-grained experts, domain diversity, and a structural routing constraint.

Mixture-of-Experts in LLMs

  • MoE work scales capacity through sparse activation, with fine-grained expert segmentation among the developments associated with DeepSeek-MoE and DeepSeek-V2.GShard and Switch Transformer scaled MoE using top-k routing and load-balancing losses.Source: §2 · pp. 2–3

Dense-to-MoE upcycling

  • Dense-to-MoE methods differ in expert source and granularity: BTX assembles coarse experts from branches, NVIDIA Upcycling uses a single source, and Drop-Upcycling randomizes some expert weights.FlexOLMo uses domain specialists for data governance but has coarse experts, independent training, and no routing constraint.Source: §2 · p. 3

Routing collapse and load balancing

  • Soft routing remedies alter gradients but not reachable selections, whereas DIVMOE’s diversity constraint structurally limits selections regardless of gradient magnitudes.DIVMOE retains a small auxiliary load-balancing loss to encourage utilization within domain groups.Source: §2 · p. 3

Continual pre-training

  • The paper uses public Dolmino-Mix-1124 continual-pretraining data, described as high enough quality for all upcycling baselines to improve over dense bases on most benchmarks.The passage notes that mismatches between original pretraining distributions and continual-pretraining mixtures can cause transient task-specific regressions.Source: §2 · p. 3

Positioning

  • Only DIVMOE occupies all three design choices summarized in Table 1, while fine-grained partitioning without domain diversity is associated with collapse.The three axes are domain-specialized initialization, fine-grained partitioning, and prevention of same-source co-activation.Source: §2 · p. 3, Table 1 · p. 3

3 DivMoE

DivMoE converts dense models into fine-grained MoE architectures through domain-specialized expert initialization and diversity-constrained routing. Its pipeline trains specialists, assembles and continually pre-trains the MoE, and optionally applies supervised fine-tuning.

3 DivMoE

  • DivMoE combines domain-specialized fine-grained expert initialization with diversity-constrained routing to enforce cross-domain expert selection.The overview depicts four specialists sliced into eight experts across four domain groups, with top-1 selection within each group followed by top-k group selection.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Domain-Specialized Fine-Grained Expert Initialization

  • Four domain-adaptively pre-trained specialists are sliced into fine-grained FFN experts, preserving base-model output behavior at initialization while adding structured diversity.With n = 4, m = 2, and k = 2, the model doubles total capacity while activating half the dense FFN parameters.Source: §3.1 · p. 4, §3 · p. 4

3.2 Diversity-Constrained Routing

  • Diversity-constrained routing selects at most one expert per domain group, guaranteeing k experts from k distinct groups when k ≤ n.The hard constraint prevents same-group co-activation by construction; unlike soft balancing losses, it does not rely on gradient pressure, while routing weights can still preserve domain emphasis.Source: §3.2 · p. 5

3.3 Training Pipeline

  • The three-stage pipeline uses public corpora for specialist initialization and MoE continual pre-training, then applies supervised fine-tuning only for the open-source MoE comparison.Stage 1 uses four domain partitions for 4×10B tokens; Stage 2 trains all upcycling baselines on the same corpus for 500B tokens, and Stage 3 applies the same SFT mixture to the compared models.Source: §3.3 · p. 6

4 Experiments

The experiments compare DIVMOE with six upcycling baselines on two base models and evaluate its open-source-model performance under controlled supervised fine-tuning. DIVMOE leads the upcycling comparisons, improves over the dense base after Stage 2 continual pre-training, and matches Moonlight-MoE in the reported full-set comparison.

4.1 Experimental Setup

  • The setup evaluates Qwen3-1.7B-Base and Llama3.2-1B against six construction baselines on 15 benchmarks spanning knowledge, reasoning, code, and science.All variants use four domain groups, two experts per group, eight experts total, and two active experts per token.Source: §4.1 · p. 6

4.2 Comparison with Upcycling Methods

  • 54.3% average accuracy on Qwen3-1.7B-Base across 11 benchmarks exceeds BTX by 3.5 points and NVIDIA Upcycling by 3.8.DIVMOE also reaches 37.4% on Llama3.2-1B, versus 35.5% for BTX.Source: §4.2 · p. 7, Table 2 · p. 7
  • DIVMOE surpasses Qwen3-1.7B-Base on every evaluated benchmark after Stage 2 CPT, including MMLU (64.5 vs. 62.6) and MATH (40.5 vs. 38.4).It also exceeds a data-matched dense baseline by approximately 2 points while activating approximately 2.4B of its 5.4B parameters per token.Source: §4.2 · p. 7
  • Fine-grained From-Scratch and Drop-Upcycling reach 20.7% and 21.8% on Qwen3-1.7B-Base, while coarse-grained Drop-Upcycling reaches 49.1%.The passage attributes the collapse to fine-grained partitioning without domain-specialized initialization.Source: §4.2 · p. 7

4.3 Comparison with Open-Source MoE Models (Controlled SFT)

  • 63.0 vs. 56.2: under the same Stage 3 SFT, DIVMOE outperforms NVIDIA Upcycling on the 7-benchmark subset.The controlled comparison uses the same Tülu-3-SFT-Mixture and Stage 3 hyperparameters.Source: §4.3 · p. 8, Table 4 · p. 8
  • With 12B total parameters, DIVMOE matches Moonlight-MoE’s 64.5% average on the full 8-benchmark set; on the 7-benchmark subset, scores are 63.0 vs. 62.3.The controlled setup applies the same Stage 3 SFT to DIVMOE and selected baselines to isolate architectural effects from data effects.Source: §4.3 · p. 8, Table 4 · p. 8

5 Ablation Studies

Ablations show that the three DIVMOE design axes work jointly, while the diversity constraint improves domain-specific benchmark performance. Routing visualizations and granularity tests further characterize expert use and the selected configuration.

5.1 Component Synergy and the Diversity Constraint

  • Removing any of the three design axes reduces performance, and removing two collapses fine-grained variants to near-random; the full combination addresses a failure no individual component solves.The constraint also improves performance on every evaluated domain-specific benchmark.Source: §5.1 · p. 8

5.2 Routing Patterns

  • Sparse Upcycling routes over 60% of tokens to two experts at layer 36, while DIVMOE balances routing across groups and preserves within-group specialization.Dataset preferences remain visible: GSM8K activates more math experts and HumanEval more code experts.Source: §5.2 · p. 9, Figure 3 · p. 9

5.3 Effect of Expert Granularity

  • m = k = 2 is the adopted balance: m = k = 4 reduces training loss but mildly overfits on MMLU and HellaSwag, while m = k = 1 plateaus at lower accuracy.The m = k = 2 setting keeps the active-parameter footprint at half of the dense base.Source: §5.3 · p. 9, Figure 4 · p. 9

6 Conclusion

DIVMOE addresses fine-grained upcycling collapse with domain-specialized initialization, diversity-constrained routing, and public-corpus continual pre-training. The paper reports complementary components, consistently strong performance, and improvements from the structural constraint on every tested domain-specific benchmark.

6 Conclusion

  • Fine-grained partitioning with single-source initialization causes downstream accuracy to drop to near-random, irrespective of the partitioning scheme.DIVMOE combines domain-specialized initialization, hard diversity-constrained routing, and public-corpus continual pre-training.Source: §6 · p. 10
  • Ablations show the proposed components are complementary and jointly necessary for robust fine-grained upcycling.Source: §6 · p. 10
  • The structural constraint improves performance on every tested domain-specific benchmark.The conclusion says this refutes the intuition that cross-domain co-activation should harm specialized tasks.Source: §6 · p. 10

A Limitations and Future Work

The experiments are bounded to base models up to 4B parameters and a fixed four-domain partition. The pipeline uses public corpora across three training stages, with Stage 3 reserved for the open-source MoE comparison.

A Limitations and Future Work

  • Experiments cover base models up to 4B parameters and MoE models up to 12B; scaling to 7B–13B base models remains future work.The domain partition is fixed to four broad domains, and learning it end-to-end is also left for future work; Stage 1 adds 40B tokens, an 8% overhead relative to from-scratch training.Source: A Limitations and Future Work · p. 14

B.1 Model Architecture

  • DIVMOE uses four domain groups, two experts per group (eight total), and activates two experts per token.The architecture table describes configurations derived from each base model with expansion factor n = 4.Source: §B.1 · p. 14, Table 7 · p. 14

B.2 Training Pipeline Details

  • Stage 1 continually pre-trains four dense base-model copies on public math, code, academic, and web subsets for 10B tokens each.This 40B-token stage is the only stage that updates the base weights as a dense model; later stages train the assembled MoE.Source: §B.2 · p. 14
  • Stage 2 trains the assembled MoE for 500B tokens on a public corpus using the same corpus, hyperparameters, and step count as every upcycling baseline.Stage 3 applies public Tülu-3-SFT-Mixture only for the open-source MoE comparison, with the same SFT also applied to NVIDIA Upcycling and OLMoE.Source: §B.2 · pp. 14–15
  • The pipeline uses publicly available data and releases the expert-slicing and routing code alongside per-domain corpus splits.The stated inputs are dense base checkpoints, Dolmino-Mix-1124, and Tülu-3-SFT-Mixture.Source: §B.2 · pp. 14–15

B.3 Training Hyperparameters

The appendix specifies shared training settings, compute accounting, and benchmark evaluation procedures. It reports a 540B-token total for DIVMOE and BTX and describes the evaluation formats and decoding choices.

B.3 Training Hyperparameters

  • Experiments use AdamW with β1 = 0.9 and β2 = 0.95, bf16 mixed precision, and a global batch size of 2M tokens.The hyperparameter table lists settings by training stage.Source: §B.3 · p. 15, Table 8 · p. 15
  • Stage 3 long-sequence training uses DeepSpeed Ulysses sequence parallelism with factor 4 and CPU offloading of optimizer states.Source: §B.3 · p. 15

B.4 Compute Budget

  • DIVMOE and BTX each use 540B tokens total, with DIVMOE’s additional 40B Stage-1 tokens counted as dense-model compute.The table compares per-method compute and reports GPU-hours measured on NVIDIA H20 (80GB) for Qwen3-1.7B-Base.Source: §B.4 · p. 15, Table 9 · p. 15

B.5 Evaluation Configuration

  • Evaluation uses greedy decoding at temperature 0 for generative tasks and likelihood-based scoring for multiple-choice tasks.The benchmark table lists each task’s format and shot count.Source: §B.5 · p. 15

C Full Upcycling Results (15 Benchmarks)

The appendix provides full upcycling comparisons across 15 benchmarks and a separate full eight-benchmark open-source MoE comparison. The latter reports equal average accuracy for DIVMOE and Moonlight-MoE.

C Full Upcycling Results (15 Benchmarks)

D Full Open-Source MoE Comparison (8 Benchmarks)

E Attention Layer Initialization Ablation

Mean attention initialization is the strongest of the three tested strategies, while learning-dynamics comparisons report a sustained DIVMOE advantage. Extended tables provide loss, perplexity, and accuracy at 10B-token intervals.

E Attention Layer Initialization Ablation

  • Mean attention initialization is best, while base-only trails by 0.4 points and random per-layer specialist selection performs worst at −1.3.The modest spread indicates that FFN expert initialization is the primary driver of DIVMOE’s advantage, with mean attention initialization adding a small, consistent boost.Source: E Attention Layer Initialization… · pp. 16–17, Table 14 · p. 17

F Learning Dynamics

  • Figure 5 reports lower training loss and validation perplexity and higher downstream performance for DIVMOE throughout Stage 2 on both evaluated base models.Source: Figure 5 · p. 17
  • DIVMOE leads NVIDIA Upcycling by 4.5 points at both 150B and 500B tokens, sustaining its advantage during Stage 2 training.Table 15 compares 15-benchmark average accuracy at extended checkpoints.Source: F Learning Dynamics · p. 17, Table 15 · p. 18

G Extended Learning Dynamics

H Granularity Ablation Details

Granularity changes both training dynamics and benchmark performance: m=k=2 offers the best stability–accuracy trade-off, while m=k=1 converges fastest but plateaus lower.

H Granularity Ablation Details

Related papers

Loading 2610.11317v1…