PaperlayerView exact revision on arXiv

Source-linked AI summary

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta, Pratik Jayarao, Neeraj Varshney, Bing Yin

arXiv:2604.19835v2cs.LGcs.AI

TL;DR

Large MoEs provide scalable capacity but are expensive to train because total-parameter memory and communication costs grow even when per-token computation stays sparse. This paper progressively expands a smaller MoE through expert duplication and router extension during continued pre-training, preserving top-K inference cost while using warm initialization and utility-guided replication. In 7B→13B experiments, the approach matched fixed-size quality while saving substantial GPU hours, with broader experiments supporting its compute-efficient use and identifying current scope boundaries.

  • Problem

    Training large MoEs from scratch is expensive because memory and inter-device communication scale with total parameter count despite sparse per-token computation.

  • Method

    Expert upcycling duplicates experts and extends the router during continued pre-training, holding top-K routing fixed so the expanded model preserves per-token FLOPs and starts from learned parameters.

  • Results

    24–32% GPU-hour savings were achieved in 7B→13B experiments, while 100% CPT closed the downstream accuracy gap to within 0.3 points across 11 benchmarks.

  • Takeaways & Limitations

    Expert upcycling is presented as a principled, compute-efficient alternative to training large sparse models from scratch, with iterated upcycling suggested as a natural extension.

  • Takeaways & Limitations

    Experiments cover m=2 upcycling on MoE architectures up to 7B parameters, leaving larger expansion factors, frontier-scale models, and multilingual or distribution-shifted CPT as open directions.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models: frontier models routinely decouple total parameters from per-token computation through sparse expert routing. Scaling laws show that under fixed active computation, model quality scales predictably with total parameters, and MoEs realize this by increasing expert count. However, training large MoEs is expensive, as memory requirements and inter-device communication both scale with total parameter count. We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts during continued pre-training (CPT). Given a trained E-expert model, the upcycling operator constructs an mE-expert model through expert duplication and router extension while holding top-K routing fixed, preserving per-token inference cost. Duplication provides a warm initialization: the expanded model inherits the source checkpoint's learned representations, starting from a substantially lower loss than random initialization. Subsequent CPT then breaks the symmetry among duplicated experts to drive specialization. We formalize the upcycling operator and develop a theoretical framework decomposing the quality gap into a capacity term and an initialization term. We further introduce utility-based expert selection, which uses gradient-based importance scores to guide non-uniform duplication, more than tripling gap closure when CPT is limited. In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours. Comprehensive ablations across model scales, activation ratios, MoE architectures, and training budgets yield a practical recipe for deploying expert upcycling, establishing it as a principled, compute-efficient alternative to training large MoE models from scratch.

1 Introduction

Expert upcycling expands MoE capacity during training by duplicating experts and extending the router while holding top-K routing fixed, preserving per-token compute. The method targets cheaper training than large-MoE-from-scratch approaches while retaining quality through warm initialization, continued specialization, and utility-guided duplication.

  • MoEs increase total expert count at fixed K to expand capacity without increasing per-token inference cost.
  • Training large MoEs remains expensive because memory and all-to-all communication costs scale with total expert count.All-to-all communication can consume 45–50% of total training time on standard GPU clusters.
  • Expert upcycling expands an E-expert model to mE experts during training by duplicating experts and extending the router while keeping active parameters and per-token FLOPs fixed.The method begins with a smaller model, transitions at step τ, and continues pre-training on the expanded model.
  • The expanded model inherits learned representations through duplication, then continued pre-training breaks duplicate symmetry and drives expert specialization.This warm initialization starts near the source model’s loss rather than from random initialization.
  • 32% GPU-hour savings at 50% CPT accompanied lower validation loss than the fixed-size baseline in 7B→13B experiments.At 100% CPT, the downstream accuracy gap closed to within 0.3 points across 11 benchmarks.
  • Utility-based expert selection uses gradient-based importance scores for non-uniform duplication and more than triples gap closure when CPT is limited.
  • Theoretical analysis decomposes the quality gap into capacity and initialization terms and supports a practical recipe across scales, activation ratios, architectures, and budgets.
  • Expert upcycling is presented as a compute-efficient alternative to training large sparse models from scratch, with progressive capacity expansion as a potential training strategy.

2 Related work

Related work studies scaling sparse MoEs, progressive network growth, checkpoint upcycling, and routing-saliency methods. Expert upcycling combines these threads by expanding already-sparse MoE capacity through learned-expert reuse without increasing active computation.

  • Scaling Mixture-of-Experts models: Sparse MoE systems grow total parameters while keeping active parameters per token fixed, improving quality-per-FLOP trade-offs at lower activation ratios.
  • Growing model capacity during training: Progressive training methods grow depth or dense width during a run, which raises active parameters and inference cost; SPARKLING applies mid-training width expansion to MoEs.
  • Growing model capacity during training: Sparse upcycling reuses an existing dense checkpoint to create an MoE, whereas expert upcycling expands capacity within an already-sparse architecture.
  • Load balancing and saliency metrics: Routing-saliency methods identify experts contributing most to loss, and expert upcycling uses these scores to choose duplication targets.The paper frames pruning and upcycling as dual uses of expert-level importance scores.

3 Expert upcycling

Expert upcycling expands an MoE during training by reusing a smaller model’s experts, reducing training cost while preserving per-token inference computation. Its warm initialization, utility-guided duplication, and continued pre-training address the quality gap to fixed-size training.

  • Method: Expert upcycling constructs an mE-expert model from an E-expert checkpoint by duplicating experts, extending the router, and fixing top-K routing.Both approaches use the same token budget and produce identical per-token FLOPs.
  • Compute efficiency: The upcycling cost is C_up = τ × s_E + (T − τ) × s_mE, compared with C_fs = T × s_mE for fixed-size training.The first τ steps run on the smaller model, whose per-step cost is lower.
  • Compute efficiency: ∼32% GPU-hour savings are achieved at τ ≈ 2/3T while matching from-scratch validation loss in 7B-scale experiments.The saving grows with the transition time and the per-step cost difference between models.
  • Quality gap closure: The quality gap decomposes into a capacity gap from early smaller-model training and an initialization gain from starting near the expanded-model optimum.Increasing τ saves compute but widens the capacity gap, whereas initialization quality is independent of τ.
  • Quality gap closure: The operator warm-starts duplicated experts by copying trained weights and approximately preserving routing, yielding an expansion loss gap below 10^-2 in practice.Bias noise and subsequent stochastic gradient diversity break duplication symmetry and drive specialization.
  • Post-expansion dynamics: Exact Net2Net-style function preservation is unavailable because top-K routing is discrete, so the method targets low initialization loss instead.This limitation motivates warm initialization rather than exact preservation.

4 Experimental setup

The experiments evaluate expert upcycling across model architectures, data stages, matched token budgets, and multiple comparison configurations. The main study emphasizes a 7B-to-13B interleaved MoE, while ablations measure validation-loss gap closure.

  • Architecture and training: The main experiments use a 20-layer interleaved MoE with approximately 7B→13B total and approximately 1B active non-embedding parameters.Interleaving limits all-to-all communication to half the layers.
  • Data: Base-model pre-training and continued pre-training use disjoint data splits to avoid leakage between stages.Small ablations use DCLM, while the 7B-scale study uses a curated instruction-following, reasoning, and mathematics mixture.
  • Evaluation protocol: Each experiment compares fixed E-expert training, E→mE upcycling, and fixed mE-expert training from scratch at matched total token budgets.The 7B-total/1B-active study reports validation loss and accuracy across 11 downstream benchmarks.
  • Evaluation protocol: Validation-loss gap closure is normalized so that a value of 1 indicates complete closure.This metric is used for the approximately 1B-total and approximately 144M-active-non-embedding ablations.
  • Upcycling procedure: At transition, optimizer states are reset, utility scores average gradients over 10 batches, and duplicate allocation uses a per-expert cap of n=3.Router-bias noise is set to δ = 10^-3.

5 Results

Expert upcycling approaches fixed-size MoE quality through progressive capacity expansion, warm initialization, and continued pre-training. Across scale, budget, duplication, and activation-ratio experiments, the method approaches fixed-size baselines with substantially lower training cost.

  • 5.1 Expert upcycling at scale: 32→64 expert upcycling saves ∼32% of GPU-hours at 50% CPT while reaching validation loss below Fixed-64 and average accuracy within 1.3 points.At 100% CPT, average accuracy is 56.4 vs. 56.7 and validation loss is 1.263 vs. 1.267, with efficiency 111.8%.
  • 5.1 Expert upcycling at scale: The upcycled 64-expert model starts at training loss 1.38, far below the randomly initialized model’s 10.5, and falls below Fixed-32 within ∼6% of pre-training.The source 32-expert model has loss 1.32 immediately after upcycling, and later CPT closes the gap to Fixed-64.
  • 5.2.1 Training budget allocation: 94–100% efficiency results from early transitions at τ/T ≤ 0.25, whereas τ/T = 0.05 underperforms slightly because specialization is not yet developed.The authors attribute the very early-transition penalty to weaker warm initialization.
  • 5.2.1 Training budget allocation: 34.7% to 98.0% efficiency is achieved as CPT increases from 10% to 100%, with at least 50% CPT needed for strong gap closure.Pre-training determines initialization quality, while post-upcycling CPT determines expert differentiation.
  • 5.2.2 Expert upcycling strategies: Utility-based duplication more than triples gap closure at 25% CPT, reaching 26.5% versus 8.2% for uniform duplication.Selective duplication outperforms uniform duplication and random initialization across CPT budgets; gradient norm performs best overall among utility strategies.
  • 5.2.3 Effect of activation ratio and comparison with sparse upcycling: At activation ratios from 25% to 3.13%, expert upcycling remains close to Fixed-mE, while sparse upcycling fails to match Fixed-E and its disadvantage widens at lower ratios.The expert-upcycling residual gap grows from 0.005 at 25% to 0.020 at 3.13%, while the method gap widens from 0.026 to 0.241.

6 Discussion

Expert upcycling expands MoE capacity during training while preserving top-K routing, and its theoretical and empirical analysis supports compute savings under stated assumptions and scope limits.

  • Discussion: 24–32% of GPU hours are saved in 7B→13B experiments, while validation loss is lower than the fixed-size baseline.At 100% CPT, downstream accuracy closes to within 0.3 points across 11 benchmarks.
  • Discussion: The method duplicates experts and extends the router mid-training while holding top-K routing fixed, preserving inference cost.Its theoretical decomposition separates a capacity gap from an initialization gain.
  • Discussion: The reported scope covers m=2 upcycling on MoE architectures up to 7B parameters, with DCLM ablations and an English-majority mixture for the 7B run.Larger expansion factors, frontier-scale models, and multilingual or distribution-shifted CPT remain open directions.
  • Discussion: The analysis compares expert upcycling with fixed-size training over shared schedules using a two-phase procedure and an expanded parameter space.The upcycling procedure trains E experts for τ steps, then mE experts for T−τ steps; the fixed-size procedure trains mE experts for all T steps.
  • Discussion: The theoretical treatment assumes bounded gradients and uses convexity for structural insight rather than tight numerical bounds.The passages explicitly note that convexity does not literally hold for deep networks.

A.2 Theorem and proof

The theorem derives an OCO-based bound by telescoping regret inequalities across the small-model and expanded-model phases, producing capacity and initialization terms under explicit assumptions.

  • A.2 Theorem and proof: The proof compares learning-rate-weighted average losses for expert upcycling and fixed-size training over the same schedule.Their regret upper bounds are denoted R_up and R_fs.
  • A.2 Theorem and proof: The theorem assumes shared initial parameters and coincident shared components of the two procedures’ optima, while convexity is acknowledged as a deep-network limitation.The convex framework is presented as qualitatively useful rather than numerically tight.
  • A.2 Theorem and proof: The derivation applies a one-step gradient-descent regret inequality and sums it phase-wise across the transition at τ.Phase 1 is represented in the expanded space through canonical lifting; Phase 2 trains directly in the expanded space.
  • A.2 Theorem and proof: The regret-bound difference decomposes into a capacity gap and an initialization-related term after comparator-loss and distance-term algebra.The G^2 contributions cancel because both procedures use the same schedule and gradient bound.

B Theoretical justification for gradient-based utility scores

The paper justifies gradient-based expert utilities through first-order loss sensitivity, showing why gradient signals can guide duplication more effectively than uniform copying while avoiding costly curvature estimates.

  • B Theoretical justification for gradient-based utility scores: Gradient norm ranks experts by how much the loss can change under a unit parameter perturbation.The score is derived from a first-order Taylor expansion around transition time τ.
  • B Theoretical justification for gradient-based utility scores: Large u_G(e) identifies experts whose parameters are most responsive under the current data distribution and routing.Replicating them gives continued pre-training greater opportunity to reduce the initialization gap.
  • B Theoretical justification for gradient-based utility scores: The scale-aware utility multiplies parameter norm by gradient norm to avoid underranking functionally important experts with large parameter norms.Plain gradient norm is not scale-invariant under uniform parameter rescaling.
  • B Theoretical justification for gradient-based utility scores: u_G and u_SAL perform similarly and both significantly outperform uniform copy-paste in the reported experiments.The result supports using gradient-based importance rather than treating all experts as equally valuable replication targets.
  • B Theoretical justification for gradient-based utility scores: First-order utilities are preferred over second-order variants because Hessian estimation is expensive and noisy, while curvature-normalized variants did not outperform them.The paper notes that diagonal Fisher approximations can introduce significant bias.

C.1 Full MoE generalization results

Full MoE experiments test expert upcycling beyond the interleaved architecture and report strong gap closure across model sizes and activation ratios.

  • C.1 Full MoE generalization results: Full MoE evaluation uses 256 experts with top-K=8, corresponding to an approximately 3% activation ratio.The architecture is evaluated at approximately the 1B total-parameter scale with gradient-norm utility-based upcycling.
  • C.1 Full MoE generalization results: Strong gap closure is observed across all tested model sizes in the full MoE evaluation.The table spans 154M to 1B total parameters and reports validation loss.
  • C.1 Full MoE generalization results: Both interleaved and full MoE architectures show strong gap closure, supporting effectiveness across MoE families and activation ratios.The full-MoE configuration uses 256 routed experts with Top-8 routing.
  • C.1 Full MoE generalization results: Table 7 reports validation loss for 256→512-expert upcycling with top-K=8 and gradient-norm utility-based duplication.Values are compared across model sizes from 154M to 1B total parameters.

D Heuristic upcycling methods and results

The appendix evaluates expert- and router-level initialization heuristics for upcycling, finding that none meaningfully outperform simple copy-paste duplication.

  • 20 initialization heuristics—10 expert-level and 10 router-level—were evaluated to seed duplicate diversity while retaining inherited capability.The evaluation covers both expert and router initialization strategies.
  • None of the evaluated heuristics meaningfully outperformed simple copy-paste duplication.

D.1 Summary of results

Across heuristic variants, validation losses remain close to the copy-paste baseline, while stronger perturbations can slightly degrade performance. The results favor preserving low loss at the upcycling boundary over injecting artificial diversity.

  • At most ∼10−3 validation-loss improvement separates heuristic variants from copy-paste on the 10-layer, 32→64 expert model.The reported comparison spans the expert- and router-level heuristic categories.
  • Several aggressive methods, including SVD mixing and orthogonalization, slightly degrade performance.
  • Copy-paste duplication preserves a low initial loss, avoiding recovery costs that can arise when perturbations disrupt the pre-upcycling solution.
  • Loss-free load balancing gives all experts gradient signal, allowing training dynamics to drive differentiation naturally.

E.2 Growing network size during training

Expert upcycling extends progressive model growth to the MoE expert-count dimension, expanding an existing sparse model rather than converting a dense model or adding externally trained domain experts. Its duplication-based initialization reuses existing capability while continued pre-training enables specialization.

  • Growing network size during training: Expert upcycling grows MoE capacity by duplicating existing experts, whereas dense progressive methods grow layers or dense width.
  • Growing network size during training: The method is complementary to approaches that add independently trained experts, compress duplicated experts, or progressively train subnetworks.
  • Growing network size during training: Exact duplication provides warm initialization by construction, avoiding the overhead of learning cross-weight mappings.
  • Growing network size during training: Unlike Sparse Upcycling and related dense-to-MoE methods, expert upcycling expands an already-sparse MoE into a larger MoE.
  • Growing network size during training: Expert upcycling relies on continued pre-training to break duplication symmetry, while Branch-Train-MiX creates diversity through independent domain-specific training.
Loading 2604.19835v2…