PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models

Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

arXiv:2507.17702v4cs.CL

TL;DR

MoE architectures make total parameter count largely independent of per-token computation, but their capacity under different architectural configurations is difficult to predict. This paper introduces Efficiency Leverage, derives a unified scaling law from more than 300 trained models, and validates it with a pilot model achieving over 7x computational efficiency against a dense counterpart.

  • Problem

    Predicting MoE capacity from architectural choices such as activation ratio and expert granularity remains unresolved despite MoE’s computational advantages.

  • Method

    The study defines Efficiency Leverage, analyzes activation ratio, expert granularity, compute budget, and other design factors across more than 300 models, then derives a unified scaling law.

  • Results

    The 17.5B-parameter Ling-mini-beta achieved over 7x efficiency versus its dense equivalent, confirming the predictive scaling framework.

  • Takeaways & Limitations

    Activation ratio and compute budget govern EL through power laws, while granularity provides a nonlinear modulation with a stable optimal range for efficient MoE design.

  • Takeaways & Limitations

    The analysis uses theoretical FLOPs and simplifying assumptions that may overlook hardware costs and interaction effects between architectural components.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from computational cost. However, this decoupling creates a critical challenge: predicting the model capacity of a given MoE configurations (e.g., expert activation ratio and granularity) remains an unresolved problem. To address this gap, we introduce Efficiency Leverage (EL), a metric quantifying the computational advantage of an MoE model over a dense equivalent. We conduct a large-scale empirical study, training over 300 models up to 28B parameters, to systematically investigate the relationship between MoE architectural configurations and EL. Our findings reveal that EL is primarily driven by the expert activation ratio and the total compute budget, both following predictable power laws, while expert granularity acts as a non-linear modulator with a clear optimal range. We integrate these discoveries into a unified scaling law that accurately predicts the EL of an MoE architecture based on its configuration. To validate our derived scaling laws, we designed and trained Ling-mini-beta, a pilot model for Ling-2.0 series with only 0.85B active parameters, alongside a 6.1B dense model for comparison. When trained on an identical 1T high-quality token dataset, Ling-mini-beta matched the performance of the 6.1B dense model while consuming over 7x fewer computational resources, thereby confirming the accuracy of our scaling laws. This work provides a principled and empirically-grounded foundation for the scaling of efficient MoE models.

1 Introduction

MoE models decouple total parameters from computational cost, but predicting capacity from architectural choices remains unresolved. This work introduces Efficiency Leverage, identifies scaling relationships governing it, and validates a unified predictor with a pilot model.

  • Metric: Efficiency Leverage quantifies the ratio of dense-to-MoE compute required to reach the same performance level, with higher values indicating greater efficiency.For example, EL of 2 means the MoE model requires half the dense model’s computational cost.
  • Findings: EL increases predictably as the expert activation ratio decreases, making activation ratio the primary architectural driver of efficiency.The relationship follows a stable power law, so greater sparsity yields consistent computational gains.
  • Findings: Expert granularity nonlinearly modulates EL, with experiments identifying an optimal range of 8 to 12 under the standard load-balancing loss.The granularity effect is independent of the total compute budget.
  • Findings: EL also follows a power law with training compute, so efficiency gains become increasingly significant at larger pre-training budgets.The study examines this relationship across more than 300 trained models, alongside other architectural factors.
  • Contribution: A unified scaling law combines compute budget, activation ratio, and expert granularity to predict EL and guide MoE architecture design.Other choices, including shared experts and MoE/dense layer arrangement, have only secondary influence and generally require little tuning.
  • Validation: Over 7x efficiency was achieved by Ling-mini-beta versus its 6.1B dense counterpart on the same 1T-token dataset, validating the predicted MoE advantage.Ling-mini-beta used 0.85B activated and 17.5B total parameters, achieved lower final training loss, and showed a slight downstream-task advantage.

2 Preliminary

The paper defines MoE sparsity and granularity measures, then establishes compute-dependent training and model-data allocation laws for fair architectural comparisons. These laws show that MoE models favor more data and smaller computational model scale than dense models at equal budgets.

  • 2.1 Mixture-of-Expert Transformers: MoE activation ratio A measures activated experts relative to total experts, while sharing ratio S measures shared experts relative to activated experts.These metrics quantify sparsity and expert utilization through A = (Ea + Es)/(E + Es) and S = Es/(Ea + Es).
  • 2.1 Mixture-of-Expert Transformers: Expert granularity G = 2dmodel/dexpert increases when a fixed parameter budget is divided among more smaller experts.This definition differs from Ludziejewski et al. (2024), producing experts half the size at the same granularity value.
  • 2.2 Scaling Laws for MoE Optimal Hyper-parameters: Optimal hyperparameters vary systematically with compute: batch size increases and learning rate decreases as the budget grows.Compared with dense models, MoE models favor a significantly larger batch size and a slightly lower learning rate at large compute scales.
  • 2.2 Scaling Laws for MoE Optimal Hyper-parameters: The predicted near-optimal hyperparameter regions generalize across activation ratios from 4.7% to 10.9% at a held-out compute budget.Near-optimal means achieving loss within 0.25% of the optimum.
  • 2.3 Scaling laws for MoE Optimal Model-Data Allocation: For compute-optimal training, model-scale and data-size allocation coefficients are both close to 0.5, indicating an approximately balanced budget split.The allocation laws are derived by minimizing loss over model scale M and data size D subject to C = M · D.
  • 2.3 Scaling laws for MoE Optimal Model-Data Allocation: At any fixed compute budget, optimal MoE training uses a smaller model scale and more data than optimal dense training.The result suggests MoEs can support larger training datasets with smaller models when data is abundant and compute is limited.
  • 2.3 Scaling laws for MoE Optimal Model-Data Allocation: The allocation laws are a principled reference for estimating training data needs, designing ablations, and developing more efficient MoE architectures.Practical training strategies may deviate from the compute-optimal allocation.

3 Efficiency Leverage: Metric for Quantifying MoE Compute-Efficiency

The paper introduces Efficiency Leverage (EL) to compare the compute required by MoE and dense architectures at matched performance, then studies how architectural choices and compute budget affect EL.

  • Definition of Efficiency Leverage: EL is the ratio of compute budgets required by dense and MoE models to achieve the same performance level.The study primarily measures performance using loss, though benchmark scores and task-specific metrics are also possible.
  • Definition of Efficiency Leverage: EL above 1 indicates that an MoE architecture reaches comparable performance with less compute than the dense model, whereas EL below 1 indicates lower efficiency.Under fixed data size, the power-law formulation approximates EL as the ratio of dense to MoE model scales.
  • Objective and Roadmap: The central research question is how MoE architectural configurations affect EL and how this relationship changes with computational budget.The investigation targets activation ratio, expert granularity, and shared expert ratio as dimensions jointly determining effective capacity.
  • Experimental Design: The study varies one MoE design dimension at a time from a baseline activating 2 of 64 experts plus one shared expert across compute budgets of 3e18 to 3e20 FLOPs.The analysis assumes these dimensions are largely independent and uses systematic ablations for tractability.
  • Methodology: The methodology establishes well-trained dense and MoE scaling conditions, analyzes core architectural dimensions, and integrates the findings into a unified relationship between configuration and EL.The stated dimensions include activation ratio, expert granularity, and shared expert ratio, among others.

4 Scaling Laws for Efficient MoE Architecture

The study derives scaling laws showing that MoE efficiency is driven mainly by activation ratio and compute budget, while granularity and sharing ratios modulate performance through identifiable optima.

  • 4.1.1 Optimal Expert Activation Ratio: Lower activation ratios consistently reduce loss and increase efficiency leverage, with the gains becoming larger at higher compute budgets.Across tested budgets, the lowest tested activation ratio, 0.8%, achieved the minimum loss; EL increased as activation ratio decreased and compute increased.
  • 4.1.2 Optimal Granularity of Experts: Expert granularity has a U-shaped relationship with loss, with an optimal range around 12 across FLOPs budgets.Poor routing balance shifts the optimum toward coarser experts and degrades performance.
  • 4.1.3 Optimal Shared Expert Ratio: A low, non-zero shared-expert ratio minimizes loss, while the EL-optimal ratio decreases from 16.7% at smaller budgets to 8.3% at larger budgets.For large-scale training with uniformly sized experts, the authors recommend the lowest possible non-zero ratio, heuristically one shared expert.
  • 4 Scaling Laws for Efficient MoE Architecture: Replacing early MoE layers with dense layers has little efficiency impact while reducing parameters and mitigating routing imbalance.The reported efficiency leverage remains close to 1 through 3e20 FLOPs, and dense-layer proportions may increase with compute budget despite limited efficiency gains.
  • 4 Scaling Laws for Efficient MoE Architecture: The unified EL scaling law combines compute budget, activation ratio, and expert granularity to predict MoE efficiency for a specified architecture.The formulation treats activation ratio as the primary efficiency driver and granularity as a nonlinear modulator, while satisfying EL = 1 for the dense-equivalent case A = 1.

5 Ling-mini-beta: More Efficient MoE Language Model

Ling-mini-beta is a 17.5B-parameter MoE model with about 0.8B active parameters, evaluated against Dense-6.1B under matched training conditions. It achieved comparable training dynamics and benchmark performance while providing over 7× training-efficiency leverage and a higher average evaluation score.

  • 5 Ling-mini-beta: More Efficient MoE Language Model: The pilot model contains 17.5B total parameters but only about 0.8B active parameters and was trained on 1T tokens.Its architecture uses granularity G = 12 and an activation ratio of 3.4%; Dense-6.1B served as the comparison model.
  • 5.2 Training Dynamics: Over 7× training-efficiency gains were achieved while Ling-mini-beta nearly matched Dense-6.1B’s final training loss.The final 100B tokens showed a loss difference of about 0.01, although Dense-6.1B converged faster early in training.
  • 5.3 Evaluation: Ling-mini-beta improved almost synchronously with Dense-6.1B and reached nearly identical final leaderboard results.The benchmark trajectories showed no decisive or lasting advantage for either model during training.
  • 5.3 Evaluation: Ling-mini-beta outperformed Dense-6.1B on several knowledge, reasoning, reading-comprehension, and coding evaluations.The reported lead exceeded 16 points on HumanEval-Plus, while performance in mathematical reasoning was slightly lower.

6 Discussion and Limitations

The study contrasts its findings with prior MoE scaling-law results and acknowledges that theoretical FLOPs and factor-isolation assumptions limit practical interpretation.

  • Comparison with Previous Works: MoE models remain more compute-efficient than dense counterparts across evaluated model sizes, contrary to Clark et al. (2022).The authors attribute the discrepancy to differences in experimental design.
  • Comparison with Previous Works: Expert granularity shows an optimal range rather than the monotonic loss reduction reported by Ludziejewski et al. (2024).The comparison uses different activation ratios and training conditions, including a 1/32 routable ratio here versus 1/64 in the prior work.
  • Comparison with Previous Works: The work extends Abnar et al. (2025) by studying granularity and shared-expert effects and deriving EL scaling laws rather than only loss scaling laws.The authors emphasize that EL is intended to be less dependent on specific training datasets.
  • Comparison with Previous Works: This study optimizes architectural parameters at fixed compute and model scale, complementing Ludziejewski et al. (2025), which optimizes model-data allocation under compute and memory constraints.Its preliminary allocation analysis was limited to one compute budget and a specific MoE architecture.
  • Limitations: The analysis uses theoretical FLOPs and assumes architectural factors act largely independently, potentially overlooking interaction effects and real-world hardware costs.Hardware specifications, infrastructure, and implementation details can separate theoretical FLOPs from wall-clock time.

7 Related Work

Prior work established scaling laws for model size, data, and compute, while MoE research studies sparse architectures and their resource-performance trade-offs.

  • Scaling Laws: Scaling-law research shows predictable power-law relationships between language-model performance, model size, dataset size, and compute budget.Chinchilla scaling laws further emphasize balancing model size and training data for compute-optimal training.
  • Mixture-of-Experts: MoE architectures decouple computational cost from parameter count, motivating studies of expert configuration and downstream performance.Recent work also reports distinct task behaviors between MoE and dense models.
  • Mixture-of-Experts: Joint scaling laws indicate that MoE models can outperform dense counterparts under memory or total-parameter constraints.This line of work complements architectural studies by examining resource allocation across model scale and data size.

8 Conclusion

The paper introduces EL to quantify MoE computational advantage, derives predictable architectural scaling behavior, and outlines extensions for practical efficiency constraints.

  • Conclusion: EL measures an MoE model’s computational advantage relative to a dense counterpart, enabling systematic analysis of architectural scaling behavior.The study bases this analysis on more than 300 trained models.
  • Conclusion: EL follows power laws with activation ratio and total compute, while expert granularity is a nonlinear modulator with a stable optimal range.Shared experts have a secondary effect, and these findings are combined into a unified EL scaling law that predicts MoE configurations.
  • Future Work: Future work should incorporate memory constraints, communication overhead, and inference latency into efficiency-aware MoE design.These extensions target distributed-training efficiency and end-to-end architecture co-design.

B Experimental Setup

The experiments follow Ling-series configurations, using a decoder-only GQA Transformer with top-k expert routing, specified optimization, and multilingual pre-training data.

  • Experimental Setup: The experimental models primarily follow the architecture and training configurations of the Ling series.This establishes the implementation context for the reported experiments.
  • Architecture and Tokenizer: The backbone is a decoder-only Transformer with grouped-query attention, alternating attention and feed-forward layers, BPE tokenization, and RoPE positional encoding.The vocabulary size is reported as 126,4 in the supplied passage.
  • Expert Routing Strategy: A routing network selects the top-Na experts for each token and combines their outputs using gating scores.Load-balancing objectives are added to support balanced expert utilization and stable training.
  • Optimizer and Scheduler: Models use AdamW with β1 = 0.9, β2 = 0.95, 0.1 weight decay, and a warmup-stable-decay learning-rate schedule.The schedule begins with linear warm-up for 1% of training steps before decay.
  • Pre-training Data: Pre-training uses a multilingual Ling corpus containing web, mathematical, programming, literary, and other text, with a 2T-token subset for experiments.The dataset composition is presented in Table 6.

C Estimating FLOPs

The paper estimates MoE training cost by combining attention, FFN, and output-projection FLOPs, then scales forward-pass cost to total training FLOPs.

  • Training FLOPs are approximated as three times forward-pass FLOPs, which sum attention, FFN, and final logit-projection costs.The estimate follows Ctrain ≈ 3 · Cfwd.
  • Attention cost accounts for GQA and all projections, parameterized by model width, batch size, sequence length, and attention head counts.
  • Dense and MoE FFN costs differ according to layer type, with MoE cost determined by activated experts and expert dimensions.Shared-expert cost is added when applicable.
  • Total forward FLOPs combine per-layer attention and FFN costs with the vocabulary-dependent final logit projection.

D Additional Experiments

Additional experiments examine routing balance, dense-layer placement, and attention–FFN compute allocation to identify robust MoE design choices and budget-dependent optima.

  • Impact of Routing Balance on the Optimal Expert Granularity: Under weakened routing balance, the study varies expert granularity while holding total parameter count constant to assess its effect on efficiency.The imbalance is induced with a load-balancing-loss coefficient of 0.001.
  • Arrangement of MoE and Dense Layers: The dense-layer experiment fixes 60 total layers and varies the first 1, 2, or 3 layers as dense while matching activated-expert dimensions to preserve FLOPs per token.
  • Arrangement of MoE and Dense Layers: At 3 × 10^20 FLOPs, the optimal dense-layer proportion shifts to approximately 2/60 or 3/60, whereas it is zero at lower budgets.
  • Compute Resource Allocation between Attention and FFN: Attention ratios between 30% and 40% are relatively stable, while broader adjustments have negligible influence on model performance.This range is described as optimal or near-optimal and aligns with mainstream open-source MoE defaults.

E Additional Evaluation Results of Ling-mini-beta

The Ling-mini-beta evaluation compares its performance with Dense-6.1B across datasets and categories, finding comparable results on most datasets.

  • Ling-mini-beta achieves comparable performance to Dense-6.1B on the majority of datasets.Figure 14 provides the overall and category-wise comparison.

F List of Experimental Models

The experiments vary MoE activation, granularity, shared-expert ratios, layer arrangements, and attention–FFN allocation through controlled configuration tables.

  • Activation-ratio experiments fix two activated experts while varying the total number of experts.
  • Granularity experiments fix the base architecture while varying total, activated, and shared experts together with expert dimension.
  • Shared-expert experiments fix 256 total experts and 12 activated pathways while varying specialized-to-shared expert allocation.
  • Layer-arrangement experiments fix 60 total layers and vary the numbers of dense and MoE layers to study their ratio and placement.
  • Attention–FFN allocation experiments hold the core MoE structure constant while varying model and expert dimensions.
Loading 2507.17702v4…