PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperparameter Scaling Laws Across MoE Sparsity

Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

arXiv:2609.08690v1cs.LGcs.AIcs.CL

TL;DR

Reliable hyperparameter transfer across increasingly sparse MoE models remains unresolved because existing scaling laws do not capture continuous sparsity effects. This paper derives unified laws incorporating activation ratio and finds they extrapolate across model scale and sparsity, predicting near-optimal hyperparameters on a held-out ultra-sparse target.

  • Problem

    Existing scaling laws do not characterize how optimal MoE hyperparameters vary continuously with sparsity, especially in the ultra-sparse regime.

  • Method

    The authors run controlled MoE pre-training experiments and compare candidate scaling forms using grouped validation across sparsity, compute, data, and expert granularity.

  • Results

    Activation ratio multiplicatively modifies power laws for optimal learning rate and batch size, and the unified form outperforms alternatives while extrapolating to near-optimal hyperparameters on a 12B-parameter target.

  • Takeaways & Limitations

    Treating activation ratio as an explicit predictor yields hyperparameter scaling laws that transfer across the tested MoE sparsity levels and expert granularities.

  • Takeaways & Limitations

    Evidence comes from one backbone, data mixture, optimizer, routing setup, and three expert-granularity or sparsity configurations, so broader transfer requires validation.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-embedding parameters, processing approximately 20 trillion tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, the optimal batch size follows a power-law relationship with training tokens $D$, whereas the optimal learning rate scales with training compute $C$ and remains robust to the allocation between model size and data. Across sparsity levels, the activation ratio $A$ enters both relationships as an additional multiplicative power-law factor. These observations lead to unified hyperparameter scaling laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from that of total expert count.

1 Introduction

The paper develops unified hyperparameter scaling laws for MoE models that explain how optimal learning rate and batch size vary with training scale and activation ratio, including ultra-sparse regimes. These laws reconcile conflicting prior findings, outperform existing alternatives, and extrapolate across sparsity and expert granularities.

  • Motivation: Prior studies disagree on whether MoE hyperparameters transfer from dense models or instead require larger batch sizes and lower learning rates, especially as sparsity reaches A = 1/64.Existing scaling laws do not characterize continuous changes in optima across sparsity.
  • Experimental scope: The study evaluates 1,800 pre-training runs across activated and total parameter scales, activation ratios, and training-token budgets, reaching 6B total parameters and approximately 20 trillion tokens.The experiments span six activated-parameter scales and cost 200,000 equivalent H800 GPU-hours.
  • Contributions: Activation ratio multiplicatively modifies the prefactors of power laws governing optimal batch size and learning rate across MoE sparsity levels.At fixed sparsity, batch size scales with training tokens, while learning rate scales with training compute.
  • Contributions: Unified scaling laws outperform existing alternatives, predict hyperparameters close to observed optima, and extrapolate jointly across sparsity levels and beyond development-data ranges.They also transfer robustly across the tested expert granularities.

2 Preliminaries

This section formalizes optimal learning-rate and batch-size selection across MoE sparsity levels and defines the scale variables used for analysis. It also describes the controlled sweep over model scale, training horizon, and activation ratio.

  • Formalization: The framework distinguishes activated parameters N, total parameters Ntot, training tokens D, per-token FLOPs M, compute C = MD, and activation ratio A, with smaller A indicating greater sparsity.It also tracks activated and total expert counts, Eact and Etot; N is treated as a parameter-scale descriptor, while C is analytically computed non-embedding training FLOPs.
  • Formalization: Optimal hyperparameters are defined jointly as the peak learning rate η and global token batch size B minimizing validation cross-entropy over a candidate search space.Because finite grids may miss the continuous optimum, analyses use both the observed optimum and a near-optimal set within 0.1% of minimum generalization error.
  • Scaling model: The analysis models each optimal hyperparameter h* as a power law of a scale variable X, with prefactor a and exponent b, equivalently a linear relation in log-log space.The formulation follows prior hyperparameter-scaling studies (Bi et al., 2024; Li et al., 2025; Ludziejewski et al., 2025; Tian et al., 2026).
  • Experimental setup: The controlled sweep varies six model scales, N, Ntot, D, and A ∈ {1, 1/4, 1/16, 1/32} while fixing the mixed data, 4,096-token sequences, backbone family, and Muon optimizer.For each group, the search covers η and B; fitting uses a shared main grid, while the held-out configuration uses a separate grid.

3 Optimal Hyperparameters Scaling Laws for Ultra-Sparse MoEs

The section establishes that optimal MoE hyperparameters require three predictive dimensions: compute C for learning rate, training tokens D for batch size, and activation ratio A for sparsity-dependent corrections. These findings motivate a unified scaling formulation with shared base exponents and multiplicative A-dependent factors.

  • Sparsity as an additional scaling dimension: Across sparsity levels, parameter counts, training tokens, and compute alone cannot explain shifts in (η∗, B∗); activation ratio A must be modeled explicitly.Matched activated or total parameter counts do not remove learning-rate shifts, and matched D, N, or C do not remove batch-size shifts.
  • Base scaling at fixed sparsity: At fixed sparsity, η∗ follows a power law in compute C and is insensitive to the allocation between per-token compute M and training duration D, while B∗ follows a power law in training tokens D.These results identify C and D as the respective base predictive variables for learning rate and batch size.
  • Sparsity as an additional scaling dimension: At fixed C, η∗ approximately follows η∗∝A^δη with δη > 0, while at fixed D, B∗ approximately follows B∗∝A^δB with δB < 0.Thus, smaller activation ratios lower the optimal learning rate and increase the optimal batch size.
  • Unified formulation: The fitted base exponents fluctuate around their means across activation ratios, supporting shared-exponent scaling with a multiplicative sparsity correction A^δh.The section notes that interaction forms remain possible and are compared in Section 4 rather than ruled out by four slice fits.
  • Mechanistic interpretation: A gradient-noise analysis explains the opposite sparsity trends: decreasing A reduces tokens per expert, increases expert-side noise, favors a larger global batch, and still lowers the optimal learning rate.Under balanced routing, each expert receives approximately AB tokens per step, while the optimal batch does not increase enough to keep AB∗ constant.

4 Fitting and Predictive Validation

The unified scaling laws are fit on development data and validated with grouped out-of-fold prediction, selecting the multiplicative form for comparable performance with fewer parameters. On a held-out target extrapolating jointly in activation ratio, data, and compute, controlled comparisons further support transfer across expert granularities and an activation-ratio effect beyond expert counts or total capacity.

  • Grouped cross-validation: The laws are fit only on activation ratios 1, 1/4, 1/16, and 1/32 across six activated-parameter scales, then tested with leave-one-activation-ratio-out and leave-one-active-scale-out folds.These folds refit optima and coefficients after removing complete loss surfaces, while absolute log-ratio errors evaluate hyperparameter-coordinate prediction rather than validation loss.
  • Fitting: Decreasing activation ratio lowers the optimal learning rate and increases the optimal batch size through positive and negative multiplicative sparsity exponents, respectively.The fitted coefficients use C = MD in non-embedding FLOPs and D and B in tokens, so coefficient values depend on these units.
  • Grouped cross-validation: The multiplicative law is selected as the working model because its predictive performance is comparable to alternatives while using one fewer parameter.For learning rate, all activation-aware families outperform scale-only; batch-size differences are smaller and grouping-dependent.
  • Held-out joint extrapolation: The held-out evaluation jointly extrapolates to N = 324M activated parameters, Ntot = 12B, A = 1/64, D = 159B, and C = 3 × 10^20 FLOPs after freezing predictions.Published-coefficient transfer and refitted-family comparisons are evaluated against the predicted setting on held-out training and validation loss surfaces.
  • Expert granularity and sparsity controls: At fixed activation ratio, doubling active and total expert counts does not materially change optimal hyperparameters, whereas changing A from 1/32 to 1/16 shifts both optimal learning rate and batch size downward.These controls indicate that the sparsity effect cannot be attributed solely to any single absolute expert count or total capacity.

5 Related Work

Prior work studies hyperparameter scaling through theory-driven parameterizations and empirical rules, including their transfer from dense models to MoEs. However, systematic scaling laws across sparsity levels remain missing, especially for frontier ultra-sparse MoEs.

  • Scaling Laws for Optimal Hyperparameters: Related work spans optimization-hyperparameter scaling and sparse MoE scaling, with theory-driven and empirically driven approaches.The µP framework and extensions use scale-aware parameterizations to preserve transfer across model-width changes.
  • Scaling Hyperparameters for MoEs: Recent studies examine dense-to-MoE transfer, reporting broadly robust learning-rate and batch-size laws alongside larger global batches and slightly lower learning rates for MoEs.These findings come from Step Law and other studies, including Li et al., 2025; Tian et al., 2026; Ludziejewski et al., 2025; Zhou et al., 2026.
  • Scaling Hyperparameters for MoEs: Existing analyses omit factors such as training horizon or cover only restricted sparsity ranges, leaving systematic hyperparameter transfer across sparsity levels unresolved.The gap is particularly important for frontier ultra-sparse MoEs spanning much wider ranges.

6 Limitations and Future Directions

The study’s evidence is limited to a narrow set of model, data, optimizer, routing, and expert-granularity configurations, while functional-form and extrapolation conclusions remain uncertain because the available groups and extrapolation targets are limited.

  • Experimental and evaluation scope: Evidence comes from one backbone, data mixture, optimizer, routing setup, and only three expert-granularity and sparsity configurations, so broader transfer requires validation.The untested dimensions include model scale, architecture, optimizer, training schedule, routing mechanism, and broader expert configurations.
  • Functional-form and extrapolation uncertainty: Overlapping cross-validation intervals and limited groups neither establish a significant advantage for the no-interaction model nor rule out activation-ratio-dependent scaling exponents.Joint extrapolation also uses only one target beyond the fitting ranges in A, D, and C, preventing isolation of extrapolation along each variable.

7 Conclusion

The conclusion identifies activation ratio A as an essential predictor of optimal learning-rate and batch-size shifts across ultra-sparse MoEs. Incorporating A into multiplicative power-law scaling yields transferable hyperparameter-selection rules.

  • Conclusion: Activation ratio A must supplement conventional scale variables to predict optimal hyperparameter shifts across sparsity levels.The conclusion frames sparsity as an explicit predictive dimension rather than solely an architectural attribute.
  • Conclusion: At fixed sparsity, optimal learning rate scales with compute C and optimal batch size scales with training tokens D.Learning rate remains stable across different M/D allocations when C is fixed.
  • Conclusion: Across sparsity levels, A modifies both scaling-law prefactors through a multiplicative power law, producing a unified hyperparameter prescription.This formulation turns sparsity into an explicit predictor for selecting hyperparameters when training ultra-sparse MoEs at scale.

A Additional Experimental Details

This section supplements Section 2.2 by detailing the fixed training controls, model scales, compute budgets, and hyperparameter search spaces used in the experiments.

  • A Additional Experimental Details: The section provides additional experimental details beyond the setup described in Section 2.2.
  • A Additional Experimental Details: It describes the fixed training controls used across the experiments.
  • A Additional Experimental Details: It specifies the model scales, compute budgets, and hyperparameter search spaces.

A.1 Training Controls · A.2 Model Configurations and Search Grids · A.3 Expert-Granularity-Controlled Configurations

The experiments hold core training controls fixed while varying model configurations, learning-rate and batch-size grids, and sparsity-related controls. Formal fitting uses specified activation ratios and scales, with separate held-out configurations for joint extrapolation and expert-granularity validation.

  • A.1 Training Controls: Training uses fixed Megatron, 4,096-token sequences, shared web/book/code data, and held-fixed engineering settings across experiments.Numerical precision and parallelism are not treated as study variables.
  • A.2 Model Configurations and Search Grids: Table 7 defines the formal fitting grid across four activation ratios, six activated-parameter scales, and compute budgets from 3 × 10^17 to 10^20 FLOPs.The formal analysis, family selection, cross-validation, and coefficient fitting exclude the held-out target.
  • A.2 Model Configurations and Search Grids: The main search evaluates η ∈ {5, 7, 10, 14, 20, 28, 40, 56} × 10^-4 and B ∈ {2^17, 2^18, ..., 2^23}, while the holdout uses narrower, shifted grids.The held-out learning-rate grid is η ∈ {3.6, 5, 7, 10, 14, 20} × 10^-4 and its batch-size grid is B ∈ {2^19, 2^20, ..., 2^23}.
  • A.2 Model Configurations and Search Grids: The held-out target has 324M activated and 12B total parameters and lies beyond fitting ranges in activation ratio, tokens, and compute.It is evaluated only after prediction coordinates are frozen and contributes to neither model selection nor coefficient estimation.
  • A.2 Model Configurations and Search Grids: All compute and Target FLOPs values use analytical non-embedding accounting, with C = MD and architecture-specific forward-and-backward FLOPs per token.The same accounting is used for reported fits, grouped validation, fixed-C slices, and held-out compute.
  • A.3 Expert-Granularity-Controlled Configurations: Expert-granularity validation compares three approximately 10M-activated-parameter configurations trained on 4.56B tokens, including controls sharing activation ratio or total expert capacity.The reference (2, 64, 384) and granularity control (4, 128, 192) share A = 1/32 and match activated and total capacity.

B Additional Experimental Results

This section reports supplementary sparsity slices and fitted coefficients for candidate functional families.

  • B Additional Experimental Results: Supplementary experiments examine additional sparsity slices and fitted coefficients across candidate functional families.

B.1 Sparsity Dependence Across Fixed-Scale Slices · B.2 Candidate Functional-Form Coefficients

Across fixed compute and token slices, sparsity-induced shifts in near-optimal learning rates and batch sizes persist across training scales. Among candidate functional forms, the additive family fits development data best in-sample, but the multiplicative law is selected for model selection.

  • B.1 Sparsity Dependence Across Fixed-Scale Slices: Sparsity-induced shifts in near-optimal learning rates and batch sizes persist across the examined fixed-compute and fixed-token training scales.The analysis extends across three fixed-C slices and three fixed-D slices rather than relying on a single reference scale.
  • B.1 Sparsity Dependence Across Fixed-Scale Slices: Figure 11 fits batch-size dependence on activation ratio A across fixed-token horizons of 4.7 × 10^9, 7.8 × 10^9, and 3.2 × 10^10 tokens.Highlighted markers show geometric means, while solid lines represent multiplicative power-law fits in A.
  • B.1 Sparsity Dependence Across Fixed-Scale Slices: The fixed-token analysis evaluates near-optimal batch sizes as functions of activation ratio A across multiple training horizons.Individual observations are shown as faint markers and aggregated observations as highlighted markers.
  • B.2 Candidate Functional-Form Coefficients: Table 9 reports fitted coefficients and base-2 log-space fit metrics for all four candidate functional families on target-aligned development surfaces.RMSE and R2 measure in-sample fit, with lower RMSE and higher R2 indicating better performance.
  • B.2 Candidate Functional-Form Coefficients: Although the additive family achieves the best in-sample metrics, the multiplicative law is selected for model selection.The reported in-sample metrics are explicitly distinguished from grouped out-of-fold errors used in the main text.
  • B.2 Candidate Functional-Form Coefficients: The candidate-family comparison evaluates RMSE and R2 in base-2 log space on the same target-aligned development surfaces.These descriptive metrics are distinct from the grouped out-of-fold errors used for model selection.

C Theoretical Analysis · C.1 Why Sparsity Increases the Optimal Batch Size · C.2 Why Sparsity Decreases the Optimal Learning Rate

A local noise model explains why increasing sparsity raises the optimal batch size but lowers the optimal learning rate. The effects are moderated by update frequency and shared parameters, under simplifying assumptions that explain signs rather than fitted coefficients.

  • C Theoretical Analysis: The analysis attributes both sparsity trends to reduced effective expert batches: as activation ratio A decreases, optimal batch size increases while optimal learning rate decreases.
  • C.1 Why Sparsity Increases the Optimal Batch Size: Each expert receives approximately AB tokens under balanced routing, so lower A increases expert-side gradient variance and favors a larger global batch.
  • C.1 Why Sparsity Increases the Optimal Batch Size: At fixed training-token budget D, larger batches reduce optimizer updates, while shared parameters still receive gradients from the full batch.
  • C.1 Why Sparsity Increases the Optimal Batch Size: The resulting model predicts a batch-size exponent with respect to D that is independent of A, while the optimal batch size rises sublinearly as A decreases.
  • C.2 Why Sparsity Decreases the Optimal Learning Rate: Lower effective expert batches increase update noise, so the optimizer favors a smaller learning rate as activation ratio A decreases.
  • C.2 Why Sparsity Decreases the Optimal Learning Rate: The sparsity-induced increase in optimal batch size only partially offsets the declining activation ratio, leaving the learning rate lower but typically shifting it modestly.
  • C.2 Why Sparsity Decreases the Optimal Learning Rate: The explanation assumes balanced routing, independent samples, and a local noise model; it captures empirical signs δB < 0 and δη > 0 without deriving fitted coefficients or requiring constant sparsity exponents.
Loading 2609.08690v1…