PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

arXiv:2610.20247v3cs.CLcs.LG

TL;DR

Adding languages can worsen performance on existing languages, and prior scaling laws leave the interaction between expert capacity and coverage unresolved. The paper fits a repetition-aware joint scaling law to per-language losses. It finds that experts substantially reduce the loss penalty of broader coverage, while broader coverage favors expert-heavy configurations across a wider compute range.

  • Problem

    Whether experts reduce the loss increase from adding languages, rather than improving performance similarly across coverage levels, remains unresolved.

    Source: §1 · p. 1

  • Method

    A repetition-aware joint scaling law predicts per-language loss from active parameters, training tokens per language, expert count, and language coverage.

    Source: §1 · p. 1, §7 · p. 8

  • Results

    83%: at matched training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages relative to a dense model.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Broader language coverage increases the relative value of expert capacity under the studied resource budgets.

    Source: §6 · p. 8

    Conclusions apply to decoder-only Top-1 MoE models with fixed expert granularity, tokenizer, corpus, sampling scheme, and E ≤32; transfer to substantially different settings is outside scope.

    Source: §8 · p. 9

Abstract

from arXiv · show

Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.

1 Introduction

Adding languages can worsen performance on existing languages, while dense capacity increases computation for every token. The paper develops a joint scaling law and uses it to study whether experts change the cost of broader language coverage.

Motivation

  • The paper targets an unresolved interaction: whether experts reduce the loss increase from adding languages or improve performance similarly across coverage levels.Existing scaling laws treat expert capacity and language coverage separately.Source: §1 · p. 1

Approach and evaluation

  • A repetition-aware joint scaling law relates active parameters, training tokens per language, experts, and language coverage.The study fits it using 26,300 per-language loss observations from 64 configurations; held-out prediction error falls 28% versus JointMoE, the strongest evaluated baseline.Source: §1 · pp. 1–2

Research questions

  • At matched training tokens per language, 32 experts reduce the loss increase from expanding from 12 to 200 languages by 83% relative to a dense model.The same direction holds across seeds; broader coverage also extends the compute range over which expert-heavy configurations are preferred.Source: §1 · p. 2

Contributions

  • The study also finds that restoring average loss after expansion can leave many original languages worse off, so average quality alone is insufficient.Its contributions include controlled comparisons at matched training tokens per language and analysis of coverage-dependent expert allocation.Source: §1 · p. 2

2 Related Work

Prior scaling laws model language-level multilingual effects or MoE capacity, but not their interaction. This work positions its per-language joint law as addressing that gap.

Scaling laws

  • Scaling laws extend from parameter-and-data relationships to MoE expert counts and multilingual coverage, but prior laws leave language coverage outside MoE scaling or expert capacity outside multilingual scaling.JointMoE combines active parameters, training data, and experts; ATLAS separates tokens per language, transfer, repetition, and coverage.Source: §2 · p. 2

Multilingual capacity

  • Multilingual models trade positive cross-lingual transfer against interference, and language-specific capacity or learned MoE routing offers ways to reduce competition.Prior work does not measure how the loss increase from adding languages changes with expert count.Source: §2 · p. 2

Research gap

  • Existing multilingual laws cannot guide splitting capacity between active parameters and experts as coverage grows, motivating a joint law that predicts per-language loss.This matters because multilingual benefits are uneven across languages.Source: §2 · p. 3

3 Scaling Law Formulation

The formulation builds from a per-language loss target to a joint law, then adds flexibility for parameter-data coupling, coverage-dependent capacity returns, and repeated data.

3.1 Problem Setting

  • The target is a function predicting each language’s validation loss from active parameters, its training tokens, expert count, and language coverage.Averaging target-language losses gives configuration mean loss.Source: §3.1 · p. 3

3.2 Core Joint Law

  • The core law combines MoE and multilingual scaling by modeling per-language loss as a function of active parameters, tokens per language, experts, and coverage.It builds on a decomposition into an asymptotic floor and model- and data-limited terms; language-specific coefficients capture differences among target languages.Source: §3.2 · p. 3
  • Each scaling exponent can vary with expert count through its own intercept and slope.This parameterization is tested against alternatives.Source: §3.2 · p. 3

3.3 Predictive Refinements

  • The refinements replace three simplifying assumptions in the core law and are evaluated on held-out configurations.The model has 12 shared coefficients plus three per target language; a composition-aware extension groups other-language tokens by language family and script.Source: §3.3 · pp. 3–4
  • A factor N^ζ couples active parameters to the data-limited term, allowing model capacity to change the benefit of more training tokens per language.The core law otherwise treats these effects as separable.Source: §3.3 · p. 3
  • The parameter-scaling exponent is allowed to vary with coverage, extending the core law’s fixed exponent to model coverage-dependent returns on active parameters.The coverage term grows with log coverage above reference K0, and γN = 0 recovers the core law.Source: §3.3 · pp. 3–4
  • The repetition refinement replaces tokens per language with effective exposure, discounting later passes relative to the first.Its repetition parameter depends on total stored parameters, allowing models with different stored capacity to discount repetition differently.Source: §3.3 · p. 4
  • The selected law combines the three refinements while keeping language-coverage exponents independent of expert count.Source: §3.3 · p. 4

4 Experimental Setup

Experiments vary model capacity, language coverage, and training horizon, then fit and evaluate per-language loss predictions across configurations.

4.1 Model Architecture and Training

  • Models use Top-1 MoE layers with 1, 4, 16, or 32 experts; increasing experts keeps active parameters approximately fixed while increasing stored parameters.The dense baseline has one expert, and models share a 64k SentencePiece Unigram tokenizer.Source: §4.1 · p. 4

4.2 Data and Sampling

  • Training uses nested mixtures of 1, 12, 50, or 200 languages, with every smaller set contained in larger sets and coverage growing roughly fourfold at each step.Languages are sampled with temperature smoothing at exponent 0.3.Source: §4.2 · p. 4

4.3 Training Grid and Fitting

  • The training grid comprises 64 configurations and 400 endpoints, yielding 26,300 per-language loss observations.A configuration has one main-experiment seed; targeted additional seeds replicate the expert-by-coverage comparison, and held-out tests withhold an entire grid edge.Source: §4.3 · pp. 4–5

4.4 Compute and Evaluation

  • Evaluation predicts unigram-adjusted validation loss and reports logical FLOPs as approximately 6ND, excluding routing and communication overheads.The study also evaluates 14 multilingual benchmarks covering language modeling, grammaticality, reasoning, and classification.Source: §4.4 · p. 5

5 Results

The repetition-aware joint law predicts held-out losses more accurately than evaluated baselines, and the results show that expert capacity changes the cost of broader language coverage. Allocation and preservation outcomes also depend on whether quality is measured on average or for each language.

5.1 Scaling Law Evaluation

  • 28%: the final repetition-aware law reduces RMSE relative to JointMoE (Dt), the strongest baseline with alternative inputs; it also beats ATLAS (N) by 65%.The law is evaluated on held-out model-size, expert-count, or coverage configurations excluded from coefficient fitting.Source: §5.1 · pp. 5–6, Table 1 · p. 6
  • On the larger 1.15B-active-parameter stress test, accounting for repetition reduces RMSE from 0.31586 to 0.17298 and outperforms the evaluated Chinchilla and ATLAS baselines.The largest errors occur for languages whose exposure substantially exceeds their available unique data.Source: §5.1 · p. 6

5.2.1 Language Coverage Cost (RQ1)

  • At matched training tokens per language, 32 experts reduce the mean loss penalty of expanding from 12 to 200 languages from 0.0796 to 0.0135, an 83% reduction.Across alternative 12-language mixtures, the reduction ranges from 76.6 to 93.3%.Source: §5.2.1 · p. 6, Figure 2 · p. 7
  • Downstream accuracy reflects the loss interaction, but the effect varies by task: expert gains are larger at K = 200 for MultiBLiMP and smaller there for HellaSwag and XStoryCloze.For HellaSwag and XStoryCloze, expert expansion improves accuracy at both coverage levels.Source: §5.2.1 · p. 7

5.2.2 Capacity Allocation (RQ2)

  • Under a 0.4B total-parameter limit, the preferred measured allocation switches from 16 experts to 4 experts at 1.70 × 10^18 FLOPs for 12 languages and 2.75 × 10^18 FLOPs for 200 languages, a 1.62× delay.At 5.1B total parameters, the predicted E = 32 →16 switch is delayed 2.70× for K = 200 versus K = 1, with the K = 200 estimate beyond the observed training horizon.Source: §5.2.2 · p. 7, Figure 3 · p. 8

5.2.3 Per-Language Preservation (RQ3)

  • Expanding from 50 to 200 languages recovers the original mean loss at 1.92× reference compute, but only 36% of original languages match or improve their reference loss.A supported configuration matches or improves all 50 languages at 2.32× compute; matching training tokens per language requires approximately 1.75× compute.Source: §5.2.3 · pp. 7–8

6 Discussion

The discussion treats language coverage as a capacity-allocation variable and distinguishes average quality from preservation for every language. Its allocation analysis uses logical compute rather than hardware-specific cost accounting.

6 Discussion

  • Broader coverage increases the relative value of expert capacity under the resource budgets studied, so coverage belongs in multilingual MoE capacity allocation.Recovering average quality can still leave many individual languages worse off; planning should specify which preservation target matters.Source: §6 · p. 8
  • The scaling-law refinements improve extrapolation to longer runs when they account for repeated data, and improve predictions under changed mixtures when they account for language composition.Scaling laws should be evaluated against the decisions they support.Source: §6 · p. 8
  • Allocation uses logical compute F ≈ 6ND and abstracts from hardware-specific memory, communication, and parallelism costs.Those hardware costs are reported separately in Appendix H.Source: §6 · p. 8

7 Conclusion

The paper presents a joint scaling law for multilingual MoE pretraining and finds that expert expansion changes how broader coverage affects loss. It concludes that capacity, data, and language coverage should be considered together.

7 Conclusion

  • At matched training tokens per language, increasing from E = 1 to E = 32 reduces the loss penalty of expanding from 12 to 200 languages by 83%.The same direction appears across additional seeds, nested coverage transitions, and external text evaluation.Source: §7 · p. 8
  • The proposed joint law relates active parameters, training tokens per language, experts, and language coverage, improving held-out loss prediction.Coverage-aware and repetition-aware terms also improve prediction on a larger-model stress test.Source: §7 · p. 8
  • Broader coverage changes both the amount of capacity needed and how it should be allocated across active and expert capacity.The paper frames multilingual MoE pretraining as a joint allocation problem that also includes training tokens per language.Source: §7 · p. 8

8 Limitations

The study’s conclusions are bounded by its training design and evaluated setting. Held-out validation does not establish transfer to substantially different architectures, corpora, or training regimes, and logical-compute allocations omit practical hardware overheads.

8 Limitations

  • The main factorial grid uses one training seed, although selected expert and coverage settings were repeated with two additional seeds.The main effect remained consistent in direction across all three seeds.Source: §8 · p. 9
  • The conclusions cover decoder-only Top-1 MoEs with fixed expert granularity, tokenizer, corpus, and sampling scheme, and at most 32 experts.Changing expert count also changes stored capacity and the expert-aware training recipe.Source: §8 · p. 9
  • Transfer to substantially different architectures, corpora, or training regimes is outside scope, and actual hardware cost also depends on communication, memory, and parallelism overheads.The scaling law was assessed on held-out configurations spanning active parameters, experts, and language coverage.Source: §8 · p. 9

Ethical Considerations

The study uses existing web-derived corpora, whose biases and data-quality limitations may carry over, and does not evaluate trained checkpoints for deployment safety, privacy, or bias. Its appendices cover robustness, language composition, capacity allocation, external evaluation, routing, and physical training costs.

Ethical Considerations

  • Average multilingual quality can remain stable even when individual languages do not, so average performance is not evidence of equal quality or representation across languages or communities.No new human participants or annotations were used; experiments rely on existing web-derived corpora, which may inherit biases and data-quality limitations.Source: Ethical Considerations · p. 9

Appendix contents

  • The appendices cover robustness of the expert–language coverage response, language composition, capacity allocation, external evaluation, routing diagnostics, and physical training costs.Source: Appendix contents · p. 13

A Experimental Details

The study combines controlled multilingual MoE experiments with a repetition-aware joint scaling-law fit, evaluating per-language loss across model size, expert count, and language coverage.

A Experimental Details

  • The core grid spans four active-parameter sizes, four expert counts, and four coverage settings, with 64 configurations and 400 evaluated endpoints.Additional seed, alternative-mixture, and larger-model runs bring the study to 101 runs and 617 checkpoints.Source: A Experimental Details · p. 13

A.1 Data and language sets

  • Language sets K ∈{1, 12, 50, 200} are nested, with English alone at K=1 and two alternative K=12 mixtures used to test composition sensitivity.The K=200 set extends K=50 with 150 FineWeb2 languages; sampling upweights low-resource languages using an exponent of 0.3.Source: §A.1 · p. 13

A.2 Architecture and training

  • The architecture varies active parameters and expert count; E=1 is the dense control, while E>1 uses dropless Top-1 routing in feed-forward blocks.Source: §A.2 · p. 14, Table 4 · p. 15
  • The original expert-aware learning-rate rule achieved the lowest mean loss among tested alternatives and beat a fixed 2×10−4 rate in all nine configurations.The check used 190M active parameters and 2B tokens across E ∈{1, 16, 32} and K ∈{1, 12, 200}.Source: §A.2 · p. 15

B.1 Final law and fitting procedure

  • The final law models repeated passes with exponential saturation: the first pass counts fully and subsequent passes are progressively discounted in effective tokens per language.The fit uses configuration-balanced Huber loss, with each configuration weighted equally, and evaluates held-out configurations excluded from fitting.Source: §B.1 · pp. 15–16

B.2 Ablations and published-law comparisons

The ablations support a joint law that models expert–coverage interactions and repetition, while composition and simpler parameterizations offer distinct trade-offs. Its selected repetition-aware formulation also predicts the larger-model evaluation better than the strongest evaluated baseline.

B.2 Ablations and published-law comparisons

  • Adding language-coverage terms lowers K = 200 RMSE from 0.06000 to 0.05511, while mean RMSE rises slightly from 0.05029 to 0.05067.The composition-aware extension further reduces mean RMSE by 0.000652 relative to the otherwise matched law without composition.Source: §B.2 · p. 16, Table 7 · p. 17
  • Allowing both coverage exponents to depend on expert count gives the lowest mean RMSE, supporting the expert–language coverage interaction in the core law.The selected 12-coefficient repetition-aware law has comparable predictive accuracy to the more flexible 14-coefficient version.Source: §B.2 · p. 17, Table 8 · p. 17, Table 9 · p. 17
  • Using stored capacity rather than active capacity for the repetition discount lowers primary-holdout RMSE from 0.03533 to 0.03332 and larger-model RMSE from 0.19054 to 0.17280.The selected law uses stored capacity for the repetition discount.Source: §B.2 · pp. 17–18

B.3 Parameterization of the number of experts

  • Log-linear expert-count exponents are retained: the transformed-count alternative changes mean RMSE only from 0.03332 to 0.03298 and has no consistent advantage.Coverage dependence in β or in both α and β also fails to improve mean RMSE.Source: §B.3 · pp. 18–19

B.4 Prediction diagnostics

  • The repetition-aware law achieves the highest explained variance across evaluated settings, but R² is supplementary because it also captures variation between languages.Without repetition adjustment, prediction errors on the larger, longer-trained model are especially high for languages with substantial training-data repetition.Source: §B.4 · p. 19, §B.5 · p. 19

B.5 Larger-model evaluation

  • RMSE is 0.17298 for the final stored-capacity repetition law, versus 0.23349 for ATLAS (N), the strongest evaluated baseline on the larger-model run.The larger model was excluded from coefficient fitting.Source: §B.5 · p. 20, Table 14 · p. 20

C.1 Training seeds and learning rate

The expert–coverage interaction persists across seeds, learning rates, interpolation choices, model sizes, training durations, and alternative language expansions. Its magnitude and even the sign of coverage’s loss difference can depend on training duration and comparison setup.

C.1 Training seeds and learning rate

  • Increasing experts from 1 to 32 reduces the loss increase from expanding 12 to 200 languages by 83.6%, 82.6%, and 66.4% across three seeds; at a fixed learning rate, the mean difference falls from 0.0708 to −0.0003.The seed comparisons match training tokens per language.Source: §C.1 · p. 20, Figure 5 · p. 20

C.2 Token matching and interpolation

  • Excluding each of the 12 languages in turn leaves the reduction in the loss increase from E = 1 to E = 32 between 77.8% and 89.3%.Comparisons use overlapping token ranges, interpolate K = 12 loss at matched training tokens per language, and weight four model sizes equally; alternative interpolation schemes preserve the main conclusions.Source: §C.2 · p. 21, Figure 6 · p. 21

C.3 Sensitivity to model size and training duration

  • At a K = 200 training horizon of 4B tokens, increasing E from 1 to 32 reduces the matched-token loss difference at both the smallest and largest active model sizes.At equal total training tokens and 589.8M active parameters, the E = 1 → 32 advantage is 0.1680 for K = 12 and 0.1618 for K = 200 at 4B tokens, but 0.1098 and 0.1960 at 32B tokens.Source: §C.3 · p. 21, Table 15 · p. 21

C.4 Other nested language expansions

  • Across six nested language expansions, the mean loss difference is smaller with 32 experts than with a dense model in all 23 supported model-size and expansion comparisons.The minimum reduction is 0.00655; English-only comparisons have less overlapping token ranges, especially for K = 1 → 200.Source: §C.4 · p. 21, Table 15 · p. 21

C.5 Resource-group robustness

  • At K = 200, E = 1 → 32 lowers mean loss by 0.1960, 0.2010, and 0.2181 for high-, medium-, and low-resource languages, respectively.For K = 50 → 200, the loss difference is smaller with E = 32 in every resource group.Source: §C.5 · pp. 21–22, Figure 7 · p. 22, Table 16 · p. 22

C.6 Alternative K = 12 compositions

  • Across three alternative K = 12 mixtures, increasing E from 1 to 32 reduces the loss increase from expanding to 200 languages by 76.6–93.3%.The effect has the same direction in every seed and evaluated horizon, and persists with alternative interpolation schemes.Source: §C.6 · p. 22, Table 17 · p. 22

D Language-Composition Analysis

The analyses test prediction on changed language mixtures, capacity allocation as coverage grows, downstream benchmark interactions, routing patterns, and physical training costs. Broader coverage shifts preferred capacity allocation, while loss benefits do not have a consistent routing or downstream-accuracy counterpart.

D Language-Composition Analysis

  • At 16B tokens, the repetition-aware law predicts loss on alternative mixtures with RMSE 0.02649, versus 0.10246 for JointMoE (D), a 74% reduction.On the seven shared languages, adding composition reduces mixture-contrast RMSE from 0.01708 to 0.01300; repetition and composition provide complementary predictive benefits.Source: D Language-Composition Analysis · pp. 22–23, Table 18 · p. 23, Table 19 · p. 23

E.1 Observed allocation crossover

  • Under a 0.4B total-parameter budget, the compute crossover between 92.8M active parameters with 16 experts and 190.3M with 4 experts rises from 1.70 × 10^18 FLOPs at K = 12 to 2.75 × 10^18 at K = 200.The K = 200/K = 12 crossover ratio is 1.62; configurations with 32 experts also remain preferred in the evaluated ranges under 0.7B and 2.2B budgets.Source: §E.1 · p. 23

E.2 Fitted allocation surfaces

  • At a 5.1B-parameter budget, the fitted E = 32 → 16 switch moves from 4.51 × 10^19 FLOPs at K = 1 to 1.22 × 10^20 at K = 200, a 2.70× delay.The K = 200 switch is extrapolated beyond observed training.Source: §E.2 · pp. 23–24, Figure 8 · p. 24

E.3 Preserving the original languages

  • For K = 50 → 200, restoring mean loss requires approximately 1.92× compute but preserves only 18 of the original 50 languages; preserving all 50 requires approximately 2.32×.These estimates use a 2 × 10^18-FLOP reference and 2.2B total-parameter budget.Source: §E.3 · p. 24, Figure 9 · p. 24

F.1 Tasks and scoring

  • The 14-benchmark evaluation includes multiple-choice, minimal-pair, classification, and monolingual text-likelihood tasks; FLORES-200 and NTREX-128 measure likelihood rather than translation quality.The comparisons hold active parameters and total training tokens fixed, not training tokens per language.Source: §F.1 · p. 24, Table 20 · p. 25

F.3 Expert–coverage interaction on shared languages

  • FLORES and NTREX show larger E = 1 → 32 loss improvements at K = 200 than at K = 12 across all four active sizes and all three 190M seeds.For the default seed, expert expansion reduces the loss increase by 22–57% on FLORES and 20–62% on NTREX, while downstream accuracy interactions are task-dependent.Source: §F.3 · p. 25, Table 21 · p. 26

G Routing Diagnostics

  • At 190M active parameters, the loss interaction has a consistent direction across three seeds, but the corresponding language–expert mutual-information interaction does not.Overall, diagnostics do not reveal a consistent association between stronger language-dependent routing and the observed loss benefits.Source: G Routing Diagnostics · p. 25

H Physical Training Costs

  • Median GPU-hours rise to 1.17, 1.44, and 1.69 times the E = 1 cost for E = 4, 16, and 32, respectively.Mean model FLOP utilization falls from 35.7% at E = 1 to 22.0% at E = 32; these measurements reflect the implementation, not an inherent MoE efficiency limit.Source: H Physical Training Costs · p. 27

Related papers

Loading 2610.20247v3…