PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, Luke Zettlemoyer

arXiv:2609.11917v1cs.LGcs.CL

TL;DR

As unique human-written text becomes scarce, the paper examines whether repeated training data harms compute-efficient MoE models differently from dense Transformers. Through controlled variation of repetition, domains, and MoE architecture, it finds that MoEs overfit faster, while masking-based regularization mitigates but does not eliminate the gap from all-unique training.

  • Problem

    The effects of repeated training data on sparse Mixture-of-Experts models remain less studied than those on dense Transformers, despite MoE adoption for compute efficiency.

  • Method

    The authors vary repetition rates, domains and mixes, expert counts and granularity, model scales, regularization methods, and internal routing and specialization measures.

  • Results

    Across domains and mixes, MoEs overfit earlier and more rapidly than dense models; their degradation depends primarily on total parameters, while dropout and output masking reduce but do not eliminate the effect.

  • Takeaways & Limitations

    Sparsity’s performance benefit is conditional on the unique-data budget, and reducing repetition-driven expert specialization is a promising direction for future methods.

Abstract

from arXiv · show

As the supply of human-written text is exhausted, it has become standard practice to repeat language model training data. Prior work has studied data repetition for densely activated Transformers, but the effects of data repetition remains largely unexplored for recently dominant sparse architectures such as Mixture-of-Experts (MoE), despite their increased compute efficiency. We vary data repetition rates across single- and multi-domain data mixes, and across MoE settings, including expert count and granularity. We consistently find, for models ranging from 80M to 1B active (8.5B total) parameters, that MoEs degrade more rapidly under data repetition. This effect increases with sparsity, dictated by total rather than active parameters. While 80M dense models can repeat data over 8x with minimal degradation, MoEs instead begin to suffer at 4x, and deteriorate rapidly, ceding their performance benefits in all-unique data settings to underperform dense models after 32x. We experiment with existing regularization methods as a potential remedy. We find that some methods, such as dropout, can mitigate overfitting. In particular, with strong masking-based regularization, MoEs are able to outperform dense models even when data is repeated more than 64 times. However, no method fully matches the performance of all-unique training data. Finally, we analyze internal mechanisms correlated with MoE overfitting in high repetition regimes, and find that MoE routing universally stabilizes early in training, and that expert specialization correlates with overfitting to repeated data. In sum, our work addresses the adverse interactions between sparsity and data repetition: we present evidence for the core mechanisms of overfitting and its potential remediation, and suggest promising avenues for future methods to reduce over-specialization in model parameters by disrupting memorization patterns.

1 INTRODUCTION

This paper studies how data repetition interacts with sparse Mixture-of-Experts models, finding that sparsity’s benefits depend on the unique-data budget and can reverse under high repetition.

  • 80M MoEs clearly degrade at 4× repetition, compared with 8× for dense models, and cede their performance advantage at 32×.
  • Higher MoE sparsity causes more catastrophic overfitting, whose patterns depend primarily on total rather than active parameters.
  • Dropout and output masking reduce repetition-driven overfitting, allowing sufficiently regularized MoEs to outperform dense models even at 64× repetition.
  • MoE routers stabilize early in training, while expert specialization increases with repetition and contributes to over-specialized parameters.
  • MoEs outperform dense models on unique data but underperform at high data repetition rates across domains and data mixes.

2 BACKGROUND

MoE Transformers replace dense feed-forward layers with routed expert networks, while architectural choices and repeated-token budgets determine how sparsity and data exposure are measured.

  • 2.1 MIXTURE OF EXPERTS LANGUAGE MODELS: Each MoE layer replaces one FFN with parallel experts and a router that selects a subset for each token.
  • 2.1 MIXTURE OF EXPERTS LANGUAGE MODELS: The router assigns scores to experts and combines the selected top-k experts using their softmax weights, decoupling total from active parameters.
  • 2.1 MIXTURE OF EXPERTS LANGUAGE MODELS: Fine-grained expert granularity is defined as the ratio of expert-FFN dimension to dense-FFN dimension, with architecture varying in granularity, total experts, and active experts.
  • DATA REPETITION: A repetition rate R equals total training tokens T divided by unique tokens U when a corpus is iterated until the total token budget is reached.
  • REGULARIZATION: Dropout randomly drops units during training, while weight decay shrinks parameters and gradient clipping rescales unusually large gradients to support generalization.

3 EXPERIMENTS

The experiments vary repetition, model sparsity, parameterization, domains, and data quality under fixed training budgets. MoEs overfit more rapidly than dense models, with effects tied to total parameters and broadly consistent across domains and data-quality settings.

  • Data repetition effects across MoE architectures: At R = 32, dense models outperform MoEs after MoEs outperform them at R ≤16, as MoEs respond more dramatically to repetition.MoEs show a noticeable performance impact at R = 4, while dense models degrade slightly at R = 8 and sharply above R = 64.
  • Overfitting dynamics: At high repetition, training loss falls below 1E-2 while validation loss rises, consistent with memorization of repeated training data.For 80M dense models this occurs at R = 512, and for 200M dense models at R = 160.
  • Sparsity and parameterization: Higher total parameters produce a sharper repetition response in MoEs, even when active parameters remain fixed.Comparisons across expert count and granularity indicate that overfitting depends more directly on total than active parameters.
  • Data-to-parameter ratios: At T/N_a = 80, repetition overfitting follows trends very similar to T/N_a = 20, indicating limited dependence on total data budget.The experiments increase total tokens per active parameter fourfold without observing a corresponding change in repetition effects.
  • Cross-domain robustness: Across web crawl, code, academic, and encyclopedic text, MoEs begin underperforming dense models at R ∈[16, 32].The single-domain experiments show similar repetition patterns despite substantial semantic differences among domains.
  • Data quality: Filtering changes repetition behavior mainly for dense models, while MoEs show similar declines across DCLM-POOL and DCLM-BASELINE mixtures.Higher filtered-data proportions improve performance, but dense models with substantial unfiltered data may deteriorate faster as repetition increases.
  • Mixed-domain repetition: Repeated-domain effects can be dampened by unrepeated data, with larger DCLM proportions reducing degradation for repeated peS2o but not StarCoder.The peS2o mixture suggests a possible regularizing effect from unrepeated data, whereas DCLM + StarCoder degradation is largely proportion-invariant.

4 REGULARIZATION TECHNIQUES FOR DATA REPETITION

The study tests regularization methods to probe and reduce performance declines caused by repeated data. Dropout and masking methods mitigate extreme-repetition effects, while gradient clipping, weight decay, and router jitter show little clear benefit.

  • Dropout: High residual dropout dramatically reduces the impact of extreme data repetition across architectures, despite hurting low-repetition performance.The experiment evaluates dropout probabilities p ∈{0.0, 0.1, 0.2, 0.4}.
  • Optimization regularization: Gradient clipping and weight decay produce performance differences within variance or noise thresholds across the tested settings.Gradient clipping gives similar performance for all four thresholds, while AdamW weight-decay differences remain small.
  • Masking methods: FFN output masking provides a smaller low-repetition penalty than residual dropout and a similar high-repetition regularizing effect.FFN output masking zeroes the entire dense FFN or MoE output for a token with probability p.
  • Expert-level regularization: Expert dropout and expert output masking behave similarly to FFN output masking because they operate only on FFN outputs or activations.Expert output masking uses intermediate granularity by independently masking each token-selected-expert output.
  • Routing regularization: Router jitter shows no clear performance impact across the tested noise scales.The sweep uses ϵ ∈{0.0, 0.1, 0.2, 0.4}.
  • Data-mixture regularization: Mixing repeated data with unrepeated data may regularize training, particularly when the domains are semantically similar.In repeated peS2o mixtures, increasing the unrepeated DCLM share slows high-repetition degradation.

5 MECHANISTIC INVESTIGATION OF DATA REPETITION

MoE routing stabilizes early, while repetition increases expert specialization and overfitting; output-dropping regularization reduces specialization and repetition sensitivity without primarily changing router ossification.

  • 5.1 MOE ROUTER OSSIFICATION: Routing stability rises from 0.02 at step 200 to 60% at step 400 and above 95% by training end across MoE configurations.Higher repetition further increases end-of-training stability, supporting nearly stationary expert token partitions.
  • 5.1 MOE ROUTER OSSIFICATION: Dropout, FFN output masking, expert dropout, and expert output masking reduce repetition-driven overfitting, whereas weight decay, gradient clipping, and router jitter have minimal effect.The four effective methods operate by dropping outputs or activations.
  • 5.1 MOE ROUTER OSSIFICATION: At R = 64, dropout models overfit far less despite only modestly differing in router ossification, indicating routing plasticity is not the primary driver.The authors instead hypothesize that the functions learned by individual experts drive overfitting.
  • 5.1 MOE ROUTER OSSIFICATION: Repetition makes expert co-activation more uniformly distributed and increases router output magnitude without a predictable change in load imbalance.These routing statistics provide additional correlates of repetition but do not establish a direct mechanism for performance degradation.
  • 5.2 MOE EXPERT SPECIALIZATION: At R = 32, expert knockout effects increase 1.1× for 16 experts and 2.3× for 128 experts, showing stronger repetition-driven specialization with more experts.The increase mirrors the larger CE degradation observed under repetition at higher expert counts.
  • 5.2 MOE EXPERT SPECIALIZATION: Dropout consistently lowers median expert knockout cost across repetition rates, consistent with reduced dependence on any single expert.The proposed explanation is that dropout encourages multiple, more varied feature representations.

6 RELATED WORKS

Prior work established repetition-related degradation and explored quality–quantity trade-offs mainly for dense Transformers, while interventions remain comparatively limited.

  • 6 RELATED WORKS: Dense-model studies report that repetition can degrade held-out loss, with tolerable repetition depending on model scale, repeat count, data filtering, and compute.Reported findings include roughly four epochs comparable to unique data in one study and conflicting quality–quantity conclusions across others.
  • 6 RELATED WORKS: Prior work has modeled repetition jointly with data quality and mixture composition, but relatively few studies have addressed mechanisms for minimizing repetition effects.The paper situates its MoE analysis within this smaller intervention literature.
  • 6 RELATED WORKS: Intervention research has examined dropout and weight decay, finding reduced high-repetition overfitting but trade-offs or retuning requirements in low-repetition settings.One study reports a roughly 70% reduction in a fitted overfitting coefficient from stronger weight decay, alongside a single-epoch loss premium.

7 CONCLUSION

Across domains and architectures, repetition causes overfitting that is worse for sparse MoEs and primarily depends on total parameters; output-dropping regularization helps but does not recover all-unique performance.

  • 7 CONCLUSION: Increasing repetition causes overfitting across dense and MoE models, with sparse MoEs deteriorating earlier and faster and degradation primarily determined by total parameters.The pattern is robust across data domains, mixes, and filtering levels, while mixing repeated data with a sufficiently large non-repeated domain may regularize.
  • 7 CONCLUSION: Output-dropping methods reduce repetition sensitivity, but no tested method fully matches performance achieved with all-unique data.Effective methods include dropout, expert dropout, FFN output masking, and expert output masking; clipping, weight decay, and router jitter show no measurable effect.
  • 7 CONCLUSION: Mechanistic analyses link repetition to early-fixed routing and increased expert specialization, while dropout reduces specialization without changing router ossification.These findings support overly specialized parameters as part of the overfitting mechanism.
  • 7 CONCLUSION: The paper recommends further masking-based methods to reduce repetition-driven overfitting by decreasing parameter specialization.This recommendation follows the reported association between specialization and repetition-related degradation.

A.1 MODEL ARCHITECTURE

The appendix provides the architecture and hyperparameter tables used for the model studies.

  • A.1 MODEL ARCHITECTURE: Table 1 lists architecture details and parameter counts for the studied models.
  • A.1 MODEL ARCHITECTURE: Table 2 lists hyperparameters for models in Sections 3–5, including defaults and investigated alternatives.

A.3 TRAINING DATA SOURCES

The appendix documents the data sources, evaluation setup, constrained-data construction, and analyses used for training and validation. It also reports seed variability across selected dense and MoE settings.

  • The OLMoE Mix combines web crawl, code, scientific, encyclopedic, and other listed sources for training.
  • Training uses held-out language-modeling validation sets and downstream tasks including BoolQ, HellaSwag, and MMLU.
  • Data-constrained experiments select U unique tokens from each domain and repeat them for R shuffled epochs.
  • The routing analysis measures expert stability, expert knockout effects, and normalized co-activation entropy.
  • Across five seeds, higher repetition slightly increases standard deviation, with greater downstream-loss variance for MoEs at higher R.

B.3 TRAINING AND VALIDATION LOSS CURVES (§3.1)

Higher repetition drives training loss toward zero while validation loss rises, indicating overfitting; routing and regularization dynamics also change at high repetition.

  • Training and validation loss: Training loss approaches 0 at higher repetition rates while validation loss rises, indicating overfitting to repeated data.At sufficiently high repetition, validation curves can peak and then fall through double descent.
  • Routing diagnostics: At high repetition, routing imbalance and load-balancing curves become outliers or periodic at larger model scales.Routing imbalance becomes an outlier above R > 128 at small scale, while 1B-scale curves can partially cycle with the repetition period.
  • Routing diagnostics: Router z-loss cycles with data repetition, with higher repetition producing an early double peak and a lower final z-loss.The reported cycles appear at 200M and 1B scale.
  • Regularization: Dropout, FFN output masking, expert dropout, and expert output masking reduce repetition-induced overfitting, whereas weight decay, gradient clipping, and router jitter have minimal effect.The masking-based methods produce the strongest reductions among the regularizers studied.
  • Routing dynamics: Router stability rises rapidly early in training, with fewer than 25% of tokens changing top-1 experts between steps 400 and 600.Stability increases with repetition up to roughly R = 16–32, where the router appears to destabilize.
  • Expert specialization: Dropout increases routing stability and expert-coactivation entropy while decreasing expert specialization.These trends are reported across dropout settings for the 200M MoE configuration.

B.9 ADDITIONAL LANGUAGE MODELING TASKS

Additional held-out language-modeling results largely reproduce the main validation-loss trends, although performance varies across domains.

  • Additional language modeling tasks: The additional evaluations use models from §3 with the extended settings described in Appendix B.2.The reported results cover the additional held-out language-modeling tasks of Appendix A.4.
  • Additional language modeling tasks: Held-out language-modeling loss largely follows the validation-loss trends across 80M, 200M, and 1B active-parameter models.The comparison spans a wide variety of domains and performance depends on the exact domain.

B.10 DOWNSTREAM TASKS

Downstream task loss loosely follows validation loss, while task accuracy is noisy and remains near chance on most tasks even at 1B scale.

  • Downstream tasks: Accuracy remains near chance on all downstream tasks except Hellaswag, even at 1B scale.The evaluation reports cross-task CE loss and Hellaswag accuracy; other task accuracies remain near chance.
  • Downstream tasks: The downstream evaluations use models from §3 with the extended settings described in Appendix B.2.The results include CE loss on all tasks and accuracy on Hellaswag.
  • Downstream tasks: Downstream task loss loosely follows validation loss across 80M, 200M, and 1B active-parameter models.The downstream-task results contain random noise but show similar overall trends.
Loading 2609.11917v1…