Efficient Large Scale Language Modeling with Mixtures of Experts

• Scaling language models improves accuracy and generalization but increases computational resource requirements, creating research and environmental concerns.Source: §1 · p. 1
• Sparse MoE models increase the number of learnable parameters without proportionally increasing computational costs.Source: §1 · p. 1
• The study trains dense and MoE models from hundreds of millions to more than one trillion parameters and compares them in zero-shot, few-shot, and fully supervised settings.Source: §1 · p. 2
• MoE models can match dense-model downstream performance at a fraction of the compute, with modest-budget models requiring almost four times less compute.Source: Figure 1 · p. 1, §1 · p. 2
• Sparse MoE models show competitive zero- and few-shot performance while using a fraction of the computation for training and inference.Source: §1 · p. 2
• Dense and sparse models exhibit different generalization behavior at scale, suggesting complementary behavior and a future research direction.Source: §1 · p. 2

• Transformer language models are pretrained by predicting, masking, or perturbing text and can be adapted to tasks through supervised fine-tuning.Source: §2.1 · p. 2
• GPT-3 showed that large language models can perform zero- and few-shot learning through in-context examples without fine-tuning, with these behaviors emerging or strengthening at scale.Source: §2.1 · p. 2, §2.3 · p. 3
• Conditional-computation strategies activate only a subset of a model for each input, reducing the computational cost of increasing model capacity.Source: §2.2 · p. 2
• Dense and MoE models are compared in roughly FLOP-matched configurations, with MoE layers using alternating dense and expert layers plus top-2 expert selection.Source: Table 1 · p. 3
• The paper contrasts zero-shot, few-shot, and fully supervised fine-tuning performance between dense and MoE models.Source: §2.3 · p. 3
• Large models require techniques including FP16 training, activation checkpointing, and fully sharded data parallel training because parameter storage can exceed one GPU’s usable memory.Source: §2.4 · p. 3

3.1 Models

• MoE models mirror dense configurations for approximately FLOPs-matched comparisons, using alternating dense and expert layers with top-2 routing across 512 experts.Each expert has capacity 2·B/E tokens; overflowed tokens pass through the residual connection.Source: §3.1 · p. 3
• Models are autoregressive decoder-only Transformers trained for 300B tokens with 2048-token contexts and Adam optimization.Training uses learning-rate warm-up and decay, weight decay of 0.01, and dropout of 0.1.Source: §3.1 · pp. 3–4

3.3 Evaluation

• Language modeling is evaluated with next-token perplexity on held-out in-domain data and The Pile's 22 out-of-domain subsets.The study reports each Pile subset's official test perplexity and their average.Source: §3.3.1 · p. 4, §3.3 · p. 4
• Perplexity evaluation chunks inputs into nonoverlapping 2048-token blocks, so the first tokens in each block lack preceding-block context.A sliding-window approach could produce better results but would be more expensive.Source: §3.3.2 · p. 4, §3.3.1 · p. 4
• Downstream evaluation compares zero-shot, few-shot, and fully supervised fine-tuning approaches on selected tasks from Brown et al. (2020).Few-shot results average 25 runs, while task selection emphasizes settings where GPT-3 showed scaling or zero-to-few-shot gains.Source: §3.3.2 · pp. 4–5

Language modeling perplexity

• MoEs outperform dense models across all language-modeling datasets, with larger gains in-domain than out-of-domain.They match dense models trained with 8–16x more compute in-domain, versus a 2–4x speedup on the Pile out-of-domain.Source: §4.1 · p. 6
• MoE efficiency decreases with scale, from roughly 4x less compute at 2–6 ZFLOPs to roughly 2x at 30 ZFLOPs.Source: §4.1 · p. 6

Zero-shot learning

• MoEs outperform dense models on every zero-shot dataset and model size, with speedups largest on HellaSwag and PIQA and more modest on ReCoRD and WinoGrande.Source: §4.2.1 · p. 7

Few-shot learning

• Few-shot improvements are smaller for MoEs than dense models: the 6.7B dense model gains 3.6 points to 69.3, while the 1.1T MoE gains 2.3 points to 70.1.Source: §4.2.2 · pp. 7–8

Supervised Fine-Tuning

• Supervised fine-tuning substantially improves dense models, but MoE fine-tuning worsens HellaSwag, PIQA, and WinoGrande while improving several other datasets.Where MoE fine-tuning helps, its accuracy approaches that of corresponding dense models; the largest models were omitted because of resource needs.Source: §4.2.3 · p. 8

• Sparse models up to 1.1T parameters offer better performance–computation trade-offs than dense models for language modeling, zero-shot learning, and few-shot learning.Source: §5 · p. 8
• At larger scale, the biggest sparse model outperforms its dense counterpart even though the dense model requires twice as much computation.The results support sparse MoE models as an alternative that saves computation and reduces model energy consumption.Source: §5 · p. 8

Bias and toxicity

• Bias and stereotyping are comparable between dense and MoE models, especially at scale, while larger models generally show worse performance.The authors identify further mitigation research as necessary and suggest that designed experts might offer greater controllability.Source: Ethical considerations · p. 9

Energy and environmental impact

• MoEs may reduce the energy and environmental costs associated with future language-model scaling by requiring less computation.The paper discusses training energy use separately and frames compute efficiency as a potential way to alleviate future environmental impact.Source: Ethical considerations · p. 9

Transparency and reproducibility

• The authors release code, pretrained models, data cards, and model cards to support research, replication, accountability, and more efficient exploration.These resources are presented as reducing the exploration burden for researchers and the environment.Source: Ethical considerations · p. 9

Configuration scope

• The study uses a single MoE configuration with dense and sparse models sharing GPT-3-like hyperparameters and model structure.This configuration may be suboptimal for scaling MoE models.Source: Limitations · p. 9
• The experiments do not vary MoE-specific choices such as the number of experts.The authors therefore do not establish how these design choices affect scaling performance.Source: Limitations · p. 9

Open questions

• The factors making some tasks and domains more favorable for MoEs remain unclear despite observed variation in performance gaps.This limits the study’s ability to explain when MoEs are most advantageous.Source: Limitations · p. 9

Full perplexity results

• Table 5 reports complete perplexity results for all different subsets of the Pile.The passage identifies the table’s coverage but does not provide individual perplexity values.Source: A Full perplexity results · p. 14

Fine-tuning procedure

• Fine-tuning uses fixed epoch counts by dataset and selects models according to validation-set accuracy.The schedules range from 6 to 100 epochs, depending on the task.Source: B Fine-tuning Settings · p. 14
• For datasets lacking validation sets, the training data are randomly divided into 80% fine-tuning and 20% per-epoch validation subsets.The same split is used when evaluation occurs on the validation set.Source: B Fine-tuning Settings · p. 14

Evaluation benchmarks

• The models are evaluated on StereoSet and CrowS-Pairs to examine whether sparse models behave differently from dense models regarding bias.StereoSet measures bias across multiple domains, while the reported metrics include LMS, SS, and ICAT.Source: C Understanding Potential Harms · p. 14

Evaluation metrics

• StereoSet defines LMS as meaningful-association preference, SS as stereotypical-association preference, and ICAT as a combined metric where higher values are better.For SS, scores near 50 indicate better performance.Source: C Understanding Potential Harms · p. 14

Interpretation caveat

• The two bias benchmarks have limitations, including unclear explanations of how some biases are measured, so their results require careful interpretation.This caveat applies to conclusions drawn from both benchmarks.Source: C Understanding Potential Harms · p. 14

Results by scale

• Both dense and MoE models generally obtain worse CrowS-Pairs scores as scale increases, with a statistically significant difference between the best and worst scores.The comparison uses the Stereotype Score metric and a bootstrap test with p < 0.05.Source: C Understanding Potential Harms · p. 14

Dense versus MoE models

• On StereoSet, comparable-FLOPs dense and sparse models perform comparably, while MoEs are slightly less biased on CrowS-Pairs without a statistically significant difference.Both model types also generally become more biased with increasing scale.Source: C Understanding Potential Harms · p. 14

Emission estimates

• Each GPU-day of training is estimated to produce 3.24 kgCO2e under the stated Azure A100 and regional carbon-efficiency assumptions.The estimate assumes a 400W GPU, carbon efficiency of 0.3 kgCO2e/kWh, and PUE of 1.125.Source: D CO2 Emission Related to Experiments · p. 14

Cost estimation

• Training-time estimates use 160 TFLOP/s per A100 GPU for dense models and 115 TFLOP/s for MoE models.The rates are based on observed training speeds for the largest models.Source: D CO2 Emission Related to Experiments · p. 14

MoE overhead

• MoE models have lower GPU utilization than dense models because all-to-all communication introduces additional overhead.The paper notes that implementation optimization could reduce this overhead.Source: D CO2 Emission Related to Experiments · p. 14

Scope and omissions

• The estimates omit infrastructure-manufacturing costs and exploratory pilot experiments.These exclusions limit the estimates’ coverage of total research costs.Source: D CO2 Emission Related to Experiments · p. 15
• Pilot experimentation is estimated to multiply total training cost by 2, because exploration and tuning commonly add substantial training runs.The authors trained and discarded pilot 6.7B dense and 1.1T MoE models during the project’s early stages.Source: D CO2 Emission Related to Experiments · p. 15

Approach

• Knowledge distillation trains small dense students to mimic larger dense or MoE teacher models.The approach is motivated by combining dense-model and sparse-model benefits.Source: E Knowledge Distillation · p. 16

Training setup

• Students use a weighted objective combining 25% standard cross entropy with 75% soft distillation loss.The distillation loss encourages students to reproduce the teacher’s logits; sequence length is reduced to 1024 tokens for faster experimentation.Source: E Knowledge Distillation · p. 16

Results

• Distilled students outperform a well-tuned dense baseline for both dense and sparse teacher models.The results are reported in Table 9, whose PPL metric is in-domain validation perplexity.Source: E Knowledge Distillation · p. 16, Table 9 · p. 16
• Students distilled from a 52B-parameter MoE teacher outperform students distilled from a 1.3B-parameter dense teacher.The dense teacher is twice as costly to train, while the MoE teacher’s advantages are partly transmitted to the dense student.Source: E Knowledge Distillation · p. 16

Scaling techniques

• Pure FP16 training, activation checkpointing, and fully sharded data parallelism make models beyond single-GPU memory feasible to train.These techniques are used because parameter storage for many models exceeds usable single-GPU memory.Source: F Techniques for Large-scale Training · p. 16

FP16 training

• Maintaining only 16-bit weights, optimizer states, and gradients reduces memory requirements by 50% without reducing model accuracy.The method discards 32-bit weights and dynamically rescales Adam states stored in 16-bit.Source: F Techniques for Large-scale Training · p. 16

Activation checkpointing

• Activation checkpointing saves memory by discarding selected forward-pass activations and recomputing them during backpropagation.The technique increases computation by 33%.Source: F Techniques for Large-scale Training · p. 17

Fully Sharded Data Parallel

• FSDP shards model parameters, optimizer state, and gradients across workers, enabling training of models with more than one trillion parameters.The paper wraps every transformer layer to balance communication efficiency against peak resident memory.Source: F Techniques for Large-scale Training · p. 17

FLOP accounting

• Training FLOPs are counted analytically with an additional forward pass assumed for activation checkpointing.Dense-model FLOPs depend on training tokens, layers, hidden dimension, sequence length, and vocabulary size.Source: G Counting FLOPs · p. 17
• MoE FLOPs add a feed-forward network at every other layer for top-2 routing, while routing-projection FLOPs are treated as negligible.The resulting quantity is independent of the number of experts.Source: G Counting FLOPs · p. 17