Source-linked AI summary
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, Claire Cui
TL;DR
Scaling dense language models increases performance but also raises computing and energy costs. GLaM uses sparsely activated Mixture-of-Experts layers to increase capacity while activating only a subset of parameters per token. Across 29 NLP tasks, GLaM achieves better overall performance than GPT-3 while using substantially less inference computation and training energy.
Problem
Further scaling dense language models requires substantial computing resources and energy.
Method
GLaM uses a sparsely activated Mixture-of-Experts architecture in which each token activates only a limited subset of experts.
Results
Across 29 NLP tasks, GLaM achieves better average performance than GPT-3 while using about half the inference FLOPs and one third the training energy.
Takeaways & Limitations
The results support MoE as a candidate for more efficient scaling of giant language models while maintaining high-quality NLP performance.
Takeaways & Limitations
MoE models require more devices because they contain more parameters, limiting resource accessibility and increasing serving costs when traffic is low.
Abstract
from arXiv · showhide
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
1. Introduction
GLaM addresses the rising cost of scaling dense language models with a sparsely activated architecture that improves efficiency while maintaining or improving performance across broad NLP evaluations.
- GLaM outperforms GPT-3 across 21 NLU and 8 NLG benchmarks on average while using about half the inference FLOPs and one third the training energy.
- Scaling language models has enabled few-shot and zero-shot generalization but is becoming prohibitively expensive and energy-intensive.
- GLaM uses sparse conditional computation to balance large model capacity with lower computation, activating 96.6B of its 1.2T parameters per token.The activated subnetwork is 8% of the total parameters.
- GLaM closes the performance gap between stereotypical and anti-stereotypical examples on WinoGender, suggesting less reliance on superficial statistical correlations.
- The authors present sparsity as a promising direction for scaling high-quality NLP models while saving energy costs.
2. Related Work
Related work establishes scaling-based in-context learning and sparsely gated architectures as foundations for GLaM, while positioning GLaM among pretrained language models by total and activated parameters.
- Pre-training and Fine-tuning: Pretraining and fine-tuning improved language understanding, but the approach requires task-specific fine-tuning.
- In-Context Few-shot Learning: GPT-3 demonstrated task-agnostic few-shot performance using text demonstrations without gradient updates.
- Sparsely Gated Networks: Mixture-of-Experts models use many weights while computing only a small subset of the computation graph at inference time.
- Table 2 compares pretrained models using total trainable parameters and activated parameters per input token.
- GLaM performs well without fine-tuning in the few-shot setting shared by GPT-3.
3. Training Dataset
GLaM is trained on a broad, high-quality corpus assembled by filtering web data and combining it with books, Wikipedia, forums, news, and public-domain social media conversations.
- The training corpus contains 1.6 trillion tokens and is designed to represent a wide range of natural language use cases.
- A text quality classifier filters webpages before they are incorporated into the final dataset.
- The final dataset combines filtered webpages with books, Wikipedia pages, forums, news pages, and public-domain social media conversations.
- Mixture weights are selected using component performance in a smaller model and to prevent small sources such as Wikipedia from being oversampled.
4. Model Architecture
GLaM replaces feed-forward sublayers in alternating Transformer layers with sparsely activated MoE layers, using learned gating to select experts for each token.
- Every other Transformer layer replaces its feed-forward component with an MoE layer composed of independent feed-forward experts.
- A softmax gating function models how well each expert can process the incoming input.
- Each token activates only two experts, giving the model greater capacity while limiting computation.
- During inference, the gating network dynamically selects the two best experts for each token.
- The architecture also uses per-layer relative positional bias and Gated Linear Units in non-MoE Transformer feed-forward sublayers.
5. Experiment Setup
The experiment evaluates GLaM variants across scales, compares sparse MoE and dense models under matched training settings, and uses zero-, one-, and few-shot benchmarks.
- Training setting: GLaM variants range from 130 million to 1.2 trillion parameters and are trained to study MoE and dense-model behavior on the same data.
- Training setting: Two selected experts balance predictive performance against training and serving efficiency, while more experts increase compute FLOPs per prediction.
- Training setting: The experiments compare MoE and dense architectures grouped by activated parameters per token, using shared learning hyperparameters across trained models.
- Training setting: Training uses sequences of up to 1024 tokens, batches of up to 1 million tokens, zero dropout, and Adafactor optimization.
- Evaluation protocol: The evaluation follows zero-, one-, and few-shot protocols across 29 datasets: 8 natural language generation and 21 natural language understanding tasks grouped into 7 categories.
- Evaluation protocol: Generative tasks use exact-match and F1 metrics with beam search, while understanding tasks select options using length-normalized log-likelihood.
6. Results
GLaM’s sparsely activated MoE models achieve strong few-shot performance while improving scaling, data, computation, and energy efficiency relative to dense models. Results also show that filtered pretraining data and larger expert capacity improve downstream performance.
- Comparison between MoE and Dense Models: GLaM (64B/64E) outperforms GPT-3 in 6 out of 7 task categories on average while activating roughly 96.6B parameters per token.This inference setting requires only half of GPT-3’s compute FLOPs for the same input.
- Effect of Data Quality: Filtered data performs consistently better than unfiltered data on both NLG and NLU tasks, with a larger filtering effect on NLG.The filtered webpages contain 143B tokens, compared with around 7T tokens in the unfiltered webpages.
- Scaling Studies: 8 NLG and 21 NLU tasks show that GLaM MoE models perform consistently better than GLaM dense models at similar effective FLOPs per token.MoE and dense models perform similarly at smaller scales, while MoE models outperform at larger scales.
- Scaling Studies: Adding more experts generally improves predictive performance for a fixed computation budget per prediction.This result supports scaling GLaM by increasing the size or number of experts while keeping activation sparse.
- Efficiency of GLaM: GLaM MoE models require significantly less training data than dense models of comparable FLOPs to achieve similar zero-, one-, and few-shot performance.GLaM (64B/64E) trained with 280B tokens outperforms GPT-3 trained with 300B tokens in 4 of 6 learning settings and matches it in the remaining setting.
- Efficiency of GLaM: 456 MWh is consumed by GLaM (64B/64E) after 600B training tokens, about 1/3 of GPT-3’s 1287 MWh energy cost.Training GLaM to reach similar or slightly higher scores than GPT-3 with 280B tokens consumed 213 MWh, or 1/6 of GPT-3’s energy cost.
7. Ethics and Unintended Biases
The paper evaluates unintended biases, coreference performance, and toxicity degeneration in GLaM, while emphasizing both the usefulness and ethical risks of large language models. Results show persistent associative biases, strong WinoGender performance, and prompt-linked toxicity behavior with important dataset and application caveats.
- Large language models lower barriers to prototyping and application development, but raise concerns about representation bias, training data, privacy, and environmental impact.The paper frames these opportunities alongside the need to assess harmful stereotypes rigorously.
- 7.1. Co-occurrence prompts: Associative biases remain evident across gender, racial, and religious identity prompts; “pretty” is strongly associated with “She” but not among the top-ten descriptions for “He”.The analysis generates sampled continuations and selects descriptive adjectives and adverbs without manual labeling.
- 7.2. WinoGender: Coreference resolution is evaluated because it supports applications including machine translation and question answering.The paper measures WinoGender in the one-shot setting to assess gendered coreference correlations.
- 7.2. WinoGender: GLaM (64B/64E) achieves 71.7% on WinoGender, compared with 64.2% for GPT-3, with equal 71.7% accuracy on stereotypical and anti-stereotypical examples.Accuracy is also close between “he” examples (70.8%) and “she” examples (72.5%).
- 7.3. Toxicity Degeneration: Model continuation toxicity closely follows prompt toxicity, reflecting strong prompt influence, while human continuation toxicity declines slightly as prompt toxicity increases.Figure 5 compares Toxicity Probability of the Prompt (TPP) with Toxicity Probability of the Continuation (TPC).
- 7.3. Toxicity Degeneration: Even for low-toxicity prompts, some generated continuations are likely to be judged toxic, and Reddit-derived prompts may not represent other domains.The paper notes that even 1 in 100 toxic suggestions could be problematic for applications.
8. Discussion
Sparse MoE models improve knowledge-oriented performance and training efficiency, but their larger total parameter counts increase hardware requirements and serving costs, especially under low traffic.
- MoE models perform better on knowledge-oriented tasks such as open-domain question answering than dense models with similar effective FLOPs.TriviaQA performance is presented as evidence of increased information capacity.
- Despite efficiency advantages, sparse models require more devices because they contain more total parameters, limiting resource accessibility and increasing serving costs when traffic is low.
9. Conclusions
GLaM uses sparse mixture-of-experts scaling to improve average few-shot performance across 29 NLP tasks while reducing computation demands. Additional experiments connect expert count with performance gains under fixed prediction-compute budgets.
- 9. Conclusions: GLaM achieves better average scores than dense counterparts with similar effective FLOPs and GPT-3 across 29 representative NLP tasks in zero-, one-, and few-shot learning.
- 9. Conclusions: The evaluation suite contains open-domain QA, inference, completion, Winograd-style, commonsense, reading-comprehension, and SuperGLUE tasks.
- 9. Conclusions: Increasing experts from 1 to 256 in 1.7B models barely increases activated parameters because of sparsity.The experiment varies experts per MoE layer while keeping the model’s active-parameter count nearly stable.
- 9. Conclusions: More experts generally improve predictive performance for a fixed computation budget per prediction.The paper attributes this gain to increased capacity and flexibility from more experts.
- 9. Conclusions: The 2D sharding algorithm partitions expert weights and activations across the TPU device network to avoid redundant data and computation.
D. Data Contamination
The paper checks pretraining–evaluation overlap and examines toxicity across model scales. It reports comparable contamination to prior work and largely scale-invariant toxicity behavior, while warning that low-prompt-toxicity generations can still be unsafe.
- D. Data Contamination: The contamination analysis measures evaluation examples sharing any n-gram with the pretraining corpus, varying n between datasets.
- D. Data Contamination: The number of validation examples appearing verbatim in training roughly matches prior work.Overlap statistics are reported in Table 6.
- E. Ethics and Unintended Biases: GLaM toxicity performance is fairly consistent across model sizes and MoE variants, although the smallest sparse 0.1B/64E model is noticeable and may be less stable.
- E. Ethics and Unintended Biases: The maximal expected toxicity probability of a continuation shows the same scale-invariant pattern, including for low-toxicity prompts.
- E. Ethics and Unintended Biases: The co-occurrence evaluation uses prompt templates covering gender, race, and religion categories.The prompts and category-specific associations are documented in Tables 7–10.
F. Energy Usage
GLaM uses substantially less training energy than GPT-3, with lower associated carbon emissions. The reported savings are attributed to the MoE architecture and computation-efficiency optimizations.
- Energy consumption: 213 MWh: GLaM training consumed one-sixth of GPT-3's 1287 MWh energy cost.The estimate uses 326W measured system power per TPU-v4 chip and a datacenter PUE of 1.11.
- Efficiency mechanisms: The reduced energy consumption is attributed to GLaM's MoE architecture and computation-efficiency optimizations from TPU-v4 hardware and GSPMD software.
- Carbon emissions: 18.7 net tCO2e: training GLaM with 280B tokens emitted far less than GPT-3's 552 net tCO2e.Complete GLaM training with 600B tokens consumed 456 MWh and emitted 40.2 net tCO2e.
G. Results on All Tasks for All Model Sizes
This section reports GLaM, GPT-3, Gopher, and dense-model results across 29 benchmarks under zero-, one-, and few-shot evaluation settings. It also includes tables examining gender, race, and religion-related co-occurrences in model responses.
- Bias-related analyses: Gender, race, and religion analyses tabulate response co-occurrences for prompts involving gender, race, and religious practitioners.
- All-task comparisons: 29 benchmarks: Table 11 compares GLaM (64B/64E), GPT-3, and Gopher across all reported tasks.
- Reference models: Computationally expensive Gopher and Megatron-NLG models are included as reference models in the comparisons.
- Zero-shot results: Zero-shot scores are reported for GPT-3 and different GLaM mixture-of-experts and dense models across all 29 benchmarks.
- One-shot results: One-shot scores are reported for GPT-3 and different GLaM mixture-of-experts and dense models across all 29 benchmarks.
- Few-shot results: Few-shot scores are reported for GPT-3 and different GLaM mixture-of-experts and dense models across all 29 benchmarks, with the number of shots tuned upward.