PaperlayerView exact revision on arXiv

Source-linked AI summary

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, Claire Cui

arXiv:2112.06905v2cs.CL

TL;DR

Scaling language models supports few-shot and zero-shot generalization, but further scaling is becoming prohibitively expensive and energy-intensive. GLaM uses a sparsely activated mixture-of-experts architecture; across 29 representative NLP tasks, it achieves better average scores than GPT-3 and dense models of similar effective FLOPs, while its largest model uses one third the energy of GPT-3 training.

  • Problem

    Further scaling for few-shot and zero-shot language-model generalization is becoming prohibitively expensive and energy-intensive.

    Source: §1 · p. 1

  • Method

    GLaM scales model capacity with a sparsely activated mixture-of-experts architecture that balances dense and conditional computation.

    Source: §1 · p. 1

  • Results

    Across 29 representative NLP tasks, GLaM achieves better average scores than similar-effective-FLOPs dense models and GPT-3 in zero-, one-, and few-shot learning, while its largest model uses one third of GPT-3 training energy.

    Source: §1 · p. 1, §9 · p. 9

  • Takeaways & Limitations

    The authors identify MoE as a strong candidate for future scaling of high-quality NLP models while saving energy costs.

    Source: §1 · p. 2

    Despite efficiency advantages, sparse models have more parameters, require more devices, and increase serving costs, especially when traffic is low.

    Source: §8 · p. 9

Abstract

from arXiv · show

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

1. Introduction

Scaling language models has enabled few- and zero-shot performance with fewer labeled examples, but further scaling is costly and energy-intensive. GLaM uses sparse conditional computation to scale model capacity while aiming for strong performance with lower compute and energy costs.

1. Introduction

  • About half the inference FLOPs and one-third the training energy accompany GLaM’s better average performance than GPT-3 across 21 NLU and 8 NLG benchmarks.The paper reports favorable comparisons across 29 public NLP benchmarks in zero-, one-, and few-shot settings.Source: Table 1 · p. 1, §1 · p. 1
  • GLaM’s largest model has 1.2T total parameters across 64 experts per MoE layer, while each token activates a 96.6B-parameter subnetwork.The activated subnetwork is 8% of the total parameters.Source: §1 · p. 1
  • GLaM’s data analysis finds that data quality should not be sacrificed for quantity when the goal is a high-quality autoregressive language model.The paper uses GLaM to study the importance of data.Source: §1 · p. 2
  • On WinoGender, the paper reports closing the performance gap between stereotypical and anti-stereotypical examples, suggesting GLaM may rely less on superficial statistical correlations.The authors describe this as the first such result to their knowledge.Source: §1 · p. 2
  • At scale, sparse decoder-only models outperform dense models with similar compute FLOPs in few-shot in-context learning, making sparsity a candidate direction for scaling NLP models while saving energy.The paper presents this as the first such result within the few-shot in-context learning setting at scale.Source: §1 · p. 2

2. Related Work

Related work spans transfer-oriented pre-training and fine-tuning, text-only few-shot prompting, and sparsely activated MoE models that scale capacity while computing only part of the network. GLaM is positioned among these approaches by its decoder-only architecture and few-shot evaluation without fine-tuning.

Language models

  • Neural language models support many NLP tasks, and word embeddings and paragraph vectors have generalized across tasks through transfer (Le & Mikolov, 2014).Source: §2 · p. 2

Pre-training and Fine-tuning

In-Context Few-shot Learning

  • GPT-3 and related work showed that scaling improves task-agnostic few-shot performance using text demonstrations without gradient updates (Brown et al., 2020; Shoeybi et al., 2019; Lieber et al., 2021; Wei et al., 2021).Source: §2 · p. 2

Sparsely Gated Networks

  • Switch-C and GLaM each have about one trillion trainable parameters, but Switch-C is encoder-decoder and mainly fine-tuned, whereas GLaM is decoder-only and performs well without fine-tuning in few-shot settings.Table 2 summarizes key differences between GLaM and related text-pretrained models.Source: §2 · pp. 2–3, Table 2 · p. 3
  • Mixture-of-Experts models can use many weights while computing only a small subset of the computation graph at inference time, with prior work scaling sparsely activated architectures (Shazeer et al., 2017; Hestness et al., 2017; Shazeer et al., 2018; Lepikhin et al., 2021; Kudugunta et al., 2021).The largest Switch-C model also has one trillion trainable parameters (Fedus et al., 2021).Source: §2 · p. 2

3. Training Dataset

GLaM’s 1.6-trillion-token training dataset combines a quality-filtered web corpus with books, Wikipedia, forums, news, and public-domain social-media conversations. Webpage quality filtering uses a classifier and Pareto sampling, while mixture weights are chosen using smaller-model performance and to avoid over-sampling small sources.

  • 1.6 trillion tokens make up a dataset combining filtered webpages with books, Wikipedia, forums, news, other sources, and public-domain social-media conversations.The social-media data was used by Adiwardana et al. (2020).Source: §3 · p. 3
  • A feature-hash linear classifier scores webpages against curated text, and Pareto sampling retains some lower-quality pages to limit classifier bias, as in Brown et al. (2020).Source: §3 · p. 3
  • Mixture weights reflect component performance in a smaller model and prevent small sources such as Wikipedia from being over-sampled.An overlap analysis found training/evaluation contamination roughly matched previous work (Brown et al., 2020).Source: §3 · p. 3

4. Model Architecture

GLaM replaces the feed-forward component in every other Transformer layer with sparsely activated MoE layers. A learned gate selects two experts per token, increasing model capacity while limiting computation.

  • Every other Transformer layer replaces its feed-forward component with an MoE layer made up of independent feed-forward experts.This lets GLaM use many more parameters in its MoE layers while activating only a subset for each input token.Source: §4 · pp. 3–4
  • For each token, the learned gate selects the two best experts, whose outputs are combined as a weighted representation.At inference, the gate dynamically chooses the experts; with E experts, the architecture offers O(E^2) combinations of feed-forward networks.Source: §4 · p. 4
  • GLaM also uses per-layer relative positional bias, gated linear units in non-MoE feed-forward sub-layers, and 2D sharding for large models.The gated linear unit multiplies two linear transformations component-wise, followed by a Gaussian Error Linear Unit activation.Source: §4 · p. 4

5. Experiment Setup

GLaM experiments compare dense and sparsely activated models across scales using shared training settings, then evaluate zero-, one-, and few-shot performance on a 29-dataset benchmark suite. The section specifies the model configurations, training procedure, evaluation protocols, and task-specific scoring rules.

5.1. Training Setting

  • GLaM variants span 130 million to 1.2 trillion parameters, with dense references matched to MoE models’ activated parameters per token and two experts selected per MoE layer.Selecting two experts balances predictive performance with training and serving efficiency; using more experts increases prediction FLOPs. GLaM (8B/64E) denotes an approximate 8B-parameter dense model with every other layer replaced by a 64-expert MoE layer.Source: §5.1 · p. 4

5.2. Hyperparameters and Training Procedure

  • All GLaM models share a training recipe with 1,024-token sequences, batches of up to 1 million tokens, and no dropout.Training uses Adafactor (Shazeer & Stern, 2018), a 256K SentencePiece vocabulary, and a 0.01-coefficient MoE load-balancing loss described in GShard (Lepikhin et al., 2021). The largest 64B/64E model was trained on 1,024 Cloud TPU-V4 chips.Source: §5.2 · p. 4

5.3. Evaluation Setting

  • Evaluation follows zero-, one-, and few-shot protocols, using development examples directly for zero-shot and random task-training examples as demonstrations for one- and few-shot.Demonstrations are concatenated with the evaluation example, following protocols suggested by Radford et al. (2018) and Brown et al. (2020).Source: §5.3 · p. 5
  • The benchmark suite contains 29 datasets—8 natural language generation and 21 understanding tasks—using Brown et al. (2020)’s task suite minus 7 synthetic tasks and 6 translation datasets.The datasets are grouped into 7 categories.Source: §5.3 · p. 5
  • Generative tasks use exact-match and F1 scores, while understanding tasks generally use accuracy based on option log-likelihood normalized by option length.Generative outputs are decoded with beam width 4; ReCoRD and COPA use non-normalized loss, MultiRC reports answer-option F1, and overall few-shot performance averages scores across datasets. TriviaQA also reports the one-shot submission’s testing-server score.Source: §5.3 · p. 5

6. Results

Across its model family, GLaM results show competitive or stronger task performance alongside sparse activation, and indicate that data quality and compute efficiency affect downstream results. The section evaluates comparisons with dense models, scaling trends, and resource use.

6.1. Comparison between MoE and Dense Models

  • GLaM (64B/64E) outperforms GPT-3 (175B) in 6 of 7 task categories on average while activating 96.6B parameters per token and requiring half GPT-3’s inference FLOPs for the same input.The comparison covers zero-, one-, and few-shot learning.Source: §6.1 · p. 6
  • By 8.6%, GLaM (64B/64E)’s one-shot TriviaQA result exceeds prior finetuned SOTA, and by 5.3% it beats few-shot GPT-3 on the testing server.The passage attributes the first comparison to Yu et al. (2022), whose approach uses additional knowledge-graph information.Source: §6.1 · p. 6, Table 5 · p. 6

6.2. Effect of Data Quality

  • Filtered-data training performs consistently better on both NLG and NLU tasks, with a larger filtering effect on NLG; the compared filtered and unfiltered webpages comprise 143B and around 7T tokens, respectively.The models use the same hyperparameters, with the filtered webpages replaced by unfiltered webpages in the second dataset.Source: §6.2 · p. 6, Figure 3 · p. 7

6.3. Scaling Studies

  • GLaM MoE performance scales with effective FLOPs per prediction and is consistently better than dense GLaM at similar FLOPs; on language-understanding tasks, MoE gains follow a similar trend.At smaller scales, MoE and dense models perform similarly, while MoE models outperform at larger scales.Source: §6.3 · p. 6, Figure 3 · p. 7
  • At a fixed computation budget per prediction, adding more experts generally improves predictive performance.GLaM MoE models activate only a fraction of their total parameters per prediction, allowing scaling through expert size or number.Source: §6.3 · pp. 6–7

6.4. Efficiency of GLaM

  • With 280B training tokens, GLaM (64B/64E) outperforms GPT-3 trained on 300B tokens in 4 of 6 learning settings and matches it on zero-shot NLG.GLaM MoE models need less data than dense models of comparable effective FLOPs to achieve similar performance; with the same training-data amount, they perform better, with the gap growing through 630B tokens.Source: §6.4 · p. 7
  • 456 MWh after 600B tokens was about one-third of GPT-3’s 1,287 MWh; 213 MWh at 280B tokens achieved similar, slightly exceeded scores at one-sixth of GPT-3’s energy cost.At similar downstream performance, sparse models take much less computation than dense models; the reported energy reduction reflects the MoE architecture and TPU-v4 and GSPMD optimizations.Source: §6.4 · p. 7

7. Ethics and Unintended Biases

The authors highlight the promise of lower-barrier language-model applications alongside ethical risks including representation bias, privacy, and environmental concerns. They examine associative bias, WinoGender coreference, and toxicity degeneration while noting the need for more rigorous harm evaluation.

7. Ethics and Unintended Biases

  • The authors identify risks including representation bias, privacy, and environmental concerns, and emphasize the need for more rigorous evaluation of harmful stereotypes.Examples of documented associations include gender and profession, negative sentiment toward racial and religious groups, and biases about people with disabilities. Measuring and mitigating harm is an active research area, but evaluation methods remain insufficiently rigorous.Source: §7 · pp. 7–8

7.1. Co-occurrence prompts

  • The analysis reports associative biases across dimensions: “pretty” is the most associated description for the term “She”, while it is not in the top-10 for the term “He”.For prompts referencing gender, religions, racial and ethnic identity, 800 outputs per prompt are generated using top-k sampling (k = 40) with temperature 1. An off-the-shelf POS tagger selects descriptive words, and no manual human labeling is used.Source: §7.1 · p. 8

7.2. WinoGender

7.3. Toxicity Degeneration

  • GLaM’s continuation toxicity probability closely follows prompt toxicity, while low-toxicity prompts can still yield continuations judged toxic.The human continuation baseline is relatively high at low prompt toxicity because the dataset samples sentences across the toxicity spectrum; batches of 25 continuations can include toxic suggestions even for low-toxicity prompts. The dataset is Reddit-derived and may not represent other domains, and applications may require much lower toxicity rates.Source: Figure 5 · p. 9, §7.3 · p. 9

7.2. WinoGender

  • GLaM (64B/64E) reaches 71.7% on WinoGender versus GPT-3’s 64.2%, with similar accuracy across gendered, stereotypical, and anti-stereotypical examples.Accuracy is 70.8% for ‘he’ examples and 72.5% for ‘she’ examples; both stereotypical and anti-stereotypical examples score 71.7%.Source: §7.2 · p. 9

7.3. Toxicity Degeneration

  • The toxicity analysis uses RealToxicityPrompts and the Perspective API to estimate continuation toxicity given prompt toxicity.For 10K randomly sampled prompts, the authors generate 25 continuations of up to 100 tokens using top-k sampling (k = 40) and temperature 1; empty continuations receive a toxicity score of 0.0.Source: §7.3 · p. 9

8. Discussion

GLaM’s sparsely activated MoE models show greater information capacity than dense models with similar effective FLOPs on open-domain QA, but their larger parameter count requires more devices and raises serving costs, especially at low traffic.

8. Discussion

  • MoE models perform better on knowledge-oriented tasks, and open-domain QA results such as TriviaQA indicate greater information capacity than dense models with similar effective FLOPs.This aligns with prior observations (Fedus et al., 2021).Source: §8 · p. 9
  • Despite in-context learning and training-efficiency advantages, sparsely activated models require more devices because they have more parameters, limiting resource accessibility and increasing serving costs especially at low traffic.Source: §8 · p. 9

9. Conclusions … C. Model Partitioning

GLaM uses sparse mixture-of-experts to achieve better average performance than dense counterparts with similar effective FLOPs and GPT-3 models across 29 representative NLP tasks in zero-, one-, and few-shot learning. Its largest model, GLaM (64B/64E), has 1.2 trillion parameters and achieves better average performance with one third of GPT-3’s training energy.

9. Conclusions

  • 1.2 trillion-parameter GLaM (64B/64E) achieves better average performance than GPT-3 models across 29 representative NLP tasks, using one third of GPT-3’s training energy.The paper proposes sparse mixture-of-experts models that also outperform dense counterparts of similar effective FLOPs.Source: §9 · p. 9

A. Benchmarks

  • The benchmarks include open-domain question answering, cloze and completion, Winograd-style, and common-sense reasoning tasks.Examples include TriviaQA, LAMBADA, Winograd, PIQA, and ARC.Source: A. Benchmarks · p. 17
  • Evaluation also covers in-context reading comprehension, SuperGLUE, and natural language inference tasks.The listed benchmarks include DROP, BoolQ, and ANLI R1–R3.Source: A. Benchmarks · p. 17

B. Scaling the Number of Experts

  • Increasing experts from 1 to 256 barely increases active parameters, and the models have almost identical FLOPs per prediction.The study starts from a 1.7B model whose MoE layers each reduce to a single feed-forward expert.Source: B. Scaling the Number of Experts · p. 17
  • At fixed computation per prediction, adding experts generally improves predictive performance over dense counterparts with similar FLOPs.Figure 6 compares average zero-, one-, and few-shot performance for models from 1.7B/1E to 1.7B/256E.Source: Figure 6 · p. 17, B. Scaling the Number of Experts · p. 17

C. Model Partitioning

  • GLaM partitions weights and computation with 2D sharding that exploits the TPU cluster’s device-network topology and places same-index experts across MoE layers on the same device.Wrapping repeated MoE Transformer modules in a while-loop control-flow statement reduces compilation time (Abadi et al., 2016a; Yu et al., 2018).Source: C. Model Partitioning · p. 17
  • The sharding divides expert weights along expert and hidden dimensions and activations along batch and model dimensions, avoiding redundant data and computation across devices.GSPMD’s compiler pass automatically determines sharding properties for the remaining tensors (Xu et al., 2021).Source: C. Model Partitioning · p. 17

D. Data Contamination · E. Ethics and Unintended Biases

The sections assess overlap between pretraining and evaluation data and analyze toxicity across model scales and prompt categories. Validation overlap roughly matches prior work, while low-toxicity prompts can still yield continuations judged toxic.

D. Data Contamination

  • Because GLaM was trained on over 1.6 trillion text tokens, the authors quantify evaluation overlap using the Wei et al. methodology, following Brown et al.Source: D. Data Contamination · p. 17
  • Validation examples appearing verbatim in pretraining roughly match prior work; Table 6 covers datasets also used in GPT-3 and defines dirty examples by any n-gram collision.Source: Table 6 · p. 18, D. Data Contamination · p. 18
  • For contamination analysis, overlap means an evaluation example has any n-gram found in pretraining, with n varying by dataset.Source: D. Data Contamination · pp. 17–18

E. Ethics and Unintended Biases

F. Energy Usage

GLaM used less training energy and produced lower emissions than GPT-3; the paper attributes its lower energy use to the MoE architecture and TPU-v4/GSPMD efficiency optimizations.

F. Energy Usage

  • 213 MWh for GLaM was 1/6 of GPT-3’s 1287 MWh energy cost (Patterson et al., 2021).The datacenter PUE was 1.11 during GLaM training; the paper attributes the reduction to MoE architecture and computation-efficiency optimizations from TPU-v4 hardware and GSPMD software.Source: F. Energy Usage · p. 18
  • 18.7 net tCO2e from GLaM’s 280B-token training compared with GPT-3’s 552 net tCO2e (Patterson et al., 2021); complete 600B-token training used 456 MWh and emitted 40.2 net tCO2e.The datacenter’s net emissions intensity at the time was 0.088 tCO2e per MWh.Source: F. Energy Usage · p. 19

G. Results on All Tasks for All Model Sizes

The section reports zero-, one-, and few-shot results for different model sizes across 29 benchmarks, with comparisons spanning GLaM, GPT-3, and reference models.

G. Results on All Tasks for All Model Sizes

Related papers

Loading 2112.06905v2…