What Language Model to Train if You Have One Million GPU Hours?

1 Introduction

• The paper situates this work in a field where emergent capabilities such as zero-shot generalization are associated with scale and scaling laws relate parameters, data, and compute.The Transformer is described as a scalable basis for practitioners to build on.Source: §1 · p. 1
• The study seeks an architecture and training setup that make the best use of a fixed 1,000,000 A100-GPU-hour budget for multilingual BLOOM.Because training large language models is costly, the authors say they cannot exhaustively explore all possible models.Source: §1 · p. 1
• The findings are presented as relevant to practitioners building 1–10B models, whose performance the authors say can improve from these findings.Source: §1 · p. 1
• The authors make pretrained models, code, and weekly meeting notes available to support reproducibility and openness.Source: §1 · p. 2
• At 1.3B parameters, the authors study how pretraining corpora, positional embeddings, activation functions, and embedding norm affect zero-shot generalization.They also examine massive multilinguality and describe how they draft the architecture for 176B-parameter BLOOM.Source: §1 · p. 2

2.1 Architecture and Pretraining Objective

• All models use a decoder-only Transformer pretrained with an autoregressive language-modeling objective.The paper follows a recommendation to train this model first, with possible later adaptation and fine-tuning.Source: §2.1 · p. 2

2.2 Experimental Setup

• Unless noted otherwise, experiments use 1.3B-parameter models pretrained on 112B tokens, with GPT-2 architecture and GPT-3 hyperparameters.The authors chose this scale as a compromise between compute cost and measuring zero-shot performance above random chance, aiming for conclusions transferable to 100B+ models.Source: §2.2 · p. 2
• The models use full attention rather than GPT-3’s alternating sparse attention, which the authors say saves negligible compute at 100B+ scale because feed-forward layers dominate compute.For the final model’s d > 10,000 and nctx = 2,048, the attention-related term in the cited compute estimate is negligible when 12d >> nctx.Source: §2.2 · p. 2

2.3 Evaluation Benchmarks

• The study measures zero-shot generalization on diverse downstream tasks because language-modeling loss may not align with task performance and may not be comparable across objectives or tokenizers.The evaluation uses the EleutherAI harness across 27 tasks; its random baseline is 33.3%.Source: §2.3 · p. 3

2.4 Baselines

• GPT-Neo, a 1.3B decoder-only autoregressive model trained on the Pile, and GPT-3 models Babbage and Curie accessed through the OpenAI API serve as baselines.The authors assume Babbage is 1.3B and Curie is 6.7B, but note that API secrecy prevents confirming the models’ identities and makes reported pretraining-token counts uncertain.Source: §2.4 · p. 3

3.1 Corpora

• The comparison covers OSCAR v1, a multilingual filtered Common Crawl corpus; AllenAI’s filtered C4 replication; and The Pile, which adds curated cross-domain sources such as academic texts and code.For The Pile, training also extended to 300B tokens to compare with GPT-3 models and validate against GPT-Neo.Source: §3.1 · p. 4

3.2 Results

• A 13B model trained on OSCAR for 300B tokens underperformed OpenAI API’s 6.7B model, which the authors attribute to OSCAR’s low-quality English data.The comparison is reported as transferring the corpus finding to larger scales.Source: §3.2 · p. 4
• The Pile-trained model outperformed GPT-Neo trained on the same dataset, while the 1.3B model remained 1.6% below the 1.3B OpenAI API model.The authors say the GPT-3 training dataset is unavailable and unreproducible, preventing further investigation of the difference.Source: §3.2 · p. 4
• Diverse cross-domain pretraining data combining web crawls with curated high-quality sources improves zero-shot generalization over Common-Crawl-only datasets; the Pile performs best, C4 is a close second, and OSCAR significantly trails.Source: §3.2 · p. 4

4.1 Positional Embeddings

• ALiBi significantly outperforms learned, rotary, and no-position alternatives for zero-shot generalization.The comparison uses models trained on OSCAR for 112 billion tokens; the results do not rely on sequence extrapolation.Source: §4.1 · p. 4, Table 2 · p. 4
• ALiBi can extrapolate effectively beyond the training sequence length, whereas rotary embeddings cannot.The authors also find that autoregressive models without positional information remain competitive, using the causal attention mask to leverage token location.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 Activation Functions

• SwiGLU slightly outperforms GELU for zero-shot generalization, but the final model initially used GELU because SwiGLU showed lower throughput.The overhead was associated with a feed-forward hidden size of 5,456 that was divisible by neither GPU warp size nor the number of streaming multiprocessors; the authors recommend SwiGLU for future models.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Embedding Norm

• Adding layer normalization after the embedding layer significantly reduces zero-shot generalization, though it may be valuable if it improves stability during training above 100 billion parameters.The authors note that such models are notoriously unstable and require considerable engineering effort to keep stable.Source: §4.3 · p. 5

Training data

• The multilingual training data uses OSCAR, with sampling probabilities that downsample frequent languages and upsample less frequent ones so all are represented.Source: §5 · pp. 5–6

Results

• The multilingual 1.3B model achieves zero-shot XNLI performance in line with XGLM, despite training on 112B tokens from 13 languages versus XGLM’s 500B tokens from 30 languages.The study also reproduces XGLM-7.5B results to validate its evaluation setup.Source: Table 5 · p. 6, §5 · p. 6

Multilingual evaluation

• English prompts are used for non-English XNLI examples, making the evaluation a test of cross-lingual zero-shot generalization without finetuning.Source: §5 · p. 6

English-only evaluation

• Multilingual pretraining significantly lowers accuracy on English zero-shot benchmarks compared with English-only pretraining on OSCAR for 112B tokens.Source: Table 6 · p. 6, §5 · p. 6

Language-specific scaling laws

• For smaller models, loss rises late in training for Malayalam, Odia, and Telugu; the authors interpret this as insufficient capacity and possible catastrophic forgetting, while the largest model’s loss decreases smoothly across languages.Scaling behavior is similar across sufficiently represented languages, with differences mostly in loss offsets.Source: §5 · pp. 6–7

6 Scaling to 176B parameters

• The team allocates 18 weeks on 52 nodes of 8 A100 GPUs, with four nodes reserved as spares, yielding 1,161,216 A100-hours and a rounded training budget of 4,500 PF-days.The rounded budget includes an approximately 10% margin for downtime and inefficiencies.Source: §6 · p. 7

6.1 Final Model Architecture

• A target of 300–400B training tokens constrains the model to about 160–200B parameters under the C = 6ND approximation.The token target is intended to avoid allocating too few tokens to low-resource languages.Source: §6.1 · p. 7
• The authors estimate a depth of 70–80 layers and prioritize high GPU throughput, noting that wide, shallow models are typically easier to parallelize across nodes up to memory constraints.Source: §6.1 · p. 7
• The team benchmarks 20 configurations and chooses among three that best fit its guidelines and offer high throughput.Source: §6.1 · p. 7
• Configuration (3) is selected for the 176B model: it provides 3% higher throughput than configuration (2) and is shallower, reducing inference latency.Configuration (1) is rejected because its attention heads are much larger than those in other models in the literature.Source: §6.1 · p. 7

Optimal scaling

• For the available compute budget, Hoffmann et al.’s scaling laws would suggest a 50B model trained for a trillion tokens, but limited high-quality multilingual data and ROOTS size would have made that recommendation difficult to follow.Figure 1 reproduces Kaplan et al. (2020) and does not account for the learning-rate schedule suggested by Hoffmann et al. (2022).Source: §7 · p. 7

Other hyperparameters

• The experiments cover positional embeddings, activation functions, and embedding norm, leaving alternatives such as attention mechanisms and optimizers for future investigation.Source: §7 · pp. 7–8

Efficient fine-tuning

• The study evaluates zero-shot use, not efficient finetuning, which the authors note may lead to different conclusions.Source: §7 · p. 8

8 Conclusion

• Complementing Common Crawl with high-quality cross-domain curated data can boost zero-shot generalization; the ablations identify ALiBi as the position-embedding choice, confirm SwiGLU’s potential, and find that embedding normalization can trade off against zero-shot generalization.Source: §8 · p. 8
• The authors identify a candidate architecture for BLOOM 176B and outline the reasoning behind its architectural parameters, including model shape.Source: §8 · p. 8, Table 8 · p. 9

A Open artefacts: models, code, and logs

• The project was conducted openly and produced open-access artefacts, including models, code, and weekly meeting notes.Source: §8 · p. 8, A Open artefacts: models, code, and logs · p. 14

8 Conclusion

• The authors hope the work will help practitioners understand modeling decisions and that its transparency will accelerate similar future work.Source: §8 · p. 8

B Multilingual scaling laws

• For sufficiently represented languages, scaling-law behavior is similar and differences are mostly in loss offsets; for poorly represented Basque, Indic, and Niger-Congo languages, fits differ mostly in offset rather than exponent.Source: Table 9 · p. 14

• The EAI harness evaluation covers the tasks and random baselines listed in Table 10.Source: Table 10 · p. 15

• Table 11 reports throughput and memory usage by model size, using equal pipeline-parallelism slots for embeddings and Transformer layers.This slot allocation matters because multilingual embeddings are large (250k).Source: Table 11 · p. 16

• The final three columns report average EAI Harness results across different billion tokens trained.Some table rows are duplicated for ease of reading.Source: Table 12 · p. 17
• The table entries include datasets, public model names, embedding types, activation functions, and embedding-norm settings.Source: E All Results · p. 18