Source-linked AI summary
The Cost of Training NLP Models: A Concise Overview
Or Sharir, Barak Peleg, Yoav Shoham
TL;DR
Training large NLP models has become increasingly expensive even as hardware and software reduce the cost of individual FLOPs. This overview analyzes how scale, training choices, repeated experimentation, and future trends shape those costs, estimating that an 11-billion-parameter T5 run exceeded $1.3 million at list price. The paper expects costs to keep rising with model ambition, while identifying efficiency improvements and cheaper compute as countervailing factors.
Problem
Large NLP-model training costs can reach millions, while the factors determining those costs remain difficult to quantify precisely.
Method
The paper reviews cost estimates and analyzes how model, data, training-volume, architecture, and experimentation choices affect NLP training expenses.
Results
An 11-billion-parameter T5 variant was estimated to cost well above $1.3 million for one list-price training run.
Takeaways & Limitations
NLP training costs are likely to keep escalating with more ambitious tasks, although cheaper compute and more efficient architectures may moderate the increase.
Takeaways & Limitations
The estimates rely on internal AI21 Labs data and cloud list prices, while inference costs are outside the article’s scope.
Abstract
from arXiv · showhide
We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).
1 Costs: Not for the faint hearted
Training costs for large NLP models can reach millions despite falling FLOP prices, because model scale and repeated experimentation multiply expenses. The estimates are approximate and depend on infrastructure and hidden costs.
- 38% lower ResNet-50 training costs were reported after combining TPUs with framework-level optimizations.The same change reduced training time by 27×, making larger-model training more feasible.
- $10k–$200k applies to a 340-million-parameter BERT model, while 1.5 billion parameters costs $80k–$1.6m.These figures use Wikipedia and Book corpora totaling 15 GB.
- Inference costs and post-training optimizations such as distillation are outside this article’s scope.The paper treats inference as related to, but separate from, training costs.
- Training-cost figures are based on internal data and cloud list prices, with discounts, preemptible systems, or on-premise implementations sometimes lowering them.The authors present the estimates as a general sense of costs rather than exact universal prices.
- More than $1.3 million was estimated for one list-price run of T5’s 11-billion-parameter variant.With several large-model runs and hundreds of small-model runs, the project estimate may reach $10 million.
2 Cost Drivers: Size Matters
The paper identifies dataset size, model size, and training volume as major cost drivers, while architecture and training scheme determine how FLOPs translate into expense. Hidden costs arise from stochastic reruns and hyper-parameter search.
- No known formula maps a given NLP performance target to a required FLOP count.The relevant variables have grown dramatically in recent years.
- Dataset size, model size, and total pre-training tokens are the three main variable categories affecting training cost.Model size is proxied by parameter count, while training volume is proxied by processed tokens.
- Total pre-training tokens equal input sequence length multiplied by training steps and batch size.This quantity captures training-volume contributions to FLOP cost beyond model size.
- BERT-style training uses fewer FLOPs than GPT-2 with comparable model, data, and step sizes.ELECTRA adds a generator model, increasing per-step cost while reducing the required number of steps.
- Multiple stochastic runs and combinatorially large hyper-parameter searches can multiply the cost of a single training episode.Running most experiments on smaller models first can reduce this hidden-cost multiplier.
3 The Future
The paper expects NLP training costs to continue rising as models, contexts, and tasks become more ambitious. It also identifies lower compute prices, efficient architectures, reduced leaderboard overfitting, and better use of data as countervailing trends.
- More ambitious NLP tasks may require larger models, longer contexts, more dimensions per token, and additional layers.Adding external knowledge sources could also expand model size.
- Raw-compute prices may continue falling because of increased competition.The paper cites large historical reductions in AWS prices as supporting context.
- Efficient architectures can reduce computational complexity or parameter counts, as illustrated by Reformer and ALBERT.Reformer reduces transformer attention complexity to O(n log n), while ALBERT uses factorization and weight sharing.
- Reducing the State-of-the-Art race could limit compute spent on repeated leaderboard runs and associated overfitting.The paper expects less of this practice because such overfitting has little value.
A NLP versus CV
The paper attributes lower computer-vision training costs to differences in data structure and task difficulty, which favor cheaper architectures and learning regimes than NLP. It cautions that this comparison is not definitive and may narrow as vision tackles more ambitious, less-defined tasks.
- Caveats: The analysis is presented as an initial discussion rather than a definitive account, and mega-scale computer-vision models have not yet become the norm because their gains may not justify added costs.The authors note limited agreement among consulted vision experts and cite FixResNet as an exception.
- Why CV training is cheaper: The paper identifies two fundamental cost differences: images versus sentences, and object recognition versus the less clearly analogous NLP tasks.Images are smooth, local, and iconic, whereas language is compositional and can couple words far apart; object recognition is relatively simple while NLP coreference remains unsolved.
- CNNs versus transformers: CNNs are cheaper-suited to computer vision because image locality and smoothness match convolution and pooling, whereas NLP relies on inherently more expensive transformers.The architectural difference follows the stated structural differences between images and sentences.
- Learning regimes: NLP devotes most training time to self-supervised language learning, requiring larger datasets and longer training cycles than supervised settings.The paper relates this imbalance to language structure and the nature of NLU and NLG tasks.
- Caveats: The comparison has important caveats: vision tasks beyond object recognition may lack smooth, local properties and require more elaborate models.Scene understanding is presented as an example of a less-defined task approaching NLP in complexity.