Source-linked AI summary
Energy and Policy Considerations for Deep Learning in NLP
Emma Strubell, Ananya Ganesh, Andrew McCallum
TL;DR
Large NLP models deliver accuracy gains but require substantial computational, financial, and environmental resources. This paper estimates those costs across NLP models and development workflows, finding that modest gains can demand substantial compute and emissions.
Problem
State-of-the-art NLP models require substantial specialized computation, limiting financial access and creating significant energy and environmental costs.
Method
The paper estimates training energy, carbon emissions, and financial costs across popular NLP models and analyzes complete development and tuning resources for LISA.
Results
At least $150k in compute produced only a 0.1 BLEU gain for NAS, while LISA development required 9998 GPU-days.
Takeaways & Limitations
Researchers should report computational requirements and prioritize more efficient algorithms, hardware, and hyperparameter-search methods.
Takeaways & Limitations
Carbon-emission estimates assume that the U.S. energy mix is comparable to that of Amazon Web Services.
Abstract
from arXiv · showhide
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
1 Introduction
Recent advances in deep learning have improved NLP accuracy, but state-of-the-art models increasingly require substantial specialized computation, creating financial, access, and environmental concerns. The paper quantifies training-related energy, carbon, and electricity costs to promote more mindful NLP research and policy.
- Motivation: The highest-scoring NLP models are often the most computationally hungry, requiring substantial computational resources for training.Advances in training techniques and hardware have enabled impressive accuracy improvements across fundamental NLP tasks.
- Motivation: Multiple specialized GPUs or TPUs increasingly replace commodity laptops or servers, limiting access to highly accurate models on financial grounds.The shift toward specialized hardware creates an equity concern for researchers without comparable resources.
- Motivation: Weeks or months of hardware operation impose substantial environmental costs because model training requires large amounts of energy.Renewable energy and carbon offsets do not eliminate concerns where electricity is not carbon-neutral or renewable capacity remains limited.
- Motivation: Model training and development likely constitute a substantial share of the greenhouse-gas emissions attributed to many NLP researchers.This motivates examining training practices as an important component of NLP’s environmental footprint.
- Contribution: The paper estimates energy requirements for popular off-the-shelf NLP models and converts them into approximate carbon emissions and electricity costs.These estimates are intended to heighten community awareness and promote mindful research practice and policy.
2 Methods
The study quantifies computational and environmental costs for popular NLP models and examines the full resources used to develop LISA, including tuning and experimentation. It measures training power, estimates completion-time energy use, and converts energy consumption into CO2 emissions using U.S. electricity assumptions.
- Study design: The analysis covers training costs for popular off-the-shelf NLP models and the complete development process for LISA, including tuning and experimentation.LISA is identified as a state-of-the-art NLP model from EMNLP 2018.
- Study design: Models are trained with default settings for at most 1 day, primarily on one NVIDIA Titan X GPU, while ELMo uses three NVIDIA GTX 1080 Ti GPUs.GPU and CPU power consumption is repeatedly sampled during training, with GPU power averaged across samples.
- Energy accounting: Energy use is estimated from reported training times and hardware, using average CPU, DRAM, and GPU power draw together with the number of GPUs.Power consumption is calculated in kilowatt-hours (kWh).
- Environmental accounting: CO2 emissions are estimated by converting energy use with EPA average U.S. emissions per kilowatt-hour, accounting for the proportions of major energy sources.The U.S. energy mix is considered comparable to Amazon Web Services, supporting the use of this conversion for cloud compute estimates.
- Models: Four freely available NLP models are analyzed using their code out-of-the-box, including Transformer, BERT, and GPT-2.The passages identify BERT base as a 110M-parameter model and report published training configurations of 16 TPU chips for 4 days or 64 Tesla V100 GPUs for 79.2 hours.
3 Related work
Prior work has characterized computational requirements in computer vision and advanced hyperparameter-search algorithms, but this paper addresses the absence of comparable analysis for NLP model R&D and tuning.
- Computer vision: Li et al. studied energy use for training and inference in popular convolutional image-classification models, including comparisons across neural-network layer types.This work provides precedent from computer vision for analyzing computational requirements of modern neural architectures.
- Hyperparameter tuning: Hyperparameter-search research has examined improved algorithms, including methods by Bergstra et al. and Snoek et al.The cited work concerns algorithms for hyperparameter search rather than NLP-specific R&D cost analysis.
- Research gap: To the authors’ knowledge, no prior analysis had quantified computation required for neural-network R&D and hyperparameter tuning in NLP.The paper positions its contribution as filling this gap.
4 Experimental results
Training modern NLP models can require substantial computational, financial, and environmental resources, with development and tuning costs far exceeding those of training a single model. The experiments also show that hardware choice affects cost efficiency and that large training runs can produce substantial carbon emissions.
- Model training costs: TPUs are more cost-efficient than GPUs for suitable workloads such as BERT, but GPU training of BERT produces emissions roughly equivalent to a trans-American flight.The reported NAS result reached 29.7 BLEU for English-to-German translation, an increase of only 0.1 BLEU.
- R&D computational requirements: 172 days of model development involved 123 hyperparameter grid searches and 4789 jobs, averaging 52 hours per job.Jobs ranged from 3 minutes to 9 days and used NVIDIA Titan X and M40 GPUs.
- R&D computational requirements: 9998 GPU-days, equivalent to 27 years of GPU time, were required for the project.This workload averaged about 60 GPUs running continuously throughout the six-month project.
- Model training costs: Training a single model was relatively inexpensive, whereas tuning for a new dataset, estimated at 24 jobs, and full R&D quickly became extremely expensive.Table 4 estimates upper and lower bounds using Google Cloud compute and raw electricity costs.
5 Conclusions
The paper concludes that NLP research should make model costs and sensitivity more comparable and pursue shared, equitable compute alongside more computationally efficient algorithms, hardware, and software. These measures address the financial, environmental, and access barriers created by large-scale compute requirements.
- Reporting and comparability: Authors should report training time, computational resources, and model sensitivity to hyperparameters when proposing models for downstream retraining or fine-tuning.Such reporting would enable direct cost-benefit comparisons across models.
- Reporting and comparability: Standardized measures of training time and model sensitivity would improve comparisons across hardware settings and tuning practices.Suggested measures include gigaflops required to converge and variance with respect to searched hyperparameters.
- Equitable access: Large-scale compute access is concentrated in industry, making recent state-of-the-art accuracy improvements unattainable to many academic researchers.The paper warns that this concentration constrains creativity and reinforces inequities based on financial resources.
- Equitable access: Approximately 58 GPUs for 172 days would cost $145,000 USD plus electricity in centralized hardware, about half the estimated cost of on-demand cloud GPUs.Shared academic compute centers could continue benefiting multiple projects after the initial investment.
- Efficiency recommendations: Industry and academia should promote computationally efficient algorithms, lower-energy hardware, and software tools that reduce energy associated with model tuning.The paper notes that easy-to-use APIs for efficient models could support this software effort.