Source-linked AI summary
Green AI
Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni
TL;DR
Deep learning research faces rapidly increasing computational, environmental, and financial costs, which can make participation more difficult. This position paper proposes evaluating efficiency alongside accuracy and reporting models’ computational price tags, arguing that these practices can support greener and more inclusive research. It concludes that Green AI should become a more prevalent option while remaining complementary to Red AI.
Problem
Deep learning progress increasingly relies on computationally intensive models, raising environmental and financial costs and barriers to participation.
Method
The paper advocates efficiency as an evaluation criterion and recommends reporting computational price tags for developing, training, and running models.
Results
The empirical analysis suggests that AI research has paid relatively little attention to computational efficiency while computational costs increase exponentially.
Takeaways & Limitations
Green AI can reduce computational expense with minimal performance reduction or improve performance through more efficient methods, while broadening research inclusiveness.
Takeaways & Limitations
The discussion notes that increasing model, dataset, and search resources can still improve downstream performance, though at an increasingly lower rate.
Abstract
from arXiv · showhide
The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [38]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive---enabling any inspired undergraduate with a laptop to write high-quality research papers. Green AI is an emerging focus at the Allen Institute for AI.
1 Introduction and Motivation
Deep learning progress has come with rapidly escalating computational costs that create environmental, economic, and participation concerns. The paper advocates Green AI, making efficiency an evaluation criterion and reporting computational price tags alongside accuracy.
- Motivation: 300,000x: training compute for state-of-the-art deep learning models increased over six years, doubling every few months.The trend extends across models from AlexNet to AlphaZero and is especially sharp among successive NLP word-embedding approaches.
- Motivation: Accuracy-focused leaderboards typically omit computational cost, overlooking the economic, environmental, and social costs of achieving reported results.
- Green AI: Green AI research seeks environmentally friendly and inclusive advances, while recognizing that Red AI has produced valuable contributions but become overly dominant.
- Proposals: The paper proposes making efficiency a common evaluation criterion alongside accuracy and related measures.
- Proposals: Reporting the computational price tag of finding, training, and running models would improve transparency and provide baselines for more efficient methods.The paper also suggests plotting accuracy against computational cost and training-set size.
- Evidence and scope: The paper’s empirical analysis suggests AI research has paid relatively little attention to computational efficiency as costs rise exponentially.It connects this pattern to diminishing returns from increased cost and advocates a simple efficiency metric.
2 Red AI
Red AI pursues state-of-the-art accuracy through massive computation, while increasing model size, data, and experiments yields diminishing returns. The paper documents these costs and advocates efficiency-oriented evaluation and reporting.
- Red AI seeks state-of-the-art accuracy by using massive computational power, effectively “buying” stronger results.
- 90% of ACL papers, 80% of NeurIPS papers, and 75% of CVPR papers in a 60-paper sample targeted accuracy.Only small portions of ACL and CVPR papers argued for efficiency, while many NeurIPS papers targeted efficiency through theoretical analyses.
- The paper examines processing cost per example, training-data size, and hyperparameter experiments as factors increasing computational expense.Equation 1 presents the result cost as growing linearly with these three quantities.
- The cost model in Equation 1 is simplified because hyperparameter assignments can have different processing costs and training epochs are omitted.
- Larger models can improve performance, but model growth exceeds the resulting performance increase, raising financial and environmental costs.
- Increasing training data and experiments also produces diminishing returns while imposing substantial computational and reproducibility costs.Common Crawl’s June 2019 collection contained 242 TB of uncompressed data, and one architecture search trained over 12,800 neural networks.
- Although additional resources still improve downstream performance, the paper identifies finding a possible saturation point as an open question.
- The paper proposes making efficiency a common evaluation criterion and reporting computational price tags as baselines for greener research.It frames computationally expensive practices as opportunities for developing more efficient techniques.
3 Green AI
Green AI proposes measuring and rewarding computational efficiency alongside accuracy, with FPO as a hardware-agnostic work metric and budget-aware reporting to support fairer comparisons and greener research.
- Green AI seeks novel results without increasing computational cost, ideally reducing it, as an alternative to accuracy-focused Red AI.
- Measures of Efficiency: Carbon emissions, electricity usage, elapsed time, and parameter counts each have limitations that hinder consistent comparison across locations, hardware, or models.Carbon depends on local infrastructure; electricity and elapsed time depend on hardware and system conditions, while parameter counts do not capture differing work per parameter.
- Measures of Efficiency: FPO estimates the total floating-point work required to generate a result by recursively costing operations from ADD and MUL.
- Measures of Efficiency: FPO is hardware-agnostic, tied to energy use, and strongly correlated with running time, enabling fairer comparisons between approaches.
- FPO Cost of Existing Models: A nearly 35% FPO increase from ResNet to ResNext produced only a 0.5% top-1 accuracy improvement on ImageNet.
- Additional Ways to Promote Green AI: The paper advocates reporting final FPO costs, budget/accuracy curves, and efficiency contributions, while tailoring savings toward training or inference according to use.Controlled ResNet comparisons likewise show that large FPO increases do not translate into large performance gains.
4 Related Work
Related work connects computationally expensive AI research to environmental, economic, and inclusiveness concerns, while noting that incomplete reporting impedes reproducibility. It also distinguishes this paper’s efficiency focus from machine-learning applications in climate and conservation.
- Computationally expensive deep NLP experiments can have substantial environmental and economic impacts, limiting participation in high-profile research areas.Greater value placed on computational efficiency could support contributions from more diverse groups, especially in academia.
- Carbon credits may offset electricity use, but the paper notes that their effectiveness relative to using less energy is unclear.Credit purchasing is voluntary, and coverage of renewable energy varies across cloud providers.
- Reporting only the best result after extensive development and tuning obscures the effort needed to reproduce or extend findings.
- Machine learning also supports emissions reduction, animal conservation tracking, and potential forest-fire management, but these applications are outside this paper’s efficiency focus.
5 Conclusion
The conclusion presents Green AI as an optional, efficiency-oriented approach intended to broaden participation without excluding accuracy-focused research. It highlights efficient models, data use, and experiment design as directions for reducing computational expense while preserving or improving performance.
- 5 Conclusion: Efficient methods could reduce computational expense with minimal performance loss or potentially improve performance as new approaches emerge.The paper also connects Green AI with a potentially more cognitively plausible direction because the brain is highly efficient.
- 5 Conclusion: Green AI is presented as a valuable option rather than an exclusive mandate, with both Green AI and Red AI contributing to the field.
- 5 Conclusion: Space- and time-efficient models can support deployment on small devices or real-time processing, but Green AI advocates a broader approach than these deployment goals.
- 5 Conclusion: Data-efficient research seeks comparable baseline performance using fewer training examples or gradient steps, including efficiency improvements during pretraining.
- 5 Conclusion: More complete reporting of hyperparameter searches, architecture evaluations, ablations, and total experiments can clarify the computational expense of obtaining final results.Hyperparameter optimization methods can reduce the expense required to reach a given performance level.