Source-linked AI summary
Power Hungry Processing: Watts Driving the Cost of AI Deployment?
Alexandra Sasha Luccioni, Yacine Jernite, Emma Strubell
TL;DR
The paper examines the environmental costs of deploying modern ML systems and conducts an extensive study across applications in natural language and computer vision. It finds that generative and multi-purpose systems require more energy and carbon than discriminative and task-specific alternatives, while training remains far more intensive than inference.
Problem
Deploying modern ML systems requires substantial computational resources, making their environmental impacts important to understand.
Method
The study measures energy requirements across datasets spanning natural-language and computer-vision applications, comparing training, finetuning, and inference requirements.
Results
Generative tasks and multi-purpose models for discriminative tasks are more energy- and carbon-intensive than discriminative and task-specific alternatives, while training remains orders of magnitude more intensive than inference.
Takeaways & Limitations
The findings highlight the need to weigh the deployment of multi-purpose generative systems against their energy and carbon costs, especially where tasks are well-defined.
Takeaways & Limitations
The study identifies the sheer amount of energy and carbon required by its experimentation as a main ethical concern.
Abstract
from arXiv · showhide
Recent years have seen a surge in the popularity of commercial AI products based on generative, multi-purpose AI systems promising a unified approach to building machine learning (ML) models into technology. However, this ambition of ``generality'' comes at a steep cost to the environment, given the amount of energy these systems require and the amount of carbon that they emit. In this work, we propose the first systematic comparison of the ongoing inference cost of various categories of ML systems, covering both task-specific (i.e. finetuned models that carry out a single task) and `general-purpose' models, (i.e. those trained for multiple tasks). We measure deployment cost as the amount of energy and carbon required to perform 1,000 inferences on representative benchmark dataset using these models. We find that multi-purpose, generative architectures are orders of magnitude more expensive than task-specific systems for a variety of tasks, even when controlling for the number of model parameters. We conclude with a discussion around the current trend of deploying multi-purpose generative ML systems, and caution that their utility should be more intentionally weighed against increased costs in terms of energy and emissions. All the data from our study can be accessed via an interactive demo to carry out further exploration and analysis.
1 INTRODUCTION
The study addresses limited understanding of inference-stage environmental costs as AI deployment expands, especially for multi-purpose systems. It compares models, tasks, and modalities to characterize energy requirements and associated carbon emissions.
- Motivation: Inference can affect the environment as much as or more than training because deployed models require substantial computational resources at scale.Although each inference uses less computation than training, user-facing systems may process billions of inferences daily.
- Motivation: Inference costs remain insufficiently quantified, including impacts from energy, carbon, water, and rare-earth-mineral mining.The passage identifies limited in-depth work on model inference and deployment costs.
- Motivation: Training efficiencies from multi-purpose models can be lost or reversed over a model’s lifetime because deployment generates frequent inference workloads.This concern is especially relevant to user-facing applications such as chat and web search.
- Study contribution: The study measures inference energy across 88 models, 10 tasks, and 30 datasets spanning natural-language and computer-vision applications.It analyzes end task, modality, model size, architecture, and task-specific versus multi-task or multi-purpose learning paradigms.
- Study contribution: The comparison finds orders-of-magnitude differences in energy required per inference across models, modalities, and tasks.The authors frame these differences as a trade-off involving the benefits of multi-purpose systems, energy cost, and carbon emissions.
2 PREVIOUS WORK
Previous research has emphasized training emissions, while inference costs and standardized comparison methods remain less developed. This study extends prior work by examining inference across architectures, tasks, and datasets.
- Prior environmental-impact studies: Existing studies have largely estimated energy and carbon during neural-network training, including comparisons of models and emissions-influencing factors.Some work has also analyzed specific model architectures and their life-cycle contributions to carbon emissions.
- Prior environmental-impact studies: Cloud-focused studies have proposed reducing ML emissions through delayed scheduling, workload elasticity, and selecting lower-carbon electricity.These approaches address deployment environments rather than establishing a unified inference-cost comparison.
- Measurement gap: There is no standardized methodology for quantifying and comparing ML energy consumption and carbon emissions, making model footprints difficult to compare systematically.Existing tools such as Code Carbon, MLCO2, and LLMCarbon use different approaches and produce different results.
- Measurement gap: Existing work has focused on dynamic power, while embodied emissions from manufacturing computing equipment are harder to incorporate because hardware transparency is limited.One estimate found embodied carbon could constitute up to 50% of an LLM’s carbon footprint.
- Inference-focused work: Inference scholarship has begun to quantify deployment costs, with prior work showing that inference can become a significant portion of a model’s overall carbon footprint.The current study pursues this line through broader testing across architectures, tasks, and datasets.
- Inference-focused work: The study examines how architecture and deployment choices produce variation in inference energy and environmental impact.It aims to show how seemingly small user choices can result in large differences in models’ environmental impacts.
3 METHODOLOGY
The methodology evaluates inference costs across diverse tasks, datasets, and model families using repeated, sequential experiments on AWS GPUs. It measures both energy consumption and carbon emissions under a fixed regional carbon intensity.
- Task and dataset selection: The study covers 10 tasks across text-to-category, text-to-text, image-to-category, image-to-text, and text-to-image modalities.The task set includes multimodal applications such as image captioning and image generation.
- Task and dataset selection: Three widely downloaded Hugging Face datasets were selected for each task and listed in Table 1.The datasets were chosen to test the selected tasks across multiple applications.
- Model selection: The evaluation sampled 88 models to represent diverse deployment use cases and compare architectures within and across tasks.Some models were task-specific or fine-tuned, while others supported zero-shot use; some incompatible models were discarded.
- Multi-purpose models: Eight multi-purpose models from the Flan-T5 and BLOOMz families were evaluated on question answering, text classification, and summarization.These tasks enabled comparison with task-specific systems under a unified model architecture, using a constant zero-shot prompting setup.
- Experimental procedure: Each task-specific model was run for 1,000 inferences on each of three datasets, with every experiment repeated 10 times.Inference was performed sequentially without batching to reflect variability in real deployment settings.
- Measurement: Experiments ran on eight NVIDIA A100-SXM4-80GB GPUs in AWS, with Code Carbon measuring inference energy and carbon emissions.All experiments used AWS us-west-2, whose average carbon intensity was 297.6 grams of CO2eq per kWh.
- Measurement: Carbon emissions vary by regional energy source even when energy use remains similar across A100 deployments, motivating separate reporting of energy and carbon.The study provides code and Code Carbon logs with hardware-level breakdowns for further analysis.
- Measurement: 754.66 kWh of energy and 178.97 kg of emissions were used for all model experimentation and evaluation.
4 RESULTS
Energy and emissions vary substantially across tasks, modalities, model sizes, and architectures. Generative and image-based tasks generally impose higher inference costs than classification tasks, while multi-purpose models often emit more than task-specific systems.
- Task and modality differences: 0.06-2.9 kWh is the energy range for image captioning and image generation, whereas generative text tasks use around 0.05 kWh per 1,000 inferences.Text-based tasks were generally more energy-efficient than image-based tasks.
- Task and modality differences: 2.9 kWh is the mean energy consumption for image generation per 1,000 inferences, compared with 0.002 kWh for text classification.Across the examined tasks, image generation was the most energy-intensive and text classification the least.
- Task and modality differences: Over 1450 times separates the energy required by the least and most energy-intensive tasks for 1,000 inferences.The least-intensive task was text classification, while the most-intensive was image generation.
- Model size and emissions: Task structure accounts for more emissions variation than model size, with a roughly 500M-parameter image-generation model producing 100g versus 0.6g of CO2eq for text-to-category models.Model size and emissions were related, but modality and task structure explained more of the observed variation.
- Multi-purpose systems: 0.3g versus 10g of CO2eq per 1,000 inferences illustrates the several-orders-of-magnitude emissions gap between task-specific and multi-purpose models on extractive question answering.The comparison concerns efficient task-specific models and multi-purpose models on a dataset such as SciQ.
- Multi-purpose systems: 11.48g is the average emissions of the 11-billion-parameter Flan-T5-XXL, lower than the smaller BLOOMz-7B.Among multi-purpose models, emissions were also associated with architecture, model size, and output length; sequence-to-sequence models averaged 2g less emissions than decoder-only models of the same size.
5 DISCUSSION
The discussion finds that multi-purpose generative models impose substantially higher inference costs than task-specific models, especially for discriminative tasks and longer outputs. It also frames the study as an initial comparison whose findings should inform more transparent and environmentally aware deployment decisions.
- Model and task comparisons: Generative tasks are more energy- and carbon-intensive than discriminative tasks, with text generation, summarization, image captioning, and image generation among the most intensive.Image-based tasks are more intensive than text-only tasks; category prediction is less intensive than content generation.
- Lifecycle costs: Training remains orders of magnitude more energy- and carbon-intensive than inference, but deployment frequency can rapidly narrow the lifecycle gap.The training/inference parity grows with model size, and hundreds of millions of inferences may correspond to one training run.
- Model and task comparisons: Multi-purpose models use more energy than task-specific models for the same discriminative tasks, especially text classification and question answering.The gap is smaller for summarization and grows with output length.
- Model and task comparisons: For extractive question answering, BERT-based models emit less than 3 times the emissions of multi-purpose models, while text classification shows a much larger gap.The cited comparison reports 0.32g CO2eq per 1,000 queries for a BERT model, versus 2.66g for Flan-T5-XL and 4.67g for BLOOMz-7B.
- Implications and scope: The authors find no convincing evidence that generative zero-shot models are necessary for well-defined tasks such as web search and navigation, given their energy requirements.They acknowledge the benefit of deploying such models for their ability to perform multiple tasks.
- Implications and scope: The paper presents its training, finetuning, and inference comparison as an initial contribution for researchers, practitioners, and policymakers assessing AI’s environmental impacts.The authors provide code and an interactive tool, while emphasizing that the study is not representative of all deployment contexts.
ETHICAL CONSIDERATIONS STATEMENT
The study acknowledges substantial energy use and carbon emissions from its experimentation, despite steps to reduce wasted resources.
- 88 models were run on 3 datasets 10 times each to ensure statistical significance of the measurements.
- 754.66 kWh of energy and 178.97 kg of CO2eq were used and emitted across experimentation and evaluation.
- Up-front experiments used smaller portions of the dataset to reduce wasted resources.
RESEARCHER POSITIONALITY STATEMENT
The authors identify their North American institutional backgrounds as a constraint on how representative their experimental planning and execution may be.
- The authors have theoretical and applied machine learning backgrounds and work at institutions based in North America.
- Their experimental planning and execution may not reflect institutions in other regions.
- Their approach may also differ from researchers at institutions with more limited access to compute.
ADVERSE IMPACTS STATEMENT
The authors clarify that their environmental analysis is intended to inform choices rather than criticize ML deployment generally or target specific organizations.
- The work could be perceived as a critique of ML deployment in general because it analyzes environmental impacts.
- The analysis could be used to argue against pursuing ML research and development or to target specific companies or organizations.
- The authors intend to provide information that helps developers and researchers make choices based on environmental footprint or energy usage.
A FULL LIST OF TASK-SPECIFIC MODELS TESTED
The tested-model inventory spans task-specific and generative systems across vision, language, sentiment, tagging, and other machine learning tasks.
- Vision models: The inventory includes image classification models such as ResNet, BEiT, ViT, ConvNeXt, and MobileNet.
- Vision-language models: Image-captioning models include ViT-GPT2, BLIP, BLIP-2, and GIT variants.
- Language models: Language-task models cover question answering, summarization, sentiment analysis, named-entity recognition, and text generation.
- Inventory scope: Table 6 lists 80 finetuned models tested for the ten analyzed tasks.
- Vision models: It includes object detection models such as DETR, YOLOS, DETA, and deformable DETR.
B MODEL EVALUATION
The evaluation plots relate model size to task performance for summarization and question answering, while indicating emissions. SQuAD v2 performance is reported using F1 for answerable questions.
- Model size is plotted against summarization accuracy, with dot size indicating emissions.
- Model size is plotted against question answering accuracy, with dot size indicating emissions.
- SQuAD v2 evaluation uses F1 for questions that have an answer.
CNN (ROUGE)
The study reports full performance metrics for 32 evaluated models, including the listed sentiment-analysis model results.
- sbcBI/sentiment_analysis_model has reported metrics of 0.81, 0.75, and 0.76.
- 32 models were evaluated, comprising 24 finetuned and 8 multi-purpose models.
- The performance-metrics table covers both finetuned and multi-purpose models.