PaperlayerView exact revision on arXiv

Source-linked AI summary

How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference

Nidhal Jegham, Marwan Abdelatti, Chan Young Koh, Lassad Elmoubarki, Abdeltawab Hendawi

arXiv:2505.09598v6cs.CYcs.AI

TL;DR

The paper addresses the absence of standardized, prompt-level evidence for the environmental costs of LLM inference, especially across commercial deployments. It introduces an infrastructure-aware benchmark combining API performance, power specifications, environmental multipliers, statistical hardware inference, and DEA; across models, energy use varies dramatically, with reasoning systems among the most intensive. The authors also acknowledge uncertainty from incomplete telemetry, non-GPU power estimation, and proprietary model-size classification.

  • Problem

    A standard prompt-level method for quantifying model-specific inference energy, water, and carbon costs is absent, while existing reports aggregate infrastructure or omit key resource categories.

  • Method

    The framework combines public API performance with GPU and system power specifications, PUE, WUE, and CIF multipliers, statistical hardware inference, cross-efficiency DEA, and dashboard visualization.

  • Results

    Energy use spans widely across 30 models: DeepSeek-R1 (DS) consumes 29.075 Wh for long prompts, around sixty five times more than LLaMA-3.1-8B at 0.443 Wh.

  • Takeaways & Limitations

    The benchmark shows that real-world inference eco-efficiency depends on deployment infrastructure as well as model design.

  • Takeaways & Limitations

    The estimates include only actively assigned GPU energy, use a fixed non-GPU utilization estimate, and classify proprietary model scale from observed API performance.

Abstract

from arXiv · show

This paper introduces an infrastructure-aware benchmarking framework for quantifying the environmental footprint of LLM inference across 30 state-of-the-art models in commercial datacenters. The framework combines public API performance data with company-specific environmental multipliers and statistical inference of hardware configurations. We additionally utilize cross-efficiency Data Envelopment Analysis (DEA) to rank models by performance relative to environmental cost and provide a dynamically updated dashboard that visualizes model-level energy, water, and carbon metrics. Results show the most energy-intensive models exceed 29 Wh per long prompt, over 65 times the most efficient systems. Even a 0.42 Wh short query, when scaled to 700M queries/day, aggregates to annual electricity comparable to 35{,}000 U.S. homes, evaporative freshwater equal to the annual drinking needs of 1.2M people, and carbon emissions requiring a Chicago-sized forest to offset. These findings highlight a growing paradox: as AI becomes cheaper and faster, global adoption drives disproportionate resource consumption. Our methodology offers a standardized, empirically grounded basis for sustainability benchmarking and accountability in AI deployment.

1 Introduction

LLMs are increasingly deployed across consumer, educational, and enterprise applications, including state-of-the-art, open-source, and reasoning systems. Their environmental costs are increasingly concentrated in large-scale inference, yet prompt-level model comparisons remain unavailable, motivating an infrastructure-aware framework.

  • 1 Introduction: LLMs now power search engines, virtual assistants, education platforms, and enterprise tools, spanning proprietary, open-source, and reasoning models.Examples include GPT-4o, Claude-3.7 Sonnet, LLaMA-3, DeepSeek-V3, DeepSeek-R1, o1, and o3-mini.
  • 1 Introduction: Inference can account for up to 90% of a model’s total lifecycle energy use, despite training receiving greater sustainability attention.Unlike training, inference occurs consistently and at large scale.
  • 1 Introduction: A standard method for quantifying inference cost at the prompt level remains absent, limiting model-specific environmental assessment.Commercial AI providers generally do not disclose the inference data needed for accurate assessment.
  • 1 Introduction: The framework combines public API latency and throughput with GPU and system power specifications and company-specific PUE, WUE, and CIF multipliers.It also uses ANOVA and Tukey HSD to estimate underlying hardware configurations and provides an interactive dashboard for visualization.
  • 1 Introduction: The paper evaluates environmental effects across open-source and proprietary models and reports findings across 30 models.The paper also includes case studies of GPT-4o annual footprint and GPT-5 adaptive routing.

2 Related Work

Prior work measures selected training or inference costs but commonly omits production-scale inference, proprietary systems, infrastructure overhead, or water and carbon categories. This paper addresses these gaps with a standardized, infrastructure-aware prompt-level benchmark using API data, power specifications, environmental multipliers, statistical inference, and DEA.

  • 2 Related Work: Prior environmental analyses omit different combinations of inference, carbon, water, system-wide energy, and infrastructure overhead.Examples include GPT-3 water analysis, BERT and GPT-2 carbon analysis, and Meta’s LLaMA reports.
  • 2 Related Work: Prompt-level inference studies measure GPU or CPU energy but generally exclude proprietary systems, broader environmental factors, or production-scale standardization.MELODI focuses on small-scale open-source models, while other work excludes API deployments and infrastructure considerations such as PUE and WUE.
  • 2 Related Work: Existing studies often focus on training or local model evaluation and lack standardized, scalable methods for production inference.They also commonly omit infrastructural overhead and resource categories such as water and carbon emissions.
  • 2 Related Work: The proposed benchmark integrates API performance, GPU and system power specifications, environmental multipliers, statistical infrastructure inference, and DEA.It estimates energy, water, and carbon costs for data-center inference and contextualizes environmental impact against functional performance.
  • 2 Related Work: The study adds GPT-4o and GPT-5 case studies covering annual footprint at scale and adaptive model routing.These extend the benchmark beyond model-level estimates to usage and deployment analyses.

3 Preliminaries

The framework uses PUE, WUE, and CIF to represent data-center energy overhead, water use, and carbon emissions. Together, these multipliers extend IT-level measurements to infrastructure-level environmental accounting.

  • 3 Preliminaries: PUE captures non-computational data-center energy overhead from cooling, lighting, and power distribution.It is defined as total data-center energy consumption divided by IT-specific energy use.
  • 3 Preliminaries: WUE measures water use per kilowatt-hour of IT energy across on-site cooling, off-site generation, and hardware-related embodied water.It may use water withdrawal or water consumption, the portion permanently lost primarily through evaporation.
  • 3 Preliminaries: CIF measures carbon emissions per kilowatt-hour of energy consumed, largely reflecting the regional electricity mix.Its accounting includes direct combustion, off-site electricity generation, and embodied emissions from manufacturing and transport.

4 Methodology

The framework estimates per-query LLM inference energy, water, and carbon costs by combining performance data, inferred hardware, and provider-specific environmental factors. It also uses DEA and a dashboard to compare models across environmental efficiency and deployment conditions.

  • Framework overview: The framework integrates model performance metrics with infrastructure-level environmental multipliers to estimate operational energy, water, and carbon emissions per query.It also evaluates eco-efficiency with DEA and supports analysis through an interactive dashboard.
  • Model and infrastructure coverage: 30 models are analyzed across OpenAI, Anthropic, Meta, and DeepSeek using estimated deployment contexts, hardware specifications, and provider-specific PUE, WUE, and CIF values.Models are generally attributed to NVIDIA DGX systems using A100, H100, H200, or H800 GPUs.
  • Per-query energy estimation: Latency and TPS data across empirical quantiles and short, medium, and long prompt configurations provide the performance inputs for probabilistic per-query energy estimation.The runtime model represents latency to first-token generation and tokens-per-second behavior.
  • Per-query energy estimation: A Gaussian copula with ρ = −0.3 generates 10,000 correlated latency–TPS samples, each representing a plausible inference scenario.The negative dependence captures the relationship typically observed between latency and TPS.
  • Per-query energy estimation: Energy estimates combine GPU and non-GPU power, utilization bounds, total inference time, and PUE before aggregating Monte Carlo draws into mean and standard-deviation outcomes.Expected energy is a weighted combination of minimum- and maximum-utilization scenarios with wmax = 0.5.
  • Hardware attribution: Models are assigned hardware classes by size, with 1, 2, 4, or 8 GPUs, while partially loaded-node idle GPU power is excluded because deployment telemetry is unavailable.Non-GPU utilization is conservatively fixed at Dnon-GPU = 0.5.
  • Eco-efficiency evaluation: Cross-efficiency DEA compares per-query energy and environmental inputs against a composite Artificial Intelligence Index spanning reasoning, knowledge, mathematics, and coding benchmarks.Reasoning and knowledge contribute 50%, mathematics 25%, and coding 25% of the index.
  • Dashboard: The automated Power BI dashboard updates from daily-scraped API data and visualizes energy, water, carbon, latency, TPS, benchmark results, and scaled query impacts.Users can filter by company, model size, query size, or sustainability metric and download the dataset.

5 Experimental Evaluation

The evaluation benchmarks 30 LLMs across energy, water, and carbon footprints, revealing wide variation driven by model scale, reasoning, and deployment infrastructure. Resource-efficient models remain low-impact for long prompts, while reasoning and inefficient infrastructure substantially increase consumption.

  • 30 LLMs are benchmarked across energy consumption, water usage, and carbon emissions, with long-form exclusions for GPT-4 and LLaMA-3 variants limited by context windows.
  • Energy Consumption: 0.443 Wh is required by LLaMA-3.1-8B for long prompts, compared with 29.075 Wh for DeepSeek-R1 (DS), approximately 65 times more.GPT-4.1 nano is the most efficient proprietary model at 0.827 Wh.
  • Energy Consumption: 0.443 Wh at 8B parameters rises to 25.202 Wh at 405B in the LLaMA family, while DeepSeek-R1 and DeepSeek-V3 consume 29.078 Wh and 13.162 Wh on DeepSeek servers versus 7.410 Wh and 3.696 Wh on Azure.The comparison indicates that hardware and datacenter efficiency materially affect measured energy use.
  • Water and Carbon Emissions: Under 4 mL of water and less than 0.3 gCO2e are consumed by the most efficient systems even for long-form prompts.These systems include LLaMA-3.2 1B, LLaMA-3.2 3B, LLaMA-3.1-8B, LLaMA-3-8B, and GPT-4.1 nano.
  • Water and Carbon Emissions: Over 200 mL of water and approximately 17 gCO2e are associated with a long DeepSeek-R1 (DS) query, versus 34 mL and 2.5 gCO2e on Azure.The Azure values represent reductions of nearly 85%.
  • Hundreds of millions of daily GPT-4o requests can aggregate into carbon emissions comparable to thousands of transatlantic flights and water use equivalent to millions of people’s annual drinking needs.

6 GPT-4o Environmental Impact Case Study

The GPT-4o case study scales per-query measurements to estimated 2025 usage, translating inference into annual energy, water, and carbon impacts. The projections show that large-scale deployment can exceed familiar household, institutional, and transportation benchmarks.

  • Energy Cost of a Single GPT-4o User Session: 0.42 Wh per short GPT-4o query exceeds a Google search’s 0.30 Wh by approximately 40%, while eight daily queries total 3.73 Wh.Medium-length daily usage reaches 9.71 Wh.
  • Estimated 2025 Annual Energy Consumption of GPT-4o Inference: 700 million daily queries are attributed conservatively to GPT-4o from a 1 billion-query daily ChatGPT baseline.
  • Estimated 2025 Annual Energy Consumption of GPT-4o Inference: 391,509 MWh at minimum and 463,269 MWh at maximum are projected for GPT-4o inference in 2025, exceeding annual electricity use of 35,000 U.S. households.The estimates use approximately 772 billion GPT-4o queries, with an assumed 80%/20% split between short and medium prompts.
  • Estimated 2025 Annual Water Consumption of GPT-4o Inference: 1,334,991 kL to 1,579,680 kL of annual water consumption are projected for GPT-4o, equivalent to over 500 Olympic-sized pools or 1.2 million people’s annual drinking needs.
  • Estimated 2025 Annual Carbon Emissions of GPT-4o Inference: 138,125 tons to 163,441 tons of annual CO2e emissions are projected for GPT-4o, comparable to emissions from 30,000 gasoline-powered cars.

7 GPT-5 Adaptive Model Routing Case Study

GPT-5’s adaptive routing selects between fast and more computationally intensive reasoning modes according to prompt complexity. Energy consumption therefore varies substantially with reasoning level, from relatively low short-query use to high long-query costs.

  • Adaptive routing automatically selects a fast variant or a more computationally intensive “Thinking” model based on prompt complexity.
  • Energy Consumption Across Reasoning Modes: 2.33 Wh for minimal reasoning rises to 17.15 Wh for high reasoning on medium-length GPT-5 queries, representing a more than sevenfold increase.
  • Energy Consumption Across Reasoning Modes: 0.67 Wh is consumed by a short, minimal-reasoning GPT-5 query, comparable to GPT-4o’s 0.42 Wh short prompt.
  • Energy Consumption Across Reasoning Modes: 33.8 Wh is consumed on average by a long, high-reasoning GPT-5 query, comparable to the upper range of the study’s most energy-intensive models.
  • Adaptive routing can optimize computational resources by matching inference depth to task complexity while amplifying the footprint of cognitively demanding prompts.

8 Discussion and Policy Implications

Inference sustainability depends on infrastructure as well as model design, while improving per-task efficiency can still increase total environmental impact when usage expands. The paper therefore calls for model-level optimization alongside systemic infrastructure regulation.

  • Infrastructure conditions can outweigh model size: GPT-4o mini uses approximately 20% more energy than GPT-4o on long queries because it relies on older A100 nodes.DeepSeek-R1 and DeepSeek-V3 also show nearly sixfold higher water and carbon consumption on proprietary servers than Azure.
  • Azure-hosted DeepSeek deployments have nearly six times lower water and carbon consumption than counterparts on DeepSeek’s own servers.The difference is associated with hardware, cooling, carbon intensity, and PUE conditions described in the passage.
  • As per-task efficiency improves, total AI usage can expand faster, amplifying net resource consumption through a pattern aligned with the Jevons Paradox.The passage links this expansion to AI’s increasing affordability and acceleration.
  • Environmental sustainability requires both model-level optimizations and systemic infrastructure regulation.Suggested routes include per-inference thresholds, architectural changes such as sparsity and quantization, and infrastructure improvements.

9 Conclusion, Limitations, and Future Work

The paper presents an infrastructure-aware framework for measuring LLM inference impacts and concludes that eco-efficiency depends on both model design and deployment conditions. It also acknowledges telemetry and scope limitations that motivate broader lifecycle and modality coverage.

  • The framework combines API performance, environmental multipliers, and statistical inference to benchmark energy, water, and carbon costs for real-world LLM inference.Cross-efficiency DEA relates environmental impact to functional performance.
  • Eco-efficiency depends not only on model design but also on infrastructure, while cheaper and faster AI can intensify environmental strain as usage expands.The GPT-4o case study is presented as evidence aligned with the Jevons Paradox.
  • The estimates conservatively include only energy drawn by actively assigned GPUs because unused capacity and non-GPU power are difficult to determine.The study also classifies undisclosed proprietary model scale from observed API performance.
  • Future work should incorporate more detailed telemetry and facility-level reporting and extend evaluation beyond text generation to image, video, and audio.The passage notes that these modalities are likely to impose greater environmental costs because of higher computational intensity.

A Batch Size Sensitivity Analysis (GPT-4o)

Batching materially changes GPT-4o’s per-prompt energy estimates, making batch size a central sensitivity in the analysis. The study uses batch size 8 as a representative midpoint while excluding Scope 3 impacts because deployment-specific attribution remains uncertain.

  • Batch Size Sensitivity: Batch size 8 is used for per-prompt energy estimates as a midpoint within providers’ typical batching range of 4 to 16.The choice balances latency constraints with energy efficiency, although batch size can significantly influence GPU and system utilization.
  • Batch Size Sensitivity: Batch size changes can significantly shift GPT-4o energy outcomes while holding the model and other parameters constant.The sensitivity analysis varies only batch size to examine plausible deployment configurations.
  • Batch Size Sensitivity: 45% lower energy per prompt results from increasing batch size from 4 to 8, followed by a further 43% reduction from 8 to 16.Using batch size 4 throughout would overstate, while batch size 16 could understate, the environmental footprint.
  • Batch Size Sensitivity: Dynamic batching can substantially shape per-prompt environmental impact, and high traffic enables large providers to use higher batch sizes without sacrificing latency.The passage connects batching decisions to traffic and latency management.
  • Scope 3 Considerations: Scope 3 includes hardware manufacturing, transportation, and end-of-life impacts, but deployment-specific attribution varies across vendors and manufacturing conditions.Microsoft’s 2023 Scope 3 emissions represented 66% of total emissions in the cited example.
  • Scope 3 Considerations: Assigning fixed embodied emissions or water per query can overestimate costs when hardware footprints are not amortized over lifespan or query volume.The paper therefore excludes Scope 3 from prompt-level comparisons because uncertainty could distort eco-efficiency results.

C Cross-effficiency DEA Results

Cross-efficiency DEA compares models using peer as well as self-selected weightings, supporting a more robust eco-efficiency ranking. Smaller reasoning models and Claude 3.7 Sonnet lead the frontier, while DeepSeek models rank lowest despite strong capabilities.

  • Evaluation Scope: Four models—Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4, and GPT-4 Turbo—were excluded because missing benchmark results prevented complete DEA evaluation.Cross-efficiency requires complete inputs and outputs for fair comparison.
  • Eco-efficiency Results: o3-mini achieves the highest cross-efficiency score at 0.884, followed by o1-mini at 0.836 and Claude 3.7 Sonnet at 0.825.GPT-4o (Mar) scores 0.789 and o3 scores 0.758.
  • Eco-efficiency Results: DeepSeek-R1 and DeepSeek-V3 record the lowest efficiency scores, at 0.067 and 0.059 respectively.Azure-hosted variants improve to 0.539 and 0.523 but remain below most OpenAI and Anthropic systems.
  • Interpretation: Eco-efficiency reflects both output quality and environmental cost, rather than capability or sustainability considered alone.OpenAI’s smaller reasoning models and Claude 3.7 Sonnet strike the strongest balance in the reported results.
Loading 2505.09598v6…