Source-linked AI summary
Measuring the environmental impact of delivering AI at Google Scale
Cooper Elsworth, Keguo Huang, David Patterson, Ian Schneider, Robert Sedivy, Savannah Goodman, Ben Townsend, Parthasarathy Ranganathan, Jeff Dean, Amin Vahdat, Ben Gomes, James Manyika
TL;DR
AI serving lacked production-scale measurements that comprehensively captured its environmental footprint. The paper proposes and applies a full-stack methodology to Google’s Gemini Apps infrastructure, finding low median per-prompt impacts and substantial one-year reductions in energy use and emissions. It argues that comprehensive metrics are needed to compare models and encourage efficiency across the serving stack.
Problem
AI serving lacked production-environment measurements covering energy, carbon emissions, and water consumption, limiting comprehensive understanding of its environmental footprint.
Method
The paper instruments Google’s production AI serving fleet and measures Gemini Apps using a full-stack methodology covering material serving energy sources and environmental metrics.
Results
0.24 Wh of energy, 0.03 gCO2e, and 0.26 mL of water are consumed by the median Gemini Apps text prompt.
Takeaways & Limitations
Comprehensive AI-serving metrics can reveal environmental impacts more accurately and support efficiency improvements across the full serving stack.
Takeaways & Limitations
The study excludes external networking, end-user devices, and model training and data storage from its inference-serving boundary.
Abstract
from arXiv · showhide
The transformative power of AI is undeniable - but as user adoption accelerates, so does the need to understand and mitigate the environmental impact of AI serving. However, no studies have measured AI serving environmental metrics in a production environment. This paper addresses this gap by proposing and executing a comprehensive methodology for measuring the energy usage, carbon emissions, and water consumption of AI inference workloads in a large-scale, AI production environment. Our approach accounts for the full stack of AI serving infrastructure - including active AI accelerator power, host system energy, idle machine capacity, and data center energy overhead. Through detailed instrumentation of Google's AI infrastructure for serving the Gemini AI assistant, we find the median Gemini Apps text prompt consumes 0.24 Wh of energy - a figure substantially lower than many public estimates. We also show that Google's software efficiency efforts and clean energy procurement have driven a 33x reduction in energy consumption and a 44x reduction in carbon footprint for the median Gemini Apps text prompt over one year. We identify that the median Gemini Apps text prompt uses less energy than watching nine seconds of television (0.24 Wh) and consumes the equivalent of five drops of water (0.26 mL). While these impacts are low compared to other daily activities, reducing the environmental impact of AI serving continues to warrant important attention. Towards this objective, we propose that a comprehensive measurement of AI serving environmental metrics is critical for accurately comparing models, and to properly incentivize efficiency gains across the full AI serving stack.
1 Introduction
As generative AI adoption expands, inference and serving increasingly contribute to AI’s environmental footprint, yet production-scale measurements have been lacking. This paper proposes a comprehensive full-stack framework and applies it to Gemini Apps.
- Motivation: Generative AI adoption has shifted environmental attention from model training toward inference and serving at global scale.The paper connects billions of user prompts with growing energy, emissions, and water impacts.
- Research gap: Existing energy estimates and empirical frameworks use differing assumptions and boundaries, producing widely varying per-prompt results.The paper introduces comprehensive metrics to address these inconsistencies.
- Research gap: The field lacks first-party environmental data from the largest AI model providers and agreement on which serving activities measurements should include.The paper identifies a measurement-boundary problem involving host systems, idle capacity, and data-center overhead.
- Contributions: The paper establishes a full-stack framework for measuring energy, emissions, and water impacts of production AI serving at Google scale.It applies the methodology to Google’s Gemini Apps product.
- Contributions: The framework defines energy, emissions, and water consumption per prompt as complementary environmental impact metrics.These metrics cover serving energy, market-based and embodied emissions, and cooling-related water consumption.
- Contributions: The study reports a 44x reduction in total emissions for the median Gemini Apps prompt over one year of development.The authors attribute this result to compounding AI-serving efficiency gains across the serving stack.
2 Related Work
Prior AI inference environmental studies span model-based estimates and empirical benchmarks, but their differing assumptions and measurement boundaries limit comparability. This paper argues for broader, production-informed metrics that compare scaled products and incentivize full-stack optimization.
- 2.2 Measured metrics: Empirical measurement improves precision but still differs in methodology and comparability across studies.Benchmarks vary in hardware, batching, operating conditions, and which components they measure.
- 2.1 Estimated metrics: Model-based inference estimates vary because they depend on hardware specifications, model parameters, and poorly constrained usage assumptions.Examples include estimates of approximately 3 Wh for a GPT-3.5 prompt and 0.3 Wh for a typical ChatGPT prompt.
- 2.1 Estimated metrics: Published energy estimates span substantially different values, making an AI assistant’s environmental impact difficult for users to understand.The paper reports order-of-magnitude variability in estimated energy per chat prompt.
- 2.2 Measured metrics: A full-stack boundary includes serving components beyond active accelerators, enabling more comprehensive energy measurement and reduction opportunities.The proposed boundary is intended to capture material energy-consuming activities across the serving stack.
- 2.2 Measured metrics: Different measurement approaches produce a 6x difference in energy per query for Llama 3.1 with 70B parameters.The paper uses this example to show why measurement boundaries are critical for interpreting results.
- 2.2 Measured metrics: The paper’s metrics aim to support meaningful comparison between scaled AI products, establish a broad measurement standard, and incentivize full-stack optimization.The authors acknowledge that this approach does not address all priorities, including hardware control, measurement ease, and visibility into serving-stack layers.
3 Methodology
The methodology measures AI-serving energy, emissions, and water across a full production serving stack, using internal telemetry and an explicit measurement boundary. It compares this Comprehensive Approach with a narrower Existing Approach and derives per-prompt metrics from fleet measurements.
- Framework: The framework measures energy, emissions, and water impacts for production AI serving using a comprehensive measurement boundary and transparent exclusions.It includes all material energy sources under Google’s operational control and defines metrics for energy, emissions, and water per prompt.
- Measurement boundary: The serving AI computer functional unit includes one or more AI accelerator trays connected to one host tray.This definition excludes external networking, end-user devices, and LLM training and data storage from the measurement boundary.
- Energy measurement: Energy is decomposed into active AI accelerator, host CPU and DRAM, idle-machine, and data-center-overhead components.The overhead component is captured using Power Usage Effectiveness, while active CPU and accelerator energy are allocated from active-machine energy using host and accelerator power shares.
- Energy measurement: Internal telemetry maps Gemini-serving models to tagged jobs and assigned machines, measuring host and accelerator power through external power supply units.The workflow identifies all models, including supporting scoring, ranking, classification, and prompt-routing models.
- Aggregation: Each energy component is divided by total Gemini Apps user prompt count over the same period to obtain energy per prompt.Measurements span at least a day, and daily or longer measurements prevent partial prompt counts from materially affecting results.
- Comparison methodology: The Existing Approach measures active accelerator energy on highly utilized standalone accelerators, whereas the Comprehensive Approach measures total energy across the serving stack.The Existing Approach also subsamples the 10% most energy-efficient data centers to enable comparison with literature benchmarks.
4 Results
Comprehensive measurement shows that Gemini Apps serving has measurable energy, emissions, and water impacts, while full-stack accounting produces higher energy estimates than narrower approaches. Over one year, software efficiency, utilization improvements, and cleaner electricity drove substantial reductions in per-prompt energy and emissions.
- Energy consumption: 0.24 Wh is the median energy consumption of a Gemini Apps text prompt under the comprehensive measurement methodology.Active AI Accelerator power contributes 0.14 Wh, host CPU and DRAM 0.06 Wh, and idle machines and PUE overhead 0.02 Wh each.
- Energy consumption: 2.4 times greater is the comprehensive per-prompt energy estimate than the 0.10 Wh estimate from the narrower existing approach.The difference reflects both a broader measurement boundary and lower machine utilization in production serving.
- Emissions efficiency gains: 47x and 36x are the reductions in Scope 2 MB emissions and Scope 1+3 emissions per prompt over 12 months, respectively.Together, these reductions correspond to a 44x reduction in total emissions per prompt.
- Emissions and water consumption: 0.03 gCO2e and 0.26 mL are the comprehensive median emissions and water consumption of a Gemini Apps text prompt.The energy use is equivalent to less than nine seconds of television viewing, while the water use equals five drops.
- Emissions efficiency gains: Energy efficiency improvements and electricity decarbonization are identified as the most impactful emissions-reduction initiatives.Electricity-related emissions dominate the per-prompt emissions breakdown relative to embodied emissions.
- Emissions efficiency gains: 44x is the reduction in total emissions per median Gemini Apps text prompt over 12 months.The reduction combines a 33x decline in energy consumption, a 1.4x decline in market-based emissions intensity, and a 36x decline in Scope 1+3 emissions per prompt.
5 Conclusions
This paper measures AI serving’s full-stack environmental impacts in production and finds that comprehensive accounting yields lower-than-many-estimated per-prompt impacts while revealing substantial year-over-year efficiency gains. It argues that standardized measurement boundaries are needed for comparable metrics and stack-wide efficiency incentives.
- Method and implications: A full-stack methodology accounts for energy, emissions, and water across AI serving infrastructure in a live production environment.The approach includes material energy sources rather than only active accelerator power or benchmark-level measurements.
- Findings: 0.24 Wh of energy, 0.03 gCO2e, and 0.26 mL of water are consumed by a median Gemini Apps text prompt under the paper’s comprehensive measurement.The measurement covers production AI serving and is more comprehensive than many previously published metrics.
- Method and implications: Standardized, comprehensive measurement boundaries are critical because narrower approaches can underestimate impacts and produce figures varying by orders of magnitude.The framework is intended to support comparison across models and providers and to incentivize efficiency across the AI serving stack.
- Findings: 33x lower energy use and 44x lower emissions for the median Gemini Apps prompt were achieved over the past year.The paper highlights software efficiency efforts and clean energy procurement as part of this improvement.
- Method and implications: Although a single prompt has low impact relative to many daily activities, global adoption makes continued reduction of AI’s environmental cost important.The paper advocates widespread adoption of comprehensive or similarly comprehensive measurement frameworks.
Appendix A: Normalization
AI serving’s overhead energy, water use, and emissions vary with location, climate, cooling technology, and time. The paper normalizes comparisons using annualized effectiveness and emissions factors together with fleetwide energy-weighted averages.
- Environmental variability: Cooling-related overhead energy varies by technology and climate, increasing during hotter and more humid periods.Air-cooled systems are generally less energy efficient than adiabatic or evaporative systems.
- Environmental variability: Water consumption varies with cooling technology and climate, from little or none for air cooling to higher use during hotter and drier periods.Adiabatic and evaporative systems consume increasingly more water under hotter and drier conditions.
- Environmental variability: Scope 2 emission factors vary seasonally with energy demand and the grid’s generation mix.This temporal variation affects reported emissions even when the AI workload is otherwise similar.
- Normalization rationale: Site and seasonal variability can skew AI benchmarking, making the same model appear more efficient under favorable cooling or data-center conditions.Advanced or optimally sited facilities can have smaller footprints than less modern or suboptimally sited facilities.
- Normalization rationale: Trailing-twelve-month or annual-average PUE, WUE, and grid emission factors, weighted fleetwide by AI-serving energy, should normalize comparisons.The paper reports applying both annualized factors and energy-weighted fleet averages.