Source-linked AI summary
Beyond State-of-the-Art: Standardising Environmental Impact Metrics for AI Research
Lachlan McGinness, Dan Pagendam, Robert Offner
TL;DR
The paper addresses limited standardisation in accounting for AI’s environmental footprint. It proposes emissions-reporting and training-efficiency approaches, and its review finds that sustainability reporting among NeurIPS 2025 papers was rare and non-standard.
Problem
Method
Results
Approximately 1% of NeurIPS 2025 papers mentioned sustainability or environmental impacts, and papers that did so used no standard reporting method.
Source: §2 · p. 3Takeaways & Limitations
The authors encourage researchers to claim the Smallest Model that Achieves the Job alongside SotA and to make choices informed by sustainability as well as model performance.
Source: §5 · p. 7Using CO2 as a research-efficiency metric may unfairly penalise researchers with less-clean power grids or less-efficient hardware, while FLOPs do not capture the full cost of all ML methods.
Source: §5 · p. 8
Abstract
from arXiv · showhide
As the capabilities and ubiquity of Large Language Models (LLMs) grow, so does their environmental footprint. Despite calls for responsible AI, the machine learning community lacks standardised practices for carbon accounting. Our automated literature review of the 5,285 papers accepted to NeurIPS 2025 reveals that reporting of environmental impact is nearly non-existent. To catalyse a shift toward sustainable AI, we define standardised sustainability metrics for evaluating model training efficiency, accompanied by simple heuristics to estimate the carbon cost of LLM inference. We implement these metrics in carbonbenchmark, a drop-in software solution for tracking and reporting emissions. Finally, to combat the pursuit of marginal accuracy gains at disproportionate environmental costs, we formalise the `Smallest Model that Achieves the Job' (SMAJ), a framework which challenges the field to prioritise computational efficiency and environmental accountability alongside traditional `State-of-the-Art' (SotA) accuracy.
1 Introduction
The introduction calls for environmental accountability in AI research through emissions transparency, standardised reporting, and choices that limit environmental costs while preserving useful outcomes. It presents training-efficiency metrics and SMAJ as steps toward that goal.
1 Introduction
- Quantifying AI’s carbon footprint is presented as the first step toward judging whether its emissions are acceptable and encouraging researchers to reduce them while achieving desirable outcomes.The authors compare this restraint to using statistical power calculations to reduce clinical-trial sample sizes.Source: §1 · pp. 1–2
- The paper identifies three needs: reliable emissions estimates from foundation models, standardised emissions reporting in research, and recognition that research advances are not justified at any environmental cost.It argues that addressing these needs can support computational efficiency, thoughtful use, and greener infrastructure.Source: §1 · pp. 1–2
- The introduction links transparency to users’ ability to weigh a foundation model’s footprint against its performance and to research organisations’ ability to weigh emissions against societal benefits.It presents transparency as a possible driver of more sustainable industry infrastructure.Source: §1 · p. 2
- The paper offers methods for estimating study footprints, advocates reporting emissions, and introduces k to compare the carbon efficiency of iterative training methods.The proposed downstream aim is to assess computational trade-offs between accuracy and carbon footprint.Source: §1 · p. 2
- The authors report theoretical and empirical results for using k to assess training efficiency and introduce SMAJ as an alternative to pursuing SotA accuracy gains without regard to computational cost.They also describe using pilot studies and statistical methods to identify SMAJ among candidate models.Source: §1 · p. 2
2 LLM Literature Review
The literature review examines LLM use and environmental-impact reporting across NeurIPS 2025 papers. It finds that sustainability reporting is rare and non-standard, and proposes requiring each paper to report its total CO2 emissions.
2 LLM Literature Review
- The automated review covered all 5,285 main-track papers and screened for LLM use, usage types, open-source status, computational cost, and sustainability.Eight LLMs were tested on a random sample of 200 papers before the remaining papers were reviewed.Source: §2 · pp. 2–3
- The review used Gemini 3.1-Flash-Lite after the five strongest models showed 90%+ agreement and statistical tests found no significant difference from Gemini 3.1 Pro for four alternatives.The four alternatives were Sonnet-4.6, Gemini 3.1-Flash-Lite, Opus-4.6, and Haiku-4.5.Source: Figure 1 · p. 3, §2 · p. 3
- Approximately 40% of NeurIPS 2025 papers used LLMs, while approximately 1% mentioned sustainability or environmental impacts; papers that did consider sustainability used no standard reporting method.Fine-tuning appeared in 22% of papers, inference and evaluation in 19%, and agentic frameworks in 17%.Source: §2 · p. 3
- The authors propose requiring every paper to report the total mass of carbon dioxide emitted by its methods and experiments.They frame this requirement as a way to support more environmentally friendly research.Source: §2 · p. 3
3 Theoretical Underpinnings of Sustainability Benchmarks
The section develops fitted learning-curve measures for training efficiency and proposes SMAJ to select computationally efficient models statistically indistinguishable from the highest-performing model.
3 Theoretical Underpinnings of Sustainability Benchmarks
- The original exponential sustainability formula breaks at accuracy A = 1 and can reward under-trained models when the highest achievable accuracy is substantially below 1.Source: §3 · p. 4
- Neither k nor β has a general closed-form solution from known accuracy and training resources, so the authors recommend least-squares fitted k as the standard, with α and β optionally reported.Source: §3 · pp. 4–5
- The simple exponential fit is constrained to pass through (0, 0), while translating t by t0 relaxes that constraint; Figure 2 shows delayed exponential and power-law fits to TinyViT’s learning curve.Source: §3 · pp. 4–5, Figure 2 · p. 4
3.1 Numerically Fitting the Learning Curve
- Across all three tasks, models with higher k required less CO2 for training; t0 was needed for exponential fits, while power law fit most plots best, especially after accuracy plateaus.Source: §3.1 · p. 5
3.2 Smallest Model that Achieves the Job
- SMAJ keeps models within the highest performer’s combined uncertainty bounds, then selects the most computationally efficient acceptable model.The paper describes small pilot experiments and statistical methods as a way to identify SMAJ among candidate models.Source: §3.2 · p. 6
4 Cost of Running LLMs
The section estimates LLM inference emissions from token count and active parameters, converting estimated computation through energy use and grid carbon intensity. Because proprietary models rarely disclose active parameters, its example uses an approximation.
4 Cost of Running LLMs
- FLOPs = 2nN estimates LLM computation from active parameters n and combined context and completion tokens N; the formula is reported accurate within 10% in some experiments.Researchers can generally obtain N, while n is known for most open-source models but often unavailable for proprietary ones.Source: §4 · p. 7
- For proprietary models, the paper approximates n from open-source models with similar benchmark performance and recommends n = 5 × 10^10 for frontier models.The authors characterize this as an imprecise, order-of-magnitude approach; Deepseek-v4-pro is cited as having approximately 50 billion active parameters.Source: §4 · p. 7
- Estimated energy use is converted to CO2 using NVIDIA energy-per-TFLOP data and average US emissions per kilowatt hour, with a factor chosen as reasonable for 2026.The authors note that most AI inference centres are currently located in the US.Source: §4 · p. 7
- Approximately 1200kg of CO2 is estimated for 210 billion tokens, assuming n = 5 × 10^10 and that the token count combines input and output.The estimate uses the paper's approximation for frontier-model active parameters.Source: §4 · p. 7
5 Recommendations, Limitations and Future Work
The authors advocate reporting emissions and weighing sustainability alongside model performance, including by recognizing the Smallest Model that Achieves the Job. They also identify limits in the metric and demonstrations that motivate further work.
5 Recommendations, Limitations and Future Work
- The authors encourage researchers to claim the Smallest Model that Achieves the Job alongside State-of-the-Art performance and urge companies to disclose foundation-model carbon footprints or information enabling estimates.They connect transparency with researchers making choices based on sustainability as well as performance.Source: §5 · p. 7
- Separating Amax from k is intended to model learning more accurately and avoid accuracy/carbon metrics that could reward under-trained models.Amax represents accuracy and k training efficiency.Source: §5 · p. 7
- Future work could set Amax to the highest-performing model when calculating other models' k values, better reflecting carbon per accuracy while accounting for diminishing returns.The authors present this as a possible refinement, not the current procedure.Source: §5 · p. 8
- carbonbenchmark has been demonstrated on only three relatively lightweight benchmarks and six established models, leaving its fit on more computationally expensive learning curves unclear.The authors say this requires further investigation; codecarbon also considers CO2, while water use and local-grid energy impacts are not included.Source: §5 · p. 8
- CO2-based efficiency comparisons may unfairly penalize researchers using dirtier power grids or less-efficient CPUs, GPUs, or RAM.The authors identify an equitable metric that captures CPU, GPU, and RAM costs as future work.Source: §5 · p. 8
A Appendix - Proofs of Theorems
The appendix establishes that the paper's exponential and power-law learning-curve forms do not generally yield closed-form training-efficiency parameters from observed accuracy and training cost. It gives the algebraic and data-fitting constraints behind those results.
A Appendix - Proofs of Theorems
- The exponential learning curve A = Amax(1 − e^−kt) has no general closed-form analytic solution for k from known A and t values.The proof uses two observations to solve for k and Amax; when the ratio of training costs is rational, the resulting equation has a solution only if n < 5, and an irrational ratio yields a strictly transcendental equation.Source: A Appendix - Proofs of Theorems · pp. 9–10
- For the exponential case, the proof reduces the rational training-cost ratio to an equation that has a solution only when n < 5; for an irrational ratio, the equation is strictly transcendental.The appendix concludes there is no general solution in either case.Source: A Appendix - Proofs of Theorems · pp. 9–10
- The power-law form A = Amax(1 − αt^−β) likewise has no general closed-form analytic solution for β from known A and t values.The proof requires three data points for α, β, and Amax; the authors also note that three arbitrary points are unlikely to fall exactly on this curve.Source: A Appendix - Proofs of Theorems · p. 10
B Appendix - Least Square Fitting with Uncertainty
The appendix describes fitting learning curves to observed training costs and accuracies, then estimating parameter uncertainty. It uses iterative numerical least squares because the fitting function is nonlinear.
B Appendix - Least Square Fitting with Uncertainty
- The fitting procedure represents observations as training costs and corresponding accuracies, with exponential, power-law, or delayed-exponential forms parameterized by θ.The listed parameter sets are (Amax, k), (Amax, α, β), and (Amax, k, t0), respectively.Source: B Appendix - Least Square Fitting with… · p. 10
- Predicted accuracies come from the selected learning-curve function, and fitting chooses parameters that minimize the sum of squared residuals.The appendix gives a predicted value for the delayed exponential form as an example.Source: B Appendix - Least Square Fitting with… · p. 10
- Because the fitting function is nonlinear, the appendix uses scipy.optimize.curve_fit iteratively; its Jacobian linearizes the function around current parameters to guide updates toward the residual minimum.The Jacobian determines the update direction and magnitude.Source: B Appendix - Least Square Fitting with… · p. 11
- After fitting, the Jacobian yields a covariance matrix whose diagonal elements provide parameter standard errors; the resulting uncertainty estimates are used for learned parameters.The appendix identifies these as estimates of uncertainty in the fitted parameters.Source: B Appendix - Least Square Fitting with… · p. 11
C Appendix - Additional Experimental Results for Learning Curve Fitting
The appendix presents learning curves and mathematical fits for five models across image, text, and tabular benchmarks.
C Appendix - Additional Experimental Results for Learning Curve Fitting
- Figure 3 evaluates ResNet18 on FashionMNIST, while Figure 4 evaluates Logistic Regression on SST-2.Both are described as learning curves and mathematical fits for computational efficiency.Source: Figure 3 · p. 11, Figure 4 · p. 11
- Figure 5 evaluates DistilBERT fine-tuned on SST-2, while Figure 6 evaluates XGBoost on the Adult tabular dataset.Both figures present learning curves and mathematical fits for computational efficiency.Source: Figure 5 · p. 12, Figure 6 · p. 12
- Figure 7 evaluates FT-Transformer on the Adult tabular dataset using a learning curve and mathematical fits.The figure evaluates computational efficiency.Source: Figure 7 · p. 12
D Appendix - Literature Review Results and Prompt
The appendix describes the literature-review prompt and its coding criteria, and reports a comparison of random and author-name-ordered paper samples.
D Appendix - Literature Review Results and Prompt
- Randomly selected and first-by-author-name samples of 200 papers showed no significant difference.The appendix reports this check when comparing models.Source: D Appendix - Literature Review Results… · p. 13
- The prompt asks models to review an attached article and classify whether it used LLMs and the types of LLM use.The listed categories include uses such as architecture or pre-training, fine-tuning, inference or evaluation, and synthetic-data generation.Source: D Appendix - Literature Review Results… · p. 13
- The prompt permits multiple usage categories, treats reasoning or task-solving as Agentic Frameworks, and says to write “None” when the paper did not use LLMs.The category examples also include synthetic-data generation and labelling data.Source: D Appendix - Literature Review Results… · p. 13
- Reviewers record open-source status, standard-benchmark training, and telemetry availability using the prompt’s stated criteria.Telemetry distinguishes controlled or rented infrastructure from managed platforms or APIs that abstract hardware.Source: D Appendix - Literature Review Results… · p. 13
- The prompt separately records whether computational cost and sustainability or environmental impact are reported.It specifies “NA” for computational cost when no experiments or compute were used.Source: D Appendix - Literature Review Results… · pp. 13–14
- Table 1 presents pairwise comparisons of LLMs against Gemini 3.1 Pro Preview using Dunnett’s test.Source: Table 1 · p. 14
E Appendix - Statistical Comparison of LLMs in Literature Review Task
The appendix compares LLM review responses with Gemini 3.1 Pro Preview using a mixed model and Dunnett’s test. It reports no strong evidence of a difference for four models.
E Appendix - Statistical Comparison of LLMs in Literature Review Task
- The GLMM models each LLM’s yes/no response to each question as Bernoulli, with question and LLM fixed effects and paper as a random effect.The response is indexed by LLM, question, and paper.Source: E Appendix - Statistical Comparison of… · p. 14
- No strong evidence indicated a difference from Gemini 3.1 Pro Preview for Haiku 4.5, Claude Opus 4.6, Claude Sonnet 4.6, or Gemini 3.1 Flash Lite.The authors retain the null hypothesis of no evidence of a difference for these four models.Source: E Appendix - Statistical Comparison of… · p. 14
- The authors fit the GLMM with glmmTMB and used Dunnett’s test to compare each LLM with Gemini 3.1 Pro Preview.They report using R’s emmeans package for the multiple comparisons.Source: E Appendix - Statistical Comparison of… · p. 14