Source-linked AI summary
Towards Sustainable Web Agents: A Plea for Transparency and Dedicated Metrics for Energy Consumption
Lars Krupp, Daniel Geißler, Paul Lukowicz, Jakob Karolus
TL;DR
Web agents’ growing capabilities make sustainability a pressing question, but their energy costs remain underexplored. The paper estimates and compares two agents, finding large design-linked differences in energy use, and calls for dedicated metrics and transparency.
Problem
Web agents’ sustainability remains underexplored despite their growing capabilities and potentially high computational costs.
Source: §1 · p. 1Method
The paper estimates and compares energy consumption for MindAct and LASER, two web agents with different design philosophies.
Source: §1 · p. 2Results
Web-agent design is associated with energy differences of orders of magnitude; LASER uses approximately 2,900 times MindAct’s lower-bound energy and 1,500 times its upper-bound energy.
Source: §3.3 · p. 5, §6 · p. 7Takeaways & Limitations
The authors advocate standardized energy metrics and transparent model-parameter disclosure to support sustainability evaluation and comparisons between web agents.
Source: §1 · p. 2, §4 · p. 6Energy estimates are coarse and conservative because trustworthy information on LLM energy costs, especially for proprietary models, is limited.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Improvements in the area of large language models have shifted towards the construction of models capable of using external tools and interpreting their outputs. These so-called web agents have the ability to interact autonomously with the internet. This allows them to become powerful daily assistants handling time-consuming, repetitive tasks while supporting users in their daily activities. While web agent research is thriving, the sustainability aspect of this research direction remains largely unexplored. We provide an initial exploration of the energy and CO2 cost associated with web agents. Our results show how different philosophies in web agent creation can severely impact the associated expended energy. We highlight lacking transparency regarding the disclosure of model parameters and processes used for some web agents as a limiting factor when estimating energy consumption. As such, our work advocates a change in thinking when evaluating web agents, warranting dedicated metrics for energy consumption and sustainability.
1 Introduction
Web agents can automate internet tasks, but their computational and energy costs raise sustainability concerns. The paper compares two agents to show how design affects energy use and advocates transparent, dedicated sustainability metrics.
1 Introduction
- Web agents’ expanding context windows increase energy consumption, adding to concerns about the sustainability of this research direction.Source: §1 · p. 1
- LASER is estimated to use 2,930 Wh per Mind2Web action and emit approximately 9.6 kg CO2 per task.The paper compares the task emissions to a car driving 39 km.Source: §1 · p. 2
- A conservative estimate puts LASER’s energy use at approximately 1,500 times MindAct’s.The comparison illustrates how different web-agent design philosophies can affect energy consumption.Source: §1 · p. 2
- The paper advocates dedicated energy metrics and a more holistic evaluation of web-agent performance.It also calls for disclosure of model parameters to improve transparency.Source: §1 · p. 2
2 Related Work
Web-agent diversity makes performance comparisons difficult, while LLM training and deployment consume energy and have broader environmental impacts. Existing estimates depend on uncertain factors, and energy use is only one part of the lifecycle picture.
2 Related Work
- Web-agent performance is often difficult to compare because benchmarks and agent designs vary.Approaches differ in inputs, preprocessing, memory, and whether they use open-source or proprietary models.Source: §2 · p. 2
2.1 Environmental Impact of LLMs
- Training large LLMs requires large-scale data centers, while public information about model details is usually limited.A cited earlier version of GPT-3 used 175 billion parameters and was trained on 570 GB of data.Source: §2.1 · p. 2
- Lannelongue et al. (2021) estimate 0.754 metric tons of CO2 for a 79-hour BERT training run on 64 Tesla V100 GPUs.The passage also reports an estimated 550 metric tons of CO2 for GPT-3 training; training estimates vary with hardware, optimization, and energy mix, and energy can also be spent on ineffective models or hyperparameter tuning.Source: §2.1 · pp. 2–3
- Inference energy depends on hardware utilization, while lifecycle impacts also include cooling water and e-waste.Energy per token is proposed as a metric for comparing sustainability, especially across quantized LLM versions.Source: §2.1 · p. 3
3 Estimating the Energy Consumption of Web Agents
The paper estimates and compares the energy use of MindAct and LASER, showing that their different designs produce sharply different costs. It also estimates task-level CO2 emissions while noting that limited public information constrains precision.
3 Estimating the Energy Consumption of Web Agents
- The comparison estimates energy from publications and source code, directly tests MindAct's open-source models, and uses Mind2Web's real-world websites as its evaluation benchmark.Mind2Web averages 1,135 HTML elements per website and 7.3 actions per task.Source: §3 · p. 3
3.1 MindAct
- 0.997432 Wh and 1.947816 Wh: MindAct's estimated lower- and upper-bound energy per action, using DeBERTa ranking followed by ten flan-T5XL questions.The estimate uses a minimum-to-maximum range of one to three times the original HTML token count for DeBERTa processing.Source: §3.1 · pp. 3–4
3.2 LASER
- LASER uses proprietary GPT-4, one-shot step-by-step prompting, state-limited actions, and memory; its raw-HTML input was inferred, and energy per token was proxied from token pricing.The authors note that OpenAI did not publish figures enabling precise energy estimates.Source: §3.2 · p. 5
3.3 Comparing MindAct and LASER
- LASER consumes approximately 2,900 times MindAct's lower-bound energy and 1,500 times its upper-bound energy; DeBERTa preprocessing uses 15,000 times less energy per token than GPT-4.The authors identify MindAct's preprocessing as important to its energy efficiency.Source: §3.3 · p. 5
3.4 Carbon Dioxide Emissions
- 3.30 g and 6.44 g: MindAct's lower- and upper-bound task emissions compare with 9,691.08 g for LASER.These estimates correspond to driving approximately 13–25 m for MindAct and 39 km for LASER with an average car.Source: §3.4 · p. 6
4 A Plea for Dedicated Metrics and Small Models
The paper calls for standardized energy reporting and energy-aware evaluation of web agents, including attention to model size and pipeline design. It favors energy metrics over CO2 reporting for cross-region comparability, while treating proposed metrics as indicators rather than exact accounting.
4 A Plea for Dedicated Metrics and Small Models
- Report tokens per action and energy cost per token for each LLM, using standardized energy metrics to compare agents and evaluate efficiency.The paper argues that these metrics support holistic evaluation as web agents become more widely used.Source: §4 · p. 6
- These metrics indicate sustainability and improve transparency, but do not provide perfectly accurate energy calculations.The paper notes that local testing to obtain the metrics is feasible especially for agents using open-source LLMs.Source: §4 · p. 6
- Reporting energy rather than CO2 makes agent results independent of the evaluation country's energy mix and permits conversion to CO2 using that mix.Source: §4 · p. 6
- Smaller models can handle predefined tasks such as candidate generation, reducing the tokens larger models process and supporting modular, more sustainable, reproducible pipelines.The paper recommends using small models for efficient preprocessing wherever possible.Source: §4 · pp. 6–7
5 Limitations
The energy estimates are limited by the availability of trustworthy information about LLM energy costs, particularly for proprietary models that cannot be run locally for measurement. The authors therefore use coarse, conservative estimates in orders of magnitude.
5 Limitations
- Trustworthy information about LLM energy costs constrains the accuracy and certainty of the calculations, especially for proprietary models that cannot be run locally to measure energy use.Source: §5 · p. 7
- The authors use coarse, conservative estimates in orders of magnitude because precise energy measurements are unavailable.Source: §5 · p. 7
- Disclosure of model energy costs would significantly improve estimate accuracy; despite the coarse estimates, the authors say they can draw implications and make generalizable recommendations.Source: §5 · p. 7
6 Conclusion
The conclusion reports that web-agent design can change energy consumption by orders of magnitude and advocates standardized energy metrics. It also identifies sustainability awareness and transparency as gaps, and recommends accounting for energy use in agent evaluation.
6 Conclusion
- The comparison of MindAct and LASER indicates that web-agent design can influence energy consumption by orders of magnitude, motivating standardized evaluation metrics.Source: §6 · p. 7
- The work identifies limited sustainability awareness and transparency as gaps in web-agent research.Source: §6 · p. 7
- The authors argue that sustainability matters because of web agents' environmental impact at scale, and recommend smaller models and efficient algorithms to reduce the data larger LLMs process.They also advocate dedicated metrics that make energy consumption comparable across agents performing the same task.Source: §6 · p. 7