PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Energy Cost of Completing Tasks with Quantized Language-Model Agents

Stephen Abkin, Aldric Yang, Alex Flesher, Nitheesh Kodarapu, Deepak Panigrahy, Aakash Tyagi, David Kebo Houngninou

arXiv:2610.32417v2cs.PF

TL;DR

Per-token energy benchmarks do not capture the repeated calls, tools, and revisions involved in completing agent tasks, and prior studies leave task-level comparisons across precisions underexplored. The paper measures energy per token and per completed task across verifiable tasks and models. It finds that 4-bit is usually cheaper per token, while task-level cost depends on the task, model, and execution budget.

  • Problem

    Prior energy studies report per-token or per-request costs, measure agents at one precision, or compare precision on single-turn queries, leaving task-level energy trade-offs across precisions uncharacterized.

    Source: §1 · p. 1

  • Method

    The study measures CPU and GPU energy per output token and per completed task for 4-bit and 8-bit models across 29 verifiable tasks, with additional budget, model, and held-out benchmark comparisons.

    Source: §1 · pp. 2–3

  • Results

    4-bit uses 6–16% less energy per output token on average, but task-level cost depends on the task, model, and execution budget and can reach 3.5 times the 8-bit cost.

    Source: §1 · pp. 2–3, §7 · p. 20

  • Takeaways & Limitations

    Precision choice should be based on energy per completed task and tokens per completed task for the model and tasks under the intended deployment conditions.

    Source: §1 · p. 3, §7 · p. 20

    The study measures one-at-a-time requests on a dedicated GPU for 24- or 30-billion-parameter models, and its tasks represent short- and medium-length work with automatically checked outcomes.

    Source: §6.2 · pp. 19–20

Abstract

from arXiv · show

Quantization reduces the memory and energy needed to serve large language models (LLMs) and AI agents for coding and data analysis can now run on locally hosted LLMs. In practice, this benefit is commonly benchmarked per output token. Yet an agent calls its model repeatedly, runs tools, and revises its work, and quantization can change both how much the agent generates and whether it succeeds. Existing energy studies either compare quantized models on single requests or measure agents at one fixed precision, overlooking how a change in precision plays out across a complete agent task. To address this gap, this paper presents the first characterization of the task-level energy trade-offs of model quantization in locally hosted agents. We develop 29 agent tasks with automatic pass/fail checks in six families, ranging from single-answer SQL analytics to multi-step repository bug fixing, and measure CPU and GPU energy per output token and per completed task for 4-bit and 8-bit versions of two open-weight models on bare-metal NVIDIA A100 nodes with llama$.$cpp. Our findings reveal that 4-bit precision generally saves energy per token, though whether it saves energy per completed task is task dependent and changes with the model and the execution budget (i.e., limits on turns, tokens, and time). We trace this discrepancy to the extra tokens needed to complete a task in full, which translates into using up to 3.5 times as much energy per completed task. These findings offer actionable guidance for developers and operators choosing model precision for agents, showing how tokens per completed task, set against the per-token saving, reveal whether lower precision completes the work with less energy.

1 Introduction

Per-token energy benchmarks may not predict the energy of a completed agent task, which includes repeated model calls, tool runs, and revisions. The paper measures both energy per token and per completed task across quantized-agent tasks to examine when the measures favor different precisions.

1 Introduction

  • Prior energy studies report per-token or per-request costs, measure agents at one precision, or compare precision on single-turn queries, leaving task-level energy comparisons underexplored.The paper argues that task energy includes model calls, tools, and waiting, so per-token cost alone does not capture the full cost of completing work.Source: §1 · p. 2
  • The study compares 4-bit Q4_K_M and 8-bit Q8_0 on 29 automatically verifiable tasks, then tests expanded budgets, a 16-bit reference, a second model, and held-out benchmarks.It examines whether per-token and per-task measures favor the same precision, what accounts for disagreements, and how budgets, models, and task sets affect the comparison.Source: §1 · p. 2
  • 4-bit uses 6–16% less energy per output token on average, yet can use more energy per completed task depending on task, model, and execution budget.On some tasks, 4-bit uses a median of 66% more energy per completed task than 8-bit, and the worst supported case uses 3.5 times as much.Source: §1 · pp. 2–3
  • A precision’s per-token saving can be outweighed by the extra tokens needed per completed task, including tokens spent on malformed tool calls, unnecessary test reruns, and failed attempts.Where the measures disagree, the lower-per-token-cost precision needs a median of 1.6–2.0 times as many tokens per completed task.Source: §1 · p. 3
  • The authors recommend measuring energy per completed task for the model and tasks an agent will run, under its deployment circumstances.Their contribution also defines a break-even identity relating energy per token to tokens per completed task.Source: §1 · p. 3

2 Background and Related Work

Quantized model files store more than their nominal weight bit width because block-level scales, offsets, and some higher-bit tensors add storage. Prior energy measures extend beyond tokens, while earlier agent studies often fixed precision or did not measure energy across precisions on the same attempts.

2.1 Quantized models for local inference

  • Q4_K_M stores most weights at 4 bits but includes shared scales and offsets plus some 6-bit tensors; Q8_0 uses 8-bit blocks with one scale and no calibration.The measured Qwen 4-bit file stores 4.88 bits per weight and is about 30% the size of its 16-bit file.Source: §2.1 · p. 3

2.2 From energy per token to energy per completed task

  • Energy per response, request, or successful outcome captures costs that per-output-token measures can miss, including response-length variation and failed attempts.Prior studies also report that thinking mode can increase tokens per problem while energy per token remains nearly unchanged.Source: §2.2 · p. 4
  • Earlier agent-energy studies held precision fixed, while the closest studies separately examined failed-run energy or precision-related tool-call behavior.This paper combines precision variation and energy measurement on the same agent attempts.Source: §2.2 · p. 4
  • Figure 2 distinguishes model-call energy from whole-task-window energy and compares tokens, energy per output token, and energy per completed task for a one-million-row business query.The figure caption reports that 4-bit is cheaper per token but costlier per completed task in this example.Source: Figure 2 · p. 5

3 Three Energy Measures

The section defines three energy measures for comparing precisions across agent tasks and derives how tokens per completion and energy outside model calls can change the per-token choice. It then specifies how to classify and quantify disagreements between per-token and per-task choices.

3.1 Attempts and the three measures

  • Energy per token, model-call energy per completed task, and operational energy per completed task distinguish token cost, completion cost, and whole-task cost.Attempts alternate model and tool calls; operational energy includes tools and waiting, while model loading and verifier energy are excluded.Source: §3.1 · pp. 5–6

3.2 The token factor, the boundary factor, and the break-even rule

  • Energy per completed task combines energy per token with tokens per completion and the operational-to-model-call energy ratio.Tokens per completion split into mean tokens per attempt and completion rate; longer attempts or lower completion rates increase the token factor.Source: §3.2 · p. 6
  • In the business query, 4-bit saves 14% per token but uses 29% more energy per completed task because its token and boundary factors exceed break-even.It needs 59% more tokens per completion, while both precisions complete all ten attempts; the boundary factor is 0.94.Source: §3.2 · p. 7

3.3 Choosing a precision by each measure

  • A task is comparable only when both precisions complete it at least three times; disagreement occurs when per-token and per-task choices differ.The intermediate model-call measure locates whether disagreement reflects tokens per completion or tool and waiting energy; the reported extra cost is relative to the per-task choice.Source: §3.3 · p. 7, Table 1 · p. 7

4 Experimental Methodology

The methodology compares precision choices under controlled, task-level conditions, using verified completions and energy attributed to model and tool activity. It specifies the testbed, task set, execution budgets, measurement checks, comparison conditions, and statistical procedures.

4 Experimental Methodology

  • Each comparison holds the machine, engine, tasks, tools, budget, and seeds constant while measuring energy, tokens, outcomes, and validity for both precisions.Valid attempts, including failed ones, contribute to each precision’s energy-per-token and operational-energy totals.Source: §4 · p. 8, Figure 3 · p. 9

4.1 Testbed and models

  • The study runs Qwen3-30B-A3B and Devstral Small 2 on bare-metal A100 machines, with both quantized model files fitting in one GPU’s memory.The main model is a 30.5-billion-parameter mixture of experts with 3.3 billion active per token; Devstral is a dense 24-billion-parameter model.Source: §4.1 · p. 8

4.2 Tasks and verification

  • The 29 tasks from 22 problems span six families, and hidden-reference automatic verifiers determine pass or fail after submission.Families range from SQL analytics and data repair to function implementation and repository repair; verifiers reject known shortcuts.Source: §4.2 · pp. 8–10

4.3 Agent protocol, budgets, and repetitions

  • Execution budgets cap context, turns, tool calls, output tokens, and time; paired precision runs use matched seeds and randomized order.About two thirds of primary-budget repository attempts reach the context or turn limit, motivating the expanded budget.Source: §4.3 · pp. 10–11

4.4 Energy measurement and attempt validity

  • CPU-package and GPU-board counters are sampled every 0.1 seconds; energy is attributed to model-call and tool-call spans within the task window.Invalid attempts are excluded from energy and completion counts and are not rerun; validity checks cover energy records, token counts, and lingering processes.Source: §4.4 · pp. 11–12

4.5 Conditions

  • Conditions compare Qwen with the primary budget, expanded-budget reruns, an expanded-budget 16-bit reference, and expanded-budget Devstral runs.Held-out evaluation adds 40 BIRD SQL questions and 37 eligible EvalPerf problems, selected independently of model performance.Source: §4.5 · p. 12

4.6 Statistical analysis

  • Task-level precision choices use bootstrap intervals or exact completion tests, while label-permutation tests estimate chance disagreements across tasks.Task and family summaries use geometric means, which can conceal task-level differences; the main analysis was fixed before the experiment, while extensions are exploratory.Source: §4.6 · pp. 12–13

5 Results

Across the three conditions that run all 29 tasks, 4-bit uses less energy per output token, yet task-level energy can favor 8-bit. Extra tokens, failed attempts, and changes in model, budget, and task family help explain why the cheaper precision per completed task varies.

5.1 RQ1: Whether the two measures favor the same precision

  • Across 29 tasks, 4-bit uses 6–16% less energy per output token than 8-bit on average, depending on condition; three comparable tasks per condition cost more per completed task beyond run-to-run variation.Source: §5.1 · pp. 13–14

5.2 RQ2: The cost of choosing by energy per token

  • On supported disagreements, the per-token choice costs a median 66% more energy per completed task, reaching 3.5 times as much as the per-task choice.Condition averages dilute task-level differences: the 4-bit/8-bit ratio per completed task is 1.17 in the main experiment and 0.92 in Qwen expanded and Devstral.Source: §5.2 · p. 15

5.3 RQ3: Where the extra energy comes from

  • The per-token-cheaper precision needs a median 1.6 to 2.0 times as many tokens per completed task, beyond the 6–16% per-token saving’s break-even range.Extra tokens arise from longer attempts and failed attempts; failures carry most of the extra energy in 7 of 10 primary-budget disagreements, while longer attempts do so under the expanded budget.Source: §5.3 · pp. 15, 17
  • Malformed tool calls, narrated plans, and repeated tests add tokens; in one business query, 4-bit used about 60% more output tokens despite the same correct answer.Across the 20 attempts, both precisions used the same schema lookup and single query; 4-bit averaged 430 output tokens versus 271 for 8-bit.Source: §5.3 · pp. 15–16

5.4 RQ4: What changes the cheaper precision per completed task

  • With the expanded budget, Qwen completes more repository repairs at 4-bit, while the two precisions remain within run-to-run variation in energy per completed task.In the repository rerun, 4-bit completes 45 of 89 attempts versus 26 of 89 for 8-bit; the energy ratio is 0.71 [0.32, 1.24].Source: §5.4 · p. 17
  • On held-out tasks, Qwen favors 4-bit per completed task on EvalPerf, while neither precision is supported as cheaper on BIRD.The EvalPerf ratio is 0.90 [0.85, 0.98]; the BIRD ratio is 0.91 [0.77, 1.09]. In Qwen expanded, 16-bit uses 31% more energy per token than 8-bit and is not clearly cheaper per completed task.Source: §5.4 · p. 17
  • 4-bit remains cheaper per token across conditions, but task-family preferences shift; SQL analytics favors 8-bit in the main experiment and 4-bit for Devstral.Table 6 reports family-level 4-bit/8-bit ratios of energy per completed task; the regression-control average can hide opposite results at different input sizes.Source: §5.4 · p. 17, Table 6 · p. 18

6 Discussion

Precision choice should be based on energy per completed task and tokens per completed task, not per-token energy alone. The recommendations are bounded by the study’s single-GPU setting, shared budgets, and fixed verifiable tasks; per-attempt records support reproducibility.

6.1 Choosing a precision in practice

  • A 6–16% per-token saving covers at most 6–19% more tokens per completed task in model-call energy, so developers should track tokens across repeated attempts.Short-function and held-out EvalPerf tasks support starting with 4-bit; SQL results vary by condition, while 4-bit completed at least as many repairs as 8-bit in every expanded-budget condition.Source: §6.1 · pp. 18–19
  • Benchmark designers should report energy per completed task beside per-token energy, charge failed attempts to completions, repeat attempts, and state the measurement boundary and budget.They should include agent-run tests in task energy but exclude the separate final-result check.Source: §6.1 · p. 19

6.2 Scope and limitations

  • The deployment study uses one request at a time on a dedicated GPU, excludes model loading, and covers two memory sizes of one GPU generation and one processor family.Shared or batched serving would divide idle and host energy among tenants.Source: §6.2 · p. 19
  • The task interfaces provide a few tool actions rather than a general shell, and malformed tool calls consume a turn; both precisions share one budget.Repairing malformed calls or constraining output with a llama.cpp grammar could reduce this cost; precision-specific budgets could give 4-bit the longer context its smaller file allows.Source: §6.2 · p. 19
  • The study uses fixed verifiable tasks; open-ended work such as web interaction, research, or planning would require judged or graded outcomes.The tasks represent short- and medium-length work, though the authors state that the same per-task accounting applies to multi-hour projects and thinking-mode models.Source: §6.2 · pp. 19–20

6.3 Reproducibility

  • Per-attempt records preserve settings, energy readings, call timing, token counts, verifier decisions, validity checks, and reasons for incomplete or invalid attempts.File hashes and manifests identify assets, scripts, machines, interfaces, and harness versions; the authors state they will release code, task definitions, verifiers, settings, and records upon acceptance.Source: §6.3 · p. 20

7 Conclusion

The conclusion finds that 4-bit is cheaper per token almost everywhere, but the cheaper precision per completed task depends on task, model, and budget. Extra tokens from agent behavior can outweigh per-token savings, so developers and benchmark designers should account for completed-task energy.

7 Conclusion

  • The cheaper precision per completed task depends on the task, model, and execution budget, even though 4-bit is cheaper per token almost everywhere.Extra tokens from malformed tool calls, narrated plans, repeated tests, and failed attempts can outweigh the per-token saving.Source: §7 · p. 20
  • Developers should track tokens per completed task across repeated attempts, and benchmark designers should report completed-task energy beside energy per token.The conclusion frames energy cost as arriving per completed task for agent hosts.Source: §7 · p. 20

A Verifier isolation and adversarial checks

The verifier checks were designed to isolate agent work and resist unsafe or invalid submissions. Every adversarial check had to pass before the verifier was used in an experiment.

A Verifier isolation and adversarial checks

  • Expected answers, reference queries, and private seeds stay outside the agent workspace, and verifier inputs are drawn from private seeds for every precision.Repository verification compares library outputs with independent implementations on 32 random graphs; function and verifier code run in sandboxed child processes.Source: A Verifier isolation and adversarial… · p. 24
  • SQL tools deny writes and access to catalogs, pragmas, and attached databases, while data-processing functions run under a syntactic allow-list.Table 7 states that every adversarial check had to pass before the experiment using that verifier.Source: A Verifier isolation and adversarial… · p. 24, Table 7 · p. 24

B How attempts ended under each budget

Attempts could end at context or time limits, but every valid sampled attempt in the main experiment finished before its deadline. The expanded deadline ended one valid Qwen 4-bit repository attempt.

B How attempts ended under each budget

  • Every valid sampled attempt in the main experiment finished before the 600 s deadline; the expanded 1,800 s deadline ended one valid attempt.That attempt was a Qwen expanded-budget 4-bit repository attempt after 97 turns; model loading took 8–12 s under the primary budget and 8–17 s under the expanded budget, at most 2% of either deadline.Source: B How attempts ended under each budget · p. 24

C Energy-record checks, model loading, and verification energy

This section checks energy-record integrity and accounts for the energy costs of model loading and verification. It also reports how often sampled attempts submit under the expanded budget.

Energy-record checks

  • Recomputed energy periods must match stored values within a relative tolerance of 10−12; other than qualifying counter wraps, any CPU-counter decrease invalidates the attempt.A decrease counts as a wrap only when it falls from the top quarter of the counter’s approximately 65.5 kJ range to the bottom quarter.Source: C Energy-record checks, model loading,… · p. 24

Model loading

  • A 4-bit load uses 0.4 to 0.7 kJ less than an 8-bit load, but Section 5.1 charges one load to every attempt as a worst case.Median loading energy per sampled attempt is 1.55 to 2.14 kJ for 4-bit and 2.00 to 2.87 kJ for 8-bit across conditions; loading costs 0.07 to 1.88 and 0.05 to 2.40 times median operational energy, respectively.Source: C Energy-record checks, model loading,… · p. 24

Submission

  • Outside repository repair, nearly all sampled attempts submit; under the expanded budget, all attempts on other families submitted in Qwen expanded and Devstral.All attempts stayed within the tool-call limit, and one reached the total output-token limit.Source: Table 8 · p. 25

Verification energy

  • The research verifier is excluded from agent energy; its median energy is at most 0.13 kJ, rising to 10.0 kJ for 4-bit repository-rerun attempts.Submitted patches run the test suite in the repository rerun.Source: C Energy-record checks, model loading,… · p. 25

D Sensitivity analyses and energy drift

The sensitivity analyses report that the Section 5 conclusions persist under alternative data handling and analysis choices. Energy-per-token drift across repetitions is small and affects both precisions similarly.

Sensitivity analyses

Missing energy records

  • All three incomplete energy records support the same conclusions once rebuilt or bounded.Table 9 rebuilds two records cut off during generation from prior readings and bounds an 8-bit record lost after verified completion by the task’s range.Source: D Sensitivity analyses and energy drift · p. 25

Partial repetitions

  • The analysis uses machine C for partial repetitions because both sampled repetitions there finished.Two conditions have partial repetitions: the eighth in Qwen expanded and a held-out repetition on machine A; held-out repetitions were assigned before runs began.Source: D Sensitivity analyses and energy drift · p. 25

Drift

  • Matched attempts’ energy per output token changed by −0.06%, +0.35%, and −0.02% per repetition on machines A, D, and B, respectively.The 4-bit and 8-bit slopes differed by less than 0.1% per repetition on every machine; machine D’s roughly 3% swings cancel in the ratio because both precisions run in the same repetition.Source: D Sensitivity analyses and energy drift · p. 25

E Held-out task selection and statistical procedures

Held-out task selection and statistical procedures are specified alongside robustness checks. The reported conclusions hold under alternative analyses, with energy ratios and disagreement counts evaluated on comparable tasks.

Held-out subsets

Function-implementation problems

  • Of 112 EvalPerf problems outside the main set, 37 met eligibility criteria and all 37 were used.Eligibility required correct canonical, fastest, and slowest reference solutions, with the slowest taking at least three times the time limit; every failure had another cause, so the time-limit and 10−6 regression thresholds left comparisons unchanged.Source: E Held-out task selection and… · p. 25

Alternative analyses

  • Table 9 reports that conclusions hold under each alternative analysis, using 4-bit/8-bit energy-per-completed-task ratios and disagreement counts for comparable tasks.Ratios include 95% bootstrap intervals.Source: Table 9 · p. 26

BIRD questions

  • The BIRD sample comprises the first 40 questions selected by a seeded shuffle of 925 questions labeled simple, covering 11 databases.The source was the June 2024 development release, with at most five questions per database.Source: E Held-out task selection and… · p. 26

Intervals

  • Condition and family summaries resample attempts within tasks, while held-out summaries resample tasks and then attempts, grouping BIRD draws by database.The procedure uses 2,000 draws for condition and family summaries and 5,000 for held-out summaries; paired draws keep both precisions together, and draws without a completion are omitted.Source: E Held-out task selection and… · p. 26

Noise baseline

  • The second noise shuffle produced 8.6, 8.2, and 4.8 disagreements on average, versus 10, 10, and 8 observed; both shuffles averaged fewer than one supported disagreement.Three or more disagreements occurred in at most 2.7% of full shuffles and 1.7% of second shuffles; the baseline estimates disagreements expected by luck alone.Source: E Held-out task selection and… · p. 26

Related papers

Loading 2610.32417v2…