PaperlayerView exact revision on arXiv

Source-linked AI summary

Joules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?

Amirreza Zeinali, Jon Saad-Falcon, Avanika Narayan, Mersad Abbasi, Jonathan S. Hartley, John Hennessy, Christopher Re, Azalia Mirhoseini

arXiv:2610.30858v1cs.LGecon.GNq-fin.EC

TL;DR

Existing approaches assess AI output quality or time saving, while energy benchmarks do not quantify the economic value of professional outputs. This paper evaluates output value and expert time saved per measured end-to-end energy across occupational tasks and AI deployments. Results show that per-joule leaders can differ from task-level leaders, and that model and accelerator choices trade off value, time, and energy.

  • Problem

    Energy benchmarks measure consumption but do not quantify the economic value of the professional outputs produced with that energy.

    Source: §1 · p. 2

  • Method

    The framework measures professional-output value and expert time saved alongside end-to-end energy to compare occupational work, models, and accelerators.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Per-joule occupational leaders differ from task-level leaders, while model and accelerator comparisons reveal distinct value, time-saving, and energy tradeoffs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Under limited energy, accounting for total benefit and return per joule can guide routing, stopping, and serving decisions across tasks, models, and accelerators.

    Source: §5 · p. 11

    The estimates use benchmark tasks, recorded expert times, and occupation median wages, and whether the same returns hold beyond the benchmark remains untested.

    Source: §1 · p. 3, §4 · p. 11

Abstract

from arXiv · show

An AI system can produce more economically valuable work while delivering less value for the energy it consumes. We therefore ask: How much economic value does an AI system create per unit of energy consumed? We formalize economic value in two ways: (1) net dollar value, the output's dollar value minus AI-system execution cost, and (2) time saving, the expert's completion time minus the AI system's completion time. Dividing each by measured end-to-end energy yields Net Value per Joule (NVJ) and Time Saving per Joule (TSJ), reported in USD/MJ and hours/MJ. We study 220 public GDPval tasks across 44 occupations, serving five open-weight models locally and evaluating four accelerator types. First, the occupation groups where AI systems succeed most often, deliver the greatest dollar value, or save the most time do not necessarily provide the greatest economic return per joule. For example, among the tested tasks, engineering work delivers about 10% more net dollar value per task than software development. Yet software development delivers about 27% more net value per joule. Second, GLM 5.2 delivers the highest average net dollar value, at $72.27 per task using 0.430 MJ, equivalent in energy to about four hours of active laptop use. By contrast, Gemma 4 31B delivers 56% of that net dollar value using just 23% of the energy, achieving approximately 2.5 times the NVJ and 2.8 times the TSJ. Third, across four single-accelerator deployments, Qwen3.6-27B's NVJ varies by a factor of 4.32 and TSJ by 9.27, with B200 leading both. Additionally, one and eight H100s deliver similar net dollar value, but eight reduce completion time by 22% at 3.24 times the energy. Our framework makes output value and expert time saving under an energy budget explicit objectives for improving models, accelerators, and software, while offering a new basis for assessing AI's potential economic effect on occupational work.

1 Introduction

The paper links judged professional-output value with measured energy, using per-joule returns to compare occupational work and AI deployments. Its results show that leaders by task-level performance can differ from leaders per joule, and that energy efficiency varies across models and accelerators.

1 Introduction

  • Energy benchmarks measure consumption without quantifying the economic value of professional outputs, so the paper measures output value alongside the energy required to produce it.It uses energy because joules are stable while API prices can change, and end-to-end energy combines model, hardware, tool use, waits, and retries.Source: §1 · p. 2
  • Per-joule leaders differ from task-level leaders: investigation leads success rate and software development leads net value per joule, while concierge and recreation lead success rate before energy accounting.Factory supervision falls from second in time saving per task to sixth in time saving per joule.Source: §1 · pp. 2–3
  • $74.19 per task is GLM 5.2’s highest average output dollar value; Gemma 4 31B uses less than a quarter of its energy for more than half the value and about 2.5 times the NVJ and 2.8 times the TSJ.GLM 5.2 saves 1.64 hours of expert time using 0.430 MJ.Source: §1 · p. 3
  • B200 leads NVJ and TSJ across four tested single-accelerator Qwen3.6-27B deployments; eight H100s use 3.24 times the energy of one while reducing completion time by 22%.One and eight H100s produce nearly identical net dollar value.Source: §1 · p. 3
  • The framework targets unsuccessful attempts and computation yielding little extra value, with benchmark-task returns still needing validation beyond this task sample.The authors propose using NVJ and TSJ to guide model, accelerator, software, and task-routing improvements under real energy constraints.Source: §1 · p. 3

2 Methods

The methods value AI outputs against expert references, judge them, measure end-to-end energy and execution cost, and aggregate results across occupations and benchmark configurations.

2.1 Dataset and expert references

  • 220 public GDPval tasks span 44 occupations in nine U.S. economic sectors; expert-only and expert-with-AI references each pair a deliverable with recorded completion time.Tasks include professional requests, source files, grading rubrics, and target deliverables.Source: §2.1 · pp. 3–4

2.2 Blinded evaluation

  • A blinded pairwise LLM judge compares the AI deliverable with an identity-masked expert reference; a win or tie earns full output value, while a loss earns zero.Figure 2 illustrates the comparison and accounting for an Accountants and Auditors task.Source: §2.2 · p. 4, Figure 2 · p. 5

2.3 End-to-end energy accounting

  • End-to-end energy covers inference, tool use, and waits; for concurrent batches, active-request attribution allocates energy by active running time, and losses and timeouts still incur energy.The batch policy represents concurrent serving rather than serial execution of isolated requests.Source: §2.3 · p. 5

2.4 Net Value per Joule and Time Saving per Joule

  • NVJ and TSJ report net dollar value and time saving per energy, while every run incurs execution cost and a loss contributes negative net dollar value.Execution cost is estimated from recorded resource allocations and published GPU, CPU, and RAM prices.Source: §2.4 · p. 6

2.5 Two levels of GDP-based aggregation

  • GDP-based weights combine sector GDP and within-sector employment; occupation weights are renormalized within categories, while benchmark outcomes use category weights after component aggregation.Wages are excluded from the weights because occupation median wage already enters output dollar value.Source: §2.5 · p. 6

3 Experiments and Results

The experiments separate differences in the work from differences in the AI system by varying occupations and expert references, comparing models on the same tasks and protocol, and varying accelerator type and count for one model.

3.1 How does economic return per joule vary across occupations? (Q1)

  • Computer & Mathematical delivers about 27% higher NVJ than Architecture & Engineering, although Engineering delivers about 10% more net dollar value per task.Personal Care & Service leads in win–tie rate, while Protective Service leads in win–tie rate per MJ; Production falls from second in time saving per task to sixth in TSJ.Source: §3.1 · p. 7, Figure 3 · p. 7

3.2 How do models trade off economic value and energy? (Q2)

  • Gemma 4 31B produces 44% less output dollar value than GLM 5.2 but uses 77% less energy, yielding about 2.5 times its NVJ and the highest TSJ among the five models.GLM 5.2 remains on the higher-benefit end of the point-estimate efficient frontiers, while Gemma 4 31B anchors the low-energy end.Source: §3.2 · pp. 8–9, Figure 5 · p. 9

3.2.1 Net value per dollar of AI cost and per AI hour

  • GPT-OSS 120B uses 68% less energy than GLM 5.2 despite similar execution cost and completion time, and delivers more per megajoule despite lower net value.GLM 5.2 instead delivers more net value per dollar of AI cost and per AI hour; Table 1 compares the three ratios using the same net-dollar-value numerator.Source: §3.2.1 · p. 10, Table 1 · p. 10

3.3 How do accelerator type and count affect return per joule? (Q3)

  • B200 leads NVJ and TSJ among tested single-accelerator deployments; eight H100s cut completion time 22% but use 3.24 times the energy for nearly unchanged output value.One H100 yields higher NVJ and TSJ, while eight yield more net value per AI hour; communication, synchronization, sequential tool calls, and aggregate GPU power limit the end-to-end gain.Source: §3.3 · pp. 10–11, Figure 6 · p. 11

4 Limitations and Future Work

The evaluation’s economic values and weights are proxies, and its energy and cost boundaries omit some deployment factors. The authors propose broader validation and testing optimization uses for NVJ and TSJ.

Benchmark scope and evaluation

  • Output dollar value uses expert completion time and occupation median wage rather than measured GDP; future work could validate this proxy with independent expert valuations and broader task samples.Source: §4 · p. 11

System coverage

  • The accelerator sweeps vary hardware type and count, while future comparisons could vary model, harness, backend, precision, batching, and retries to isolate effects and test portability beyond OpenHands.Source: §4 · p. 11

Measurement and deployment

  • Energy covers metered GPU, CPU, and DRAM, while execution cost uses published rates; deployment studies could include facility overhead and expert review, coordination, liability, and adoption costs.Source: §4 · p. 11

Toward optimization

  • Future work could test NVJ and TSJ as training signals and for routing and stopping, with computation spent only when expected value or time saving warrants the energy, subject to quality and reliability.Source: §4 · p. 11

5 Conclusion

Joules-to-GDP connects judged professional work with the energy used to produce it, making computation allocation explicit under limited-energy deployment. In the Qwen3.6-27B comparison, eight H100s deliver similar value but lower return per joule than one.

5 Conclusion

  • The evaluated tasks and systems illustrate a framework rather than a comprehensive survey, making computation allocation an explicit choice across tasks, models, and accelerators when deployment energy is limited.Comparing total benefit with return per joule can guide routing, stopping, and serving decisions toward higher economic value and greater expert time savings within that budget.Source: §5 · p. 11
  • Eight H100s deliver similar value but lower return per joule than one for Qwen3.6-27B, using the expert-only reference and GDP weights.Whiskers show 95% task-composition intervals.Source: Figure 7 · p. 12

Reproducibility Statement

The reproducibility materials are intended to support regeneration of reported results from recorded inputs, while the paper situates its work among occupational, benchmark, and energy-efficiency research. Its expert comparisons use supplied reference conditions with stated limits.

Reproducibility Statement

  • The planned release includes task-level annotations, blinded judgments, completion times, execution costs, energy measurements, harness and analysis code, seeds, and versioned source inventories.The task data and five model weights are public; end-to-end reruns require compatible hardware and serving software and may produce different outputs or energy measurements.Source: Reproducibility Statement · p. 12

A Related Work

  • Related work covers professional-output and occupational benchmarks, field studies of worker outcomes, cost and energy benchmarks, and energy-normalized measures of economic value and time saving.The paper frames NVJ and TSJ as extending resource accounting beyond benchmark scores or win–tie rates alone.Source: A Related Work · p. 17

B Expert References and Annotations

  • The expert-only and expert-with-AI references are distinct supplied conditions, and pairwise judged quality does not establish absolute task quality or realized economic savings.The expert-with-AI condition is workflow-specific, not representative of every form of assisted professional work; the expert-only reference remains primary.Source: B Expert References and Annotations · p. 18

C Benchmark Contamination

The evaluation documents its contamination checks, judging and energy-measurement procedures, and cost boundaries. Sensitivity analyses show that energy attribution can change category rankings, while the tested alternative node tariffs leave category NVJ order unchanged.

C Benchmark Contamination

  • The contamination checks found no direct copying in evaluated outputs, but do not establish every model’s training-data contents.GPT-OSS 120B and 20B preceded public GDPval; the other three model cards do not mention GDPval.Source: C Benchmark Contamination · p. 18

D Judge and Evidence Pipeline

  • The judge receives task specifications, rubrics, source evidence, and identity-masked candidates through a pipeline combining sanitized semantic text and rendered visual evidence.All verdict records used in the analysis parsed without error.Source: D Judge and Evidence Pipeline · p. 18

D.1 Judge consistency

  • The three judges agree on 72–77% of exact outcomes; for GLM 5.2, Gemini’s win–tie versus loss decision matched across two outputs on 171 of 220 tasks (77.7%).In blinded expert review, experts agreed with Gemini on 23 of 32 decisions (71.9%).Source: §D.1 · p. 18, Table 2 · p. 19

E Execution and End-to-End Energy Accounting

  • End-to-end energy is measured across concurrent batches from immediately before launch until the final run completes, using counter differences or integrated sampled power and summing assigned accelerators.The metered components are GPU, CPU, and DRAM; deployments use vLLM with 20 concurrent requests and stated context and output limits.Source: E Execution and End-to-End Energy… · p. 19, Table 3 · p. 19

F Energy-Attribution Sensitivity

  • At P20, P50, P80, and P90 request-completion milestones, estimated pooled batch energy is 37.9%, 52.4%, 71.1%, and 83.9%; intermediate estimates assume constant power.Only full-batch energy is directly measured, and the figure does not test stopping because unfinished-request outcomes are unobserved.Source: F Energy-Attribution Sensitivity · p. 20, Figure 8 · p. 20
  • Protective Service remains first in win–tie rate per MJ and Computer & Mathematical first in NVJ under both energy-allocation rules, but the TSJ leader changes and eight NVJ and nine TSJ category ranks change.Both rules conserve measured batch totals but neither establishes the energy drawn by an individual concurrent request.Source: F Energy-Attribution Sensitivity · p. 20, Table 4 · p. 21

G Calculating AI-System Execution Cost

  • Execution cost uses published resource rates, while the calculation excludes unobserved external API charges, durable storage, expert review, development, coordination, and adoption costs.These factors belong in a broader deployment assessment alongside measured energy and delivered-work value.Source: G Calculating AI-System Execution Cost · p. 21

G.1 Execution-cost sensitivity

  • At three published node tariffs, the category NVJ order is unchanged for the same recorded GLM 5.2 tasks; the alternative-provider columns are hypothetical calculations, not provider runs.Source: Table 5 · p. 22, §G.1 · p. 22

H Economic Weights and Wage Robustness

The benchmark weights occupations by sector GDP share and within-sector employment, while wages enter output-dollar valuation rather than the weights. Because GDP shares, employment, wages, work practices, and time-saving estimates change over time, results should be recomputed as work changes.

Economic weights

  • Sector GDP shares set weights across sectors, and employment shares allocate each sector’s weight among its evaluated occupations.The 44 occupations are grouped into 13 categories using the O*NET-SOC taxonomy, which supplies classification rather than weights.Source: H Economic Weights and Wage Robustness · p. 22
  • Wages enter output dollar value through expert completion time multiplied by matched median hourly wages, but are not used again in GDP-based weights.Reusing wages in weights would give high-wage occupations additional influence; wage-bill weights would emphasize compensation paid to those occupations.Source: H Economic Weights and Wage Robustness · p. 22, Figure 9 · p. 23

Wage robustness

  • Mean rather than median wages raise NVJ while leaving TSJ unchanged when GDP-based occupation weights and other inputs are held fixed.The headline analysis uses median hourly wages, and the paper notes that economic inputs and work practices change over time.Source: H Economic Weights and Wage Robustness · p. 23

I Occupation-Category Results

Occupation rankings depend on which return metric is used: win–tie frequency per joule, dollar value per win or tie, and time saving per win or tie play different roles. The reported category comparisons describe evaluated tasks, not a population ordering.

I Occupation-Category Results

  • Architecture & Engineering leads net dollar value and time saving per task, while category results are reported with 95% task-composition intervals.Table 11 reports output dollar value and time saving per win or tie, components that help explain differing returns per joule.Source: I Occupation-Category Results · pp. 23–24, Table 9 · p. 25

J Why Occupation Rankings Differ

J.1 Full Occupation-Category Results

The full category results form economic-return ratios from weighted outcome and resource components, while sensitivity analyses vary task composition and the expert reference. These analyses qualify how category rankings and reported returns depend on the evaluated tasks and reference choices.

J.1 Full Occupation-Category Results

  • GDP-weighting underlying components before forming ratios reproduces GLM 5.2’s 37.69 net value per AI dollar and 191.3 USD per AI hour.Tables report category-normalized components and occupation-level point estimates; all attempts contribute cost and energy.Source: §J.1 · p. 26, Table 13 · p. 27, Table 14 · p. 27

J.2 Task-composition sensitivity

  • Computer & Mathematical retains the NVJ lead in 76.5% of one-task-deletion portfolios, while Architecture & Engineering retains the TSJ lead in 40.4%.These frequencies describe the specified task removals, not the probability that a category would lead across all professional work.Source: §J.2 · pp. 28–29, Table 16 · p. 29

K.1 Separating reference judgments from time valuation

  • Changing both judgment and time references reduces NVJ by 70.2% and TSJ by 65.0% when substituting the expert-with-AI reference.Averaging the two switching orders attributes 56.7% of the NVJ decline to reference judgments and 43.3% to time valuation.Source: Table 18 · p. 30, §K.1 · p. 30

K.2 Repeated runs: Pass@k and four-run win–tie thresholds

Repeated attempts and stricter win–tie thresholds change measured returns, while model and accelerator comparisons show that energy profiles can shift return-per-joule rankings. In these analyses, all scheduled runs incur energy and cost, and four-run thresholds hold total energy fixed.

K.2 Repeated runs: Pass@k and four-run win–tie thresholds

  • Additional runs improve win–tie coverage but reduce NVJ and TSJ because the added wins or ties do not offset the energy of every run.The pass@k comparison charges all scheduled attempts, including runs after an earlier win or tie; it does not evaluate early stopping.Source: §K.2 · p. 30, Figure 10 · p. 31, Table 19 · p. 32
  • At the ≥1/4 threshold, 0.80 hours (65.0%) of the 1.24 GDP-weighted hours saved per task comes from the expert-with-AI-to-AI-system transition.The remaining 0.43 hours (35.0%) is the difference between the two expert completion times.Source: §K.2 · pp. 31, 33, Figure 11 · p. 32, Table 22 · p. 33
  • Against the expert-with-AI reference, stricter four-run thresholds make NVJ negative at the two strictest thresholds.All four runs’ energy is retained while the required number of wins or ties changes.Source: §K.2 · pp. 30, 32, 33, Table 21 · p. 33

L Five-Model Comparison

M Accelerator Comparisons

  • Among four single accelerators, B200 has the highest NVJ and TSJ; under the same tasks, expert-only reference, pass@1 rule, and GDP weights, one versus eight H100s use 0.159 versus 0.516 MJ.One and eight H100s produce output dollar values of $64.79 and $67.81, respectively, with NVJ of 402.16 versus 123.39 USD/MJ and TSJ of 7.243 versus 2.432 hours/MJ.Source: Table 25 · p. 35, M Accelerator Comparisons · p. 35

N Additional Joules-to-GDP Tasks

The Joules-to-GDP collection adds 83 Snorkel AI tasks with detailed materials and annotations, extending occupation coverage and supporting evaluation of longer, source-intensive workflows. The paper’s reported results still use the 220-task GDPval split; the collection is not GDP-weighted.

N Additional Joules-to-GDP Tasks

  • 83 additional tasks include requests, grading rubrics, source files, expert-only and expert-with-AI deliverables, and time annotations, alongside the 220 public GDPval tasks.The materials are prepared for release by the Scaling Intelligence Lab and HazyResearch; the additional tasks are in the joules_to_gdp split.Source: N Additional Joules-to-GDP Tasks · p. 35
  • The extension adds 36 O*NET occupations absent from GDPval, bringing coverage to 80 occupation codes, and targets recent, source-intensive workflows.Median expert-only time estimates are 4.0 hours versus 2.34 hours for GDPval, and the median number of reference files is six versus one.Source: N Additional Joules-to-GDP Tasks · p. 35
  • The collection supports future tests of generalization to more occupations and longer workflows, but is not GDP-weighted and the paper’s results use the 220-task GDPval split.The extension includes 2026-dated source materials; its time estimates characterize expected expert effort.Source: N Additional Joules-to-GDP Tasks · p. 35
Loading 2610.30858v1…