PaperlayerView exact revision on arXiv

Source-linked AI summary

Green Code or Costly Code? Evaluating the Energy Costs of Iterative LLM-Based Code Optimizations

Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer

arXiv:2610.32891v1cs.SEcs.AI

TL;DR

The paper tests whether LLM-optimized code is more energy-efficient than human references and whether execution savings offset optimization energy. Across iterative optimization experiments, 42 of 44 configurations use more execution energy per problem than the canonical reference.

  • Problem

    It is unclear whether execution-time savings from LLM code optimization offset generation energy or make the resulting code more energy-efficient than human-written solutions.

    Source: §I · pp. 1–2

  • Method

    The study measures iterative self-optimization across 11 models, four decoding configurations, and 1,000 EffiBench problems.

    Source: • RQ2. How does the energy cost of… · p. 2, §VII · p. 11

  • Results

    42 of 44 configurations consume 4% to 886% more execution energy per problem than the canonical human reference.

    Source: • RQ2. How does the energy cost of… · p. 2

  • Takeaways & Limitations

    Mixture-of-Experts models can use substantially less inference energy than equivalently sized dense models without sacrificing correctness, while output-token limits risk correctness collapse.

    Source: §VII · p. 11

    EffiBench’s short, sub-second problems are unrepresentative of long-running workloads where break-even may more plausibly favor optimization.

    Source: §VI · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used to optimize code for performance. This practice often assumes that improvements in execution speed naturally translate into overall lower energy consumption. We challenge this assumption empirically by measuring whether performance-optimized code produced by LLM self-refinement pipelines is also energy-efficient relative to canonical human reference solutions, and whether the energy cost of the optimization process can be offset by subsequent execution-time savings. We evaluate 11 open-source models across four decoding configurations on all 1,000 problems of EffiBench, a state-of-the-art performance benchmark, while collecting per-process, hardware-level energy measurements throughout. Inference energy spans 0.79 to 8.68 MJ across model-variant pairs, which is roughly equivalent to the energy of 250 to 2800 GPT-4.1 queries. Limiting the maximum number of output tokens has the largest impact, reducing energy use by up to 69% compared to the default configuration. On the execution side, 42 of 44 configurations produce code that consumes more energy per problem than the canonical human reference, with overheads ranging from +4% to +886%. The two exceptions achieve only marginal gains, with break-even points on the order of 10^6 to 10^7 executions. Mixture-of-Experts architectures consume substantially less inference energy than equivalently-sized dense models without sacrificing correctness, making architectural choice a more consequential sustainability lever than decoding parameters. Performance optimization and energy efficiency are not always aligned goals: LLM self-refinement consistently produces faster-seeking but energy-heavier code relative to what a human engineer would write. All artifacts are available at https://doi.org/10.5281/zenodo.23598170.

I. INTRODUCTION

The paper asks whether LLM-driven performance optimization is also energy-efficient and whether execution savings can repay generation energy. It evaluates an iterative self-refinement pipeline against human reference solutions and finds that optimized code usually uses more execution energy.

I. INTRODUCTION

  • The study addresses whether generation energy is offset by execution savings and whether LLM-optimized code is more energy-efficient than human-written code.The motivation distinguishes inference energy from the execution energy of the generated code.Source: §I · pp. 1–2
  • The study uses EffiLearner to generate optimized solutions and compares them with canonical human-written references across 1,000 EffiBench problems.The evaluation is limited to open-source models deployed on-premise, enabling hardware-level inference-energy measurement.Source: §I · p. 1

• RQ2. How does the energy cost of generating optimized code vary across open-source LLMs and decoding configurations?

  • Inference energy ranges from 0.79 to 8.68 MJ across model–configuration pairs, while limiting maximum output length reduces energy by 13–69% relative to default.The study reports these findings across its evaluated models and decoding configurations.Source: • RQ2. How does the energy cost of… · p. 2
  • 42 of 44 configurations consume 4% to 886% more execution energy per problem than the canonical human reference.Only two configurations outperform the reference, and their break-even points require approximately 3.4 × 10^6 and 6.4 × 10^7 executions.Source: • RQ2. How does the energy cost of… · p. 2

II. RELATED WORK

Prior work separately examines model inference energy, generated-code efficiency, and optimization break-even costs. The paper situates itself at their intersection, including controlled evaluation of decoding parameters for code optimization.

II. RELATED WORK

  • The related-work landscape separates general NLP inference energy, generated-code runtime efficiency against human references, and break-even analysis against models’ own unoptimized outputs.The paper frames its study as joining these previously separate lines of work.Source: §II · p. 2, A. Inference-Pipeline Energy · p. 2

A. Inference-Pipeline Energy

  • Prior work measures general inference energy and runtime power feedback, but does not connect inference and execution energy through break-even analysis on a standard benchmark.The cited iterative power-feedback work targets bespoke HPC kernels rather than EffiBench.Source: A. Inference-Pipeline Energy · p. 2

B. Effect of Model Configuration on Inference Energy

  • Earlier output-token studies report correlations with energy, but do not isolate decoding parameters through controlled ablations; the cited framework comparison studies neither code generation nor optimization.The controlled configuration gap also includes temperature and context-window size.Source: B. Effect of Model Configuration on… · pp. 2–3

C. Runtime Energy of Generated Code and the Break-Even Point

Prior studies find persistent efficiency gaps between LLM-generated and human-written code, while earlier break-even analyses use models’ own unoptimized outputs as the baseline. This paper targets the remaining comparison against canonical human solutions.

C. Runtime Energy of Generated Code and the Break-Even Point

  • Across prior benchmarks and studies, generated code generally remains less efficient than human solutions, with reported gaps in runtime, memory, and energy.The cited work also finds that LLMs outperform human solutions in only a small share of LeetCode responses and that the gap persists across platforms.Source: C. Runtime Energy of Generated Code… · p. 3
  • EffiLearner feeds execution-overhead profiles back into iterative refinement, but did not measure inference energy or whether the investment is recovered at execution.Earlier BEP analysis compared optimized code with each model’s own unoptimized output rather than a canonical human reference.Source: C. Runtime Energy of Generated Code… · p. 3

D. Gap Analysis

  • No prior study jointly measures inference energy across controlled decoding settings, evaluates output against canonical human solutions, and applies break-even analysis within multi-epoch refinement.The paper identifies this combination on a standardized benchmark as its research gap.Source: D. Gap Analysis · p. 3

III. METHODS

  • The methods section proceeds from dataset and refinement framework to models, hardware, instrumentation, and the use of the setup to answer each research question.This is the section’s stated organization.Source: §III · p. 3

A. Dataset

The study uses EffiBench as its human-reference dataset, applies iterative profiling-guided refinement, and evaluates configurations that vary practitioner-adjustable decoding and memory levers.

A. Dataset

  • EffiBench contains 1,000 problems with canonical solutions drawn from top-starred submissions, providing a strong human reference for the energy comparison.The benchmark’s algorithmically challenging problems make execution-energy differences measurable; repairs addressed defects in 122 generated test suites, after which all canonical solutions passed.Source: A. Dataset · p. 4

B. Self-Refinement Framework

  • EffiLearner iteratively generates code, profiles execution time and memory, and returns those overhead profiles to the model for refinement.Profiling averages 60 measurements per script; revised candidates are checked against open tests, while final evaluation uses held-out private tests.Source: B. Self-Refinement Framework · p. 4

C. Models and Parameter Configurations

  • The study evaluates four configurations that change at most one parameter, including sampling temperature, context-window size, and maximum output length.The reduced-context setting caps the window at 64k tokens, and the reduced-output setting halves the maximum output length to 1,024 tokens; Table II lists the evaluated models.Source: C. Models and Parameter Configurations · pp. 4–5, Table II · p. 4

D. Experimental Setup

The experiments measure both the energy used to generate optimized code and the energy used to execute it, at different aggregation levels. Thread-level instrumentation and repeated runs are used to capture process energy and steady-state execution energy.

D. Experimental Setup

  • The server used two Intel Xeon Gold 6438Y+ processors, two NVIDIA L40 GPUs, 1 TB of memory, and locally served models; the full pipeline ran for about 30 days.Processes were pinned to physical cores, while several system settings remained at their operating-system defaults.Source: D. Experimental Setup · p. 5

E. Energy Measurement

  • Metrion attributes CPU, DRAM, and GPU energy to solution processes at thread level, accounting for frequency scaling, SMT contention, and hardware topology.It combines RAPL, nvidia-smi, and hardware performance counters, allocating sampled package energy to active processes using unhalted cycles.Source: E. Energy Measurement · p. 5

1) Measurement Protocol:

  • Each model-configuration pair completed six full runs over 1,000 problems, and reported generation-plus-execution energy is their mean.GPU energy was recorded continuously during inference, whose runs lasted hours to days.Source: 1) Measurement Protocol · p. 5
  • Each solution was executed 60 times in one persistent Python interpreter, with the mean reported as execution energy to reflect steady-state use.A 15-second pause and CPU-load and temperature admission criteria were used between executions.Source: 1) Measurement Protocol · p. 5

F. Approaches to the Research Questions

2) RQ2 – Cross-Model and Parameter Variation:

The analysis compares energy and optimization quality across decoding configurations and models, then assesses whether execution savings against canonical human solutions can recover generation energy. Break-even is reported only when optimized solutions use less execution energy on average.

2) RQ2 – Cross-Model and Parameter Variation:

3) RQ3 – Break-Even of Generation vs. Execution Savings:

  • RQ3 compares optimized solutions with EffiBench’s top-rated human-written canonical solutions, aggregating results by model and configuration rather than by task.The comparison characterizes the overall energy trade-off of deployment, not which individual problems benefit from refinement.Source: 3) RQ3 – Break-Even of Generation vs.… · p. 6
  • Break-even is the execution count at which cumulative savings offset aggregate generation energy; no break-even exists when mean optimized execution energy is not lower than the canonical baseline.In that case, optimization is a net energy loss in aggregate.Source: 3) RQ3 – Break-Even of Generation vs.… · p. 6

IV. RESULTS

Across the reported results, self-optimization reduces execution-time and memory metrics, while correctness and inference energy vary across models. The results also report lower inference energy for the MoE model than for a larger dense counterpart.

optimized code using LLMs?

  • After self-optimization, execution-time reductions reach 29.0%, with Nemotron-Nano showing the largest relative improvement.Qwen models, which begin below or near normalized execution time 1.0, improve by 1.5–1.7%.Source: optimized code using LLMs? · p. 6
  • After optimization, normalized maximum memory usage falls by up to 2.1%, while normalized total memory usage falls by up to 36.6% for Nemotron-Nano.Ministral-3B, Ministral-8B, and GLM-4.7-Flash reach normalized total memory usage below 1.0; the table defines values below 1.0 as beating the canonical reference.Source: optimized code using LLMs? · p. 6, Table III · p. 7
  • Qwen3.5-27B has the highest pass@1 at 81.0%, while GLM-4.7-Flash and GLM-4.6V-Flash have the lowest reported pass@1, at 25.02% and 25.42%.Source: optimized code using LLMs? · p. 6
  • Ministral-3B uses 1.27 MJ of inference energy, while Qwen3.5-27B uses 8.58 MJ; Qwen3.5-35B-A3B uses 3.12 MJ despite having more total parameters.The passage attributes the latter comparison to its MoE architecture, which activates only a subset of parameters per token; CPU accounts for 59–65% of Etotal across models.Source: optimized code using LLMs? · p. 6, Table IV · p. 7
  • Self-optimization reduces execution time, maximum memory, and total memory across all models; Nemotron shows the largest relative improvements, while Qwen achieves the highest correctness.Inference energy is strongly driven by model size, but MoE models use considerably less energy than equivalently sized dense models; GLM combines high inference cost with the lowest correctness.Source: optimized code using LLMs? · p. 7

B. RQ2: How does the energy cost of generating optimized

Output-token limits reduce inference energy, but decoding variants also affect execution quality. Qwen3.5-4B offers the strongest correctness-efficiency trade-off across variants, while higher generation costs can undermine larger models’ efficiency.

1) RQ2a: How do decoding parameters affect energy consumption across models?:

  • mt1024 has the lowest total inference energy across models, reducing the mean from 3.37 MJ to 2.26 MJ, or by up to 69% relative to default.Its reduction ranges from 6.08 MJ → 1.87 MJ (−69.3%) for GLM-4.7-Flash to 1.38 MJ → 1.33 MJ (−3.8%) for Qwen3.5-4B.Source: 1) RQ2a: How do decoding parameters… · p. 7, Figure 1 · p. 8
  • Default decoding yields NET values of 0.95–1.05 across models, while non-default variants span 0.93–2.51, with the upper extreme driven by Nemotron-Nano mt1024.NMU varies little by variant (1.002–1.088), whereas NTMU reaches 3.19 for Nemotron-Nano mt1024 and 1.52 for Ministral-8B ctx65536.Source: 1) RQ2a: How do decoding parameters… · p. 7

2) RQ2b: Which model-configuration pairs offer the most favorable trade-off between inference energy and optimization quality?:

  • Qwen3.5-4B leads across all four variants at 37–40 % MJ−1 while keeping NET below 1.0; Qwen3.5-27B reaches 80.98% pass@1 but only ∼9.4 % MJ−1.Qwen3.5-4B ctx65536 is the best single configuration at 40.15 % MJ−1; GLM-4.7-Flash and Nemotron-Cascade mt1024 are least efficient at 4.1 and 3.7 % MJ−1, respectively.Source: 2) RQ2b: Which model-configuration… · pp. 7–8, Figure 2 · p. 8

C. RQ3: Does the energy invested in generation translate

The study compares last-epoch code’s per-execution CPU energy with canonical human solutions and examines whether savings can repay generation energy. Most configurations consume more execution energy than the canonical reference, and the few break-even thresholds are very large.

a) Canonical solutions dominate at execution time

  • 42 of 44 configurations consume more execution energy per problem than the canonical reference; only Qwen3.5-9B t0.5 and Nemotron-Cascade mt1024 undercut it.The two exceptions save 0.65 J and 0.02 J per problem, respectively.Source: C. RQ3: Does the energy invested in… · p. 8
  • The mean deltas are 0.43 J for mt1024 and 0.44 J for t0.5, versus approximately 0.65 J for default and ctx65536.Even the most favourable configuration, Qwen3.5-9B t0.5, achieves only a marginal gain, driven by a single model-variant pair.Source: C. RQ3: Does the energy invested in… · p. 8

b) Optimization does not reliably improve upon its own base generation

  • Only 20 of 44 configurations (45.5%) reduce execution energy relative to their own base generation, while 24 (54.5%) regress.The largest regressions are Nemotron-Nano t0.5 (−0.54 J per problem) and Nemotron-Nano default (−0.44 J per problem).Source: C. RQ3: Does the energy invested in… · pp. 8–9

c) Break-even analysis

  • 42 of 44 configurations have no break-even point; Qwen3.5-9B t0.5 needs approximately 3.4×10^6 executions per problem and Nemotron-Cascade mt1024 approximately 6.4×10^7.A break-even point exists only when optimized code uses less execution energy than the canonical solution on average.Source: C. RQ3: Does the energy invested in… · p. 9

D. Carbon Footprint of the Optimization Loop

The paper converts measured energy to operational carbon emissions using a global-average grid intensity and reports generation and execution footprints. For the two configurations that save execution energy against canonical solutions, per-execution carbon savings are negligible relative to generation investment.

D. Carbon Footprint of the Optimization Loop

  • The conversion uses a global-average grid intensity of γ = 217 gCO2eq/kWh; region-specific estimates can be scaled by γlocal/217.The stated global value lies within the 23.92–625.57 kg/MWh range across 365 regional grids tracked by eco2AI.Source: D. Carbon Footprint of the… · p. 9
  • For optimizing 1,000 problems with the default configuration, total generation footprint ranges from 76.8 gCO2eq for Ministral-3B to 517.0 gCO2eq for Qwen3.5-27B.Table VI reports the five-epoch optimization-loop carbon footprint.Source: D. Carbon Footprint of the… · p. 9, Table VI · p. 10
  • Execution-side emissions are three orders of magnitude smaller than generation; Qwen3.5-9B t0.5 saves ∼39 µgCO2eq per execution against a generation investment of 133 gCO2eq.The resulting break-even point is approximately 3.4 × 10^6 executions, identical to the energy-denominated result because the conversion is linear in γ.Source: D. Carbon Footprint of the… · pp. 9–10

A. Generation energy as a hidden cost of optimization

LLM self-optimization usually increases execution energy relative to canonical human solutions, while model architecture matters more than decoding settings for inference energy. The study also identifies workload, baseline, pipeline, language, model-access, and lifecycle boundaries on these findings.

A. Generation energy as a hidden cost of optimization

  • 42 of 44 configurations consume more execution energy per problem than the canonical human reference; the two exceptions need millions of executions to break even.The authors conclude that performance optimization and energy efficiency are not aligned goals.Source: A. Generation energy as a hidden cost… · p. 10

B. Model and architecture choice matter more than decoding parameters

  • MoE Qwen3.5-35B-A3B achieves near-identical correctness to dense Qwen3.5-27B at less than half the inference energy, while output caps can cause correctness collapse.The authors identify model and architecture as more consequential than decoding configuration.Source: B. Model and architecture choice… · p. 10

C. Self-optimization does not reliably reduce execution energy across epochs

  • 54.5% of configurations regress in execution energy across the optimization trajectory, and the authors recommend selecting for execution energy alongside correctness.The pipeline replaces solutions without an execution-energy gate.Source: C. Self-optimization does not reliably… · p. 10

D. Practical implications

  • The authors recommend reserving self-optimization for high-frequency hot-path code and using smaller models when correctness allows.Qwen3.5-27B’s generation footprint is approximately seven times Ministral-3B’s for the same task.Source: D. Practical implications · p. 10

VI. LIMITATIONS AND FUTURE WORK

  • EffiBench’s short, sub-second problems do not represent long-running workloads, where break-even may be more favorable to optimization.The canonical solutions are also highly efficient competitive-programming submissions, not average-practitioner code; their authoring cost is not counted.Source: §VI · p. 11
  • The pipeline replaces solutions without an execution-energy gate, limiting attribution to model capability; all experiments also use Python, whose interpreted execution shapes the measured profiles.The authors note that the execution-energy gap may narrow in compiled languages such as C++ or Rust.Source: §VI · p. 11
  • The study excludes frontier models served through commercial APIs and measures operational energy only, omitting embodied hardware and other lifecycle costs.Hardware-level energy attribution motivates the open-source-only model selection.Source: §VI · p. 11

VII. CONCLUSION

The paper measures inference and execution energy for LLM-based code self-optimization across open-source models and EffiBench problems. Most optimized code consumes more execution energy than canonical solutions, while model architecture strongly shapes inference cost.

VII. CONCLUSION

  • 42 of 44 model-variant pairs consume more execution energy than the canonical reference, and the two exceptions require millions of executions to break even.More than half of configurations also regress in execution energy relative to their base epoch.Source: §VII · p. 11
  • MoE models can reduce inference energy relative to equivalently sized dense models without sacrificing correctness, while output-token limits can risk correctness collapse.The conclusion identifies model size and architecture as primary drivers of inference-energy variation.Source: §VII · p. 11
  • The authors recommend treating generation energy as a first-class cost, favoring small or MoE models when correctness allows, and targeting frequently executed hot-path code.They call for future work on compiled languages, long-running workloads, and energy-aware refinement pipelines.Source: §VII · p. 11

Related papers

Loading 2610.32891v1…