Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

• Prior work suggested multi-agent capability gains, but simultaneous evaluation across tasks, models, and hardware of performance and environmental cost was missing.Source: §1 · p. 2
• NA has the highest average accuracy on four of five tasks, while MA leads vulnerability detection; moving from NA to MA raises average energy use 6.36× and latency 6.07×.Source: §1 · p. 3
• Only one of 66 Pareto-optimal configurations is MA, while 36 (55%) are NA; energy and latency correlate at r=0.987 in log–log space.Source: §1 · p. 3
• Model and prompt effects vary by task, with prompting shifting log-parsing accuracy from 3.4% to 54.3%; hardware matters more for heavier configurations.Source: §1 · p. 3
• The study compares configurations across five software-engineering tasks, instruction-tuned open-weight models, heterogeneous hardware, and accuracy, latency, and energy measures.Source: §1 · pp. 3–4
• Its contributions include a multidimensional empirical comparison and design guidelines for sustainable, task-aware developer tools.Source: §1 · p. 4

2.1 Multi-Agent LLM Systems

• Surveys organize multi-agent systems around roles, memory, planning, and interaction protocols, while frameworks provide orchestration, communication, and tool integration [57] [37] [61].Source: §2.1 · p. 4

2.2 Multi-Agent LLM Systems in Software Engineering

• Software-engineering agents use specialization and iterative feedback for code generation and repair, but orchestration and coordination raise sustainability concerns that prior frameworks do not explicitly address.Source: §2.2 · pp. 5–6

2.3 Energy & Sustainability in LLMs

• Sustainability work examines inference energy, emissions, latency, and water use, while optimization approaches span quantization, routing, decoding, and hardware-aware methods.Source: §2.3 · p. 6

2.4 Gaps in the Literature

• The literature lacks broad comparisons across agentic configurations and tasks, joint evaluation of performance and sustainability metrics, and evidence about when simpler non-agentic designs suffice.Source: §2.4 · p. 7

3.1 Research Questions

• The research questions examine configuration accuracy, energy consumption, and inference latency across tasks and hardware, then synthesize their trade-offs.Source: §3.1 · pp. 7–8

3 Study Design

• The study spans 1,840 runs: 1,800 in the core factorial design and 40 additional GPT-OSS runs; four configurations increase from direct queries to a four-agent workflow.Source: Table 1 · p. 8, Figure 1 · p. 11

3.2.1 Code Generation.

• HumanEval provides 164 handwritten Python problems with unit tests for evaluating functional correctness in code generation.Source: §3.2.1 · p. 8

3.2.2 Technical Debt Detection.

• MLCQ code-smell reviews were filtered for ambiguity, minor severity, and class/function overlap, yielding a randomly shuffled 385-sample evaluation subset.Source: §3.2.2 · p. 9, Table 2 · p. 9

3.2.3 Code Vulnerability Detection.

• PrimeVul Pair contains 386 functions, with each vulnerable function paired with a benign counterpart sharing at least 80% of its string content.Source: §3.2.3 · pp. 9–10

3.2.4 Log Parsing and Analysis.

• HDFS-v1 supplies 385 log messages for parsing and 385 block-ID sessions for analysis; stratified sampling preserves the roughly 3% anomaly rate.Source: §3.2.4 · p. 10, Table 3 · p. 10

3.2 Tasks & Datasets

• Five datasets cover complementary software-lifecycle stages and vary in reasoning complexity, input/output structure, and evaluation criteria.Source: §3.2.4 · p. 10, Table 3 · p. 10

3.3 System Configurations

• The configurations are representative, literature-informed patterns rather than an exhaustive design space or task-optimal architecture.The progression keeps the experimental space tractable while isolating costs and benefits associated with increasing orchestration.Source: §3.3 · pp. 11–12
• The four configurations span a direct stateless LLM call, one autonomous agent, two collaborating agents, and a coordinated four-agent workflow.The study uses this progression to examine the marginal costs and benefits of added agent abstraction, adversarial feedback, and role-based reasoning.Source: §3.3 · pp. 11–12
• Multi-agent roles vary by task, using generic Actor-Critic patterns for structured tasks and specialized analytical roles for code and log tasks.Examples include a Programmer, Moderator, and Review Board for code generation, and a Security Researcher, Code Author, Moderator, and Review Board for vulnerability detection.Source: §3.3 · pp. 12–14

3.4 Implementation Details

• Experiments used Ollama with a 4096-token default context, temperature 0, and AG2 workflows; prompts used zero-shot or few-shot examples.Few-shot prompts included three manually selected examples that were not drawn from evaluation datasets.Source: §3.4 · p. 14

3.5 Models

• Five models from Gemma, Qwen, and DeepSeek-Coder families were evaluated, with gpt-oss:20b restricted to Server-SMU because of its memory requirements.The selection includes general-purpose and code-specialized models, ranging from compact 4B models to a 20B model.Source: §3.5 · pp. 14–15

3.6 Hardware

• The three platforms span two server-grade systems and one workstation, varying in CPU, memory, GPU type, and GPU memory.Server-SMU enabled runs that exceeded the model-size or GPU-memory capacity of the other platforms.Source: §3.6 · p. 15

3.7 Energy Measurement

• CodeCarbon estimated process-attributable CPU, GPU, and memory energy; emissions used Canada's lower-carbon grid as a conservative lower-bound estimate.Process tracking excludes idle and other non-utilized machine power, so reported energy is not full-system consumption.Source: §3.7 · p. 16

3.7.2 Reporting.

• Each configuration-task pair had three independent trials, with mean energy and standard deviation reported to reflect run-to-run variability.Execution duration was also recorded as the end-to-end inference latency metric.Source: §3.7.2 · p. 17, §3.7 · p. 17

3.8.1 Latency Measurement.

• End-to-end latency runs from prompt submission to final output and includes model initialization, agent orchestration, and inference.Phase timestamps support a breakdown of time spent in loading, reasoning, and tool use; results are averaged over three runs.Source: §3.8.1 · p. 17

3.8.2 Accuracy Measurement.

• Accuracy is an umbrella term for task performance, measured with task-appropriate metrics; code generation uses Pass@1 because one solution is requested.Pass@k estimates the probability that at least one of the top-k generated solutions passes all unit tests.Source: §3.8.2 · pp. 17–18
• Log parsing uses exact-match accuracy, edit similarity, and LCS similarity, while technical-debt classification also reports Macro F1 and per-class F1.For classification tasks, the study describes performance with accuracy, precision, recall, and F1-score based on confusion-matrix components.Source: §3.8.2 · p. 18

3.9.1 Independent Variables (Factors We Manipulate).

• The manipulated factors are agent configuration, software-engineering task, LLM model, hardware platform, and prompting strategy.The levels include NA, SA, DA, and MA workflows; five tasks; three hardware platforms; and zero-shot or few-shot prompting.Source: §3.9.1 · p. 19

3.9.2 Dependent Variables (Measured Outcomes).

• Measured outcomes are process-level energy consumption, end-to-end inference latency including coordination, and task-specific output accuracy.Accuracy may use Pass@1, parsing metrics, or classification metrics depending on the task.Source: §3.9.2 · p. 19

3.10 Experiment Execution

• Each trial follows the same initialization, execution, and termination sequence to reduce uncontrolled variability and cross-run interference.Figure 3 depicts the workflow’s initialization, execution, and measurement phases.Source: §3.10 · p. 19, Figure 3 · p. 20
• Models are freshly loaded for each run, then kept resident during that run to avoid both biased cold-start measurements and mid-run reloads.Ollama is started immediately before execution and terminated afterward; keep_alive is set to −1 during the run.Source: §3.10 · pp. 19–20
• Agent-level caching is disabled, runs are isolated, and configurations are repeated three times, except GPT-OSS, which is run once due to resource demands.The reported results are arithmetic means, with standard deviations calculated to assess stability.Source: §3.10 · p. 20

3.11 Data Analysis

• The five-task experiment records energy, processing time, task-specific accuracy, and configuration details across 1,840 runs.Each row corresponds to an experimental run; GPT-OSS configurations were run once rather than in triplicate.Source: §3.11 · pp. 20–21
• Cleaning and grouping the records by task, agent, model, hardware, and prompt produces 640 configuration-level summaries with means and standard deviations.Each summary includes up to three replicate runs.Source: §3.11 · p. 21
• Box plots summarize energy distributions, scatter plots examine energy–accuracy relationships, and Pareto curves show the efficiency–accuracy trade-off boundary.The box plots show medians, interquartile ranges, and whiskers extending to 1.5 times the IQR.Source: §3.11 · p. 21

3.12 Replication Package

• The replication package supplies code, workflows, prompts, evaluation-data preparation scripts, environment configurations, measurement and analysis scripts, and aggregated results.It provides acquisition instructions and preprocessing scripts instead of restricted artifacts, and notes that exact energy and latency reproduction may require comparable hardware.Source: §3.12 · p. 21

4.1.1 Effect of agentic configuration.

• NA leads code generation, technical debt detection, log parsing, and log analysis, while MA leads vulnerability detection; other tasks show small or low-scoring configuration differences.For code generation, Pass@1 ranges from 94.2% for SA to 96.8% for NA; technical debt Macro F1 ranges from 40.0% to 42.8%.Source: §4.1.1 · pp. 22–23
• Agent configuration significantly affected accuracy only for vulnerability detection (p=0.0162), while energy consumption differed significantly by configuration on every task.Accuracy differences were not significant for code generation, technical debt detection, log parsing, or log analysis.Source: §4.1.1 · p. 23

4.1.2 Effect of hardware platform.

• 2.5 percentage points was the largest hardware-level spread for code generation, compared with 2.0 for vulnerability detection and under one for each other task.The hardware-balanced comparison excludes GPT-OSS and shows hardware was not a major driver of RQ1 effectiveness.Source: Figure 5 · p. 24, §4.1.2 · p. 24

4.1.3 Effect of prompt strategy.

• Few-shot prompting raises log-parsing accuracy from 3.4% to 54.3%, while zero-shot leads code generation at 97.3% versus 92.7%.Zero-shot also performs best for technical debt detection and log analysis; few-shot is marginally better for vulnerability detection.Source: §4.1.3 · pp. 24–25, Table 12 · p. 26

4.1.4 Effect of LLM model.

• No model leads every task: G3-4B leads code generation (99.8% Pass@1), GPT-OSS technical debt (58.6% Macro F1), Q3-4B vulnerability (57.8% F1), and Q2.5C-7B both log tasks.The results also report that 4B models provide the strongest local-model accuracy for code generation and vulnerability detection, while Q2.5C-7B leads the log tasks.Source: §4.1.4 · pp. 25–26, Table 13 · p. 26
• In vulnerability detection, MA performs strongly for several models, whereas DA performs well with Q3-4B but is much weaker with several other models.The observed effect of agent configuration therefore varies by both model and task.Source: §4.1.4 · p. 26

4.1.5 Per-class analysis: Technical Debt Detection.

• Technical-debt mean F1 is 62.7% for Long Method and 8.1% for Feature Envy, the most difficult smell, despite competitive aggregate Macro F1.Best agent configuration varies by smell class, with differences small relative to standard deviations; model and prompt winners also vary by class, so the results do not support a general benefit from more agentic detection.Source: §4.1.5 · p. 27, Table 14 · p. 27

4.1.6 Metric sensitivity: Log Parsing.

• Few-shot prompting raises log-parsing accuracy from 3.4% to 54.3%, while NA remains strongest among agent configurations at 34.5% exact-match accuracy.Edit similarity ranges from 63.3% to 69.6% and LCS similarity is around 89–91%; partial-credit metrics therefore reveal structural similarity in many failed templates, but no hidden benefit from more agents.Source: §4.1.6 · p. 28, Table 15 · p. 28

4.2.1 Effect of agentic configuration.

• On average, SA, DA, and MA use 1.20×, 3.06×, and 6.36× NA’s energy, respectively; NA is lowest-energy in 11 of 15 task–hardware pairs.MA is never the lowest-energy configuration. Vulnerability detection is an exception to the limited general performance gains: MA has the highest average F1, while DA remains less accurate than NA and SA despite higher energy on most platforms.Source: §4.2.1 · pp. 28–29, Table 16 · p. 29, Figure 7 · p. 29

4.2.2 Effect of hardware platform.

• Hardware changes both energy levels and component shares: MA averages 1.210 kWh on Server-STF versus 0.588 kWh on Server-SMU, while CPU and RAM account for 52% and 42% on Server-STF and Workstation-SMU.Server-SMU energy is 97% GPU-attributed, showing that hardware effects are not a constant multiplier across platforms or agent configurations.Source: Table 16 · p. 29, §4.2.2 · p. 30, Figure 8 · p. 30

4.2.3 Effect of prompt strategy.

• Energy effects of prompting and model choice are task-specific: few-shot reduces energy for technical debt and log parsing, while code-generation energy rises and low-energy models differ by task.Few/Zero energy ratios for technical debt are 0.63–0.66, and few-shot uses 24–41% less energy for log parsing. CG-7B is lowest-energy in seven of 15 task–hardware pairs and Q3-4B in six; on Server-SMU, Q2.5C-7B uses 6.4× CG-7B’s energy (0.569 vs. 0.088 kWh).Source: §4.2.3 · p. 30, §4.2.4 · p. 31, Table 17 · p. 31

4.3 Inference Latency Comparison (RQ3)

• On average, SA, DA, and MA are 1.23×, 3.32×, and 6.07× slower than NA; log parsing on Workstation-SMU reaches a 160× slowdown for MA.For that log-parsing pair, MA takes 50 186 seconds versus 313 seconds for NA.Source: §4.3.1 · p. 33, Table 20 · p. 34, Figure 10 · p. 34

4.3.2 Effect of hardware platform.

• Hardware and prompt effects on latency vary: MA averages 6 220 s on Server-STF versus 21 002 s on Workstation-SMU, while few-shot speeds technical debt and log parsing but can slow other tasks.Few-shot reduces latency by 28–37% for technical debt and 23–41% for log parsing; it increases latency for code generation on some platforms and is consistently slower for log analysis.Source: §4.3.2 · p. 34, §4.3.3 · pp. 34–35, Table 21 · p. 35

4.3.4 Effect of LLM model.

• Model choice strongly affects latency: Q2.5C-7B is 136.5× slower than Q3-4B for log parsing on Workstation-SMU, while energy and latency correlate at r = 0.987 in log–log space.CG-7B and Q3-4B are among the fastest models and often among the lowest-energy; model-level latency differences can exceed hardware differences.Source: §4.3.4 · p. 35, §4.3.5 · pp. 35–36, Table 22 · p. 36

4.3.5 Energy–latency relationship.

• Log-transformed energy and latency are strongly correlated (Pearson r=0.987, p<10^-100), so longer-running configurations generally consume more energy, without establishing causality.Source: Figure 12 · p. 37

4.4 Trade-Off Analysis and Design Guidelines (RQ4).

• RQ4 compares accuracy and energy in its task-level projection, then selects balanced recommendations from a three-objective Pareto front that also includes latency.The balanced score uses min–max-normalized accuracy, energy, and latency with equal weight; ties are broken by raw accuracy.Source: §4.4 · p. 37, §4.4.1 · pp. 37–38, Figure 13 · p. 38

4.4.1 Pareto front analysis.

• Of 66 Pareto-optimal configurations, 36 (55%) use NA, 23 (35%) SA, 6 (9%) DA, and 1 (2%) MA, showing lightweight designs dominate the frontier.DA appears for log parsing and vulnerability detection, while the sole MA Pareto point is for log analysis.Source: §4.4.1 · p. 38
• NA is balanced-best in 11 of 15 task–hardware pairs (and 11 of 12 outside code generation), while SA wins code generation on all three platforms at 100% Pass@1.Recommendations balance normalized accuracy, energy, and latency across the three-dimensional Pareto front.Source: §4.4.1 · p. 38, Table 23 · p. 39
• For vulnerability detection, MA achieves the highest average F1-score (55.4%), improving 5.7 percentage points over NA but appearing on only one task–hardware Pareto front.For technical debt detection, NA wins across accuracy, energy, and latency on all hardware platforms.Source: Table 24 · p. 39, §4.4.2 · p. 39

4.4.2 Task-specific analysis.

• For log parsing, few-shot prompting raises accuracy from 3.4% to 54.3% while NA remains balanced-best; for log analysis, SA minimizes cost but NA has highest F1 (5.5–7.5%).The low absolute F1-scores make practical performance differences between log-analysis configurations small.Source: §4.4.2 · pp. 39–40
• The design guidance is to start with NA, adopt heavier workflows only for measured task-specific benefits, and match models and prompt examples to each task.MA is not supported as a default once energy and latency are considered alongside accuracy, despite its vulnerability-detection accuracy improvement.Source: §4.4.2 · p. 40

5.1 Insights across RQs

• NA has the highest average accuracy on four of five tasks, while MA leads vulnerability detection; moving from NA to MA raises average energy 6.36× and latency 6.07×.Only one of 66 Pareto-optimal configurations is multi-agent.Source: §5.1 · p. 40
• No model or prompt is a universal choice: their accuracy and energy effects vary by task, and Figure 14 shows differing task-level performance and energy leaders.Few-shot prompting shifts log-parsing accuracy from 3.4% to 54.3%.Source: §5.1 · p. 41, Figure 14 · p. 41
• Hardware differences affect accuracy by less than 3 percentage points, but MA runs 3.4× slower on Workstation-SMU than Server-STF, with the energy gap in the same direction.Hardware matters more for heavier DA/MA workloads than NA/SA workloads.Source: §5.1 · p. 41
• Energy and latency are tightly coupled (r = 0.987), while accuracy depends on task-appropriate model and prompt choices rather than energy use.For local inference on the same hardware, roughly constant power underlies the energy–latency relationship.Source: §5.1 · pp. 41–42

5.2 Design Guidelines for Sustainable Agentic LLM Systems

• Start with NA and add agents only when a measurable task-specific benefit justifies their cost; validate tool configurations against a non-agentic baseline.SA is a reasonable next step when a benefit is observed; DA/MA are reserved for cases such as vulnerability detection.Source: §5.2 · p. 42, §5.3 · p. 42
• Match the model to the task before tuning architecture: smaller models are not automatically more energy-efficient, and the best choice depends on task needs.For example, Q2.5C-7B can use more than two orders of magnitude more energy than the best model on log parsing.Source: §5.2 · p. 42
• Few-shot prompting can improve output-structure-sensitive tasks, but its effects vary: it helps log parsing and technical debt detection while hurting code generation.For log parsing, it raises accuracy from 3.4% to 54.3% and reduces energy by 24–41%; heavy workloads also make hardware choice more consequential.Source: §5.2 · p. 42

5.3 Implications

• Multi-task, multi-objective evaluation reveals trade-offs that accuracy-only comparisons can miss; sustainability reporting should pair reference-platform energy with transparent carbon-intensity assumptions.The 1,840-run campaign used approximately 651.7 kWh; its emissions estimate depends on the assumed carbon intensity, and high-cost configurations account for a disproportionate share.Source: §5.3 · pp. 42–43, §5.2 · p. 42

6.1 Internal Validity

• GPT-OSS was evaluated on one hardware platform without repeated runs, so its results warrant additional caution.Other configurations had three repeated runs, which capture some but not all long-term hardware and system variability.Source: §6.1 · p. 43

6.2 External Validity

• The five tasks cover only a subset of developer workflows and curated datasets, limiting generalization to noisier real-world data and other programming languages.The code-generation, technical-debt, and vulnerability findings may not generalize to languages with different syntax, conventions, or LLM training coverage.Source: §6.2 · pp. 43–44
• Evaluation covered only small local LLMs up to 20 billion parameters; results may differ for frontier models and cloud deployments.Public frontier services do not permit direct local energy measurement, and cloud deployments may differ in batching, virtualization, infrastructure, and carbon intensity.Source: §6.2 · p. 44
• Findings reflect the selected prompts and agent implementations, and more specialized or tuned multi-agent designs could yield different task-specific accuracy results.The authors characterize their configurations as realistic practitioner choices, not optimal designs.Source: §6.2 · p. 44

6.3 Construct Validity

• “Accuracy” denotes task-specific metrics that are not directly interchangeable, so cross-task averages summarize relative benchmark behavior rather than universal software-engineering capability.Metrics include Pass@1, Macro F1 and per-class F1, F1-score, and parsing accuracy.Source: §6.3 · p. 44
• Technical-debt classification metrics do not capture severity, maintainability impact, or disagreement among human reviewers.Macro F1 and per-class F1 mitigate the risk that plain accuracy obscures minority smell-class performance, but still reduce code quality to fixed-label classification.Source: §6.3 · p. 44
• CodeCarbon estimates process energy from CPU, GPU, and memory signals rather than measuring wall-socket energy, with uncertainty potentially depending on hardware idle power.Source: §6.3 · p. 44
• Reported accuracy reflects both model performance and output extraction, since format deviations and alternative normalization could change measured results.This issue occurred particularly in multi-agent configurations, where final outputs followed multiple intermediate reasoning steps.Source: §6.3 · pp. 44–45

7 Conclusion

• MA improves average vulnerability-detection accuracy, but NA or SA is sufficient and often strongest for the other tasks; MA consumes 6.36× as much energy and runs 6.07× as long as NA.Source: §7 · p. 45
• MA appears in only one of 66 Pareto-optimal configurations, and energy and latency are tightly coupled (r = 0.987 in log–log space).Source: §7 · p. 45
• Sustainable system design should start with a lightweight baseline and add agents only when a task-specific benefit is measured.The authors also recommend reporting accuracy alongside energy and latency to assess full cost–benefit trade-offs.Source: §7 · p. 45

A Technical Debt Per-Class Breakdowns

• Table 25 reports F1 by smell class and agent configuration, averaged over models, hardware, and prompts.Source: Table 25 · p. 50, A Technical Debt Per-Class Breakdowns · p. 50
• Table 26 breaks down smell-class F1 by LLM model, averaged over agent configurations, hardware, and prompts.Source: Table 26 · p. 50, A Technical Debt Per-Class Breakdowns · p. 50
• Table 27 breaks down smell-class F1 by prompt strategy, averaged over agent configurations, models, and hardware.Source: Table 27 · p. 50, A Technical Debt Per-Class Breakdowns · p. 50