PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints

Zhiyu Fan, Kirill Vasilevski, Dayi Lin, Boyuan Chen, Yihao Chen, Zhiqing Zhong, Jie M. Zhang, Pinjia He, Ahmed E. Hassan

arXiv:2509.09853v2cs.SEcs.AI

TL;DR

Existing SWE leaderboards largely emphasize solution accuracy rather than resource use, despite practical constraints on cost and time. SWE-Effi evaluates scaffold–LLM systems with metrics that relate issue resolution to resource consumption. Its analysis finds that effectiveness depends on scaffold–model synergy, with trade-offs across resource budgets and costly unresolved attempts.

  • Problem

    Existing AI-for-SWE leaderboards focus largely on resolve rate and do not capture whether systems use resources cost-effectively under practical constraints.

    Source: §1 · p. 1

  • Method

    SWE-Effi introduces holistic resource-effectiveness metrics and applies them to five scaffolds paired with three LLMs on a representative SWE-bench subset.

    Source: §1 · p. 2

  • Results

    Effectiveness depends on scaffold–LLM synergy, and the analysis identifies a trade-off between token-budget and time-budget effectiveness alongside expensive failures.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings highlight that resource effectiveness matters for project budgets and for the feasibility of large-scale reinforcement learning, where low latency is critical.

    Source: §6 · p. 9

    The exploratory evaluation covers 15 scaffold–LLM combinations and 50 of the initial 500 SWE-bench-Verified samples, rather than an exhaustive system assessment.

    Source: §6 · p. 9

Abstract

from arXiv · show

The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resolution and feature addition. Existing AI for software engineering leaderboards (e.g., SWE-bench) focus solely on solution accuracy, ignoring the crucial factor of effectiveness in a resource-constrained world. This is a universal problem that also exists beyond software engineering tasks: any AI system should be more than correct - it must also be cost-effective. To address this gap, we introduce SWE-Effi, a set of new metrics to re-evaluate AI systems in terms of holistic effectiveness scores. We define effectiveness as the balance between the accuracy of outcome (e.g., issue resolve rate) and the resources consumed (e.g., token and time). In this paper, we specifically focus on the software engineering scenario by re-ranking popular AI systems for issue resolution on a subset of the SWE-bench benchmark using our new multi-dimensional metrics. We found that AI system's effectiveness depends not just on the scaffold itself, but on how well it integrates with the base model, which is key to achieving strong performance in a resource-efficient manner. We also identified systematic challenges such as the "token snowball" effect and, more significantly, a pattern of "expensive failures". In these cases, agents consume excessive resources while stuck on unsolvable tasks - an issue that not only limits practical deployment but also drives up the cost of failed rollouts during RL training. Lastly, we observed a clear trade-off between effectiveness under the token budget and effectiveness under the time budget, which plays a crucial role in managing project budgets and enabling scalable reinforcement learning, where fast responses are essential.

1 Introduction

Existing SWE leaderboards emphasize resolve rate while largely ignoring the resource constraints that shape practical deployment. SWE-Effi adds resource-effectiveness metrics and uses them to reassess popular systems, finding that effectiveness depends on scaffold–model synergy and can be undermined by expensive failures.

1 Introduction

  • Resolve-rate-focused leaderboards leave resource use largely unmeasured, despite practical questions about time, compute cost, and whether systems are affordable to run regularly.The introduction frames cost-effectiveness as a concern extending beyond software engineering.Source: §1 · p. 1
  • High token consumption and slow rollouts can hinder accessible deployment and large-scale reinforcement-learning training, where repeated trials make scaffold costs consequential.The introduction describes agents using massive token budgets for marginal leaderboard gains and notes that latency per rollout can impede RL training.Source: §1 · p. 2
  • SWE-Effi introduces metrics for holistic effectiveness and re-evaluates systems on SWE-bench to complement accuracy-focused benchmarks with resource-constraint analysis.The study analyzes five scaffolds paired with three LLMs on a representative subset, and presents its findings as exploratory rather than definitive.Source: §1 · p. 2
  • Effectiveness depends on how a scaffold integrates with its base model: SWE-Agent’s EuTB is 21.8% with Qwen3-32B but 5.1% with GPT-4o-mini.For the same system, unresolved attempts consume on average over four times more resources than successful ones.Source: §1 · p. 2

2 Related Work

Prior SWE evaluations cover issue resolution and other software-engineering tasks, but the cited benchmarks generally emphasize correctness rather than system efficiency. Issue-resolution approaches include agentic tool-using systems and procedural pipelines, and scaffold performance is coupled to the underlying LLM.

2.1 AI System Evaluation on Software Engineering Tasks

  • SWE-bench evaluates LLMs on real GitHub issues, while SWE-bench-Verified improves benchmark quality through expert selection of instances.Other cited work extends SWE evaluation to additional languages, multimodal data, and new task sets.Source: §2.1 · pp. 2–3
  • Despite extensive SWE benchmarks, the cited evaluations do not systematically measure AI-system efficiency and often focus on solution correctness.The discussion spans GitHub issue resolution, test generation, code optimization, GUI testing, and vulnerability exploitation.Source: §2.1 · p. 3

2.2 LLM-based methods for GitHub Issue Resolution

  • Agentic methods let LLMs plan and use tools, whereas procedural methods invoke LLMs at scripted checkpoints in a fixed resolution pipeline.Examples include SWE-Agent and OpenHands among agentic scaffolds, and Agentless [9] and AutoCodeRover among procedural methods.Source: §2.2 · p. 3
  • There is no consensus on which issue-resolution approach is better, and scaffold performance and resource efficiency are coupled with the underlying LLM’s strengths and weaknesses.Source: §2.2 · p. 3

3 The SWE-Effi Resource Effectiveness Evaluation Metrics

SWE-Effi separates direct trial measurements from derived resource-effectiveness scores. It standardizes inference time to reduce the influence of external API conditions, then scores task resolution against resource consumption under defined budgets.

3 The SWE-Effi Resource Effectiveness Evaluation Metrics

  • SWE-Effi organizes evaluation into raw Core Performance Metrics and derived Resource Effectiveness Metrics for holistic assessment of SWE systems.Source: §3 · p. 3

3.1 Core Performance Metrics: The Foundational Measurements

  • Core measurements include resolve rate, local CPU time, LLM calls and token counts, and normalized inference time.These direct trial measurements provide the inputs for subsequent efficiency analysis.Source: §3.1 · pp. 3–4

3.1.1 Derivation of Normalized Inference Time

  • A regression model trained on 515,041 API-call logs, with R² of 0.79 on validation data, predicts normalized inference time from input and output token counts.The model uses GPT-4o-mini as its performance baseline because its API performance was considered stable and widely representative.Source: §3.1.1 · p. 4

3.2 Resource Effectiveness Metrics: Scoring Efficiency

  • The model’s normalized inference time provides a hardware-agnostic comparison by applying the regression to each call’s input and output token counts.Effectiveness scores use AUC of cumulative resolved tasks against resource consumption, normalized to 0–1; four scores cover tokens, dollar cost, CPU time, and inference time under specified caps.Source: §3.1.1 · p. 4, §3.2 · p. 4

4 Experiment Setup

The study compares five SWE scaffolds across three LLMs under controlled, sequential execution. Its benchmark is a stratified random subset of 50 SWE-bench-Verified issues.

4 Experiment Setup

  • The experiment evaluates five popular open-source scaffolds, including both procedure-based and agentic systems, using default settings except for SWE-Agent’s $1-per-issue API budget.Parallel processing was disabled to measure sequential scaffold time, and logging was augmented to profile local CPU and backend LLM inference costs.Source: §4 · p. 5
  • Three LLMs span proprietary and open-source options: GPT-4o-mini-2024-07-18, Llama-3.3-70B-Instruct, and Qwen3-32B.The selection was intended to cover different types and sizes relevant to varying resource conditions.Source: §4 · p. 5
  • All scaffold-model combinations were tested on 50 randomly selected SWE-bench-Verified issues, stratified to preserve the dataset’s project distribution.Source: §4 · p. 5

5 Evaluation of AI System SWE Capability and Efficiency

The evaluation compares resource and core-performance metrics averaged across resolved and unresolved attempts, then examines model–scaffold interactions and resource trade-offs. Its observations cover reasoning efficiency, growing prompts, and the higher costs of failures.

5.1 Observation 1: SWE Scaffold Performance Is Highly Model-Dependent

  • Base-model choice can outweigh scaffold design: SWE-Agent resolves 28% with Qwen3-32B but 10% with GPT-4o-mini at over 18× the input-token cost; AutoCodeRover remains competitive.AutoCodeRover with Qwen3-32B resolves 38% using 14.7 API calls and 55.5K input tokens; the authors describe effectiveness as dependent on scaffold–LLM synergy.Source: Table 1 · p. 6, Table 2 · p. 6, §5.1 · pp. 6–7
  • Agentless leads resolution at 48% with Qwen3-32B, while using 83.1 API calls, 209.4 seconds of inference, and 727.9 seconds of CPU time.Its patch generation and testing design trades off effectiveness for robustness.Source: §5.1 · p. 7

5.2 Observation 2: High-Quality Reasoning Minimizes Iterations and Saves Tokens

  • Although Qwen3-32B uses more tokens per call, AutoCodeRover earns higher effectiveness scores than with Llama-3.3-70B, with fewer calls contributing to lower total cost.The cited example is 15 API calls for Qwen3-32B versus 38 for Llama-3.3-70B.Source: §5.2 · p. 7

5.3 Observation 3: Token Snowball: Cascading Amplification of Invalid Context

  • The Token Snowball Effect describes roughly linear prompt growth as calls accumulate, with stale context adding tokens and potentially burdening reasoning.The showcase compares three scaffolds; the authors contrast long, high-token SWE-Agent trajectories with earlier termination by AutoCodeRover paired with Qwen3-32B.Source: §5.3 · p. 7, Figure 1 · p. 8

5.4 Observation 4: Failing is Far More Expensive Than Succeeding

  • Unresolved attempts tend to consume more resources; SWE-Agent with GPT-4o-mini uses over 8.8 million tokens and 658 seconds on failure, versus 1.8 million tokens and 167.2 seconds on success.The authors report more than a fourfold increase in both token cost and inference time for this failed attempt.Source: §5.4 · pp. 7–8, Table 3 · p. 8, Figure 2 · p. 9
  • When stuck, agents can enter repetitive, expensive loops until an external budget limit; the authors identify futility detection and progress-aware stopping or redirection as future research directions.Source: §5.4 · pp. 8–9

6 Discussion and Conclusion

The analysis finds that effectiveness depends on scaffold–model synergy, with a trade-off between token-budget and time-budget effectiveness and costly failure loops. Its findings are exploratory and SWE-bench-based, though the authors suggest the identified issues may extend to other complex, multi-step AI tasks.

6 Discussion and Conclusion

  • Effectiveness depends on scaffold–model synergy, while token-budget and time-budget effectiveness trade off and expensive failure loops hinder real-world use and RL training.The authors identify these as implications for project budgets and the feasibility of large-scale reinforcement learning.Source: §6 · p. 9
  • The evaluation is exploratory, limited to 15 scaffold–LLM combinations and 50 of 500 SWE-bench-Verified samples because of resource, time, and cost demands.The authors describe the work as an introduction and initial insights, not an exhaustive evaluation.Source: §6 · p. 9
  • Although experiments use SWE-bench, the authors suggest token snowballing and expensive failures may affect autonomous AI systems tackling complex, multi-step problems in other domains.They propose lightweight frameworks that avoid expensive failure modes as potentially more practical and evolvable foundations for reinforcement learning.Source: §6 · p. 9

7 Appendix

The appendix lists model prices used for cost-budget calculations and adds resource-efficiency analyses. These compare runtime composition and request counts across systems, and show resolve-rate curves against token and CPU-runtime consumption.

7.1 Costs used for Effectiveness under Cost Budget (EuCB) Calculations

  • GPT-4o-mini costs $0.15 per million input tokens and $0.6 per million output tokens; Llama-3.3-70B-Instruct costs $0.38 and $0.12, respectively; Qwen3-32B costs $0.10 and $0.30.The calculations use prices from the cheapest provider found on July 11, 2025.Source: §7.1 · p. 11

7.2 Additional Resource Efficiency Evaluation

  • AutoCodeRover, SWE-Agent, and Agentless-Mini spend most runtime on inference, while OpenHands and Agentless spend most on CPU tasks such as tests.Agentless-Mini’s inference-heavy runtime is attributed to its two-stage design; Agentless’s CPU-heavy runtime reflects numerous reproduction and regression tests.Source: Figure 3 · p. 12
  • GPT-4o-mini with SWE-Agent has a request-count spike attributed to lower token prices; Qwen3 generally uses similar or fewer requests, especially with OpenHands and AutoCodeRover.The passage notes that GPT-4o-mini and Llama-3.3-70B have similar request counts across scaffolds except with SWE-Agent.Source: Figure 4 · p. 13
  • Resolve-rate curves plot per-issue input-token and output-token usage for EuTB calculations, and per-issue CPU runtime for comparison.Figures 5 and 6 show token usage; Figure 7 shows CPU-bound processing time.Source: Figure 5 · p. 13, Figure 6 · p. 14, Figure 7 · p. 14

Related papers

Loading 2509.09853v2…