PaperlayerView exact revision on arXiv

Source-linked AI summary

EET: Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents

Yaoqi Guo, Ying Xiao, Jie M. Zhang, Mark Harman, Yiling Lou, Yang Liu, Zhenpeng Chen

arXiv:2601.05777v2cs.SE

TL;DR

SE agents can be costly because repeated iterations reuse growing conversation histories and may continue with little benefit. EET retrieves structured experience from prior resolutions to guide early termination during patch generation and selection. Across three agents on SWE-bench Verified, it reduces total cost by 19.3%–55.1% while keeping resolution-rate loss to at most 0.2%.

  • Problem

    Repeated agent iterations reuse conversation history, increasing computational cost, while iterations on difficult or unsolvable issues may add little benefit.

    Source: §1 · p. 1

  • Method

    EET stores structured experience from prior issue resolutions and retrieves relevant experience to guide early termination during patch generation and selection.

    Source: §1 · pp. 1–2

  • Results

    19.3%–55.1% lower total cost (31.8% on average) across three SE agents on SWE-bench Verified, with resolution-rate loss at most 0.2%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Removing experience injection substantially reduces resolution rate despite further reducing cost, indicating its role in EET’s cost-performance trade-off.

    Source: §5.3 · p. 8

    Evaluation is limited to SWE-bench Verified, and EET may fail to retrieve relevant experience for novel issues or domains with sparse historical data.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Software engineering (SE) agents powered by large language models are increasingly adopted in practice, yet they often incur substantial monetary cost. We introduce EET, an experience-driven early termination approach that reduces the cost of SE agents while preserving task performance. EET extracts structured experience from prior issue-resolution executions and leverages it to guide early termination during patch generation and selection, reducing unproductive iterations. We evaluate EET on the SWE-bench Verified benchmark across three representative SE agents. EET consistently reduces total cost by 19%-55% (32% on average), with negligible loss in resolution rate (at most 0.2%). These efficiency gains are achieved, on average, by identifying early-termination opportunities for 11% of issues and reducing API calls, input tokens, and output tokens by 21%, 30%, and 25%, respectively. We release the code, prompts, and data at https://github.com/IanWalls/EET.

1 Introduction

SE agents can incur substantial costs through repeated iterations, especially when they persist on difficult or unsolvable issues. EET uses structured experience to guide early termination, reducing costs while preserving task performance.

1 Introduction

  • Cost efficiency is a practical-adoption concern, with 53% of surveyed software engineers reporting agent cost as a barrier.The paper also connects monetary cost with energy consumption and environmental impact.Source: §1 · p. 1
  • Repeated tool invocations feed the full conversation history back as context, making computational costs grow super-linearly; iterations on difficult or unsolvable issues may add little benefit.The paper describes this growth as “token snowball.”Source: §1 · p. 1
  • EET captures structured experience from prior issue resolutions and retrieves relevant experience to guide early termination during patch generation and selection.The stored information covers issues, agent execution trajectories, and outcomes of prior attempts.Source: §1 · pp. 1–2
  • On SWE-bench Verified, EET reduces total cost by 19.3%–55.1% (31.8% on average) across Agentless, Mini-SWE-Agent, and Trae Agent, with resolution-rate loss at most 0.2%.On average, EET identifies early-termination opportunities for 11.3% of issues and reduces API calls, input tokens, and output tokens by 20.8%, 29.9%, and 25.1%, respectively.Source: §1 · p. 2

2 Related Work

Prior work addresses agent efficiency through prompt-level methods, architecture changes, and runtime constraints, while memory methods generally aim to improve task performance. EET instead uses structured experience to reduce agent cost.

2 Related Work

  • LLM-based SE agents use tools and iterative execution to explore codebases and generate patches; prior work reports substantial token consumption alongside strong performance.Examples discussed include Mini-SWE-Agent and Trae Agent.Source: §2 · p. 2
  • Efficiency methods include prompt compression and retrieval, architecture strategies such as plan caching and delegation, and runtime controls such as token budgets and turn limits.The paper notes that these approaches typically reduce cost while degrading task performance.Source: §2 · p. 2
  • Prior agent-memory work uses shared or hierarchical memory and retrieval to improve performance, whereas EET uses structured experience specifically to reduce cost.The paper notes that frequent retrieval and context expansion can add computational cost, citing MemoryBank as an example.Source: §2 · p. 2

3 Methodology

EET builds an experience base from historical issue resolutions and uses retrieved experience to make stopping decisions during patch generation and selection. It combines compact representations of successful resolutions with confidence-based checks at different execution stages.

3 Methodology

  • EET records historical trajectories and resolution results in an experience base, then retrieves relevant objects to guide tool use and stopping during patch generation and selection.At selection, the patch and its trajectory are evaluated against retrieved experience to guide selection and decide whether more patches are needed.Source: §3 · p. 2

3.1 Experience Generation

  • Experience objects summarize an issue, its resolution trajectory, and its outcome; EET stores successful, test-validated resolutions and adds confidence and rationale fields.It compresses trajectories into key steps and retrieves the top-1 experience above a similarity threshold using TF–IDF.Source: §3.1 · pp. 3–4

3.2 Early Termination

  • Relevant historical experience helps EET interpret intermediate execution signals to make dynamic stopping decisions in both patch generation and patch selection.Existing agents otherwise commonly stop only on an LLM completion signal or when an iteration budget is exhausted.Source: §3.2 · p. 4

3.2.1 Early Termination of Patch Generation

  • During patch generation, EET checks confidence after code-modification and test-execution milestones, stopping when confidence exceeds a threshold or the agent’s iteration budget is reached.The confidence estimate uses retrieved experience to assess whether current modifications suffice to resolve the issue.Source: §3.2.1 · p. 4

3.2.2 Early Termination of Patch Selection

  • During patch selection, EET scores each generated patch using the patch, its execution trajectory, and retrieved experience, then compares confidence with upper and lower thresholds.High confidence stops further patch generation; low confidence halts additional iterations as unlikely to resolve the issue; intermediate scores can lead to more patches.Source: §3.2.2 · pp. 4–5
  • Patch-generation checks avoid unnecessary cost within a single patch, while experience-driven selection checks prevent wasteful generation of additional patches.The paper presents these as complementary early-termination mechanisms.Source: §3.2.2 · p. 5

4 Evaluation Setup

The evaluation asks whether EET affects task performance and cost, how it compares with baselines, and how its components contribute. It uses SWE-bench Verified, three SE agents, and two LLM backends, measuring resolution and multiple cost dimensions.

4.1 Research Questions (RQs)

  • The evaluation addresses EET’s effects on performance and cost, its comparison with baselines, and the individual effects of experience injection and early termination.Source: §4.1 · p. 5

4.2 Datasets

  • SWE-bench Verified contains 500 real-world Python repository tasks with unit tests used to validate patches; 207 non-overlapping SWE-bench Lite tasks are used exclusively to generate experience.Agents receive issue descriptions and repository access, while tests are withheld for evaluation.Source: §4.2 · p. 5

4.3 SE Agents

  • The study uses Agentless, Mini-SWE-Agent, and Trae Agent with GPT-5-mini and DeepSeek-V3.2, applying EET to the phases each agent supports.EET applies to Agentless patch selection, Mini-SWE-Agent patch generation, and both Trae Agent phases.Source: §4.3 · p. 5

4.4 Baseline Methods

  • The baselines are turn-control, which limits turns with a one-time extension, and Naive-EET, which retrieves raw trajectories and patches instead of structured experience.The turn-control initial and extended limits follow the 25th and 50th percentiles of SWE-bench Lite turn counts.Source: §4.4 · pp. 5–6

4.5 Evaluation Metrics

  • Performance is measured by resolution rate, while efficiency is measured by API calls, input and output tokens, and total monetary cost.Source: §4.5 · p. 6

4.6 Implementation Details

  • Implementation uses 100 randomly selected SWE-bench issues excluded from Verified and Lite as validation data; hyperparameter-tuning procedures are provided in Appendix B.Source: §4.6 · p. 6

5 Results

Across the evaluated agents, EET reduces cost while largely preserving task performance. Comparisons with baselines and ablations show how its cost–performance balance relates to dynamic termination and experience guidance.

5.1 RQ1: Effectiveness of EET

  • 20.8% fewer API calls, 29.9% fewer input tokens, and 25.1% fewer output tokens on average.The respective reductions ranged from 7.9%–29.9%, 13.6%–51.8%, and 3.7%–51.0%.Source: §5.1 · p. 7
  • 11.3% of issues saw early termination on average, and EET reduced cost in both patch generation and patch selection.Early termination ranged from 8.6% to 14.0% across agents.Source: §5.1 · p. 7
  • With zero repository overlap, Trae Agent’s cost fell from $12.7 to $9.6 while its resolution rate remained 14.0% on 50 issues.Patches with confidence > 90 had pass rates of 63.6%–92.6%, compared with 8.7%–13.8% for confidence < 40.Source: §5.1 · p. 7
  • 19.3%–55.1% lower total cost (31.8% average) with resolution-rate loss of at most 0.2% across six configurations.For Agentless, resolution rates increased by 7.8% with GPT-5-mini and 7.2% with DeepSeek-V3.2; the 0.2% decrease for Trae Agent with DeepSeek-V3.2 was not statistically significant.Source: §5.1 · p. 7

5.2 RQ2: Comparison with Baselines

  • Turn-control cut total cost by 41.4% versus EET’s 31.8%, but reduced resolution rate by 10.7% while EET improved it by 2.7% on average.EET also outperformed Naive-EET in cost savings and performance preservation; its structured experience objects provide lower-overhead, more effective guidance than raw trajectories.Source: §5.2 · p. 8

5.3 RQ3: Ablation Study

  • Without experience injection, resolution rate fell by 10.4 percentage points, while total cost decreased by an additional 30.7%.The ablation was limited to Trae Agent with GPT-5-mini; the authors attribute the performance drop to losing historical guidance that helps distinguish unproductive from promising iterations.Source: §5.3 · p. 8
  • Disabling early termination increased resolution rate by 0.4 percentage points and total cost by 3.1%.API calls, input tokens, and output tokens increased by 9.2%, 7.3%, and 8.0%, respectively.Source: §5.3 · p. 8
  • The ablations show a trade-off: experience injection supports resolution, while early termination lowers cost with a slight resolution-rate decrease when disabled.Removing experience injection reduced resolution despite further cost savings; disabling early termination slightly improved resolution but increased cost.Source: §5.3 · p. 8

6 Conclusion

EET uses structured experience from prior executions to guide early termination during patch generation and selection. Across multiple agents and LLM backends, it reduces cost while maintaining resolution performance.

6 Conclusion

  • EET reduces SE-agent cost while preserving task performance through experience-guided early termination during patch generation and selection.Its approach targets redundant iterations and is described as integrable into diverse agents without fundamental redesigns.Source: §6 · p. 9
  • Across evaluated agent paradigms and LLM backends, EET reduces cost by 31.8% on average, with resolution-rate loss of at most 0.2%.The gains are associated with early-termination opportunities, fewer API calls, and lower token usage.Source: §6 · p. 9
  • EET applies structured experience from prior executions to reduce unnecessary computational cost while maintaining or improving the likelihood of successful issue resolution.The method is presented as a general optimization that requires no fundamental agent redesign.Source: §6 · p. 9

Limitations

The paper identifies limits in evaluation scope, historical-data availability, and domain coverage. Figure 2 provides an example experience object for one Django issue.

Limitations

  • Evaluation is limited to SWE-bench Verified, and EET may lack relevant experience for novel issues or domains with sparse historical data.The authors identify cold start as a limitation and plan to extend evaluation to additional benchmarks and, if feasible, industrial settings.Source: Limitations · p. 9
  • EET is evaluated on SE agents; extending it to non-programming domains such as general multi-step reasoning remains future work.The authors describe its design philosophy as domain-agnostic because it operates at context generation and budget-aware execution rather than task-specific logic.Source: Limitations · p. 9

A Experience Object Example

  • Figure 2 presents an experience object for the django__django-16910 issue, constructed from a Mini-SWE-Agent execution with GPT-5-mini.The supplied text identifies the example but does not describe the figure's individual fields.Source: A Experience Object Example · p. 10, Figure 2 · p. 11

B Hyper-Parameter Tuning of EET

EET uses tuned thresholds for experience retrieval and early termination during patch generation and selection. The reported choices balance cost reduction with resolution performance on validation data.

Tuning of τsim

  • τsim = 0.15 achieves both the highest resolution rate and lowest cost among the tested similarity thresholds and is adopted in subsequent evaluations.The threshold controls experience-retrieval quality; values from 0.1 to 0.3 were tested in steps of 0.05 on validation data using Mini-SWE-Agent with GPT-5-mini.Source: B Hyper-Parameter Tuning of EET · p. 11, Table 5 · p. 11

Tuning of τgen

  • τgen = 90 reduces cost while maintaining a resolution rate comparable to no early termination, so it is adopted for subsequent experiments.The tested settings ranged from 70 to 90 in increments of 5, plus a no-early-termination condition.Source: B Hyper-Parameter Tuning of EET · p. 12, Table 6 · p. 12

Tuning of τsel

  • The patch-selection thresholds are set to a lower boundary of 40 and an upper boundary of 90 to minimize cost without compromising performance.The lower threshold marks when an issue is considered too difficult to resolve; the upper threshold marks when a patch is assumed correct and needs no regeneration. The thresholds were tuned on Trae Agent with GPT-5-mini.Source: B Hyper-Parameter Tuning of EET · p. 12, Table 7 · p. 12

C Cost Reduction in Patch Generation and Selection

EET reduces iteration costs in patch generation and selection. The section reports fewer API calls per patch and fewer generated patches, then identifies Table 11 as a comparison of EET and baseline methods.

C Cost Reduction in Patch Generation and Selection

  • For Mini-SWE-Agent with DeepSeek V3.2, EET reduces API calls per patch from 61.7 to 58.9 for resolved issues and from 79.5 to 68.7 for unresolved issues.Except for Mini-SWE-Agent with GPT-5-mini, API calls per patch decrease across the other configurations; reductions are more pronounced for unresolved issues. Agentless is excluded because it follows a fixed workflow for individual patches.Source: C Cost Reduction in Patch Generation… · pp. 12–13, Table 9 · p. 13
  • With GPT-5-mini, Agentless generates 2.0 patches for resolved issues and 2.7 for unresolved issues under EET, down from four patches originally.EET adaptively reduces generated patches for resolved and unresolved issues across the configurations reported for agents that perform patch selection.Source: C Cost Reduction in Patch Generation… · p. 13

D Detailed Comparison with Baselines

  • Table 11 compares EET and two baseline methods on task performance and cost across different SE agents.The supplied passage states the table's comparison scope but gives no specific values or winning method.Source: D Detailed Comparison with Baselines · p. 13

E Case Studies

The case studies illustrate three ways retrieved experience contributes: faster bug localization, steering away from a symptomatic fix, and early termination after a correct patch is available.

Case 1

  • For django-11211, retrieved experience directs Mini-SWE-Agent to GenericForeignKey.get_prefetch_queryset, avoiding unrelated ORM exploration without changing the final fix.The resulting patch normalizes ID types before matching.Source: E Case Studies · p. 13

Case 2

  • For astropy-12907, retrieved experience steers Trae Agent from adjusting matrix indices toward the recursion bug underlying incorrect separability for nested CompoundModels.The patch unpacks sub-models to recover the separability matrix's diagonal-block structure.Source: E Case Studies · p. 13

Case 3

  • For xarray-7233, EET's confidence estimator scores Agentless's first patch highly enough to stop further patch generation.The case's savings come from avoiding downstream work rather than improving localization.Source: E Case Studies · p. 14

F Cross-Repository Transferability

The analyses test EET’s transfer beyond familiar repositories and examine how its benefits vary with experience-base size, issue characteristics, and retrieval settings. They also report the accounting boundary for experience-generation cost.

F Cross-Repository Transferability

  • With zero repository overlap, EET reduced average cost from $12.7 to $9.6 (24.4%) while resolution remained 14.0%.The authors interpret this as evidence that retrieved experience captures generalizable debugging patterns rather than relying on memorized project-specific cues.Source: F Cross-Repository Transferability · pp. 14–15, Table 12 · p. 14

G Scalability with Experience Base Size

  • As the experience base grew, cost reduction and successful experience injections increased monotonically, while resolution remained stable or slightly improved.The evaluation used Mini-SWE-Agent with GPT-5-mini on 50 issues, varying the base from 50 entries to the full set.Source: G Scalability with Experience Base Size · p. 15, Table 13 · p. 15

H Fine-Grained Analysis by Issue Difficulty

  • Simple issues achieved substantially higher resolution rates, early-termination rates, and cost savings than complex issues.The 500 SWE-bench Verified issues were divided into 194 simple issues requiring under 15 minutes and 306 complex issues requiring at least 15 minutes.Source: H Fine-Grained Analysis by Issue… · p. 15

I Fine-Grained Analysis by Issue Type

  • Bug-fix issues had higher resolution rates, higher early-termination rates, and larger cost savings than feature-request issues across agents and backbones.The analysis covered 443 bug-fix issues and 57 feature-request issues; the authors describe bug fixes as relatively simpler and more localized.Source: I Fine-Grained Analysis by Issue Type · p. 15

J Experience Generation Cost

  • Experience-generation cost is a one-time cost excluded from the per-task inference cost reported in RQ1 and RQ2.The reported generation cost depends primarily on trajectory verbosity, with autonomous agents such as Trae naturally incurring more than lightweight Agentless.Source: J Experience Generation Cost · p. 15

K Top-k Ablation on Retrieved Experiences

  • Injecting more than one experience entry gradually diminished EET’s effectiveness, so the authors use top-1 retrieval in the main experiments.They suggest multiple competing trajectories may introduce confusion, noise, and additional input-token cost; successful-outcome experiences were the most informative.Source: K Top-k Ablation on Retrieved… · p. 15

L Retrieval Method Comparison

The retrieval comparison finds nearly identical results for TF-IDF and embedding-based retrieval, supporting TF-IDF as EET’s lightweight default. The section also reports that raw LLM confidence tracks patch pass rates and identifies the artifact licenses and model-policy adherence.

L Retrieval Method Comparison

M LLM Confidence Calibration

  • Confidence scores above 90 corresponded to 63.6%–92.6% pass rates, while scores below 40 corresponded to 8.7%–13.8%.Across representative agent-model combinations, actual pass rates increased monotonically with raw LLM confidence.Source: M LLM Confidence Calibration · p. 17, Table 19 · p. 17

N License for Artifacts

  • The evaluated artifacts are distributed under the MIT License, and the experiments adhere to the usage policies for GPT-5-mini and DeepSeek-V3.2.Source: N License for Artifacts · p. 17

Related papers

Loading 2601.05777v2…