PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentless: Demystifying LLM-based Software Engineering Agents

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang

arXiv:2407.01489v2cs.SEcs.AIcs.CLcs.LG

TL;DR

Existing autonomous software agents use complex tool interaction and planning despite limitations in current LLMs, motivating whether such complexity is necessary. AGENTLESS tests this question with a simple localization, repair, and patch-validation process, achieving the highest performance among open-source techniques on SWE-bench Lite while minimizing cost.

  • Problem

    Current LLM-based software engineering agents use complex autonomous tool use and planning despite limitations in current LLM abilities, raising whether such complexity is necessary.

  • Method

    AGENTLESS uses a three-phase process of hierarchical localization, candidate patch repair, and patch validation without autonomous LLM tool usage or planning.

  • Results

    AGENTLESS achieves the highest performance among open-source techniques on SWE-bench Lite while minimizing cost.

  • Takeaways & Limitations

    SWE-bench Lite contains problematic issues, motivating SWE-bench Lite-S as a more rigorous benchmark after their removal.

  • Takeaways & Limitations

    AGENTLESS performance might not generalize beyond the SWE-bench Lite evaluation dataset.

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.

1 Introduction

AGENTLESS asks whether complex autonomous agents are necessary for repository-level software engineering and answers with a simple, controlled alternative. It combines hierarchical localization, patch generation, and validation, achieving strong benchmark performance while motivating a cleaner evaluation set.

  • Benchmark: SWE-bench Lite evaluates bug fixing on 300 real-world GitHub issues paired with Python repositories.Each task requires modifying the repository to fix a bug or introduce a feature, while Lite focuses on bug fixing.
  • Motivation: Agent-based approaches can suffer from complex tool design and uncontrolled decision planning because current LLM abilities differ from those of human developers.Incorrect tool use can reduce performance and waste queries, while large action spaces make autonomous decisions difficult to check.
  • Approach: AGENTLESS uses localization, repair, and patch validation without letting the LLM autonomously plan actions or operate complex tools.Localization is hierarchical, repair samples candidate patches, and validation uses generated reproduction tests plus regression tests.
  • Benchmark: SWE-bench Lite contains 4.3% exact-patch descriptions, 10.0% insufficient-information problems, and 5.0% misleading solutions.These categories motivated SWE-bench Lite-S, which excludes problematic issues for more rigorous evaluation.
  • Results: 32.00% and 96 correct fixes: AGENTLESS outperformed all open-source approaches on SWE-bench Lite.The approach also achieved comparably low cost.

2 Background and Related Work

Prior work largely develops autonomous agents and specialized techniques for localization, repair, and testing. AGENTLESS instead presents a simpler, interpretable, and cost-effective alternative that avoids autonomous tool use and planning.

  • Agent-based systems: Agent-based systems such as Devin, OpenDevin, SWE-agent, and Aider iteratively use repository tools, terminals, editors, or repository maps.These systems generally let LLM agents act, observe feedback, and plan subsequent steps.
  • Agentless approach: AGENTLESS uses defined localization, repair, and patch-validation stages instead of agent-controlled future actions or complex tools.The paper frames this simplicity as a competitive alternative to autonomous agent-based systems.
  • Research context: LLM-based software-engineering research spans fault localization, automated program repair, fuzzing, and unit-test generation.The paper situates AGENTLESS at the intersection of these established research directions.
  • Technical ingredients: AGENTLESS combines prompting-based and embedding-based localization with sampled patch generation and reproduction-test-based selection.Its design draws on established fault localization, automated repair, and LLM-based test-generation research.

Approach

AGENTLESS solves repository-level issues through a fixed three-phase pipeline: hierarchical localization, candidate patch repair, and validation-oriented selection. It compresses context and uses simple representations to make each stage more focused and economical.

  • Localization: AGENTLESS begins with repository structure and issue text, then ranks suspicious files using LLM prompting and embedding-based retrieval.Irrelevant folders are filtered before retrieval adds files with code snippets relevant to the issue.
  • Localization: The localization hierarchy narrows suspicious files to related classes, functions, variables, and finally fine-grained edit locations.Skeleton representations expose declarations, signatures, fields, methods, and selected comments without including full file contents.
  • Repair: Repair supplies code windows around localized edits and asks the LLM for Search/Replace diffs rather than complete rewritten snippets.The format contains original search text and replacement text, focusing generation on small edits.
  • Repair: AGENTLESS samples multiple candidate patches, beginning with a greedy patch and then using higher-temperature samples.Small diffs are intended to reduce generation cost and hallucination risk while increasing patch variety.

3.3 Patch Validation

AGENTLESS validates candidate patches with generated reproduction tests and existing regression tests, then normalizes and ranks surviving patches. This fixed selection process avoids delegating validation decisions to an autonomous agent.

  • Patch filtering: Candidate patches must pass existing regression tests, while reproduction tests and LLM assessment help identify relevant regression behavior.This conservative filtering removes patches that fail the regression suite even if reproduction behavior appears favorable.
  • Reproduction tests: AGENTLESS generates complete tests that reproduce the reported issue and print whether it is reproduced or resolved.A test is selected only when it reproduces the issue on the original repository.
  • Patch selection: AGENTLESS normalizes patches by parsing and unparsing abstract syntax trees before computing a canonical textual diff.The normalized patches are used for majority-based final selection.
  • Design principle: The pipeline is a simple step-by-step combination of existing techniques for localization, repair, and validation rather than a new technique in isolation.Its distinction from autonomous agents is the absence of complex environment interactions and agent-directed planning.

4 Experimental Setup

The experiments evaluate AGENTLESS on SWE-bench Lite against agent-based and agentless baselines using resolution, cost, token, and localization metrics.

  • 300 self-contained SWE-bench Lite problems provide the evaluation setting for AGENTLESS and baselines.
  • AGENTLESS is implemented with GPT-4o, greedy decoding by default, temperature 0.8 for sampling, and embedding retrieval using text-embedding-3-small.
  • 26 agent-based approaches, including open-source and commercial or closed-source tools, serve as comparison baselines.
  • The evaluation also includes a SWE-bench RAG baseline that retrieves relevant files with BM25 and directly generates a patch.
  • Reported metrics include resolved percentage, average cost, average tokens, and correct-location percentages at line, function, and file levels.

5 Evaluation

The evaluation reports results on SWE-bench Lite and distinguishes approaches whose source code is not released.

  • Table 1 reports results for AGENTLESS and prior approaches on SWE-bench Lite.
  • Closed-source approaches are identified because their source code is not released.

5.1 Performance on SWE-bench Lite

AGENTLESS achieves strong SWE-bench Lite performance and low cost, while its localization and reproduction-test analyses expose important evaluation details.

  • Performance results: 32.00%: AGENTLESS solves 96 of 300 SWE-bench Lite problems and achieves the highest performance among open-source approaches.
  • Performance results: $0.70: AGENTLESS’s average cost is lower than that of most prior agent-based approaches.
  • Issue overlap: Figure 5 compares issues uniquely solved by AGENTLESS with top-performing closed-source and open-source approaches.
  • Localization performance: Correct-location percentages correlate heavily with solve rate, although OpenCSG StarShip reaches 90.0% at file level despite a 23.67% solve rate.
  • Reproduction test results: AGENTLESS conservatively requires regression-test passage first, removes patches failing regression tests, and falls back to regression results when reproduction tests cannot select a patch.
  • Localization steps: Table 2 reports each localization step’s retained ground-truth locations, average lines of code, and average dollar cost.

5.2 Ablation study on components of AGENTLESS

AGENTLESS ablations show that hierarchical localization, multiple repair candidates, and reproduction-test filtering jointly improve performance while controlling context and cost. Increasing samples helps until majority voting plateaus, whereas considering all patches reveals additional potential.

  • Localization: 81.7% correct file localization results from combining prompting-based and embedding-based retrieval, outperforming either method alone.The individual methods locate the ground-truth file in 78.7% and 67.7% of cases, respectively.
  • Localization: Hierarchical edit-location localization preserves accuracy while reducing context, whereas directly mapping files to edit locations worsens both performance and cost.Merging multiple location samples increases ground-truth coverage but adds repair-phase context; separate samples support downstream repair.
  • Repair: 96 fixes are achieved when separately sampled edit-location sets each generate candidate patches, exceeding the more than 88 fixes obtained from greedy locations.Different location sets can expose distinct ground-truth locations and useful repair context, especially with test filtering and selection.
  • Repair: 80 correct fixes are obtained with one greedy patch sample per location set, while performance plateaus around 40 samples under majority voting.Considering all patch samples raises the possible solved issues to 126 (42.0%), indicating additional re-ranking potential.
  • Patch validation: 96 correct fixes result after adding reproduction-test filtering to regression-test selection, up from 77 with majority voting and 81 with regression filtering alone.Generating reproduction tests adds cost because they are not provided in the original repository; increasing candidates to 40 per issue reaches the same final performance.

6 Additional Analysis on SWE-bench Lite

The analysis classifies SWE-bench Lite problems, identifies benchmark-quality concerns, and constructs SWE-bench Lite-S to enable more rigorous comparisons of software-development approaches.

  • Problem Classification: 10.0% of SWE-bench Lite problems lack enough information to solve the issue reliably.These include unspecified function names or error strings and descriptions allowing multiple interpretations, only some of which match the developer test.
  • Problem Classification: 4.3% of issues provide the exact ground truth patch, while 9.7% describe the exact steps needed for the correct solution.These issues can be substantially easier because the solution is supplied as code or natural language.
  • Problem Classification: 5.0% of issues contain proposed solutions or steps that do not reflect the ground truth patch.Such misleading descriptions can direct approaches toward incorrect repairs.
  • Problem Classification: The classification reveals unsolvable questions, misleading solutions, and substantial differences in problem difficulty within SWE-bench Lite.These issues were not properly considered by the benchmark creation process or prior approaches.
  • SWE-bench Lite-S: SWE-bench Lite-S contains 249 problems after removing issues with exact patches, misleading solutions, or insufficient information.The filtering is intended to eliminate less reasonable problems and normalize benchmark difficulty.
  • SWE-bench Lite-S: SWE-bench Lite-S provides a more accurate reflection of tool capability than the original benchmark, although overall approach rankings remain roughly similar.The comparison uses the filtered benchmark alongside the original 300 SWE-bench Lite problems.
  • Category-Based Analysis: Test generation is important for patch selection, while closed-source agents perform better than AGENTLESS when no location clue is provided.The latter advantage is attributed to agent-based tools’ ability to use complex code-search tools.
  • SWE-bench Verified: AGENTLESS solves 194 of 500 SWE-bench Verified problems, achieving a 38.80% solve rate.This result is reported alongside comparisons with prior agent-based approaches.

7 Threats to Validity

The paper identifies threats to validity from possible training-data leakage and limited evidence about generalization beyond SWE-bench Lite.

  • Internal Validity: GPT-4o may have seen SWE-bench Lite ground truth patches during training, but its closed-source training data cannot be verified.Fully addressing this threat would require retraining GPT-4o from scratch, which the authors consider infeasible.
  • External Validity: AGENTLESS’s performance may not generalize to datasets beyond SWE-bench Lite.The authors plan to evaluate it on additional benchmarks, while noting SWE-bench Lite’s popularity and problem diversity.

8 Conclusion

The paper presents AGENTLESS as a simple alternative to autonomous software agents and pairs its evaluation with a more rigorous filtered benchmark.

  • Conclusion: AGENTLESS solves software-development problems through localization, repair, and patch validation without autonomous tool usage or planning.The approach is designed as a simpler alternative to prior agent-based methods.
  • Conclusion: AGENTLESS achieves the highest performance among evaluated open-source techniques while minimizing cost, and motivates SWE-bench Lite-S by removing problematic problems.The paper also contributes a detailed classification of SWE-bench Lite problems.
Loading 2407.01489v2…