PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoCodeRover: Autonomous Program Improvement

Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, Abhik Roychoudhury

arXiv:2404.05427v3cs.SEcs.AI

TL;DR

AutoCodeRover addresses the challenge of autonomously improving software from real-life GitHub issues, beyond automated code generation. It combines LLM agents with AST-based, iterative code search and optional fault localization to retrieve context and generate patches. On SWE-bench, it reports 19% efficacy, while the benchmark-based resolution criterion may allow overfitting.

  • Problem

    Real-life program repair requires resolving GitHub issues despite limited test availability and without assuming buggy locations are already known.

  • Method

    AutoCodeRover uses LLM agents with repository-aware AST code search, iterative context retrieval, and optional spectrum-based fault localization to generate patches.

  • Results

    AutoCodeRover achieved 19% efficacy on the SWE-bench benchmark, according to the paper’s reported headline result.

  • Takeaways & Limitations

    The workflow supports autonomous software engineering by using program structure to guide issue resolution and patching.

  • Takeaways & Limitations

    SWE-bench resolution is based on patches passing pull-request tests, so a reported resolution may be overfitting.

Abstract

from arXiv · show

Researchers have made significant progress in automating the software development process in the past decades. Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding. Nevertheless, software engineering involves the process of program improvement apart from coding, specifically to enable software maintenance (e.g. bug fixing) and software evolution (e.g. feature additions). In this paper, we propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. In contrast to recent LLM agent approaches from AI researchers and practitioners, our outlook is more software engineering oriented. We work on a program representation (abstract syntax tree) as opposed to viewing a software project as a mere collection of files. Our code search exploits the program structure in the form of classes/methods to enhance LLM's understanding of the issue's root cause, and effectively retrieve a context via iterative search. The use of spectrum-based fault localization using tests, further sharpens the context, as long as a test-suite is available. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent. In addition, AutoCodeRover achieved this efficacy with significantly lower cost (on average, $0.43 USD), compared to other baselines. We posit that our workflow enables autonomous software engineering, where, in future, auto-generated code from LLMs can be autonomously improved.

1 Beyond Automatic Programming

AutoCodeRover extends LLM-based coding toward autonomous program improvement by combining repository-aware code search with iterative context retrieval and debugging guidance. It targets bug fixing and feature addition while emphasizing AST-based program structure and reports 19% efficacy on SWE-bench lite.

  • Motivation and scope: The system addresses the gap between automated coding and broader software maintenance or evolution tasks.The motivation includes improving automatically generated code and automating program repair.
  • Motivation and scope: AutoCodeRover targets autonomous program improvement for bug fixing and feature addition in existing software projects.The approach augments LLMs with context knowledge from the code repository.
  • Software-engineering orientation: Context retrieval proceeds iteratively, with the LLM refining its issue understanding as code-search results become available.Search APIs run locally over AST-based representations and return class signatures and method implementations.
  • Debugging guidance: Spectrum-based fault localization can prioritize code context from suspicious methods when an accompanying test suite is available.It uses passing and failing test control flow to assign method-level suspiciousness scores.
  • Software-engineering orientation: AutoCodeRover represents projects with abstract syntax trees and searches classes, methods, and code snippets to retrieve relevant context for LLM reasoning.Its code search is designed to resemble human software-engineering activity and improve context use.
  • Reported results: 19% efficacy was achieved on SWE-bench lite with 300 GitHub issues, within 4 minutes.The paper also reports that two-thirds of autonomously produced patches were correct and acceptable.

2 Relevant Literature

Prior program-repair methods often assume known buggy locations and high-quality tests, whereas real-life GitHub issues require locating and understanding changes from natural-language descriptions. SWE-bench lite evaluates this end-to-end challenge on 300 issues from 11 large Python projects.

  • Program Repair: Traditional automated program repair generates patches intended to pass a given test suite using search-based, semantic-based, or pattern-based techniques.Search-based systems such as GenProg use predefined mutation operators or search patch spaces.
  • Program Repair: Recent LLM-based repair commonly assumes perfect fault localization and focuses on prompting models to generate patches for preselected statements.These methods target functional bugs, type bugs, and vulnerabilities across languages and models.
  • Program Repair: General real-life issue resolution remains challenging because existing repair techniques rely on high-quality tests and do not exploit natural-language problem specifications.The cited challenge concerns resolving issues from scratch rather than repairing predefined locations.
  • LLM agents and benchmark: SWE-bench lite contains 300 real-life software-engineering tasks collected from 11 popular large Python repositories.Each task pairs a GitHub issue with the corresponding human pull request and tests.
  • LLM agents and benchmark: Unlike code-generation benchmarks, SWE-bench lite requires reasoning about locations, root causes, repair strategies, and code changes in mature repositories.Its tasks may be bug fixes or feature additions specified through GitHub issues.
  • LLM agents and benchmark: AutoCodeRover differs from file-oriented agents by using software structure to infer specification and guide patching.The comparison passage contrasts it with SWE-agent’s shell-based file operations.

3 Motivating Example

The Django motivating example shows AutoCodeRover retrieving increasingly specific AST context before selecting the method responsible for the reported validation error. A separate patch-generation agent then uses that context to draft the modification.

  • Issue and workflow: The motivating task django-13933 requests a Django feature addition concerning ModelChoiceField behavior.The example is included in SWE-bench lite.
  • Issue and workflow: AutoCodeRover separates context retrieval from patch generation when resolving the issue.Context retrieval covers Steps 1–3, followed by patch generation in Step 4.
  • Iterative context retrieval: The retrieval agent searches the project AST for names inferred from the natural-language issue, including ModelChoiceField and ModelMultipleChoiceField.The search returns relevant class and method context from the local Django codebase.
  • Iterative context retrieval: Iterative search reveals that ModelChoiceField lacks clean and exposes to_python and validate as relevant methods.These results refine the agent’s initial understanding of the issue.
  • Patch localization: The agent selects to_python because it raises the relevant exception without including the invalid value in the error message.The selected location and gathered context are passed to the patch-generation agent.
  • Patch localization: The patch-generation stage modifies the error handling so a value can be integrated into the error message.The workflow uses the identified buggy location and retrieved context to draft the patch.

4 AI Program Improvement Framework

AutoCodeRover autonomously analyzes GitHub issues, retrieves structurally relevant code context, and generates patches. Its framework combines AST-based, iterative code search with optional test-based fault localization to identify buggy locations.

  • Overview: AutoCodeRover analyzes an issue and codebase, retrieves relevant context, and generates a patch for human vetting.The workflow is designed for realistic maintenance, where developers submit issues and maintainers review generated patches.
  • Overview: The framework operates in two main stages: context retrieval followed by patch generation.The retrieval stage gathers relevant snippets, while patch generation extracts precise snippets and crafts the final modification.
  • Context Retrieval APIs: AST-based retrieval APIs search classes, methods, and snippets locally, returning signatures or implementations as context for the LLM.Class signatures can limit context length, while method implementations provide concrete code details.
  • Context Retrieval APIs: Stratified search invokes selected retrieval APIs iteratively, adding results to context until the agent judges the issue sufficiently understood or identifies buggy locations.This avoids incomplete context from a single search while limiting distraction from executing every invocation at once.
  • Debugging Integration: When a test suite is available, spectrum-based fault localization assigns suspiciousness scores to program locations and guides more precise retrieval.SBFL uses control-flow differences between passing and failing tests, but its accuracy depends strongly on test-suite quality.

5 Experiment Setup

The evaluation measures AutoCodeRover on real-life GitHub issues using established SWE-bench benchmarks and compares it with LLM-based agent baselines. Experiments repeat runs to reduce the effect of LLM randomness and assess generated-patch correctness in the official environment.

  • Benchmark: AutoCodeRover is evaluated on SWE-bench and SWE-bench lite, containing 2294 and 300 real-life GitHub issues, respectively.Each issue supplies a natural-language description and its corresponding buggy codebase.
  • Baseline and Evaluation Metric: The study compares AutoCodeRover with the LLM-based agent systems Devin and Swe-agent.Swe-agent is compared using its publicly available original results, while Devin uses the most relevant reported technical-report result.
  • Baseline and Evaluation Metric: Experiments are repeated three times to reduce the influence of LLM randomness.The setup explicitly accounts for nondeterministic model behavior when reporting performance.
  • Implementation and Parameters: Generated-patch correctness is evaluated in the official SWE-bench Docker environment on an x86_64 Linux server running Ubuntu 20.04.The foundation model is GPT-4 with temperature 0.2 and a 1024-token maximum output.

6 Evaluation

AutoCodeRover resolves real-life GitHub issues across SWE-bench settings, with performance varying by repetition, comparison system, and use of debugging information. Its evaluation also examines cost, patch correctness, complementary coverage, and failure categories.

  • Overall effectiveness: 19% of SWE-bench lite task instances were resolved by AutoCodeRover @1, compared with 18% by Swe-agent.AutoCodeRover @3 increased the resolved rate to 26% on the 300-instance benchmark.
  • Overall effectiveness: 12.42% of full SWE-bench tasks were resolved by AutoCodeRover @1 and 17.96% by @3, while Swe-agent resolved 12.47%.AutoCodeRover @1 took 248 seconds per task and @3 took 701 seconds per task.
  • Comparison with Devin: 12.63% of tasks were resolved by AutoCodeRover @1 and 18.77% by @3 on Devin’s 570-task subset, exceeding Devin’s result.The passage also reports that AutoCodeRover took much less time than Devin.
  • Detailed Comparison with Swe-agent: AutoCodeRover uniquely resolved 26 task instances, while Swe-agent uniquely resolved 23, indicating complementary coverage.The AutoCodeRover advantage is attributed to fine-grained AST-level context search, whereas failures included unimplemented search APIs.
  • Time / Token Cost: AutoCodeRover averaged 195 seconds, 37k tokens, and $0.43 per SWE-bench lite task, versus 245k tokens and $2.51 for Swe-agent.Across three repetitions, AutoCodeRover averaged 520 seconds per task, below the 30–60-minute limit considered tolerable for automated repair.
  • Plausible / Correct patches: 65.4% of AutoCodeRover’s plausible patches were correct, with 51 correct patches among 78 plausible patches.Most overfitting patches modified the same methods as developer patches but made incorrect code changes.
  • RQ2: Fault Localization: SBFL increased resolved SWE-bench lite tasks from 57 to 66, raising the resolved rate from 19% to 22%.ACR-sbfl also uniquely resolved 7 task instances not resolved in the other runs.
  • Challenges on real-life tasks: 26.0% of SWE-bench lite issues were resolved, while 29.3% had correct method-level locations but incorrect patches.Other failures involved wrong methods or classes in the correct file, wrong files, or no applicable patch.

7 Discussion on Experiments and Improvements

AutoCodeRover’s results are competitive while emphasizing program-structure-guided patching, and the paper outlines improvements involving reproduction tests, semantic artifacts, and human involvement.

  • Position on Experiment Results: AutoCodeRover reached 18.83% on full SWE-bench as of 12th July 2024, alongside Factory Code Droid’s 19.27% resolve rate.The authors emphasize both low cost and the method’s use of program specifications to guide patching.
  • Issue Reproducer: AutoCodeRover can use issue descriptions or SBFL when an accompanying test suite is available, but the latter scenario is not always practical.User-provided reproduction scripts are identified as a possible alternative source of validation and guidance.
  • Issue Reproducer: Future work could generate bug-reproduction tests from GitHub issues to validate patches and support regeneration when patches fail.This proposal targets cases where issue descriptions contain concrete reproduction information.
  • Semantic Artifacts: Static call-graph and forward data-dependence analyses could extend context retrieval beyond methods initially identified from the issue description.These semantic artifacts are proposed as additional ways to find relevant methods.
  • Human Involvement: The current system leaves context-location and retrieval-termination decisions to the foundation LLM, motivating flexible interfaces and criteria for human involvement.The authors note that maintainers commonly discuss pull requests over multiple rounds before creation.

8 Threats to Validity

The paper identifies randomness and possible patch overfitting as threats to the evaluation, addressing them through repeated experiments, replication materials, and manual verification.

  • Threats to Validity: LLM randomness may produce different results across runs, so the main evaluation was repeated three times and a replication package was released.These measures address the validity threat associated with nondeterministic generation.
  • Threats to Validity: A patch passing all pull-request tests may still be overfitting, so the authors manually checked semantic equivalence with developers’ patches.The evaluation’s resolution criterion is therefore supplemented by author verification.

9 Perspectives

The paper positions AutoCodeRover as a response to untrusted LLM-generated code, combining structured, iterative, and test-guided search to infer developer intent for code improvement.

  • Perspectives: LLM-generated code may be incorrect or vulnerable, motivating autonomous processes for improving automatically generated and manually written code.The perspective connects code-generation promise with the need for trustworthy improvement processes.
  • Perspectives: AutoCodeRover integrates AST-based program representations, iterative structure-aware code search, and test-based fault localization when tests can be constructed.The approach aims to inject a software-engineering outlook into LLM-based autonomous code improvement.
  • Perspectives: AutoCodeRover treats project structure as a source for inferring developer intent during automated program modifications.The paper identifies intent or specification inference as the central challenge in program repair.
  • Perspectives: Future software-engineering workflows may involve developers vetting LLM conversations across the software lifecycle rather than holding one narrowly defined role.The paper also anticipates greater emphasis on trust if autonomous software engineering becomes commonplace.

DATA AVAILABILITY

The authors provide public access to AutoCodeRover’s implementation, experimental patches and LLM conversation histories, and replication scripts.

  • DATA AVAILABILITY: The project website provides AutoCodeRover’s implementation, generated patches, LLM conversation history, and experiment replication scripts.The materials are publicly accessible at the cited GitHub repository.
Loading 2404.05427v3…