PaperlayerView exact revision on arXiv

Source-linked AI summary

RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

Islem Bouzenia, Premkumar Devanbu, Michael Pradel

arXiv:2403.17134v2cs.SEcs.AI

TL;DR

Automated program repair seeks to reduce the substantial effort of manually fixing bugs, while existing LLM approaches use fixed prompts or hard-coded feedback loops. RepairAgent instead uses an autonomous LLM agent that invokes repair tools and adapts its actions, successfully fixing 164 Defects4J bugs, including 39 not fixed by prior work.

  • Problem

    Existing LLM-based repair approaches use one-time prompts or hard-coded feedback loops that do not let the model gather broader bug and code information.

  • Method

    RepairAgent lets an LLM autonomously plan and invoke repair-specific tools through dynamically updated prompts, middleware, and finite-state guidance.

  • Results

    164 bugs were successfully fixed in Defects4J, including 39 bugs not fixed by prior work.

  • Takeaways & Limitations

    RepairAgent establishes a state-of-the-art autonomous approach for program repair and is intended to support future agent-based software-engineering techniques.

  • Takeaways & Limitations

    Defects4J provides at least one failing test case for every bug, unlike some real-world repair scenarios without a-priori error-revealing tests.

Abstract

from arXiv · show

Automated program repair has emerged as a powerful technique to mitigate the impact of software bugs on system reliability and user experience. This paper introduces RepairAgent, the first work to address the program repair challenge through an autonomous agent based on a large language model (LLM). Unlike existing deep learning-based approaches, which prompt a model with a fixed prompt or in a fixed feedback loop, our work treats the LLM as an agent capable of autonomously planning and executing actions to fix bugs by invoking suitable tools. RepairAgent freely interleaves gathering information about the bug, gathering repair ingredients, and validating fixes, while deciding which tools to invoke based on the gathered information and feedback from previous fix attempts. Key contributions that enable RepairAgent include a set of tools that are useful for program repair, a dynamically updated prompt format that allows the LLM to interact with these tools, and a finite state machine that guides the agent in invoking the tools. Our evaluation on the popular Defects4J dataset demonstrates RepairAgent's effectiveness in autonomously repairing 164 bugs, including 39 bugs not fixed by prior techniques. Interacting with the LLM imposes an average cost of 270,000 tokens per bug, which, under the current pricing of OpenAI's GPT-3.5 model, translates to 14 cents of USD per bug. To the best of our knowledge, this work is the first to present an autonomous, LLM-based agent for program repair, paving the way for future agent-based techniques in software engineering.

I. INTRODUCTION

RepairAgent treats program repair as an autonomous LLM-agent task, addressing limitations of fixed and iterative repair loops by invoking tools and adapting actions to gathered information and feedback. Its tools, dynamic prompting, middleware, and state-machine guidance support this approach, which fixes 164 Defects4J bugs, including 39 not fixed by prior work.

  • Motivation: Current LLM-based repair uses one-time prompts or hard-coded iterative feedback loops that restrict the model’s repair context.These approaches typically provide buggy code and test feedback but do not let the model gather broader bug or code information.
  • Approach: RepairAgent is an autonomous LLM-based agent that plans and executes tool calls to gather bug information, find repair ingredients, and validate fixes.The LLM chooses which tool to invoke next using gathered information and feedback from earlier fix attempts.
  • Approach: RepairAgent combines human-developer-oriented repair tools, a dynamically updated prompt format, and middleware for LLM–tool communication.The tool set covers repair steps such as reading code and searching the code base, while the prompt updates with invoked commands and their results.
  • Evaluation: 164 bugs were successfully fixed, including 39 bugs not fixed by prior work.The evaluation covered all 835 bugs in Defects4J, and the fixed set included bugs requiring changes to more than one line.
  • Contribution: RepairAgent is presented as the first autonomous LLM-based agent for program repair and as a basis for future agent-based software-engineering techniques.The authors also state that they will release its implementation as open source.

II. BACKGROUND ON LLM-BASED, AUTONOMOUS AGENTS

RepairAgent frames program repair as an autonomous LLM-agent task, where the model plans tool-using actions rather than following a fixed query or feedback loop. Its middleware repeatedly executes model-selected commands and feeds their outputs into a dynamic prompt.

  • LLM-based agents autonomously plan and execute action sequences toward a goal, including invocations of external tools.
  • RepairAgent’s middleware initializes the agent, parses and executes its tool calls, and integrates outputs into the next prompt.The process continues iteratively until the bug is fixed or a predefined budget is exhausted.
  • Each repair cycle queries the LLM, post-processes its response, executes the suggested command, and updates the dynamic prompt.
  • The dynamic prompt is a sequence of static and dynamic sections whose contents may remain constant or change across cycles.

C. Dynamic Prompting of the Repair Agent

RepairAgent uses a dynamically updated prompt to define the autonomous repair agent’s role, goals, and operating context. The prompt directs the agent from bug localization and understanding through increasingly complex fixes and iteration.

  • 1) Role:: The prompt defines the agent as an autonomous Java-bug repair specialist whose objective is to understand and fix bugs without user assistance.
  • 2) Goals:: The agent pursues five persistent goals: locating the bug, gathering information, suggesting simple fixes, suggesting complex fixes, and iterating until a fix is found.
  • 2) Goals:: Fault localization and test execution help pinpoint the bug’s location, unless localization information is already supplied.
  • 2) Goals:: The agent analyzes bug-associated code to understand the defect before proposing repairs.
  • 2) Goals:: Repair proceeds from simple fixes to more complex fixes when earlier attempts prove ineffective.
  • 2) Goals:: The agent repeatedly gathers information and proposes fixes until it finds a repair.

3) Guidelines:

RepairAgent’s guidelines shape the agent’s reasoning and tool use during repair. A finite state machine structures movement from understanding the bug, through information gathering, to validated fix attempts.

  • 3) Guidelines:: The prompt accommodates single-line and multi-line bugs and supplies recurring fix patterns with natural-language descriptions and buggy-to-fixed examples.
  • 3) Guidelines:: The guidelines require comments above modified code, a clearly defined next tool step, and efficient selection under a limited invocation budget.
  • 3) Guidelines:: The finite state machine constrains which tools are available at each point and mirrors states a human developer would traverse when fixing a bug.
  • 4) State Description:: In “Understand the bug,” the agent collects failing-test and bug-location information, formulates hypotheses, and can revise them during repair.
  • 4) State Description:: In “Collect information to fix the bug,” the agent searches for repair ingredients and reads relevant code before attempting a fix.
  • 4) State Description:: In “Try to fix the bug,” each attempt modifies the code base and is validated by executing the test cases.
  • 4) State Description:: The agent may return to earlier states to establish a new hypothesis or gather additional information, then reaches “Done” through a success command.

6) Gathered Information:

The agent maintains gathered information as prompt memory and responds in a structured format that supports tool execution. Its tools let it autonomously inspect code, search for repair ingredients, and choose developer-like actions.

  • 6) Gathered Information:: A prompt section records information returned by tool invocations, allowing the agent to recall findings from previous cycles.
  • Each response contains thoughts describing the next-command reasoning and a command specifying the tool invocation to execute.
  • An example response shows the agent requesting more bug information by searching the code base with keywords.
  • 8) Last Executed Command and Result:: The prompt includes the last executed command, its output, and the number of cycles executed and remaining.
  • The agent autonomously chooses among repair tools inspired by those developers use in integrated development environments.
  • Code-reading tools provide focused file ranges and broader class-and-method structure to help the agent understand relevant code.

2) Search and generate code:

RepairAgent searches existing code and generates new method bodies to gather repair ingredients, then applies and validates candidate patches. Its testing workflow can revert failed changes and evaluate multiple fix variants.

  • 2) Search and generate code:: RepairAgent searches the code base for keywords and similar API calls to understand bugs and retrieve repair ingredients.Approximate keyword matching locates related methods, while similar-call search helps avoid nonexistent method calls.
  • 2) Search and generate code:: A separate tool generates a method body from the preceding code and method signature by invoking another LLM.The code-generating LLM operates independently of RepairAgent’s dynamic prompt.
  • 3) Testing and Patching:: The run tests tool executes the project test suite and returns a cleaned report indicating whether tests passed or failed.Cleaning removes irrelevant stack-trace entries to reduce prompt noise.
  • 2) Search and generate code:: The agent can express a bug hypothesis to enter information collection or discard it to return to bug understanding.These actions connect the search-and-testing workflow to RepairAgent’s state-based control.
  • 3) Testing and Patching:: After gathering sufficient information, the write fix tool applies JSON-formatted insertions, deletions, and modifications to the code base.The tool is intended for arbitrary complex bugs, including multi-line and multi-file changes.
  • 3) Testing and Patching:: Failed patches are automatically reverted, and the tool samples up to 30 variants, removes duplicates, and tests each remaining variant.This workflow targets fix attempts that are almost correct.

4) Control:

RepairAgent uses control tools and middleware to manage the agent’s state, interpret LLM responses, and execute valid commands safely. The middleware also detects malformed or repeated actions and refreshes the agent’s context after execution.

  • 4) Control:: Control tools let RepairAgent express or discard bug hypotheses, moving between information-collection and bug-understanding states.Discarding a no-longer-viable hypothesis returns the agent to understanding the bug.
  • 4) Control:: At each cycle, middleware queries the LLM with the current prompt and parses its proposed tool invocation.The process repeats as the agent interacts with available tools.
  • 4) Control:: Middleware heuristically maps malformed tool names, argument names, and argument values to the expected format.It uses substring or Levenshtein matching and retries through a new cycle when mapping fails or is ambiguous.
  • 4) Control:: Repeated invocations of the same tool with identical arguments are detected and reported back to the agent.The middleware then enters another cycle rather than executing the duplicate command.
  • 4) Control:: Valid commands run in an isolated environment to prevent tool executions from interfering with the host or RepairAgent.

3) Updating the Prompt:

The evaluation examines RepairAgent’s effectiveness, cost, component influence, and tool use using broad bug datasets and iterative repair baselines. The setup defines how datasets, comparisons, and patch correctness are assessed.

  • 3) Updating the Prompt:: The prompt-updating middleware refreshes state descriptions, available tools, gathered information, and the last executed command after each tool result.
  • IV. IMPLEMENTATION: RepairAgent is implemented with Python 3.10, Docker isolation, AutoGPT, GPT-3.5-0125, and ANTLR for Java interaction.
  • V. EVALUATION: The evaluation asks how effectively RepairAgent fixes real-world bugs, what it costs, and how its components and tool use influence behavior.
  • A. Experimental Setup: The Defects4J evaluation covers all 835 real-world bugs from 17 Java projects across versions 1.2 and 2.0.Using the complete dataset is intended to assess generalization across projects and bugs without restricting bug complexity.
  • A. Experimental Setup: A secondary evaluation samples 100 of GitBug-Java’s 199 bugs from 55 projects, whose fixes postdate the GPT-3.5 training cutoff.
  • A. Experimental Setup: RepairAgent is compared with ChatRepair, ITER, and SelfAPR, whose iterative baselines use feedback from previous patch attempts without an autonomous LLM agent.
  • A. Experimental Setup: The study reports plausible and correct patches, defining plausibility as passing all tests and correctness through syntactic or manual semantic comparison with developer fixes.

B. RQ1: Effectiveness

RepairAgent produces plausible and correct fixes across Defects4J, including fixes outside the baseline union and complex multi-line or multi-file repairs. Its effectiveness is especially pronounced for fixes requiring more than one line.

  • B. RQ1: Effectiveness: 186 bugs receive plausible fixes, while 164 receive correct fixes; 116 exactly match developer patches and 48 are semantically consistent.The results span bugs from different projects and therefore support generalization across multiple code-base domains.
  • B. RQ1: Effectiveness: 164 correct fixes include 49 bugs requiring more than one changed line, demonstrating repairs beyond single-line edits.
  • B. RQ1: Effectiveness: RepairAgent fixes 90 Defects4J v2.0 bugs versus ChatRepair’s 48 and repairs 39 bugs missed by all three baselines.Those 39 bugs include 18 single-line, 20 multi-line, and one multi-file bug.
  • B. RQ1: Effectiveness: RepairAgent is particularly more effective on bugs requiring multi-line fixes, which the authors attribute to retrieving repair ingredients and editing arbitrary lines and files.
  • B. RQ1: Effectiveness: For an exclusively repaired bug, the find similar api calls tool retrieves a related call whose differing field value becomes a repair ingredient.

4) Generalization and External Validity:

RepairAgent generalizes to GitBug-Java but performs better on single-line than multi-line, multi-file bugs. Its costs and effectiveness also depend on localization, search tools, state-machine guidance, and long-term memory.

  • Generalization: 19 plausible and 13 correct fixes were found on GitBug-Java, including 9 correct fixes among 19 single-line bugs.For 81 multi-line and multi-file bugs, only 4 correct fixes were found.
  • Costs: RepairAgent uses a median of 270,000 tokens per bug, costing approximately 14 cents under GPT-3.5 pricing.Fixed bugs consumed 21,000 tokens, compared with 315,000 for unfixed bugs.
  • Ablations: Without search tools, RepairAgent fixes half as many bugs and incurs doubled costs because it reads longer code sequences that saturate the prompt.
  • Ablations: Removing state-machine guidance reduces fixes and increases costs because the agent often proposes fixes before collecting bug information.
  • Ablations: Long-term memory improves effectiveness by preserving useful information across cycles and avoiding repeated queries, wrong names, and repeated commands.
  • Fault localization: With spectrum-based GZoltar fault localization, RepairAgent fixes 16 bugs for 29 dollars, a 25% capability drop and 81% cost increase.

E. RQ4: Usage of Tools by the Agent

RepairAgent uses tools adaptively across repair cycles, combining bug understanding, code search, fix application, and test feedback. Its logs show that information retrieval and validation are central to the repair process.

  • Tool usage: RepairAgent makes an average of 35 tool invocations per bug and uses the full range of available tools.The most frequently called tool is write fix, averaging six invocations per bug.
  • Bug understanding: The agent retrieves failing-test results, similar code, code-structure details, and feedback from previous fix attempts to understand bugs.
  • Validation: Applying a fix triggers test execution, revealing which test cases still fail and supplying feedback for subsequent repair actions.
  • Observed behavior: RepairAgent sometimes proposes complex fixes for simple bugs and edits only part of the required locations in multi-line, multi-file bugs.The paper suggests trying simpler candidate fixes first and using partial fixes to help developers.

B. Threats to Validity and Limitations

The paper identifies threats involving data leakage, missing failing tests, fault localization, and nondeterministic LLM outputs. It also positions RepairAgent as an early autonomous LLM-based approach for program repair.

  • Threats to validity: GPT-3.5 may have seen parts of the evaluated Java projects during training, creating a potential data-leakage threat.The GitBug-Java evaluation is presented as evidence on bugs guaranteed not to be in the training data.
  • Threats to validity: Defects4J provides at least one failing test case per bug, whereas real-world scenarios may lack a-priori error-revealing tests.The paper identifies evaluation without such tests as future work.
  • Threats to validity: Inaccurate or imprecise fault localization could produce suboptimal repair suggestions or incorrect diagnoses.
  • Threats to validity: LLM nondeterminism may produce different RepairAgent outcomes across consecutive runs, although the large evaluation mitigates this risk.
  • Scope and contribution: RepairAgent is presented as the first LLM-based autonomous agent for program repair and contributes repair-specific tools and an interactive prompt format.
Loading 2403.17134v2…