PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study on LLM-based Agents for Automated Bug Fixing

Xiangxin Meng, Zexiong Ma, Pengfei Gao, Chao Peng

arXiv:2411.10213v2cs.SEcs.AI

TL;DR

The paper addresses the limited systematic evidence about how leading LLM-based agents differ in automated bug fixing. It compares six systems on SWE-bench Verified across repair, localization, and reproduction, and concludes that improving model reasoning and agent-flow design should be priorities.

  • Problem

    Systematic analyses of fault localization, repair capabilities, and performance differences among LLM-based agents were lacking.

    Source: §1 · p. 1

  • Method

    The authors compare six leading systems on SWE-bench Verified, examining overall repair, file- and symbol-level localization, and bug reproduction.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Analysis of the six systems points to improving LLM reasoning and agent-flow design, with attention to issue quality, stack traces, and reproduction correctness.

    Source: §1 · pp. 1–2, §8 · p. 12

  • Takeaways & Limitations

    The authors conclude that more effective bug-fixing agents require attention to both model capability and agentic workflow design.

    Source: §8 · p. 12

    The evaluation treats patches passing Fail-to-Pass tests as correct despite possible test incompleteness, and output-divergence-based reproduction checks may misclassify non-triggering scripts.

    Source: §6 · p. 11

Abstract

from arXiv · show

Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWE-bench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.

1 Introduction

LLM-based agents combine language-model analysis with environmental interaction to edit, execute, and validate bug fixes, but their repair and localization differences have not been systematically analyzed. This study compares six leading systems and concludes that both model reasoning and agent workflow need further improvement.

1 Introduction

  • Systematic analysis of agent fault-localization and repair capabilities, and of performance differences among tools, was lacking despite systems solving different cases.Issue-description quality, defect dependencies, and system design are identified as factors related to variation.Source: §1 · p. 1
  • The study compares six top-performing systems on SWE-bench Verified, examining shared successes and failures, fault localization at file and symbol levels, and bug reproduction.It also analyzes differences in the instances each system can solve.Source: §1 · p. 1
  • The authors recommend stronger model reasoning to identify relevant issue information and repair locations, alongside agent-flow checks for patch completeness and global impact.They also call for attention to issue quality, multiple suspicious stack-trace locations, and mitigating output randomness.Source: §1 · pp. 1–2
  • The paper’s stated contributions cover agent repair effectiveness, fault-localization differences, bug-reproduction impact, and future research directions.It presents these as the first such study, to the authors’ knowledge.Source: §1 · p. 2

2 Background

The background introduces SWE-bench and six leading LLM-based bug-fixing systems. It describes the benchmark’s real-world repository tasks and outlines varied agent designs, from tool use and search to accumulated interaction experience and multi-agent patch selection.

2 Background

  • The background introduces SWE-bench and then the leading LLM-based bug-fixing systems.Source: §2 · p. 2

2.1 SWE-bench Benchmark

  • SWE-bench contains real-world GitHub issue and pull-request tasks across 12 Python repositories; SWE-bench Verified is a 500-instance release intended to improve evaluation quality.The benchmark requires work across large codebases, interaction with execution environments, and handling long contexts.Source: §2.1 · p. 2

2.2 Leading LLM-based Bug Fixing Systems

  • LLM-based bug-fixing systems edit code repositories based on issue reports; this study compares six systems’ designs and performance.The paper aims to identify shortcomings and guide future adaptive, high-reliability tools.Source: §2.2 · p. 2
  • W&B Programmer uses five parallel rollouts and an o1 crosscheck; Blackbox uses ReAct-style repository tools and tests, while CodeStory uses Monte Carlo Tree Search.These descriptions illustrate different approaches to exploration, validation, and candidate selection.Source: §2.2 · p. 2
  • Learn-by-interact reuses offline interaction trajectories, Devlo tests candidates with reproduction and edge-case tests, and Emergent E1 selects among patches from multiple expert agents.Source: §2.2 · pp. 2–3
  • The paper plans to analyze localization accuracy across strategies and how system components relate to final patch-generation accuracy.The stated aim is to inform more reliable bug-fixing systems.Source: §2.2 · p. 3

3 Study Design

The study frames three questions around repair effectiveness, fault localization, and bug reproduction, then assesses issue-description quality across five dimensions. It uses an LLM-based scoring process and checks that process against regex rules for structured cues.

3.1 Research Questions

  • The study asks how systems perform in repair, how they differ in fault localization, and how effective reproduction is; reproduction scripts are assessed by output changes around the golden patch.Because scripts were difficult to collect from existing systems, the authors designed a repair agent with a mainstream workflow and collected its scripts and execution commands.Source: §3.1 · p. 3

3.2 Issue Quality Assessment Criteria

  • Issue descriptions are scored across five dimensions inspired by Agentless, with code-symbol localization broader than the function-level granularity used there.The assessment uses DeepSeek-R1 and defines code symbols to include classes, functions, class methods, and top-level statements.Source: §3.2 · p. 3
  • File-level localization compares issue-description cues with golden files and selects the maximum score when a case modifies multiple files.The rubric distinguishes references such as file names and paths from absent information.Source: §3.2 · pp. 3–4
  • Symbol-level localization uses AST-extracted modified symbols in file_path::symbol_name form, while line scoring matches issue cues against line-numbered golden patches.For cases with multiple modified symbols, the maximum symbol-level score is retained.Source: §3.2 · pp. 3–4
  • Reproducible-example scoring detects whether issue descriptions contain runnable code, partial code, or natural-language reproduction steps, without judging semantic correctness.The method explicitly limits this dimension to detecting presence rather than evaluating whether an example truly reproduces the defect.Source: §3.2 · p. 4
  • Resolve-solution scoring distinguishes exact patches and complete or partial natural-language fixes from absent or misleading solution information.The rubric compares the described solution with the golden patch.Source: §3.2 · p. 4

3.3 Reliability of the Scoring System

  • Across 500 issues, DeepSeek R1 and regex rules disagreed in 34 cases; inspection found that DeepSeek identified debugging signals that regex missed in varied formats and natural-language reports.The authors regard this comparison as support for using DeepSeek R1 to score issue quality.Source: §3.3 · pp. 4–5

4 Analysis & Results

Across three research questions, the analysis compares six systems’ repair performance, fault localization, and bug-reproduction scripts on SWE-bench Verified. Results show substantial variation across systems and cases, stronger file-level than symbol-level localization, and a relationship between reproduction-script quality and issue-description quality.

4.1 RQ1: Effectiveness of Systems

  • 404 of 500 cases were solvable by at least one system, 96 by none, and 181 by all six; all five issue-quality indicators scored higher in the all-resolve set.The average Quality of Resolve Solutions score in the all-resolve set was 1.95 times that of the no-one-resolve set; symbol-, file-, and line-location scores differed by 1.30, 1.26, and 1.22 times, while reproducible-example quality differed by 1.01 times.Source: §4.1 · p. 5, Table 1 · p. 6
  • 323 cases were resolved by W&B Programmer, the most among six systems, versus 286 for Emergent E1; W&B Programmer’s 64.6% coverage was a 13.0% improvement.The benchmark contains 500 cases; Figure 1 also shows how solvable cases differ across tool combinations.Source: §4.1 · p. 5, Figure 1 · p. 6
  • Learn-by-interact repaired 12 cases no other system repaired, while 30 cases repaired by other systems were missed by it; its approach uses historical interaction trajectories in prompts.The authors suggest that varied workflows and patch voting may uncover or resolve additional repairable cases.Source: §4.1 · pp. 6–7

4.2 RQ2: Effectiveness of FL

  • W&B Programmer localized at least one buggy file in 448 cases and all buggy files in 394, leading the file-level results and improving 31.0% over Learn-by-interact in the at-least-one-file setup.The tool order in the all-files setup was unchanged except that Devlo and Blackbox AI Agent swapped positions; W&B Programmer’s improvement over Learn-by-interact was 25.5%.Source: §4.2 · p. 7, Figure 2 · p. 8
  • At symbol level, W&B Programmer localized at least one symbol in 396 cases, 28.6% more than Learn-by-interact; it localized 50 more cases in the all-symbols setup, a 20% improvement.W&B Programmer ranked highest in symbol-prediction F1, Midwit Agent in precision, and Blackbox AI Agent second in instance-level localization.Source: §4.2 · pp. 7, 9, Figure 3 · p. 8
  • File- and symbol-level localization were positively correlated with bug-fixing performance, but symbol-level localization showed more variation and greater room for improvement than file-level localization.Systems may pursue different symbol-localization strategies, prioritizing broader recall or precise identification of the first erroneous symbols.Source: §4.2 · p. 9
  • Learn-by-interact ranked last in localization but fourth in bug fixing; successful file and symbol localization corresponded to repair probabilities of 82.2% and 85.1%, respectively.The probabilities were 281/342 after localizing at least one erroneous file and 262/308 after localizing at least one erroneous code symbol.Source: §4.2 · p. 9

5 Discussion

The discussion calls for stronger LLM reasoning and more deliberate Agentic-flow design, while noting the study’s Python-only scope and the need for fuller developer evaluation.

5.1 Large Language Model

  • LLMs should better identify issue-relevant information and choose among potential repair locations, while Agentic flows should address issue quality, suspicious stack-trace locations, patch completeness, global impact, and output variability.The discussion also calls for more symbol-level localization and better reproduction scripts through relevant artifacts and model reflection.Source: §5.1 · p. 10, §5.2 · p. 10

5.2 Agentic Flow

  • Agentic-flow design should also increase dynamic repair centered on reproduction, patch generation, and validation, and improve reproduction scripts with relevant artifacts and model reflection.The discussion emphasizes mechanisms for checking patch completeness and global impact, and for avoiding or using model-output diversity.Source: §5.2 · p. 10

5.3 General Applicability

  • The analysis uses only Python repositories in SWE-bench Verified, and the authors say multilingual benchmarks remain important future work.They argue their insights may generalize because the systems use language-agnostic tools and show no signs of Python-specific tuning.Source: §5.3 · p. 10

5.4 Qualitative Evaluations from Developers

  • Manual review of 30 patches per system found context-aware, syntactically correct, often clean and readable patches that reuse existing structures and include meaningful comments.A more comprehensive evaluation with developer feedback remains future work.Source: §5.4 · pp. 10–11

6 Threats to Validity

The study’s findings are bounded by test-suite coverage, stochastic and inaccessible systems, and possible misclassification of reproduction scripts.

Test validity

  • Fail-to-Pass tests may miss whether a patch fully resolves an issue, but the study treats passing those tests as correctness.The authors call for improved SWE-bench test cases to make evaluations more reliable.Source: §6 · p. 11

Comparative validity

  • Closed-source systems prevent standardized model and decoding settings or controlled ablations with identical LLM backbones, so model-capacity differences contribute to performance variation.The authors analyze submitted patches and trajectories as each agent’s best performance and note that repeated runs to address stochasticity are impractical.Source: §6 · p. 11

Assessment procedure

  • The study uses DeepSeek-R1 for issue-quality assessment, refines prompts against manual annotations, and selects the most frequent of five annotations per metric and instance.These procedures were intended to bring the LLM evaluation closer to software-expert judgments.Source: §6 · p. 11

Reproduction validity

  • Output divergence can mark a reproduction script as related to a bug even when it does not trigger that bug, creating possible false positives.The authors accepted this limitation because automating checks such as log-pattern or exception-type verification is practically complex within limited agent interactions.Source: §6 · p. 11

7 Related Work

Related work spans language models for software tasks, fault localization and program repair methods, and agent frameworks that interact with development environments.

7.1 Large Language Models

  • LLMs are applied across software tasks including code generation, translation, repair, fault localization, incident analysis, summarization, and testing.The section distinguishes encoder-only, decoder-only, and encoder-decoder architectures and notes task-specific fine-tuning.Source: §7.1 · p. 11

7.2 Fault Localization and Program Repair

  • Fault localization work includes dynamic methods such as SBFL and MBFL and static semantic or syntactic analysis, while APR spans search-based, semantics-based, and pattern- or learning-based approaches.Recent LLM-based APR work often uses APR-specific prompts to guide patch generation.Source: §7.2 · p. 11

7.3 Agents for Software Development

  • Agent frameworks support planning and iterative tool use, with systems such as SWE-agent and AutoCodeRover providing repository interfaces or APIs for locating and modifying code.Devin and OpenDevin are described as early end-to-end agent frameworks for software engineering tasks.Source: §7.3 · p. 12

8 Conclusion

The paper analyzes six top SWE-bench Verified bug-fixing systems, their fault-localization performance, and bug-reproduction tasks. It concludes that further work should improve LLM reasoning and Agentic-flow design, including attention to issue and stack-trace quality and reproduction correctness.

Conclusion

  • The analysis of six top SWE-bench Verified systems covers bug-fixing performance, fault localization at different levels, and bug reproduction, and points to reasoning and Agentic-flow improvements.The conclusion highlights issue quality, stack traces, and reproduction correctness as considerations for future agent design.Source: §8 · p. 12

Related papers

Loading 2411.10213v2…