PaperlayerView exact revision on arXiv

Source-linked AI summary

MarsCode Agent: AI-native Automated Bug Fixing

Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, Chao Peng

arXiv:2409.00899v2cs.SEcs.AI

TL;DR

Real-world bug fixing challenges LLMs because it requires understanding complex codebases and inter-file dependencies. MarsCode Agent combines multi-agent workflows and code-retrieval tools; on SWE-bench, it outperforms most existing automated tools in accuracy and efficiency.

  • Problem

    Real-world bug fixing challenges LLMs because it requires understanding complex codebases, interdependencies among files, and context-specific issues.

    Source: §1 · p. 1

  • Method

    MarsCode Agent uses multi-agent workflows to route issues to static or dynamic repair and combines code knowledge graphs with language server protocols for code retrieval and navigation.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    MarsCode Agent outperforms most existing automated bug-fixing tools in accuracy and efficiency; on SWE-bench Lite, it solved 102 instances, a 34% solve rate.

    Source: §1 · p. 2, §4.2 · p. 10

  • Takeaways & Limitations

    On SWE-bench, MarsCode Agent demonstrated the ability to fix a wide range of bugs without human intervention.

    Source: §1 · p. 2

    The authors identify reducing LLM call costs, supporting dynamic debugging in real user workspaces, and improving localization and modification accuracy as areas for further work.

    Source: §5 · p. 11

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have shown significant potential to automate various software development tasks, including code completion, test generation, and bug fixing. However, the application of LLMs for automated bug fixing remains challenging due to the complexity and diversity of real-world software systems. In this paper, we introduce MarsCode Agent, a novel framework that leverages LLMs to automatically identify and repair bugs in software code. MarsCode Agent combines the power of LLMs with advanced code analysis techniques to accurately localize faults and generate patches. Our approach follows a systematic process of planning, bug reproduction, fault localization, candidate patch generation, and validation to ensure high-quality bug fixes. We evaluated MarsCode Agent on SWE-bench, a comprehensive benchmark of real-world software projects, and our results show that MarsCode Agent achieves a high success rate in bug fixing compared to most of the existing automated approaches.

1 Introduction

Real-world bug fixing demands understanding complex, interdependent codebases, while traditional rule- and heuristic-based tools can be limited in scope and adaptability. MarsCode Agent combines multi-agent workflows and code tools to automate repairs, and it outperforms most existing automated tools on SWE-bench in accuracy and efficiency.

1 Introduction

  • Bug fixing in real-world projects requires understanding complex codebases, inter-file dependencies, and context-specific issues, challenges that limit traditional rule- and heuristic-based approaches.Bug fixing includes identifying defect causes, generating correct patches, and validating that fixes do not introduce new issues.Source: §1 · p. 1
  • MarsCode Agent uses multi-agent collaboration to allocate static or dynamic solving pipelines according to the problem, adapting to varied bug-fixing challenges.The framework also combines code knowledge graphs and language server protocols for code-entity retrieval, relationship retrieval, and definition-and-reference navigation.Source: §1 · p. 2
  • For code editing, MarsCode Agent uses conflict-based edit descriptions and static syntax checking; for dynamic debugging, it provides Docker sandbox capabilities for reproduction, logging, and test execution.These tools are presented as developer-like debugging capabilities.Source: §1 · p. 2
  • The authors present LLM-based agents as having potential to automate and enhance software engineering tasks, including feature development.They describe the framework as a step toward autonomous software maintenance and hope it will inspire further research.Source: §1 · p. 2
  • MarsCode Agent outperforms most existing automated bug-fixing tools in accuracy and efficiency on SWE-bench, fixing a wide range of bugs without human intervention.SWE-bench is described as a diverse benchmark of real-world software projects.Source: §1 · p. 2

2 Background and Related Work

The background reviews LLMs and agent frameworks for software engineering, fault localization and automated repair approaches, and SWE-bench as a benchmark of complex real-world tasks. It frames codebase-scale evaluation as demanding because tasks require coordinated changes across files and interaction with execution environments.

2 Background and Related Work

  • The background surveys LLMs, fault localization, automated program repair, software-development agents, and SWE-bench as context for the paper’s approach.It identifies fault localization and automated repair as central software-engineering applications.Source: §2 · p. 2

2.1 Large Language Models

  • LLMs support code generation and understanding across software-engineering tasks, while code-specific and instruction-following models are developed for coding use.The cited task areas include program synthesis, translation, repair, fault detection and localization, incident analysis, summarization, and testing.Source: §2.1 · pp. 2–3

2.2 Fault Localization

  • Fault localization methods include dynamic execution-based approaches, which can be resource-intensive, and static semantic or syntactic approaches, which offer faster detection with lower resource use.The section also notes multiple-fault and combined dynamic-static techniques.Source: §2.2 · p. 3

2.3 Automated Program Repair

  • Automated program repair includes search-based, semantics-based, and pattern- or learning-based approaches, with recent LLM work often prompting models to generate patches for buggy statements.Search-based methods use predefined mutation operators, while semantics-based methods solve repair constraints from test specifications.Source: §2.3 · p. 3

2.4 Agents for Software Development

  • Software-development agents use tools and interfaces to iteratively perform tasks; examples include repository interaction and APIs for locating code that needs modification.The discussion includes Devin/OpenDevin, SWE-agent, and AutoCodeRover.Source: §2.4 · p. 3

2.5 SWE-bench

  • SWE-bench contains 2,294 tasks from GitHub issues and pull requests across 12 Python repositories, emphasizing multi-file codebase work and execution-environment interaction.The passage reports that Claude 2 achieved a 1.96% success rate at the time of publication.Source: §2.5 · p. 3

3 Approach

The approach introduces MarsCode Agent as a framework organized around multi-agent collaboration, code indexing, and code-editing tools. Figure 1 is identified as the multi-agent collaborative framework.

3 Approach

  • MarsCode Agent’s approach is organized around a multi-agent collaborative framework.Figure 1 is titled “Multi-agent Collaborative Framework.”Source: §3 · p. 3, Figure 1 · p. 4
  • The approach includes code indexing tools designed for agents.Source: §3 · p. 3
  • The approach also includes code-editing tools designed for agents.Source: §3 · p. 3

3.1 Multi-agent Collaborative Framework

The framework uses different agent roles and static or dynamic workflows because repair tasks vary in their need for code review, execution, and debugging. Dynamic repair reproduces and tests issues in a Docker sandbox; static repair generates and votes on normalized candidate patches.

3.1 Multi-agent Collaborative Framework

  • Repair scenarios range from test failures and unexpected outputs to feature extensions and simple defects, so a fixed approach may not suit all tasks.Some issues can be addressed by reviewing relevant code, while deeper exceptions or complex logic errors may require execution and variable tracking.Source: §3.1 · p. 4
  • Six roles divide the work: Searcher retrieves code, Planner selects a workflow, Reproducer reproduces issues, Programmer edits, Tester verifies, and Editor proposes and votes on repairs.Source: §3.1 · p. 5
  • Agent toolsets are assigned selectively rather than granting every tool to every role, limiting responsibilities to improve execution stability and quality.Table 1 is identified as showing tool support for agent tasks.Source: §3.1 · p. 5, Table 1 · p. 5
  • In dynamic repair, Reproducer creates an issue-matching script, Tester runs it, and Programmer iterates on repairs until the Tester confirms resolution.A diff captures the repair when resolved; otherwise test output returns to the Programmer, and the runtime sandbox uses Docker.Source: §3.1 · p. 5
  • For static repair, Editor generates multiple candidates in one LLM request, normalizes them using AST, then merges and votes to select the final solution.Source: §3.1 · p. 5

3.2 Code Indexing

MarsCode Agent indexes repositories using code knowledge graphs, LSP-based navigation, and general project search tools. Together, these capabilities support code-entity retrieval and repository-level analysis.

3.2 Code Indexing

  • The indexing toolset supports multiple code-search requirements and the code knowledge graph covers 12 common programming languages.Supported languages include C, C#, C++, Java, Kotlin, JavaScript, TypeScript, TSX, Rust, Go, Python, and Lua.Source: §3.2 · p. 5, §3.2.1 · p. 7

3.2.1 Code Knowledge Graph

  • The code knowledge graph organizes code, documentation, and repository information as uniquely identified nodes and typed relationships.Nodes represent entities such as variables, functions, classes, and files; attributes record their location, type, and name.Source: §3.2.1 · p. 6
  • Graph retrieval combines entity recognition, embedding similarity, and keyword search, then merges and fine-ranks the candidate lists.The resulting entity list is returned to the agent, supporting repository-level knowledge Q&A.Source: §3.2.1 · p. 6

3.2.2 Language Server Protocol

  • The graph handles many in-project definition and reference searches, but does not accurately retrieve entities outside the target project; LSP supports global retrieval and more precise navigation among same-named entities.The LSP is used for code retrieval on the user’s machine.Source: §3.2.2 · p. 7
  • To form LSP requests despite imprecise agent positions, MarsCode searches for identifiers at or near the given line, or in files the agent has browsed, and uses the first successful response.These request strategies are prioritized in that order.Source: §3.2.2 · pp. 7–8

3.2.3 Other General Indexing Capabilities

  • General project-file and identifier search tools complement the graph and LSP within a consistent code-retrieval toolset.These include find-file and grep capabilities.Source: §3.2.3 · p. 8

3.3 Code Editing

The section describes why several code-editing formats proved unreliable and identifies desired properties for edit descriptions. Static diagnostics compare the original and modified files and return success or further diagnostic information.

3.3.1 Reflections on Code Editing

  • Unified diffs can fail when LLMs miscalculate line numbers, line-range edits can target the wrong span, and full-file rewrites are costly and poorly suited to long files.The authors report that even GPT-4 often gives incorrect modification ranges; full-file rewriting is described as economically unfeasible per edit and nearly unusable for long files.Source: §3.3.1 · p. 8
  • The desired edit descriptions avoid strict format validation and line-number calculations while remaining stably applicable after processing.The section also lists concise descriptions as an approach explored to reduce token and time costs.Source: §3.3.1 · pp. 8–9

3.3.2 Static Code Diagnostics

  • Static diagnostics run LSP checks on original and modified files and compare results for newly introduced errors, especially Fatal and Error levels.The workflow applies the patch to produce modified content before checking both versions.Source: Figure 4 · p. 9, §3.3.2 · p. 9
  • If diagnostics find no new errors, the workflow returns success and the unified diff; otherwise, it sends diagnostic information back for further adjustments.The diagnostic results guide the next response to the agent.Source: §3.3.2 · p. 9

4 Experimental Results and Analysis

MarsCode Agent is evaluated on SWE-bench Lite, a 300-instance subset selected from SWE-bench for clearer, relatively easier resolution. It solves 102 instances (34%) and reports file and segment localization alongside static and dynamic solve outcomes.

4.1 Dataset Overview: SWE-bench Lite

  • SWE-bench Lite selects 300 of SWE-bench’s 2,294 issues from 12 Python repositories for complete descriptions, clear solution logic, and relative ease of resolution.The dataset evaluates agents given a codebase and issue description, with a patch as the goal.Source: §4.1 · pp. 9–10

4.2 MarsCode Agent Results

  • 34% solve rate (102 instances) is MarsCode Agent’s reported result on SWE-bench Lite.The result is presented in Table 2.Source: Table 2 · p. 10, §4.2 · p. 10
  • 88.3% file localization (265/300 instances) exceeds Aide’s 78%, while MarsCode Agent located target segments in 206 instances.The comparison concerns localization; Aide’s reported solve rate is 43%.Source: Figure 5 · p. 10, §4.2 · p. 11
  • Dynamic solving resolved 32 instances at a 38.1% solve rate, versus 70 instances at 32.4% for static solving.The Planner Agent assigned 84/300 instances (28%) to dynamic solving and 216/300 (72%) to static solving; the paper attributes the slightly higher dynamic solve rate to defect reproduction and verification.Source: Figure 7 · p. 11, §4.2 · p. 11

5 Final Remarks

MarsCode Agent combines LLMs with code analysis and achieves a 34% solve rate on SWE-bench Lite. The authors identify cost, collaboration, debugging-setting, and accuracy improvements as future work.

5 Final Remarks

  • MarsCode Agent achieved a 34% solve rate on SWE-bench Lite, which the authors describe as an improvement in solving real-world software engineering problems.The authors frame the result as demonstrating the framework’s performance on the benchmark.Source: §5 · p. 11
  • The framework combines LLM capabilities with code analysis for fault localization, candidate patch generation, and patch validation.Source: §5 · p. 11
  • Future work targets lower LLM call costs, better user-agent collaboration, dynamic debugging in real workspaces, and more accurate localization and code modifications.The stated aim for real workspaces includes avoiding environmental contamination.Source: §5 · p. 11
  • The authors present the SWE-bench Lite results as evidence of LLMs’ potential to advance software engineering.Source: §5 · p. 11
Loading 2409.00899v2…