Source-linked AI summary
MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, Yu Cheng
TL;DR
GitHub issue resolution is difficult because repository-level changes must preserve existing functionality while LLMs struggle with long contexts and complex localization. The paper analyzes these factors and proposes MAGIS, a four-agent framework for planning, repository navigation, coding, and review. On SWE-bench, MAGIS achieves a 13.94% resolved ratio and an eight-fold gain over direct GPT-4 application.
Problem
Over 95% of SWE-bench instances remain unresolved by LLMs even when required file paths are provided, highlighting the difficulty of repository-level GitHub issue resolution.
Method
MAGIS uses collaborating Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents, with repository-memory support for locating and understanding files.
Results
13.94% resolved ratio gives MAGIS an eight-fold performance gain over direct GPT-4 application on SWE-bench.
Takeaways & Limitations
MAGIS’s superiority on SWE-bench points toward multi-agent collaboration as a promising direction for integrating LLMs into software evolution workflows.
Takeaways & Limitations
SWE-bench covers 12 Python repositories and cannot fully reflect the diversity of GitHub software, including specialized fields and programming paradigms.
Abstract
from arXiv · showhide
In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code. Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving Github issues, particularly at the repository level. To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. Motivated by the empirical findings, we propose a novel LLM-based Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. This framework leverages the collaboration of various agents in the planning and coding process to unlock the potential of LLMs to resolve GitHub issues. In experiments, we employ the SWE-bench benchmark to compare MAGIS with popular LLMs, including GPT-3.5, GPT-4, and Claude-2. MAGIS can resolve 13.94% GitHub issues, significantly outperforming the baselines. Specifically, MAGIS achieves an eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.
1 Introduction
GitHub issue resolution requires repository-wide code changes while preserving existing functionality, exposing limitations in direct LLM application. MAGIS addresses these challenges through coordinated software-evolution agents and outperforms evaluated LLM baselines on SWE-bench.
- GitHub issues require repository-wide code changes that integrate new capabilities while maintaining existing functionality.The paper illustrates the scale of this challenge with Django, which supports over 1.6M projects and has 34K issues.
- LLMs remain constrained by long repository contexts, making direct use of an entire repository difficult for issue resolution.
- Over 95% of SWE-bench instances remain unresolved by LLMs even when required file paths are provided.
- MAGIS coordinates Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents for repository-level issue resolution.The agents respectively coordinate the process, locate files, implement changes, and review code changes.
2 Empirical Study
The empirical study examines why direct LLM use struggles with GitHub issue resolution, focusing on file and line localization and code-change complexity. It finds that accurate line localization is positively related to resolution, while greater change complexity tends to hinder it.
- GPT-4 resolves less than 2% of SWE-bench issues, despite scoring 67.0 on HumanEval for function-level code generation.
- Locating the Files to be Modified: Selecting more files can reduce Claude-2’s resolved ratio from 1.96 to 1.22 as recall increases from 29.58 to 51.06.The paper attributes this decline to irrelevant files or limited capacity for processing longer contexts.
- Locating the Lines to be Modified: Line-localization coverage estimates how accurately generated modified-line ranges overlap reference modified-line ranges across hunks.The coverage ratio uses the intersection length in the numerator and the number of modified reference lines in the denominator.
- Locating the Lines to be Modified: 0.5997 coefficient on Claude-2 indicates a statistically significant positive relation between coverage-ratio improvements and issue-resolution probability.The reported relationship has P-value < 0.05 and identifies line localization as a key factor.
- Complexity of the Code Changes: Negative correlations for modified files and functions indicate that increasing these complexity indices is associated with lower issue-resolution likelihood.The study reports statistically significant correlations across several complexity indices for all three LLMs.
3 Methodology
MAGIS transforms GitHub issue resolution into a collaborative workflow involving specialized agents for repository exploration, planning, coding, and review. Its planning process narrows relevant files, decomposes issues into tasks, builds flexible Developer teams, and coordinates dependencies before execution.
- Agent Role Design: MAGIS assigns four specialized roles—Manager, Repository Custodian, Developer, and QA Engineer—to collaborate on GitHub issue resolution.The framework is inspired by collaborative software workflows and is tailored to software evolution.
- Planning: The workflow comprises locating code files, team building, and a kick-off meeting before collaborative coding begins.Three role agents participate in planning: the Repository Custodian, Manager, and Developer.
- Planning: The Repository Custodian locates relevant files by ranking repository files with BM25, then filtering candidates using issue relevance.The framework maintains repository evolution memory to reuse summaries and compact descriptions of file changes.
- Planning: The Manager decomposes an issue into file-level tasks and recruits Developer agents with roles tailored to those tasks.Each Developer handles a sub-task rather than the entire issue, simplifying the repository-level modification process.
- Planning: A kick-off meeting validates task assignments, identifies parallelizable work and dependencies, and produces an executable main work plan.The meeting uses circular discussion among the Manager and Developers to reduce conflicts and improve coordination.
4 Experiments and Analysis
MAGIS is evaluated on a SWE-bench subset using GPT-4 as its base model and compared with existing LLM baselines. The analysis examines overall resolution, component contributions, planning quality, line localization, code-change complexity, and QA effects.
- 4.1 Setup: SWE-bench contains 2,294 issues from 12 popular Python repositories, and the study evaluates MAGIS on the previously used 25% subset.GPT-4 is selected as the base model for comparison with reported SWE-bench results.
- 4.2 How Effective is Our Framework? (RQ 2): 13.94% resolved ratio makes MAGIS eight-fold more effective than GPT-4 and more than two-fold better than Claude-2 on SWE-bench.The comparison uses the same dataset and reports the overall performance in Table 2.
- 4.2 How Effective is Our Framework? (RQ 2): 8.71 resolved ratio remains five times higher than the base LLM without QA or hints, while adding QA or hints separately raises it by 1.92 or 1.57.The ablation studies bypass QA or remove pull-request comment hints to assess their contributions.
- 4.3 How Effective is Our Planning Process? (RQ 3): MAGIS consistently outperforms BM25 in recall across selected-file counts, identifying more relevant files with fewer selections.The Repository Custodian’s recall-versus-file-number curve evaluates this planning component.
- 4.3 How Effective is Our Planning Process? (RQ 3): Most Manager-generated task-description correlation scores are 3 or above, indicating alignment with the reference code change.GPT-4 evaluates the correlation using criteria defined in Table 6.
- 4.4 How Effective is Our Coding Process? (RQ 4): MAGIS more frequently achieves line locating coverage near 1 than baselines, and resolved ratios increase across higher coverage intervals.The analysis states that accurately locating lines is important for issue resolution, with larger benefits at higher coverage.
- 4.4 How Effective is Our Coding Process? (RQ 4): GPT-4 shows significant negative correlations between issue resolution and the numbers of modified files and functions, while MAGIS exhibits weaker negative correlations.The study uses logistic regression to relate complexity indices to resolution.
- 4.5 How Effective is Our QA Engineer? (RQ 5): The QA Engineer increases resolved ratio by 1.57% with hints and 3.31% without hints.The ablation results substantiate the QA Engineer’s contribution to outcomes.
5 Related Work
Prior work has developed LLM-based multi-agent systems for complex task completion and automated programming. This literature also includes methods for bug localization and automatic program repair, while GitHub issues commonly involve bug fixing.
- LLM-based Multi-Agent Systems: MetaGPT simulates software teams’ Standardized Operating Procedures and reports leading HumanEval and MBPP scores.It represents one example of LLM-based multi-agent systems for complex programming tasks.
- LLM-based Multi-Agent Systems: ChatDev decomposes requirements into atomic tasks and uses mutual communication and self-reflection in a virtual development company.These mechanisms are presented as ways to mitigate limitations in LLM-based software development.
- Automated Program Repair: Other research addresses GitHub issue components through bug localization and automatic program repair methods.Most GitHub issues are described as requirements, with many belonging to bug fixing.
6 Conclusion
The paper studies why direct LLM application struggles with repository-level GitHub issue resolution and proposes MAGIS, a collaborative multi-agent framework. MAGIS is evaluated on SWE-bench, with the supplied passages emphasizing its effectiveness and code-line locating challenges.
- MAGIS uses collaboration among well-designed agents to address challenges in direct LLM-based GitHub issue resolution.The framework is presented as an LLM-based multi-agent approach for software development.
- Coverage ratio compares the line ranges changed by generated code with the line ranges in the reference change.Reference and generated modifications are represented as sets of line intervals across hunks.
- Low coverage ratios dominate the three evaluated LLMs, indicating difficulty locating the code lines requiring modification.The observation concerns coverage of generated changes against reference changes in terms of code-line location.
A.3 Analysis on Complexity of the Code Change
The complexity analysis contrasts how different LLMs respond to code-change complexity and illustrates a multi-agent meeting used to organize issue resolution. Claude-2 shows a distinct correlation pattern from GPT-3.5 and GPT-4.
- A.3 Analysis on Complexity of the Code Change: Claude-2 has much lower negative correlations for file and function counts than GPT-3.5 and GPT-4.The passage interprets this pattern as indicating a more efficient approach to generating code changes.
- A.3 Analysis on Complexity of the Code Change: Claude-2 still shows significant negative correlations for hunks, added LoC, deleted LoC, and changed LoC.Thus, its distinct pattern is limited to the reported file and function indices.
- A.3 Analysis on Complexity of the Code Change: The kick-off meeting assigns a specific issue, discusses tasks, and determines the workflow sequence among the Manager and Developer agents.The depicted participants include Oliver CodeLead, Django Database Specialist, and Alex Rossini.
C Metrics
The metrics evaluate whether generated changes can be applied and pass tests, while also measuring how effectively the framework locates files and code lines. File-number curves connect recall with the average search effort.
- Applied and resolved ratios: Applied ratio measures the proportion of generated code changes that can be applied to the existing repository using Git tools.The corresponding set D contains changes successfully applied with “git apply,” while I contains all test instances.
- Applied and resolved ratios: Resolved ratio measures changes that are successfully applied and pass the relevant test cases.The test definition includes cases passed by the old repository and tests designed for new requirements.
- File locating metrics: File-locating recall is the proportion of required files that are successfully located.The metric is defined for each issue-resolution instance using located file paths and reference paths requiring modification.
- File locating metrics: The file-number curve reports the average number of files that must be located before reaching a given recall score.It represents both locating effectiveness and the processing effort across instances.
D Comparison Result on SWE-bench Lite
MAGIS is compared with contemporary methods on SWE-bench lite, a recommended canonical subset used here under budget constraints. It achieves the highest reported resolved ratio, while ablations show effects from QA and hints.
- Overall comparison: SWE-bench lite provides the common evaluation set for comparison with AutoCodeRover and SWE-Agent under budget constraints.The subset is described as a canonical, recommended portion of SWE-bench.
- Overall comparison: 25.33% resolved ratio is achieved by MAGIS, the highest value among the reported SWE-bench lite baselines.The comparison is presented in Table 4.
- Ablation comparison: 23.33% resolved ratio is obtained without QA, slightly below MAGIS’s full configuration.Performance drops further under the other two ablation settings.
- Ablation comparison: The results highlight the impact of QA and hints on MAGIS performance.The passage connects these components with the observed ablation differences.
E Comparison with Devin
On 140 shared SWE-bench instances, MAGIS resolved more issues than Devin and completed resolved cases substantially faster, although the comparison settings were not fully equivalent.
- 21 issues (15%) were resolved by MAGIS, compared with 18 (12.86%) resolved by Devin within the shared pool of 140 instances.The comparison is qualified because Devin’s underlying LLM and tool access differed from MAGIS’s setting.
- MAGIS relied solely on the shell, while Devin used the entire repository, common developer tools, environmental feedback, and potentially internet access.These differences make the resolution-rate comparison not entirely equitable.
- 72% of Devin’s resolved instances required more than 10 minutes, whereas MAGIS finalized each resolved issue in approximately 3 minutes.MAGIS processed each instance in under 5 minutes on average.
F Statistics on the Generated Code Changes
The generated changes for resolved issues were close in structural complexity to human reference solutions, while unresolved cases involved larger and more difficult modifications. Unresolved outputs also showed a tendency toward deleting more lines and varied substantially across repositories.
- Resolved instances: Resolved-instance code changes differed from human reference solutions by no more than 0.3 on average for files, functions, hunks, and deleted lines.This indicates similar code-change complexity for resolved instances.
- Resolved instances: Added lines were especially higher than the reference distribution in resolved instances, often because generated changes contained more comments.The modified-line distribution is summarized in Fig. 8.
- Applied but not resolved instances: Unresolved instances produced applicable changes spanning up to 13 files, 28 hunks, line 7,150, and a single modification of 9,367 lines.These changes were applicable but did not pass all potential tests.
- Applied but not resolved instances: Unresolved instances tended to delete more lines and add fewer lines than human-written changes, suggesting different repair strategies.The observed pattern may reflect a preference for removing potentially problematic code rather than adding new code.
- Resolved versus unresolved instances: Unresolved cases contained more files, hunks, and changed lines than resolved cases, indicating that more complex modification scenarios remain challenging.The paper states that performance on these complex issues requires further enhancement.
- Repository-level statistics: Resolved ratios varied significantly across repositories, ranging from as high as 40% in some repositories to nearly 0% in others.The paper attributes this variation to differences in repository difficulty and summarizes it in Fig. 13.
K Limitation
The study identifies prompt design and dataset diversity as limitations affecting the validity, fairness, and general applicability of its evaluation.
- Prompt design can affect LLM performance and the validity and fairness of the results, while completely eliminating prompt bias is difficult.The paper links this difficulty to dataset-instance biases and API-resource limitations.
- SWE-bench covers 12 Python repositories, which may not represent the diversity of software projects, architectures, or programming paradigms on GitHub.Applying the findings to other repositories may require further validation.