Source-linked AI summary
CodeR: Issue Resolving with Multi-Agent and Task Graphs
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, Qianxiang Wang
TL;DR
SWE-bench issue resolving remains highly non-trivial because it requires deep reasoning over large repositories and incomplete issue descriptions, while prior approaches primarily focus on single agents. CODER combines a multi-agent framework, pre-defined task graphs, role-specific actions, LLM-generated reproduction tests, coverage-guided retrieval, and fault localization. CODER achieved 28.33% on SWE-bench lite at one attempt, addressing 84 of 300 issues.
Problem
SWE-bench issue resolving remains highly non-trivial because it requires deep reasoning over large repositories and incomplete issue descriptions, while prior approaches primarily focus on single agents.
Source: §1 · p. 1Method
CODER combines a multi-agent framework, pre-defined task graphs, role-specific actions, LLM-generated reproduction tests, coverage-guided retrieval, and fault localization.
Source: §1 · p. 2, §2 · p. 2, §3.2 · p. 5Results
CODER achieved 28.33% on SWE-bench lite at one attempt, addressing 84 of 300 issues.
Source: §4.2 · p. 8Takeaways & Limitations
Task graphs convert planning into a simpler decision task for LLMs, ensure exact plan execution, and allow additional software-engineering skills to be added through JSON-defined plans.
Source: §6 · p. 10
Abstract
from arXiv · showhide
GitHub issue resolving recently has attracted significant attention from academia and industry. SWE-bench is proposed to measure the performance in resolving issues. In this paper, we propose CodeR, which adopts a multi-agent framework and pre-defined task graphs to Repair & Resolve reported bugs and add new features within code Repository. On SWE-bench lite, CodeR is able to solve 28.33% of issues, when submitting only once for each issue. We examine the performance impact of each design of CodeR and offer insights to advance this research direction.
1 Introduction
CodeR addresses the difficulty of repository-level issue resolving by combining multiple specialized agents with pre-defined task graphs. On SWE-bench lite, it achieves a 28.33% one-attempt resolved rate.
Motivation
- Issue resolving remains challenging because SWE-bench requires deep reasoning over large repositories and incomplete issue descriptions.SWE-bench-lite removes low-quality descriptions but remains highly non-trivial.Source: §1 · p. 1
- Prior issue-resolving systems primarily use single agents or dynamically determine subsequent steps, whereas CODER explicitly separates roles and pre-defines execution plans.The paper identifies free-agent communication, information loss, and complex multi-agent plans as implementation challenges.Source: §1 · p. 1
Design rationale
- CODER limits each agent to a subtask and associated actions, pre-plans manageable tasks, and uses task graphs instead of plain-text plans.The design aims to reduce decision complexity and avoid reliance on imperfect instruction following and long-context memorization.Source: §1 · p. 2
Results
- The proposed CODER multi-agent framework renewed the state-of-the-art on SWE-bench lite to 28.33% (85/300), with only one submission per issue.Source: §1 · p. 2
Contribution
- CODER introduces a multi-agent framework with task graphs to resolve reported bugs and add repository features.Its graph plans assign roles and actions that can be parsed and executed strictly, while allowing human plan injection.Source: §1 · p. 2
- CODER also uses LLM-generated reproduction tests and repository tests to obtain coverage information for contextual retrieval and fault localization with BM25.This is presented as a second contribution alongside the multi-agent framework and benchmark result.Source: §1 · p. 2
2 Framework
CODER organizes issue resolution around five collaborating agents with distinct responsibilities and role-specific actions. Its task graphs coordinate plans, downstream summaries, testing, and repository edits.
Agent roles
- CODER uses five agents—Manager, Reproducer, Fault Localizer, Editor, and Verifier—to collaboratively resolve GitHub issues.The Manager selects plans and submits patches, while the other agents reproduce, localize, edit, and verify solutions.Source: §2 · p. 2
- The Reproducer creates or adapts tests from issue information, while the Fault Localizer identifies potentially responsible code regions.The Reproducer may extract test inputs and use LLMs to generate test sequences; fault localization uses specialized tools.Source: §2 · p. 2
- The Editor uses upstream information and contextual search before performing iterative repository edits, and the Verifier runs reproduced or integration tests.Integration tests are built-in repository unit tests rather than official SWE-bench issue-verification tests.Source: §2 · p. 2
Actions
- CODER adds actions for plan selection, similar-issue retrieval, fault localization, test execution, action summarization, and basic shell commands.These actions complement reused actions from SWE-agent and AutoCodeRover.Source: §2 · pp. 2–3
Action permissions
- Each role receives a distinct action set, while all roles can use basic shell commands; the Manager alone can plan and submit.This assigns permissions according to role-specific responsibilities.Source: §2 · p. 3
3 Methodology
CodeR addresses repository-level issue resolving with pre-defined, strictly executable task graphs and specialized multi-agent roles. Its methodology combines flexible planning, coverage-guided fault localization, and role-specific prompts for handling diverse issue complexity.
3.1 Task Graphs for Planning
- Pre-defined task graphs decompose issue resolving into executable subtasks, allowing plans to be added, removed, or tuned without changing agent code.Each plan specifies an entry agent, participating roles, subtasks, and downstream transitions; execution ends when the Manager is activated or the budget is exceeded.Source: §3.1 · pp. 4–5
- Four plans support different issue-solving strategies, but experiments use only the loop-free standard flow and direct-resolution plan for cost savings and fast evaluation.The other plans add testing feedback or test-driven execution with ground-truth tests.Source: §3.1 · p. 5
3.2 Fault Localization Specialized for Issue Resolving
- SBFL depends on failing tests and is unavailable when reproduction or coverage collection fails, so CodeR uses reproducer-generated tests when possible.The filtering process leaves 140 reproducible issues and 104 issues with non-empty coverage data.Source: §3.2 · pp. 5–6
- Fault localization combines issue-description retrieval with spectrum-based fault localization because their distinct information sources can complement one another.The resulting suspiciousness score linearly combines BM25 relevance with test-coverage-based scores.Source: §3.2 · pp. 5–6
- Combining BM25 with SBFL improves fault-localization precision by more than 10%, while method-level localization balances agent-useful location information with good precision.The authors select λ = 0.99 as the final setup after testing the combination factor on a small reproduced-issue subset.Source: §3.2 · p. 6
3.3 Prompt Engineering
- CodeR’s five roles—manager, reproducer, fault localizer, editor, and verifier—separate planning, reproduction, localization, modification, and verification responsibilities.The Manager selects plans and interprets execution summaries, while downstream agents perform specialized issue-resolution subtasks.Source: §3.3 · p. 7
- Role-specific system and instance prompts configure CodeR’s five agents and their inter-agent communication, following designs inspired by SWE-agent [8].The system prompts define role identities, responsibilities, and actions, while instance prompts include the raw issue and resolution guidance.Source: §3.3 · p. 7
4 Experiments
Experiments evaluate CODER on SWE-bench lite using standardized metrics and implementation settings, then compare it with existing methods and analyze its design through ablations. CODER achieves the strongest reported one-attempt result, while performance and cost vary with patch-generation strategy, planning, multi-agent structure, and fault localization.
4.1 Experimental Setup
- SWE-bench lite contains 300 self-contained instances focused on functional bug fixes, and evaluation reports resolved rate, average requests, and average tokens/cost.The study focuses on SWE-bench lite for faster, easier, and more cost-effective evaluation.Source: §4.1 · p. 7
4.1.1 Implementation Details
- CODER uses GPT4-preview-1106 across distinct agent roles, role-specific history windows, and an $8 maximum cost per issue.The Manager uses full history, while several other roles use only the last five histories.Source: §4.1.1 · p. 8
4.2 Results
- 28.33% of SWE-bench lite issues are resolved by CODER in one attempt, exceeding Aider at 26.33% and SWE-agent + GPT-4 at 18.00%.The comparison covers 300 issues and is reported as a new benchmark record.Source: §4.2 · p. 8, Table 4 · p. 9
- Implicit patch generation outperforms direct patch generation: CODER resolves 28.33%, versus 19.00% for AutoCodeRover and 2.67% for RAG+GPT-4.The paper attributes the difficulty of explicit generation to strict patch formatting and sensitivity to line numbers.Source: §4.2 · p. 8
- CODER’s pre-planning improves the reported result by 10.33% over SWE-agent + GPT-4, while increasing requests, tokens, and cost at an acceptable rate.Plans are represented as structured graphs and executed strictly by the agent roles.Source: §4.2 · p. 8
- Removing the multi-agent task graph reduces resolved rate from 22% to 10% on 50 issues, while removing fault localization causes a performance drop and cost increase.These ablations test the contribution of the framework components on a randomly selected subset.Source: §4.2 · pp. 8–9, Table 5 · p. 9
5 Related Works
Related work spans retrieval and editing pipelines, test-based automated program repair, and general multi-agent systems. CODER combines multi-turn information collection with fine-grained, decoupled roles and actions for repository-level issue resolving.
Automatic Issue Resolving
Test-based Automated Program Repair
- Test-based automated program repair validates generated patches against tests but can be limited by weak test suites, large search spaces, and omission of issue descriptions.Prior methods guide search using program structure, heuristics, learning, synthesis, or semantic analysis.Source: §5 · p. 10
- CODER uses a multi-turn framework to collect necessary information on demand before generating fixed code.This distinguishes its issue-resolving process from approaches that treat repair primarily as direct code-to-code translation.Source: §5 · p. 10
Artificial Intelligence (AI) Agents
- CODER defines decoupled roles such as reproducer, programmer, and tester, each paired with fine-grained actions for collaborative issue resolution.The design extends agent-based automation toward specialized repository-level interactions.Source: §5 · p. 10
6 Conclusion and Future Works
CODER concludes that human-inspired plans represented as executable task graphs are central to its issue-resolving performance. Future work will expand the plan library and add advanced software-engineering skills through the graph interface.
6 Conclusion and Future Works
- CODER’s pre-specified task graphs simplify planning into decisions for LLMs and guarantee exact execution of the selected plan.Human experts provide the predefined plans that guide issue resolution.Source: §6 · p. 10
- Task graphs allow capabilities such as fault localization, similar-issue mining, and web search to be added through JSON without code changes.The graph interface is presented as a way to incorporate advanced software-engineering skills.Source: §6 · p. 10
- Future work will build a more comprehensive set of plans to resolve more issues.The paper also suggests that more sophisticated plans could yield further improvements.Source: §6 · p. 10