Source-linked AI summary
Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey
Caihua Li, Lianghong Guo, Yanlin Wang, Daya Guo, Wei Tao, Zhenyu Shan, Mingwei Liu, Jiachi Chen, Haoyu Song, Duyu Tang, Hongyu Zhang, Zibin Zheng
TL;DR
Issue resolution is a difficult repository-level software-engineering task, and existing surveys have not adequately covered its fragmented literature. This paper surveys 175 papers and resources through a taxonomy of data, methods, and analysis, identifying challenges and directions while maintaining an open-source repository. Its synthesis emphasizes the need for more reliable, reproducible, and practical issue-resolution systems, but the review may miss niche or nascent work because it relies on citation tracking and snowballing.
Problem
Issue resolution remains a difficult repository-level task, while prior surveys primarily focus on code generation rather than this domain.
Method
The paper conducts a systematic survey of 175 papers and online resources using a taxonomy organized around data, methods, and analysis.
Results
The survey synthesizes issue-resolution data construction, methodologies, analyses, applications, challenges, and future directions in a structured overview.
Takeaways & Limitations
The paper provides a maintained open-source repository tracking datasets, implementations, and developments to support continued research.
Takeaways & Limitations
Citation tracking and snowballing may overlook niche or nascent works, and the survey prioritizes high-level summaries over exhaustive detail.
Abstract
from arXiv · showhide
Issue resolution, a complex Software Engineering (SWE) task integral to real-world development, has emerged as a compelling challenge for artificial intelligence. The establishment of benchmarks like SWE-bench revealed this task as profoundly difficult for large language models, thereby significantly accelerating the evolution of autonomous coding agents. This paper presents a systematic survey of this emerging domain. We begin by examining data construction pipelines, covering automated collection and synthesis approaches. We then provide a comprehensive analysis of methodologies, spanning training-free frameworks with their modular components to training-based techniques, including supervised fine-tuning and reinforcement learning. Subsequently, we discuss critical analyses of data quality and agent behavior, alongside practical applications. Finally, we identify key challenges and outline promising directions for future research. An open-source repository is maintained at https://github.com/DeepSoftwareAnalytics/Awesome-Issue-Resolution to serve as a dynamic resource in this field.
1 Introduction
Issue resolution extends software engineering automation from function-level generation to navigating and modifying complex, multi-file repositories. This survey addresses the fragmented literature with a structured synthesis of data, methods, and analysis.
- The task responds to LLM limitations in dynamic development-environment interaction and human collaboration.
- SWE-bench defined issue resolution as automatically navigating multi-file repositories to produce changes that resolve reported issues.
- Existing surveys largely focus on code generation rather than the more complex issue-resolution problem.
- The survey analyzes 175 papers and online resources using a taxonomy organized around data, methods, and analysis.
- It also identifies challenges and future directions and maintains an open-source repository for the community.
2 Task formulation
Issue resolution models a method that uses an issue description, codebase, and available environment information to synthesize a patch. The patch is applied to the codebase and evaluated by executing the corresponding tests.
- An issue-resolution instance comprises an issue description D, codebase C, and corresponding tests T.
- The method synthesizes a patch P using the observable issue description and codebase while exploring supplementary environment information E.
- Evaluation applies P to C, executes T on the modified codebase C′, and records a binary resolution outcome.
- Overall performance is computed as the average resolution outcome across dataset instances.
3 Data
Issue-resolution data serves both evaluation and training, spanning collected real-world instances and synthesized examples. The field is moving toward interactive environments, trajectories, and automated pipelines to improve scale and realism.
- Datasets are classified into evaluation and training sets, with construction based on real-world collection or data synthesis.
- SWE-bench pairs issues with pull requests and repository snapshots, but invalid tests and underspecified descriptions motivated SWE-bench Verified.
- Evaluation datasets increasingly cover multiple programming languages, multimodal information, enterprise complexity, and broader software-engineering scenarios.
- Training data includes static issue–PR instances, interactive execution environments, and trajectories containing tool calls and feedback loops.
- Automated collection pipelines configure repositories, build Docker environments, execute tests, and parse results across workflow-based and agent-based systems.
- Automated synthesis rewrites code, derives requirements from tests, paraphrases descriptions, and injects bugs to expand training data while reducing verification or storage costs.
4 Methods
LLM-based issue-resolution methods combine frameworks, tool modules, memory, and inference-time scaling, alongside training-based approaches. Frameworks range from autonomous agents and multi-agent collaboration to predefined workflows, while tools support the repair pipeline.
- Training-free methods: Training-free methods are organized into frameworks, plug-and-play modules, and inference-time scaling strategies.
- Frameworks: Single-agent systems use tool interaction for repository navigation, code editing, and test execution, but unrestricted autonomy can create redundant actions and high costs.
- Frameworks: Multi-agent frameworks allocate roles and tasks, while graph-based structures address limited explicit modeling of collaboration in text-based exchanges.
- Frameworks: Workflow architectures enforce predefined steps to improve stability, efficiency, and reproducibility for issue-resolution processes.
- Tool modules: Specialized tools support bug reproduction, fault localization, code search, patch generation, patch validation, and test generation.
- Modules: Memory modules retain historical context through hierarchical or episodic-semantic storage, enabling retrieval of repository knowledge and transferable repair strategies.
- Inference-time scaling: Inference-time scaling expands exploration beyond linear execution paths, addressing the limited coverage of complex solution spaces.
5 Analysis
Analysis research complements methodology development by examining existing data and methods, exposing limitations and informing future directions. Recent work also investigates benchmark validity and agent behavioral pathologies.
- Empirical analysis of existing data and methods reveals limitations and informs future research directions.
- Benchmark defects can inflate agent success rates through solution leakage, ambiguous issues, and weak test suites.
- Behavioral studies examine excessive internal deliberation, which can produce analysis paralysis and rogue actions, while visual interfaces streamline long trajectory inspection.
6 Application
Software-engineering AI applications have progressed from localized IDE assistance to autonomous systems handling complex enterprise workflows.
- Software-engineering AI has progressed from localized IDE assistance to fully autonomous systems handling complex enterprise workflows.
7 Challenges and Opportunities
The survey identifies challenges spanning computational cost, evaluation quality, scope, safety, reward design, context management, and patch validation. It proposes broader, more reliable, efficient, and human-centered issue-resolution systems.
- Concurrent rollouts and parallel validation impose substantial storage and computational costs, motivating lightweight sandboxing and optimized resource scheduling.
- Evaluations emphasizing resolve rates can obscure API costs and inference time, motivating joint effectiveness-and-efficiency assessment.
- Current benchmarks rarely evaluate visually dependent tasks, motivating multimodal benchmarks and specialized code-centric models.
- Unsafe agent behaviors, including codebase deletion and evaluation cheating, motivate safer frameworks and stronger safety alignment.
- Outcome-level binary rewards make credit assignment ambiguous across multi-turn interactions, motivating finer-grained process rewards.
- Evaluation reliability is threatened by leakage, ambiguous descriptions, solution hints, and insufficient test coverage, while context rot raises costs and degrades performance.
- Because real-world development lacks gold tests, agents need intrinsic validation and human-centric interfaces for reviewing generated solutions.
- Issue-resolution research remains concentrated on implementation and integration, leaving requirements analysis and architectural design insufficiently addressed.
8 Conclusion
The paper surveys LLM-based issue resolution through a structured synthesis of 175 papers and online resources. It organizes the field around data, methods, and analysis, identifies future directions, and maintains an open-source repository.
- The survey synthesizes 175 publicly available papers and online resources on LLM-based issue resolution.
- A tailored taxonomy structures prior work along the dimensions of data, methods, and analysis.
- The paper distills challenges, outlines directions toward reliable and practical systems, and maintains a repository tracking datasets, implementations, and developments.
9 Limitations
The survey favors high-level coverage over exhaustive detail and acknowledges that its citation-tracking and snowballing search may miss niche or emerging work.
- The survey prioritizes high-level summaries over exhaustive details because of space constraints.
- Its citation-tracking and snowballing methodology was thorough but may overlook niche or nascent works.
- The authors plan to continuously update their open-source repository as the field evolves rapidly.
A Appendix
The appendix includes a 2025 preprint on best practices for building rigorous agentic benchmarks.
- The cited work addresses best practices for building rigorous agentic benchmarks.
- The work is authored by Sekhon, Jacob Steinhardt, and six others.
- The reference identifies the work as a 2025b arXiv preprint, arXiv:2507.02825.
A.1 Related work
Related work spans code generation, automated software construction and maintenance, environment setup, and surveys dedicated to issue resolution. These strands provide benchmarks, tools, and conceptual coverage for repository-level engineering.
- Code generation: Code-generation research began with function-level benchmarks such as HumanEval, while newer initiatives aim to capture real-world development nuances.
- Automated software generation: Automated software generation constructs executable systems from natural-language requirements across the software development life cycle.
- Automated software maintenance: Automated maintenance contributes bug reproduction, fault localization, code search, and test generation tools to issue-resolution agents.
- Automated software maintenance: These maintenance tools provide precise error locations, relevant code context, and verification mechanisms for resolution.
- Automated environment setup: Automated environment-setup research configures runtime environments for entire repositories and develops in parallel with issue-resolution data construction.
- Related surveys: Existing surveys focus mainly on code generation or other software-engineering tasks, whereas this paper surveys the full issue-resolution spectrum.
A.2 Detailed discussions on background
Issue resolution benchmarks formalize repository-level repair as producing and testing patches from issue descriptions and codebases. The survey organizes datasets, methods, and analyses while tracing expansion toward broader languages, modalities, training strategies, and autonomous applications.
- Task definition: Issue resolution represents each task as an issue description D, codebase C, and tests T, with the model generating a patch P from D and C.The tests remain hidden during generation; evaluation applies P to C and executes T on the resulting codebase C′.
- Task definition: Evaluation applies the proposed patch to C and counts a task as resolved only when the patch applies correctly and all tests pass.The benchmark reports Resolve Rate as the percentage of successfully resolved tasks.
- Dataset construction: Data construction proceeds through repository selection, attribute filtering, execution-based validation, and manual verification.Execution checks use reproducible build and test environments, while manual inspection assesses clarity, self-containment, and suitability.
- Dataset construction: The initial SWE-bench pipeline produced 2,294 instances but faced manual-heavy environment setup, data contamination, limited scalability, and limited dynamism.Later work therefore emphasizes dynamic data collection and synthetic dataset generation.
- Dataset scope: Issue-resolution datasets increasingly broaden evaluation across programming languages, modalities, repositories, scale, and reproducible execution environments.Multilingual benchmarks cover languages including Java, C++, Go, Rust, JavaScript, and TypeScript, while multimodal datasets add visual context.
- Training and applications: Reinforcement-learning approaches combine policy-update algorithms, reward design, and rollout scaffolds, with process rewards helping stabilize smaller models during long-horizon debugging.The survey also connects specialized models to established agent frameworks and reports SFT and RL pipelines as performance-oriented approaches.
- Training and applications: Applications span developer augmentation, autonomous multi-step engineering, and ecosystem integration with security governance and interoperable multi-agent tooling.Reported examples include up to 55% productivity improvement for AI assistants and a 12x efficiency improvement during a Nubank migration.