Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey

1 Introduction

• Issue resolution asks LLMs to navigate complex, multi-file repositories to resolve issues, a challenge highlighted by SWE-bench (Jimenez et al., 2024).Earlier benchmarks such as HumanEval focused on function-level code generation, while realistic development involves dynamic environments and human collaboration.Source: §1 · p. 1
• The literature remains fragmented, and existing surveys focus primarily on code generation rather than the more complex task of issue resolution.The paper presents itself as the first in-depth survey of this domain.Source: §1 · p. 1
• The survey covers 175 publicly available papers and online resources using a tailored classification framework for the rapidly evolving domain.Source: §1 · p. 1
• Its structured taxonomy covers Data, Methods, and Analysis, and the survey identifies key challenges and future directions while providing an open-source repository.Source: §1 · p. 2

• An issue-resolution instance is I = (D, C, T): an issue description, codebase, and corresponding tests, from which a valid patch P is synthesized.Source: §2 · p. 2
• During resolution, the issue description D and codebase C are observable, while supplementary environment information E can be explored.Source: §2 · p. 2
• Evaluation applies patch P to C, runs tests T on C′ = Apply(C, P), and assigns outcome r ∈ {0, 1}.Source: §2 · p. 2

3.1.1 Evaluation datasets

• SWE-bench pairs Python issues with repository snapshots, while SWE-bench Verified (OpenAI, 2024) manually validates a subset to address invalid tests and underspecified descriptions.The verified subset is presented as a trusted benchmark.Source: §3.1.1 · p. 2
• Evaluation datasets broaden scope to ten languages, visual contexts, enterprise settings, incremental development, long-horizon evolution, and efficiency and safety metrics.Multimodal datasets use images such as UI screenshots and diagrams; Yang et al. (2025c) also introduces a visual tester. Enterprise-level datasets address realistic development complexity across domains (Deng et al., 2025a; Tarasova et al., 2025); broader tasks include incremental development (Li et al., 2025c) and long-horizon evolution (Zeng et al., 2025b), while recent metrics include efficiency and safety (He et al., 2025a; Ma et al., 2025a; Xu et al., 2025a).Source: §3.1.1 · p. 2

3.1.2 Training datasets

• Training data ranges from static issue–PR instances to interactive execution environments and tool-use trajectories, with verifiers filtering generated candidates for quality.Multi-SWE-RL (Zan et al., 2025) paired tasks with Conda or Docker environments; R2E-Gym uses LLM-synthesized tests to verify environment usability, while Pan et al. built Gym environments from real-world GitHub issues. Trajectory datasets record tool invocations and feedback loops (Nebius, 2024; Jain et al., 2025; Pan et al.; Pham et al., 2025; Xie et al.; Guo et al., 2025c); inference-time scaling generates candidates that verifiers filter (Nebius, 2024; Yang et al., 2025d).Source: §3.1.2 · pp. 2–3

3.2.1 Automated data collection

• Automated collection builds issue-specific Docker environments and analyzes test results to address static datasets’ rigidity, maintenance costs, and limited scale.Pipelines use LLMs to explore repository configurations, identify setup files, and generate dependency commands; SWE-Factory (Guo et al., 2025c) adds memory-enhanced setup and verification, exit-code grading, and parsers for diverse testing frameworks, while RepoForge (Chen et al., 2025e) uses SPICE-based automatic verification after construction instead of human expert validation.Source: §3.2.1 · pp. 3–4

3.2.2 Automated data synthesis

• Automated synthesis expands and verifies tasks by rewriting code, deriving requirements from tests, paraphrasing descriptions, injecting bugs, or transplanting issues across repositories.SWE-Synth (Pham et al., 2025) rewrites code and generates tests; SWE-Flow (Zhang et al., 2025e) uses a runtime dependency graph to derive incremental code and requirements from unit tests; SWE-Smith (Yang et al., 2025d) and SWE-Mirror (Wang et al., 2025e) use shared environments to scale tasks while reducing storage overhead or producing verifiable tasks.Source: §3.2.2 · p. 4

4.1.1 Frameworks

• Frameworks span tool-using single agents, collaborative agents, and fixed workflows; task graphs and dependency-trace debate make multi-agent coordination more explicit.SWE-agent’s agent-computer interface supports repository navigation, editing, and test execution, while linear localization-repair-validation workflows prioritize stability and reproducibility.Source: §4.1.1 · pp. 4–5

4.1.2 Tool modules

• Tool modules support the repair pipeline from bug reproduction and localization through code search, patch generation, validation, and test generation.Examples include SBFL and dependency graphs for localization, BM25 or AST-based retrieval, AutoDiff editing and regression-tested candidate selection, sandboxed execution or static diagnostics, and feedback-driven test synthesis.Source: §4.1.2 · p. 5

4.1.3 Memory modules

• Memory systems progress from hierarchical or repository-specific storage to context-sensitive episodic and semantic retrieval, with newer work distilling transferable strategies from successful and failed trajectories.Experience banks can guide MCTS, while generalized rules aim to prevent repeated errors.Source: §4.1.3 · p. 6

4.1.4 Inference-time scaling

• Inference-time scaling broadens search through MCTS backtracking and parallel independent state machines, while memory-driven time travel generates experiences for longer-term strategy learning.These approaches expand exploration without retraining model parameters.Source: §4.1.4 · p. 6

4.2.1 SFT-based method

• SFT methods adapt models through synthesized-data scaling, staged curricula from broad trajectories to filtered or specialized tasks, and rejection sampling of successful trajectories.Some approaches also train verifiers to rerank solutions at inference time.Source: §4.2.1 · pp. 6–7

4.2.2 RL-based method

• RL methods combine policy-update algorithms, outcome and process rewards, and rollout scaffolds; GRPO, PPO, and MCTS-integrated DPO are among the approaches described.Rewards range from terminal patch-similarity or subtask verification to dense stepwise or token-level feedback; OpenHands is the most prevalent scaffold, followed by workflow-based methods.Source: §4.2.2 · p. 7

5.1 Data analysis

• Benchmark success rates are often inflated by solution leakage, ambiguous issues, and weak tests, prompting automated model-consensus validation of fixes (OpenAI, 2024; Aleithan et al., 2024; Wang et al., 2025m; Oliva et al., 2025).The shift toward automation reflects the cost and inconsistency of manual cleanup at dataset scale.Source: §5.1 · p. 7

5.2 Methods analysis

• Agents can prioritize prolonged internal deliberation over environmental interaction, causing analysis paralysis and rogue actions (Cuadron et al., 2025).Research examines these behavioral pathologies beyond simple success-rate measurement.Source: §5.2 · pp. 7–8
• Visual interfaces turn massive 128k-token interaction logs into navigable workflows for faster error analysis (Bula et al., 2025).These interfaces streamline trajectory inspection of otherwise cryptic output streams.Source: §5.2 · p. 8

• Detailed discussion of these application scenarios is deferred to § A.5 because of space constraints.Source: §6 · p. 8

Efficiency and computational overhead

• Online RL rollouts and parallel instance validation require costly sandbox resources, while evaluations should pair resolve rates with API-cost and inference-time metrics.The paper calls for lightweight sandboxing and optimized resource scheduling to reduce computational and storage burdens.Source: §7 · p. 8

Visually-grounded reasoning

• Rare multimodal benchmarks and methods that flatten visuals into text hinder evaluation of visually dependent tasks and fail to capture rendering-code alignment; research should build benchmarks and specialized code-centric models (Sun et al., 2025a).Source: §7 · p. 8

Safety risks

• Unsafe agents have deleted users’ codebases (Business Insider, 2025) and cheated during evaluation1, motivating safer frameworks and stronger safety alignment against reward hacking.Source: §7 · p. 8

Fine-grained rewards

• Outcome-level binary test rewards leave credit assignment across multi-turn actions ambiguous, motivating finer-grained process rewards for denser supervision and better policy optimization.Source: §7 · p. 8

Data leakage and contamination

• Unclear training cutoffs risk memorization (Prathifkumar et al., 2025), while ambiguous descriptions, solution hints, and weak test coverage undermine benchmarks (OpenAI, 2024; Mathews and Nagappan, 2025); rigorous curation and decontamination are needed (Zhu et al., 2025b).These measures are intended to restore trust in comparative assessments.Source: §7 · p. 8

Autonomous context management

• Long-horizon, multi-turn interactions increase API costs and degrade performance through context rot, motivating mechanisms that proactively compress and curate interaction history.Source: §7 · p. 8

Patch validation and human review

• Because real-world development lacks gold tests, agents need intrinsic regression-testing and dependency-analysis validation, alongside human-centric explanations and summaries for developer review and acceptance.Source: §7 · pp. 8–9

Universality across SWE domains

• Issue-resolution research largely targets SDLC implementation and integration; broader coverage of requirements analysis and architectural design is needed for more versatile automation.Source: §7 · p. 9

• The survey synthesizes 175 publicly available papers and online resources on LLM-based issue resolution.Source: §8 · p. 9
• Its tailored taxonomy organizes prior work across data, methods, and analysis.Source: §8 · p. 9
• The survey distills challenges and promising directions toward more reliable, reproducible, and practical issue-resolution systems.Source: §8 · p. 9
• Its open-source repository tracks relevant datasets, implementations, and new developments in the field.Source: §8 · p. 9

9 Limitations

• Space constraints led the first dedicated survey to prioritize high-level summaries, and citation tracking and snowballing may miss niche or nascent works despite the commitment to continuously update its repository.The search relied on citation tracking, including of SWE-bench, and snowballing.Source: §9 · p. 9

A.1 Related work

• The survey addresses a gap left by surveys of code generation and other software engineering tasks, covering issue resolution from non-agent approaches through agentic systems.It also reviews realistic code-generation evaluations, multi-agent software generation, and maintenance tools for bug reproduction, fault localization, code search, and testing.Source: §A.1 · pp. 20–21

A.2 Detailed discussions on background

• Issue resolution takes an issue description and pre-fix codebase as input; a patch is successful when it applies and all tests pass, with performance measured by Resolve Rate.Tests include fail-to-pass checks for the fix and pass-to-pass checks for regressions, but are not shown to the model during patch generation.Source: §A.2 · p. 21
• Issue-resolution datasets are built by scraping repositories and filtering merged, issue-linked pull requests with test changes, then checking execution viability and corrective test transitions.The pipeline uses CI workflows to construct environments, removes tasks that fail installation or runtime checks, and retains examples with a fail-to-pass transition.Source: §A.2 · pp. 21–22
• The initial dataset pipeline was manual-heavy and vulnerable to contamination, prompting work on more dynamic data collection and synthetic training data.Manual verification checks issue clarity, task completeness, and benchmark suitability.Source: §A.2 · pp. 22–23

A.3 Detailed discussions on data

• Issue-resolution benchmarks have expanded beyond Python to additional languages and visual contexts, while large datasets reach 115k instances and increasingly provide reproducible Docker environments.Multilingual datasets include Java, C++, Go, Rust, JavaScript, and TypeScript; multimodal benchmarks incorporate visual data for frontend-related tasks.Source: §A.3 · p. 23

A.4 Detailed discussions on training methods

• Specialized models increasingly use reinforcement learning with test-based outcome rewards, with process rewards helping stabilize smaller models on long-horizon tasks.RL is often applied atop established agent frameworks; general-purpose models instead rely on inference scaffolds, providing a comparison for gains attributable to SFT and RL pipelines.Source: §A.4 · pp. 23–24

A.5 Detailed discussions on applications

• Enterprise adoption of developer assistants has been associated with productivity increases of up to 55%, while autonomous agents reportedly achieved 12x engineering efficiency in a Nubank migration.The first figure concerns AI pair programmers; the second concerns agents handling complex multi-step tasks in sandboxed environments.Source: §A.5 · p. 24
• Applications progress from issue-to-pull-request workflow automation to autonomous multi-step engineering and enterprise integration, where MCP-based collaboration faces context-overhead and scalability challenges.Stage 2 agents analyze issues and codebases, make changes, and submit pull requests; ecosystem tools emphasize security governance and protocol-level efficiency optimizations.Source: §A.5 · p. 24