PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model-Based Agents for Software Engineering: A Survey

Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, Yiling Lou

arXiv:2409.02977v2cs.SEcs.AI

TL;DR

Software-engineering research needs a consolidated account of how LLM-based agents are applied and designed across its complex task landscape. This paper surveys 124 papers from software-engineering and agent perspectives, finding broader and stronger applicability for agents than standalone LLMs while noting uneven evidence across strategies.

  • Problem

    Existing research on LLM-based agents for software engineering spans diverse tasks and architectures, creating a need for a consolidated survey of the field.

  • Method

    The paper conducts a comprehensive survey of 124 papers, organizing them by software-engineering activities and agent components, systems, and interactions.

  • Results

    LLM-based agents generally show stronger performance and wider applicability than standalone LLMs across software-engineering tasks.

  • Takeaways & Limitations

    The survey provides a structured overview of agent applications and designs while identifying open challenges and future research directions.

  • Takeaways & Limitations

    Uneven publication status across tasks and strategies introduces uncertainty about the effectiveness and generality of surveyed techniques.

Abstract

from arXiv · show

The recent advance in Large Language Models (LLMs) has shaped a new paradigm of AI agents, i.e., LLM-based agents. Compared to standalone LLMs, LLM-based agents substantially extend the versatility and expertise of LLMs by enhancing LLMs with the capabilities of perceiving and utilizing external resources and tools. To date, LLM-based agents have been applied and shown remarkable effectiveness in Software Engineering (SE). The synergy between multiple agents and human interaction brings further promise in tackling complex real-world SE problems. In this work, we present a comprehensive and systematic survey on LLM-based agents for SE. We collect 124 papers and categorize them from two perspectives, i.e., the SE and agent perspectives. In addition, we discuss open challenges and future directions in this critical domain. The repository of this survey is at https://github.com/FudanSELab/Agent4SE-Paper-List.

1 INTRODUCTION

This survey examines LLM-based agents for software engineering by organizing existing work across software-engineering tasks and agent architectures. It synthesizes 124 papers and discusses open challenges and future directions.

  • Motivation: LLM-based agents extend standalone LLMs with environmental perception and external-tool use, increasing their versatility for software engineering.The survey positions agents as LLM-centered systems that interact with surrounding environments and resources.
  • Contributions: The survey synthesizes 124 papers on LLM-based agents for software engineering.Its contributions include categorization from both the software-engineering and agent perspectives.
  • Contributions: It also identifies research opportunities and future directions in this emerging domain.The survey structure reserves discussion for potential research opportunities.
  • Survey scope: The survey analyzes applications across requirements engineering, code generation, static checking, testing, debugging, and end-to-end development and maintenance.This provides a software-engineering perspective on how agents tackle individual tasks and complete workflows.
  • Survey scope: From the agent perspective, it examines foundation LLMs, planning, memory, perception, action, multi-agent systems, and human-agent collaboration.The analysis covers agent roles, collaboration mechanisms, information flows, applications, and human interaction.

2 BACKGROUND AND PRELIMINARY

LLM-based agents combine planning and memory with perception and action to pursue software-engineering goals, while multi-agent and human-agent systems support more complex collaborative work. The background also frames software engineering as a complex, iterative, changing, and collaborative lifecycle addressed by the surveyed research.

  • Basic LLM-based agents: LLM-based agents typically comprise planning, memory, perception, and action components that connect an LLM-controlled brain with its environment.Planning and memory support internal control, while perception and action enable environmental interaction.
  • Basic LLM-based agents: Planning decomposes complex tasks into subtasks, schedules them, and can revise plans using environmental or human feedback.Agents may generate an initial plan through reasoning strategies or adjust an existing plan.
  • Basic LLM-based agents: Memory records historical thoughts, actions, and observations, allowing agents to reuse prior records and experience during complex tasks.Memory representation and retrieval or update mechanisms affect agent efficiency and effectiveness.
  • Basic LLM-based agents: Perception receives textual, visual, or auditory environmental information, while action executes decisions and uses external tools to expand the action space.Tool use provides access to external resources beyond text-only interaction.
  • Advanced agent systems: Multi-agent systems assign distinct roles and expertise, enabling communication and information sharing for more complex tasks.Human-agent coordination additionally incorporates human requirements, feedback, expertise, and guidance.
  • Software engineering: Software engineering spans requirements, design, coding, static checking, testing, and maintenance across a complex, changing, iterative, collaborative lifecycle.These characteristics motivate methods and tools that support individual engineers and team collaboration.

3 SURVEY METHODOLOGY

The survey defines its scope around LLM-based agents for SE and synthesizes research through database searching, snowballing, author feedback, and collaborative screening. The resulting collection contains 124 papers, combining peer-reviewed sources with emerging preprints.

  • Scope: The survey covers LLM-based agents applied across SE tasks throughout the software life cycle.Its scope includes requirements engineering, software design, code generation, software quality assurance, and software improvement.
  • Scope: The paper synthesizes existing research rather than conducting additional experimental analysis.It is positioned as a comprehensive survey intended to provide a quick overview of a rapidly evolving field.
  • Paper collection: The collection process combines keyword searching, snowballing, and author feedback with independent review and third-author arbitration.The first two authors independently assess relevance, and a third author resolves disagreements until consensus.
  • Keyword searching: 57 DBLP searches produced 10,362 hits, from which 67 relevant papers were identified through manual filtering.The searches were conducted on July 1, 2024, using iteratively refined keywords.
  • Collection outcome: 124 papers were collected after keyword searching, snowballing, and author feedback expanded the initial set.Snowballing added 41 papers, while author feedback produced additional recommendations and corrections.
  • Collection outcome: Approximately 75% of the references are peer-reviewed publications, while the remainder are arXiv preprints.The survey describes this mix as balancing scholarly reliability with timely coverage of emerging work.

4 ANALYSIS FROM SE PERSPECTIVES

The survey organizes LLM-based agents along the software development and maintenance life cycle, distinguishing individual SE tasks from end-to-end development and maintenance processes. Most collected agents target individual tasks, particularly code generation and code quality assurance.

  • Task coverage: The survey maps SE activities along the common software development and maintenance life cycle.Figure 4 presents the distribution of collected agents across these activities.
  • Task coverage: LLM-based agents address both individual SE tasks and end-to-end software development or maintenance processes.End-to-end development can generate complete programs from requirements, while end-to-end maintenance can generate patches for user-reported issues.
  • Distribution: Most collected agents focus on individual-level SE tasks, especially code generation and code quality assurance.Code quality assurance includes static checking and testing.

4.1 Requirements Engineering

LLM-based agents support requirements engineering through multi-agent role-playing, collaboration, and tool-enhanced single-agent feedback loops. The survey reports quality gains from external verification while identifying continuing problems with correctness, human involvement, and requirements evolution.

  • Requirements engineering phases: Requirements engineering includes elicitation, modeling, negotiation, specification, verification, and evolution.These phases cover collecting, representing, coordinating, documenting, validating, and adapting requirements.
  • Multi-agent collaboration: Multi-agent approaches assign roles such as users, stakeholders, requirements engineers, modelers, checkers, and documenters to simulate RE collaboration.Examples span elicitation, specification, analysis or negotiation, validation, modeling, verification, and documentation.
  • Tool-enhanced single-agent strategy: Tool-enhanced single-agent methods iteratively refine generated artifacts using feedback from external verification or analysis tools.SpecGen combines an LLM-based agent with OpenJML, mutating failed specifications before re-verification.
  • Tool-enhanced single-agent strategy: 15.84%, 47.01%, and 53.76% are SpecGen's reported improvements over purely LLM-based methods, Houdini, and Daikon, respectively.These comparisons concern specification generation quality.
  • Strategy comparison: Multi-agent collaboration supports one or more RE phases but raises quality-assurance challenges, whereas single-agent tool enhancement yields higher quality with limited task scalability.The trade-off contrasts collaborative coverage with external-tool feedback and the constraints of a single-agent setup.
  • Challenges: RE agents may generate vague, irrelevant, or incorrect requirements and often lack effective human-agent interaction and requirements-evolution mechanisms.The survey links these challenges to insufficient domain knowledge, reduced stakeholder involvement and trust, and limits in iterative long-term development.

4.2 Code Generation

LLM-based agents extend code generation beyond standalone models through planning, iterative refinement, and feedback from models, tools, and humans. The survey compares prompt-engineering and agentic strategies while identifying reliability, efficiency, and scalability challenges.

  • Code Generation with Planning: LLM-based agents transform one-time code generation into a plan-generate-refine process intended to improve generation correctness.Plans may use natural language, pseudocode, intermediate code, or code skeletons.
  • Code Generation with Planning: Prompt engineering decomposes code generation into step-by-step subtasks, with zero-shot and few-shot chain-of-thought as common strategies.Few-shot chain-of-thought provides completed task examples, whereas zero-shot chain-of-thought does not.
  • Code Generation with Planning: Agentic strategies dynamically adapt code-generation plans using historical thoughts, actions, observations, environmental feedback, and self-reflection.CodePlan detects affected repository snippets and adapts its modification plan accordingly.
  • Code Generation with Planning: Two planners do not surpass a single planner in Flows, while combining traditional chain-of-thought with agentic planning improves code-generation accuracy in RAT.The comparison distinguishes collaborative planning from hybrid reasoning strategies.
  • Code Generation with Planning: Prompt engineering plans are determined upfront, whereas agentic strategies iteratively revise plans from environmental feedback and are therefore more dynamically adaptive.Prompt engineering is broadly applicable to instruction-following LLMs, while agentic strategies require LLM-based agents.
  • Code Generation with Iterative Refinement: Tool feedback supplies static analysis, runtime monitoring, and performance evaluation that help mitigate hallucination and randomness in generated code.External tools provide diagnostic support across multiple dimensions of code quality.
  • Code Generation with Iterative Refinement: Human feedback clarifies ambiguous requirements and helps keep generated code consistent with user intent and expectations.Feedback can be incorporated at decision points such as requirement analysis.
  • Code Generation with Iterative Refinement: Hybrid feedback combines complementary model and tool strengths but may remain vulnerable to cascading errors because the agent still supplies the final feedback.Model feedback can add contextual relevance, while tool feedback can improve precision.

4.3 Static Code Checking

LLM-based agents support static code checking by detecting and reporting bugs or vulnerabilities through specialized roles, external knowledge, traditional analysis, and multi-agent collaboration. The survey highlights shallow tool integration, demanding reasoning requirements, and persistent false positives as major challenges.

  • Static Bug Detection: Static code checking examines code without execution and can identify bugs, vulnerabilities, and code smells before extensive testing.Static analysis commonly supports automated bug or vulnerability detection.
  • Static Bug Detection: Agents for static bug detection detect input programs and produce bug or vulnerability reports through detector, validator, ranker, and assistant roles.Validators confirm or filter detections, while rankers order suspicious results.
  • Static Bug Detection: Multi-agent vulnerability detection uses specialized agents that discuss findings and seek consensus, mirroring collaborative debugging processes.Examples include developer-tester discussion and multi-auditor smart-contract analysis.
  • Additional Knowledge from Tool Execution: Tool invocation enhances bug analysis with retrieved demonstrations, document and web search, code context, vulnerability knowledge, and execution tools.LLM4Vuln identified 14 zero-day vulnerabilities in four pilot bug bounty programs.
  • Combined with Traditional Static Bug Detection: Traditional static analysis is combined with agents to improve detection, using fast but imprecise analysis followed by symbolic execution or agent-based handling of undecided bugs.UBITect’s symbolic-execution stage discards 40% of bugs because of time or memory limits.
  • Comparison of Enhancement Strategies: The main enhancement strategies are multi-agent collaboration, knowledge enhancement, and integration of static analysis tools.Collaboration divides labor, while knowledge and tools are especially important for single-agent approaches.
  • Code Review: Process-based multi-agent code review organizes specialized agents into sequential stages resembling traditional human review pipelines.Stages can include information gathering, code analysis, revision, and documentation.
  • Code Review: CodeAgent implements a waterfall-like review pipeline with four stages and six agents assigned different organizational and technical roles.The stages are information synchronization, code review, code alignment, and documentation.

4.4 Testing

LLM-based testing agents use tools, environmental feedback, and iterative refinement to improve test validity, coverage, and fault detection across unit and system testing. The survey organizes these agents by testing target and highlights challenges in context acquisition and tool integration.

  • Unit Testing: LLM-based unit-testing agents generate initial tests for a program under test and iteratively refine them using compilation, testing, and static-analysis feedback.The refinement targets include reducing execution or compilation errors, increasing coverage, and improving fault detection.
  • Unit Testing: Coverage-oriented agents use uncovered code information, execution feedback, and code-query tools to generate tests for difficult branches.CoverUp uses SlipCover and abstract syntax trees, while TELPA targets hard-to-cover branches through iterative feedback.
  • Unit Testing: 99.3% test coverage was achieved on HumanEval through iterative test generation, execution, and revision, comparable to human-written tests.
  • System Testing: System-testing agents incorporate domain knowledge and dynamically explore integrated software systems whose interacting components and execution paths challenge standalone LLMs.The surveyed system-testing applications include fuzzing and GUI testing across diverse software systems.
  • Challenges: Testing agents remain constrained by complex project context and by weak integration between agent workflows and mature traditional testing tools.The survey identifies context augmentation, unified tool integration, and broader testing strategies as ongoing challenges.

4.5 Debugging

LLM-based debugging agents address fault localization, program repair, and unified debugging through tool use, iterative feedback, and multi-agent collaboration. The survey emphasizes improved project-level reasoning alongside unresolved verification, integration, and efficiency challenges.

  • Fault Localization: Fault-localization agents analyze buggy code and error logs with cooperating expert roles and repository tools such as static analysis and code retrieval.Multi-agent systems can divide expertise among reviewers, architects, and test engineers, while single-agent systems can invoke specialized functions.
  • Program Repair: Program-repair agents generate candidate patches, validate them with compilation and tests, and iteratively refine them from execution feedback until acceptance criteria are met.Environmental feedback may include compilation errors, runtime failures, and test outcomes.
  • Program Repair: 164 bugs in Defects4j were repaired by RepairAgent, including 39 bugs not fixed by prior techniques.
  • Unified Debugging: Unified-debugging systems coordinate localization and repair, allowing downstream repair results to refine upstream localization through iterative, fine-grained analysis.FixAgent also uses a crafter to generate additional tests and a revisitor to assess symptoms and patch rationale.
  • Challenges: Debugging agents remain challenged by correctness verification, costly tool integration, and multi-agent overhead; AutoSD takes approximately five times longer than a standalone LLM to generate a patch.Many systems still produce plausible patches whose correctness beyond available tests remains difficult to establish.

4.6 IT Operations

LLM-based agents extend IT operations by interacting with live systems, tools, and domain documents to diagnose incidents collaboratively. The survey categorizes diagnosis strategies and identifies scheduling heterogeneous evidence as a central challenge.

  • General Framework: IT-operations agents respond to anomaly alerts by dispatching diagnostic experts that interact with target systems, tools, and domain-specific reference documents.
  • Collaborative Diagnosis: RCAgent aggregates multiple reasoning trajectories with embedding-based similarity voting through controller and specialized expert agents.
  • Collaborative Diagnosis: mABC decomposes root-cause analysis among specialized agents and uses blockchain-inspired voting to reach consensus on causes and remediation.
  • Collaborative Diagnosis: D-Bot combines offline diagnostic knowledge extraction, automatic prompt construction, collaborative experts, tree search, and majority voting for database-anomaly diagnosis.Voting selects the most promising reasoning path when alternative hypotheses branch.
  • Challenges: IT-operations agents must schedule experts and integrate heterogeneous data from dimensions such as CPU, memory, workload, and I/O behavior.

4.7 End-to-end Software Development

End-to-end software-development agents use specialized roles and adapted process models to cover the software life cycle from requirements through quality assurance. The survey reports stronger and more stable benchmark performance for Scrum, while noting that evaluation remains limited by benchmark complexity and standardization.

  • Overview: Multi-agent systems extend beyond individual SE tasks by assigning specialized roles across requirements engineering, architecture, coding, testing, debugging, review, deployment, and assistance.
  • Software Development Process Model: Current end-to-end agents commonly adapt waterfall workflows by adding iterations to selected phases because LLM randomness and hallucination can reduce accuracy.
  • Software Development Process Model: Scrum achieved the best and most stable performance on function-level code-generation benchmarks, followed by Test-Driven Development.
  • Agent Roles: End-to-end frameworks typically use predefined roles, although some systems derive task-specific roles from a small set of meta-roles.Role categories include managers, requirements engineers, designers, developers, quality-assurance experts, testers, debuggers, reviewers, deployment engineers, and assistants.
  • Agent Evaluation: Evaluation is limited because many benchmarks use small function-level tasks, while real-world end-to-end development involves more complex project-scale requirements and outputs.The survey also identifies a lack of standardized benchmarks and metrics for open-ended software generation.

4.8 End-to-end Software Maintenance

End-to-end software maintenance agents commonly follow issue localization, patch generation, and patch verification, with optional preprocessing, reproduction, and planning stages. Evidence suggests dynamic verification and patch ranking are associated with higher resolve rates, while localization and verification remain challenging.

  • Existing agents use three mandatory phases—issue localization, patch generation, and patch verification—with optional preprocessing, issue reproduction, and task decomposition.
  • Preprocessing can construct repository knowledge structures, generate test templates, or select human-designed workflows before maintenance begins.
  • Localization strategies include retrieval, autonomous navigation, spectrum-based scoring, and Monte Carlo Tree Search over repository knowledge graphs.
  • 17.00% to 20.33%: AutoCodeRover’s issue resolution rate increased when spectrum-based fault localization was applied.
  • Patch verification remains difficult because reproduction tests may be unavailable or incorrect, and regression tests can produce false negatives for semantically correct patches.
  • 9.89%: SWE-bench-java-verified’s best reported resolve rate fixed 9 of 91 issues using SWE-agent and DeepSeek-Coder.
  • Pure autonomous localization does not necessarily improve performance, and traditional fault-localization approaches can surpass more complex agentic designs.
  • Dynamic patch verification and patch ranking generally correspond to higher resolve rates, with four of the five highest-rate agents using both strategies.

5 ANALYSIS FROM AGENT PERSPECTIVE

The agent-perspective analysis organizes LLM-based software engineering agents around planning, memory, perception, action, and foundation models. It surveys planning forms and challenges, memory mechanisms, and the strong context and reasoning demands placed on foundation models.

  • The agent perspective examines planning, perception, memory, action, foundation models, multi-agent systems, and human coordination.
  • Planning decomposes complex tasks, schedules subtasks, and can revise plans using environmental or human feedback.
  • Planning strategies vary by planner count, planning turns, path count, and plan representation, including natural-language, semi-structured, and graph forms.
  • Planning: Two planners did not surpass one planner in Flows, while multi-path planning additionally increases time and token costs.
  • Planning: Planning can hallucinate nonexistent methods or variables, repeat steps, and become less reliable as task complexity increases.
  • Memory: Short-term memory sustains current-task trajectories in multi-turn interactions and helps agents track progress and avoid repeating mistakes.
  • Foundation LLMs: Long context windows and strong multi-step reasoning remain critical because end-to-end maintenance requires coherent reasoning across extended interactions.
  • Foundation LLMs: Architecture and agent design cannot always bridge performance gaps between foundation LLMs.

5.2 Multi-agent System

Multi-agent systems assign specialized roles and coordinate their work across software engineering tasks. The survey characterizes role types, hierarchical collaboration, managerial functions, and scalability bottlenecks.

  • 59.7% of surveyed SE agents are multi-agent systems, which divide specialized roles and coordinate agents for complex, multi-phase tasks.
  • Role-playing assigns duties, available actions, attributes, and constraints so agents specialize in corresponding tasks.
  • Manager Roles: Manager roles make decisions, decompose and assign tasks, plan work, and oversee team coordination.
  • Manager Roles: Managers may dynamically assemble teams by selecting role compositions to optimize costs and meet project demands.
  • Multi-agent systems include requirement-analysis, design, development, quality-assurance, testing, debugging, and deployment roles.
  • Software Quality Assurance Roles: Quality-assurance roles review code, generate tests, and diagnose failures or unexpected behaviors.
  • Collaboration Structures: Layered collaboration decomposes tasks into stages whose agents sequentially pass intermediate results to later stages.
  • Performance Issues and Solutions: Scaling dynamically derived agent teams can create performance bottlenecks, so systems may limit the number of derived agents.

5.3 Human-Agent Collaboration

LLM-based SE agents increasingly incorporate humans at planning, requirements, development, and evaluation stages, but collaboration remains largely agent-centric and underexplored.

  • Human-Agent Collaboration: Human participation commonly occurs during planning, requirements refinement, development, and evaluation.These interventions help users revise workflows, clarify intent, correct agent errors, and validate outputs.
  • Planning Phase: Human-AI collaboration during planning outperforms other AI feedback-based approaches in Flows.However, revising system designs may require expertise and is optional in some agents.
  • Requirements Phase: Requirements refinement addresses ambiguity that can otherwise create a gap between agent outputs and user intent.ClarifyGPT generates targeted questions and incorporates human responses to refine requirements.
  • Discussion: Overall, existing systems usually place human involvement at predefined points rather than supporting human-driven workflows throughout the process.The survey identifies broader human-agent collaboration as an open challenge, including scenarios involving multiple people.

6 RESEARCH OPPORTUNITIES

The survey identifies research opportunities in evaluation, human-agent collaboration, perception, task coverage, and software-oriented models, prioritizing standardized benchmarks and metrics.

  • Evaluation of Agents for SE: Evaluation should combine diverse metrics with higher-quality, more realistic benchmarks for SE agents.The survey prioritizes standardized benchmarks and metrics because their absence limits meaningful comparisons.
  • Metrics: Fine-grained metrics are needed to expose intermediate decisions and failure points beyond black-box task success.Current evaluations often measure benchmark success rates without explaining why agents fail.
  • Evaluation of Agents for SE: Only 46.7% of surveyed papers explicitly quantify agent efficiency using measures such as time, tokens, cost, or feedback loops.These costs matter for large repositories, complex documentation, and intricate workflows.
  • Benchmarks: Current benchmarks may contain vague issue descriptions and tasks that are substantially smaller and simpler than real-world software projects.The survey cites SWE-bench tasks, including 77.8% completable by an experienced engineer within an hour, as evidence of this gap.
  • Human-Agent Collaboration: Human involvement remains underexplored across the full software life cycle, including architecture design, test generation, code review, and maintenance.Effective interfaces must display intermediate outputs and collect feedback efficiently.
  • Perception Modality: Most SE agents rely on textual or visual perception, while voice commands, gestures, and other modalities remain underutilized.Only one surveyed work had used a multimodal LLM, and agents for design, verification, and feature maintenance are also lacking.

7 DISCUSSION

LLM-based agents extend standalone LLMs through environmental interaction, planning, and feedback, but the survey’s conclusions remain limited by uneven validation and publication status.

  • Disparity of LLM-based Agents and Standalone LLMs in SE Tasks: Standalone LLMs remain effective for many single-phase tasks, but hallucination, stochasticity, and limited adaptability restrict their coverage.Agents address these limitations through planning, feedback, and environmental interaction.
  • Disparity of LLM-based Agents and Standalone LLMs in SE Tasks: LLM-based agents generally outperform standalone LLMs on effectiveness while supporting broader, environment-dependent SE tasks.Standalone LLMs are easier to adopt but lack dynamic interaction and plan adjustment.
  • Threats to Validity: Manual screening may omit relevant papers, potentially affecting survey comprehensiveness and biasing its findings.Two authors independently reviewed papers and a third resolved disagreements, but subjective judgment remains a validity threat.
  • Threats to Validity: Uneven publication status introduces uncertainty about the effectiveness and generality of surveyed agent techniques.Several strategies rely partly on preprints or unpublished work, including multi-agent requirements engineering and visual-input approaches.
  • Threats to Validity: Evidence remains insufficient for some knowledge-enhanced bug detection methods and memory formats.The survey calls for more empirical and peer-reviewed studies to assess robustness, generalizability, and memory-format effectiveness.
  • Threats to Validity: Iterative coverage improvement in unit testing has weaker empirical support than iterative refinement for compilation or execution errors.Only one of three cited studies on coverage improvement was published, so effectiveness remains insufficiently established.

8 CONCLUSION

The paper surveys 124 papers on LLM-based agents for software engineering from both SE and agent perspectives and discusses open challenges and future directions.

  • 8 CONCLUSION: The survey analyzes LLM-based agents across software development and maintenance activities and their agent-component designs.It also discusses open challenges and future directions for the domain.
Loading 2409.02977v2…