Large Language Model-Based Agents for Software Engineering: A Survey

SE perspective

• LLMs already support software development and maintenance tasks including program generation, testing, debugging, and program improvement [4],,, [7], [8], [9], [10],, [12], [13], [14], [15],, [17],, [19], [20], [21],,.Recent surveys report promising performance across these Software Engineering activities [2], [3].Source: §1 · p. 1

• LLM-based agents augment standalone LLMs with perception, external resources, tools, multi-agent collaboration, and human interaction for complex real-world goals.Unlike standalone LLMs, these agents use LLMs as central controllers and can autonomously perceive and act on environments.Source: §1 · p. 1
• The survey collects 124 papers and categorizes LLM-based agents for SE from both Software Engineering and agent perspectives.It provides a comprehensive and systematic review of the domain.Source: §1 · p. 1

SE perspective

• From the SE perspective, it covers individual activities such as requirements engineering, code generation, static checking, testing, and debugging, as well as end-to-end development and maintenance.This perspective summarizes how LLM-based agents tackle SE tasks.Source: §1 · p. 1

Agent perspective

• From the agent perspective, it analyzes foundation LLMs, planning, memory, perception, action, multi-agent roles and collaboration, information flows, applications, and human-agent collaboration.The survey summarizes component characteristics for LLM-based agents applied to SE and discusses research opportunities and future directions.Source: §1 · p. 1

2 BACKGROUND AND PRELIMINARY

• The section establishes the background needed to study LLM-based agents for software engineering and frames the survey around their design, application, and coordination capabilities.These foundations connect agent components and collaboration mechanisms with the complexity of software development and maintenance.Source: §2 · p. 2, §2.3 · p. 3

2.1 Basic Framework of LLM-based Agents

• LLM-based agents combine planning, memory, perception, and action to pursue goals, with planning and memory forming the reasoning core and perception and action mediating environmental interaction.Planning decomposes tasks and revises plans using feedback; memory stores and retrieves prior execution records; perception handles multimodal inputs; action uses external tools to extend capabilities.Source: §2.1 · p. 2

2.2 Advanced LLM-based Agent Systems

• Advanced agent systems address complex tasks through role-specialized agents that collaborate or compete, and through human guidance that supplies requirements, feedback, expertise, and joint decision-making.Multi-agent systems communicate and share progress across subtasks, while human-agent coordination improves alignment with human preferences.Source: §2.2 · p. 3

2.3 Software Engineering

• Software engineering spans requirements, design, coding, checking, testing, and maintenance, while facing complexity, changing requirements and codebases, iterative development, and cross-role collaboration.LLM-based agents can address these challenges through perception, iterative mechanisms, planning, action, and multi-agent systems, supporting both single-phase and end-to-end development and maintenance tasks.Source: §2.3 · p. 3

2.4 Related Surveys

• Unlike surveys of general LLM-based agents or LLMs for software engineering, this work covers broader and end-to-end SE tasks while organizing systems by agent architectures and interaction modes,, [29],,,,.It surveys existing single-agent and multi-agent systems rather than presenting only a vision of future multi-agent applications.Source: §2.4 · pp. 3–4

3.1 Survey Scope

• The scope covers SE tasks across the software life cycle and LLM-based agents that autonomously and iteratively perceive environmental feedback and act in real time.The survey is positioned as a comprehensive survey for rapid field overview; it includes collected experimental results but performs no additional experiments.Source: §3.1 · p. 4

3.2 Paper Collection

• Paper collection uses independent reviews by the first two authors, third-author arbitration, and three stages: keyword searching, snowballing, and author feedback collection.Inclusion and exclusion criteria guide relevance decisions throughout the collaborative process.Source: §3.2 · p. 4

3.2.1 Keyword Searching

• 57 DBLP searches on July 1, 2024 produced 10,362 hits, from which manual filtering identified 67 relevant papers using iteratively refined search keywords.The final query combined agent or LLM terms with SE-related terms, synonyms, and variations.Source: §3.2.1 · p. 4

3.2.2 Snowballing

• Backward and forward snowballing continued until no new relevant papers were found and added 41 papers to the collection.Backward snowballing examined references, while forward snowballing used Google Scholar to find citing papers.Source: §3.2.2 · pp. 4–5

3.2.3 Author Feedback Collection

• Author feedback contacted 321 authors, yielding 36 valid responses, including 29 recommended papers, of which 16 were included after relevance filtering.Respondents also confirmed descriptions and suggested publication-status, method-description, and formatting revisions.Source: §3.2.3 · p. 5

3.3 Statistics of Collected Papers

• The final survey contains 124 papers spanning software engineering, artificial intelligence, and human-computer interaction venues, with approximately 75% of references peer-reviewed.The remaining citations are arXiv preprints, balancing established scholarly sources with emerging work.Source: §3.3 · p. 5

• End-to-end development agents generate complete programs from requirements across requirements engineering, design, code generation, and quality assurance, whereas maintenance agents generate patches through debugging and feature maintenance.The surveyed systems therefore span multiple activities rather than a single isolated task.Source: §4 · pp. 5–6
• Unlike standalone LLMs specialized for single SE tasks and inadequate for complex end-to-end processes [2], [3], LLM-based agents provide autonomy and flexibility through modular capabilities, coordination, and human interaction.The cited agent components are planning, memory, perception, and action.Source: §4 · p. 6
• Most collected agents target individual SE tasks, especially code generation and code quality assurance, while a smaller portion addresses end-to-end development or maintenance.Figure 4 reports task-category counts and records end-to-end systems only at the end-to-end level, not under their component tasks.Source: §4 · p. 6

4.1 Requirements Engineering

• LLM-based RE agents simulate users, stakeholders, engineers, modelers, checkers, and documenters across elicitation, modeling, negotiation, specification, and verification, with external validation tools supporting iterative refinement.This framework targets complete requirements documents rather than isolated RE tasks.Source: §4.1 · p. 6, Figure 5 · p. 7

4.1.1 Multi-agent Collaboration Strategy.

• Multi-agent collaboration assigns distinct personas and shared-workspace roles to coordinate RE activities, uncover hidden needs, negotiate priorities, and produce or verify requirement models and specifications.Elicitation reduces costs compared with conventional user studies, while MARE coordinates stakeholder, collector, modeler, checker, and document agents.Source: §4.1.1 · p. 7

4.1.3 Comparison of Multi-agent and Tool-enhanced Single-agent Strategy.

• Multi-agent methods cover individual or multiple RE phases through team-like communication, whereas tool-enhanced single agents produce higher-quality artifacts through external feedback but have limited task scalability.The two strategies trade collaborative coverage against verification-driven quality and single-agent scalability.Source: §4.1.3 · pp. 7–8

4.1.4 Challenges of LLM-based Agents in RE

• LLM-based RE agents still generate vague, irrelevant, or incorrect requirements, because domain knowledge is insufficiently integrated, while human–agent collaboration and requirements evolution remain under-supported.Replacing stakeholders with agents can reduce involvement and trust, and current agents lack mechanisms for iterative long-term requirements change, as shown in Table 3.Source: Table 3 · p. 7, §4.1.4 · p. 8

4.2 Code Generation

• Agents transform one-time code generation into a “plan-generate-refine” process that improves correctness through planning and diverse feedback sources.Feedback may come from the model, external tools, humans, or combinations of these types.Source: §4.2 · p. 8

4.2.1 Code Generation with Planning

• Planning methods include broadly applicable prompt engineering and adaptive agentic strategies, with code-based representations and iterative environmental feedback extending plan expressiveness.Prompt engineering typically fixes plans upfront, whereas agentic strategies revise plans dynamically; few-shot CoT often uses static formatting examples, while advanced strategies remain underexplored. Flows found that two planners did not outperform one, while RAT improved over basic CoT.Source: §4.2.1 · pp. 8–9

4.2.2 Code Generation with Iterative Refinement

• Iterative refinement uses peer- or self-reflection, execution and analysis tools, retrieval, human clarification, and hybrid feedback to progressively improve generated code.Model feedback supports interpretation and error detection, tool feedback supplies diagnostics and external knowledge, human feedback confirms intent, and hybrid methods combine tool precision with model explanations; LDB outperformed methods analyzing only final outputs, while natural-language feedback produced 2–17% absolute gains per additional turn.Source: §4.2.2 · pp. 10–11

4.2.3 Common Failure Causes of LLM-based Agents in Code Generation.

• Agents commonly fail through poor multi-agent coordination, unreliable tests, cascading feedback errors, and degraded reasoning as long-context interaction histories grow.Self-Refine attributed 33% of failed cases to incorrect error localization and 61% to inappropriate fixes; weak tests can cause false positives, false negatives, and insufficient coverage.Source: §4.2.3 · pp. 11–12

4.2.4 Challenges of LLM-based Agents in Code Generation.

• Key challenges are unreliable or insufficient tests, the cost of iterative refinement, and dependence on external tools whose retrieved or analyzed information may be inaccurate.These limitations affect test coverage and correctness, efficiency and cost, and the reliability of environmental feedback.Source: §4.2.4 · p. 12

4.3 Static Code Checking

• Static code checking identifies quality issues without executing code, including bugs, vulnerabilities, and code smells, making it useful in continuous integration before extensive testing.Common practices include automated static bug detection and peer code review.Source: §4.3 · p. 12

4.3.1 Static Bug Detection

• LLM-based static code-checking agents combine multi-agent collaboration, knowledge enhancement, and traditional static-analysis tools to balance LLM semantics with rule-based precision.These strategies can also be combined, as in LLM4DFA, which uses multi-agent collaboration and static-analysis tools.Source: §4.3.1 · pp. 12–14

4.3.2 Code Review

• Multi-agent code-review systems either orchestrate sequential workflows or specialize agents by task, supporting coordinated end-to-end review or focused objectives such as bug detection and optimization.CodeAgent demonstrates process-based review, while Rasheed et al. use specialized review, bug-report, code-smell, and optimization agents; both approaches show effectiveness or potential across review tasks.Source: §4.3.2 · pp. 14–15

4.3.3 Challenges of LLM-based Agents in Static Code Checking.

• LLM-based agents still face shallow integration with static-analysis tools, demanding intermediate representations and strong reasoning while lacking reliable automated filtering of false positives.These demands can necessitate high-performing proprietary models such as GPT-4,, and false positives limit practical reliability,,.Source: §4.3.3 · p. 15

4.4 Testing

• Standalone LLM-generated tests commonly contain compilation, execution, oracle, and coverage problems, motivating agent-based testing for stronger practical quality assurance.High-quality tests must be syntactically and semantically correct while covering as many software states as possible.Source: §4.4 · p. 15, §4.4.1 · p. 15

4.4.1 Unit Testing

• Agents iteratively generate and refine unit tests using compilation, execution, static-analysis, coverage, and mutation feedback to improve correctness, coverage, and fault detection.Common workflows generate initial tests, verify them with external tools, and revise them based on feedback; AutoDev achieved 99.3% coverage on HumanEval, comparable to human-written tests.Source: §4.4.1 · pp. 15–16

4.4.2 System Testing

• System-testing agents incorporate domain knowledge and dynamically explore complex software systems across kernels, compilers, mobile applications, web APIs, fuzzing, and penetration testing.Examples use source-code analysis, iterative specification repair, GUI perception and memory, API reflection loops, and modular tool-using workflows to generate and validate system-level tests.Source: §4.4.2 · pp. 16–18
• Mobile-testing agents support higher-level objectives through long-term memory, multi-agent coordination, GUI interaction, bug replay, unusual-input generation, and multimodal functional-bug detection.GPTDroid, DroidAgent, AXNav, InputBlaster, VisionDroid, AdbGPT, and XUAT-Copilot respectively combine perception, planning, execution, feedback, memory, or specialized agent roles for mobile testing tasks.Source: §4.4.2 · pp. 17–18

4.4.3 Challenges of LLM-based Agents in software testing

• Testing agents still struggle to augment broad code and documentation context, tightly integrate traditional tools, and coordinate multidimensional objectives such as quality, coverage, security, and assertion generation.Current systems often use mature testing tools alongside agents rather than within unified strategies, and most remain focused on narrow task-specific goals; multi-agent coordination is promising but underexplored.Source: §4.4.3 · pp. 18–19

4.5 Debugging

• LLM-based debugging agents organize software debugging into fault localization, program repair, and unified debugging, which respectively identify buggy elements, generate patches, and connect both phases.Fault localization uses failure symptoms to identify buggy code, while repair generates fixes; unified debugging lets each phase refine the other,,.Source: §4.5 · p. 19

4.5.1 Fault Localization

• Fault-localization agents combine cooperating expert roles with repository tools to explore suspicious code, while AgentFL divides project-level localization into comprehension, navigation, and confirmation.Agents may invoke static analysis, code retrieval, dynamic instrumentation, and codebase navigation; AutoFL [13] instead enhances a single agent with four specialized function calls.Source: §4.5.1 · p. 19

4.5.2 Program Repair

• Program-repair agents iteratively generate, test, and refine patches using compilation, execution, and test feedback because one-shot LLM patches can fail or overfit.RepairAgent repaired 164 Defects4J bugs, including 39 not fixed by prior techniques, while ChatRepair, and CigaR refine patches from environmental feedback; CigaR also reboots after failed invocations to escape dead ends.Source: §4.5.2 · pp. 19–20

4.5.3 Unified Debugging

• Unified-debugging agents use division of labor and fine-grained analysis to let localization and repair influence each other iteratively, as in FixAgent and LDB.FixAgent coordinates a Localizer, Repairer, Crafter, and Revisitor, whereas LDB analyzes variable changes across program blocks during test execution to locate and repair bugs.Source: §4.5.3 · p. 21

4.5.4 Common Failure Causes of LLM-Based Agents in Debugging

• Common failures include inefficient exploration of complex project context, incoherent reasoning and verification, context overflow, incomplete or logically incorrect edits, and tool-incompatible invocations.In AutoFL [13], excessive exploration of project-specific structures left insufficient budget for inspecting potentially buggy code. In AutoSD, an unhit breakpoint led the agent to incorrectly blame the test instead of revisiting its hypothesis.Source: §4.5.4 · p. 21

4.5.5 Challenges of LLM-based Agents in Debugging

• Debugging agents still struggle with correctness signals, tool integration, collaboration overhead, and semantic patch verification, leaving plausible patches that require manual inspection.AutoSD takes approximately five times longer than a standalone LLM to generate a patch,, and intermediate execution signals alone remain insufficient for reliably assessing correctness, [13],.Source: §4.5.5 · p. 21

4.6 IT Operations

• A controller dispatches anomaly-diagnosis tasks to tool-equipped expert agents that explore root causes through target-system interaction and produce diagnostic reports or solutions.The pipeline uses predefined toolkits and domain-specific reference documents while agents pursue different exploration paths. [Fig. 12]Source: §4.6 · p. 22, Figure 12 · p. 22

4.6.1 Self-Consistency with Embedding Voting.

• RCAgent shares preliminary reasoning steps across trajectories and aggregates outcomes by embedding similarity, improving inference efficiency and robustness.Its controller coordinates thought-action-observation cycles, while specialized expert agents perform code and log analysis.Source: §4.6.1 · p. 22

4.6.2 Agent Chain with Blockchain-Inspired Voting.

• mABC uses a decentralized Agent Chain of specialized agents and blockchain-inspired voting to reach reliable consensus on root causes and remediation.An Alert Receiver prioritizes alerts, a Process Scheduler assigns subtasks, and a Solution Engineer proposes strategies using historical knowledge.Source: §4.6.2 · p. 22

4.6.3 Tree Search with Majority Voting.

• D-Bot combines offline diagnostic knowledge extraction, automatically matched prompts and tools, tree-search reasoning, and majority voting to explore competing database-anomaly hypotheses.Multiple reasoning agents assess evidence independently, and the highest-consensus branch guides further exploration, preserving diversity and robustness against individual errors.Source: §4.6.3 · p. 22

4.6.4 Challenges of LLM-based Agents in IT Operations.

• IT-operations agents must schedule experts across heterogeneous system dimensions, plan iterative exploration efficiently, and expand beyond root-cause analysis to performance analysis and optimization.Broader coverage requires integration with operational tools such as logging systems and performance profilers.,Source: §4.6.4 · p. 22

4.7.2 Role Specialization of Software Development Team

• End-to-end agents cover requirements, architecture, coding, quality assurance, deployment, and support through specialized roles that may be predefined or task-adaptively created.Roles include managers, requirements engineers, designers, developers, testers, debuggers, reviewers, deployment engineers, and assistants; most systems manually predefine roles, while others dynamically recruit or spawn them.Source: §4.7 · p. 23, §4.7.2 · p. 24

4.7 End-to-end Software Development

• Multiagent synergy enables autonomous end-to-end development beyond individual software phases, analogous to real-world teams handling the software development life cycle.The surveyed systems address complete procedures such as developing a Snake Game application from scratch, and Table 11 summarizes existing approaches.Source: §4.7 · pp. 22–23, Table 11 · p. 23

4.7.1 Software Development Process Model

• Scrum achieved the best and most stable performance on function-level code-generation benchmarks, followed by Test-Driven Development (TDD).Current agents adapt Waterfall and Agile process models; Waterfall pipelines commonly add feedback iterations to revise outputs after testing,,, [4].Source: §4.7.1 · p. 23

4.7.3 Collaboration Mechanism in Multi-agent

• End-to-end systems predominantly use vertical collaboration, while some phases add horizontal discussion, parallel pipelines, structured artifacts, or shared memory for coordination.Vertical workflows assign and integrate phase outputs; communication commonly uses direct dialogue but may exchange documents, diagrams, structured formats, or information through shared memory.Source: §4.7.3 · p. 25

4.7.4 Agent Evaluation

• Evaluation combines function-level and project-level benchmarks, but project tasks remain limited in complexity and many methods use self-constructed datasets, reducing comparability.Metrics include pass rate, Pass@K, program similarity, execution or generation cost, manual refinement effort, and generated-program scale, while robustness, security, cost, and workflow measures remain underexplored.Source: §4.7.4 · pp. 25–26

4.7.5 Challenges of LLM-based Agents in End-to-end Software Development.

• Most agents still follow linear waterfall-style workflows, target mainly Python or basic Web applications, and lack standardized benchmarks and evaluation metrics for open-ended software generation.These limitations hinder iterative adaptation, broader language and domain support, and reliable assessment of qualities such as reusability and architectural soundness.Source: §4.7.5 · pp. 26–27

4.8 End-to-end Software Maintenance

• Existing systems follow mandatory issue localization, patch generation, and patch verification phases, optionally adding preprocessing, issue reproduction, task decomposition, and patch ranking.The pipeline is summarized in Fig. 15, with different agents using varied strategies across these phases.Source: §4.8 · p. 27, Figure 15 · p. 28

4.8.1 Preprocessing.

• Preprocessing builds repository context through knowledge graphs, hierarchical file trees, expert-selected workflows, or generated test templates that support later localization and reproduction.LingmaAgent constructs a repository knowledge graph, Agentless creates a tree-like project structure, CODER selects human-defined workflows, and MASAI generates a test template.Source: §4.8.1 · p. 27

4.8.2 Issue Reproduction.

• Issue reproduction agents generate executable tests from issue descriptions, while MASAI improves success through a two-stage process that first analyzes the repository’s testing setup and existing tests.Generated tests should ideally fail on the buggy version and pass on the fixed version, making reproduction difficult but useful for localization and verification.Source: §4.8.2 · p. 27

4.8.3 Issue Localization.

• Issue localization combines retrieval, navigation, spectrum analysis, and simulation to identify relevant code from user descriptions, with SBFL-based AutoCodeRover improving resolution from 17.00% to 20.33%.Retrieval is often coarse-grained, navigation explores repository structure, SBFL uses passing and failing tests, and LingmaAgent applies MCTS over a repository knowledge graph; CODER combines suspiciousness and BM25 scores.Source: §4.8.3 · p. 28

4.8.4 Task Decomposition / 4.8.5 Patch Generation.

• Agents decompose issues into file-level or fine-grained resolution tasks, then generate patches from issue descriptions, suspicious code, cross-file context, and ancillary function summaries.MAGIS delegates file-level tasks to a development team, while LingmaAgent outlines fine-grained resolution steps; retrieval APIs and function summaries refine patch-generation context.Source: §4.8.4 · p. 28, §4.8.5 · p. 28

4.8.6 Patch Verification / 4.8.7 Patch Ranking.

• Patch verification uses code review, static checking, dynamic test execution, and regression testing, but unreliable or unavailable reproduction tests motivate patch ranking and multi-agent selection.SpecRover combines reproduction and regression tests, while MASAI, Agentless, SpecRover, and DEIBase rank candidate patches using agents, voting, or review committees.Source: §4.8.6 · pp. 28–29, §4.8.7 · p. 29

4.8.8 Agent Evaluation / 4.8.9 Challenges of LLM-based Agents in End-to-end Software Maintenance.

• Real-world benchmarks mine user-reported GitHub issues because Defects4J is limited to 357 bugs from five projects with 92.41% of patches changing one file; remaining challenges include multimodal issue understanding, reproducibility, and reliable correctness verification.SWE-bench contains 2,294 issues across 12 Python repositories, while SWE-bench-java-verified includes 91 issues across six Java projects and reports a best resolve rate of 9.89% (9/91).Source: §4.8.8 · pp. 29–30, §4.8.9 · p. 30

4.8.8 Agent Evaluation

• Dynamic patch verification and patch ranking generally achieve higher resolve rates, whereas purely autonomous localization can perform poorly and traditional localization can surpass complex agentic approaches.SWE-agent performed worst on SWE-bench Lite despite fully autonomous localization; four of the five highest-repair-rate agents used both dynamic verification and ranking, while Agentless surpassed many agentic approaches.Source: §4.8.8 · p. 30

5.1 Agent Framework

• The survey organizes SE agents around planning, perception, memory, action, and foundation-model effects, grounded in a common LLM-agent framework,, [29].This framework forms the basis for the agent-perspective analysis of existing LLM-based systems for software engineering.Source: §5.1 · p. 30

5.1.1 Planning

• Planning strategies trade efficiency against reliability: single planners reduce overhead, whereas multi-planner collaboration corrects errors but suits complex tasks despite added token and time costs.Single-planner methods have broader applicability but greater hallucination risk; multi-planner systems integrate specialized perspectives and mutual correction,,,,,,.Source: §5.1.1 · p. 31
• Multi-turn and multi-path planning improve adaptation and trial-and-error, but accumulating trajectories, stochasticity, and extra costs can cause context overflow, hallucinations, or limited practicality.Multi-turn planning revises plans using environmental feedback, while multi-path planning generates, selects, switches, or aggregates alternatives,,.Source: §5.1.1 · pp. 31–32
• Planning remains difficult to evaluate and execute reliably: hallucinations, weak complex-task decomposition, and limited fine-grained assessment persist, while human plan fragments substantially improve competitive-programming performance from 26.9% to 74.5% and 47.5% to 80.8%.The reported gains occur on novel problems in Flows, while CoCoST finds planning contributes less as individual function-code complexity increases.Source: §5.1.1 · p. 32

5.1.2 Memory

• Memory preserves task trajectories and intermediate artifacts so agents can maintain coherent reasoning across iterative SE development and maintenance workflows.Short-term memory stores dialogue, action-observation-critique records, or intermediate outputs; long-term memory distills prior-task experiences through summaries, shortcuts, or selected vital data.Source: §5.1.2 · pp. 32–33
• Memory may be agent-specific or shared, with shared memory functioning as an information-exchange hub for intermediate results, role-based artifacts, and decentralized discussions.MetaGPT uses a shared message pool, while FlowGenscrum stores discussion comments from participating agents for later user-story extraction.Source: §5.1.2 · p. 33
• Memory writing distills or filters trajectories to reduce context and token burden, while reading uses reflection, retrieval, subscription, and recency, relevance, or similarity criteria.Examples include sliding-window storage in Reflexion, shortcut extraction in Co-Learning, and role-based message retrieval in MetaGPT and AgileCoder.Source: §5.1.2 · pp. 34–35
• Memory design still faces abstraction and context-matching problems, and its effectiveness lacks fine-grained evaluation beyond limited ablation studies and final task outcomes.Future evaluation could measure historical-question accuracy and memory-related costs independently of overall task performance,,.Source: §5.1.2 · p. 35

5.1.4 Action

• Agent systems use retrieval, file and GUI operations, static and dynamic analysis, testing, fault localization, and version-control tools to interact with software artifacts and improve outputs.Retrieval draws on Web or local knowledge bases; analysis tools provide structural and runtime information, while testing and quality checks supply feedback for validation, generation, repair, and improvement.Source: §5.1.4 · pp. 36–37
• Tool-invocation overhead varies with tool type and task complexity, so systems limit processing time, search results, or invocation counts for expensive operations.Method-level generation may take seconds, whereas system-level testing, fault localization, and end-to-end development can take minutes to hours.Source: §5.1.4 · p. 38

5.1.5 Foundation LLMs.

• Foundation LLMs generally need instruction following, planning, multi-turn dialogue, and tool-use capabilities, while fine-tuning requirements can restrict some agents to open-source models.Tool-using models must understand specifications, propose calls, and incorporate results; systems such as ToolCoder and ToolGen require model modification for API-related behavior.Source: §5.1.5 · pp. 38–39
• Foundation LLMs’ context windows can remain insufficient for repository-level tasks, while complex multi-turn maintenance requires strong long-term reasoning and planning.CodeS reports that even a 200K-token context window may be insufficient; end-to-end maintenance agents rely on models such as GPT-4, GPT-4o, and Claude-3.5-Sonnet.Source: §5.1.5 · p. 39
• Although foundation LLMs are replaceable in principle, architecture cannot always bridge model-quality gaps, making comparable foundation-model capability important in practice.Self-Refine performs well with GPT-series models but struggles with Vicuna-13B, including failures to produce correctly formatted feedback, repeated outputs, and hallucinated conversations.Source: §5.1.5 · p. 39

5.2 Multi-agent System

• 59.7% of surveyed SE agents are multi-agent systems that divide specialized roles and coordinate agents to address complex, end-to-end tasks.These systems are especially suited to activities spanning multiple software-engineering phases.Source: §5.2 · p. 39

5.2.1 Agent Roles

• Agent roles span managers, requirements analysts, designers, developers, quality-assurance specialists, deployment agents, and assistants, with managers planning, assigning, coordinating, and organizing teams.Managers may dynamically create or assemble agents, while downstream roles transform requirements into designs, code, tests, reviews, and fixes.Source: §5.2.1 · pp. 39, 41

5.2.2 Collaboration Mechanism

• Collaboration uses layered, circular, star-like, tree-like, and mesh structures, but scaling agent populations can cause bottlenecks and performance saturation.Systems limit derived agents or task scale; one study reports saturation on benchmarks including MMLU and HumanEval regardless of collaboration structure.Source: §5.2.2 · p. 42

5.2.3 Information Flow.

• Information flows through widely used unidirectional transfer or bidirectional chat, with some systems combining staged data handoffs and same-stage dialogue.Unidirectional transfer supports decoupled individual contexts, whereas bidirectional chat enables collaborative discussion through shared dialogue history.Source: §5.2.3 · p. 43

5.2.4 Real-world Applications

• Real-world applications include commercial systems such as MultiDevin and Amazon Bedrock, alongside open-source frameworks CrewAI, Swarm, and AgentScope.MultiDevin supports a manager Devin and up to ten worker Devins for parallel tasks, while the open-source systems provide role assignment, orchestration, handoffs, or distributed messaging.Source: §5.2.4 · p. 43, Table 15 · p. 43

Overview

• Human participation in LLM-based SE agents primarily occurs during planning, requirements, development, and evaluation to address workflow bottlenecks and align outputs with user intent.Most systems still aim for maximum automation, but prior studies report bottlenecks during software development,.Source: §5.3 · p. 43

Planning Phase

• During planning, users can inspect and revise workflows or provide oracle plans, while design revision remains optional when it requires substantial expertise,,,.Flows found that human-AI collaboration using a brief natural-language oracle plan outperformed other AI feedback-based approaches.Source: §5.3 · pp. 43–44

Requirements Phase

• During requirements refinement, systems detect ambiguity, generate targeted clarification questions, and incorporate human responses or user-assessed use-case revisions to better match intent,.ClarifyGPT uses code consistency checks before producing the final code solution.Source: §5.3 · p. 44

Development Phase

• During development, users refine generated repositories or correct sub-step code outputs, helping agents fix errors and guide subsequent actions,.CodeS Extended supports editing at each layer of a three-layer repository sketch.Source: §5.3 · p. 44

Evaluation Phase

• During evaluation, users manually test final systems and use test reports to trigger refinements, as in AISD and Prompt Sapper.Human intervention occurs at the acceptance-testing phase.Source: §5.3 · p. 44

Discussion

• Research remains limited because collaboration is usually predefined as fixed, iteration-specific intervention points, with human-driven workflows and multi-person team scenarios still unexplored.AISD exemplifies the agent-centric pattern by letting users revise use cases, adjust plans, and validate software within each development cycle.Source: §5.3 · p. 44

Evaluation of Agents for SE

• Fine-grained evaluation should complement final success rates by measuring erroneous actions, debugging iterations, backtracking, phase progress, and individual agent modules.Current evaluations mainly assess task success on benchmarks, making failures and decision processes difficult to diagnose; trustworthy attributes and efficiency also require greater attention.Source: §6 · p. 44
• More realistic, high-quality benchmarks are needed because vague issue descriptions, small generated projects, and relatively easy tasks poorly represent real-world software engineering demands.77.8% of SWE-bench tasks can be completed by an experienced engineer within an hour, underscoring the benchmark’s gap from real-world complexity.Source: §6 · p. 45

Human-Agent Collaboration

• Human participation should extend beyond requirements, planning, coding, and evaluation to architecture design, test generation, code review, and end-to-end maintenance.Effective collaboration also requires interfaces that organize intermediate outputs and collect feedback without inefficiently presenting complex repositories as flat file lists.Source: §6 · p. 45

Perception Modality

• Agents should explore multimodal perception beyond predominantly textual inputs, including images, voice commands, and user gestures, to improve coding-assistant flexibility and accessibility.Only one cited work, VisionDroid, had used a multimodal LLM, while image-based perception remains nascent.Source: §6 · p. 45

Applying Agents for More SE Tasks

• Future agents should address underexplored software engineering phases such as design, verification, and feature maintenance, which require architectural understanding, system reasoning, and informed decisions.These demands extend beyond basic code generation and may exceed the capabilities of traditional LLM-controlled agents.Source: §6 · p. 45

Training Software-oriented LLMs for SE Agents

• Software-oriented LLMs should train on lifecycle data beyond code, including architecture, developer communication, historical changes, and dynamic runtime information.Such data could support models better suited to software engineering and provide foundations for agents covering a wider range of tasks.Source: §6 · p. 45

SE Expertise in Building Agents

• Agent design should more comprehensively integrate SE tools, process models, and quality-assurance techniques to improve effectiveness, robustness, efficiency, interpretability, replicability, and trustworthiness.Trustworthiness assurance must consider the backbone LLM, attached modules, underlying software, and complete systems across diverse tasks; Agentless and OpenAI show that simpler SE workflows can outperform more autonomous agents.Source: §6 · p. 46

Priorities of different Research Directions

• Standardized task-specific benchmarks and metrics should receive highest priority, while richer collaboration, modalities, tasks, and SE expertise are feasible nearer-term directions and software-oriented LLM training remains long-term.Training requires extensive lifecycle data, human effort, new documentation practices, suitable data formats, and effective methodologies.Source: §6 · p. 46

7.1 Disparity of LLM-based Agents and Standalone LLMs in SE Tasks

• LLM-based agents generally achieve stronger SE performance than standalone LLMs by reducing hallucination and stochasticity through planning, feedback, and environmental interaction.Studies report consistently higher pass@1 across method-, competition-, and repository-level code-generation benchmarks, with similar findings in testing, debugging, and static code checking,,,,, [13],,,.Source: §7.1 · p. 46
• Standalone LLMs are easier to adopt but cover fewer complex, environment-dependent tasks, whereas agents extend applicability to IT operations, end-to-end development, and maintenance [2], [3].Standalone LLMs lack dynamic environmental interaction, while LLM-based agents have been applied to broader SE scenarios,,,,,.Source: §7.1 · p. 46

7.2 Threats to Validity

• The survey may omit relevant studies because manual screening involves subjective judgment despite independent reviews and third-author disagreement resolution.Such omissions could reduce comprehensiveness and bias the survey’s findings.Source: §7.2 · p. 47
• Evidence is also insufficient for some memory formats: structured messages have only one peer-reviewed example among cited works, while image storage has been explored only in unpublished VisionDroid.More empirical and peer-reviewed studies are needed to establish the effectiveness and generality of these techniques.Source: §7.2 · p. 47
• Evidence remains limited for multi-agent requirements engineering, knowledge-enhanced bug detection, iterative coverage improvement, and visual input because surveyed works have uneven or low peer-reviewed publication status.Two of four requirements-engineering works were not published in peer-reviewed venues,,. Only one of four knowledge-enhanced bug-detection works,,,, one of three iterative-coverage works,,, and one of three visual-input works,, were published.Source: §7.2 · p. 47

8 CONCLUSION

• The survey analyzes LLM-based agents across software development and maintenance activities and examines agent-component design for SE, while identifying open challenges and future directions.Its coverage spans both applications in SE and the design of LLM-based agent components.Source: §8 · p. 47