PaperlayerView exact revision on arXiv

Source-linked AI summary

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, Huaming Chen

arXiv:2408.02479v2cs.SEcs.AIcs.CL

TL;DR

Prior surveys cover only parts of software engineering and do not clearly distinguish LLMs from LLM-based agents. This survey reviews both across six domains, comparing tasks, benchmarks, and metrics, and analyzes 139 collected papers and their experimental models.

  • Problem

    Prior surveys cover only some software-engineering tasks and do not clearly distinguish LLM applications from LLM-based agents.

    Source: A. Existing Works · p. 2

  • Method

    The survey reviews LLMs and agents across six software-engineering domains, analyzing their tasks, benchmarks, metrics, and experimental models.

    Source: §I · p. 2, §XI · p. 46

  • Results

    The authors report analyzing 139 collected papers and comparing LLM and agent contributions across the six domains.

    Source: §XI · p. 46

  • Takeaways & Limitations

    ConDefects and EvalPlus are described as emerging benchmarks intended to mitigate data leakage and improve real-world representativeness, with fine-grained evaluation potentially suiting future multi-agent testing.

    Source: 1) Benchmarks · p. 44

Abstract

from arXiv · show

With the rise of large language models (LLMs), researchers are increasingly exploring their applications in var ious vertical domains, such as software engineering. LLMs have achieved remarkable success in areas including code generation and vulnerability detection. However, they also exhibit numerous limitations and shortcomings. LLM-based agents, a novel tech nology with the potential for Artificial General Intelligence (AGI), combine LLMs as the core for decision-making and action-taking, addressing some of the inherent limitations of LLMs such as lack of autonomy and self-improvement. Despite numerous studies and surveys exploring the possibility of using LLMs in software engineering, it lacks a clear distinction between LLMs and LLM based agents. It is still in its early stage for a unified standard and benchmarking to qualify an LLM solution as an LLM-based agent in its domain. In this survey, we broadly investigate the current practice and solutions for LLMs and LLM-based agents for software engineering. In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. We review and differentiate the work of LLMs and LLM-based agents from these six topics, examining their differences and similarities in tasks, benchmarks, and evaluation metrics. Finally, we discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering. We anticipate this work will shed some lights on pushing the boundaries of LLM-based agents in software engineering for future research.

I. INTRODUCTION

LLMs offer useful software-engineering capabilities but face limits that LLM-based agents address through tools and greater autonomy. This survey distinguishes the two across six software-engineering domains, responding to gaps in prior surveys’ coverage and comparisons.

I. INTRODUCTION

  • LLMs’ limited context, hallucinations, and inability to use external tools constrain their use in dynamic software-engineering work, while agents combine LLMs with tools and resources for more autonomous tasks.The survey gives autonomous debugging, refactoring, and adaptive test generation as examples of agent capabilities.Source: §I · p. 1
  • The survey presents itself as the first review of the integration and transformation of LLMs into agents in software engineering, spanning six domains from requirements to security and maintenance.Its domains include code generation, autonomous learning and decision-making, software design and evaluation, and test generation.Source: §I · p. 2

A. Existing Works

  • Earlier surveys cover only some software-engineering tasks and do not clearly distinguish LLM applications from LLM-based agents,,.The paper also notes that earlier surveys span publication periods with substantial content disparities, while requirements-engineering research was sparse in prior coverage.Source: A. Existing Works · pp. 2–3
  • The authors report collecting 139 relevant papers and offer a comparative overview of LLM and agent contributions across the six software-engineering domains.They note that the field’s novelty and lack of standardized benchmarks motivate a detailed review intended to guide future research.Source: A. Existing Works · p. 3

B. Methodology

The survey searches DBLP and arXiv for recent work, filters papers by relevance and stated selection criteria, and manually screens the results. It reports 139 relevant papers and describes their distribution across software-engineering domains and venues.

1) Database Selection and Rationale:

  • The survey searches DBLP and arXiv for publications from the latter half of 2023 through December 2024, citing DBLP’s curated coverage and arXiv’s rapid access to preprints.The authors say IEEE Xplore and ACM Digital Library could complement the search with more peer-reviewed conference and journal papers.Source: B. Methodology · pp. 3–4, 1) Database Selection and Rationale · p. 4

2) Paper Retrieval Process:

  • Retrieval combines software-engineering topic keywords with LLM-related terms using Boolean searches, with OR within topics and AND between topic and LLM terms.The paper describes this as a keyword-based process intended to identify relevant software-engineering work.Source: 2) Paper Retrieval Process · p. 4

3) Selection Criteria:

  • Selection requires substantial work focused on software engineering and LLMs, while excluding papers unrelated to those topics or shorter than seven pages.The authors also describe checking titles, abstracts, and, when necessary, full texts against inclusion and exclusion criteria.Source: 3) Selection Criteria · p. 4

4) Manual Screening and Paper Distribution:

  • 139 relevant papers were identified; the domain-distribution figure counts some papers in multiple fields, so its total instances exceed 139.Figure 2 presents the distribution across six software-engineering domains and the proportion of agent studies.Source: 4) Manual Screening and Paper… · p. 4
  • 59.7% of papers came from established peer-reviewed venues, while arXiv accounted for 40.3%; reported venues include NeurIPS at 10.1% and ICSE at 6.5%.The survey also reports ESEC/FSE at 3.6% combined, ICLR at 2.9%, and IEEE RE Conference at 2.9%.Source: 4) Manual Screening and Paper… · pp. 4–5, Figure 3 · p. 5

C. Overall Structure of the Work

The paper introduces LLM and agent foundations, then compares studies across six software-engineering domains, followed by a general discussion and conclusion.

C. Overall Structure of the Work

  • The paper introduces LLM and agent foundations, compares studies across six software-engineering domains, and concludes with a general discussion.Section II covers background; Sections III–IX compare datasets, tasks, benchmarks, and metrics, Section X discusses, and Section XI concludes.Source: C. Overall Structure of the Work · p. 5

III. PRELIMINARIES

  • The preliminaries cover LLM-based agents, including single- and multi-agent systems, their applications, and distinctions in software engineering.They also introduce foundational LLM concepts, framework evolution, and architectures.Source: §III · p. 5

A. Large Language Model

  • The LLM overview traces NLP development from early dialogue systems through neural networks and recurrent models to Transformers and GPT.It describes Transformers’ self-attention and positional encoding, then GPT’s pretraining and fine-tuning approach.Source: A. Large Language Model · p. 5

B. Model Architecture

  • The architecture overview contrasts encoder-decoder, encoder-only, and decoder-only designs, illustrated by CodeT5+, BERT, GPT, and LLaMA.The passages associate encoder-decoder models with translation, encoder-only models with contextual analysis, and decoder-only models with text generation and sequence prediction.Source: B. Model Architecture · p. 6

C. Large Language Model Based Agent

  • LLM-based agents extend language models with interaction, tools, retrieval, or experience-based learning, while data augmentation supplies varied training examples.The described augmentation approaches include synonym replacement, back-translation, paraphrasing, and synthetic data generation; RAG retrieves relevant indexed documents to augment prompts.Source: C. Large Language Model Based Agent · pp. 6–7

D. Single and Multi-Agents

The section contrasts single- and multi-agent systems, then proposes criteria for distinguishing LLM-based agents in software engineering.

D. Single and Multi-Agents

  • Single agents suit tasks accepting general answers, but long inputs can produce inconsistent responses, and hallucinations cannot be fully eliminated.Prompt engineering and tool interventions can mitigate hallucination effects but not entirely eliminate them.Source: D. Single and Multi-Agents · pp. 7–8
  • Multi-agent systems coordinate specialized LLMs or agents for complex tasks, dividing work across models with different areas of focus.The survey notes growing use of multi-agent systems in software-engineering experiments.Source: D. Single and Multi-Agents · p. 8
  • The described multi-agent benefits include shared context, division of labor, cross-checking outputs, and scaling through specialized agents.The passages associate these capabilities with more coherent responses, error correction, and handling complex tasks.Source: D. Single and Multi-Agents · p. 8

E. LLM in Software Engineering

  • Software-engineering examples include agent-based repair, simulated agents, and multi-agent document generation, alongside work across testing, design, security, and maintenance.The cited examples include RepairAgent for automatic repair and a multi-agent system assigning roles in document generation.Source: E. LLM in Software Engineering · p. 8
  • An LLM architecture qualifies as an agent when it meets most criteria; reasoning, planning, tool use, and evaluation are fundamental, while learning and multi-turn interaction are advanced but optional.The criteria synthesize recent literature and are intended to support consistent evaluation of diverse frameworks.Source: E. LLM in Software Engineering · p. 8

IV. REQUIREMENT ENGINEERING AND AND DOCUMENTATION

The section surveys LLM and LLM-agent applications across requirement engineering, contrasting task-focused generation and assessment with collaborative, iterative workflows. It also notes that agent research remains limited and concentrated on generating and refining requirements documents.

A. LLMs Tasks

  • GPT-3.5 Few-Shot achieved the best overall classification performance in a five-dataset comparison, while GPT-4 Zero-Shot was strong for OnlyFunctional requirements.The comparison included PROMISE, Dronology, ReqView, Leeds Library, and WASP; PRCBERT also achieved an F1 score of 96.13% on PROMISE.Source: A. LLMs Tasks · p. 9
  • ChatGPT can assist requirement elicitation, with effectiveness varying by domain expertise and task complexity; GPT-4 artifacts rated highly for understandability and specifiability under RUST.Studies also report that the human-generated SRS was overall superior, although CodeLlama sometimes outperformed GPT-4 in specific categories.Source: A. LLMs Tasks · pp. 9–10
  • SpecGen generated 70% of program specifications, while PathOCL improved OCL constraint generation against UML-Augmented prompting on Validity@K and Correctness@K.PathOCL injects contextual navigation paths into GPT-4 prompts; SpecGen uses mutation operators and heuristic selection.Source: A. LLMs Tasks · p. 10
  • Prompt patterns can produce more structured, modular results, while BERT-based methods address missing requirement terms and contextual term ambiguity.BERT’s MLM achieved 82% precision for predicted missing terms; BERT and K-means clustering were used to detect terms with different contextual meanings.Source: A. LLMs Tasks · p. 10

B. LLM-based Agents Tasks

  • AISD iteratively refines use cases and code with user feedback, raising use case pass rates to 75.2% versus 24.1% without human involvement; agent prototypes also support safety requirements and agile user stories.The safety-requirements prototype completed within a day across three design iterations, compared with months manually.Source: B. LLM-based Agents Tasks · p. 11
  • Agent workflows extend from Graph-RAG compliance checks, which achieved an 87.93% F1-score, to role-specific systems that automate requirement tasks with human expertise retained for validation.The role-specific workflow includes Product Owner, QA, Developer, and Manager agents.Source: B. LLM-based Agents Tasks · p. 11

C. Analysis

  • The survey describes agent-based requirements work as limited and focused mainly on collaborative generation and refinement, with agents using shared databases to improve documents from different professional perspectives.Figure 5 contrasts an agent architecture with prompt engineering and LLMs alone, while omitting more detailed architectural aspects.Source: C. Analysis · pp. 11–12, Figure 5 · p. 12

D. Benchmarks

The survey describes limited public and standardized datasets for requirement engineering, alongside a shift toward workflow-oriented agent benchmarks and behavioral evaluation. It also introduces code generation and software development as a broader application area for LLMs and agents.

D. Benchmarks

  • Requirement-engineering studies lack abundant public datasets, and many rely on author-collected data, practical examples, or case studies.The section describes datasets including small task-specific collections and CAASD’s 72 development tasks, as well as synthetic user stories evaluated with six agile teams.Source: D. Benchmarks · p. 12
  • Agent requirement-engineering datasets often rely on practical projects and case studies, lacking standardization and large scale, despite their flexible, targeted validation.The survey expects broader and deeper development as more public datasets are constructed and shared.Source: D. Benchmarks · pp. 12–13
  • Benchmarking shifts from small, isolated LLM tasks toward realistic agent workflows involving multi-turn reasoning, collaboration, traceability, and system-level criteria.SRS Broker and SRS Aero [82] address regulatory compliance and interdependencies; role-driven settings include Sami et al..Source: D. Benchmarks · p. 13

E. Evaluation Metrics

  • LLM evaluation commonly uses classification metrics and human ratings of generated requirements, whereas agent evaluation adds response time, standards alignment, interpretability, and role-based feedback.AISD also uses system-level measures such as use case pass rate, code consistency, and human-LLM iteration count.Source: E. Evaluation Metrics · p. 13

V. CODE GENERATION AND SOFTWARE DEVELOPMENT

  • The survey presents code generation as an established LLM application and notes that agents can undertake complex generation, autonomous learning, and continuous refinement in development settings.It cites GitHub Copilot as an LLM-integrating tool associated with programming efficiency and code quality.Source: §V · p. 14

A. LLMs Tasks

LLM code-generation work spans direct generation, feedback-driven refinement, and interactive tools, while agent systems add clarification, role specialization, context management, and tool use for broader development workflows.

A. LLMs Tasks

  • SQL-PaLM reached 77.3% test accuracy and 82.7% execution accuracy; CodeGeeX outperformed other multilingual models on HumanEval-X.SQL-PaLM used few-shot prompting and instruction fine-tuning for text-to-SQL tasks.Source: A. LLMs Tasks · p. 15

B. LLM-based Agents Tasks

  • GitHub Copilot developers completed HTTP-server tasks 55.8% faster, while SWE-agent's agent-computer interface achieved 12.5% pass@1 on SWEbench and 87.7% on HumanEvalFix.SWE-agent's interface provides LM-friendly commands, immediate feedback, and guardrails for common software tasks.Source: A. LLMs Tasks · p. 15, B. LLM-based Agents Tasks · p. 18

A. LLMs Tasks

  • TICODER achieved 84% vs. 40% correctness in a 15-programmer study and 45.73% average absolute pass@1 gains across MBPP and HumanEval within five interactions.The framework generates tests for users to validate, then refines and ranks code suggestions from their responses.Source: A. LLMs Tasks · p. 15
  • ALGO uses an LLM-generated brute-force oracle to verify candidate programs, increasing one-shot pass rate 8× over Codex and 2.6× over CodeT.Its verifier compares outputs on generated inputs to guide an efficient coder.Source: A. LLMs Tasks · p. 16

B. LLM-based Agents Tasks

  • SDM assigns ChatGPT agents to analysis, coding, and testing, with reported HumanEval improvement up to 29.9% over GPT-4; MapCoder adds retrieval, planning, coding, and debugging roles.MapCoder's debugging agent iteratively refines code using sample input-output pairs.Source: B. LLM-based Agents Tasks · p. 16
  • SoA surpassed Reflexion by 5% in HumanEval Pass@1, while L2MAC manages memory and execution context for large-codebase generation and scored 90.2% Pass@1 on HumanEval.SoA scales the number of self-organized agents with task complexity; L2MAC uses a control unit to check syntactic and functional correctness.Source: B. LLM-based Agents Tasks · pp. 16–17
  • MetaGPT simulates a waterfall lifecycle through SOP-based agent collaboration, while AGILECoder coordinates Agile roles in sprints and outperformed prior multi-agent systems on pass@1 and executability.AGILECoder's Dynamic Code Graph Generator tracks cross-file dependencies; its results were especially strong on large-scale, repository-level tasks.Source: B. LLM-based Agents Tasks · p. 17
  • ClarifyGPT detects ambiguous requirements through divergent sampled completions, asks targeted questions, and raises Pass@1 by 11.66% with GPT-4 and 15.00% with ChatGPT.It refines the requirement after responses from users or a high-fidelity simulator, then prompts the LLM to generate code.Source: B. LLM-based Agents Tasks · p. 18

C. Analysis

The survey contrasts single-model workflows with agent systems that coordinate roles, integrate tools, and simulate development lifecycles; it also describes differences in benchmark choice and evaluation metrics.

C. Analysis

  • Agent frameworks distribute planning, coding, testing, and debugging across roles, using iterative feedback to extend context and address cross-file dependencies.Traditional LLM methods often rely on a single model and human-provided tests or clarifications; agent systems can model waterfall or agile cycles.Source: C. Analysis · p. 18
  • Figure 6 contrasts specialized roles coordinating iterative implementation, testing, and refinement with traditional LLMs processing prompts or analyses in isolation.The figure's agent workflow highlights continuous coordination and tool-driven synergy, while single-model approaches can still perform well on established pass@k benchmarks.Source: C. Analysis · pp. 18–19, Figure 6 · p. 19

D. Benchmarks

  • HumanEval and MBPP are common code-generation benchmarks, while agent studies also favor multi-turn, iterative, and tool-use evaluations such as ToolBench and APIBench.The section also describes cross-language and cross-domain datasets, including HumanEval-X and CodeSearchNet.Source: D. Benchmarks · p. 19

E. Evaluation Metrics

  • LLM evaluations include Pass@k, BLEU, completion time, and execution rate, whereas agent evaluations add collaboration, cost, human revision, and user-satisfaction measures.The survey notes that both paradigms use traditional accuracy and efficiency metrics, but agent evaluation also considers the multi-agent system and broader development process.Source: E. Evaluation Metrics · p. 20

VI. AUTONOMOUS LEARNING AND DECISION MAKING

The section reviews LLM and agent work in autonomous learning and decision-making, including feedback-based debugging, voting inference, creativity assessment, and LLM-as-judge evaluation.

VI. AUTONOMOUS LEARNING AND DECISION MAKING

  • The survey describes majority voting as a way to improve inference accuracy and reviews 23 LLM-agent studies on autonomous learning and decision-making.It also cites automated debugging and self-correction as LLM applications in this area.Source: §VI · p. 20

A. LLMs Tasks

  • Voting-inference performance initially improves with more LLM calls, then declines beyond a threshold, while call costs continue to rise.The reported nonmonotonic trend is attributed to the diversity of query difficulties.Source: A. LLMs Tasks · p. 20
  • SELF-DEBUGGING reported a 212% accuracy increase on Spider and TransCoder by analyzing execution results and natural-language explanations to debug code.AutoSD generates explainable patches, while ALGO uses generated brute-force oracles and output comparisons to guide code synthesis without external supervision.Source: A. LLMs Tasks · pp. 20, 22
  • LLMs showed limitations in combinatorial, exploratory, and transformative creativity, while GPT-4 judgments were highly consistent with human judgments across tasks.The latter evaluation used MT-Bench and Chatbot Arena.Source: A. LLMs Tasks · p. 22

B. LLM-based Agents Tasks

The survey describes LLM-based agents as supporting collaborative task execution, tool use, and learning from interaction, in contrast to LLMs’ more static, task-specific operation. It presents agent work across decision-making, code generation, and software-engineering workflows.

B. LLM-based Agents Tasks

  • Multi-agent approaches span discussion, collaboration, and role-based coordination, with studies reporting gains in reasoning accuracy, coding performance, and WebShop performance.CMD reports improved commonsense and mathematical reasoning; AGENTVERSE reports collaboration gains especially in coding; BOLAA outperforms other architectures in WebShop.Source: B. LLM-based Agents Tasks · pp. 22–23
  • Reflexion improves through language feedback rather than weight updates, while ExpeL collects experience from training tasks and recalls past successes and failures for cross-task transfer.Reflexion’s first-pass success rate on HumanEval Python programming increased from 80.1% to 91.0%; ExpeL does not require parameter updates.Source: B. LLM-based Agents Tasks · p. 22
  • AgentLite and GPTSwarm provide development frameworks, Iris supplies context-aware programming tutoring, and GoEx adds reversible, isolated execution for agent actions.GPTSwarm reports up to 90.2% improvement on GAIA over the best existing methods; GoEx supports actions across RESTful APIs, databases, and file systems.Source: B. LLM-based Agents Tasks · pp. 23–24
  • CodeAct, CODEAGENT, Agile, and ToolCoder connect code generation with decision-making through control-flow reasoning, task decomposition, coordination, or structured tool-planning scaffolds.ToolCoder converts user tasks into Python function scaffolds for modular decomposition, tool selection, and action planning.Source: B. LLM-based Agents Tasks · p. 24

C. Analysis

  • The analysis contrasts LLMs’ static, well-defined tasks with agents’ management of multiple tasks involving dynamic decisions, interaction, and multimodal environments.Examples include learning from experience, multi-agent discussion, and real-time decisions.Source: C. Analysis · p. 24
  • LLMs’ learning is described as bounded by training data and parameters, whereas agents use interaction, feedback, and experience; ExpeL’s memory and insights pools support later decisions.The survey identifies ExpeL as combining ReAct and Reflexion and illustrates the framework in Figure 7.Source: C. Analysis · pp. 24–25, Figure 7 · p. 24

D. Benchmarks

The survey distinguishes task-specific datasets and metrics for LLMs from the more interactive settings and broader evaluations used for agents. It also describes software design and evaluation as overlapping with development, requirements, and autonomous decision-making.

D. Benchmarks

  • LLM benchmarks include Defects4J, MMLU, TransCoder, and MBPP for specific domains, while agent benchmarks include HotpotQA, ALFWorld, FEVER, WebShop, and MGSM for complex or interactive tasks.The survey characterizes LLM datasets as typically focused on static tasks and agent datasets as emphasizing multitasking, dynamic settings, and multimodal inputs.Source: D. Benchmarks · p. 25

E. Evaluation Metrics

  • LLM evaluations commonly use accuracy, sample efficiency, patch metrics, and creativity measures, while agent evaluations also assess perception, cognition, action, task success, practitioner feedback, efficiency, and accuracy.The survey presents the broader agent metrics as reflecting multimodal, collaborative, and practical application settings.Source: E. Evaluation Metrics · p. 26

VII. SOFTWARE DESIGN AND EVALUATION

  • Software design and evaluation overlap with code development and requirements engineering, and the survey notes shared use of tools, APIs, reinforcement learning, and prompt engineering across related topics.Table VIII is identified as covering evaluation metrics in autonomous learning and decision-making.Source: §VII · pp. 26–27, Table VIII · p. 27

A. LLMs Tasks

The survey reviews LLM applications in software tasks and agent frameworks, then compares their scope and benchmarks. It reports task-specific LLM results alongside agent approaches for collaboration and more complex workflows.

A. LLMs Tasks

  • 100% success rates were reported for ChatGPT on both log summarization and pronoun resolution, while EvaluLLM uses pairwise output comparisons and win rates instead of traditional reference-based evaluation.The survey also notes relatively poor ChatGPT performance on code review and vulnerability detection.Source: A. LLMs Tasks · p. 27
  • In a comparison with Mockplus, RaWi improved precision@k by 40%; separately, GPT-4 generated HLS designs with predefined errors for Chrysalis, a resource for evaluating and optimizing LLM-based HLS debugging assistants.Source: A. LLMs Tasks · p. 28
  • ChatGPT answered about 77.5% of software-testing-course questions, but only 55.6% were correct or partially correct and explanation correctness was 53.0%.Source: A. LLMs Tasks · p. 28

B. LLM-based Agents Tasks

  • ChatDev uses a chat-based multi-agent framework to coordinate software development, and experiments report an average design-and-generation time of 409.84 seconds at a cost of $0.2967 while reducing code vulnerabilities.The survey also describes agent orchestration frameworks including Flows and HuggingGPT.Source: B. LLM-based Agents Tasks · p. 28
  • Flows reports increases of 21 percentage points in problem-solving rate and 54 percentage points in human-AI collaboration rate on competitive coding tasks.HuggingGPT instead uses ChatGPT to orchestrate task execution by models available through Hugging Face.Source: B. LLM-based Agents Tasks · p. 29

C. Analysis

  • The analysis says LLM applications tend to automate specific tasks, with relatively limited research on higher-level software design, while agents can coordinate roles and adapt across design workflows.The survey notes LLMs remain slightly superior in efficiency and cost for text generation and vulnerability detection.Source: C. Analysis · p. 29

D. Benchmarks

  • LLM research tends to use public, standardized datasets, whereas agent studies more often use customized or unspecified settings; agent benchmarks also cover multimodal and dynamic multi-agent scenarios.LLMARENA is described as assessing spatial reasoning, strategic planning, and risk assessment in dynamic multi-agent environments.Source: D. Benchmarks · pp. 29–30

E. Evaluation Metrics

LLMs are presented as a more cost-saving choice for test generation alone, while LLM-based agents are used for complex tasks involving collaboration and iterative optimization. The survey contrasts their evaluation metrics and reviews testing applications across both approaches.

E. Evaluation Metrics

  • Agent evaluations add collaboration- and interaction-oriented measures to traditional metrics, reflecting their emphasis on multi-agent coordination and decision-making.The survey cautions that metric choice depends on the task and dataset.Source: E. Evaluation Metrics · p. 30

A. LLMs Tasks

  • 24 proof-of-concept vulnerabilities were found in 55 applications by ChatGPT-generated security tests, outperforming TRANSFER and SIEGE.The study varied prompt styles and templates.Source: A. LLMs Tasks · p. 30
  • LLMs can generate test suites through varied prompting and modeling approaches, but ChatGPT averaged 55.4% statement coverage across 207 Java classes versus EvoSuite’s 74.2%.ChatGPT outperformed EvoSuite in generating diverse test data, semantically rich call chains, and context-aware inputs in 17.9% of classes.Source: A. LLMs Tasks · p. 32

B. LLM-based Agents Tasks

  • Multi-agent test-generation systems divide work across specialized agents; AgentCoder reached an 89.9% pass rate on MBPP, while TestChain with GPT-4 achieved 71.79% accuracy on LeetCode-hard and 90.24% on HumanEval.TestChain agents generate diverse inputs, map inputs to outputs through ReAct dialogue chains, and use a Python interpreter to obtain test outputs.Source: B. LLM-based Agents Tasks · p. 32
  • Conversational testing frameworks such as SocraTest use multi-step interaction and middleware to generate and optimize test cases with testing tools.The survey describes this as supporting more advanced automated testing capabilities.Source: B. LLM-based Agents Tasks · p. 32
  • In WeChat Pay acceptance testing, XUAT-Copilot achieved 88.55% Pass@1 and 93.03% Complete@1 across 450 test cases.Its Pass@1 rate was 22.65% for a single-agent system and 81.96% without the reflection component.Source: B. LLM-based Agents Tasks · p. 33

C. Analysis

  • LLMs suit rapid, specific test-generation tasks, while agents target more complex tasks through collaboration and iterative optimization, at greater cost and with system performance sensitive to individual models.The survey notes that a single poorly performing model can affect the entire agent system.Source: C. Analysis · p. 33

D. Benchmarks

The survey describes different dataset emphases for LLM and agent research, alongside varying evaluation practices. Its software security and maintenance review covers program repair, penetration testing, vulnerability detection, and security evaluation.

D. Benchmarks

  • LLM test-generation studies use task-specific datasets, while agent studies draw on broader programming benchmarks and system-level evaluations, including WeChat Pay UAT cases.Agent studies also place more emphasis on qualitative evaluation and user feedback concerning system effectiveness.Source: D. Benchmarks · pp. 33–34

E. Evaluation Metrics

  • LLM studies commonly report bug reproduction, coverage, precision, and recall, whereas agent evaluations also assess qualitative effects, collaboration, and system efficiency.The survey associates these differences with single-task generation versus more complex agent workflows.Source: E. Evaluation Metrics · p. 34

IX. SOFTWARE SECURITY AND MAINTENANCE

  • Software security and maintenance research reviewed here includes vulnerability detection, automated repair, penetration testing, and robustness evaluation, with agent work also addressing dynamic repair tasks.The survey names RITFIS and NAVRepair as examples of agent-related work.Source: §IX · p. 34, A. LLMs Tasks · p. 34

A. LLMs Tasks

  • WizardCoder raised ROC AUC from 0.66 for CodeBERT to 0.69, while GRACE reported a 28.65% F1 improvement over baseline models on vulnerability detection.GRACE uses code property graphs with semantic, syntactic, and lexical similarities.Source: A. LLMs Tasks · pp. 34–35
  • A 7B model scored 68.26% F1 on BigVul but 3.09% on PRIMEVUL, and a study of 2,500 GPT-4-generated PHP applications identified vulnerabilities including injection and XSS risks.The survey reports that existing benchmarks overestimated code-language-model vulnerability detection performance and that generated applications often lacked sanitization.Source: A. LLMs Tasks · p. 36
  • 26% higher repair accuracy was reported for NAVRepair than existing methods on C/C++ program repair.NAVRepair combines AST node-type information with CWE-derived vulnerability templates.Source: A. LLMs Tasks · p. 36
  • CodeLlama improved by 11% on evalrepair-C++ and 8% on EvalRepair-Java for its first 10 repair suggestions, while SRepair repaired 300 Defects4J single-function errors with at least 85% improvement over previous techniques.The survey also reports RTT’s repair effects across benchmarks and PyDex’s evaluation on 286 student programs.Source: A. LLMs Tasks · pp. 36–37
  • PENTESTGPT’s task completion rate was 228.6% higher than GPT-3.5 and 58.6% higher than GPT-4 across evaluations involving 13 targets and 182 subtasks.RITFIS adapted 17 DNN testing methods to assess robustness of LLM-based software against natural-language inputs.Source: A. LLMs Tasks · p. 37

B. LLM-based Agents Task

The survey reviews LLM-based agents for debugging, program repair, smart-contract security, offensive security, and security engineering. It contrasts these systems with traditional LLM approaches, emphasizing agent autonomy, collaboration, and dynamic information use.

B. LLM-based Agents Task

  • FixAgent fixed 78 of 79 QuixBugs bugs, including nine previously unfixed bugs, while LDB improved HumanEval accuracy from 73.8% to 82.9%.FixAgent assigns localization, repair, and analysis to specialized agents; LDB verifies program blocks against task descriptions.Source: B. LLM-based Agents Task · p. 37
  • RepairAgent correctly repaired 164 of 186 Defects4J bugs, while AutoCodeRover and MAGIS use structured context retrieval or specialized-agent collaboration for repository-level issues.MAGIS models a team with Manager, Repository Custodian, Developer, and QA Engineer roles.Source: B. LLM-based Agents Task · p. 38
  • ACFIX repaired 94.92% of access-control vulnerabilities versus 52.54% for GPT-4, and TrustLLM achieved 91.21% F1 and 91.11% accuracy on smart-contract auditing.ACFIX mines RBAC practices and uses debate between generator and verifier agents to check patches.Source: B. LLM-based Agents Task · p. 38
  • An LLM-based agent achieved an 87% success rate in exploiting disclosed, unpatched vulnerabilities when given CVE descriptions.Another agent application automated high-level planning and low-level vulnerability discovery in penetration testing.Source: B. LLM-based Agents Task · p. 38
  • The review also describes a concept-guided strategy in which agents interact with system-model diagrams for safety-analysis tasks.The passage places this work within security engineering and software-system reliability.Source: B. LLM-based Agents Task · p. 38

C. Analysis

  • The comparison attributes agents’ added flexibility to multi-agent collaboration, runtime information tracking, and dynamic analysis, while traditional LLM approaches more often rely on fixed prompts or feedback loops.The MOREPAIR and RepairAgent comparison illustrates model optimization versus tool-assisted inspection, iterative refinement, and self-reflection.Source: C. Analysis · pp. 38–39, Figure 9 · p. 39

D. Benchmarks

The survey finds that benchmarks and evaluation metrics reflect different emphases in LLM and agent research, while model use varies across topics and agent studies often draw on a limited model range.

D. Benchmarks

  • Agent studies pair shared benchmarks with specialized interactive datasets, while LLM studies more often assess static tasks; self-collected benchmarks add flexibility where public datasets may not fit.Both use datasets such as Defects4J, CVE, and HumanEval, but agent work also uses specialized security benchmarks and self-collected datasets.Source: D. Benchmarks · pp. 39–40

E. Evaluation Metrics

  • LLM evaluation emphasizes task-specific correctness and efficiency, whereas agent metrics more often measure overall task completion, time, cost, and API use.LLM studies use measures such as ROC AUC and F1; agent studies also consider completion time and number of API calls.Source: E. Evaluation Metrics · p. 40

A. Experiment Models

  • Across 139 reviewed papers, the survey identified 82 unique models, with GPT-3.5, GPT-4, LLaMA2, and Codex among those frequently used.Closed-source models are also used for comparison or data augmentation, sometimes alongside locally deployed models.Source: A. Experiment Models · p. 40
  • In autonomous learning and decision making, GPT-4 appeared in 12 of 23 papers and GPT-3.5 in 16 of 23, with agent studies often testing multiple models per task.The survey links the higher usage frequency in this topic to evaluating human-like behavior, decision making, and reasoning.Source: A. Experiment Models · p. 42
  • Agent studies use a relatively limited variety of models, while code-generation research draws on a wider range and CodeLlama appears across several software tasks.The survey suggests agent frameworks can reuse general-purpose models, while citing CodeLlama in code, test, and security work.Source: A. Experiment Models · p. 42

B. Topics Overlapping

The survey contrasts where LLM and agent research concentrates, describes overlapping software engineering topics and benchmark choices, and finds that their evaluation metrics differ in emphasis and interpretation.

B. Topics Overlapping

  • Requirement engineering and security account for nearly half of LLM papers, while autonomous learning makes up nearly 30% of agent papers, whose research also emphasizes code generation and development.Agent research in requirements engineering and test generation remains relatively limited.Source: B. Topics Overlapping · p. 42
  • The most frequent topic overlaps are code generation with testing and code generation with autonomous learning, while autonomous learning also intersects substantially with security and maintenance.The survey notes that requirements and code generation, and testing and security, each overlap in one paper.Source: B. Topics Overlapping · pp. 42–43

1) Benchmarks:

  • Public benchmark statistics omit many custom or domain-specific datasets, while HumanEval and MBPP are widely used for code-generation evaluation across both LLMs and agents.The survey identifies FEVER and HotpotQA in agent studies for fact verification and multi-hop reasoning.Source: 1) Benchmarks · pp. 43–44
  • Defects4J is a staple for LLM code repair but rare in agent studies, which increasingly use benchmarks such as SWE-bench, ProjectDev, or API-Bank for multi-stage workflows and tool invocation.ConDefects and EvalPlus are designed to mitigate data leakage and improve real-world representativeness.Source: 1) Benchmarks · p. 44

2) Evaluation Metrics.:

  • Accuracy is the most frequent listed LLM metric at 20.9%, while agent studies most often use Success Rate at 20.0% and also track efficiency, cost, and win rate.The survey says Accuracy can refer to static classification for LLMs and dynamic subtask completion for agents; agent Success Rate often reflects end-to-end multi-step workflows.Source: 2) Evaluation Metrics · p. 44
  • Both paradigms use overlapping metrics, but their execution modes shape metric interpretation, and interactive studies also use qualitative measures such as alignment and understandability.The survey points to context-aware evaluation frameworks as a need for assessing LLM-based agents.Source: 2) Evaluation Metrics · p. 44

D. Challenges and Opportunities

The survey identifies unresolved challenges in defining, evaluating, and deploying LLM-based agents, alongside research directions for more integrated and practical software engineering systems.

D. Challenges and Opportunities

  • The lack of a shared agent definition and evaluation protocol limits reproducibility, horizontal comparison, and a cohesive theoretical foundation.Studies differ on required autonomy, planning, and tool use, and a unified evaluation benchmark is absent.Source: D. Challenges and Opportunities · p. 45
  • Multi-agent systems face synchronization, recovery, and context-sharing problems, while tool integration remains fragile across workflows and lacks evaluation under latency, failure, or partial observability.Role-based collaboration can improve performance, but state consistency and semantic alignment remain open issues.Source: D. Challenges and Opportunities · p. 45
  • Agents rarely transfer knowledge across tasks, interactive simulation data remain scarce, and opaque reasoning and planning hinder trust, particularly in critical settings.Static benchmarks such as HumanEval and MBPP do not capture multi-turn, multi-role behaviors such as negotiation or tool-driven reasoning.Source: D. Challenges and Opportunities · p. 45

E. Opportunities for Future Research

  • The survey proposes a unified agent taxonomy and evaluation framework, with shared definitions and multidimensional measures supporting connected software-development pipelines.It identifies CAASD and API-Bank as foundations for protocol-driven task construction.Source: E. Opportunities for Future Research · p. 45
  • Future work could strengthen long-term planning through structured memory, experience tracking, and self-evaluation.The survey reports that agents with these mechanisms outperform stateless systems in complex debugging and requirement construction tasks.Source: E. Opportunities for Future Research · p. 45
  • Human-in-the-loop workflows and multimodal capabilities could extend agents to role-specific collaboration, visual debugging, GUI operations, and embedded-system control.The survey presents these as directions for moving beyond text-only environments.Source: E. Opportunities for Future Research · pp. 45–46
  • Practical deployment requires project-level context and dependency tracking, with role-aware collaboration as a route beyond function- or module-level completion.The survey notes that most current agents remain focused on function- or module-level code completion rather than real-world project tasks.Source: E. Opportunities for Future Research · p. 46

XI. CONCLUSION

  • Across six software engineering topics, the survey finds that agents have brought extensive research activity with task, benchmark, and metric emphases that differ from traditional LLMs.The review categorizes 139 papers and analyzes their applications, models, datasets, and evaluation metrics.Source: §XI · p. 46

Related papers

Loading 2408.02479v2…