PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Code Generation with LLM-based Agents

Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, Ge Li

arXiv:2508.00083v2cs.SEcs.AIcs.CLcs.LG

TL;DR

Complex software engineering tasks exceed the autonomous planning, environment interaction, validation, and self-correction capabilities of single-response LLM code generation. This survey systematically synthesizes agent technologies, SDLC applications, evaluation methods, and tools, while identifying challenges including unstable context engineering and error cascading. It concludes that development-environment integration and code quality remain central research problems.

  • Problem

    Single-response LLM code generation struggles with complex software engineering tasks requiring decomposition, environment interaction, validation, and iterative self-correction.

  • Method

    The survey organizes LLM-based code generation agents methodologically across single-agent and multi-agent systems and reviews their technologies, SDLC applications, evaluations, and tools.

  • Results

    The survey identifies applications spanning code implementation, debugging and repair, testing, and requirements clarification, alongside mainstream evaluation methods and benchmarks.

  • Takeaways & Limitations

    Code generation agents are characterized by autonomy, broader software-development task coverage, and an engineering-practice focus across the software development lifecycle.

  • Takeaways & Limitations

    Current agents remain limited on domain-specific tasks requiring deep professional reasoning and structured domain knowledge.

Abstract

from arXiv · show

Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm. Distinct from previous code generation techniques, code generation agents are characterized by three core features. 1) Autonomy: the ability to independently manage the entire workflow, from task decomposition to coding and debugging. 2) Expanded task scope: capabilities that extend beyond generating code snippets to encompass the full software development lifecycle (SDLC). 3) Enhancement of engineering practicality: a shift in research emphasis from algorithmic innovation toward practical engineering challenges, such as system reliability, process management, and tool integration. This domain has recently witnessed rapid development and an explosion in research, demonstrating significant application potential. This paper presents a systematic survey of the field of LLM-based code generation agents. We trace the technology's developmental trajectory from its inception and systematically categorize its core techniques, including both single-agent and multi-agent architectures. Furthermore, this survey details the applications of LLM-based agents across the full SDLC, summarizes mainstream evaluation benchmarks and metrics, and catalogs representative tools. Finally, by analyzing the primary challenges, we identify and propose several foundational, long-term research directions for the future work of the field.

1 INTRODUCTION

LLM-based code generation agents address the limits of single-response code generation by autonomously managing broader, engineering-oriented software workflows. This survey reviews their technologies, applications, evaluation methods, tools, and remaining challenges.

  • Motivation: LLM-based agents address native LLM limitations in task decomposition, environment interaction, code validation, and continuous self-correction.These limitations hinder autonomous completion of complex tasks involving cross-file context, dynamic debugging, and iterative optimization.
  • Core distinctions: Unlike passive code generation models, agents can manage development workflows from requirements through implementation, testing, refactoring, and iterative optimization.This shifts developers toward defining tasks, supervising processes, and reviewing final results.
  • Core distinctions: Code generation agents extend beyond bounded snippets to ambiguous requirements and broader software development tasks.Their scope includes entire-project coding, testing, refactoring, and optimization using real-time feedback.
  • Survey scope: The survey organizes recent work by key technologies, single-agent and multi-agent systems, SDLC applications, evaluation methods, and deployed tools.It aims to provide a technical knowledge foundation while integrating recent advances and challenges.
  • Survey scope: The survey differs from prior software-engineering agent surveys by emphasizing methodological classification and integrating rapidly developing research advances and challenges.The authors argue that technical methods are common across varying application scenarios.

2 LITERATURE COLLECTION AND TECHNOLOGICAL DEVELOPMENT TREND ANALYSIS

The survey combines broad academic retrieval, bilingual searches, preprint coverage, and snowballing to study a rapidly expanding research area. Its publication analysis indicates growing attention across software engineering, NLP, and AI venues.

  • Literature collection: The retrieval covered ACM, IEEE, SpringerLink, Google Scholar, DBLP, and CNKI to collect relevant literature comprehensively.The study targeted authoritative academic databases.
  • Literature collection: The search used bilingual Chinese-English keywords across titles, abstracts, keywords, and index terms, with quality restrictions on venues.The strategy targeted combinations of code generation, software development, LLMs, and agents.
  • Literature collection: Preprint platforms and forward-backward snowball searches expanded coverage of recent developments.The authors specifically included high-quality arXiv work because of the field’s rapid development.
  • Literature collection: 447 candidate literature pieces were initially obtained before screening.The screening process applied criteria concerning duplication, preprint impact and innovation, and publication completeness.
  • Technological development trend analysis: Since 2023, related publications have increased year over year and appeared across leading software engineering, NLP, and AI venues.The distribution indicates growing attention from multiple disciplinary fields.

3 BACKGROUND KNOWLEDGE

Code generation maps requirements and other inputs to executable source code, while LLM-based agents add planning, tools, memory, reflection, and iterative interaction. The survey distinguishes centralized single-agent systems from collaborative multi-agent systems.

  • Code Generation in Software Engineering: Code generation converts structured or unstructured inputs into source code that should be syntactically legal, semantically consistent, and executable.Inputs include natural-language requirements, design documents, and existing code snippets.
  • Code Generation in Software Engineering: Traditional code generation struggles with the contextual demands of open, complex, and diverse modern software development tasks.The survey identifies insufficient contextual understanding as a fundamental limitation.
  • Large Language Models: LLMs learn programming syntax and paradigms from code and documentation, enabling code generation from natural-language descriptions as well as completion and refactoring.Their training supports mappings between natural-language descriptions and code logic.
  • LLM-based Agents: LLM-based agents integrate planning, memory, tool usage, reflection, perception, decision-making, and action to execute tasks autonomously.Planning decomposes complex tasks into manageable sub-goals, while memory includes short- and long-term components.
  • Agent system categories: Single-agent systems complete tasks through one centralized agent, while multi-agent systems coordinate multiple agents through communication, collaboration, negotiation, and role-based division of labor.Multi-agent roles can include analyst, programmer, and tester.
  • LLM-based Agents: LLMs provide passive single-response generation, whereas agents support active planning, stateful interaction, execution, observation, and adjustment.This distinction targets complex, ambiguous, and multi-step collaborative software development tasks.

4 KEY TECHNOLOGIES AND METHODS

The survey classifies core methods into single-agent code generation and multi-agent code generation systems. It organizes the technologies chronologically and examines workflows, agent information interaction, and collaborative optimization.

  • Core classification: The chapter divides LLM-based code generation agent methods into single-agent methods and multi-agent systems.Single-agent methods provide the foundation for constructing multi-agent systems.
  • Multi-agent systems: Multi-agent methods are analyzed through workflow arrangement, efficient information interaction and management, and collaborative optimization.These aspects describe how multiple agents become a more capable overall system.
  • Organization: The key technologies and methods are organized chronologically as shown in Figure 2.The chronology covers the field’s technological development trajectory.

4.1 Single-Agent Code Generation Methods

Single-agent code generation methods increasingly combine explicit planning, multi-path exploration, external tools and retrieval, and iterative reflection to address complex software tasks. These approaches expand generation beyond one-shot responses through structured reasoning, execution feedback, and self-improvement.

  • Planning and Search: Explicit planning decomposes complex code-generation problems into high-level steps before executable code is produced.Self-Planning introduced a planning phase, while later methods added clustering, self-revision, search, and tree-structured exploration.
  • Planning and Search: Single-agent methods evolved from linear, single-path reasoning toward multi-path exploration and structured planning with execution-based evaluation and pruning.GIF-MCTS samples multiple generation paths, PlanSearch evaluates candidate plans in parallel, and CodeTree and Tree-of-Code organize exploration and refinement hierarchically.
  • Tool Integration and Retrieval Enhancement: External tools and retrieval extend agents’ perception and execution by supporting API search, code navigation, testing, repository retrieval, and domain-specific simulation.Examples include ToolCoder, CodeNav, AUTOPATCH, AnalogCoder, and related systems that use tool feedback or retrieved code and context.
  • Tool Integration and Retrieval Enhancement: ROCODE forms a closed loop of generation, error detection, and adaptive backtracking by monitoring compilation output during generation.The mechanism uses real-time feedback to initiate backtracking when compilation problems are detected.
  • Reflection and Self-Improvement: Reflection and self-improvement methods review intermediate outputs and iteratively revise code using natural-language feedback, execution results, modular reuse, or multi-solution evaluation.Self-Refine performs self-evaluation without additional training, while CodeChain promotes reuse of verified submodules and later methods incorporate automatic repair and execution feedback.

4.2 Multi-Agent Code Generation Systems

Multi-agent code-generation systems coordinate specialized agents through sequential, hierarchical, circular, or self-evolving workflows. Their effectiveness also depends on role prompts, shared context and memory, and emerging team-level collaborative optimization.

  • Multi-Agent System Workflows: Multi-agent workflows mainly follow pipeline-based division of labor, hierarchical planning-execution, self-negotiation circular optimization, or self-evolving structural updates.These workflow types organize how agents divide tasks, implement solutions, exchange feedback, and adjust system structure.
  • Pipeline-Based Division of Labor: Pipeline systems assign agents to sequential software-development stages, such as requirement analysis, coding implementation, and testing.Self-Collaboration follows a waterfall-like three-stage process, while other systems model project-management and quality-assurance roles for GitHub tasks.
  • Hierarchical Planning-Execution: Hierarchical systems decompose goals through higher-level agents and delegate concrete implementation to lower-level agents.PairCoder separates Navigator planning from Driver implementation, while FlowGen and MAGE use staged or layered structures across software and hardware code generation.
  • Hierarchical Planning-Execution: Self-organizing structures can dynamically expand or contract the number of agents according to task complexity and resource usage, improving management of large and heterogeneous code-generation tasks.SoA uses dynamic agent scheduling, and MAGE decomposes high-level hardware goals into micro-operations assigned to different agents.
  • Self-Negotiation Circular Optimization: Circular optimization systems use repeated negotiation, reflection, evaluation, testing, and repair to improve solution quality or robustness.MapCoder repeats multi-agent interaction, while AutoSafeCoder combines coder, static analyzer, and fuzzer feedback for security-oriented revisions.
  • Context Management and Memory Technologies: Multi-agent systems manage collaboration through role-specific prompts and shared context spaces containing task descriptions, historical modifications, and intermediate products.The blackboard model lets agents read and update structured shared memory, while L2MAC separates instruction registers and file storage to organize longer code contexts.
  • Collaborative Optimization: Collaborative optimization trains multiple agents with team-level interaction modeling, using mutual assessment and feedback to refine prompts, code, tests, and repair advice.CodeCoR repeatedly evaluates and prunes low-quality outputs across four dedicated agents; Lingma SWE-GPT collects multi-stage behavioral data for optimization.

5 APPLICATIONS OF LLM-BASED CODE GENERATION AGENTS IN SOFTWARE DEVELOPMENT TASKS

LLM-based code generation agents now support diverse software-development tasks beyond code production, including debugging, testing, refactoring, optimization, and requirement clarification. Applications increasingly combine autonomous planning, tool use, iterative verification, and multi-agent collaboration across function-level, repository-level, and end-to-end development.

  • Code-generation agents support multiple SDLC stages, including implementation, debugging and repair, test generation, refactoring, optimization, and requirement clarification.
  • Function-Level Code Generation: Function-level systems use planning, chain-based revision, process models, pair programming, search, reflection, and execution-based checking to improve code-generation workflows.
  • Repository-Level Code Generation: Repository-level generation targets large codebases spanning multiple modules and files, including incremental feature development based on existing structures and responsibilities.
  • Automated Debugging and Program Repair: Automated repair is progressing from test-driven patch generation toward autonomous defect diagnosis and semantic repair.
  • Automated Debugging and Program Repair: 164 defects were automatically repaired on Defects4J by RepairAgent, which uses 14 developer-oriented repair tools coordinated through finite state machines.
  • Automated Test Code Generation: TestPilot achieved 52.8% branch coverage on 25 npm packages and 1,684 API functions, improving 27% over Nessie.
  • Automated Test Code Generation: Automated testing agents generate tests and perform execution, evaluation, analysis, and iterative coverage-guided test generation.
  • Automated Code Refactoring and Optimization: Refactoring and optimization systems combine LLM semantic understanding with static analysis, test feedback, performance monitoring, or specialized detection tools.

6 EVALUATION METHODS AND BENCHMARKS

Evaluation has shifted from isolated code snippets toward interactive, multi-step tasks in complete software environments. The survey organizes assessment around functional correctness, process efficiency, cost, execution trajectories, tool use, and non-functional software quality.

  • Evaluating code-generation agents requires assessing problem solving in complex, dynamic software-development scenarios beyond syntax or pass rates.
  • Benchmarks have evolved from independent unit tasks toward real software-development environments requiring interaction, planning, and execution.
  • Method/Class-level Code Generation Benchmarks: Method- and class-level benchmarks test whether models transform natural-language requirements into syntactically and functionally correct code snippets.
  • Method/Class-level Code Generation Benchmarks: HumanEval established objective, automatable code-execution testing by pairing programming problems with unit tests.
  • Agent evaluation covers functional correctness, process efficiency, and non-functional software quality.
  • Functional Correctness Metrics: Pass@k counts a problem as successful when at least one of k independently generated samples passes all unit tests.
  • Evaluation Metrics for Code Generation Agents: Task success rate measures whether an agent solves a problem, while cost and latency capture API usage, token consumption, computational resources, and response time.
  • Evaluation Metrics for Code Generation Agents: Trajectory efficiency measures the number and effectiveness of actions, and tool-usage accuracy measures whether agents select correct tools with valid parameters.

7 DEPLOYED CODE GENERATION AGENT TOOLS

Deployed code generation agent tools span a progression from developer assistance to collaborative codebase work and increasingly autonomous software development. Representative tools differ in their architectures, context handling, and supported engineering workflows.

  • Tool Evolution: Deployed tools are categorized as co-pilots, collaborators, or autonomous teams according to their degree of human involvement and workflow automation.Co-pilots assist developers, collaborators understand entire codebases and interact deeply, while autonomous teams target end-to-end development with humans acting more like clients or managers.
  • Representative Tools: GitHub Copilot combines retrieval-augmented context construction with cloud sandbox execution to connect coding assistance with Issues, pull requests, and code commits.Its architecture integrates programming assistance with broader GitHub workflow automation.
  • Representative Tools: Devin aims to automate planning, execution, debugging, and deployment through terminal, editor, and browser tools, but reported use reveals low success rates, loops, and hallucinations.These problems expose difficulties handling complex edge cases in real-world software engineering projects.
  • Representative Tools: Cursor uses persistent vector indexing and a local-first architecture to support low-latency global context awareness and multi-file refactoring.Its agents directly interact with local file systems and terminals during collaboration with developers.
  • Representative Tools: Tongyi Lingma combines codebase knowledge graphs with Monte Carlo Tree Search for repository navigation and fault localization.The approach supports multi-file dependency analysis and automated program repair.
  • Representative Tools: Claude Code uses up to 200K-token context windows, hybrid reasoning, and extended thinking to plan and execute complex coding, refactoring, and debugging tasks.It is designed as a terminal-native agent oriented toward autonomous development-team workflows.

8 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies limitations spanning agent capabilities, context and collaboration reliability, continual updating, tool use, deployment, security, ownership, and evaluation. It proposes future work addressing these constraints through better understanding, coordination, knowledge maintenance, flexible tooling, safeguards, and comprehensive assessment.

  • Limitations of Agent Core Capabilities: Agents struggle with domain-specific terminology and professional reasoning because structured domain knowledge bases and specialized training remain limited.These gaps can produce biased understanding, logical collapse, or hallucinations beyond the agents’ knowledge boundaries.
  • Limitations of Agent Core Capabilities: Ambiguous human instructions can cause misunderstandings, motivating stronger intent reasoning, context awareness, and interactive clarification mechanisms.Clarification is intended to maintain alignment with human goals.
  • Limitations of Agent Core Capabilities: Large projects expose limitations in long-context modeling, cross-file dependency analysis, and overall software architecture understanding.These limitations constrain effectiveness on projects with massive files and complex dependency relationships.
  • Limitations of Agent Core Capabilities: Agents cannot yet fully use UI sketches, architecture diagrams, and flowcharts for code generation, especially in frontend and interface implementation tasks.Multimodal understanding and generation is identified as an important future research direction.
  • Limitations of Agent Core Capabilities: Context defects such as poisoning, distraction, confusion, and conflict can make generated code inconsistent or incorrect, especially in multi-step tasks.The survey describes context engineering as dynamically delivering appropriate information and tools at the right time and in the right format.
  • Robustness and Updatability Challenges of Agent Systems: In multi-agent systems, small upstream deviations can propagate through downstream decisions and amplify into systemic task failures.This error cascading effect threatens multi-agent system reliability.
  • Robustness and Updatability Challenges of Agent Systems: As agent counts increase, interaction relationships grow exponentially, creating risks of communication bottlenecks, responsibility ambiguity, goal drift, and unstable execution.The survey frames robust management of large agent teams as a complex systems engineering challenge.
  • Robustness and Updatability Challenges of Agent Systems: One-off training leaves agents’ knowledge bases fixed as programming languages and frameworks evolve, while continual learning without catastrophic forgetting remains unresolved.The survey identifies efficient incorporation of new knowledge as an open need.

9 CONCLUSION

The survey frames LLM-based code generation agents as a new software-development paradigm defined by autonomy, broader task coverage, and engineering practicality. It reviews their development, methods, lifecycle applications, tools, challenges, and future research directions.

  • 9 CONCLUSION: The paper distinguishes code generation agents through autonomy, broader task coverage, and engineering practicality.These features separate agents from traditional code generation methods.
  • 9 CONCLUSION: The survey traces the technology’s development and analyzes its core components from a methodological perspective.
  • 9 CONCLUSION: It summarizes representative applications and deployed tools across the software development lifecycle.
  • 9 CONCLUSION: Integration with development environments and ensuring code quality remain central technical challenges for future research.
  • 9 CONCLUSION: Resolving existing challenges could free developers from repetitive coding and support greater focus on creative problem formulation and system-level design.
Loading 2508.00083v2…