Source-linked AI summary
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
Junda He, Christoph Treude, David Lo
TL;DR
Software engineering tasks span multiple domains, creating a need for collaborative systems that extend beyond a single LLM-based agent. This paper systematically reviews LMA applications, evaluates current capabilities and limitations through two case studies, and proposes a research agenda focused on stronger agents and better synergy. The case studies show efficient success on a moderately complex Snake game but substantial difficulty on the more complex Tetris task.
Problem
Single LLM-based agents face limitations on real-world software engineering problems that span multiple domains and require varied expertise.
Method
The paper reviews 71 primary studies, conducts two case studies, and develops a structured research agenda for LMA systems in software engineering.
Results
The Snake case averaged 76 seconds and $0.019 per attempt, while ChatDev required ten attempts for Tetris and still lacked completed-row removal.
Takeaways & Limitations
The findings support LMA systems for moderately complex software engineering tasks while motivating work on individual capabilities and agent synergy.
Takeaways & Limitations
Current benchmarks often evaluate isolated tasks and overlook collaboration, integration, peer review, and collective decision-making in software engineering.
Abstract
from arXiv · showhide
Integrating Large Language Models (LLMs) into autonomous agents marks a significant shift in the research landscape by offering cognitive abilities that are competitive with human planning and reasoning. This paper explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems for addressing complex challenges in software engineering (SE). By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects. In this paper, we conduct a systematic review of recent primary studies to map the current landscape of LMA applications across various stages of the software development lifecycle (SDLC). To illustrate current capabilities and limitations, we perform two case studies to demonstrate the effectiveness of state-of-the-art LMA frameworks. Additionally, we identify critical research gaps and propose a comprehensive research agenda focused on enhancing individual agent capabilities and optimizing agent synergy. Our work outlines a forward-looking vision for developing fully autonomous, scalable, and trustworthy LMA systems, laying the foundation for the evolution of Software Engineering 2.0.
1 INTRODUCTION
LLM-based multi-agent systems address software engineering’s multidisciplinary complexity through specialized collaboration, autonomous task decomposition, and cross-agent validation. This paper reviews 71 primary studies, evaluates current systems through two case studies, and proposes a research agenda for autonomous, scalable, and trustworthy LMA systems.
- LLMs bring planning and reasoning abilities nearing human levels to autonomous agents, motivating their integration into LMA systems.
- LMA systems use specialized agents and collaboration mechanisms such as debate, discussion, and validation to address problems spanning multiple domains.These mechanisms support divergent thinking, factuality, reasoning, and thorough validation.
- Benefits of LMA Systems in Software Engineering: LMA systems can decompose high-level software requirements into subtasks and implementation steps, increasing autonomy while freeing developers for strategic and creative work.
- Benefits of LMA Systems in Software Engineering: Cross-agent examination can detect and correct faults early, helping mitigate unreliable LLM outputs and improve solution robustness.The paper compares this process with code reviews and automated testing frameworks.
- Benefits of LMA Systems in Software Engineering: LMA systems scale to complex software projects by adding specialized agents and reallocating tasks as technologies and project needs evolve.Distributed intelligence and collaborative agent frameworks are presented as ways to manage complexity beyond individual developers or traditional teams.
- The study reviews 71 primary studies, conducts two case studies, and identifies research gaps leading to a structured agenda for advancing LMA systems in software engineering.The agenda targets individual agent capabilities and agent collaboration and synergy.
2 PRELIMINARY
The preliminary section defines autonomous agents, LLM-based agents, and multi-agent systems, then characterizes LMA systems by their agents and orchestration platforms. It organizes orchestration around coordination, communication, planning, and learning, while distinguishing agent roles and configurations.
- 2.1 Autonomous Agent: An autonomous agent independently operates in dynamic environments through autonomy, perception, goal-driven intelligence, social ability, and learning capabilities.
- 2.2 LLM-Based Agent: An LLM-based agent is formally represented as ⟨L,O,M,P,A,R⟩, with the LLM serving as its cognitive core and rethink evaluating outcomes and feedback after actions.The supplied passages explicitly define L as the Large Language Model and R as Rethink.
- 2.3 LLM-Based Multi-Agent Systems: A multi-agent system contains interacting intelligent agents that communicate, coordinate, and share knowledge to solve problems beyond a single agent’s capability.
- 2.3 LLM-Based Multi-Agent Systems: An LMA system comprises an orchestration platform and LLM-based agents, with the platform managing interactions and information flow.The platform facilitates coordination, communication, planning, and learning.
- 2.3 LLM-Based Multi-Agent Systems: Orchestration platforms specify coordination models, communication mechanisms, and planning and learning styles, including centralized, decentralized, hierarchical, cooperative, competitive, CPDE, and DPDE arrangements.
- 2.3 LLM-Based Multi-Agent Systems: Agents may be predefined or dynamically generated, homogeneous or heterogeneous, and represented as graph nodes whose edges encode interactions.
3 LITERATURE REVIEW
The review organizes LMA research across SDLC stages and surveys multi-agent approaches for code generation, quality assurance, and requirements engineering. It uses database keyword searches, screening criteria, and snowballing to identify relevant primary studies.
- The review covers LMA applications in requirements engineering, code generation, quality assurance, and software maintenance.
- The search combines LMA-related keywords with phase-specific software engineering terms in DBLP title searches.
- The study excludes short, duplicate, inaccessible, non-English, non-experimental, and non-software-engineering papers, as well as single-agent LLM studies.
- Backward and forward snowballing continued to a transitive-closure fixed point and added 30 relevant papers.
- Requirements Engineering: Requirements-engineering frameworks use specialized agents to elicit, model, verify, specify, evaluate, and prioritize requirements or user stories.
- Code Generation: Code-generation systems commonly combine role specialization with iterative feedback among orchestrators, programmers, reviewers, testers, and information retrievers.
- Code Generation: Agent Forest instead samples independent candidate outputs and selects the one with the highest cumulative similarity score.
- Quality Assurance: LMA systems support software quality assurance through testing, vulnerability detection, bug detection, and fault localization using specialized agents and cross-validation.
4 CASE STUDY
Two ChatDev case studies examine autonomous generation of Snake and Tetris games through designing, coding, and testing phases. Snake met all prompt requirements after one retry, whereas Tetris required ten attempts and still lacked row removal.
- ChatDev uses specialized CEO, CTO, programmer, reviewer, and tester roles across designing, coding, and testing phases.
- Snake Game: Snake became playable and met all prompt requirements on the second attempt after the first attempt failed.
- Snake Game: 76 seconds and $0.019 per attempt were the average Snake development time and cost.
- Tetris Game: Tetris produced functional gameplay only on the tenth attempt after the first nine attempts failed.
- Tetris Game: The generated Tetris game met most prompt requirements but lacked completed-row removal.
- Tetris Game: 70 seconds and $0.020 per attempt were the average Tetris development time and cost.
- Summary of Findings: The case studies characterize LMA systems as efficient for moderately complex tasks and promising for rapid prototyping, while exposing difficulties with deeper reasoning and abstraction.
5 RESEARCH AGENDA
The research agenda frames LMA development in software engineering as a nascent field with unresolved challenges and envisions two phases for its future development.
- The agenda identifies many critical challenges in the nascent LMA-for-software-engineering field and proposes a two-phase development vision.
5.1 Phase 1: Enhancing Individual Agent Capabilities
Phase 1 focuses on strengthening individual LLM-based agents through specialized SE roles, competency assessment, targeted training, and improved prompting languages. The agenda emphasizes role-specific capability development and prompts that balance expressiveness, robustness, usability, and cross-model adaptability.
- Phase 1: Phase 1 improves individual agents’ adaptability and specialized software-engineering skills.The agenda frames individual-agent capability improvement as the first development phase for LMA systems.
- Refining Role-Playing Capabilities: Current systems simulate generic developers and product managers, but general-purpose LLMs lack nuanced SE expertise for roles such as vulnerability detection and security auditing.The paper identifies deficiencies in accurately detecting and repairing vulnerabilities as evidence for domain-specific specialization.
- Refining Role-Playing Capabilities: Role refinement requires identifying high-value SE roles, mapping their competencies, evaluating performance, and enhancing agents through targeted training.The proposed process combines market analysis, stakeholder engagement, value-addition modeling, competency frameworks, realistic tasks, gap analysis, expert consultation, and iterative refinement.
- Refining Role-Playing Capabilities: LMA roles are specialized collaborative tasks and should not be assumed to match traditional software-engineering roles exactly.The paper distinguishes agent-oriented role design from direct replication of conventional SE positions.
- Advancing Prompts: A specialized prompting language should reduce ambiguity while remaining expressive for complex SE workflows and simple enough for users to adopt.The agenda also highlights the need to adapt prompts across LLM models and versions rather than rely on manual adjustment and experimentation.
- Advancing Prompts: AOP-based prompting is presented as an intermediate step toward an AI-oriented language with grammar tailored to LLMs.The paper does not claim that AOP-based prompting is the final solution.
5.2 Phase Two: Optimizing Agent Synergy
Phase Two addresses agent synergy by optimizing collaboration among agents and between humans and agents. Its agenda covers task allocation, cooperative evaluation, scalable coordination, dynamic adaptation, and secure information sharing for complex software projects.
- Phase Two: Phase Two focuses on leveraging diverse agent strengths through internal collaboration and external human intervention.The phase is organized around optimizing synergy rather than only improving isolated agent capabilities.
- Human-Agent Collaboration: Human-agent task allocation should combine human creativity, critical thinking, ethical judgment, and domain knowledge with agents’ rapid data processing, repetitive-task accuracy, and pattern detection.The paper identifies intuitive interfaces, interaction protocols, and human-to-agent ratio models as supporting mechanisms.
- Evaluating the LMA Systems: Existing benchmarks measure isolated LLM task performance but often omit collaborative activities such as joint requirements gathering, code integration, and peer review.The paper calls for multi-agent benchmarks that simulate realistic cooperative software-engineering scenarios.
- Evaluating the LMA Systems: Proposed benchmarks should test collaborative design, task delegation, conflict negotiation, component integration, peer review, and proactive clarification requests.These tasks evaluate coordination, collective decision-making, conflict handling, quality assurance, and uncertainty management.
- Evaluating the LMA Systems: Collaboration metrics should assess communication efficiency, ambiguity resolution, conflict management, adherence to best practices, and overall project success.The paper argues that evaluation must cover the collaboration process as well as final outputs.
- Scaling and Dynamic Adaptation: Scaling agent populations creates communication bottlenecks, information overload, and memory challenges, requiring efficient protocols and message prioritization.Dynamic adaptation can adjust agent numbers, create specialized or replicated agents, and provide newcomers with project context from shared knowledge sources.
- Security and Data Sharing: Multi-organizational projects must reconcile siloed data, privacy concerns, proprietary restrictions, regulatory requirements, and limited transparency in data sources and processes.Secure private-data sharing is therefore a distinct research question for LMA systems.
6 DISCUSSION
LMA systems differ from MoE by enabling explicit communication, iterative refinement, tool feedback, and human intervention among specialized agents. The review also acknowledges risks in study selection and describes mitigation through broad database searching and snowballing.
- A Comparison with the Mixture of Experts Paradigm: MoE models have high resource consumption because their many experts enlarge parameter counts and make training more resource-intensive and time-consuming.Optimizing the gating mechanism for expert selection adds considerable training overhead.
- A Comparison with the Mixture of Experts Paradigm: Unlike MoE’s internal expert routing, LMA systems let specialized agents communicate, exchange information, and iteratively refine outputs.MoE dynamically activates experts based on input but does not provide interaction or communication between them.
- A Comparison with the Mixture of Experts Paradigm: LMA systems can incorporate feedback from compilers, static analyzers, and testing frameworks while supporting human validation and guidance throughout the process.These mechanisms align LMA workflows with collaborative software engineering practices.
- Threat to Validity: The literature search may have excluded relevant studies despite using DBLP, automated querying, and forward and backward snowballing to broaden coverage.The authors identify inadvertent exclusion as a potential threat to validity and describe multiple mitigation steps.
7 CONCLUSION AND FUTURE WORK
The paper reviews LMA applications across the software development lifecycle, uses two case studies to examine practical capabilities and challenges, and proposes a structured research agenda. Future directions target stronger specialized agents, better agent synergy, scalable orchestration, and protected data exchange.
- CONCLUSION AND FUTURE WORK: The review covers LMA applications across requirements engineering, code generation, software quality assurance, and maintenance.It examines applications across different stages of the software development lifecycle.
- CONCLUSION AND FUTURE WORK: Two case studies illustrate practical uses and challenges of current LMA systems, informing the paper’s proposed research agenda.The agenda is aimed at advancing LMA integration in software engineering.
- CONCLUSION AND FUTURE WORK: Immediate priorities include specialized datasets, pre-training tasks, and prompting strategies that improve agents’ representation of specialized software engineering roles.These efforts target the agents’ cognitive functions and decision-making skills.
- CONCLUSION AND FUTURE WORK: Longer-term priorities include allocating tasks between humans and agents, scaling orchestration for multifaceted projects, and protecting privacy and confidentiality during information exchange.The proposed agenda combines human-agent coordination, scalable methodologies, and data management or access controls.