Source-linked AI summary
ChatDev: Communicative Agents for Software Development
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, Maosong Sun
TL;DR
Existing software-development methods improve isolated waterfall phases with technically distinct designs, leaving a fragmented process. ChatDev unifies design, coding, and testing through specialized LLM agents directed by a chat chain and communicative dehallucination. The framework improves software completeness, executability, and consistency, while natural-language communication aids design and programming-language communication aids optimization.
Problem
Deep-learning methods for separate software-development phases require unique designs, creating technical inconsistencies and a fragmented, less efficient process.
Method
ChatDev uses specialized LLM-powered agents, a chat chain for subtask sequencing and communication targets, and communicative dehallucination for more precise exchanges across design, coding, and testing.
Results
ChatDev improves software completeness, executability, and consistency with requirements; natural-language communication supports system design, while programming-language communication helps software optimization.
Takeaways & Limitations
The study presents linguistic communication as a unifying bridge for multi-agent collaboration and autonomous task-solving among LLM agents.
Takeaways & Limitations
The approach is more suitable for prototype systems than complex real-world applications, especially when software requirements are unclear or insufficiently detailed.
Abstract
from arXiv · showhide
Software development is a complex task that necessitates cooperation among multiple members with diverse skills. Numerous studies used deep learning to improve specific phases in a waterfall model, such as design, coding, and testing. However, the deep learning model in each phase requires unique designs, leading to technical inconsistencies across various phases, which results in a fragmented and ineffective development process. In this paper, we introduce ChatDev, a chat-powered software development framework in which specialized agents driven by large language models (LLMs) are guided in what to communicate (via chat chain) and how to communicate (via communicative dehallucination). These agents actively contribute to the design, coding, and testing phases through unified language-based communication, with solutions derived from their multi-turn dialogues. We found their utilization of natural language is advantageous for system design, and communicating in programming language proves helpful in debugging. This paradigm demonstrates how linguistic communication facilitates multi-agent collaboration, establishing language as a unifying bridge for autonomous task-solving among LLM agents. The code and data are available at https://github.com/OpenBMB/ChatDev.
1 Introduction
ChatDev addresses fragmented software development by unifying specialized LLM agents through language-based, multi-turn collaboration across design, coding, and testing. Its chat chain structures communication, while communicative dehallucination supports more precise information exchange and reduces coding hallucinations.
- Software development requires diverse roles to communicate across requirements analysis, development, and debugging, while prior deep-learning methods target isolated waterfall phases.These phases include design, coding, and testing.
- Technical inconsistencies between phase-specific methods leave software development fragmented and less efficient.Different phases require unique designs for data, training, and inference.
- ChatDev integrates multiple LLM-powered software agents with specialized roles across design, coding, and testing.The framework uses language-based communication to connect these lifecycle phases.
- A chat chain divides each phase into smaller subtasks and guides agents’ multi-turn communication about creative ideas or source code.This chain links natural-language and programming-language subtasks in a sequential workflow.
- Communicative dehallucination lets agents request specific details before responding directly, improving information exchange and reducing coding hallucinations.The mechanism addresses hallucination risks in software generation through a defined communication pattern.
- ChatDev improves software completeness, executability, and consistency with requirements; natural language supports system design, while programming-language communication helps software optimization.The paper presents linguistic communication as a unifying bridge for multi-agent autonomous task-solving.
2 Related Work
Related work motivates ChatDev through LLMs’ language and role-playing capabilities and autonomous agents’ memory, planning, and tool-use abilities. Prior software-development research applies deep learning to separate waterfall phases and to natural- and programming-language tasks.
- LLMs integrate extensive knowledge expressed in language and demonstrate strong role-playing abilities.These capabilities underpin recent progress in autonomous-agent research.
- Autonomous agents extend LLM capabilities with memory, multistep planning, and strategic tool use.These features are discussed as central abilities of current autonomous-agent systems.
- Software-development studies apply deep learning to natural-language requirements and to system development and debugging in programming languages.The cited work spans requirements-related language tasks and programming-language development activities.
- Waterfall-model research segments software development into discrete phases such as design, coding, and testing, improving selected phases with deep learning.This phase-based framing provides the related-work context for ChatDev’s integrated framework.
3 ChatDev
ChatDev is a chat-powered framework that coordinates specialized agents across design, coding, and testing through structured multi-turn communication. It combines sequential task decomposition, role-based prompting, phase-aware memory, and communicative dehallucination to develop and refine software.
- ChatDev: The framework addresses the difficulty of converting textual requirements into functional software in one step by organizing development as sequential, language-mediated collaboration.The design responds to the need for cooperation among diverse software-development roles and to hallucination risks in programming.
- ChatDev: ChatDev integrates specialized software agents into sequential design, coding, and testing phases, with subtasks linked through a chat chain.The coding phase includes code writing and completion, while testing is further segmented into subtasks.
- ChatDev: Agents communicate through multi-turn dialogues until reaching consensus, producing solutions ranging from textual specifications to source code.The chain guides communication and transfers completed solutions between subtasks and phases.
- ChatDev: ChatDev uses inception prompting to initialize, sustain, and conclude instructor–assistant exchanges while specifying roles, objectives, tools, protocols, and termination conditions.Role customization assigns system messages to the instructor and assistant, and prompting occurs at the start of each subtask round.
- ChatDev: Short-term memory maintains dialogue continuity within a phase, whereas long-term memory carries previous phase solutions forward without transmitting complete communication histories.Sharing only subtask solutions reduces information overload while preserving cross-phase context continuity.
- ChatDev: Communicative dehallucination reverses roles when needed so the assistant requests precise information before applying targeted modifications to problematic code.Multiple communication rounds address individual issues and practically aid in reducing incomplete, unexecutable, or requirement-inconsistent code.
4 Evaluation
ChatDev is evaluated against single- and multi-agent baselines using software-quality metrics, pairwise preferences, software statistics, ablations, and communication analyses. It outperforms the baselines, while multi-agent collaboration improves software outcomes at the cost of additional time and tokens.
- Overall Performance: ChatDev outperforms all baseline methods across the reported software-quality metrics.The evaluation compares LLM-based development methods using measures including completeness, executability, consistency, and quality.
- Overall Performance: 0.3953 is ChatDev’s Quality score versus 0.1523 for MetaGPT.The authors attribute this improvement to autonomous cooperative communication and iterative source-code refinement across natural and programming languages.
- Overall Performance: ChatDev achieves higher average win rates than other baselines in both GPT-4 and human pairwise evaluations.Agent-generated solutions were compared in pairs by human participants and GPT-4.
- Overall Performance: Multi-agent development is slower and consumes more tokens than the single-agent method, while producing more code files and a larger codebase.These software statistics may enhance functionality and integrity, and agent dialogues can yield functional enhancements not explicitly specified in requirements.
- Ablation Study: Removing communicative dehallucination decreases all metrics, while code completion improves Completeness and testing is critical for Executability.The largest performance impact occurs when agent roles are removed from system prompts.
- Communication Analysis: 57.20% of communications use natural language, with design relying on natural-language discussion and code reviews receiving substantial post-design communication.Reviewers most commonly identify “Method Not Implemented” issues, accounting for 34.85% of code-review discussions.
5 Conclusion
ChatDev integrates fragmented software-development phases into a cohesive multi-agent communication system and uses language-based collaboration to support software optimization.
- ChatDev integrates multiple LLM-powered agents across fragmented waterfall phases through a cohesive communication system.Its chat chain organizes communication targets, while dehallucination addresses coding hallucinations.
- The framework highlights multi-turn communication as beneficial for software optimization.
6 Limitations
ChatDev’s autonomous agents remain constrained by vague requirements, limited evaluation coverage, and the computational costs of multi-agent interaction.
- Without clear, detailed requirements, agents may produce simple, low-information software implementations.The paper gives vague Snake-game guidelines as an example of this limitation.
- The framework is currently more suitable for prototype systems than complex real-world applications.
- General-purpose software evaluation remains highly complex, and the study’s metrics omit functionality, robustness, safety, and user-friendliness.The paper emphasizes completeness, executability, consistency, and overall quality, while noting that manual verification at scale is impractical.
- Compared with single-agent approaches, multiple agents require more tokens and time, increasing computational demands and environmental impact.The authors suggest improving agent capabilities with fewer interactions.