Source-linked AI summary
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, Jürgen Schmidhuber
TL;DR
Existing LLM-based multi-agent systems struggle with coherent and accurate problem solving on complex tasks. MetaGPT addresses this by encoding software-development SOPs, specialized roles, structured communication, and executable feedback into a meta-programming framework, achieving state-of-the-art benchmark performance and 100% task completion in reported software-project evaluations.
Problem
Existing LLM-based autonomous-agent systems oversimplify complex workflows and struggle with effective, coherent, and accurate collaborative problem solving.
Method
MetaGPT encodes human-like software-development SOPs through specialized roles, structured intermediate outputs, message sharing, and executable runtime feedback.
Results
MetaGPT achieves state-of-the-art performance on HumanEval and MBPP, with reported Pass@1 scores of 85.9% and 87.7%, and a 100% task completion rate in software-project evaluations.
Takeaways & Limitations
MetaGPT provides a flexible platform for developing LLM-based multi-agent systems with role definition, message sharing, and workflow-oriented collaboration.
Takeaways & Limitations
The system does not yet fully support scenarios such as UI and frontend development, and remains challenged by diverse and complex real-world requirements.
Abstract
from arXiv · showhide
Remarkable progress has been made on automated problem solving through societies of agents based on large language models (LLMs). Existing LLM-based multi-agent systems can already solve simple dialogue tasks. Solutions to more complex tasks, however, are complicated through logic inconsistencies due to cascading hallucinations caused by naively chaining LLMs. Here we introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together. On collaborative software engineering benchmarks, MetaGPT generates more coherent solutions than previous chat-based multi-agent systems. Our project can be found at https://github.com/geekan/MetaGPT
1 INTRODUCTION
MetaGPT addresses the difficulty of coherent, accurate collaboration among LLM agents by encoding human-like SOPs, specialized roles, and structured intermediate outputs. It reports state-of-the-art code-generation results and broad task completion in software-development evaluations.
- Existing LLM-based autonomous-agent systems oversimplify complex workflows and struggle with effective, coherent, and accurate collaborative problem solving.
- MetaGPT encodes human-like SOPs into a meta-programming framework for LLM-based multi-agent collaboration.
- Structured requirements documents, design artifacts, flowcharts, and interface specifications support more consistent communication and target code generation.
- Specialized agents support automatic requirement analysis, system design, code generation, modification, execution, and runtime debugging.
- 85.9% and 87.7% in Pass@1 are reported on HumanEval and MBPP, respectively, while MetaGPT achieves a 100% task completion rate in software-project evaluations.
- Executable feedback contributes a 5.4% absolute improvement on MBPP, and extensive results report state-of-the-art performance on HumanEval and MBPP.
2 RELATED WORK
Related work develops LLM-based agents through dialogue, cooperation, competition, planning, strategy, and behavior simulation. MetaGPT instead brings human practice into multi-agent frameworks, while Figure 2 illustrates structured communication and executable feedback.
- Recent LLM-based multi-agent frameworks improve problem-solving abilities by integrating discussions among multiple agents.
- Some related systems emphasize cooperation and competition for planning and strategy, while others model LLM-based economies or open-world human behavior.
- MetaGPT aims to introduce human practice into multi-agent frameworks rather than focus on open-world human behavior simulation.
- Figure 2 depicts agents publishing structured messages to a shared pool, subscribing by profile, and iteratively checking generated code with executable feedback.
3 METAGPT: A META-PROGRAMMING FRAMEWORK
MetaGPT organizes LLM agents through specialized roles, sequential software-development SOPs, structured communication, publish-subscribe messaging, and executable feedback. These mechanisms decompose complex work, direct information flow, and iteratively test generated code.
- MetaGPT combines role specialization, workflow management, structured communication, publish-subscribe interfaces, and executable feedback in an LLM-based multi-agent framework.
- Agents in Standard Operating Procedures: Unambiguous roles decompose complex work into smaller tasks handled by agents with diverse skills and specialized outputs.
- Agents in Standard Operating Procedures: MetaGPT defines Product Manager, Architect, Project Manager, Engineer, and QA Engineer roles with role-specific profiles, goals, constraints, contexts, and skills.
- Agents in Standard Operating Procedures: The software-development SOP establishes a sequential workflow in which requirements become a PRD, system design, task distribution, and implementation deliverables.
- Structured Communication Interfaces: Structured communication replaces unconstrained dialogue with role-specific documents and diagrams containing the information needed by downstream agents.
- Publish-Subscribe Mechanism: A shared message pool lets agents publish and retrieve structured messages directly, while subscriptions filter information according to role-specific interests.
- Iterative Programming with Executable Feedback: Because non-executable review may not ensure runtime correctness, executable feedback runs code and tests, uses execution and debugging memory, and permits up to 3 retries.
4 EXPERIMENTS
MetaGPT is evaluated on public code-generation benchmarks and a challenging software-development dataset using functional, practical, cost, and capability comparisons. It outperforms prior approaches on the reported benchmarks and shows advantages in software-development quality and efficiency, while ablations examine roles, SOP-like designs, and executable feedback.
- Experimental Setting: HumanEval contains 164 programming tasks, MBPP contains 427 Python tasks, and SoftwareDev contains 70 representative software-development tasks.SoftwareDev spans mini-games, image-processing algorithms, and data visualization.
- Experimental Setting: The evaluation uses unbiased Pass@k for HumanEval and MBPP, while SoftwareDev measures executability, cost, code statistics, productivity, and human revision cost.Executability is rated from 1 for non-functional code to 4 for flawless code.
- Main Result: On SoftwareDev, MetaGPT outperforms ChatDev in nearly all reported metrics, including executability, time, code statistics, and human revision cost.MetaGPT achieves executability 3.75 and takes 503 seconds; it uses 126.5/124.3 tokens per code line versus ChatDev’s 248.9, while requiring more total tokens.
- Ablation Study: Adding roles beyond Engineer improves revisions and executability, although more roles slightly increase expenses.The ablation reports that excluding certain roles produces unworkable code.
5 CONCLUSION
MetaGPT combines SOPs, specialized roles, workflow management, message sharing, and executable feedback to support LLM-based multi-agent problem solving. Extensive experiments report state-of-the-art performance on multiple benchmarks and motivate human-inspired techniques for future multi-agent systems.
- MetaGPT models agents as a simulated software company using SOPs, role specialization, workflow management, and message sharing.The framework also includes an executable feedback mechanism intended to enhance code generation quality during runtime.
- MetaGPT achieves state-of-the-art performance on multiple benchmarks.
- The integration of human-like SOPs motivates future research on human-inspired techniques for artificial multi-agent systems.
A.1 SELF-IMPROVEMENT MECHANISMS
The paper explores self-improvement by having agents revise role constraint prompts from prior software-development feedback. This mechanism is limited because it does not yet update structured communication interfaces, while broader teamwork and agent-service platforms provide related context.
- Limitations: MetaGPT executes each software project independently, so the main-text version does not learn from experience across projects.The stated limitation is that project teams do not become more compatible and successful over time through accumulated development experience.
- Self-improvement mechanism: Before each project, agents review previous feedback and adjust their constraint prompts to improve the multi-agent system.Each agent evaluates received feedback and summarizes how its constraint prompt can improve.
- Limitations: Summary-based optimization modifies role-specialization constraints rather than structured communication interfaces.The paper identifies extending optimization to communication protocols as future work.
- Agent platforms: AgentStore lets users subscribe to agents, pay for usage, purchase capabilities, and combine agents that follow MetaGPT development and communication protocols.Within MetaGPT, the platform supports collaboration among multiple agents on more complex tasks or projects.
B.2 METAGPT DEVELOPMENT PROCESS
MetaGPT’s software-development process converts a user request into standardized artifacts and role-specific implementation work. Product management, architecture, task allocation, engineering, and quality assurance form a sequential workflow that produces a functional application.
- Product Manager: The Product Manager creates a PRD containing goals, user stories, competitive analysis, requirements, and a requirement pool before handing it to the Architect.A competitive quadrant chart is also produced.
- Architect: The Architect translates requirements into technical specifications, architecture diagrams, interface definitions, file and class structures, and sequence-flow documentation.The Architect’s documentation is passed to the Project Manager for task allocation and execution.
- Engineer: Engineers implement the assigned file structure using the documented requirements, implementation approach, libraries, and shared project knowledge.The example uses Tkinter for the GUI and Pillow for image handling, with specified files and interfaces.
- Project Manager: The Project Manager decomposes the project into a task list and assigns separate code-file tasks to Engineers.
- QA Engineer: The QA Engineer generates unit tests and reviews the Engineer’s code to identify and fix bugs.The workflow includes iterative testing, debugging, and up to 3 retries before resuming or completing programming.
- Output: MetaGPT ultimately generates a functional application named “Drawing App.”
C.1 DETAILS OF THE SOFTWAREDEV DATASET
The SoftwareDev dataset contains 70 diverse software-development tasks, with 11 task descriptions displayed in Table 8 and seven used in the paper’s main experiments.
- The SoftwareDev dataset includes 70 diverse software-development tasks.The tasks cover varied scopes, including mini-games, image-processing algorithms, and data visualization.
- Table 8 presents the names and detailed prompts of 11 tasks, of which the first seven are used in the main experiments.
C.2 ADDITIONAL RESULTS
Additional experiments evaluate MetaGPT’s executability, backend dependence, instruction sensitivity, GPT-model performance, and qualitative software-design outputs. Across these settings, structured workflows and feedback support stronger execution and more comprehensive generated systems.
- Quantitative results: 3.9 average executability surpasses ChatDev’s 2.1, while AutoGPT scores 1.0 and fails to generate executable code.Executability is graded from 1, complete failure, to 4, perfect match with expectations.
- Qualitative results: Compared with chat-based collaboration, MetaGPT’s structured messaging and feedback mechanisms reduce communication-information loss and improve code execution.The framework transforms abstract requirements into detailed class and function designs through specialized roles and SOPs.
- Quantitative results: MetaGPT’s performance on SoftwareDev is superior with GPT-4 than with GPT-3.5 or Deepseek Coder 33B as agent backends.The comparison uses five randomly selected SoftwareDev tasks.
- Instruction levels: Detailed prompts produce better software projects with lower productivity ratios, while simple inputs still achieve an executability rating of 3.8.Productivity is defined as token usage divided by total code lines; lower is better.
- GPT variants: GPT-4 achieves 67% HumanEval as the stated baseline, while GPT-4 and GPT-3.5-Turbo show sensitivity to prompting, parsing, and post-processing settings.The experiments use gpt-4-0613 and gpt-3.5-turbo-0613 across five trials with different settings.
- Qualitative results: The Architect agent produces system interface designs and program call flows that support division of labor for complex recommender-system development.The outputs contain system-module designs and interaction sequences for downstream Engineers.
D.1 LIMITATION
The paper identifies limitations in MetaGPT’s scenario coverage, user control, and ability to meet diverse real-world requirements. It also describes design goals addressing context overload and hallucinated software generation.
- System limitations: MetaGPT does not fully support UI and frontend scenarios because it lacks corresponding agents and multimodal tools.This is presented as a current system-side limitation.
- System limitations: Despite generating more code than comparable frameworks, MetaGPT remains challenged by the diverse and complex requirements of real-world applications.The paper states this as a boundary on practical applicability.
- User limitations: Users face difficulty interrupting each agent’s running process or setting an agent’s starting checkpoint.The limitation concerns human control over execution.
- Design responses: MetaGPT addresses code hallucinations by focusing LLMs on granular tasks such as requirement analysis and package selection.The paper identifies incomplete functions, missing dependencies, and undiscovered bugs as software-generation problems.
- Design responses: MetaGPT uses a global message pool and subscription mechanism to filter irrelevant context and streamline communication.The design targets information overload in software-design scenarios and SOP-based communication.