Source-linked AI summary
AutoAgents: A Framework for Automatic Agent Generation
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, Yemin Shi
TL;DR
Existing multi-agent systems often rely on predefined agents, limiting adaptation to complex tasks. AutoAgents dynamically synthesizes task-specific specialized teams and plans, then coordinates refinement and collaboration; experiments report stronger performance across tasks, although erroneous outcomes can remain because role generation and planning are imperfect.
Problem
Existing multi-agent approaches commonly rely on predefined agents, while complex tasks require intensive knowledge, reasoning, diverse expertise, and long-horizon planning.
Method
AutoAgents dynamically generates specialized agents and execution plans for each task, using staged drafting, coordinated execution, observer feedback, and agent refinement.
Results
AutoAgents outperforms single agents and other groupings across tasks requiring diverse skills, and improves knowledge acquisition by 10% over the Standard method across all reported experiments.
Takeaways & Limitations
The findings support dynamically assigning roles and coordinating specialized agents as a way to address complex tasks requiring diverse skills.
Takeaways & Limitations
AutoAgents may still produce erroneous outcomes because role generation and planning arrangements are not always rational.
Abstract
from arXiv · showhide
Large language models (LLMs) have enabled remarkable advances in automated task-solving with multi-agent systems. However, most existing LLM-based multi-agent approaches rely on predefined agents to handle simple tasks, limiting the adaptability of multi-agent collaboration to different scenarios. Therefore, we introduce AutoAgents, an innovative framework that adaptively generates and coordinates multiple specialized agents to build an AI team according to different tasks. Specifically, AutoAgents couples the relationship between tasks and roles by dynamically generating multiple required agents based on task content and planning solutions for the current task based on the generated expert agents. Multiple specialized agents collaborate with each other to efficiently accomplish tasks. Concurrently, an observer role is incorporated into the framework to reflect on the designated plans and agents' responses and improve upon them. Our experiments on various benchmarks demonstrate that AutoAgents generates more coherent and accurate solutions than the existing multi-agent methods. This underscores the significance of assigning different roles to different tasks and of team cooperation, offering new perspectives for tackling complex tasks. The repository of this project is available at https://github.com/Link-AGI/AutoAgents.
1 Introduction
AutoAgents addresses the limited adaptability of multi-agent systems that rely on predefined roles by dynamically generating specialized agents and coordinating them for different tasks. It combines customized team formation, execution planning, refinement, and collaboration to improve complex task solving.
- LLMs struggle with tasks requiring intensive knowledge, reasoning, diverse domain knowledge, and long-horizon planning.
- Existing multi-agent systems often depend on handcrafted or user-specified agents and human supervision, restricting collaborative applications.
- AutoAgents dynamically generates and coordinates specialized agents to construct task-specific AI teams.
- The framework uses Drafting and Execution stages, with predefined agents synthesizing the team and plan before inter-agent collaboration and feedback produce the outcome.
- AutoAgents evaluates complex tasks quantitatively and through case studies, including knowledge acquisition, reasoning, software development, and collaborative mechanisms.
2 Related Work
Related work develops autonomous and generated-agent systems, but existing approaches do not fully support automatic agent generation for broad collaborative applications. AutoAgents is positioned as a framework for dynamically forming task-oriented agent teams.
- Autonomous-agent systems use LLMs to pursue objectives, while some multi-agent systems assign distinct responsibilities and coordinate agents toward common goals.
- Auto-GPT operates in isolation and does not support multi-agent collaboration.
- Existing approaches are described as not supporting automatic agent generation, limiting the scope of collaborative applications.
- Several agent-generalization methods generate agent profiles, but some remain dependent on restricted predefined agents or vary only profiles.
3 The Framework for Automatic Agent Generation
AutoAgents forms task-specific teams and plans in a Drafting Stage, then executes and refines them through coordination, self-refinement, collaborative refinement, and memory-based knowledge sharing.
- Drafting Stage: The Drafting Stage analyzes the task to synthesize a customized agent team and execution plan, while the Execution Stage refines the plan through collaboration and feedback.
- Drafting Stage: Three predefined roles—Planner, Agent Observer, and Plan Observer—iteratively improve agent selection and plan quality through communication.
- Agent Generation: Each generated agent is specified through a prompt, description, toolset, and suggestions to establish its expert identity and operating constraints.
- Execution Stage: Execution uses self-refinement for specialized individual work and collaborative refinement for knowledge sharing and interdisciplinary tasks.
- Knowledge Sharing: Long-term, short-term, and dynamic memory record action histories and provide specialized information for subsequent actions.
- Execution Stage: The Action Observer allocates tasks, verifies agent outputs, and dynamically adapts the execution plan according to execution status.
4 Experiments
AutoAgents is evaluated on open-ended question answering and Trivia Creative Writing, alongside ablations and a software-development case analysis. Across these settings, results support adaptive agent generation, collaborative execution, self-refinement, observers, and dynamic memory.
- Experimental Setup: The experiments evaluate AutoAgents on Open-ended Question Answering and Trivia Creative Writing, with additional case studies for complex practical scenarios.Open-ended answers are evaluated with FairEval and HumanEval; Trivia Creative Writing measures integration of trivia answers into coherent stories.
- 4.1 Open-ended Question Answer: AutoAgents outperforms individual LLM models on both FairEval-based LLM evaluation and human evaluation for open-ended questions.The framework synthesizes multiple expert models to produce more comprehensive, nuanced answers with more elaborate explanations and justifications.
- 4.2 Trivia Creative Writing: 10% improvement across all Trivia Creative Writing experiments distinguishes AutoAgents from the Standard method without Agent Generation.AutoAgents also surpasses SSP, which uses agent generation through a different approach.
- 4.3 Further Analysis: 3% lower overall performance without observers shows that collaborative discussion affects agent generation and execution-plan allocation.Observer feedback broadens the generated team from programmers alone to include game design, UI design, and testing experts.
- 4.3 Further Analysis: 3% lower performance without self-refinement indicates that individual agents’ iterative improvement contributes to the multi-agent framework.Self-refinement is used for specialized roles and individual tasks within AutoAgents.
- 4.3 Further Analysis: Collaborative refinement and dynamic memory improve performance, with ablations showing declines of 2% and 1%, respectively.Collaborative refinement supports cross-domain knowledge fusion, while dynamic memory incorporates historical action information into prompts.
5 Conclusion
AutoAgents automatically synthesizes collaborative specialized agents and applies task decomposition, delegation, and refinement to complex problem-solving. The paper reports advantages over single agents and other groupings, alongside a software-development case study.
- AutoAgents automatically synthesizes collaborative specialized agents for complex tasks.
- The framework decomposes tasks into drafting and execution phases and delegates subtasks to different agents.
- AutoAgents surpasses single agents and other groupings across tasks requiring diverse skills.
- A software-development case study illustrates the framework’s versatility and potential benefits.
A More Examples
The examples show AutoAgents combining self-refinement, collaborative refinement, dynamic agent generation, and a universal prompt format across complex tasks.
- Self-refinement: Self-refinement has an agent iteratively improve specialized task outputs, while programmer examples move from pseudocode to program files.
- Collaborative refinement: Collaborative refinement combines information from multiple fields to produce more coherent and consistently improved creative writing.
- Dynamic agents: AutoAgents generates agents from three distinct domains for open-ended questions, enabling more elaborate answers than GPT-4 in the illustrated comparison.
- Dynamic agents: The Action Observer coordinates interactions among generative agents and summarizes essential information during task execution.
- Prompt design: AutoAgents uses one universal prompt format across tasks, whereas AgentVerse and SSP implement task-specific enhancements.
B Human Evaluation
The human-evaluation materials ask volunteers to assess response quality using specified criteria and an overall 1-to-10 score.
- Volunteers evaluate responses for helpfulness, relevance, accuracy, and level of detail.
- Overall performance is scored on a 1-to-10 scale, with higher scores indicating better performance.
- Evaluators provide an explanation before assigning the score and are instructed to avoid presentation-order bias.
C Discussion
AutoAgents remains limited by role-generation quality, model dependence, and finite memory, despite its adaptive expert construction. Future work therefore targets stronger team recruitment, more professional agents, broader LLM compatibility, dynamic correction, and improved memory.
- Limitations: AutoAgents may still produce erroneous outcomes because role generation and planning arrangements are not always rational.Collaborative discussions improve these components but do not eliminate the issue.
- Limitations: Role differences mainly derive from prompt and tool-use variations, which do not sufficiently distinguish expert roles.The paper calls for more expert knowledge and professional role agents for professional problems.
- Limitations: AutoAgents relies heavily on GPT-4 and adapts poorly to some earlier LLMs.The authors propose more reasonable prompts to improve compatibility across different LLMs.
- Future Work: Future multi-agent systems should detect and rectify errors quickly while dynamically adjusting plans toward desired outcomes.The initial LLM-generated plan may be insufficient and can lead to erroneous final outputs.
- Future Work: Existing agents’ memory capacity is limited by LLM token counts, leaving efficient memory retrieval and storage unresolved.The paper identifies high-quality memory mechanisms as an open question.
- Future Work: Generated agents’ professional skills are effective but could improve through retraining or an on-demand Agent Bank.The paper leaves more professional agent construction open for exploration.
D Prompts
The prompts specify how AutoAgents creates, inspects, and refines expert roles and execution plans. They emphasize task decomposition, role suitability and cooperation, non-overlapping expertise, explicit tools and constraints, progressive plans, and observer-based review.
- D Prompts: The framework documents prompts for Planner, Plan Observer, Role Observer, Action Observer, and Custom Agent components.These prompts are designed to elicit desired agent behaviors across scenarios and tasks.
- D.1 Planner: The Planner prompt begins by understanding and decomposing the task before selecting suitable existing roles and tools.It also requires role goals, constraints, tool assignments, execution suggestions, and generated prompt templates.
- D.1 Planner: New expert roles must avoid duplicated functions and specify distinct expertise, goals, constraints, tools, suggestions, and prompt templates.The requested role information follows a structured JSON format.
- D.1 Planner: The Planner requires a progressive multi-step execution plan assigning at least one expert role to each step.Each step should describe contributions, collaboration, expected outputs, and inputs for subsequent steps.
- D.1 Planner: The planner prompt requires a final independent language-expert step and a numbered plan identifying expert roles.The prompt also uses task context, existing roles, tools, suggestions, and a specified output format.
- D.2 Agent Observer: The Agent Observer checks whether selected roles satisfy task requirements, cooperate effectively, and retain complete original information.It also reviews created roles for duplication, completeness, meaningful naming, and specific expertise.
- D.2 Agent Observer: The Agent Observer outputs inspection results and clearly reports errors or suggestions, or writes “No Suggestions” when none exist.Its prompt supplies the task, existing roles, selected roles, and created roles as inputs.
- D.3 Plan Observer: The Plan Observer reviews whether execution plans contain connected detailed steps, assigned roles, expected outputs, next-step inputs, and a final language expert.It starts by analyzing the human’s problem before evaluating the plan.