Source-linked AI summary
Large Language Model based Multi-Agents: A Survey of Progress and Challenges
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang
TL;DR
LLM-based multi-agent research lacks a consolidated account of how agents, environments, communication, capabilities, applications, and evaluation resources fit together. This survey systematically organizes those dimensions, reviews problem-solving and world-simulation applications, and discusses datasets, benchmarks, challenges, and future opportunities. It presents the field as rapidly developing while identifying substantial gaps in multimodal settings and comprehensive evaluation benchmarks.
Problem
The rapidly evolving LLM-MA field needs an overview of how agents interact with environments, communicate, acquire capabilities, and operate across applications and evaluation settings.
Method
The survey positions, differentiates, and connects LLM-MA systems across interfaces, agent characterization, communication, capability acquisition, applications, frameworks, datasets, and benchmarks.
Results
The survey organizes LLM-MA research into problem-solving and world-simulation applications and synthesizes commonly used resources, challenges, and future opportunities.
Takeaways & Limitations
The survey provides a resource for researchers studying LLM-MA systems across research fields, including their applications, datasets, benchmarks, and open challenges.
Takeaways & Limitations
Existing evaluation focuses heavily on individual-agent understanding in narrow scenarios, while comprehensive benchmarks remain lacking for several domains.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have achieved remarkable success across a wide array of tasks. Due to the impressive planning and reasoning abilities of LLMs, they have been used as autonomous agents to do many tasks automatically. Recently, based on the development of using one LLM as a single planning or decision-making agent, LLM-based multi-agent systems have achieved considerable progress in complex problem-solving and world simulation. To provide the community with an overview of this dynamic field, we present this survey to offer an in-depth discussion on the essential aspects of multi-agent systems based on LLMs, as well as the challenges. Our goal is for readers to gain substantial insights on the following questions: What domains and environments do LLM-based multi-agents simulate? How are these agents profiled and how do they communicate? What mechanisms contribute to the growth of agents' capacities? For those interested in delving into this field of study, we also summarize the commonly used datasets or benchmarks for them to have convenient access. To keep researchers updated on the latest studies, we maintain an open-source GitHub repository, dedicated to outlining the research on LLM-based multi-agent systems.
1 Introduction
LLM-based multi-agent research builds on single-agent reasoning by combining specialized agents and their interactions for complex problem solving and world simulation. This survey organizes the field’s interfaces, agent profiles, communication, capability acquisition, applications, resources, and future challenges.
- Motivation: LLM-based multi-agent systems specialize LLMs into distinct agents and enable their interactions to address complex problem-solving and world-simulation tasks.The approach leverages collective intelligence, specialized profiles, planning, discussion, and decision-making among autonomous agents.
- Survey goals: The survey aims to give readers a comprehensive overview of LLM-MA systems and the field’s latest research trends and applications.It also maintains an open-source GitHub repository as a continuing resource for the rapidly evolving field.
- Applications and resources: Applications are organized into two primary streams: multi-agents for problem-solving and multi-agents for world simulation.The survey also presents implementation frameworks, usable datasets and benchmarks, and future research challenges.
- Survey framework: The survey analyzes how LLM-MA systems align agents with collaborative task-solving environments through a schema covering interfaces, profiling, communication, and capability acquisition.These perspectives address how agents interact with environments, behave, exchange messages, collaborate, and develop problem-solving abilities.
2 Background
The background contrasts LLM-based single-agent systems with multi-agent systems and introduces the capabilities that support autonomous decision-making. Single agents use thought, tools, and memory, whereas multi-agent systems emphasize diverse profiles, interactions, and collective decisions.
- Single-Agent Systems: Decision-making Thought enables an LLM-based agent to decompose complex tasks, reason through subgoals, explore paths, and learn from past experiences.These capabilities support more autonomous and effective problem solving.
- Single-Agent Systems: Tool-use allows LLM-based agents to leverage external tools and resources, improving their functional operation across diverse and dynamic environments.
- Single-Agent Systems: Memory preserves and retrieves information through short in-context learning or long-term external vector databases.This supports contextual coherence and learning from interactions over prolonged periods.
- Multi-Agent Systems: LLM-MA systems emphasize diverse agent profiles, inter-agent interactions, and collective decision-making rather than only internal mechanisms and environment interactions.Collaboration among multiple autonomous agents supports tackling more dynamic and complex tasks.
3 Dissecting LLM-MA Systems: Interface, Profiling, Communication, and Capabilities
The survey analyzes LLM-MA systems through their environment interfaces, agent profiles, communication mechanisms, and capability acquisition. It also identifies orchestration as a critical challenge as systems scale to larger numbers of agents.
- System Architecture: LLM-MA architectures are analyzed through four aspects: the agents-environment interface, agent profiling, agent communication, and agent capability acquisition.These aspects describe how systems interact with contexts, define agents, coordinate communication, and develop capabilities.
- Agents-Environment Interface: Agents interact with sandbox, physical, or no environment interfaces, receiving environmental feedback that guides strategy adjustment.Sandbox environments include code interpreters and game rules, while physical environments impose real-world entities, physics, and constraints.
- Agent Profiling: Agent profiles specify traits, actions, skills, characteristics, capabilities, behaviors, and constraints tailored to particular goals and roles.Examples include players in games and product managers or engineers in software development.
- Agent Communication: Agent communication is organized by cooperative, debate, or competitive paradigms and by structures such as layered, decentralized, centralized, and shared-message-pool networks.Agents exchange application-specific text, including code segments, analyses, suspicions, and strategies.
- Capability Acquisition: Capability acquisition combines feedback about action outcomes with memory, self-adjustment, and other strategies for adapting to complex problems.Agents may store prior interactions, retrieve useful memories, modify goals or planning strategies, and learn from feedback or communication logs.
- Challenges: Scaling LLM-MA systems to larger agent populations makes managing diverse agents an increasingly critical orchestration problem.The survey identifies agent orchestration as a challenge requiring further discussion.
4 Applications
LLM-MA applications are summarized in two broad categories: problem solving and world simulation. The field is rapidly growing, with new applications appearing almost daily and an open-source repository tracking recent work.
- Application Categories: The survey categorizes LLM-MA applications into problem solving and world simulation.These two streams organize the applications summarized in Table 1.
4.1 LLM-MA for Problem Solving
For problem solving, LLM-MA systems coordinate agents with specialized expertise across software development, embodied tasks, science experiments, and science debate. Their collaboration supports role-based workflows, physical task planning, human-supervised experimentation, and consensus-oriented reasoning.
- Overview: LLM-MA problem-solving systems harness agents with specialized expertise to address complex tasks collaboratively.The survey discusses software development, embodied agents, science experiments, and science debate.
- Software Development: Software-development systems emulate roles such as product managers, programmers, testers, and engineers within collaborative workflows.Agents may interact with code interpreters, other agents, or humans to iteratively refine generated code.
- Embodied Agents: Embodied-agent applications model heterogeneous robots that cooperate on complex planning and manipulation tasks, including warehouse management.LLMs can serve as action planners while agents with different capabilities coordinate physical tasks.
- Science Experiments: Science-experiment systems place human experts at the center because experiments are costly and LLM agents can hallucinate.Human oversight is used to process agent information and provide guidance.
- Science Debate: Science-debate systems have agents independently analyze problems, debate across multiple rounds, and converge on a consensus answer.Reported applications include MMLU, mathematics, StrategyQA, factual-accuracy tasks, commonsense reasoning, and medical diagnosis.
4.2 LLM-MA for World Simulation
LLM-MA world simulations span social, gaming, psychological, economic, recommendation, policy, and disease-propagation settings. These systems use role-playing, profiles, communication, and interaction to model behaviors, decisions, and emergent phenomena.
- Overview: World simulation is a major LLM-MA application spanning social sciences, gaming, psychology, economics, policy-making, and related fields.Role-playing abilities support the depiction of diverse roles and viewpoints in simulated worlds.
- Societal Simulation: Societal simulations model social behavior and communication, ranging from a 25-agent interactive sandbox to a simulated community of 1,000 personas.These systems generate behavioral data for social-science analysis and populate virtual communities with social phenomena.
- Gaming: Gaming environments use natural-language communication to create controlled, scalable settings for studying reasoning, cooperation, persuasion, deception, and leadership.Applications include the Prisoner’s Dilemma, Battle of the Sexes, Werewolf, and Resistance Avalon.
- Psychology: Psychological simulations either apply experiments to independent agents or study emergent behavioral patterns among interacting agents using psychological theories and statistical analysis.The two approaches differ in whether agents communicate with one another.
- Economy: Economic and financial simulations assign agents endowments, information, and preferences to explore decisions in markets and other economic contexts.Applications include macroeconomic simulation, information marketplaces, financial trading, and virtual towns with restaurant and customer agents.
- Other Applications: Recommendation, policy, and disease simulations model user-item interactions, policy effects on communities, and human responses to disease outbreaks.Examples include filter-bubble analysis, virtual governments, and behaviors such as self-quarantine and isolation.
5 Implementation Tools and Resources
The survey reviews open-source frameworks, datasets, and benchmarks supporting LLM-based multi-agent research. It highlights distinct framework designs and identifies domain-specific gaps in current evaluation resources.
- Multi-Agents Framework: The survey introduces MetaGPT, CAMEL, and AutoGen as open-source frameworks for complex task-solving through multi-agent collaboration.These frameworks differ in their approaches and applications.
- Multi-Agents Framework: MetaGPT embeds standard operating procedures and role-specific assembly-line workflows to structure agent operations and reduce hallucination in complex tasks.
- Multi-Agents Framework: CAMEL uses inception prompting to guide autonomous agent cooperation toward tasks consistent with human objectives and to generate conversational data.
- Multi-Agents Framework: AutoGen supports highly customizable agent interactions defined through natural language and code across technical and consumer-focused domains.
- Datasets and Benchmarks: The survey summarizes commonly used datasets and benchmarks, which differ across research applications.
- Datasets and Benchmarks: Problem-solving benchmarks mainly evaluate planning and reasoning through multi-agent cooperation or debate, whereas world-simulation benchmarks assess realism or agent behavior.
- Datasets and Benchmarks: Comprehensive benchmarks remain needed for science-team experiments and economic modeling to assess LLM-based multi-agent systems in these complex fields.
6 Challenges and Opportunities
The survey identifies challenges in extending LLM-based multi-agent systems beyond text, coordinating collective learning, scaling agent populations, and evaluating emergent capabilities. It also points to broad future applications across complex real-world domains.
- Advancing into Multi-Modal Environment: Most LLM-based multi-agent research remains focused on text-based environments, leaving multimodal interaction with images, audio, video, and physical actions underdeveloped.Multimodal systems must process diverse data types and support non-textual interaction among agents.
- Addressing Hallucination: Hallucinations can propagate through interconnected agents when misinformation from one agent is accepted and relayed by others.Mitigation must address both individual inaccuracies and information flow across the network.
- Learning and Capability Acquisition: LLM-based multi-agent learning depends mainly on immediate environmental or human feedback, making reliable interactive environments difficult to design and limiting scalability.
- Acquiring Collective Intelligence: Current memory and self-evolution methods adjust agents individually, overlooking synergistic effects and leaving coordinated optimization of collective intelligence unresolved.
- Scalability: Increasing the number of LLM-based agents substantially raises computational and memory requirements while complicating coordination, communication, and orchestration.
- Evaluation and Benchmarks: Existing evaluation often measures individual-agent reasoning in narrow scenarios, overlooking emergent multi-agent behaviors and lacking comprehensive domain benchmarks.Gaps are identified for science-team experiment operations, economic analysis, and disease-propagation simulation.
- Future Opportunities: LLM-based multi-agent systems may support future problem-solving in finance, education, healthcare, environmental science, and urban planning, although current role-playing capabilities have limitations.
7 Conclusion
The survey organizes LLM-based multi-agent systems by their environments, agent profiles, communication strategies, and capability acquisition, while reviewing applications, resources, challenges, and opportunities. It aims to serve as a research resource for exploring their potential.
- Conclusion: The survey systematically reviews LLM-based multi-agent systems across agent-environment interfaces, agent characterization, communication, and capability acquisition.
- Conclusion: It summarizes applications in problem-solving and world simulation and highlights commonly used datasets, benchmarks, challenges, and future opportunities.
- Conclusion: The survey is intended as a resource for researchers across fields and to inspire further exploration of LLM-based multi-agent systems.