Source-linked AI summary
A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application
Shuaihang Chen, Yuanxing Liu, Wei Han, Weinan Zhang, Ting Liu
TL;DR
As LLM-MAS research grows, existing reviews struggle to cover emerging work comprehensively. This survey synthesizes the field through a definition, an application taxonomy, resource analysis, and discussion of challenges and future directions.
Problem
As related LLM-MAS studies continue to grow, existing reviews struggle to capture them comprehensively.
Source: §1 · p. 1Method
The survey reviews 125 papers, defines LLM-MAS, organizes applications into three categories, and analyzes resources, challenges, and future directions.
Source: §1 · pp. 1–2, §7 · p. 8Results
Takeaways & Limitations
The survey presents its taxonomy and challenge discussion as a basis for understanding existing research and considering potential future directions.
Source: §7 · p. 8The review provides brief method summaries rather than exhaustive technical details and may miss important work from venues outside its primary collection.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
LLM-based Multi-Agent Systems ( LLM-MAS ) have become a research hotspot since the rise of large language models (LLMs). However, with the continuous influx of new related works, the existing reviews struggle to capture them comprehensively. This paper presents a comprehensive survey of these studies. We first discuss the definition of LLM-MAS, a framework encompassing much of previous work. We provide an overview of the various applications of LLM-MAS in (i) solving complex tasks, (ii) simulating specific scenarios, and (iii) evaluating generative agents. Building on previous studies, we also highlight several challenges and propose future directions for research in this field.
1 Introduction
LLM-based multi-agent systems build on generative agents’ reasoning and natural-language interaction to address complex decision-making. This survey reviews 125 papers, organizes applications by purpose, and identifies resources, challenges, and future directions.
1 Introduction
- LLMs enable generative agents to reason and make long-trajectory decisions without training, while offering flexible, explainable natural-language interaction with humans.The introduction contrasts these capabilities with traditional reinforcement-learning agents’ constrained parameters and lack of general knowledge.Source: §1 · p. 1
- The survey reviews 125 papers and groups LLM-MAS applications into solving complex tasks, simulating specific scenarios, and evaluating generative agents.It describes task decomposition as a way multi-agents can improve performance, simulation as a domain sandbox, and evaluation as dynamic assessment.Source: §1 · p. 1
- Its stated contributions are a purpose-based application taxonomy, analysis of open-source frameworks and benchmarks or datasets, and discussion of challenges and future research.Figure 1 distinguishes material aligned with previous surveys from contributions introduced in this study.Source: §1 · p. 2, Figure 1 · p. 2
2 Core Components of LLM-MAS
LLM-MAS comprise generative agents interacting and collaborating in a shared environment. The survey describes agents through their capabilities and communication, and environments through tools, rules, and intervention interfaces.
2 Core Components of LLM-MAS
- LLM-MAS are collections of generative agents that interact and collaborate within a shared environmental setting.The survey treats generative agents and the environment as the system’s core components.Source: §2 · p. 2
2.1 Generative Agents
- Generative agents use role profiles, memory, planning, and actions to perceive and influence their environment, and can communicate to collaborate or reach consensus.Memory retrieves relevant histories for later actions and supports long-term behavior despite limited LLM context windows; communication may use natural language or custom content, trading interpretability and flexibility against ease of optimization.Source: §2.1 · pp. 2–3
2.2 Environment
- The environment provides tools that translate action instructions into outcomes, rules that structure interactions, and interfaces for external intervention.Intervention may let a human or model read system information or interrupt the system to prevent uncontrolled behavior.Source: §2.2 · p. 3
3 LLM-MAS for Solving Complex Tasks
The survey frames LLM-MAS as suited to complex, multi-role tasks and reviews reasoning frameworks, communication optimization, resources, and evaluation approaches.
3 LLM-MAS for Solving Complex Tasks
- Multiple agents can divide complex, multi-step tasks and be trained independently; the survey reports that collaboration can improve overall performance over a single agent.The stated motivation is that such tasks are difficult for a single agent.Source: §3 · p. 3
3.1 Representative LLM-MAS for Solving Complex Tasks
- Reasoning frameworks are grouped into multi-stage problem solving, collective decision-making through voting or debate, and self-refinement through reflection.The survey also notes frameworks for natural-science applications and self-adaptation, while reported exploration of cooperation scaling found no significant pattern.Source: §3.1 · p. 3
- Communication optimization addresses fully connected systems’ risks of combinatorial explosion and privacy disclosure through faster exchange and distributed discussion.Examples of speed optimization include non-verbal communication or shorter generation; distributed discussion tackles tasks when agents lack complete information.Source: §3.1 · pp. 3–4
3.2 Resources of LLM-MAS for Solving Complex Tasks
- The resource discussion covers datasets and open-source systems, including SRDD for evaluating generated software and frameworks such as MetaGPT, AgentScope, and Swarm.AgentScope centers on message exchange and supports local or distributed deployment; Swarm is described as an experimental, lightweight orchestration framework.Source: §3.2 · p. 4
3.3 Evaluation of LLM-MAS for solving complex task
- Task-specific evaluation uses measures such as task steps, QA accuracy, game win rates, recruitment success, and navigation success, while system runtime can measure operational cost.The survey lists additional measures including Spearman correlation, TrueSkill, and manual rankings of generated behavior.Source: §3.3 · p. 4
4 LLM-MAS for Simulating Specific Scenarios
The survey presents LLM-MAS as a way to simulate specific scenarios, with environment design shaping simulation properties. It reviews representative domains, resources, and measures of simulation fidelity.
4 LLM-MAS for Simulating Specific Scenarios
- LLM-MAS simulation applies agents to specific scenarios; natural-language communication can be more intuitive for people than rule-based methods, and the environment shapes simulation properties.The survey characterizes the environment as central to the simulation.Source: §4 · p. 4
4.1 Representative LLM-MAS for Simulating Specific Scenarios
- Social simulations cover town life, emotional propagation, recommendation-driven information cocoons, social movements, urban systems, hospitals, and games.Stanford Town simulates one day in the lives of 25 agents; hospital simulation is described as scaling doctor-agent training to tens of thousands of cases in days.Source: §4.1 · p. 5
- Physical-domain examples include mobility, transportation, and wireless networks, though the survey says multi-generative-agent research in this area is limited.One wireless framework lets on-device agents interact with the environment and exchange knowledge to solve a complex task together.Source: §4.1 · p. 5
4.2 Resources for LLM-MAS simulation
- Simulation resources include open-source systems, code, and benchmarks; simulation evaluation requires realistic data, with dense datasets preferred for assessing capability.The survey describes an ideal dense dataset as one with fewer users for the same scale of data.Source: §4.2 · p. 6
4.3 Evaluation of LLM-MAS simulation
- System-level simulation evaluation includes consistency with reality and similarity in information-dissemination trends over time.The survey relates consistency to meaningful simulation outcomes, agent training and evaluation, and data validity; dissemination comparisons can track events, awareness, emotional density, or support rates.Source: §4.3 · p. 6
5 LLM-MAS for Evaluating Generative Agents
The section reviews how LLM-MAS are used to evaluate and train generative agents, and points readers to resources for evaluation research.
5 LLM-MAS for Evaluating Generative Agents
- LLM-MAS evaluation faces constrained evaluation abilities, vulnerable benchmarks, and unobjective metrics, while researchers remain unsure how to design suitable indicators and methods.The survey also identifies SFT, reinforcement learning, and synthesized training data as three aspects of generative-agent training.Source: §5 · p. 6
5.1 Representative LLM-MAS for Evaluating Generative Agents
- Researchers use LLM-MAS scenarios to examine agents’ strategic capabilities and, with MuMA-ToM, their understanding of human interaction in a real home environment.Examples include long-term strategy, leadership, and competitiveness; MuMA-ToM uses video and text descriptions.Source: §5.1 · p. 6
- Training work uses LLM-MAS to enhance SFT data, support multi-agent reinforcement learning, co-train agents on split trajectories, and provide alignment rewards.The survey describes Long- and Short-Trajectory Learning as maintaining synergy while preserving each agent’s fine-grained performance, and a multi-agent alignment scheme as addressing instability and reward gaming concerns in reward-based RL optimization.Source: §5.1 · pp. 6–7
5.2 Resources of LLM-MAS for evaluations
- Table 3 is presented as a reference listing evaluation studies’ code, datasets, and benchmarks.The survey says the table is intended to serve future researchers.Source: §5.2 · p. 7
6 Challenges and Future Directions
The survey identifies challenges in generative agents, interactions, and LLM-MAS evaluation, then outlines corresponding research directions.
6.1 Challenges posed by generative agents
- Generative agents may fail to align with mock objects, hallucinate during interactions, and forget complex input because of limited long-text capability.The survey says enhancement methods can alleviate hallucination but cannot solve it; it points to alignment, hallucination reduction, long-text capability, and more complex reasoning as future directions.Source: §6.1 · p. 7
6.2 Challenges posed by interactions
- Repeated LLM queries for each action make inference slow, and this efficiency problem is magnified as systems scale, especially with large action spaces.SoMoSiMu-Bench replaces edge generative agents with rule-based agents, but the problem remains unsolved for generative agents with complex action spaces.Source: §6.2 · pp. 7–8
- Because each round depends on the preceding round, errors can accumulate; rule-based intermediate correction has been used, but substantial room for improvement remains.The survey also reports that IOA’s Internet-like communication architecture enhanced scalability and adaptability to dynamic tasks.Source: §6.2 · p. 8
- Researchers are working to reduce LLM-MAS communication costs, but the survey describes these efforts as still at an early stage.Examples include the alignment-based OPTIMA method and AgentScope’s industrialized parallel messaging method.Source: §6.2 · p. 8
6.3 Challenges of Evaluating for LLM-MAS
- Current population-level evaluation struggles to provide sufficiently detailed, specific, and direct indicators, and distribution comparisons with real environments omit running-process details.The survey proposes studying large-scale LLM-MAS to evaluate and discover scale effects.Source: §6.3 · p. 8
- Different LLM-MAS of the same kind cannot be compared without benchmarks, and a common framework for individual and total-based evaluation is lacking.The survey anticipates that common test benchmarks and evaluation methods will emerge in future research.Source: §6.3 · p. 8
7 Conclusion
The survey organizes LLM-MAS research across task-solving, simulation, and evaluation, and identifies challenges and possible future directions.
7 Conclusion
- The survey reviews LLM-MAS across task-solving, simulation, and evaluation, provides a taxonomy of techniques and their development histories, and proposes challenges and future directions.The taxonomy is intended to draw connections among existing research.Source: §7 · p. 8
Limitations
The survey’s coverage is constrained by brief method summaries and its selection of research venues.
Limitations
- Page limits prevent exhaustive technical detail, and the survey’s focus on *ACL, NeurIPS, ICLR, AAAI, and arXiv may miss important work from other venues.The authors say they plan to update the survey and supplement overlooked work.Source: Limitations · p. 9