Source-linked AI summary
MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, Jiaxuan You
TL;DR
Existing benchmarks inadequately capture coordination and competition in multi-agent interactions. MultiAgentBench introduces a diverse benchmark and MARBLE framework that evaluate task completion, coordination, and competition across protocols and planning strategies. The benchmark reports emergent social behaviors and a 3% improvement in milestone achievement rates from cognitive planning.
Problem
Existing evaluation paradigms focus mainly on single-agent tasks and overlook dynamics intrinsic to multi-agent coordination and competition.
Method
MultiAgentBench and MARBLE evaluate six diverse interactive scenarios using milestone-based KPIs, coordination and competition metrics, multiple topologies, and planning strategies.
Results
The experiments highlight emergent social behaviors, and cognitive planning improves milestone achievement rates by 3%.
Takeaways & Limitations
The framework provides evaluation insights into task success, coordination quality, communication, competition, and emergent multi-agent behavior.
Takeaways & Limitations
The benchmark covers selected domains and models rather than the full range of real-world settings and available model families.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. Code and datasets are public available at https://github.com/MultiagentBench/MARBLE.
1 Introduction
MultiAgentBench addresses the lack of benchmarks capturing multi-agent coordination and competition by evaluating LLM-based systems across diverse interactive scenarios. It measures task performance alongside coordination quality using milestone-based metrics and supports varied communication and planning strategies.
- Existing benchmarks mainly evaluate isolated reasoning and generation, overlooking dynamics intrinsic to multi-agent interactions.
- MultiAgentBench evaluates LLM-based multi-agent systems across diverse task-solving and simulation scenarios.
- The benchmark covers domains from collaborative coding to gaming and measures both coordination dynamics and competitive interactions.
- Its KPI tracks milestone progress and individual contributions, while MARBLE supports star, chain, tree, and graph communication topologies.
- 3%: cognitive planning improves milestone achievement rates in the benchmark.
2 Related Work
Prior work applies multi-agent systems across research, software, gaming, and other domains, while recent studies distinguish cognitive from population scaling. These lines of work also leave persistent challenges in communication, emergent behavior, and scalability.
- Multi-agent systems support scientific research, software engineering, and gaming applications.
- Applications also extend to social deduction, game theory, healthcare, business, education, and urban planning.
- Communication, emergent behavior, and scalability remain challenges despite progress in multi-agent applications.
- Recent work identifies cognitive scaling and population scaling as complementary paradigms for multi-agent systems.
- Cognitive scaling studies adaptive architectures and self-organizing communication, whereas population scaling uses larger collectives and diverse collaboration patterns.
3 Methodology
MARBLE combines modular coordination, agent graphs, cognitive state, communication protocols, planning strategies, and milestone-based evaluation. MultiAgentBench evaluates collaborative and competitive scenarios using flexible milestones and agent-level KPI tracking alongside task scores.
- 3.1 Framework Design: MARBLE coordinates an Agent Graph, Cognitive Module, and Coordinate Engine to support adaptive collaboration and strategic task execution.
- 3.1 Framework Design: The Agent Graph represents agents and explicitly defined collaboration, supervision, or negotiation relationships that constrain subsequent communication and coordination.
- 3.1 Framework Design: The Cognitive Module updates personas, inter-agent relationships, and reasoning strategies so agents can adapt their strategies in complex environments.
- 3.1 Framework Design: The framework distinguishes planners, who allocate and strategize, from actors, who execute tasks through environment and agent interactions.
- 3.1.1 Coordination Engine: Star, tree, graph-mesh, and chain protocols provide centralized or decentralized coordination, with chain handoffs suited to dependent tasks but limiting parallel processing.
- 3.1.1 Coordination Engine: Planning uses vanilla prompting, chain-of-thought, group discussion, and cognitive self-evolving strategies, with iterative feedback and task updates.
- 3.2 Benchmark Design: The benchmark spans adapted and human-verified LLM-generated tasks, including research, Minecraft, database analysis, coding, Werewolf, and bargaining.
- 3.2 Benchmark Design: Each task uses flexible milestones monitored by an LLM detector, which records milestone achievement and contributing agents for KPI calculation.
4 Experiment Setup
The experiments compare five function-calling models across task-completion and simulation scenarios, examining task performance, coordination, and protocol effects. gpt-4o-mini generally achieves the strongest task performance, while coordination quality and task success do not always align.
- Experimental Design: Five models are evaluated across diverse scenarios using Task Score and Coordination Score as primary performance axes.The setup uses maximum-token and temperature settings, with up to 5 iterations for research and 20 for Minecraft.
- Model Performance: gpt-4o-mini records a Research Task Score of 84.13%, exceeding Meta-Llama-3.1-8B at 80.87% and Meta-Llama-3.1-70B at 80.80%.In Coding, gpt-4o-mini also records a Task Score of 65.10.
- Model Performance: Meta-Llama-3.1-70B combines a Minecraft Coordination Score of 75.00 with a Task Score of 0.21, showing that coordination alone does not ensure task execution.gpt-4o-mini is described as balancing high task scores with moderate-to-high coordination scores more robustly.
- Model Performance: Meta-Llama-3.3-70B achieves Coordination Scores of 72.00 in Research and 76.30 in WereWolf, yet trails gpt-4o-mini in Task Score across several scenarios.The findings emphasize considering task-specific abilities and coordination skills together.
- Coordination Protocols: The graph-based protocol performs best in Research for task performance, planning efficiency, and token usage, while Tree has the lowest task and coordination scores.Star and Graph yield similar task scores; Cognitive Evolving Planning has superior coordination and a task score comparable to COT.
5 Ablation Study
The ablations examine coordination protocols, planning prompts, iteration limits, and team size. Results indicate that graph coordination and cognitive planning are favorable, while excessive iterations and larger teams can introduce coordination costs.
- Iteration Settings: Task and coordination scores increase from 1 to 7 iterations, then coordination drops sharply at 10 iterations.At 20 iterations, task score recovers while coordination remains nearly unchanged beyond 7 iterations.
- Coordination Protocols: Graph coordination performs best in Research for task performance, planning efficiency, and token usage, whereas Tree performs poorly across task and coordination metrics.Star and Graph produce similar task scores, while Tree has high token consumption.
- Planning Strategies: Cognitive Evolving Planning achieves the strongest coordination and a task score comparable to COT, while group discussion scores worst across the evaluated measures.Figure 6 compares average Research metrics across planning prompt strategies.
- Implications: The ablations support adaptive iteration and moderate team expansion as ways to balance task execution with coordination efficiency.Further increases in team size may counterbalance task-performance gains through additional coordination challenges.
- Agent Numbers: Increasing agents from 1 to 7 decreases overall KPI, although coordination improves significantly from 1 to 3 agents.Average task score rises more gradually, suggesting that larger teams can add coordination challenges.
6 Emergent Behaviors Analysis
The emergent-behavior analysis identifies coordination patterns arising under shared goals, information asymmetry, and role conflicts. These include selective information sharing, trust-polarized collaboration, and role-driven strategy changes.
- Emergent Coordination: Goal-driven “aha-moments” mark agents aligning actions toward shared objectives and catalyze spontaneous multiagent dynamics.The paper links these behaviors to novel coordination strategies and adaptive collective intelligence.
- Strategic Information Sharing: Agents selectively disclose information based on trust and context, but excessive caution by the Seer and Witch can cause missed sharing opportunities and failure.Research agents also evolve proposal details by combining their contributions.
- Trust-Polarized Collaboration: Role identities can split collaboration: suspicious villagers may turn against allies, while werewolves can create false consensus through deception and internal execution.The analysis connects excessive village caution with werewolf exploitation of confusion.
- Role-Driven Strategy Iteration: Roles adapt strategies during play, with the Seer shifting toward leadership and the Witch moving from hoarding information to taking risks.These shifts are presented as task-objective-driven changes in decision-making.
7 Conclusion
The paper introduces MultiAgentBench and MARBLE to evaluate LLM-based multiagent systems across interactive scenarios. Its evaluation extends beyond task success to structured coordination, communication, and competition assessments, while experiments surface emergent social behaviors.
- Benchmark: MultiAgentBench and MARBLE provide a benchmark for evaluating LLM-based multiagent systems across diverse interactive scenarios.
- Evaluation: The framework measures coordination quality through structured planning, communication scores, and competition-driven assessments in addition to task success.
- Findings: Experimental results highlight emergent social behaviors and provide insights for future multiagent research.
8 Limitations
The benchmark and framework have several documented scope boundaries, including limited scenario and model coverage, coarse ablations, incomplete competition mechanisms, and mainly well-defined tasks.
- Scenario and Model Coverage: The benchmark covers selected domains and models rather than the full spectrum of multi-agent settings and current model families.Future work is proposed for open-world environments, richer social cognition, task-oriented dialogue, and additional models such as DeepSeek.
- Ablation Studies: The current analysis emphasizes overall coordination and competition, leaving finer-grained component effects underexplored.Future ablations are suggested for long-term, short-term, and shared memory, along with different multi-agent workflow methods.
- Competition Mechanisms: Competitive tasks do not fully capture multi-party negotiations, repeated strategic play, or stochastic interaction.The authors identify transitions between cooperative and adversarial roles in evolving environments as a future direction.
- Open-Ended and Ill-Defined Tasks: Most framework tasks have well-defined objectives, whereas real-world applications often involve open-ended or ambiguous contexts without clear success criteria.Future extensions could examine exploratory and non-goal-oriented scenarios.
A.1 Contributions
The listed contributors covered the benchmark’s core implementation, environments, evaluation, memory, reasoning, and paper-writing responsibilities.
- Contributions: Kunlun Zhu led the main codebase design, research environment, coordination engine, evaluator basics, and main paper writing.
- Contributions: Hongyi Du contributed milestone generation, the Werewolf framework, analysis and data generation, emergent behavior, limitations, related work, human evaluation, and appendix materials.
- Contributions: Zhaochen Hong contributed environment basics, the communication module, the database environment, and appendix writing.
- Contributions: Xiaochen Yang contributed the memory module, Minecraft environment, and related appendix writing.
- Contributions: Shuyi Guo contributed the evaluator prompt, bargaining environment, and bargaining-related appendix writing.
- Contributions: Zhe Wang contributed the reasoning-agent module, coding environment, and coding-related appendix writing.
A.2 More Details on Multi-agent framework design
The framework combines configurable agent profiles and relationships with communication, action, environment, and memory modules to support adaptive multi-agent interaction.
- Configuration Module: The configuration module ingests task, persona, profile, role, and database information to construct agent attributes and coordination relationships.Agent profiles include capabilities, constraints, and personality traits; relationships may be hierarchical, collaborative, or adversarial.
- Environment Module: The environment module supports coding, research, and negotiation scenarios in which agents select function-calling actions and receive state updates.A domain-specific toolbox provides functions such as code editors and search engines.
- Memory Module: The memory module stores global knowledge and collective decisions in shared memory while retaining personal experiences and local observations in individual memory.Individual memory is divided into long-term and short-term segments, with FIFO management and retrieval-augmented generation for prompt construction.
- Communication Module: The communication module supports context-aware exchanges, structured dialogue, information sharing, role negotiation, plan coordination, and balancing collaboration with competition.
- Action Module: The action module executes agent plans through function calling and structured outputs, feeding outcomes and observations into individual and shared memories.This feedback loop supports adaptation to evolving task requirements and strategy refinement.
A.3 Human Evaluation
The human evaluation compares prompt-based and human ratings of planning and communication in Werewolf tasks, using multiple annotators and correlation analyses. The reported scores are generally closely aligned across five models.
- Evaluation Setup: The evaluation computes Kendall’s, Pearson’s, and Spearman’s correlation coefficients with corresponding p-values for planning and communication scores.
- Evaluation Setup: 60 Werewolf tasks across five LLMs were rated by six NLP-familiar annotators, with each task evaluated independently by two annotators and averaged.
- Results: Table 2 compares human and prompt-based machine evaluation scores for communication and planning in the Werewolf scenario.
- Results: The largest communication-score difference is within 0.38, with gpt3.5-turbo scoring 3.75 by humans versus 4.00 by the machine.
Task Overview
The evaluation shows substantial variation in multi-agent performance across models and scenarios, with long-run Werewolf outcomes strongly favoring llama3.3-70B. Case studies further indicate that trust, timely information sharing, and coordinated role use are important for successful village play, while stronger reasoning alone does not guarantee victory.
- Daily Outcomes: 0.3754 completion ratio was the highest achieved by llama3.3-70B, indicating stronger fulfillment of short-term objectives than competing models.gpt-3.5-turbo and gpt-4o-mini recorded ratios around 0.22–0.25.
- Daily Outcomes: 0.2802 was the only positive villager net score, achieved by llama3.3-70B within a single day–night cycle.Other models produced negative values, reflecting weaker daily confrontations or less effective use of cooperative roles.
- Full-Run Results: Around 35% was llama3.3-70B’s highest villager win rate, while llama3.1-8B and gpt-4o-mini achieved 1.15% and about 3.09%.These results suggest difficulty sustaining decisive collaboration across multiple night/day cycles.
- Full-Run Results: 35% villager win rate made llama3.3-70B the strongest model in full-game Werewolf simulations, exceeding gpt-4o’s 24.73%.It also achieved a positive net score of 0.4511 versus gpt-4o’s −2.1946.
- Case Studies: Improved model capabilities enhanced collaboration and villager performance, but even gpt-4o versus gpt-4o produced a less-than-ideal villager win rate.The case study attributes failure in adversarial settings to uncertainty, deception, and insufficient trust and cooperation.
- Case Studies: Timely role disclosure and information sharing supported successful coordination: the Witch’s sheriff bid rallied villagers, while the Seer’s revelations enabled successive wolf eliminations.The sheriff badge’s transfer and extra voting power contributed to the villagers’ eventual victory.