MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

1 Introduction

• The authors identify a gap in quantitative assessment of true interaction capabilities in multi-agent systems.Prior benchmarks emphasize understanding and reasoning in environments, while work on specific social skills has limited scope and quantitative metrics.Source: §1 · pp. 1–2
• The metrics span cognition, adaptability, rationality, and collaboration, including judgment, reasoning, self-awareness, deception, cooperation, and coordination.Cooperation measures communication and agreeability; coordination measures collaboration facilitation.Source: §1 · p. 2
• The proposed competition-based benchmark covers over 100 cases in five scenarios and uses seven metrics to evaluate multi-agent abilities.The benchmark evaluates LLM agents by having them compete against a fixed type of LLM.Source: §1 · p. 2
• GPT-o1, GPT-4, and GPT-3.5 are reported as the top performers, ahead of PaLM 2, Claude 2, and Cohere.The models also show distinct profiles in judgment, rationality, and cooperation.Source: §1 · p. 2
• PGM-aware agents outperform their vanilla versions by 37% on average across the evaluated abilities.The approach integrates LLMs with symbolic reasoning.Source: §1 · p. 2

2 Related Work

• Chain of Thought (Wei et al., 2022), Tree of Thought (Yao et al., 2023a), Graph of Thought (Yao et al., 2023b; Besta et al., 2023), and ReAct (Yao et al.) are recent works that improve LLM reasoning; API-bank (Li et al., 2023b) benchmarks tool-augmented LLMs.Source: §2 · p. 2
• LLM-powered agents have been developed for complex tasks and multi-agent settings, including Auto-GPT and Generative Agents (Park et al., 2023).Generative Agents describes a sandbox with 25 AI agents simulating human actions and recording experiences.Source: §2 · p. 3
• Benchmarks cover single-agent games or real-life environments, while multi-agent social studies still lack quantifiable measurements of social abilities.The cited work includes Liu et al. (2023), Wu et al. (2023), and Gioacchini et al. (2024) on single agents, and Agashe et al. (2023) and Abdelnabi et al. (2023) on social abilities.Source: §2 · p. 3

3 Benchmark

• The benchmark measures LLMs in multi-agent competitions and proposes seven quantitative metrics across cognition, adaptability, rationality, and collaboration.The benchmark includes more than 100 cases across five scenarios.Source: §3 · p. 3

3.1 Scenarios

• Two social-deduction games assess cognition and adaptability, while three game-theory scenarios focus on rationality and collaboration.The scenarios include Chameleon, Undercover, cost sharing, multi-turn Prisoner’s Dilemma, and Public Good.Source: §3.1 · pp. 3–4

3.2 Competition Settings

• Challenger LLMs compete against the same GPT-4-powered defender in identical game settings, with capabilities evaluated using intermediate results and win rates.The benchmark ranks challengers by their win rates against the defender.Source: §3.2 · p. 4

3.3 Evaluation Metrics

• Win Rate averages performance across seven challenger roles; Judgment uses correct votes, while Reasoning assesses correct role and deduction predictions.Judgment targets understanding global information from partial information; Reasoning compares deductions with gold situations and other players’ true subjective deductions.Source: §3.3 · p. 4
• Deception measures successful blending or incorrect secret-word guesses, while Self-Awareness measures correct role identification and consistent behavior.The deception metric weights incorrect code guesses by λ = 0.25 because not all games trigger them.Source: §3.3 · p. 4
• Cooperation measures contribution to collective efficacy in cost-sharing games, while Coordination tracks constructive proposals that support successful collaboration.The number of successful collaborations proposed by the challenger is used in the cost-sharing games.Source: §3.3 · pp. 4–5
• Rationality measures agents’ ability to optimize their own interests according to the rules of the game-theory scenarios.Its formulation uses decisions across Prisoner’s Dilemma and Public Good competitions.Source: §3.3 · p. 5

4 PGM-Aware Agent

• LLMs struggle with ambiguous relationships and causal reasoning in multi-agent inference; the paper integrates probabilistic graphical models with LLMs to address this challenge.The authors describe PGMs as Bayesian tools for representing dependencies between random variables.Source: §4 · p. 5

4.1 PGM Structure

• The PGM represents conditional dependencies among agents and uses two-hop understanding: the agent analyzes from its own perspective and from other agents’ perspectives.In the example, the PGM-aware player maintains interpretations of the global status from each player's perspective.Source: §4.1 · pp. 5–6
• The PGM can use text-represented probabilities as well as numerical ones, structuring multi-party, multi-hop understanding for LLMs.This choice reflects the text-based input and output nature of LLMs.Source: §4.1 · p. 6

4.2 LLM Decision with PGM

• The PGM-aware agent makes decisions conditioned on both the PGM and game context, using a prompt to guide the decision step.The decision prompt receives the PGM acquired in the preceding step.Source: §4.2 · p. 6

5 Experiments

• Challenger LLMs played against the same GPT-4 defender, and models were ranked by win rate; all participating LLMs used temperature 0.The fixed defender and temperature were used to support comparability and reduce game randomness.Source: §5 · p. 6

5.1 LLM Leaderboard

• GPT-4-turbo led with a 57.2% win rate, GPT-4 followed at 53.3%, and GPT-4-turbo exceeded Llama-2-70B by more than threefold in overall multi-agent capabilities.The benchmark's area sizes are reported as directly proportional to winning rates.Source: §5.1 · p. 6
• GPT-4 scored 90% in Judgment and 75.0% in Deception; in game-theory scenarios, it reached 66.7% Coordination and 78.1% Rationality.The reasoning gap among models was narrow, while deception abilities showed significant disparities.Source: §5.1 · p. 6

5.2 PGM Enhancement Performance

• 37% average ability improvement; PGM-aware methods raised win rates by 6.57% across scenarios, with most significantly better in 3–4 of 7 abilities (p<0.05).The t-test comparison found significant gains for most PGM-aware agents in three or four of the seven abilities.Source: Figure 4 · p. 7, §5.2 · p. 7

5.3 Analysis

• In Chameleon, base Llama-2-70B and GPT-4 lost by voting for Player 1; their PGM variants shifted outcomes to non-chameleon wins.Llama-2-70B’s first two PGM estimates indicated the wrong chameleon, while GPT-4+PGM’s analysis aligned its conclusions and explanations without the cited hallucination.Source: §5.3 · p. 8
• GPT-3.5-turbo+PGM increased Win Rate and reduced cost simultaneously; GPT-4+PGM increased Win Rate while keeping cost slightly lower.Without PGM, GPT-3.5-turbo won in Win Rate while GPT-4 performed better in Cost.Source: Table 2 · p. 8, §5.3 · pp. 8–9
• In Prisoner’s Dilemma, GPT-4+PGM won more but scored lower than GPT-3.5-turbo+PGM; in Public Good, PGM models had higher Win Rates but lower payback.The paper treats betrayal and reduced public contribution as rational choices in those scenarios.Source: Table 2 · p. 8, §5.3 · p. 9

6 Discussion: Generalization of Benchmark

• The benchmark can extend beyond the paper’s scenarios to tasks where multi-agent participants have local views and must infer global information to make decisions toward a goal.The targeted abilities are Cognition, Adaptability, Rationality, and Collaboration.Source: §6 · p. 9
• Judgment assesses unknown information, Reasoning compares an agent’s perspective with actual and others’ views, and Self-awareness applies when roles are undisclosed.Deception measures influence through false information; Cooperation and Coordination measure agreement and proposal quality, respectively.Source: §6 · p. 9
• Rationality is defined as the proportion of decisions that maximize an agent’s outcomes.Source: §6 · p. 9

7 Conclusion

• The framework quantitatively assesses seven abilities in LLM multi-agent environments: judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality.Source: §7 · p. 9
• GPT-4-turbo emerged as the most capable evaluated model, outperforming others by a threefold margin.Source: §7 · p. 9
• PGM enhancement amplified the models’ inherent abilities by 37%.The conclusion presents this as evidence of PGM’s potential to enhance LLM capabilities.Source: §7 · p. 9

Limitation

• The evaluation is preliminary; its game scenarios and topic settings need substantial expansion.The paper also says whether the metrics help in real-life scenarios was not explored, and integrating PGM-enhanced abilities through fine-tuning requires further exploration.Source: Limitation · p. 9
• Whether the proposed metrics help in real-life scenarios remains unexplored.Source: Limitation · p. 9
• Integrating PGM-enhanced abilities into LLMs through fine-tuning requires further exploration.Source: Limitation · p. 9

• The five scenarios are mostly controlled through prompts, so the authors report a low possibility of offensive outputs.Source: Ethical Considerations · p. 9
• The authors checked that the scenario data contain no personal data or unsuitable content and follow artifact-use licenses.Source: Ethical Considerations · pp. 9–10

A.1 Competition Settings

• The challenger plays each role in Chameleon and Undercover, while game-theory win rates measure successful agreement in Cost Sharing or player victory.The challenger’s role-specific win rates are calculated separately for the two social-deduction games.Source: §A.1 · p. 11, §A.2 · p. 12
• Settings comprise 20 topics each for Chameleon and Undercover and 21 settings for each game-theory scenario.The social-deduction topics were selected after simulations to balance role win rates; the game-theory tasks use constructed settings.Source: §A.1 · pp. 11–12

A.2 Win Rate Definition

• Chameleon and Undercover win-rate definitions use outcome-specific role credits, while game-theory win rate is the testing player’s competition-win ratio.Source: §A.2 · p. 12

A.3 PGM Enhancement Performance

• For PGM-aware agents, the radar chart depicts LLMs’ performance on seven metrics, and larger polygon areas correlate with higher average game-winning rates, supporting the proposed metrics’ effectiveness for assessing capabilities.Source: Figure 6 · p. 12

A.4 More Case Studies

• PGM analyses supported role inference and deceptive clues; GPT-4+PGM won with consistently good clues across two rounds, while GPT-3.5-turbo+PGM lost.The authors conclude that a more capable LLM can benefit more from the PGM structure.Source: §A.4 · p. 14

A.5 PGM prompts

• The study compared text-based PGM analysis with direct probability-matrix output and mainly selected text-based analysis because it helped all tested LLMs, including Llama-2-70B.For GPT-4 and GPT-3.5-turbo, both prompt types worked well; Llama-2-70B performed much better with text-based analysis.Source: §A.5 · p. 14

A.6 Defect and Investment Tendency

• Across tested LLMs, average total investments almost all exceeded 100; GPT-4 gave reasonable investment, while across seven multipliers [1,1.2,1.5,1.8,2,2.5,3], investment showed no increasing tendency.The authors interpret the first pattern as a lack of arithmetic awareness and the second as insensitivity to increased possible payback in the communal setting.Source: §A.6 · pp. 14–15

A.7 Game rules and Prompt Design

• The appendix lists game rules and prompts for the five scenarios in Tables 5 and 6.Source: §A.7 · p. 16, Table 5 · p. 17, Table 6 · p. 18