Source-linked AI summary
Theory of Mind for Multi-Agent Collaboration via Large Language Models
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, Katia Sycara
TL;DR
The paper asks whether LLM-based agents can collaborate and reason about teammates’ mental states in interactive tasks. It evaluates them in a cooperative text game against MARL and planning baselines, finding collaborative and high-order ToM capabilities alongside planning failures that explicit belief states can mitigate.
Problem
Evidence about LLM abilities in multi-agent collaboration and interactive Theory of Mind remains limited.
Method
The study evaluates LLM agents in a cooperative embodied text game with dynamic ToM tasks, comparing them with MARL and planning-based baselines.
Results
LLM-based agents show collaborative behavior and high-order ToM, while GPT-4 teams achieve full scores and belief representations improve efficiency and reduce invalid actions.
Takeaways & Limitations
Explicit belief-state representations provide a structured way to improve LLM collaboration and ToM inference in these task scenarios.
Takeaways & Limitations
The evaluation uses a relatively simple environment with five nodes and five bombs and homogeneous teams of three agents.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities in multi-agent collaborations remains largely unexplored. This study evaluates LLM-based agents in a multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi-Agent Reinforcement Learning (MARL) and planning-based baselines. We observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limitations in LLM-based agents' planning optimization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit belief state representations to mitigate these issues, finding that it enhances task performance and the accuracy of ToM inferences for LLM-based agents.
1 Introduction
The study examines LLM-based agents’ embodied collaboration, belief maintenance, and Theory of Mind in interactive multi-agent tasks. It evaluates these capabilities against existing concerns about LLM cognition and develops methods for collaborative assessment.
- The study evaluates LLMs’ collective behavior and proficiency in multi-agent collaboration.
- The text-based game tests embodied interaction, exploration, world-belief maintenance, and collaboration among agents.
- Theory of Mind is evaluated as reasoning about concealed mental states relevant to social interaction and collaboration.
- The study compares LLM-based agents’ embodied interaction with reinforcement-learning and planning-based baselines.
- It identifies collaboration failures and proposes explicit belief-state representations as a prompt-engineering mitigation.
- The evaluation extends ToM assessment to interactive teamwork involving dynamic beliefs and communication among multiple agents.
2 Related Work
Prior work studies LLM reasoning, planning, and Theory of Mind largely through static text tasks or exploratory social environments. This study places ToM in dynamic interactive teamwork and connects explicit belief representations to collaboration.
- LLMs show reasoning and plan-generation abilities, but researchers report limitations in actionable planning for real-world interaction.
- Prior ToM evaluations use unexpected-transfer and unexpected-contents tests, with leading LLMs exceeding 90% on reported cases.
- Other findings show difficulty with complex ToM involving communication or second-order beliefs.
- This study evaluates ToM during interactive teamwork, where observations and communication dynamically update agents’ mental states.
- Prior work uses ToM and explicit intention representations to enhance artificial-agent collaboration, motivating explicit belief representations here.
- Research on human collaboration has not established how findings transfer to hybrid human-agent or fully automated teams.
3 Multi-agent Collaboration Tasks
The paper introduces a cooperative search-and-rescue game in which three agents locate and defuse color-coded bombs across a connected-room environment. Performance reflects both coordination quality and completion efficiency.
- The environment simulates a search-and-rescue mission requiring collaborative problem solving by embodied agents.
- Three agents coordinate locations, cutter colors, and bomb sequences to safely defuse bombs in an unexplored environment.
- The evaluation uses five rooms and five bombs, with single-, double-, and triple-phase bombs across three possible colors.
- Each successfully defused phase awards 10 points, with 90 points possible per mission.
- Team score measures coordination quality, while rounds to completion measures collaboration efficiency.
- The environment was initially implemented for multi-agent reinforcement-learning agents using the Gym API.
- LLM agents interact sequentially through natural-language observations, actions, and per-round communication.
4 LLM-based Embodied Agents
LLM-based embodied agents interact with the task through a text interface that translates observations and language into actions. Agents communicate, update explicit beliefs, and plan using limited interaction history.
- The experiments evaluate gpt-3.5-turbo-0301 and gpt-4-0314 as embodied agents in a text-based game.
- Agents receive game rules and only the previous two rounds of history, totaling 4096 tokens.
- The framework contains three LLM agents, a text interface, and a task environment connected through encoded actions and returned observations.
- Agents choose actions and send messages after receiving environmental observations through the interface.
- Because agents alternate messaging, teammates receive communication and feedback through subsequent observations.
- Input-size limits prevent retaining the full interaction history, creating a need to preserve long-term task information.
- Explicit textual belief states are updated from observations and preserved for subsequent action planning.
- Belief-state update rules are zero-shot, although an initial description demonstrates the intended format.
5 Experiments
The experiments evaluate three-agent LLM teams with and without explicit belief states, alongside MARL and planning baselines, in a partially observed bomb-defusal task with embedded ToM questions.
- Teams of three agents were evaluated across four conditions formed by GPT-4 or ChatGPT and presence or absence of belief-state representation.
- Each trial randomized starting locations, room connections, bomb distributions, and sequences, while agents relied on partial observations and communication to share information.
- Baselines: MAPPO provided a learned MARL baseline, using recurrent actor-critic networks and intermediate rewards for correct or incorrect wire-cutter choices.
- Baselines: The CBS-based planner jointly generates assignments and collision-free paths subject to precedence and temporal constraints while maximizing a user-defined reward.
- Theory of Mind evaluation: ToM evaluation covered introspection, first-order inference about others’ hidden states, and second-order inference about others’ beliefs about mental states.
- Theory of Mind evaluation: The ToM questions adapted the Sally–Anne paradigm by testing whether agents recognized teammates’ awareness of state or belief changes from observations and communication.
6 Results
LLM teams showed strong collaboration and ToM abilities, especially with GPT-4 and explicit beliefs, but remained less efficient than the optimal planner because of long-context failures and task-state hallucinations.
- Task performance: The CBS Planner completed tasks in 6.0 rounds, whereas MAPPO required 11.0 rounds after 45 million training timesteps.
- Task performance: GPT-4 teams achieved full scores, while explicit belief representation improved efficiency from 28.3 to 12.3 rounds; ChatGPT failed all experiments with an average team score of 43.3.
- Basic embodied interactions: All LLM teams exceeded the random baseline, and agents achieved over 80% accuracy on world knowledge and situation modeling in introspection tasks.
- Emergent collaborative behaviors: GPT-4+Belief teams exhibited communication-based coordination, including emergent leadership, delegation, helping, conflict resolution, and information sharing.
- Systematic failures: LLM collaboration was less efficient than the optimal baseline because agents overlooked long-horizon context and generated invalid actions despite relevant information appearing earlier in the prompt.
- Theory of Mind inference: GPT-4+Belief agents answered nearly 70% of second-order ToM questions correctly, while GPT-4 surpassed ChatGPT across all three ToM levels.
- Theory of Mind inference: Communication-heavy ToM cases remained difficult because inaccurate messages and false intentions produced false beliefs that agents struggled to track.
7 Discussions
LLM-based agents show substantial collaborative and planning abilities, but systematic failures in long-horizon reasoning and task-state tracking limit their effectiveness. Explicit belief states improve performance while ToM remains limited in dynamic, communication-heavy teamwork.
- With suitable prompt engineering, LLM-agent teams perform comparably to state-of-the-art MARL algorithms despite decentralized, zero-shot operation.MARL agents receive task-specific training and a centralized critic, whereas LLM-based agents process actions step by step.
- LLMs exhibit emergent social interaction skills when placed in multi-agent collaborative tasks.The study links this behavior to interactive settings in which human language is learned and used.
- Neglecting long-horizon context and hallucinating about task state can rapidly disseminate misinformation and hinder collaboration.These systematic failures cause agents to form widespread false beliefs through communication.
- Explicit belief-state representations improve team performance by helping agents track task-related information and update beliefs from observations.The findings suggest that LLMs can revise textual belief descriptions based on observed information.
- LLM ToM capabilities remain limited in interactive teamwork involving dynamic beliefs and intensive communication.The evaluation targets reasoning conditions more complex than traditional static ToM tests.
8 Conclusions
The study finds that LLM-based agents can perform complex embodied collaboration at a level comparable with state-of-the-art reinforcement learning while displaying emergent collaboration and high-order ToM. It also identifies systematic failures and proposes explicit belief states as a mitigation.
- LLM-based agents handle complex multi-agent collaborative tasks at a level comparable with a state-of-the-art reinforcement learning algorithm.
- The agents show evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities.
- Explicit belief states are proposed as a prompt-engineering method to mitigate systematic failures limiting LLM-agent performance.The representation incorporates world knowledge into the model input.
Limitations
The study’s limitations concern evaluation scope, task and team simplicity, human-centered deployment, and the approximation used to establish ToM ground truth. The proposed belief-state approach could also be extended to represent other agents’ beliefs.
- The evaluation uses a relatively simple environment with five nodes and five bombs, so larger and more constrained tasks remain to be tested.
- The current team contains three agents with homogeneous policies, leaving heterogeneous and human-agent teams for future evaluation.Human-centered evaluation would address trust, transparency, and human-agent co-training.
- The ToM ground truth is only an approximation because human annotators infer answers from a global task-state and interaction-history view.This assumption may be ambiguous when agents hold false beliefs or experience miscommunication.
- Using each agent’s maintained belief state as ground truth is proposed as an alternative ToM evaluation method.
- The belief-state method could extend from introspective belief to first-order or second-order beliefs about other agents’ world knowledge.Such representations could support direct comparisons between agents’ beliefs instead of Sally-Anne-style questions.
A Prompts
The appendix specifies a cooperative search-and-rescue game in which three players navigate a five-room graph, communicate observations, maintain beliefs, and defuse colored-sequence bombs using role-specific tools.
- A.1 Task context: The task places three players in a five-room search-and-rescue environment with five hidden bombs to defuse.
- A.1 Task context: Rooms form a connected graph centered on room 0, with additional links among rooms 5, 6, and 8 and between rooms 3 and 8.
- A.1 Task context: Bombs require one to three color-coded tools applied in the correct phase sequence.
- A.1 Task context: Alpha, Bravo, and Charlie receive red-green, green-blue, and blue-red wire cutters, respectively.
- A.1 Task context: Each round, agents may move, inspect a bomb, or apply a tool, while also sending a message to both teammates.
- A.1 Task context: Agents observe only their current room, teammate messages, the round, score, and teammate locations; teammates’ actions and consequences require explicit communication.
- A.1 Task context: Prompts require fixed-format action and communication responses, including commands for movement, inspection, tool use, and team messages.
B Environment feedback for Error correction
Environment feedback identifies invalid actions, explains why movement or bomb operations cannot proceed, and prompts agents to reason about their own and others’ knowledge. These checks support error correction and Theory of Mind inference during collaboration.
- Error correction: Invalid actions receive feedback explaining the specific constraint that prevented execution.Examples include nonadjacent movement and attempting operations where no bomb is present.
- Error correction: Bomb-operation feedback identifies incorrect tool sequences and missing tools, while suggesting alternative actions or teammate assistance.Agents are told to apply another colored tool first or ask teammates who possess the required tool.
- Theory of Mind queries: The environment queries whether an agent knows room contents, bomb states, defusal events, or bomb sequences.These prompts explicitly target the agent’s knowledge of task-relevant state and history.
- Theory of Mind queries: Additional queries assess whether another player knows those facts, extending knowledge checks from self-state to teammate state.The queried facts include room contents, bomb changes, defusal events, and bomb sequences.
- Higher-order Theory of Mind: Higher-order queries ask whether a player is aware that the agent knows or has changed particular task information.These checks use observations and previous history to evaluate nested awareness.