PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied LLM Agents Learn to Cooperate in Organized Teams

Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L. Griffiths, Mengdi Wang

arXiv:2403.12482v2cs.AIcs.CLcs.CYcs.MA

TL;DR

The paper addresses how to organize embodied LLM agents amid redundant communication and coordination problems. It introduces prompt-based team structures and a Criticize-Reflect process, finding that leadership improves team efficiency and iterative prompt refinement can improve it further.

  • Problem

    LLM agents can over-report, follow instructions indiscriminately, and send redundant messages or interrupt teammates, while free-flowing interaction that could scale to humans remains an open problem.

    Source: §1 · pp. 1–2

  • Method

    The paper organizes embodied LLM-agent teams through prompts and uses a Criticize-Reflect framework in which a critic evaluates episodes and a coordinator proposes revised organizational prompts.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 5

  • Results

    Hierarchically organized teams with a designated or elected leader have superior team efficiency, which can be further improved by Criticize-Reflect.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    With proper organization, LLM agents can display cooperative behaviors such as constructive suggestions, help-seeking, and reporting task progress.

    Source: §1 · p. 2

    The work uses a single environment and lacks human evaluation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as integral tools for reasoning, planning, and decision-making, drawing upon their extensive world knowledge and proficiency in language-related tasks. LLMs thus hold tremendous potential for natural language interaction within multi-agent systems to foster cooperation. However, LLM agents tend to over-report and comply with any instruction, which may result in information redundancy and confusion in multi-agent cooperation. Inspired by human organizations, this paper introduces a framework that imposes prompt-based organization structures on LLM agents to mitigate these problems. Through a series of experiments with embodied LLM agents and human-agent collaboration, our results highlight the impact of designated leadership on team efficiency, shedding light on the leadership qualities displayed by LLM agents and their spontaneous cooperative behaviors. Further, we harness the potential of LLMs to propose enhanced organizational prompts, via a Criticize-Reflect process, resulting in novel organization structures that reduce communication costs and enhance team efficiency.

1 Introduction

The paper studies how organizational structures can help embodied LLM agents coordinate, addressing redundant communication and interruptions in unstructured teams. It finds benefits from designated leadership and from iteratively improving organizational prompts.

1 Introduction

  • The framework organizes LLM agents into varied team structures to study how organizational structures affect multi-agent coordination.The study asks what role structures play and how to optimize them for efficient coordination.Source: §1 · p. 2
  • Uncoordinated agents sent redundant, repetitive messages and interrupted others’ actions, motivating the study of organizational structures for efficient collaboration.The introduction describes free-flowing LLM-agent interaction that can potentially scale to include humans as an open problem.Source: §1 · p. 2
  • With a designated leader, three-agent teams improved efficiency by up to 30% with almost no extra communication cost, and the benefit also held in five-agent cases.The paper also reports that human leaders coordinated agent teams better than AI leaders.Source: §1 · p. 2
  • Criticize-Reflect prompts led agents to form novel team structures with reduced communication cost and improved efficiency.The framework reflects on team performance and generates improved organizational prompts.Source: §1 · p. 2

2 Related Works

Related work covers LLM agents, multi-agent cooperation, and prompt optimization. The paper distinguishes its focus on embodied teams of three or more agents and organizational structures that can be optimized for communication efficiency.

2.1 LLM Agents

  • Prior LLM-agent research covers translating environment observations into language, turning reasoning into actions, prompting methods, fine-tuning, and techniques to mitigate model constraints.The cited techniques include ReAct, Reflexion, chain-of-thought reasoning, external tools and documents, and skill libraries.Source: §2.1 · p. 3

2.2 Multi-Agent Cooperation

  • Multi-agent work spans communication, planning, leadership, team dynamics, text-based LLM collaboration, and role assignment; embodied LLM cooperation has also been studied.The paper contrasts prior fixed structures and two-agent or one-human-one-agent settings with its focus on optimizing structures for teams of three or more.Source: §2.2 · p. 3

2.3 Prompt Optimization

  • Prompt optimization includes language-model heuristic search, soft prompt tuning, and prefix tuning; this work targets interpretable prompts written in natural language.The paper draws on prior work by Yang et al. [62], Zhou et al. [73], and Pryzant et al. [38].Source: §2.3 · p. 3

3 Method

The method extends embodied LLM agents to organized teams of three or more, with separate communication and action phases. A Criticize-Reflect loop revises organizational prompts using episode histories and environment costs; experiments use a VirtualHome housekeeping setting.

3.1 Architecture and Multi-Agent Communication

  • The architecture extends embodied LLM agents to teams of three or more and separates communication from action, allowing flexible team communication patterns.Agents take turns broadcasting, selecting one or multiple recipients for distinct messages, or remaining silent; each agent keeps communication history.Source: §3.1 · p. 4

3.2 Criticize-Reflect Method for Improving Organizational Structure

  • The Critic evaluates episode dialogue and action histories, while the Coordinator combines its analysis with environment cost metrics to propose a different organizational prompt for the next episode.The loop iteratively criticizes and reflects on prompts to discover more effective organizational structures.Source: §3.2 · p. 5

3.3 Environment Setup

  • Experiments use VirtualHome-Social housekeeping tasks, where agents have room-limited observations but can explore other rooms and communicate without a range limit.Team efficiency is measured by time steps to completion and average communicated tokens per step; runs use randomized initial states.Source: §3.3 · p. 5

4 Main Results

Designated leadership improved team performance with little added communication cost, while leadership election, human leaders, and model differences revealed important trade-offs. Criticize-Reflect generated novel organizational structures that improved efficiency, and organized teams displayed distinct cooperative behaviors.

4.1 A Designated Leader Enhances Performance

  • Across 20 random seeds, a performance comparison found a significant 9.76% improvement (t(38)=1.71, p<.05); designated leaders improved 3×GPT-4 by 5.28% and 1×GPT-4+2×GPT-3.5-turbo by 9.61%.The latter improvements had p=.20 and p=.08, respectively. Compared with disorganized teams, designated-leader teams had slightly increased or even lower communication costs.Source: §4.1 · p. 6
  • In 3×GPT-4 teams, electing a leader improved efficiency over a fixed leader but substantially increased communication cost; leadership was reelected about every nine steps.Agents often voted for others, though their voting rationale was unclear on most occasions.Source: §4.1 · p. 6
  • Human leaders improved task completion time and communication efficiency relative to GPT-4 leaders in teams of three GPT-4 agents.Three human players participated in the experiment.Source: §4.1 · p. 7

4.2 Leadership and Open Communication Matters

  • GPT-4 led mixed teams more efficiently than GPT-3.5-turbo in both three- and five-agent settings; constructive feedback improved 3×GPT-4 efficiency but lowered GPT-3.5-turbo efficiency, while increasing communication cost in both.The feedback comparison reports t(38) = 0.87, p = 0.14 for 3×GPT-4 and t(38) = 0.27, p = 0.40 for GPT-3.5-turbo.Source: §4.2 · p. 7

4.3 Emergence of Cooperative Behaviors

  • Under hierarchy, leaders devoted over 60% of their communication to orders, while followers shared and requested more information; disorganized agents issued orders but requested guidance in under 10% of dialogues.Cooperative behaviors were classified as information sharing, leadership and assistance, and requests for guidance; the GPT-4 classifier had 91.67% accuracy on 20 human-labeled dialogue samples.Source: §4.3 · p. 8

4.4 Novel Organizational Structures

  • Removing the Critic and using Reflection alone caused a performance decline, with the Coordinator producing vague organizational prompts.The comparison supports the role of the dual Criticize-Reflect architecture.Source: §4.4 · pp. 8–9
  • Criticize-Reflect improved task efficiency for all three tested teams; communication cost rose slightly for 3×GPT-3.5-turbo but fell for 3×GPT-4 and the mixed team.The Critic evaluates the team trajectory, individual behaviors, and leadership rankings to inform prompt improvement.Source: §4.4 · p. 8, Figure 8 · p. 9
  • Criticize-Reflect proposed chain, dual-leader, and dynamic structures; the dynamic prompt outperformed a fixed GPT-4 leader across six new tasks, with marginal benefits on easy tasks.The advantage was stronger on the three hard tasks.Source: Figure 9 · p. 9, §4.4 · p. 9

5 Conclusion

Hierarchically organized teams with designated or elected leaders showed superior efficiency, which Criticize-Reflect could further improve. The evaluation is limited to one environment and lacks human evaluation.

5 Conclusion

  • Hierarchically organized teams with designated or elected leaders showed superior team efficiency, further improved by Criticize-Reflect.The paper develops an embodied multi-LLM-agent architecture to support these organizational structures.Source: §5 · pp. 9–10
  • The evaluation is limited to one environment and lacks human evaluation; VirtualHome also cannot hold hundreds of agents.The paper identifies broader environments and larger organizations as future work.Source: §5 · p. 10

A Prompt Templates

  • Actor and Communicator prompts include the organization instruction, current goal, progress and available actions, latest 12 dialogue messages, and latest 20 action steps.The organization instruction may be manually designed or automatically generated.Source: A Prompt Templates · p. 15

B.1 Details of the Critic

  • The Critic reviews episode trajectories and ranks leadership by communication, conflict resolution, flexibility, and strategy, without scores because criteria vary by episode.Its assessments inform the Coordinator’s redesign of organizational prompts.Source: §B.1 · p. 19

B.2 Details of the Coordinator

  • Coordinator prompts specify organizational structure through topology, role assignment, and behavioral rules.Topology can be decentralized, centralized with a specific leader, or pyramid-shaped; role assignments describe each agent’s duty and leadership status.Source: §B.2 · p. 19

C.1 Complete list of basic experimental results

The appendix reports results across group settings and organization instructions, including leadership and communication-cost differences between LLM teams. Scaling experiments show communication costs rising nearly linearly, while adding agents does not always improve completion time.

C.1 Complete list of basic experimental results

  • GPT-4 exhibited poorer leadership than Llama2-70B in the 1×GPT-4+2×Llama2-70B group; Table 1 covers performance across organization instructions.The teams containing Llama2-70B also had much higher communication costs than teams containing GPT-3.5-turbo.Source: §C.1 · p. 20, Table 1 · p. 20

C.2 Scaling up the team size

  • Communication costs increased in a nearly linear way as team size grew, but task completion time did not always improve with more agents.The authors suggest the approach will not have dimension explosion when scaling up.Source: §C.2 · p. 21, Table 2 · p. 21
  • 60.67±15.06 (9 agents) was worse than 43.00±2.16 (7 agents), with the apartment potentially too crowded for nine agents.Table 2 reports performance for different team sizes.Source: §C.2 · p. 21, Table 2 · p. 21

C.3 Across Task Generalizability

The paper tests whether the dynamic-leadership prompt found on Prepare_Afternoon_Tea generalizes across other tasks. Figure 10 groups tasks by typical step counts and reports experiments using a mixed GPT-4/GPT-3.5-turbo team.

C.3 Across Task Generalizability

  • The dynamic-leadership prompt discovered on Prepare_Afternoon_Tea was tested across tasks to assess generalizability.The experiments used a 1×GPT-4+2×GPT-3.5-turbo team over two seeds per task.Source: §C.3 · p. 21, Figure 10 · p. 21
  • Figure 10 separates hard tasks with more than 60 typical steps from easy tasks with fewer than 60 typical steps.The hard tasks are read_book, put_dishwasher_hard, and prepare_food; the easy tasks are put_dishwasher_easy, put_fridge, and setup_table.Source: Figure 10 · p. 21

D Emergent Cooperative Behaviors in an Organization

The section summarizes cooperative behaviors observed by examining messages between agents, with Table 3 presenting typical behaviors.

D Emergent Cooperative Behaviors in an Organization

E Ineffective Communication

The appendix describes ineffective communication, leader election, and a human-led collaboration example with two GPT-4 agents.

E Ineffective Communication

F.1 Examples of Election

  • Agents preferred voting for others five times more than nominating themselves and reached consensus soon, though some election choices reflected hallucinations.The elected leader could nominate others for the next round; Figure 11 illustrates the voting and negotiation process.Source: §F.1 · p. 24, Figure 11 · p. 24

F.2 Examples of Human-AI Collaboration

  • The authors conducted an example experiment with one human leader and two GPT-4 agents to illustrate human-AI collaboration.Figure 12 depicts the collaboration.Source: §F.2 · p. 24, Figure 12 · p. 24

F.3 Examples of Correction

The examples show agent corrections of leader instructions, compare GPT-4 and GPT-3.5-turbo leadership, and illustrate a nine-agent pyramid.

F.3 Examples of Correction

  • A prompt inviting agents to correct the leader yields corrections of unnecessary orders and clarifications of what an agent is holding.The examples also show an agent sharing its next plan after clarifying the leader’s mistaken recollection.Source: §F.3 · p. 25, Figure 13 · p. 25

F.4 Examples of Leadership Comparison

  • GPT-4’s leadership instructions are more specific, clear, and holistic than GPT-3.5-turbo’s.Figure 14 compares the two models’ leadership.Source: Figure 14 · p. 25, §F.4 · p. 25

F.5 Examples of Scaling Up

  • A team of nine agents forms a pyramid structure, with one primary leader and two designated vice leaders.Figure 15 depicts the prompt’s leadership assignments.Source: §F.5 · p. 26, Figure 15 · p. 26

F.6 Examples of Failure Cases

The examples include agent failure cases, behavior labels for dialogues, prompts generated with and without a critic, and broader-impact considerations.

F.6 Examples of Failure Cases

  • Failure examples include repeating a leader’s order instead of completing a subtask and hallucinating that another wine is unnecessary.Figure 16 describes both cases for GPT-3.5-turbo agents.Source: Figure 16 · p. 26, §F.6 · p. 26

G Examples of Cooperative Behaviors Classification by Humans and GPT-4

H Examples of New Prompts after Reflection

I Broader Impacts

  • The authors identify potential applications in robotics, virtual assistants, and disaster response, where coordination is crucial.These are presented as potential applications of more effective, autonomous multi-agent systems.Source: I Broader Impacts · p. 28
  • For high-stakes human-LLM cooperation, the authors say mandatory termination measures such as human approval are still needed.They note that current ability to bound and evaluate LLM behavior remains immature, so natural-language instructions alone are not enough.Source: I Broader Impacts · p. 28
Loading 2403.12482v2…