Source-linked AI summary
Building Cooperative Embodied Agents Modularly with Large Language Models
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan
TL;DR
The paper studies cooperative embodied agents operating with decentralized control, partial observations, costly communication, and long-horizon multi-objective tasks. It introduces CoELA, a cognitive-inspired modular architecture using LLMs for perception-linked memory, communication, planning, and execution, and reports that GPT-4-driven CoELA outperforms strong planning-based methods while showing effective communication.
Problem
Cooperative embodied agents must collaborate under decentralized control, partial observations, costly communication, and long-horizon multi-objective tasks.
Method
CoELA integrates LLM capabilities into a cognitive-inspired modular framework with perception, memory, communication, planning, and execution modules.
Results
GPT-4-driven CoELA surpasses strong planning-based methods with more than 40% efficiency improvements and emergent efficient communication across C-WAH and TDW-MAT.
Takeaways & Limitations
The experiments and user study indicate that LLM-based embodied agents can cooperate effectively with agents and humans through natural-language communication.
Takeaways & Limitations
CoELA still lacks effective reasoning about low-level actions because high-level planning abstracts away their execution.
Abstract
from arXiv · showhide
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. While previous research either presupposes a cost-free communication channel or relies on a centralized controller with shared observations, we harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework that integrates with perception, memory, and execution. Thus building a Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish long-horizon tasks efficiently. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune a CoELA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoELA communicating in natural language can earn more trust and cooperate more effectively with humans. Our research underscores the potential of LLMs for future research in multi-agent cooperation. Videos can be found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.
1 INTRODUCTION
The paper frames decentralized embodied cooperation as a difficult problem involving partial observations, costly communication, and long-horizon multi-objective tasks, then introduces CoELA as a modular LLM-based solution. Across C-WAH and TDW-MAT, GPT-4-driven CoELA outperforms strong planning-based baselines and supports effective cooperation.
- Cooperative embodied agents must handle complex perception, partial observation, long-horizon planning, and natural-language communication.
- The target setting requires decentralized agents to cooperate efficiently despite raw observations, costly communication, and multiple long-horizon objectives.
- CoELA integrates five cognitive-inspired modules for perception, memory, communication, planning, and execution.The modules extract information, maintain knowledge of the world and other agents, generate communication, form high-level plans, and produce primitive actions.
- CoELA is evaluated in the C-WAH and TDW-MAT embodied environments.
- More than 40% efficiency improvements over strong planning-based baselines are reported for GPT-4-driven CoELA, alongside emergent efficient communication.
- The paper also reports a user study examining effective and trustworthy human–AI cooperation with LLM-based agents.
2 RELATED WORK
Related work spans multi-agent cooperation, communication efficiency, visually rich and embodied domains, and LLM-based language agents for sequential decision-making. These lines of research establish relevant capabilities while leaving cooperative embodied interaction under decentralized, costly-communication conditions insufficiently explored.
- Multi-agent research has developed platforms for cooperation and communication across diverse tasks.
- Other studies target communication efficiency, visually rich cooperation, and grounding communication in environments.
- The surveyed directions motivate studying LLM-driven cooperation with other agents or humans under decentralized settings with costly communication.
- Language-agent research uses LLMs for sequential decision-making and demonstrates planning capabilities, including in embodied environments.
3 COOPERATIVE PLANNING UNDER DEC-POMDP-COM
The paper formulates cooperative embodied interaction as an extension of DEC-POMDPs with communication actions, partial observations, task sub-goals, action costs, discounting, and a planning horizon. It instantiates this formulation with two decentralized agents solving long-horizon rearrangement tasks in indoor multiroom environments.
- The setting extends DEC-POMDPs with agents, states, actions, observations, transitions, goals, rewards, discounting, and a planning horizon.
- Each agent’s action set includes world actions and communication actions, while observations include sensor-derived world observations and received messages.
- The transition model specifies state changes after joint actions, and the reward combines action costs with progress across multiple sub-goals.
- The formulation assumes noise-free broadcast communication and focuses on two agents, while stating that the methods and experiments generalize beyond two agents.
- The instantiated environment has two decentralized embodied agents completing a long-horizon indoor multiroom rearrangement task.
- CoELA’s framework is organized around five modules: Perception, Memory, Communication, Planning, and Execution.
4 BUILDING COOPERATIVE EMBODIED AGENTS MODULARLY WITH LLMS
CoELA is a modular cognitive architecture that processes raw observations, updates multiple forms of memory, communicates selectively, plans collaboratively, and executes actions. Its memory design separates world knowledge, past experience, and procedures for carrying out plans.
- At each interaction step, CoELA perceives raw sensory observations and updates its memory with extracted information.
- The architecture contains Perception, Memory, Communication, Planning, and Execution modules built around LLM reasoning and language generation.
- Memory Module: Semantic memory stores the semantic map, task progress, and the states of the agent and others.The paper cautions that this world knowledge can become inaccurate when other agents change object states without awareness.
- Memory Module: Episodic memory records action and dialogue histories as actions are executed and messages are sent or received.
- Memory Module: Procedural memory contains environment-specific knowledge for carrying out high-level plans, including code and neural-model parameters.
- Communication Module: The Communication Module uses LLM free-form language generation to produce task-conditioned messages rather than inefficient casual conversation.
5 EXPERIMENTS
CoELA is evaluated in decentralized embodied environments with costly communication, where it demonstrates stronger cooperation, efficient communication, and useful modular components, while retaining important reasoning and grounding limitations.
- Experimental settings: CoELA is evaluated on TDW-MAT and C-WAH, two embodied multi-agent environments supporting cooperation and communication.TDW-MAT uses transport tasks with RGB-D observations and low-level navigation, interaction, and communication actions; C-WAH covers household activities under symbolic and visual observations.
- Collaborating with AI agents: CoELA improves cooperation with baseline agents, reaching 0.69 TR and 36% EI versus 0.61 TR and 29% EI for RHP with another RHP.On C-WAH, cooperating with CoELA achieves 45% efficiency improvement versus 33% when cooperating with the same MHP.
- Collaborating with AI agents: Two CoELA agents further boost TR to 0.71 and 0.85 on TDW-MAT without and with Oracle Perception, while fine-tuned CoLLAMA reaches 0.70 TR.Replacing GPT-4 with LLAMA-2 causes a significant performance drop, whereas fine-tuning improves performance and surpasses GPT-4 on the Stuff subtask.
- Cooperative behavior: CoELA exhibits information sharing, help requests and responses, adaptive planning, and decisions about when not to communicate.These behaviors are presented as evidence of effective communication and cooperation, although communication still fails in some cases.
- Collaborating with humans: Natural-language communication improves human-agent collaboration: trust scores are 6.3 for CoELA versus 4.7 without communication, with p=0.0003.Humans also prefer collaborating with CoELA over MHP’s template-language communication and cooperate more efficiently with it.
- Ablations and limitations: Removing memory nearly doubles task steps, while replacing GPT-4 with GPT-3.5 causes more reasoning errors, implausible plans, and unuseful messages.Removing the Execution Module slows inference substantially, and trials perform poorly and struggle to finish tasks.
- Ablations and limitations: CoELA lacks 3D spatial information and effective low-level action reasoning, while complex reasoning mistakes can still cause planning failures.These limitations can yield time-consuming exploration plans, plausible but ineffective decisions, and failures from incorrect goal counting or misunderstood rules.
6 CONCLUSION
The paper presents CoELA, a modular LLM-based framework for cooperative embodied agents, and evaluates its capabilities, limitations, and future potential for multi-agent cooperation.
- CoELA integrates LLMs into a modular framework so agents can plan, communicate, and collaborate in decentralized, partially observed, costly-communication settings.The framework targets multi-objective, long-horizon embodied tasks involving other agents and humans.
- Experiments in two embodied multi-agent environments show the effectiveness of the proposed framework and several cooperative behaviors.The evaluated environments are C-WAH and TDW-MAT.
- A CoLLAMA fine-tuned from LLAMA-2 with agent-collected data achieves promising performance for cooperative embodied agents.
- Natural-language communication enables CoELA to cooperate better with humans and earn more trust from them.
- The paper analyzes current LLM limitations and identifies potential solutions for improving embodied cooperative agents.
A.1 PERCEPTION MODULE
The framework converts raw visual observations into structured spatial and object information, stores world and interaction history in memory, and uses LLMs for communication, planning, and execution.
- Perception Module: CoELA’s Perception Module extracts useful information from raw sensory observations for downstream higher-order reasoning.
- Perception Module: 512 × 512 RGB and depth images are segmented, projected into 3D coordinates, and accumulated into a top-down semantic map.The map records semantic classes, occupied area, and explored area on 0.125m × 0.125m grid cells.
- Perception Module: The fine-tuned Mask-RCNN achieves 81.4% mAP@50 on the test set.It was fine-tuned using 53K randomly sampled 512×512 RGB images from training environments.
- Memory Module: Semantic, episodic, and procedural memory store knowledge and experience about the world, other agents, actions, dialogues, and environment-specific plans.
- Communication Module: The Communication Module generates accurate, helpful, brief messages from task instructions, goals, states, action history, and dialogue history.The prompt also discourages repetitive messages and uses seed dialogues to elicit effective communication behavior.
- Planning Module: The Planning Module retrieves memory, constructs an executable high-level Action List, and uses LLM reasoning to select an action.Zero-shot chain-of-thought prompting encourages additional reasoning before answer extraction.
- Execution Module: The Execution Module converts high-level plans into environment-specific primitive actions, reducing planning burden, inference time, and cost.This separation lets the Planning Module focus on overall task solving and remain more generalizable across environments.
- Working Example: A TDW-MAT working example shows the pipeline from RGB-depth perception through state extraction, memory updates, planning, communication, and action execution.
B.1 THREEDWORLD MULTI-AGENT TRANSPORT
TDW-MAT extends a transport challenge with multi-agent cooperation, natural-language communication, richer objects and placements, and realistic embodied environments.
- Environment: TDW-MAT extends the ThreeDWorld Transport Challenge with more object types, more realistic placements, and communication between agents.It is built on the TDW virtual-world simulation platform and uses replicant humanoid embodiments.
- Tasks: TDW-MAT includes food-transporting and stuff-transporting tasks across four floorplans with three layouts each.Two floorplans are used for training and two for testing.
- Tasks: Agents transport as many target objects as possible to a goal position within 3000 frames, using containers as tools.A container carries at most three objects, while an agent without containers carries two objects at a time.
- Observation Space: The main observation consists of egocentric RGB and depth images, supplemented by auxiliary observations such as oracle perception, pose, and messages.Figure 9 illustrates RGB, depth, and oracle perception outputs.
- Action Space: The action space contains navigation, object interaction, and communication actions, each requiring several frames.Listed actions include movement, turning, grasping, putting objects in containers, dropping objects, and sending messages.
B.2 COMMUNICATIVE WATCH-AND-HELP
C-WAH extends Watch-And-Help with communication and evaluates cooperative household activities under symbolic and visual observation settings.
- Environment: C-WAH enables agents to send messages alongside other actions, with each message-taking action consuming one timestep and messages limited in length.
- Tasks: The benchmark contains five household tasks: preparing afternoon tea, washing dishes, preparing a meal, putting groceries away, and setting a dinner table.Each task includes several predicate-based subgoals.
- Tasks: Agents must satisfy all subgoals within 250 time steps, with each task containing 3 to 5 subgoals.
- Observation Space: C-WAH supports symbolic observations that expose object information in the agent’s room, including location, status, name, and relationships.
- Observation Space: Visual observations provide egocentric RGB and depth images plus auxiliary information such as oracle perception and agent position.The RGB images have screen size 256 × 512 and a 60-degree field of view.
- Action Space: The action space adds message sending to the original Watch-And-Help actions, including navigation, turning, grasping, opening, closing, and putting.Messages can contain no more than 500 characters.
C ADDITIONAL DETAILS ON EXPERIMENTS
The experiments compare CoELA with transformer- and hierarchy-based baselines, while also describing data collection and fine-tuning for an open language model.
- Multi-Agent-Transformer: MAT uses a centralized decision transformer to generate actions from oracle semantic maps and agent information.Its action set includes exploration, navigation to target objects or containers, and navigation to goal places; actions last up to 50 frames or until completion.
- RL training: The RL agents were trained for 2e5 frames and then evaluated on test sets.Training used hidden-layer dimension 64, learning rate 7e −4, and 10 PPO epochs.
- Rule-based Hierarchical Planner: RHP combines rule-based high-level planning with A-star navigation using occupancy and semantic maps from visual observations.
- MCTS-based Hierarchical Planner: MHP combines an MCTS-based high-level planner with regression planning and adapts subgoals based on the other agent’s observed intention.
- Open-model fine-tuning: CoELA training data comprised 2k GPT-4-driven trajectories, manually filtered to 572 high-quality examples for LoRA fine-tuning of LLAMA-2-13b-chat.Fine-tuning used a batch size of 384, maximum sequence length 2048, maximum learning rate 4e−4, and 30 epochs.
C.4 ADDITIONAL QUALITATIVE ANALYSIS OF THE AGENT BEHAVIORS
Qualitative analyses show CoELA coordinating through progress and intent sharing, help requests, adaptive labor division, and selective silence during cooperation.
- Communication: CoELA shares progress and intentions with partners through generated communication.This behavior demonstrates the Communication Module’s ability to decide what information to send while using language generation.
- Help coordination: CoELA requests help when needed and responds to partners’ requests by changing its actions.For example, one agent reports a target object when its container is full, and the other retrieves the objects.
- Human interaction: A Human–CoELA example shows natural-language coordination producing a complete division of the exploration trajectory.
- Plan adaptation: CoELA adapts plans after incorporating information about another agent’s circumstances.In the example, Bob changes his labor-division plan after Alice explains that she is already in the kitchen.
- Communication efficiency: CoELA sometimes avoids communicating when acting alone is more efficient for completing the final goal object.The same behavior was also observed in humans performing the task.
C.5 ADDITIONAL DETAILS ON THE HUMAN EXPERIMENTS
The human-experiment materials include an effective Human–CoELA interaction example, communication templates, prompt examples, and discussion of cooperation behavior.
- Human–CoELA interaction: A human shares progress and proposes labor division, while CoELA responds with its future plan, producing a complete exploration division.
- Discussion: CoELA’s cooperation tendency can improve coordination but may reduce efficiency when the other cooperator is malicious.
- Planning prompts: The planning materials provide example prompts for C-WAH and TDW-MAT reasoning modules.The cited tables are identified as example prompts for the two environments.
- Progress sharing: Template communication supports progress sharing so agents update task completion and avoid repeating satisfied subgoals.Messages are sent after achieving a new placement subgoal and parsed by recipients to update progress tracking.
- Intent sharing: Intent sharing communicates changed subgoals so agents can coordinate plans and avoid selecting the same subgoal.The receiving agent extracts the sender’s new subgoal and updates its belief about the sender’s intent.
- Belief sharing: Belief sharing communicates newly observed scenes, objects, and containers so agents can update object-location beliefs.Received messages are processed as information about object-location distributions, and multiple communication types may be combined into one message for efficiency.