Building Cooperative Embodied Agents Modularly with Large Language Models

• CoELA combines perception, memory, communication, planning, and execution modules to extract observations, track the world and agents, choose messages and timing, plan, and generate primitive actions.Source: §1 · p. 2
• GPT-4-driven CoELA outperformed strong planning-based baselines by more than 40% efficiency improvements and exhibited emergent efficient communication.Source: §1 · p. 2
• LLAMA-2 underperformed, but LoRA fine-tuning on limited agent-collected data yielded promising CoLLAMA performance, while human users trusted natural-language CoELA more.Source: §1 · p. 2
• The authors formalized decentralized household cooperation with partial observation, costly communication, and long-horizon multi-objective tasks, instantiated in C-WAH and TDW-MAT.Source: §1 · p. 2, Figure 1 · p. 2

Multi-Agent Cooperation and Communication

• Prior platforms span multi-agent tasks and efficient or grounded communication, but may omit communication or use continuous vectors or discrete symbols rather than natural language.This work instead studies agents communicating efficiently without a presupposed free channel, including communication with humans.Source: §2 · pp. 2–3

Language Agents

• LLM language-agent research includes direct plan generation, environment-feedback monologues, and grounded instruction following (Liang et al., 2022; Huang et al., 2022b; Ahn et al., 2022).The broader literature explores LLMs for sequential decision-making and planning, including in embodied environments.Source: §2 · p. 3
• Unlike multi-LLM self-talk or debate (Li et al., 2023b; Du et al., 2023; Wang et al., 2023c), CoELA requires decentralized agents to plan costly communication for complex, long-horizon, multi-objective tasks.The passage also contrasts this setting with Park et al. (2023), whose LLM society uses memories to simulate human behavior.Source: §2 · p. 3
• The work evaluates open LMs such as LLAMA-2 and fine-tunes CoLLAMA with LoRA on agent-collected embodied-environment data to demonstrate promising performance.Source: §2 · p. 3

3 COOPERATIVE PLANNING UNDER DEC-POMDP-C

• The DEC-POMDP extension includes communication actions and teammate messages, joint state transitions, multiple sub-goals, and a team reward that accounts for action costs and goal completion.The formalization also includes a discount rate and planning horizon.Source: §3 · p. 3
• The discussion focuses on two agents and noise-free broadcast communication, while the methods and experiments are described as generalizable to more than two agents.Source: §3 · p. 3
• The task instantiation has two decentralized embodied agents, including possible human participants, cooperating on long-horizon indoor rearrangement through navigation, interaction, and natural-language communication actions.Sub-goals can specify counts, such as placing two plates on a dinner table.Source: §3 · pp. 3–4

4.1 FRAMEWORK OVERVIEW

• CoELA combines five modules—perception, memory, communication, planning, and execution—in a cognitive-inspired framework for embodied agents.At each step, it processes raw observations, updates memory, generates a candidate message, selects a plan, records it in episodic memory, and executes it through stored procedures.Source: §4.1 · p. 4

4.2 PERCEPTION MODULE

• The Perception Module segments RGB observations, builds 3D point clouds from RGB-D input, extracts key-object states, and constructs a local semantic map.The paper describes training Mask-RCNN to predict segmentation masks from RGB images.Source: §4.1 · p. 4

4.3 MEMORY MODULE

• CoELA separates memory into semantic, episodic, and procedural stores for knowledge about the world, past experience, and how to carry out plans.Semantic memory tracks the map, task progress, and agent states but may be inaccurate when others change objects outside the agent’s awareness; episodic memory records action and dialogue history, while procedural memory contains environment-specific plan implementations and neural-model parameters.Source: §4.1 · p. 5

4.4 COMMUNICATION MODULE

• The Communication Module conditions an LLM message generator on retrieved task, map, agent-state, action-history, and dialogue-history information to avoid inefficient casual chatting.It converts retrieved information into templated text and adds a prompt note and two seed messages to encourage effective communication.Source: §4.1 · p. 5

4.5 PLANNING MODULE

• The Planning Module prompts an LLM to choose an executable high-level plan from an available Action List, using retrieved information and zero-shot chain-of-thought prompting.The formalized list is intended to focus LLM reasoning and support plan generation without few-shot demonstrations.Source: §4.1 · p. 5

4.6 EXECUTION MODULE

• The Execution Module translates high-level plans into environment-specific primitive actions, addressing LLM weakness in low-level control while reducing inference time and cost.Stored procedures let the LLM planner focus on overall task-solving and generalizable cooperation decisions.Source: §4.1 · p. 5

5.1 EXPERIMENTAL SETUP

• CoELA experiments tested TDW-MAT transport tasks with ego-centric RGB-D observations and C-WAH household tasks under symbolic and visual observations.TDW-MAT used 24 episodes across six scenes, with a 3000-frame horizon; C-WAH used 10 episodes across five activity types, with a 250-step horizon. Both involved decentralized agents with communication.Source: §5.1 · pp. 5–6

5.2 BASELINES

• The baselines included hierarchical planners for C-WAH and TDW-MAT, plus MAT, which used centralized decisions from shared observations.CoELA used GPT-4, with Mask-RCNN perception; the experiments also tested LLAMA-2 and LoRA-fine-tuned CoLLAMA. MAT received oracle semantic maps and agent states in this setting.Source: §5.2 · p. 6

5.3.1 COLLABORATING WITH AI AGENTS

• Two cooperating CoELAs reached 0.71 and 0.85 TR on TDW-MAT without and with Oracle Perception, while fine-tuned CoLLAMA reached 0.70 TR and surpassed GPT-4 on Stuff.Replacing GPT-4 with LLAMA-2 caused a significant performance drop; the authors describe fine-tuned CoLLAMA as competitive with GPT-4.Source: §5.3.1 · pp. 6–7, Table 1 · p. 7
• 0.69(36%) vs. 0.61(29%) Transport Rate (Efficiency Improvement) on TDW-MAT and 45% vs. 33% Efficiency Improvement on C-WAH favor cooperation with CoELA over the respective same-baseline pairings.The paper reports higher transport rate and efficiency improvement when RHP cooperates with CoELA rather than another RHP, and a 45% versus 33% performance boost on C-WAH when cooperating with CoELA rather than MHP.Source: §5.3.1 · p. 6, Table 2 · p. 6, Table 1 · p. 7

5.3.2 COLLABORATING WITH HUMANS

• 6.3 vs. 4.7 trust scores favored CoELA with natural-language communication over CoELA without communication (p=0.0003), and humans preferred it to MHP’s template language.CoELA also performed better than MHP when collaborating with humans; without communication, CoELA’s performance dropped. The study included 8 subjects and 80 trials across four scenarios.Source: §5.3.2 · pp. 7–8, Figure 4 · p. 8

5.4 ANALYSIS

• Replacing GPT-4 with GPT-3.5 increased task-completion steps, with more state-reasoning errors, implausible plans, and unhelpful messages.The authors attribute the gap to GPT-4’s more advanced reasoning and Theory of Mind abilities.Source: §5.4 · p. 8
• CoELA displayed information sharing, help requests and responses, plan adaptation, and restraint in communicating, but disabling AI-agent communication caused no significant performance drop.Communication is time-consuming and requires accurate models of other agents and robust understanding of ambiguous language; the authors report that current LLMs do not yet master this reliably. Natural-language communication also helped human-agent cooperation.Source: §5.4 · p. 8
• Removing memory nearly doubled completion steps, while removing the Execution Module slowed inference and left trials struggling to finish tasks.The Memory Module stores and updates knowledge and experience about the scene and other agents; removing the Execution Module left the Planning Module to make low-level controls directly at every step.Source: §5.4 · pp. 8–9, Figure 4 · p. 8

6 CONCLUSION

• CoELA combines LLMs with a modular framework to plan, communicate, and collaborate with agents and humans in challenging multi-agent settings.The settings involve decentralized control, complex partial observation, costly communication, and multi-objective long-horizon tasks.Source: §6 · p. 9
• Experiments in two extended embodied multi-agent environments show the framework's effectiveness and several cooperative behaviors.Source: §6 · p. 9
• The authors fine-tune CoLLAMA from LLAMA-2 using data collected by their agents in embodied environments, demonstrating promising performance for building better cooperative embodied agents.Source: §6 · p. 9
• CoELA's natural-language communication can help it cooperate better with humans and earn more trust from them.Source: §6 · p. 9
• The work analyzes limitations of current LLMs and highlights potential solutions for building better embodied cooperative agents.Source: §6 · p. 9
• The authors identify designing stronger LLM-based embodied agents for multi-agent cooperation as a promising future direction.Source: §6 · p. 9

A.1 PERCEPTION MODULE

• Fine-tuned Mask-RCNN reaches 81.4% mAP@50 on the test set; TDW-MAT RGB/depth observations are converted into a voxel-derived top-down semantic map for memory.The map uses 0.125m × 0.125m cells and represents semantic classes, occupied area, and explored area.Source: §A.1 · p. 16

A.2 MEMORY MODULE

• Semantic, episodic, and procedural memories store world/task/agent state, action and dialogue histories, and environment plans/model parameters, respectively.Semantic state may be inaccurate when others change objects unseen; episodic storage retains only the last K actions and D dialogues, while procedural memory is not updated except through model fine-tuning.Source: §A.2 · p. 17

A.3 COMMUNICATION MODULE

• The Communication Module prompts an LLM with instructions, goals, state descriptions, action history, and dialogue history to generate messages constrained to be accurate, helpful, brief, and nonrepetitive.The prompt uses designed components and seed messages to guide effective communication behavior.Source: §A.3 · p. 17

A.4 PLANNING MODULE

• The Planning Module prompts an LLM with retrieved state descriptions and an Action List of available high-level plans, using zero-shot chain-of-thought to support its choice.The multi-choice Action List is intended to make executable plans easier to select without few-shot demonstrations.Source: §A.4 · p. 18

A.5 EXECUTION MODULE

• The Execution Module converts high-level plans into robust primitive actions, using an A-star planner for shortest-path navigation and object interaction in rearrangement tasks.For a new environment with a different action space, the procedural knowledge needs to be rewritten.Source: §A.5 · p. 18

A.6 A WORKING EXAMPLE ON TDW-MAT

• A TDW-MAT example shows the modules’ end-to-end flow from RGB/depth perception and semantic memory updates through message generation and LLM planning to A-star navigation and action.The example’s selected plan is to explore the current room, followed by the primitive action “Move forward 0.5m”.Source: §A.6 · pp. 18, 20, Figure 7 · p. 19

• TDW-MAT extends the ThreeDWorld Transport Challenge with natural-language cooperation, more object types and realistic placements, and the TDW platform’s latest replicant humanoid embodiment.Source: §B.1 · p. 20
• The food- and stuff-transport tasks each use 10 targets and 2–5 containers across four floorplans with three layouts each, split between training and test.Each task has six target types and three container types; object placement reflects common-sense room contexts.Source: §B.1 · p. 20, Figure 8 · p. 20
• Agents must move as many targets as possible to a goal within 3000 frames; containers carry up to three objects, versus two without a container.Source: §B.1 · p. 20
• Agents observe egocentric RGB and depth images plus optional oracle perception, pose, and team messages, and can move, turn, grasp, put in, drop, or message.RGB and depth images are 512 × 512 with a 90-degree field of view; messages are limited to 500 characters per frame.Source: §B.1 · p. 21, Figure 9 · p. 21

B.2 COMMUNICATIVE WATCH-AND-HELP

• C-WAH adds agent-to-agent messaging to Watch-And-Help, with each message taking one timestep and a limit of 500 characters.Sending a message consumes a timestep like other actions.Source: §B.2 · pp. 21–22
• The benchmark includes five household tasks with 3–5 predicate-defined subgoals each, all of which must be satisfied within 250 timesteps.Tasks range from preparing afternoon tea and washing dishes to putting away groceries and setting up a dinner table.Source: §B.2 · p. 21
• Agents receive either symbolic room-level object information or visual observations including egocentric RGB and depth images plus auxiliary observations.Symbolic observations include object location, status, name, and relationships; visual observations also include oracle perception, agent position, and messages.Source: §B.2 · p. 22
• The action space retains Watch-And-Help movement and object-manipulation actions and adds an action for sending messages to other agents.The listed actions include walking, turning, grasping, opening, closing, and putting objects.Source: §B.2 · p. 22

C.1 TRAINING DETAILS ON THE MULTI-AGENT TRANSFORMERS

• MAT treats multi-agent reinforcement learning as sequence modeling and uses a centralized decision transformer to generate actions.Source: §C.1 · p. 23
• MAT receives a 12 × 24 nine-channel oracle semantic map and agent information, then chooses among four high-level actions that last up to 50 frames or until completion.The actions are explore, navigate to the nearest target object or container, and navigate to the goal place.Source: §C.1 · p. 23

C.2 ADDITIONAL DETAILS ON OTHER BASELINES

• The Rule-based Hierarchical Planner selects exploration, object pickup, container pickup, or placement plans using human-defined rules, with A-star navigation and random frontier waypoints.Source: §C.2 · p. 23
• The MCTS-based Hierarchical Planner combines an MCTS high-level planner with regression planning and adapts its subgoal by inferring the other agent’s intention from observation.Source: §C.2 · p. 23

C.3 ADDITIONAL DETAILS ON CoLLAMA

• CoLLAMA was fine-tuned on 572 manually filtered high-quality trajectories selected from 2,000 GPT-4-driven CoELA trajectories across 10 TDW-MAT training episodes.The selected data featured effective communication and reasoning traces for collaborative decision-making.Source: §C.3 · p. 23
• CoLLAMA uses LoRA to fine-tune LLAMA-2-13b-chat for 30 epochs (approximately 60 steps), with batch size 384, maximum sequence length 2048, and maximum learning rate 4e−4.Source: §C.3 · p. 23

• CoELA shares progress and intentions with teammates, with its Communication Module using LLM dialogue generation to decide what information to send.The examples show communication about progress and plans across several situations.Source: §C.4 · p. 23
• CoELA recognizes when it needs help and responds to others’ requests, including by asking a teammate to collect objects when its own container is full.It also answers teammates’ requests and questions.Source: §C.4 · p. 23
• CoELA adapts its plan after a teammate shares relevant circumstances, revising a proposed division of labor when the teammate is already in the kitchen.The teammate’s location was information Bob had not previously known.Source: §C.4 · p. 24
• CoELA sometimes withholds communication when acting alone is more efficient, such as collecting the last goal object; the same behavior was observed in humans.Source: §C.4 · p. 24

C.5 ADDITIONAL DETAILS ON THE HUMAN EXPERIMENTS

• In an example, CoELA responds to a human’s progress update and proposed labor division with its own plan, producing a perfect division of the exploration trajectory.The authors present this as evidence of CoELA’s potential to work cooperatively with humans.Source: §C.5 · p. 24

D ADDITIONAL DISCUSSIONS

• CoELA coordinated without arguing back and forth in the experiments, but this cooperative tendency may reduce efficiency when a cooperator is malicious.The authors suggest that instruction-following and trust in cooperators may contribute to this behavior.Source: D ADDITIONAL DISCUSSIONS · p. 24
• The work targets multi-agent cooperation with decentralized control, complex observations, costly communication, and long-horizon, multi-objective tasks.The authors contrast this setting with prior work on LLM-based embodied planning.Source: D ADDITIONAL DISCUSSIONS · p. 24

E EXAMPLE PROMPTS

• Example prompts for the planning module on C-WAH and TDW-MAT are provided in Tables 4 and 6, respectively.The table captions label these as examples for the Reasoning Module.Source: E EXAMPLE PROMPTS · p. 25, Table 4 · p. 25, Table 6 · p. 27

F TEMPLATE LANGUAGE COMMUNICATION FOR MCTS-BASED HP AGENT

• Progress-sharing messages report completed subgoals, which recipients extract to update task progress and avoid repeating already-satisfied subgoals.The planner sends this message when an agent performs a put action, indicating that a new subgoal has been achieved.Source: F TEMPLATE LANGUAGE COMMUNICATION FOR… · p. 28
• Intent-sharing messages report an agent’s current subgoal when its plan changes, helping others avoid selecting the same subgoal.Recipients extract the other agent’s new subgoal and update their beliefs about its intent.Source: F TEMPLATE LANGUAGE COMMUNICATION FOR… · p. 28
• Belief-sharing messages convey newly found goal objects and container findings so recipients can update object-location beliefs; agents can combine the three message types into one message to improve efficiency.The example messages are sent when an agent enters a new room.Source: F TEMPLATE LANGUAGE COMMUNICATION FOR… · p. 29