PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication

Huao Li, Hossein Nourkhiz Mahjoub, Behdad Chalaki, Vaishnav Tadiparthi, Kwonjoon Lee, Ehsan Moradi-Pari, Charles Michael Lewis, Katia P Sycara

arXiv:2409.17348v2cs.MA

TL;DR

MARL agents can learn shared communication protocols, but those protocols are often not interpretable to humans or teammates they were not trained with. LangGround aligns MARL communication with natural-language embeddings using synthetic data from embodied LLM teamwork. The paper reports maintained task performance, faster emergence of communication, and zero-shot generalization in ad-hoc teamwork.

  • Problem

    MARL protocols learned from task rewards are often not interpretable to humans or agents outside the co-trained team, limiting their use in ad-hoc teamwork.

    Source: §1 · p. 1

  • Method

    LangGround aligns MARL communication with natural-language embeddings using synthetic communication data generated by embodied LLM agents in interactive teamwork.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Language grounding maintains task performance and accelerates communication emergence, while learned protocols show zero-shot generalization to unseen teammates and novel task states.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The aligned communication space supports translating between high-dimensional embeddings and natural-language sentences for ad-hoc teamwork.

    Source: §1 · p. 2

    The evaluation of more complicated task environments at scale and of different MARL algorithms, backbone LLMs, and word embeddings remains future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Multi-Agent Reinforcement Learning (MARL) methods have shown promise in enabling agents to learn a shared communication protocol from scratch and accomplish challenging team tasks. However, the learned language is usually not interpretable to humans or other agents not co-trained together, limiting its applicability in ad-hoc teamwork scenarios. In this work, we propose a novel computational pipeline that aligns the communication space between MARL agents with an embedding space of human natural language by grounding agent communications on synthetic data generated by embodied Large Language Models (LLMs) in interactive teamwork scenarios. Our results demonstrate that introducing language grounding not only maintains task performance but also accelerates the emergence of communication. Furthermore, the learned communication protocols exhibit zero-shot generalization capabilities in ad-hoc teamwork scenarios with unseen teammates and novel task states. This work presents a significant step toward enabling effective communication and collaboration between artificial agents and humans in real-world teamwork settings.

1 Introduction

MARL agents can learn shared communication protocols, but these protocols are often difficult for humans or unseen teammates to interpret. LangGround uses embodied LLM-generated teamwork data to align MARL communication with natural language.

1 Introduction

  • Learned MARL protocols can be hard for humans and agents outside the training team to interpret, while task-focused optimization may neglect language complexity and informativeness.Prior work also leaves relations between communication symbols unexplored, with only a few approaches targeting semantic spaces for zero-shot communication of unseen states.Source: §1 · p. 2
  • LLMs can generate teamwork descriptions grounded in natural language, but their weak grounding in task environments can prevent actionable plans.Prior LLM grounding work does not address teamwork and communication among multiple embodied agents.Source: §1 · p. 2
  • LangGround aligns MARL communication with natural-language embeddings using synthetic data from embodied LLM teamwork, with both language and environment learning signals.The paper evaluates ad-hoc teamwork with unseen teammates and novel task states; it presents the pipeline as compatible with different MARL architectures and LLMs.Source: §1 · p. 2

2 Related Work

Prior communication research includes reward-optimized MARL, emergent communication, and language-grounding approaches, but challenges remain in interpretability and interactive ad-hoc teamwork. This work places its contribution against those gaps.

2.1 Multi-Agent Communication

  • Reward-driven communication methods learn protocols end-to-end, but learned messages can overfit to co-trained interlocutors and fail to be understood by unseen teammates or humans.Other approaches stabilize learning through shared parameters or centralized control, or ground communication in individual observations using representation learning.Source: §2.1 · p. 2
  • Emergent communication commonly uses reference or Lewis signaling games, while learning interpretable protocols in more complex, multi-round tasks remains challenging.Even when communication emerges, protocols may be non-human-interpretable or semantically drift from human language.Source: §2.1 · pp. 2–3

2.2 Human-Interpretable Communication

  • Human-interpretable RL communication has used language models, aligned tokens, and semantic prototypes, but alignment remains non-trivial and interactive ad-hoc team evaluation was unreported.The closest cited work alternates imitation and self-play in reference games, rather than training MARL agents for interactive team tasks.Source: §2.2 · p. 3

2.3 Language-Grounded Reinforcement Learning

  • Language-grounded RL has used instructions, expert trajectories, game manuals, and commands, but prior work did not use communication messages to guide information sharing in multi-agent teamwork.Source: §2.3 · p. 3
  • CICERO separates its RL and language-model modules, whereas this work trains individual MARL agents end-to-end with RL and supervised-learning losses.Source: §2.3 · p. 3

3 Preliminaries

The paper formulates cooperative communication as a decentralized partially observable Markov decision process and defines language learning around expert communication data. Training combines team reward with supervision toward a target language.

3 Preliminaries

  • The cooperative setting is a decentralized partially observable Markov decision process in which agents act and communicate from local observations and prior messages to maximize total expected return.The formulation specifies agents, global states, actions, messages, transitions, observations, rewards, and a discount factor.Source: §3 · p. 3
  • The target language is a mapping from observations to communication messages, and its dataset contains observation-action pairs from expert embodied-LLM trajectories.Agents are trained to use that language to collaborate with experts in ad-hoc teamwork and should generalize beyond examples in the dataset.Source: §3 · p. 4
  • The learning objective combines environment reward with supervised data so agents both maximize team performance and use language similar to the dataset.Source: §3 · p. 4

4 Language Grounded Multi-agent Communication

LangGround collects grounded communication from embodied LLM agents and aligns MARL communication with language grounds. Its MARL training combines task-driven communication with supervised imitation of LLM messages.

4 Language Grounded Multi-agent Communication

  • LangGround consists of collecting grounded communication from LLM agents and aligning MARL communication with language grounds.Figure 1 also depicts translating aligned communication vectors into natural-language messages via cosine-similarity matching.Source: §4 · p. 4, Figure 1 · p. 4

4.1 Grounded communication from LLM agents

  • Embodied LLM agents receive English descriptions of local observations and teammates’ previous messages, then produce actions and communication for the task environment.Their expert trajectories form a dataset mapping individual observation-action pairs to natural-language communication messages for MARL supervision.Source: §4.1 · pp. 4–5
  • A parallel text environment gives LLM agents information equivalent to RL agents, while prompts omit explicit coordination or communication strategies.The resulting behavior comes from the backbone LLM and its interaction in the task.Source: §4.1 · pp. 4–5

4.2 Multi-agent Reinforcement Learning with aligned communication

  • MARL agents use independent LSTM controllers and probabilistic gates to decide whether to communicate, then share a mean communication vector to inform action selection.Source: §4.2 · p. 5
  • A supervised loss aligns each MARL communication vector with the word embedding of an LLM reference message selected for the agent’s observation and action.The communication is shaped jointly by the RL policy-loss signal and imitation of messages in dataset D; λ scales the supervised loss.Source: §4.2 · p. 5

5 Experiments

Experiments use Predator Prey and Urban Search & Rescue to represent complementary coordination demands, and compare LangGround with communication baselines under a shared training architecture.

5.1 Environments

  • Predator Prey requires agents to combine partial observations, while Urban Search & Rescue adds heterogeneous teammates and temporally dependent behaviors requiring communication of observations, intentions, and requests.The tasks are selected to represent different collaborative demands.Source: §5.1 · p. 5
  • Figure 2 tracks episode length until task completion across training timestamps; lower episode length is better, and shaded areas show standard errors over three seeds.Source: Figure 2 · p. 6

5.2 Experiment setups

  • LangGround is compared with IC3Net, aeComm, VQ-VIB, protoComm, and noComm, which respectively provide an ungrounded architectural ablation, observation-reconstruction grounding, interpretable-token approaches, and a no-communication control.The passage identifies aeComm as a state-of-the-art communication method and VQ-VIB and protoComm as human-interpretable approaches.Source: §5.2 · p. 6
  • All methods share a centralized-training, decentralized-execution architecture, with shared action-policy and observation-encoder parameters during training.REINFORCE trains the gating function and policy network; communication messages are 256-dimensional continuous vectors.Source: §5.2 · p. 6

6 Results

Across the evaluated tasks, LangGround maintains competitive task performance while producing communication aligned with human language. Its communication also shows semantic structure and zero-shot generalization, including better performance than selected baselines with unseen LLM teammates.

6.1 Task performance

  • LangGround achieves on-par performance with state-of-the-art communication methods, including comparable Predator Prey results and better USAR completion than all baselines in fewer steps.In Predator Prey, it matches IC3Net and aeComm on ppv1 and is comparable to aeComm on ppv0; USAR results also show lower variance than other methods.Source: §6.1 · pp. 6–7

6.2.2 Human interpretability

  • LangGround gains on both cosine similarity and BLEU compared with baselines trained without language alignment.The comparisons use agent messages and reference messages from dataset D across 100 ppv0 evaluation episodes.Source: Table 1 · p. 8, §6.2.2 · p. 8

6.3.1 Zero-shot generalizability

  • Agents complete tasks at four unseen prey locations, and their messages for those locations resemble natural-language sentences generated by LLMs.The agents were trained from scratch without exposure to, or language grounding for, those situations.Source: §6.3.1 · p. 8, Table 3 · p. 9
  • On 10-by-10 Predator Prey, increasing the share of grounded states from 25% to 100% improves both team performance and communication alignment.Alignment is also evaluated on ungrounded states; the reported results show alignment there across grounding levels.Source: Figure 5 · p. 8, Figure 6 · p. 8, Table 2 · p. 8

6.3.2 Ad-hoc teamwork between MARL and LLM agents

  • LangGround teams with unseen LLM teammates outperform noComm and aeComm in at least two of three evaluation scenarios, though homogeneous teams perform better.The authors attribute the ad-hoc advantage over those baselines to information sharing through language aligned with human language.Source: Table 4 · p. 9, §6.3.2 · pp. 9–10

7 Discussion

LangGround aligns MARL communication with human-language embeddings using synthetic messages from embodied LLMs in interactive teamwork. The discussion reports preserved task performance and faster communication emergence, while noting trade-offs and future evaluation needs.

7 Discussion

  • LangGround grounds MARL communication in human-language embeddings using synthetic data from embodied LLMs in interactive teamwork scenarios.The pipeline combines language-grounding and environment-reinforcement signals to shape communication.Source: §7 · p. 10
  • Predator-Prey favors informativeness and better human-language alignment, whereas USAR favors task utility, faster convergence, and less interpretability to unseen teammates.The authors frame these patterns as a utility–informativeness trade-off.Source: §7 · p. 10
  • Language grounding preserves task performance and accelerates communication emergence, which the authors attribute to expert LLM trajectories that also guide action learning.The dataset supports behavior cloning and provides semantic representations of observations.Source: §7 · p. 10
  • LangGround’s interchangeable modules include the word-embedding model, MARL communication model, message source, and translation module; message-source hallucinations do not directly affect task performance.Datasets may also come from rule-based agents or human participants if they communicate effectively in collaborative tasks.Source: §7 · p. 10
  • The authors position the pipeline as an initial step toward natural-language collaboration between artificial agents and humans, and plan larger-scale, more complex evaluations.Future work also proposes replacing static dataset D with online LLM queries to capture beliefs, intents, and requests.Source: §7 · p. 10

A Implementation details

The implementation uses embodied LLM agents, MARL agents, and natural-language interfaces across Predator Prey and USAR. The section specifies agent interaction, training settings, environment configurations, and interface behavior.

A.1 Embodied LLM agents

  • Embodied LLM agents interact through prompts, text observations, and teammate messages; prompts avoid prescribing coordination, and the GPT-4 backbone uses temperature 0.Agents receive their own observations and teammate messages, select actions, and can broadcast messages; the appendix also gives USAR prompt and interaction examples.Source: §A.1 · pp. 15–16

A.2 LangGround and baseline MARL agents

  • Training uses 2,000 epochs for ppv0 and USAR and 500 for ppv1; the supervised-learning weight λ is 1 in Predator Prey and 10 in USAR.The agents use an LSTM action policy with hidden size 256; communication vectors are exchanged once per timestamp.Source: §A.2 · p. 16

B Environment details

  • Predator Prey is a partial-observation gridworld where predators search for stationary prey, while USAR requires specialists to coordinate bomb searches and defusals.The environments respectively test coordinated navigation and teamwork with distinct agent capabilities.Source: Figure 7 · p. 16

B.1 Predator Prey

  • Predators receive partial views of the grid and must communicate observations to navigate together; episodes end when all reach the prey or the step limit is exceeded.The task uses stationary prey and rewards predators for reaching its location.Source: §B.1 · pp. 16–17

B.2 USAR

  • USAR assigns three agents different wire cutters and partial room observations, requiring coordination to inspect and defuse color-sequenced bombs.The action-space size depends on the number of rooms and cutters; successful defusal rewards 10 times the bomb’s number of phases.Source: §B.2 · p. 17

B.3 Environment configurations

  • The configured Predator Prey maps are 5 by 5 with three predators and one prey; USAR has five rooms, five bombs, three colors, and a 100-step limit.Predator Prey has a 20-step limit and vision range 0 or 1; USAR’s maximum mission score is 90.Source: §B.3 · p. 17

B.4 Text game interface

  • The text interface converts game-state features into templated natural-language observations and maps agent responses to actions using keyword matching.Invalid or ambiguous actions trigger rule-based feedback messages.Source: §B.4 · pp. 17–18

C.1 LangGround dataset

The paper collects LLM-generated expert trajectories to create grounding data, then describes how MARL agents are paired with an unseen LLM teammate for ad-hoc evaluation.

C.1 LangGround dataset

  • Dataset D contains GPT-4 embodied-agent trajectories: 50 USAR episodes and 2,550 observation-action/message pairs, plus 1,893 pairs for ppv0 and 2,493 for ppv1.Natural-language messages are embedded into 256-dimensional vectors to align with MARL communication vectors.Source: §C.1 · p. 18

C.2 Ad-hoc teamwork

  • Ad-hoc teams pair two MARL agents with one unseen GPT-4-turbo agent, embedding LLM messages for MARL agents and translating MARL vectors into English by cosine-similarity matching.The agents interact sequentially through a text-wrapped Gym environment.Source: §C.2 · p. 18

D Additional Experiment Details

LangGround achieves the highest reported topographic similarity among the compared baselines, indicating communication patterns more similar to human language by this measure.

D.1 Topographic similarity

  • ρ = 0.67: LangGround has the highest topographic similarity among the baselines, a measure associated with compositionality and generalization.The measure correlates observation-space object distances with communication-space signal distances.Source: §D.1 · p. 18

D.2 Additional ablation study

The additional ablations examine how reinforcement-learning and supervised-learning signals contribute to task performance and communication alignment. Their effects differ: excessive weight on either signal impairs the objective optimized by the other.

D.2 Additional ablation study

  • The ablation aims to attribute RL and SL signals to the agent’s action and communication outputs in the multi-objective optimization.Because the agent is trained end-to-end using both losses and its hidden state as the communication vector, separating action and communication reasoning is difficult; the authors offer indirect evidence of joint contributions.Source: §D.2 · pp. 18–19
  • Removing the gating function harms LangGround more than IC3Net, suggesting both RL and SL signals influence communication content and timing.Figure 8 presents the gating-function and language-grounding ablation.Source: §D.2 · p. 19, Figure 8 · p. 19
  • Too much SL loss prevents task-policy optimization, whereas too much RL loss prevents human-language alignment.The authors relate the two signals to task utility and alignment, respectively.Source: §D.2 · p. 19, Figure 9 · p. 19, Figure 10 · p. 19
Loading 2409.17348v2…