PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning

Seth Karten, Siva Kailas, Huao Li, Katia Sycara

arXiv:2302.14276v1cs.LGcs.AIcs.MA

TL;DR

Observing an expert’s actions may not be enough for agents to coordinate, and heterogeneous partners need ways to align intent through communication. This paper combines compositional messages with an information-bottleneck and contrastive objective; it reports better performance than referential-only methods and develops communication for social-learning settings.

  • Problem

    Expert-action observation may be insufficient for coordination, while heterogeneous agents need communication to model intent and align policies.

    Source: §1 · p. 1

  • Method

    The method combines compositional messaging with an information-bottleneck objective that balances message complexity against task-relevant information conveyed to other agents.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    The compositional, contrastive method outperforms methods grounded only in referential information and produces independently composed concepts in variable-length messages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that structured messages reduce sample complexity, can align heterogeneous agents, and support learning communication while observing an expert’s policy.

    Source: §1 · p. 2

    The objective conditions on input or observation data as a decentralized training objective.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Explicit communication among humans is key to coordinating and learning. Social learning, which uses cues from experts, can greatly benefit from the usage of explicit communication to align heterogeneous policies, reduce sample complexity, and solve partially observable tasks. Emergent communication, a type of explicit communication, studies the creation of an artificial language to encode a high task-utility message directly from data. However, in most cases, emergent communication sends insufficiently compressed messages with little or null information, which also may not be understandable to a third-party listener. This paper proposes an unsupervised method based on the information bottleneck to capture both referential complexity and task-specific utility to adequately explore sparse social communication scenarios in multi-agent reinforcement learning (MARL). We show that our model is able to i) develop a natural-language-inspired lexicon of messages that is independently composed of a set of emergent concepts, which span the observations and intents with minimal bits, ii) develop communication to align the action policies of heterogeneous agents with dissimilar feature models, and iii) learn a communication policy from watching an expert's action policy, which we term `social shadowing'.

1. INTRODUCTION

The paper develops emergent communication for social learning, where agents use messages to align heterogeneous policies and coordinate beyond what expert-action observation alone may provide. Its compositional, contrastive approach aims to produce informative, compact messages while supporting policy alignment and social shadowing.

1. INTRODUCTION

  • Social learning uses observed expert or novice behavior to learn an action policy, but observing expert actions may not suffice for coordination (Jaques et al., 2019; Ndousse et al., 2021).Communication can help agents model others’ intent and align heterogeneous theories of mind.Source: §1 · p. 1
  • Existing emergent communication can produce degenerate protocols with joined concepts or little information, which are difficult to interpret and can degrade performance.The paper frames its approach as a way to prepare a messaging lexicon for social-learning scenarios.Source: §1 · p. 1
  • The paper tests whether structured messages improve learning, align heterogeneous experts’ policies, and let a novice learn communication by observing an expert’s actions.This last setting is termed social shadowing.Source: §1 · p. 2
  • The proposed compositional, contrastive communication outperforms methods grounded only in referential information, while enabling variable-length messages with independently composed concepts.The authors also report reduced sample complexity for the unsupervised objective.Source: §1 · p. 2
  • The authors position structured emergent messages as a way to reduce sample complexity and help novices learn from experts, while communication can align heterogeneous agents.They motivate this setting with agents or frameworks that develop task-specific feature spaces before interacting with partners.Source: §1 · p. 2

2. RELATED WORK

Related work distinguishes signaling by whether information is intentionally communicated and contrasts learned emergent messages with expert-defined symbols. It also motivates task-specific grounding and compositional messages as ways to improve informativeness and express richer observations.

2.1. Multi-Agent Signaling

  • Emergent communication deliberately shares learned information and can scale to complex observations and many agents, whereas symbolic communication relies on expert-defined messages (Grupen et al., 2022; Sokota et al., 2022).The section distinguishes implicit communication and signaling from explicit signaling, which seeks to directly influence other agents’ behavior (Li et al., 2021).Source: §2.1 · p. 2

2.2. Emergent Communication

2.3. Natural Language Inspiration

  • Communication tokens range from high-information continuous vectors to discrete one-hot tokens and clustered prototypes, but these approaches select a single token that must be strung into messages.VQ-VIB uses variational properties to give continuous tokens clustering and greater informativeness (Tucker et al., 2022).Source: §2.3 · p. 3

3. Preliminaries

The paper formulates communication in a decentralized, partially observable multi-agent decision process and organizes each policy into encoding, communication, and decoding stages. Its information-bottleneck objective trades message complexity against task-relevant information conveyed to other agents.

3. Preliminaries

  • The setup is a Dec-POMDP-Comm, with states, joint actions, communications, transitions, observations, rewards, and discounting; communication contributes to observations in the partially observable setting.The observation function maps messages and local state to each agent’s observation distribution.Source: §3 · p. 3

3.1. Architecture

  • Each policy has observation encoding, communication, and action decoding stages, with task-dependent network choices and on-policy REINFORCE or decentralized MAPPO training.Communication itself includes message encoding, message passing, and decoding, with multiheaded attention for selecting important messages.Source: §3.1 · p. 3

3.2. Objective

  • The information bottleneck balances encoded-information complexity against relevant information received from other agents, with contrastive learning signaling utility and token independence further promoting compositionality.The objective is expressed as a Lagrangian combining utility and complexity bounds.Source: §3.2 · pp. 3–4

4. Complexity through Compositional Communication

The method builds messages from tokens intended to encode relevant input information with minimal redundancy. Information bounds, variable-length sequencing, and an attention-based generator support this compositional communication design.

4. Complexity through Compositional Communication

  • Messages comprise fixed-bit tokens under a total sequence budget; each token should represent the hidden state while adding information not already carried by preceding tokens.The generator samples tokens from a categorical posterior and maps them to nearest-neighbor vocabulary entries.Source: §4 · p. 4

4.1. Independent Information

  • The method bounds interaction information among tokens to encourage low overlap, consistent with the result that arbitrary token pairs have low or near-zero mutual information.The proposition’s proof is in Appendix A.1.Source: §4.1 · p. 4, §4.3 · p. 4

4.2. Input-Oriented Information

  • An upper bound on mutual information between the composed message and encoded input is used as a Lagrangian term to promote message complexity control.The bound is expressed as a sum of expected KL divergences between token posteriors and a prior.Source: §4.2 · p. 4

4.3. Sequence Length

  • Because repeated tokens are redundant, the method can use a self-supervised end-of-sequence token to set variable message length.The token-independence argument also indicates that irrelevant information need not be selected.Source: §4.3 · p. 4

4.4. Message Generation Architecture

  • The generator expands compressed tokens into hidden-state space, uses softmin attention to reduce similarity with prior tokens, and samples new tokens from a variational distribution.Equation 1 is cited as allowing direct message generation at execution.Source: §4.4 · p. 5

5. Utility through Contrastive Learning

The section frames communication utility as mutual information between agent j’s message and agent i’s outcome, and uses a contrastive objective to lower-bound that quantity.

5. Utility through Contrastive Learning

  • The Markov Network links agent j’s H_j to message M_j, then to agent i’s Y_i, with H_i also informing Y_i.The objective is to satisfy I(M_j; Y_i) through contrastive learning.Source: §5 · p. 5
  • Utility mutual information is lower bounded by a contrastive NCE-binary objective.The proposition’s proof is in Appendix A.1.Source: §5 · p. 5
  • The result indicates that gradient information must flow backward across agents along communication-edge connections.Source: §5 · p. 5

6. Experiments and Results

The experiments evaluate emergent communication for input-oriented information, task utility, heterogeneous-agent alignment, and social shadowing across 10 seeds. They test referential and coordination tasks, including high-dimensional pixel data and teams of agents with varying skill levels.

6.1. Environments

  • The blind traffic junction requires ten agents to coordinate without observing one another, while Pascal VOC asks two agents to identify two image labels from raw pixels.The Pascal VOC evaluation uses 534 images with exactly two labels drawn from a closed subset of five classes.Source: §6.1 · p. 6

6.2. Baselines

  • Baselines are no communication, policy-loss-only rl-comm, observation-grounded ae-comm, and VQ-VIB, which combines input grounding with a mutual-information objective for low-entropy messages.Source: §6.2 · p. 6

6.3. Input-Oriented Information Results

  • At β=0.1, tokens map bijectively to emergent concepts without redundancy; stronger complexity and independence losses increase sample complexity, while β=1 does not converge.The table defines minimum redundancy as 1.0, corresponding to a bijection.Source: §6.3 · p. 6, Table 1 · p. 6
  • 2.3x fewer message bits result from the derived regularization, for an 8x reduction versus ae-comm; tokens may compress to 10 bits each, yielding 20-bit messages.The reported independence loss is around 1e-4.Source: §6.3 · pp. 6–7

6.4. Communication Utility Results

  • Our compositional, contrastive method outperforms methods grounded only in input-oriented communication, achieving higher task success with lower sample complexity in the blind traffic junction.The contrastive update can spike to high success before convergence, leaving room for an adaptive loss.Source: Figure 3 · p. 7, §6.4 · p. 7

6.5. Heterogeneous Alignment Through Communication

  • On Pascal VOC, the method significantly outperforms ae-comm and no-comm, with agents guessing more than two of four labels across the pair.The authors attribute this to composable, mutually independent concepts that are easier to discriminate.Source: Figure 4 · p. 7, §6.5 · p. 7

6.6. Social Shadowing

  • Social shadowing learns an action policy with almost half the sample complexity of online reinforcement learning while observing an expert's actions.During shadowing, the agent updates its communication policy with the contrastive objective rather than learning the expert's action policy.Source: Figure 5 · p. 8, §6.6 · p. 8

7. Discussion

The discussion argues that communication can help agents understand intent, align policies, and coordinate, with expert examples offering a way to bootstrap communication. It connects independent, compact emergent messages to coordination and interpretability.

7. Discussion

  • Contrastive communication is presented as an optimal critic for coordination, while expert examples from social-shadowing episodes may bootstrap communication.The discussion distinguishes referential objectives from the harder multi-agent challenge of coordination.Source: §7 · p. 8
  • Minimizing the independence objective yields tokens with little overlapping information, and the complementary objectives are not trivially minimized during training.The authors connect this design to benefits of reinforcement learning in multi-agent settings.Source: §7 · p. 8
  • Mutual-information regularization produces small messages for sparse communication and discrete emergent concepts that can be made human-interpretable through post-hoc analysis.The discussion frames this as a step toward white-box machine learning in multi-agent settings.Source: §7 · p. 8
  • The framework is proposed as a step toward zero-shot communication and future ad-hoc teaming with agents and humans.The authors describe these social scenarios as future inspiration for ad-hoc teaming.Source: §7 · p. 8

A.1. Proofs

The proofs bound information in composed messages and relate the contrastive objective to mutual information. They also connect the critic to goal-conditioned values interpreted through future-state occupancy.

A.1. Proofs

  • The compositional-message analysis assumes token independence and uses variational approximations to bound interaction information and message information.The message is represented as a vector of tokens, with a Gaussian approximation and a standard-Gaussian reference.Source: §A.1 · pp. 11–12
  • The contrastive NCE-binary objective lower-bounds mutual information between messages and coordination information.The proof identifies the objective with NCE-binary and constrains its critic with a sigmoid.Source: §A.1 · pp. 12–14
  • The goal-conditioned Q-function is equivalent to the discounted occupancy probability of a target state.The critic optimizing the objective is related to this Q-function up to a multiplicative constant.Source: §A.1 · pp. 12–13
  • The critic compares the encoding of the current state and message with the encoding of a future rollout state.The critic is defined as a similarity between those encodings.Source: §A.1 · p. 13
Loading 2302.14276v1…