PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Ground Multi-Agent Communication with Autoencoders

Toru Lin, Minyoung Huh, Chris Stauffer, Ser-Nam Lim, Phillip Isola

arXiv:2110.15349v1cs.LGcs.AIcs.CLcs.MA

TL;DR

Decentralized agents need a shared understanding of communication symbols, yet finding a protocol through unguided exploration is difficult. This paper uses observation autoencoding to ground messages, and reports improved performance across coordination tasks compared with baselines.

  • Problem

    Decentralized agents lack a common grounding for communication symbols, making meaningful protocols difficult to learn amid high-variance exploration.

    Source: §1 · pp. 1–2

  • Method

    Agents independently auto-encode observations and use the learned representations to ground messages in a fully decentralized communication framework.

    Source: §4 · p. 3

  • Results

    The proposed method achieves much better performance on a suite of coordination tasks compared with baselines.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings support observation autoencoding as a common grounding for decentralized communication across varied MARL coordination tasks.

    Source: §1 · p. 2, §6 · p. 9

    The method assumes agents share the same model architecture and autoencoding loss, and may fail when agents need to select information to communicate.

    Source: §6 · p. 10

Abstract

from arXiv · show

Communication requires having a common language, a lingua franca, between agents. This language could emerge via a consensus process, but it may require many generations of trial and error. Alternatively, the lingua franca can be given by the environment, where agents ground their language in representations of the observed world. We demonstrate a simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that a standard representation learning algorithm -- autoencoding -- is sufficient for arriving at a grounded common language. When agents broadcast these representations, they learn to understand and respond to each other's utterances and achieve surprisingly strong task performance across a variety of multi-agent communication environments.

1 Introduction

Meaningful communication requires agents to share an understanding of their symbols, but decentralized agents lack a common grounding and face difficult exploration. The paper proposes grounding communication in learned observation representations and reports improved coordination performance.

1 Introduction

  • Prior approaches use supervision, centralized learning, or differentiable communication, unlike natural communication protocols that emerge without supervision or a centralized coordinator.The paper also notes that decentralized models perform poorly even on simple communication tasks or with additional inductive biases.Source: §1 · p. 1
  • Without common grounding, agents’ symbols lack shared meaning, making consistent protocols difficult to discover and random messages add to high-variance learning.The paper notes that these difficulties increase with the time horizon, communication space, and number of agents.Source: §1 · pp. 1–2
  • Observation autoencoding gives decentralized agents a common grounding, easing communication and improving coordination on tasks that are nearly unsolvable without communication.The authors identify this as a way to avoid learning grounding from scratch through random exploration.Source: §1 · p. 2
  • The paper formulates communication grounding as representation learning and reports that autoencoder-based communication consistently outperforms baselines across varied MARL environments.It frames the findings as evidence for the importance of visual grounding in emergent communication.Source: §1 · p. 2

2 Related Work

Prior decentralized communication methods face exploration and applicability challenges, while many approaches use structured state inputs. This work addresses pixel-based communication with learned representations.

2 Related Work

  • Decentralized MARL communication poses a difficult exploration problem, and centralized learning has not solved tasks that rely on emergent communication.Prior analyses and empirical results describe communication as exacerbating sparse, high-variance rewards.Source: §2 · p. 2
  • Differentiable communication permits agents to optimize one another’s communication policies through gradients, but imposes constraints on communication that limit applicability.Choi et al. [7] explore generating messages that the model itself can interpret, a high-level idea similar to this paper’s.Source: §2 · p. 2
  • Other decentralized methods require access to other agents’ policies or proximity, or depend on task-specific hyperparameter tuning with limited effectiveness.These constraints hinder application across a wider range of tasks.Source: §2 · p. 2
  • Unlike many communication studies using structured state inputs, this work learns protocols directly from raw pixels and reports effective representation-learning support across MARL task settings.The authors characterize pixel data as unstructured and ungrounded [3] [7] [26].Source: §2 · pp. 2–3

3 Preliminaries

The paper models communication-enabled MARL as a decentralized partially observable decision process. Its experiments use cooperative objectives and independently parameterized policies optimized with A3C.

3 Preliminaries

  • The model is a partially observable Markov decision process with state, action, communication, and observation spaces, and each agent observes only part of the underlying state.Agents also receive communicated messages from the previous time step.Source: §3 · p. 3
  • The work considers a fully cooperative setting in which each agent maximizes the total expected return of all agents.The objective is specified over a finite horizon with discount factor γ.Source: §3 · p. 3
  • Policies are optimized with A3C and Generalized Advantage Estimation, without centralized training or self-play, using an independent policy for each agent.The authors note that other policy optimization algorithms could also be used.Source: §3 · p. 3

4 Grounding Representation for Communication with Autoencoders

The method uses self-supervised observation autoencoding to provide agents with a common grounding for decentralized communication. A speaker generates messages from image features, while a listener combines image and message features to select actions.

4 Grounding Representation for Communication with Autoencoders

  • A self-supervised representation-learning task lets agents independently auto-encode their observations to learn a common grounding, without extra architectural bias or supervision.The approach is intended to make messages understandable and meaningful to listeners, even when agents need not use identical symbols.Source: §4 · pp. 3–4

4.1 Speaker Module

  • The speaker encodes raw pixels into image features, then an autoencoder maps those features to communication symbols and reconstructs the input state from the quantized message.The reconstruction loss is optimized jointly with the listener’s policy-gradient loss; a straight-through estimator differentiates through quantization.Source: §4.1 · p. 4

4.2 Listener Module

  • The listener encodes previous messages, concatenates message and image features, and uses a GRU policy to predict an action distribution and expected returns.The policy-gradient loss is jointly optimized with the speaker’s autoencoder reconstruction loss.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate observation autoencoding as a way to ground communication across three multi-agent environments. The analyses compare communication baselines, test joint reinforcement-learning training, and examine learned messages and listener behavior.

5.1 Environments

  • The evaluation spans CIFAR referential image exchange, FindGoal cooperative search, and ordinal RedBlueDoors coordination.The environments cover different communication settings, including referential or non-referential and ordinal or non-ordinal tasks.Source: §5.1 · pp. 5–6

5.2 Baselines

  • Baselines include no communication, reinforcement-learning communication, reinforcement learning with signaling and listening biases, and a policy trained atop autoencoded representations.Source: §5.2 · p. 6

5.3 The Effectiveness of Grounded Communication

  • CIFAR Game baselines stay near random while ae-comm earns higher reward; ae-comm reaches optimal RedBlueDoors strategy after 150k training and speeds FindGoal completion.FindGoal agents can solve the task without communication, but communication improves their performance.Source: §5.3 · pp. 6–7
  • Autoencoding-based communication consistently outperforms baselines across all environments.Table 1 reports average reward for CIFAR Game and RedBlueDoors, and episode length for FindGoal.Source: §5.3 · p. 7, Table 1 · p. 7, Figure 3 · p. 7

5.4 The Role of Autoencoding

  • Adding reinforcement-learning policy training to the autoencoder representation degrades performance across environments, except FindGoal, where performance stays about the same.The authors hypothesize that weak correlation between visual observations and communication rewards can produce conflicting gradient updates and worsen high-variance optimization.Source: §5.4 · p. 8, Figure 4 · p. 8

5.5 Analyzing the Effects of Communication Signals on Agent Behavior

  • In RedBlueDoors, message clusters align with task phases, and listener entropy drops after milestones for ae-comm agents but not agents trained without autoencoding.The clusters include messages associated with no visible doors and with the red door opened.Source: §5.5 · pp. 8–9, Figure 5 · p. 9
  • Adding communication improves task performance most substantially for ae-comm, which uses representation learning to support positive signaling and measures positive listening through reward gains.The positive-signaling and positive-listening metrics are suggested by [29].Source: §5.5 · p. 9

6 Discussion and Societal Impacts

The paper grounds decentralized communication through autoencoding and reports stronger coordination-task performance than baselines. It also identifies constraints on agent homogeneity, information selectivity, and cooperative settings.

6 Discussion and Societal Impacts

  • Agents trained with the proposed method achieve much better performance across coordination tasks than baselines.Table 2 reports performance gain with communication and says ae-comm agents improve more than baselines.Source: §6 · p. 9, Table 2 · p. 9
  • Autoencoder-trained agents’ messages are effectively used by other agents, as reflected in policy entropy across 256 episodes.The figure measures entropy over task progress for FindGoal and RedBlueDoors.Source: Figure 6 · p. 10
  • The framework extends computational inquiry into language acquisition, language evolution, and social learning, and may support improved coordination and cooperation in artificial agents.The authors contrast this with work relying on passive observations of other agents [47] [48].Source: §6 · p. 10
  • The method assumes agents share a model architecture and autoencoding loss, and may fail when they need to selectively communicate information.The authors also characterize their work as a partial step that does not address joint speaker-listener optimization.Source: §6 · p. 10
  • The evaluated environments are fully cooperative; real-world deployment would require excluding adversaries or developing robust counterstrategies.The authors note adversaries could use communication policies to lie, spread misinformation, or manipulate other agents.Source: §6 · p. 10

A Implementation Details

The implementation uses partially observed grid environments and a shared encoder–communication–policy model pipeline.

A.1 Environment

  • FindGoal uses three agents on a 15×15 grid with 7×7 views; RedBlueDoors uses two agents on a 10×10 grid with 3×3 views.FindGoal has one goal, 25 obstacles, and a 512-step limit; RedBlueDoors has opposite-side red and blue doors and a 2048-step limit.Source: §A.1 · p. 13

A.2 Model

  • The model encodes images with a CNN, maps encoded observations to 10-dimensional communication messages, embeds messages, and feeds features to a GRU policy.The image encoder outputs 32 channels at 3×3; the message encoder outputs 128 features, and the GRU hidden size is 128.Source: §A.2 · p. 13

B Training Details

Experiments used Adam, with a hyperparameter sweep to select settings.

B.1 Hyperparameters

  • Adam used a learning rate of 0.0001, β1 = 0.9, β2 = 0.999, and ϵ = 10^-8; sweeps covered the listed alternative values.The learning-rate sweep was 0.01, 0.001, 0.0001, and 0.00001; β1, β2, and ϵ were also varied.Source: §B.1 · p. 13

B.2 Compute Resources

Experiments ran on a four-GPU internal cluster; the additional-baseline results use repeated-seed evaluation.

B.2 Compute Resources

  • Experiments ran on an internal cluster with four NVIDIA GeForce RTX 2080 Ti GPUs.Source: §B.2 · p. 13
  • Additional-baseline results report mean and 95% confidence intervals from 100 episodes per seed across 10 random seeds.The reported metric is average reward for CIFAR Game and RedBlueDoors, and average episode length for FindGoal.Source: Table 3 · p. 14

C Additional Experiments

The added baselines compare autoencoded messages with policy-network states sent directly as communication. Full-latent communication performs close to ae-comm, while ae-comm uses low-dimensional discrete utterances.

C.1 fc-comm and latent-comm Baselines

  • Fc-comm sends discretized policy-network states without autoencoding, matching ae-comm messages in shape and range.A fully connected layer is added to the policy states for this match.Source: §C.1 · p. 14
  • Latent-comm sends pre-policy-head latent states through a fixed-size discrete channel, using either a reduced final hidden layer or its first N features.Outputs are quantized identically and use a straight-through estimator [1].Source: §C.1 · p. 14
  • Full-latent-comm performs close to ae-comm, which achieves similar performance using low-dimensional discrete utterances.The paper presents full-latent-comm as an upper bound and notes the potential practical utility of low-bandwidth communication channels.Source: §C.1 · p. 14

C.2 Training Emergent Communication via Policy Optimization

Policy optimization on top of autoencoding remained difficult: extending training and alternating the objectives did not improve performance. Task performance improved as representation loss decreased, but not substantially until that loss was sufficiently low.

C.2 Training Emergent Communication via Policy Optimization

  • Performance did not improve under RL optimization, even after training for up to 1 million iterations.The authors describe policy optimization on top of autoencoding as an unresolved problem.Source: §C.2 · p. 14
  • Alternating autoencoding and RL training froze one component while the other trained to stable loss, but extended training still did not improve performance.Figure 7 visualizes the alternating training segments, beginning with the autoencoding objective.Source: Figure 7 · p. 15, §C.2 · p. 15

C.3 Autoencoder Loss

  • Task performance improved as representation-learning loss decreased, with no drastic task-performance gain until representation loss was sufficiently low.Figure 8 shows both agent performance and autoencoder loss improving over learning.Source: §C.3 · p. 15, Figure 8 · p. 15
Loading 2110.15349v1…