PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations

Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pineau, Jakob Foerster

arXiv:2010.15896v2cs.MAcs.AI

TL;DR

Emergent communication has largely used symbolic channels, whose protocols may not generalize to independently trained partners. This paper studies embodied agents communicating through joint motion and evaluates training approaches that use energy as a latent feature. The authors report around 61% zero-shot performance for one combined training setup, while identifying optimization challenges.

  • Problem

    Emergent communication has primarily used symbolic channels, and the resulting protocols may not generalize to independently trained partners.

    Source: §1 · p. 1

  • Method

    The paper trains embodied agents to communicate intent through articulated motion, testing energy-based latent inputs and sender pretraining alongside trajectory-based training.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    Around 61% zero-shot performance is reported for 10 intents when self-play uses full trajectories and the external observer uses energy input, combined with sender pretraining.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that physical grounding can support protocols that generalize to novel partners under their assumptions, though current approaches remain brittle.

    Source: §6 · p. 9

    Protocol learning remains difficult because many local optima and combinatorial energy ordering can yield low-loss solutions without the desired ordering, especially as intents grow and their likelihoods overlap.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Effective communication is an important skill for enabling information exchange and cooperation in multi-agent settings. Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels. One limitation of this setting is that it does not allow for the emergent protocols to generalize beyond the training partners. Furthermore, so far emergent communication has primarily focused on the use of symbolic channels. In this work, we extend this line of work to a new modality, by studying agents that learn to communicate via actuating their joints in a 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a common-knowledge energy cost, these agents can find protocols that generalize to novel partners. We also explore and analyze specific difficulties associated with finding these solutions in practice. Finally, we propose and evaluate initial training improvements to address these challenges, involving both specific training curricula and providing the latent feature that can be coordinated on during training.

1 INTRODUCTION

The introduction extends emergent communication from symbolic channels to embodied agents that signal through articulated motion, addressing communication with independently trained partners. It motivates energy costs and non-uniform intent frequencies as shared structure for zero-shot communication, while noting that learning such protocols is difficult.

1 INTRODUCTION

  • Symbolic cheap-talk protocols can be mutually incompatible across independently trained partners, making zero-shot coordination difficult.For example, independently permuting channel bits and adapting receiver policies can preserve payouts while making senders and receivers incompatible.Source: §1 · p. 1
  • The paper studies referential communication through agents’ articulated joint motion, with each message represented by a trajectory in a simulated 3D world.This extends prior grounded-robot language work, which the authors distinguish from their use of deep reinforcement learning for articulated motion [Steels et al., 2012; Spranger, 2016].Source: §1 · pp. 1–2
  • Physical motion requires energy, and non-uniform intent frequencies can support protocols that use lower-energy trajectories for more common intents.The paper identifies the Zipf distribution as a proxy for many real-world distributions associated with human activity.Source: §1 · p. 2
  • Learning these protocols is difficult because sender–receiver lock-in, latent-structure discovery, and local optima complicate optimization and energy ordering.The authors frame the work as an initial exploration of these difficulties and training improvements, with potential relevance to intermediate settings between self-play and zero-shot coordination.Source: §1 · pp. 2–3
  • Around 61% zero-shot performance is achieved with full trajectories during self-play, energy-only input for the external observer, and sender pretraining, across 10 intents.Sender pretraining alone yields around 35% for 10 intents; providing energy during training without that combination yields around 34%, equivalent to the most frequent Zipf-distribution class.Source: §1 · p. 2

2 BACKGROUND

The background introduces a centralized-training, decentralized-execution reinforcement-learning framework and defines zero-shot coordination as test-time coordination with novel partners. It also describes the policy-gradient approach used for continuous actions.

2.1 MULTI-AGENT REINFORCEMENT LEARNING

  • The protocol-learning problem is formalized as a decentralized partially observable Markov decision process, with centralized training and decentralized execution.Agents share state and action spaces, receive private observations, and aim to maximize shared return.Source: §2.1 · p. 3
  • For continuous action spaces, the paper uses a model-based policy-gradient approach with differentiable dynamics and SVGinfinity.Stochastic value gradient methods compute the policy gradient through backpropagation.Source: §2.1 · p. 3

2.2 ZERO-SHOT COORDINATION

  • Zero-shot coordination evaluates whether agents can coordinate at test time with independently trained, novel partners rather than overfit to training partners.The paper connects this setting to generalization to unseen agents, such as new human partners.Source: §2.2 · p. 3

3 PROBLEM SETTING

The problem setting treats communication as a referential game in which independently trained senders and receivers are paired at test time. Agents produce joint-motion trajectories that observers use to infer intent.

3 PROBLEM SETTING

  • Cross-play pairs independently trained senders and receivers who have not encountered or influenced one another during training, providing an out-of-distribution test of zero-shot communication.Sender–receiver pairs first train protocols in self-play.Source: §3 · p. 3
  • Agents’ states include joint configurations and discrete intents, while their communication actions are joint angular velocities observable by other agents.These actions have no effect on the environment.Source: §3 · p. 3
  • A differentiable forward-kinematics model generates motion trajectories, which an observer uses to predict the actor’s intent; cross-entropy supplies the shared return.The observer’s loss is backpropagated to the actor under centralized training.Source: §3 · pp. 3–4
  • Gaussian action noise supports imperfect observability and exploration while regularizing the continuous channel into a coarse-grained space that keeps intent trajectories distinguishable.The passage says this noise prevents trajectory collapse.Source: §3 · p. 4

4 METHODOLOGY

The methodology addresses the difficulty of finding embodied communication protocols that generalize to new partners by inducing latent structure in motion. It tests whether observers can use that structure to decode intents across independently trained agents.

4.1 INDUCING IMPLICIT LATENT STRUCTURE THROUGH PHYSICAL ENERGY EXERTION

  • Energy regularization and a Zipf distribution over intents encourage lower-energy trajectories for more frequent intents, potentially inducing an energy-to-intent mapping.The paper notes that the global optimum would order energy values increasingly and intents decreasingly, yielding a presumed one-to-one mapping.Source: §4.1 · pp. 4–5
  • Finding the intended energy ordering is difficult because the optimization has many local optima, and nearby intent frequencies can make ordering harder as the number of intents grows.Improving a local solution may require changing actions for multiple intents concurrently.Source: §4.1 · p. 4

4.2 PROVIDING EXPLICIT LATENT STRUCTURE

  • Explicit latent trajectory features are intended to reduce message dimensionality while preserving informativeness, allowing an observer that infers the encoding structure to decode intents.With a Zipfian intent distribution, the paper frames this as maximizing mutual information between goals and latent features.Source: §4.2 · p. 5

4.3 THIRD-PARTY OBSERVER EVALUATION

  • The evaluation trains an external observer on one set of independently trained actors and tests whether it can interpret intents from a disjoint set of unseen actors.Policies are frozen and the observer is randomly initialized; success on unseen partners is treated as zero-shot communication.Source: §4.3 · p. 5

5 EXPERIMENTS AND RESULTS

Experiments test whether energy-based latent structure supports generalizable protocols and whether observers can discover that structure. Across the tasks, energy structure supports zero-shot coordination, but explicit discovery by agents remains critical.

5.1 EFFICACY OF INDUCING IMPLICIT STRUCTURE IN PROTOCOL LEARNING

  • Training partners reached near-perfect protocol performance, but none of the conditions immediately generalized to unseen partners; adding energy slowed convergence but reduced exertion by several orders of magnitude.Figure 2 contrasts training-partner success with out-of-distribution actor–observer pairs.Source: §5.1 · pp. 5–6, Figure 2 · p. 6

5.1.1 EFFICACY OF IMPLICIT STRUCTURE IN DISCRETE-CHANNEL PROTOCOLS

  • CP performance was 0.15±0.12 with Zipf only versus 0.58±0.22 with Zipf + energy, despite comparable SP performance.SP performance was 0.97±0.003 versus 0.97±0.002, respectively.Source: §5.1.1 · p. 6

5.2 IMPACT OF LATENT STRUCTURE FOR ZERO-SHOT COORDINATION

  • Near-perfect zero-shot coordination on the two-intent task occurred only when the external observer received explicit energy values at test time.This held independently of the information provided during observer training.Source: §5.2 · p. 7, Table 1 · p. 7
  • Intent-conditioned energy distributions had little overlap, enabling observers given energy values to decode messages independently of actors’ motion styles.The authors interpret this as an informative relationship between intent and energy exertion.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3 EXAMINING COMMUNICATION GENERALIZATION WITH A LEARNING CURRICULUM

  • For five- and ten-intent tasks, zero-shot performance was maximized with an energy curriculum and explicit latent structure, while trajectory input without curriculum never beat the max-class baseline.The torque curriculum notably improved generalization only for trajectory-input observers on the ten-intent task.Source: Table 2 · p. 8, §5.3 · pp. 8–9
  • As intent counts rise, the number of possible energy orderings grows combinatorially, and training repeatedly repositions less-likely intents, risking local optima and coordination disruption.Intent 1 moved little, while intents 4 and 5 shifted more often and could remain incorrectly ordered.Source: §5.3 · p. 9, Figure 6 · p. 9

5 EXPERIMENTS AND RESULTS

  • The experiments identify energy-based latent structure as advantageous for generalizable protocols, while discovery of that structure remains a separate requirement for zero-shot coordination.Source: §5.3 · p. 9

6 CONCLUSION AND FUTURE WORK

The paper’s embodied agents can learn protocols that generalize to novel partners under mild assumptions, but current approaches remain brittle. The authors identify improved use of environmental common knowledge as a direction for greater robustness.

6 CONCLUSION AND FUTURE WORK

  • Under mild assumptions, physical grounding should let embodied agents learn communication protocols that generalize to novel partners.The paper describes this as a first exploration in high-dimensional simulated environments.Source: §6 · p. 9
  • Current approaches are more brittle than hoped, leaving open challenges for continuous-action communication in embodied agents.Source: §6 · p. 10
  • The authors hypothesize that robustness may improve by distinguishing agent-specific or shared knowledge from common knowledge usable for coordination.They propose seeking the maximum coordination possible from the environment’s available common knowledge, or “grounding.”Source: §6 · p. 10

A.1 EMBODIED AGENT MORPHOLOGIES

Agents use articulated bodies whose joints obey a predefined topology and kinematic constraints. In this setup, joints rotate in place, and forward kinematics ensures valid transitions between joint states.

A.1 EMBODIED AGENT MORPHOLOGIES

  • An articulated body is a tree of rigid links connected at joints, with a root joint and predefined topology constraining its motion.Source: §A.1 · p. 12
  • The root joint has six degrees of freedom, while other joints have three rotational degrees of freedom; the study constrains all joints to rotate.As a result, agents cannot navigate through the environment and can only move their bodies in place.Source: §A.1 · p. 12
  • A differentiable forward-kinematics model adapted from Holden et al. (2017) ensures transitions between joint states satisfy the agent’s kinematic constraints.Source: §A.1 · p. 12

A.2 PROTOCOL TRAINING

Training pairs intent-conditioned sender trajectories with a receiver that predicts the intent. The agents share a recognition-based return and update their policy and discriminator parameters.

A.2 PROTOCOL TRAINING

  • For each sampled intent batch, the sender generates a trajectory by producing policy actions from the current state and target intent.The batch is sampled from a predefined intent distribution.Source: §A.2 · p. 12
  • Exploration and imperfect actuation are modeled by adding Gaussian noise to actions before forward kinematics computes the next state.Source: §A.2 · p. 12
  • The receiver predicts the intent from the trajectory, and the shared return is the negative loss between the actual and predicted intents.Policy and discriminator parameters are then updated.Source: §A.2 · p. 12

A.3 EXPERIMENTAL SETUP

Experiments use an intent-recognition referential task with populations of embodied agents and noisy observations of joint motion. External-observer evaluation tests whether a new discriminator can recognize trajectories from held-out actors.

A.3 EXPERIMENTAL SETUP

  • The intent-recognition task uses 10 concepts and populations of 10 agents, with batches sampled from uniform or Zipf distributions.Receivers observe sender joint positions and orientations with Gaussian observation noise; policy and discriminator are feed-forward networks.Source: §A.3 · p. 12
  • Agents use a four-joint robot-arm morphology with unrestricted joint motion, angle values wrapped to [0, 2π], and a five-step horizon.The study upsamples frames for visualization.Source: §A.3 · pp. 12–13
  • External-observer evaluation trains a newly initialized discriminator on 80% of agents and tests it on trajectories from the remaining 20%.Experiments used 2- and 5-concept tasks for 500 iterations and the 10-concept task for 1,000 iterations; learning stabilized after about 100 iterations.Source: §A.3 · p. 13

A.4 DISCRETE DOMAIN: TASK 1 – ADDITIONAL RESULTS

Task 1 compares protocols learned in the Zipf-only baseline and Zipf-plus-energy condition, then examines whether they generalize to unseen receivers. The comparison emphasizes that self-play success alone does not ensure zero-shot coordination.

A.4 DISCRETE DOMAIN: TASK 1 – ADDITIONAL RESULTS

  • Task 1 assigns each of 10 discrete sender actions a unique energy value, Φ(a) = id(a).Figures 7a and 8a show protocols from baseline and experimental self-play; Figures 7b and 8b show pairings with unseen receivers.Source: §A.4 · p. 13
  • Without consistent structure across independently trained pairs, generalization to novel agents in cross-play becomes exceedingly difficult.The comparison confirms this difficulty even with small discrete action spaces.Source: §A.4 · p. 13
  • Side-by-side self-play and cross-play comparisons test whether independently learned policies contain structure that unseen agents can exploit.The comparison includes locally optimal, compatible sender-receiver policies from self-play.Source: §A.4 · p. 13

A.5 DISCRETE DOMAIN: TASK 2 (PROTOCOL LEARNING GIVEN ACTION DEGENERACIES)

Task 2 tests zero-shot coordination when multiple discrete actions share energy costs, a setting with more ways for self-play partners to choose equally low-cost actions. Energy-based structure helps cross-play relative to the Zipf-only baseline, but coordination remains difficult.

A.5 DISCRETE DOMAIN: TASK 2 (PROTOCOL LEARNING GIVEN ACTION DEGENERACIES)

  • With 5 intents, 17 actions map to five energy values: one action has the unique lowest cost, while each other value maps to four actions.Thus several actions can share an energy cost.Source: §A.5 · p. 13
  • Equal-cost actions let self-play partners assign different low-energy actions to different intents, creating multiple optimal protocols and making zero-shot coordination harder.The challenge grows when intent likelihoods are close: distinct trajectories may have comparable low energy, making energy-based intent inference difficult for a new receiver.Source: §A.5 · p. 13
  • 0.20±0.07 for the zipf (only) condition versus 0.35±0.2 for the zipf + energy penalty condition in cross-play success.The experimental condition remains below the approximately 0.44 probability of selecting the most frequent class.Source: §A.5 · pp. 13–14, Figure 12 · p. 20
  • With Zipf plus energy, zero-shot coordination is somewhat more successful than in the baseline, though still more challenging than communication with training partners.Figure 10 illustrates learned experimental protocols; Figure 12 compares self-play and cross-play outcomes.Source: §A.5 · p. 14, Figure 10 · p. 18, Figure 12 · p. 20
  • The energy condition does not exceed the approximately 0.44 probability of selecting the most frequent class, underscoring the difficulty of learning zero-shot policies in this task.The authors report that action degeneracies make coordination harder than in the simpler task with unique energy values.Source: §A.5 · p. 14
  • In the Zipf-only baseline, senders learn effective self-play policies, but substantial variation across senders leaves little evident common structure for novel partners.Figure 9 shows baseline protocols and self-play performance; Figure 11 contrasts self-play and cross-play, where communication is more challenging.Source: Figure 9 · p. 17, Figure 11 · p. 19
Loading 2010.15896v2…