PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Discrete Communication in Semantic Spaces

Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia Sycara, Michael Lewis, Julie Shah

arXiv:2108.01828v3cs.LGcs.CLcs.MAcs.RO

TL;DR

One-hot discrete messages do not encode relationships between tokens, limiting desirable communication properties such as zero-shot understanding. The paper introduces learnable discrete tokens in a continuous semantic space and finds benefits in noisy self-play, novel human communication, and human interpretation of agent messages.

  • Problem

    One-hot tokens lack relationships between messages, while zero-shot understanding is a desirable but limited property of discrete emergent communication.

    Source: §1 · p. 1

  • Method

    The paper gives agents learnable discrete tokens in a continuous space, using a prototype-based architecture and word-embedding alignment for human communication.

    Source: §2.1 · p. 2, §3.3 · p. 4

  • Results

    Prototype tokens clustered in human-understandable patterns, supported robust team performance under environment noise, and enabled responses to novel communication and human interpretation of unlabeled agent messages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments demonstrate that prototype-based discrete communication can support noisy self-play and zero-shot understanding of both human messages and agent tokens.

    Source: §7 · p. 9

    The technique adds learnable parameters relative to one-hot agents, which may increase learning difficulty.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

1 Introduction

Agents can learn communication through reinforcement learning, but one-hot tokens lack relationships that could support desirable language properties. The paper proposes discrete tokens learned within a continuous semantic space and reports benefits across noisy self-play and human-agent communication.

1 Introduction

  • Reinforcement-learning agents can learn to communicate by broadcasting numerical vectors, while one-hot messages have been studied as a discrete lexicon.Researchers examine emitted tokens to study their meanings and properties such as compositionality or zero-shot understanding.Source: §1 · p. 1
  • One-hot tokens are orthogonal and equally distant, so they do not encode relationships between tokens.The authors argue this prevents agents from learning some desirable language properties.Source: §1 · p. 1
  • The paper proposes a novel architecture for discrete tokens learned within a continuous space and analyzes communication value decision-theoretically.It compares this approach with standard one-hot communication in discrete emergent communication.Source: §1 · p. 2
  • Prototype agents learned semantically meaningful token clusters, achieved robust team performance under environment noise, and used novel communication vectors.They also responded to novel English phrases and humans interpreted their unlabeled tokens in a reference game.Source: §1 · p. 2

2 Related Work

This work studies finite-message communication among decentralized agents without a task-specific language model. It relaxes the usual one-hot constraint by learning discrete tokens in a continuous semantic space, drawing on word embeddings and side information used in zero-shot classification.

2.1 Emergent Communication

  • Emergent communication studies how agents learn protocols for task performance, often using real-valued messages; this work focuses on finite messages among decentralized agents.Such agents do not access other agents’ weights or gradients during training or execution.Source: §2.1 · p. 2
  • The proposed architecture relaxes the one-hot constraint while retaining a finite message set and no task-specific language model.The authors aim to support use of novel communication and human understanding of learned messages.Source: §2.1 · p. 2

2.2 Natural Language Word Embeddings

  • Word embeddings represent discrete words in a semantic space where words with similar meanings can occupy similar locations.The paper draws an analogy between one-hot emergent messages and early one-hot word representations, which had limited generalizability.Source: §2.2 · p. 2

2.3 Zero-Shot Classification

  • Zero-shot classification uses side information, such as language, to classify new classes with few or no training examples; this paper instead applies side information to emergent communication.In experiments aligning tokens with word embeddings, the authors exploit side information for zero-shot understanding.Source: §2.3 · p. 3

3 Technical Approach

The technical approach models communication as part of decentralized partially observable multi-agent reinforcement learning, then learns discrete messages as selectable prototypes in a continuous space. Decision-theoretic analysis identifies when one-hot tokens are optimal, while lexicon-setting and supervised grounding offer ways to constrain learned behavior.

3.1 Emergent Communication in Multi Agent Reinforcement Learning

  • In a decentralized partially observable Markov decision process, agents choose policies to maximize expected discounted reward, with communication included among their actions.The framework defines state transitions, observations, rewards, and a discount factor for multiple agents.Source: §3.1 · p. 3

3.2 Decision-Theoretic Noisy Discrete Communication

  • Under equal token and communication dimensions, identity rewards, uniform token priors, and Gaussian noise, optimal tokens maximize pairwise squared distance; one-hot vectors are optimal within a unit hypercube.Relaxing uniform error costs or priors, or having unequal numbers of tokens and dimensions, can make one-hot communication suboptimal.Source: §3.2 · p. 4

3.3 Discrete Prototype Communication

  • The prototype architecture selects a discrete one-hot index from policy logits and multiplies it by a learnable token matrix to produce the communication token.The hard argmax is used in the forward pass, with the Gumbel-softmax operation used to calculate gradients in the backward pass.Source: §3.3 · p. 4
  • Prototype communication adds learnable parameters beyond one-hot agents, increasing flexibility but potentially making learning harder; reported experiments show net benefits.Behavior can also be constrained by manually fixing the token matrix or by adding supervised loss from observation-to-communication examples to the policy loss.Source: §3.3 · p. 4, §3.4 · pp. 4–5

3.4 Opportunities for Behavior Shaping

  • Lexicon-setting fixes a manually specified token matrix, while grounding uses a small supervised dataset to align observations with desired communications.Even with hand-specified tokens, their associated meanings are not known in advance; grounding is used to support a common understanding for human-agent communication.Source: §3.4 · pp. 4–5

4 Experiment Preliminaries

The experiments compare prototype-based communication with continuous and one-hot baselines, using self-play reward and reference-game understanding measures. The hypotheses predict benefits for prototype agents in noisy self-play, zero-shot interpretation of novel human communication, and human interpretation of agent tokens.

4.1 Baselines

  • The baselines are continuous real-valued vectors bounded by tanh and one-hot vectors selected with the Gumbel-softmax trick from c possible tokens.All methods use MADDPG, a policy-gradient method that updates network weights to maximize expected discounted reward.Source: §4.1 · p. 5

4.2 Hypotheses

  • Evaluation measures self-play reward and reference-game accuracy for agents interpreting novel human messages and humans identifying targets from unlabeled agent messages.The hypotheses predict higher prototype self-play reward in noisy environments, zero-shot agent understanding with word-embedding tokens, and accurate human interpretation of prototype tokens.Source: §4.2 · p. 5
  • The authors expect semantically similar tokens to cluster, supporting noisy-environment robustness, human understanding, and zero-shot interpretation of novel communication.These are proposed motivations for the hypotheses, not reported outcomes in this section.Source: §4.2 · p. 5

5 Agent Self-Play Experiments

Prototype-based communication improved robustness to noise in the triangle and 9 points environments, with learned tokens clustering according to landmark structure. In the uniform environment, continuous communication outperformed both discrete methods, while prototype interpolations often produced interpolated behavior.

5 Agent Self-Play Experiments

  • With z<c in triangle, prototypes spread across distant hypercube corners for robust communication and handled skewed target priors better than one-hot.One-hot instead learned to move to the location between the two most likely landmarks.Source: §5 · p. 6
  • In 9 points, nine prototypes encoded nine landmarks in a 3D channel; under noise, same-cluster tokens grouped, trading more within-cluster errors for fewer cross-cluster errors.The 9-prototype, 3D setting matched the reward of nine prototypes in 9D or nine one-hot tokens; in the noisy setting, prototype reward was -100 versus -96 for 3D one-hot.Source: §5 · pp. 6–7, Figure 2 · p. 6
  • In the uniform environment, continuous communication outperformed both discrete methods, while prototype and one-hot performed similarly and discretized continuous values.Seeker trajectories over 200 episodes showed the discrete pattern of locations.Source: §5 · p. 7, Table 2 · p. 7, Figure 3 · p. 7
  • When given pairwise prototype interpolations, prototype-based seekers often moved to the mean of the destinations associated with the two endpoint tokens.Figure 4 plots mean seeker destination across pairwise interpolations from 0% to 100%.Source: §5 · p. 7, Figure 4 · p. 7
  • Prototype tokens provided greater robustness to environmental noise in triangle and 9 points, with learned token separation and clustering visible in the prototypes.The self-play reward results summarize five runs.Source: Table 1 · p. 6, Figure 2 · p. 6, §5 · p. 7

6 Human-Agent Experiments

The human-agent experiments test zero-shot understanding in both directions: agents interpret novel human communication, and humans interpret unlabeled agent tokens. Prototype listeners support both forms, while participants also identify held-out tokens above chance.

6 Human-Agent Experiments

  • In the reference game, a speaker sent a vector after viewing a CIFAR10 image, and a listener chose between that target and a different-class distractor under a reward matrix favoring within-group confusions over cross-group errors.For the human-understanding experiment, participants saw eight labeled learned tokens before choosing which of two held-out-class images an unlabeled token referred to.Source: §6 · p. 8, §6.2 · p. 9

6.1 Zero-Shot Agent Understanding

  • The agent-understanding experiment withheld one animal and one vehicle class, training five teams for each of 24 excluded-class combinations.Evaluation tested listener accuracy on 500 trials using self-generated messages, class-label embeddings, or embeddings of human-written image captions.Source: §6.1 · p. 8
  • Prototype listeners supported zero-shot communication with novel messages, outperforming continuous and random one-hot baselines when both images and messages were novel.The authors also report that the embedding method affected agent performance and that reward-matrix and environment-noise ablations contributed to zero-shot understanding.Source: §6.1 · p. 9

6.2 Zero-Shot Human Understanding of Tokens

  • Participants selected the correct image for an unlabeled held-out token 70% of the time across five prototype models, versus 66% with BERT-based embeddings.The BERT comparison had χ2(1, N = 2093) = 3.21, p = .073.Source: §6.2 · p. 9

7 Contributions

The paper presents prototype-based communication as an approach to discrete emergent communication and reports semantic token spaces supporting zero-shot understanding. It frames the work as an initial connection to word embeddings and identifies further analysis and bias scrutiny as open needs.

7 Contributions

  • Prototype agents produced semantically meaningful communication spaces in noisy environments with non-identity reward matrices, supporting zero-shot understanding.The paper also reports increased robustness to environment noise and human-subject results on novel human communication and human interpretation of emergent tokens.Source: §7 · p. 9
  • The authors characterize their work as a first step connecting emergent communication and word embeddings, with further experiments needed on environmental factors that contribute to zero-shot understanding.They note that techniques for analyzing word embeddings could also be applied to emergent communication tokens.Source: §7 · p. 9
  • The authors caution that word-embedding methods and emergent tokens may reinforce biases, so desirable token properties require attention.Source: §7 · p. 9

Checklist

The checklist records affirmative responses about reproducibility materials, training details, error bars, and compute disclosures. It also reports asset attribution and documentation of human-subject procedures and limitations.

Checklist

  • The checklist marks code, data, and instructions for reproducing main experiments, training details, error bars, and compute-resource information as included or documented.It points to the supplement or a URL, appendices, reported standard errors, and Appendix C, respectively.Source: Checklist · p. 12
  • The checklist reports attribution and licensing for existing assets, while marking new assets as not applicable and stating that the data were unpublished and non-identifying.Source: Checklist · pp. 12–13
  • The checklist says human-subject materials, potential risks and approvals, and participant compensation details are documented in Appendix F.Source: Checklist · p. 13

A Appendix: Optimality of One-Hot Encodings

The appendix shows that one-hot tokens are locally optimal only under specific assumptions, and gives counterexamples where alternatives achieve higher reward. It also describes the particle environments used in the experiments.

A Appendix: Optimality of One-Hot Encodings

  • Under an identity reward matrix, uniform token prior, unit-hypercube constraints, and Gaussian additive noise, one-hot tokens are locally optimal.The proof establishes a local optimum, not a unique global optimum.Source: A Appendix: Optimality of One-Hot… · pp. 13–14

A.1 Uniform prior; non identity reward

  • When confusing tokens 0 and 1 still earns reward 1, making them identical reduces confusion with token 2; at σ = 0.5, 100,000 simulations gave Talt mean reward 0.96 versus 0.91 for one-hot tokens.Source: §A.1 · p. 14

A.2 Non-uniform prior; identity reward

  • With an identity reward but non-uniform priors, an alternative arrangement achieved reward 0.96 versus 0.92 for one-hot tokens at σ = 0.5.Because tokens 0 and 1 were more likely than token 2, distinguishing the first two mattered more than separating token 2.Source: §A.2 · p. 15

B Appendix: Particle Environment Details

  • The triangle, 9-points, and uniform environments used particle physics, with landmarks arranged around the origin or sampled within a square.The 9-points layout separated landmarks enough to encourage unique tokens while retaining visible clusters; the uniform environment used 100-timestep episodes.Source: B Appendix: Particle Environment Details · p. 15

C Appendix: Training Details and Hyperparameters

The appendix records shared training choices and experiment-specific settings for particle and human-agent studies. It also notes an unstable one-hot baseline in the under-parametrized 9-points case and details the supervised and human-interpretability protocols.

C Appendix: Training Details and Hyperparameters

  • Across experiments, agents used MADDPG, Adam with learning rate 0.01, and three-layer MLP policies; noisy training used identical noise models and zero-noise evaluation.Unless otherwise noted, τ = 0.01; trials used their zero-indexed trial number as the random seed, and the complete suite took approximately 48 hours.Source: C Appendix: Training Details and… · pp. 15–16

C.1.1 Triangle

  • Triangle runs used 10,000 episodes of 50 timesteps, with updates every two episodes and batch size 1,024.Training took approximately five minutes.Source: §C.1.1 · p. 16

C.1.2 9 Points

  • In the 9-points setting, one-hot agents with three tokens for nine targets showed unstable training; baselines used the best saved model from each run.Changing learning rates and τ over three orders of magnitude did not resolve the instability; training used 5,000 episodes and 50-timestep episodes.Source: §C.1.2 · p. 16

C.1.3 Uniform

  • Uniform-environment runs used 5,000 episodes of 100 timesteps; communication methods used different token counts, and interpolation locations averaged 20 trials per level and token pair.Prototype and continuous methods used c = 3, one-hot used c = 5, and initial σ values of 0.01 or 0.05 showed no large behavioral differences.Source: §C.1.3 · p. 16

C.2 Human-Agent Experiments

  • Human-agent experiments used a 2-timestep reference game and CIFAR10 features from a pretrained classifier, reduced to 10 dimensions with PCA.The classifier achieved 95% accuracy; PCA was fit on the training set and was found sufficient for high task performance.Source: §C.2 · p. 16

C.2.1 Zero-Shot Agent Understanding

  • For zero-shot agent understanding, speakers were pretrained on 100 supervised image-token examples, while communication received Gaussian noise with σ = 0.05.Agents nearly reached perfect reference accuracy in training; 1,000 supervised examples had minimal effect on task performance.Source: §C.2.1 · pp. 16–17, Table 4 · p. 17
  • AMT captions were sampled from captions for images of the correct class, a choice the authors say likely underestimated their discriminative power.Captions came from CIFAR10 training images, while evaluation used test images, so no exact image-caption matches existed.Source: §C.2.1 · p. 17

C.2.2 Zero-Shot Human Understanding of Tokens

  • For human interpretation, prototype speakers received a class-specific supervised loss on their softmax outputs, while the token matrix remained unguided.Participants interpreted a held-out token using labeled communication-space diagrams and selected between images from the two held-out classes.Source: §C.2.2 · p. 17

D Appendix: Reference Game Results

The appendix tests how reference-game behavior varies with training data, game components, and natural-language inputs, then describes two ways of constructing communication embeddings. Noise and a non-identity reward matrix support zero-shot understanding, while embedding choices shape which similarities agents can use.

D Appendix: Reference Game Results

  • Training with 1,000 rather than 100 supervised examples had minimal effect on task performance.The authors report similar behavior across a large range of supervised dataset sizes.Source: D Appendix: Reference Game Results · p. 17
  • Noise and a non-identity reward matrix were both useful for the best zero-shot understanding of human-generated labels.The authors relate noise to learning a communication space rather than isolated tokens, and the non-identity reward matrix to semantic relationships between tokens.Source: D Appendix: Reference Game Results · pp. 17–18, Table 5 · p. 18
  • Animal or vehicle inputs worsened in-distribution performance but had no or a small positive effect on zero-shot performance.These labels were outside the training distribution and were poor at distinguishing between two animals, for example.Source: D Appendix: Reference Game Results · p. 18, Table 6 · p. 18

E Appendix: NLP Embeddings

  • BERT sentence embeddings projected with task-independent PCA remained informative of class, with vehicle words clustering together.The authors describe this pattern in the 2D visualization and infer that the 3D PCA embeddings were at least as informative.Source: E Appendix: NLP Embeddings · p. 18, Figure 5 · p. 19
  • Hand-crafted embeddings clustered animals and vehicles and enabled better zero-shot performance than BERT-based models, but their token proximity caused noise-related confusion.The embeddings were designed to encourage those category similarities, not optimized for the reference game.Source: Table 7 · p. 19, E Appendix: NLP Embeddings · p. 20

F Appendix: User Study Details

The user studies collected human-generated image labels and tested whether participants could interpret agents’ zero-shot communication. Four of five prototype-based models outperformed BERT-based embeddings in the human interpretation experiment, though the overall difference was marginally significant.

F Appendix: User Study Details

  • For the image-label study, 51 workers produced 1,257 valid annotations for 500 CIFAR-10 images sampled across ten classes.Workers were asked for three labels per image; some images were difficult to distinguish because of low resolution.Source: F Appendix: User Study Details · p. 20
  • Four of five prototype-based models outperformed BERT-based embeddings in human interpretation, with a marginally significant overall difference.After excluding Proto-1, which failed to converge to high reward, the prototype models significantly outperformed BERT embeddings.Source: Table 8 · p. 21, F Appendix: User Study Details · p. 21
  • The annotations included class synonyms and image-specific attributes, alongside some mistaken classifications associated with low-resolution images.Examples included “puppy” and “canine” as dog synonyms, and “furry,” “small,” and “white” as image features.Source: F Appendix: User Study Details · p. 21
  • The interpretation study retained 2,093 valid responses from 253 workers after excluding unusually short or long completion times.Participants evaluated five trained models using the method and one BERT-based embedding across animal-vehicle image pairs.Source: F Appendix: User Study Details · p. 21
  • Participants acted as listeners in a reference game, choosing between two held-out-class images based on an agent’s communication.The interface showed communications in a 2D plane with eight labeled nodes and one unlabeled communication node.Source: F Appendix: User Study Details · p. 21, Figure 8 · p. 23
Loading 2108.01828v3…