PaperlayerView exact revision on arXiv

Source-linked AI summary

Biases for Emergent Communication in Multi-agent Reinforcement Learning

Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, Thore Graepel

arXiv:1912.05676v1cs.MAcs.CLcs.LG

TL;DR

Independent agents can struggle to learn communication without centralized training because joint exploration is difficult. The paper adds positive-signalling and positive-listening losses to encourage use of the channel; across a simple and a temporally extended environment, these biases ease learning or make communication emerge more consistently.

  • Problem

    In temporally extended reinforcement-learning tasks, independent agents have difficulty learning emergent communication without centralized training, in part because joint exploration is difficult.

    Source: §1 · p. 1

  • Method

    The paper adds auxiliary positive-signalling and positive-listening losses to RIAL to encourage speakers to vary messages and listeners to respond to them.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    The biases ease the learning problem in a simple environment and increase the consistency with which agents learn to communicate in both simple and temporally extended environments.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The losses are intended to get agents using the communication channel, after which MARL can find a useful protocol.

    Source: §3 · p. 3

    The experiments cover fully cooperative environments, leaving open whether the approach helps in partly cooperative or competitive settings where communication can expose agents to exploitation.

    Source: §5 · p. 9

Abstract

from arXiv · show

We study the problem of emergent communication, in which language arises because speakers and listeners must communicate information in order to solve tasks. In temporally extended reinforcement learning domains, it has proved hard to learn such communication without centralized training of agents, due in part to a difficult joint exploration problem. We introduce inductive biases for positive signalling and positive listening, which ease this problem. In a simple one-step environment, we demonstrate how these biases ease the learning problem. We also apply our methods to a more extended environment, showing that agents with these inductive biases achieve better performance, and analyse the resulting communication protocols.

1 Introduction

The paper addresses how independent agents can learn a shared communication protocol without an agreed language, a problem for decentralized multi-agent reinforcement learning. It proposes positive-signalling and positive-listening losses and evaluates them in one-step and temporally extended environments.

1 Introduction

  • Without an agreed communication protocol, agents must learn a language over their shared channel to maximize utility; straightforward RIAL implementations perform poorly.RIAL learns message policies through environmental reward.Source: §1 · p. 1
  • RIAL supports decentralized training and execution with discrete symbols, while differentiable communication models are less generally applicable or realistic and can model agents as a single entity.The paper builds on RIAL while targeting its joint-exploration difficulty.Source: §1 · p. 2, §1.1 · p. 2
  • The paper adds auxiliary positive-signalling and positive-listening losses to RIAL, finding that they facilitate communication in a one-step task and make it emerge more consistently in a treasure-hunt gridworld.The experiments also analyze how the biases change learning and interpret the resulting protocol.Source: §1 · p. 2

1.1 Related Work

  • Prior work includes decentralized Q-learning that converges in deterministic tabular settings but does not trivially extend to stochastic environments or function approximation, and centralized critics that use other agents’ training information.The paper notes that centralized critics violate its independence assumptions and may not scale well.Source: §1.1 · p. 2

2 Setting

The paper studies agents in partially observable Markov games that share a communication channel and learn through individual experience. Agents independently optimize their policies for a joint discounted reward, making the cooperative learning problem non-stationary and non-Markov for each agent.

2 Setting

  • Agents receive partial observations and other agents’ previous messages, then choose environment actions and messages as the game state transitions and individual rewards are generated.The setting is a partially observable Markov game with stochastic state transitions.Source: §2 · pp. 2–3
  • In the fully cooperative setting, agents receive the same reward at each timestep and maintain separate action and message policies based on their experience.Policies may depend on each agent’s trajectory of experience.Source: §2 · p. 3
  • Each agent independently optimizes its own action and message policies for discounted cumulative joint reward, with no inter-agent communication beyond environmental actions and messages.Independent reinforcement learning makes each agent’s problem non-stationary and non-Markov and creates joint exploration and coordination challenges.Source: §2 · p. 3

3 Shaping Losses for Facilitating Communication

The section identifies a communication-channel non-use problem and proposes shaping losses to encourage speakers to vary messages across situations and listeners to respond differently to messages. The biases are intended to get agents using the channel, not to directly produce high-reward communication.

3 Shaping Losses for Facilitating Communication

  • Agents may settle on random messages that listeners ignore, and entropy costs can reinforce this non-use; when meaningful communication begins, agents can find at least a locally optimal communication solution.The difficulty is particularly relevant when the no-communication task has a single deterministic optimal policy.Source: §3 · p. 3
  • The shaping losses are not expected to directly yield high reward because many policies maximizing the biases are unhelpful; their intended role is to start channel use so MARL can find a useful protocol.Positive signalling alone is insufficient for useful communication with a limited channel, and positive listening can encourage unhelpful action conditioning before a good protocol develops.Source: §3 · p. 3

3.1 Bias for positive signalling

  • Positive signalling encourages messages to vary with the speaker’s trajectory while remaining diverse overall, using a target conditional entropy to avoid problematic direct mutual-information optimization.The authors motivate this alternative by noting infinite gradients at mutual-information-maximizing policies and disconnected low-entropy policy regions; the target-entropy loss has finite gradients around its minima and can have a connected set of minimizing policies.Source: §3.1 · pp. 3–4

3.2 Bias for positive listening

  • Positive listening encourages a listener’s actions to depend on the speaker’s messages, extending causal influence of communication across multiple timesteps so actions can reflect all previous messages.The training loss uses the L1 distance between message-conditioned and unconditioned listener policies, which the authors report as more stable than the other divergence considered.Source: §3.2 · pp. 4–5

4 Empirical Analysis

The paper evaluates positive-signalling and positive-listening biases in a one-step digit-summing task and a multi-step Treasure Hunt environment. Both biases improve communication-related task performance, though Treasure Hunt runs often remain at local optima.

4 Empirical Analysis

  • The paper presents Treasure Hunt as a multi-step test with a ceiling for agents that do not use communication, and reports that the biases enable agents to learn communication there.The resulting protocols are analyzed, including interventions to observe their effects on listener behavior.Source: §4 · pp. 5–6

4.1 Summing MNIST digits

  • The one-step task gives the speaker one of 20 messages and the listener one of 19 sum actions, with both agents rewarded when the action matches the digits' sum.Agents are trained independently with REINFORCE; the experiments quantify each agent's potential reward gain from using communication as a proxy for its learning signal.Source: §4.1 · p. 6
  • In digit-summing, either bias leads to greater reward and increased speaker and listener communication power; listening lifts speaker power first, while signalling lifts listener power first.Without a bias, both powers stay low and final reward is 0.1, matching the no-communication outcome.Source: §4.1 · p. 6, Figure 2 · p. 7, Table 1 · p. 7

4.2 Treasure Hunt

  • Treasure Hunt pairs a finder that sees but cannot reach the treasure with a collector that can reach it but must explore to locate it.The optimal solution has the finder communicate the treasure's position to the collector.Source: §4.2 · p. 7
  • In Treasure Hunt, both biases increase reward and make useful protocols more consistently discoverable, but runs still often settle for local optima.Among runs with communication, both biases together correspond to higher average reward and better local optima.Source: §4.2 · p. 8, Table 2 · p. 8
  • An extra speaker reward for influencing the listener produced no significant task-reward improvement over communication without that additional bias in Treasure Hunt.In digit-summing, the extra positive-listening reward likewise provided no gain over the no-bias baseline because it did not make messages vary with the speaker's input.Source: §4.1 · p. 7, §4.2 · p. 8
  • In Treasure Hunt protocol analyses, finder messages correlated with treasure location; a symbol in the median run indicated treasure in the rightmost tunnel about 75% of the time.The best-reward run used a more complex protocol, with symbols correlating with one or more tunnels.Source: §4.2 · pp. 8–9
  • In Treasure Hunt interventions, overriding messages with a constant symbol led the collector to reach the indicated tunnel faster than without an override.Visit times were compared over 100 episodes, with 95% confidence intervals.Source: Table 3 · p. 8, §4.2 · p. 9

5 Conclusion

The paper introduces speaker-side positive-signalling and listener-side positive-listening losses for decentralized communication. In simple and temporally extended environments, the losses improve the consistency of learning useful communication, while broader settings remain open.

5 Conclusion

  • In a simple environment, the two shaping losses ease the other agent’s learning problem and increase consistency in learning useful communication protocols.Positive signalling is applied on the speaker’s side and positive listening on the listener’s side.Source: §5 · p. 9
  • In a temporally extended environment, the losses also increase the consistency with which agents learn to communicate.Source: §5 · p. 9
  • The evidence covers fully cooperative, otherwise simple environments; whether the approach scales to larger domains or helps in non-cooperative settings remains open.The authors note that communication may expose agents to exploitation in settings that are not fully cooperative.Source: §5 · p. 9

A Environment details

Treasure Hunt uses a pixel-map of tunnels in which two agents search for treasure. The layout, limited observations, and shared reward define the environment’s exploration and coordination task.

A Environment details

  • The map is an 18-by-24 grey rectangle with horizontal tunnels on the second rows from the top and bottom, joined by four separated vertical tunnels.The vertical tunnels are randomly selected from positions at least three pixels apart and stop two pixels above the bottom tunnel.Source: A Environment details · p. 12
  • Treasure is placed at the bottom of a random tunnel, with one agent placed randomly in each horizontal tunnel.Source: A Environment details · p. 12
  • Episodes last 500 timesteps; agents have four directional actions and a no-op, see a centered 5-by-5 area, and cannot move through grey walls.When either agent reaches the treasure, both receive reward 1 and the treasure respawns at a random tunnel bottom.Source: A Environment details · p. 12

B Treasure hunt communication protocols

The Treasure Hunt protocols link transmitted symbols to treasure locations and listener actions. Median- and best-reward runs show different message patterns, with listener movement corresponding to the message–tunnel correlations.

B Treasure hunt communication protocols

C.1 MNIST sums experiments

The MNIST sums experiments use an existing classifier architecture and REINFORCE to test communication biases rather than optimize image-classification performance. Hyperparameter sweeps select settings for the reported experiments.

C.1 MNIST sums experiments

  • The experiments use an existing MNIST classifier architecture, which the authors did not optimize because their goal was to study communication biases.Source: §C.1 · p. 13
  • The classifier has two convolutional layers with 32 and 64 channels, followed by max pooling and a 1,024-neuron hidden linear layer.Both convolutional layers use kernel size 5, stride 1, and ReLU activations; the pooling uses stride and kernel size 2.Source: §C.1 · p. 13
  • Agents use REINFORCE with Adam and combine its loss with entropy regularization and positive-signalling and positive-listening losses.For the listener, the received message is concatenated with the flattened convolutional output before the hidden linear layer.Source: §C.1 · p. 13
  • Hyperparameters are selected through sweeps, with five runs per setting and 50 runs for the final settings; no tested no-bias setting improved over the no-communication policy.The no-bias sweep varied action and message entropy bonuses from 0.0 to 0.3.Source: §C.1 · pp. 13–14

C.2 Treasure Hunt experiments

The Treasure Hunt experiments specify the agents’ shared architecture, optimization setup, and hyperparameter sweeps across five settings. The final hyperparameters are summarized in Table 9.

C.2 Treasure Hunt experiments

  • Each agent uses a convolutional layer, a two-layer MLP, an LSTM, and output layers for action and message policies and the value function.The other agent’s message is concatenated with the flattened convolutional output before the hidden linear layer.Source: §C.2 · p. 14
  • RMSProp uses an initial learning rate of 0.001, annealed by 0.99 every million steps, with η = 0.99 and ϵ = 10^-6.Source: §C.2 · p. 14
  • The sweeps vary action entropy costs, MLP and LSTM sizes, positive-listening loss weights, positive-signalling loss weights, and target entropy.The stated candidate values are in the passage; the selected values are not provided here.Source: §C.2 · pp. 14–15
  • The combined-bias setting reuses hyperparameters from the best positive-signalling and positive-listening runs, while social-influence reward weights are swept separately.The social-influence weights are 0.01, 0.03, 0.1, 0.3, and 1.0.Source: §C.2 · p. 15
  • For each sweep, five runs select settings that most often exceed the no-communication baseline, followed by 50 runs with the final settings.Table 9 is identified as the summary of final Treasure Hunt hyperparameters.Source: §C.2 · p. 15, Table 9 · p. 15

D Multi-step CIC ablation

This ablation contrasts multi-timestep CIC, which removes messages from the trajectory, with single-timestep CIC, which removes only the final message. Single-timestep CIC did not improve results over the no-speaker-side-bias baseline.

D Multi-step CIC ablation

  • For multi-timestep CIC, the listener trajectory without messages removes every message while retaining observations and actions.The full trajectory includes observations, actions, and messages; the message-free trajectory is used in the CIC definition.Source: D Multi-step CIC ablation · p. 15
  • Single-timestep CIC removes only the final message, and its action-distribution estimate is obtained by rolling out the policy with that message zeroed and fitting with cross-entropy.The actual observations and actions are retained in the rollout.Source: D Multi-step CIC ablation · p. 15
  • Single-timestep CIC, with or without positive signalling, did not improve results over the baseline without speaker-side bias.Figure 5 presents this comparison.Source: Figure 5 · p. 16, D Multi-step CIC ablation · p. 16

E Statistical methodology

The section reports 95% confidence intervals, with procedures specified for Bernoulli proportions and average-performance graphs.

E Statistical methodology

  • All intervals are 95% confidence intervals; Bernoulli proportions use the Wilson approximation, while performance graphs use means of per-run rewards across 100 training windows.Graph intervals are calculated for the mean reward across runs in each window.Source: E Statistical methodology · p. 16
Loading 1912.05676v1…