PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Curricula and Emergent Implicit Signaling

Niko A. Grupen, Daniel D. Lee, Bart Selman

arXiv:2106.11156v3cs.MAcs.AIcs.LG

TL;DR

Emergent communication research has largely focused on language-like protocols, while implicit communication in emergent agent interactions is less understood. The paper studies implicit signaling in decentralized pursuit-evasion using curriculum-driven learning; its method completes the task at a lower pursuer-speed ratio than baselines and exchanges more implicit information.

  • Problem

    Implicit communication is less well-understood in emergent agent-agent interactions, and studying it under standard definitions entails difficult assumptions about agents’ mental states.

    Source: §1 · p. 2

  • Method

    The approach combines curriculum-driven learning with implicit signals generated by agents’ physical positions and passively observed by teammates.

    Source: §1 · p. 2, §4.2 · p. 6, §6 · p. 8

  • Results

    At a speed ratio of 0.5, the method completes pursuit-evasion while baselines fail at 0.8; it exchanges up to 0.375 bits per time-step on average, versus a baseline maximum of 0.15 bits.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results provide evidence that implicit communication contributes to the success of the curriculum-driven strategy.

    Source: §6 · p. 8

    The pursuit-evasion setup constrains pursuer speed to at most the evader’s speed.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Emergent communication has made strides towards learning communication from scratch, but has focused primarily on protocols that resemble human language. In nature, multi-agent cooperation gives rise to a wide range of communication that varies in structure and complexity. In this work, we recognize the full spectrum of communication that exists in nature and propose studying lower-level communication. Specifically, we study emergent implicit signaling in the context of decentralized multi-agent learning in difficult, sparse reward environments. However, learning to coordinate in such environments is challenging. We propose a curriculum-driven strategy that combines: (i) velocity-based environment shaping, tailored to the skill level of the multi-agent team; and (ii) a behavioral curriculum that helps agents learn successful single-agent behaviors as a precursor to learning multi-agent behaviors. Pursuit-evasion experiments show that our approach learns effective coordination, significantly outperforming sophisticated analytical and learned policies. Our method completes the pursuit-evasion task even when pursuers move at half of the evader's speed, whereas the highest-performing baseline fails at 80% of the evader's speed. Moreover, we examine the use of implicit signals in coordination through position-based social influence. We show that pursuers trained with our strategy exchange more than twice as much information (in bits) than baseline methods, indicating that our method has learned, and relies heavily on, the exchange of implicit signals.

1 INTRODUCTION

The introduction motivates studying implicit signals in decentralized coordination and presents a curriculum-driven approach for difficult, sparse-reward pursuit-evasion. The method outperforms analytical and learned baselines and is associated with greater exchange of implicit information.

1 INTRODUCTION

  • Emergent communication has largely targeted language-like protocols, while implicit communication remains less understood in emergent agent-agent interactions.Implicit signals arise from an agent’s physical position and are passively observed by teammates, without specialized sender-receiver architectures.Source: §1 · pp. 1–2
  • Naive decentralized learning fails in pursuit-evasion because learning is non-stationary, rewards are sparse, and slower pursuers struggle to gain early experience.Agents must learn environmental behavior and teamwork simultaneously, while capture is the only source of reward.Source: §1 · p. 2
  • The approach combines velocity-based environment shaping with a behavioral curriculum that seeds agents with successful single-agent experience before team learning.Pursuer velocity is initially set above the evader’s and then decreased as strategies develop.Source: §1 · p. 2
  • At a speed ratio of 0.5, the method completes pursuit-evasion, while each baseline fails at a speed ratio of 0.8.The ratios compare pursuer speed with evader speed.Source: §1 · p. 3
  • Capture-location distributions indicate that the method learns structured coordination while allowing pursuers to adjust their positions dynamically relative to the evader.The analysis compares rotational symmetry and rotational invariance in capture distributions.Source: §1 · p. 3
  • Pursuers trained with the method exchange up to 0.375 bits per time-step on average, versus a baseline maximum of 0.15 bits.The authors interpret this as evidence that the method learns and relies heavily on implicit signals.Source: §1 · p. 3

2 RELATED WORK

Related work covers language-like emergent communication, implicit interaction, decentralized multi-agent learning, curricula, and pursuit-evasion. The paper positions its approach at their intersection.

2.1 Emergent Communication

  • Emergent communication research includes language-like protocols and complex situated tasks, while implicit interaction is less studied in emergent behavior; the paper bridges this work with MARL.Action-space communication can be non-verbal yet explicit, unlike the fully implicit signals studied here.Source: §2.1 · p. 3

2.2 Multi-Agent Reinforcement Learning

  • Decentralized learning can be unstable because agents learn individually in a non-stationary environment, while centralized methods share training information; the paper prioritizes decentralized learning.Its curriculum combines velocity-based environment shaping with single-agent experience to bootstrap multi-agent learning.Source: §2.2 · p. 3

2.3 Pursuit-Evasion

  • Pursuit-evasion with equally fast or faster evaders highlights the need for coordinated motion and communication, and the paper studies learned strategies in this setting.Prior work includes control strategies derived from value functions and strategies learned through MARL.Source: §2.3 · p. 4

3 PRELIMINARIES

The preliminaries define decentralized multi-agent learning and the pursuit-evasion setting, then establish how implicit positional signals and instantaneous coordination are understood and measured.

3.1 Markov Game

  • A Markov game represents multi-agent decisions through joint action and observation spaces, state transitions, and a reward function.Each agent seeks to maximize its total reward over time.Source: §3.1 · p. 4

3.2 Deep Deterministic Policy Gradients

  • DDPG is an off-policy actor-critic method for continuous actions, and agents perform its updates individually in a decentralized manner.The actor uses deterministic policy gradients; the critic minimizes a loss from replay-buffer transition tuples.Source: §3.2 · p. 4

3.3 Potential Field Navigation

  • Potential-field navigation turns the negative gradient of a potential between an agent and its goal into a motion control law.Because actions are headings, only the force direction affects agent motion.Source: §3.3 · p. 4

3.4 Pursuit-evasion

  • Pursuit-evasion pits n pursuers against one evader, with the pursuers rewarded for capture and the evader's potential encouraging escape from pursuers.The evader's objective favors the largest bisector between pursuers when possible; pursuer speed is constrained to be no greater than evader speed.Source: §3.4 · p. 4

3.5 Implicit Communication

  • For these RL agents, an implicit signal is positional information teammates observe in the state or observation, unlike a message sent through a dedicated channel.The adopted distinction between implicit and explicit communication turns on deliberate information sharing.Source: §3.5 · pp. 4–5

3.6 Instantaneous Coordination

  • Instantaneous Coordination measures mutual information between one agent's action and another's next-step action, serving as a measure of position-based social influence.Without explicit communicative actions, an agent's action determines its next position, which teammates observe; test-time estimates use Monte Carlo samples.Source: §3.6 · p. 5

4 METHOD

The method addresses sparse-reward, decentralized learning by pairing a velocity-based environment curriculum with a behavioral curriculum. Together, these stages shape task difficulty and give agents successful single-agent experience before team coordination.

4 METHOD

  • CD-DDPG combines a curriculum over agent velocities with a behavioral curriculum that bootstraps cooperative learning from successful single-agent experience.The method targets the sparse rewards and instability that make decentralized pursuit-evasion learning difficult.Source: §4 · p. 5

4.1 Velocity Ratio Curriculum

  • The velocity curriculum gradually lowers the pursuer-to-evader speed ratio, creating increasingly difficult pursuit-evasion environments.The initial ratio can exceed 1.0, giving pursuers an early speed advantage before the ratio decays.Source: §4.1 · p. 5

4.2 Behavioral Curriculum

  • Only CD-DDPG successfully learns team coordination in the early-training ablation as the decreasing speed ratio makes the task harder.The figure characterizes learning by sustained reward rather than improved reward.Source: Figure 2 · p. 6
  • The behavioral curriculum first uses a supervisory policy to collect successful single-agent experience, then switches to standard exploration so agents can learn coordination.Without this curriculum, agents must learn environmental interaction and teamwork simultaneously.Source: §4.2 · p. 6

5 RESULTS

The experiments test CD-DDPG’s curricula in decentralized pursuit-evasion and examine whether implicit signals emerge during coordination. CD-DDPG captures at lower pursuer speeds than the comparison strategies, while its pursuers show structured but adjustable positioning and stronger measured influence.

5.1 Multi-Agent Curricula

  • Using both curricula is crucial: CD-DDPG learns from early capture experience, while removing either curriculum fails to gain consistent traction.No Curriculum never receives positive reward; No Behavioral produces inconsistent reward spikes, and No Velocity performs worse than No Curriculum.Source: §5.1 · p. 6

5.2 Properties of Emergent Teamwork

  • CD-DDPG is evaluated against strategies spanning independent pursuit, static-role coordination, and position-responsive pincer coordination.The evaluation treats outperforming a strategy as surpassing its level of coordination, implicit-communication capacity, or both.Source: §5.2 · pp. 6–7, Table 1 · p. 6

5.2.1 Capture Success.

  • CD-DDPG succeeds at a pursuer-to-evader speed ratio of 0.5, while competing methods fail below 0.8.Figure 3 reports capture success as a function of velocity.Source: Figure 3 · p. 7, §5.2.1 · p. 7

5.2.3 Position-based Social influence.

  • CD-DDPG pursuers exchange up to 0.375 bits of information per time-step on average, compared with a baseline maximum of 0.15 bits.The measured IC increases as the pursuer-to-evader speed ratio decreases, while competing methods remain stagnant.Source: Figure 5 · p. 8, §5.2.3 · p. 8

5.2.2 Relative position during capture.

  • CD-DDPG combines structured coordination with dynamic adjustments to pursuers’ relative angles, unlike the fixed roles of CD-DDPG (Partial).Greedy and Pincer yield uniform capture distributions, while CD-DDPG (Partial) shows strong role assignment; Figure 4 displays relative-angle distributions as heatmaps, one row per strategy.Source: §5.2.2 · pp. 7–8, Figure 4 · p. 7

5.2.4 High-influence moments.

  • At speed ratios of 0.7 or lower, CD-DDPG’s high-influence moments rise to nearly 30%, against 10–15% across most methods and ratios.A high-influence moment is a timestep when IC exceeds the mean IC for the trajectory.Source: §5.2.4 · p. 8, Figure 6 · p. 8

6 CONCLUSION

The paper frames implicit-to-explicit communication as a spectrum and proposes curriculum-driven learning for difficult multi-agent environments. Its toroidal pursuit-evasion setting supports repeated pursuit without strict initialization conditions or special rewards.

6 CONCLUSION

  • The curriculum-driven strategy enables pursuers to coordinate and capture a superior evader, with evidence suggesting implicit communication contributes to its success.The work also connects emergent communication to the spectrum from implicit to explicit communication.Source: §6 · p. 8

A.1 Pursuit-Evasion Environment

  • When pursuers are faster than the evader, straight-line chasing solves the planar game without requiring coordination.When pursuers are no faster, the evader has a significant advantage and capture usually requires strict initialization conditions.Source: §A.1 · p. 10
  • Toroidal pursuit-evasion permits random initialization and repeated pursuit without strict initialization conditions or special rewards.Periodic boundary conditions prevent the evader from running away indefinitely when pursuers are no faster.Source: §A.1 · p. 10

A.2.1 Derivation.

The evader’s strategy selects a heading by optimizing a potential based on the relative positions of the pursuers. Distance weighting lets it adjust its bisector choice when formations are symmetric.

A.2.1 Derivation.

  • The evader chooses a heading by optimizing its objective over pursuers’ relative polar coordinates.The objective uses pursuer angles and distances to select an escape direction.Source: §A.2.1 · p. 10
  • The unit-test examples show the evader selecting a heading that minimizes its cost for specified pursuer angles.With pursuers at 0, π/2, and π, the selected heading is −π/2; with angles 0, π/2, and −π/2, either −π or π is selected.Source: Figure 8 · p. 10
  • Distance-modulating the cost lets the evader adjust its bisector based on pursuer distances, helping when it is caught in symmetric formations.The strategy follows the negative gradient of the potential at maximum speed.Source: §A.2.1 · p. 10

A.2.2 Unit Tests.

Unit tests illustrate how the evader’s objective directs it toward the largest opening between pursuers. The cosine-based objective favors headings that split the largest bisector.

A.2.2 Unit Tests.

  • With three pursuers at relative angles 0, π/2, and π, the cost-minimizing evader heading is −π/2.The tests set all pursuer distances to 1 to remove radius modulation.Source: §A.2.2 · p. 10
  • With pursuers at 0, π/2, and −π/2, either −π or π minimizes cost and moves the evader toward the largest opening.The two headings represent equivalent directions around the circle.Source: §A.2.2 · p. 11
  • Summing the cosine terms incentivizes the evader to head through the largest bisector between pursuers.The objective oscillates between −1 and 1 over a period of π.Source: §A.2.2 · p. 11

A.3 Pincer Pursuit Explained

Pincer selects active pursuers through discrete optimization over replicated torus states. The search tends to favor symmetric triangular formations, but its computational approach does not scale well with the number of pursuers or replications.

A.3 Pincer Pursuit Explained

  • The discrete Pincer search selects active pursuers by maximizing the objective across replicated torus states.Each pursuer is restricted to selecting among replications of itself, reducing the search space; experiments use k = 1.Source: §A.3 · p. 11
  • The maximizing pursuer set tends to favor symmetric triangular formations.Figure 9 illustrates the replicated environment for k = 1.Source: §A.3 · p. 11
  • The discrete optimization approach does not scale well with the number of pursuers or the torus-unrolling parameter.The authors report finding a sufficient maximizer with low k values, using k = 1 in their experiments.Source: §A.3 · p. 11

B QUALITATIVE RESULTS

CD-DDPG pursuers appear to adapt their positions to one another as they approach the evader, sometimes moving away from it to preserve group formation. The authors interpret this as encouraging evidence of implicit communication.

B.1 Animalistic Coordination

  • Pursuers appear to adjust their positions in response to fellow pursuers’ movements while closing in on the evader.The authors report this from post-hoc qualitative analysis of CD-DDPG trajectories.Source: §B.1 · p. 11
  • Pursuers sometimes move away from the evader to maintain the integrity of their group formation.The authors contrast this with what a less coordinated strategy would do.Source: §B.1 · pp. 11–12
  • The authors interpret these behaviors as encouraging evidence that implicit communication has emerged among CD-DDPG pursuers.They say this explains the performance difference from potential-field strategies, which lack a basis for small-scale adaptive movements.Source: §B.1 · p. 12

B.2 Phantom Coordination

Greedy pursuers can produce temporary formations that raise the IC score despite acting independently, because the measure averages across trajectories. The appendix also specifies the potential-field and policy-learning hyperparameters.

B.2 Phantom Coordination

  • At pursuer-to-evader speed ratio 0.4, greedy pursuers form temporary triangular patterns around an evader they cannot capture, producing apparent coordination despite independent actions.The evader zig-zags within the formation until an opening allows escape; the authors call the IC effect phantom coordination.Source: §B.2 · p. 12, Figure 11 · p. 12
  • The IC score can rise for greedy pursuers because it averages independent actions together with trajectory segments that appear highly coordinated.The measure is computed between consecutive time-steps and averaged over whole trajectories.Source: §B.2 · p. 12

C.1 Potential-Field Hyperparameters

  • The Greedy potential-field uses an attractive-pull strength of k_att = 1.5 in all experiments.Source: §C.1 · p. 12

C.2 Policy Learning Hyperparameters

  • Actors use two 128-unit hidden layers and critics three; actor and critic learning rates are 1e−4 and 1e−3, respectively.Training also uses gradient clipping at 0.5, Polyak averaging with τ = 0.001, a 500000-entry buffer, batches of 512, and γ = 0.99.Source: §C.2 · p. 12

C.3 Experiments

Experiments use a staged training and evaluation schedule, and the supplementary assets include the new toroidal pursuit-evasion environment. The environment extends Lowe et al.’s open-sourced pursuit-evasion environment [44].

C.3 Experiments

  • Models train for 50000 epochs of 500 steps, with 1000 warm-up epochs and pursuer velocity decayed over 15000 epochs.Test performance averages 100 independent trajectories; the Figure 2 ablation uses 7500 epochs and 10 random seeds.Source: §C.3 · p. 12

C.4 Assets

  • The new toroidal pursuit-evasion environment is included in the supplementary material and extends the environment introduced by Lowe et al. [44].The original environment is open-sourced under the MIT license.Source: §C.4 · p. 12
Loading 2106.11156v3…