Source-linked AI summary
Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, Nando de Freitas
TL;DR
MARL coordination and meaningful emergent communication remain difficult, and some prior intrinsic social-motivation approaches rely on environment-specific rewards or access to others’ rewards. This paper rewards agents for counterfactual causal influence on peers’ actions, relating the incentive to mutual information. Across SSD experiments, influence improves collective return and communication, and can be computed independently using learned models of other agents.
Problem
MARL coordination remains difficult, and meaningful emergent communication has been challenging even with centralized training; some prior intrinsic social-motivation methods also require environment-specific rewards or access to others’ rewards.
Source: §1 · p. 1Method
Results
Across three SSD experiments, influence consistently yields higher collective return than powerful agents such as A3C and improved baselines, while experiments also demonstrate improved communication and independent reward computation.
Source: §9 · pp. 8–9Takeaways & Limitations
In the studied SSDs, influence rewards support coordination and communication without requiring a centralized controller or access to another agent’s reward function.
Source: §9 · p. 9Harvest’s collective reward can be spuriously high when an agent fails to collect apples, so those random seeds are excluded from reported results.
Source: §5.4 · p. 23
Abstract
from arXiv · showhide
We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information between their actions. Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols. The influence rewards for all agents can be computed in a decentralized way by enabling agents to learn a model of other agents using deep neural networks. In contrast, key previous works on emergent communication in the MARL setting were unable to learn diverse policies in a decentralized manner and had to resort to centralized training. Consequently, the influence reward opens up a window of new opportunities for research in this area.
1. Introduction
The introduction frames coordination and meaningful communication as persistent MARL challenges, and proposes causal influence as intrinsic social motivation for addressing them.
1. Introduction
- Prior MARL work often relied on centralized training for coordination, while meaningful emergent communication remained difficult even with centralized training.Earlier intrinsic social-motivation approaches also relied on environment-specific rewards or access to other agents’ rewards.Source: §1 · p. 1
- The paper rewards agents for causal influence on peers’ actions, assessed by counterfactual changes in their behavior and related to mutual information between actions.The authors hypothesize that this incentive will promote coordinated behavior.Source: §1 · p. 1
- Across three experiments, influence supports coordination; results include higher collective reward, more effective communication, and independent training using learned models of other agents.The model predicts other agents’ actions so each agent can estimate its influence reward without a centralized controller or access to another agent’s reward function.Source: §1 · p. 2
2. Sequential Social Dilemmas
- The study uses Cleanup and Harvest, two games with limited apple resources that require agents to coordinate harvesting for cooperation.The paper identifies them as a public-goods game and a public-pool resource game, respectively.Source: §2 · p. 2
3. Multi-Agent RL for SSDs
- Agents independently maximize individual rewards without shared weights, in a partially observable multi-agent setting.The supplied setting description also says agents see only part of the true state.Source: §3 · pp. 2–3
4. Basic Social Influence
The basic influence reward measures how counterfactual changes to one agent’s action alter another agent’s planned behavior. In Experiment I, influence improves collective reward and can produce action-based communication.
4. Basic Social Influence
- The influence reward measures the discrepancy between another agent’s marginal policy and its policy conditioned on the influencer’s action.The marginal policy is obtained by sampling counterfactual actions; the discrepancy captures how much the other agent changes its planned action distribution because of the influencer.Source: §4 · p. 3
- In expectation, influence rewards incentivize mutual information between agents’ actions, which the authors connect to more coordinated behavior.The reward is a Monte Carlo estimate of mutual information across independently sampled trajectories.Source: §4 · p. 3
- Explicit dependencies across agents’ actions can reduce policy-gradient variance by lowering the conditional variance of an agent’s gradients.The passage presents this as a potential benefit for convergence in larger MARL tasks.Source: §4 · p. 3
- The basic formulation assumes centralized training to compute influence and unidirectional influence between disjoint groups; later sections relax both assumptions.Source: §4 · p. 3
4.1. Experiment I: Basic Influence
- Influence significantly outperforms baseline and ablated agents in both games, and is essential for meaningful learning in Harvest.The experiments measure total collective reward; the authors infer that higher collective reward in these SSDs reflects more effective cooperation.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
- In a Cleanup example, an influencer’s movement toward an unseen apple signals its presence and changes another agent’s planned-action distribution.The influencer learned to use its actions as a binary code for apple presence or absence, yielding emergent communication in Cleanup and similarly in Harvest.Source: §4.1 · p. 4
- Whether influence produces cooperation may depend on the environment and task, and on tuning the trade-off between environmental and influence rewards.The authors note that influence could also be used non-cooperatively, although it increased cooperation in these experiments.Source: §4.1 · p. 4
5. Influential Communication
The section trains an explicit communication policy with social influence rewards, then tests whether those rewards produce useful communication and coordination. Influence-trained agents achieve higher collective reward, communicate more consistently, and show stronger coordination during influential moments, with results qualified by repeated interaction.
5. Influential Communication
- Each agent emits a discrete symbol, and an added policy head learns what to send using influence reward alongside environmental reward.The reward is r=αe+βc; messages are supplied to other agents at the next timestep.Source: §5 · p. 4, Figure 3 · p. 5
5.1. Experiment II: Influential Communication
- Influence-trained communication agents learn faster and achieve significantly higher collective reward for most of training in both Cleanup and Harvest.In Cleanup, the optimal setting used α=0, suggesting influence alone can suffice to train the communication policy.Source: §5.1 · p. 5, Figure 4 · p. 5
- Influence agents communicate more unambiguously about their actions, while high communication coordination appears only during influential moments.Influence exceeded its mean in less than 10% of timesteps; baseline agents showed almost no communication coordination.Source: §5.1 · pp. 5–6, Figure 5 · p. 5
- Agents more often influenced by communication also earn higher individual task reward in both games: Cleanup ρ=.67, p<0.001, and Harvest ρ = .34, p<0.001.The authors interpret this correlation as evidence that messages transmit information useful to listeners.Source: §5.1 · p. 6
- The communication findings may depend on repeated interactions, where unreliable messages can cost speakers influence over time; this may not hold in one-shot interactions.The authors report empirical evidence for meaningful protocols within the repeated-interaction setting.Source: §5.1 · p. 6
6. Modeling Other Agents
The section replaces centralized access to other agents’ policies with an internal Model of Other Agents (MOA) that predicts their actions. In the experiments, influence rewards computed with the MOA improve collective reward, enabling independent social learning without a centralized controller.
6. Modeling Other Agents
- The MOA predicts other agents’ next actions from their previous actions and the agent’s egocentric state view, using observed trajectories and cross-entropy loss.Its predictions support counterfactual estimates of how alternate actions would affect others.Source: §6 · p. 6, Figure 6 · p. 6
- Because MOA predictions implicitly model other agents’ internal states and environment transitions, influence rewards are given only when the target agent is in view.The authors state that estimates are more accurate when the other agent is visible.Source: §6 · p. 6
6.1. Experiment III: Modeling Other Agents
- The MOA lets agents compute influence rewards internally and learn socially but independently, using observed actions without a centralized controller or access to others’ rewards.The comparison framework includes a version where the MOA is an auxiliary task rather than the source of reward.Source: §6 · p. 6, §6.1 · p. 7
- Influence agents using an internal MOA achieve higher collective reward than the A3C baseline and exceed prior state-of-the-art returns of 275 for Cleanup and 750 for Harvest.The auxiliary-task MOA improves over A3C, but the influence model performs consistently better; A3C baselines are described as unable to learn.Source: §6.1 · p. 7, Figure 7 · p. 7
7. Related work
Related work addresses social rewards, emergent communication, modeling other agents, and mutual-information-based motivations. The paper distinguishes its counterfactual influence reward from approaches requiring known goals, analytical computation, or centralized control.
7. Related work
- Prior social-reward approaches used hand-crafted fairness penalties or cooperativeness-based rewards, while classifying actions as cooperative or not was deemed unsuitable for complex, long-term SSD strategies.Other work optimized collective reward through access to others’ rewards, an assumption the authors call potentially unrealistic.Source: §7 · p. 7
- The paper notes that intrinsic motivation need not mean internal motivation; other people can be intrinsically motivating.Source: §7 · p. 7
- Emergent communication studies report that selfish agents often fail to use ungrounded cheap-talk channels effectively, whereas prosocial, curious, or hand-crafted motivations can support communication.The authors position social influence as another way to encourage communication in complex environments.Source: §7 · p. 7
- Machine theory-of-mind work predicts agents’ actions and can model false beliefs, while LOLA models how an agent’s policy affects others’ parameter updates.Source: §7 · p. 7
- Prior counterfactual methods use centralized controllers for advantage estimation or reward shaping, while convergence cross mapping estimates of causality can degrade under stochastic effects.Source: §7 · pp. 7–8
- The proposed reward is linked to empowerment through mutual information between agents’ actions, but differs from approaches requiring a known categorical goal or analytical computation in simple grid worlds.Transfer empowerment was reported to keep agents near players and did not use reinforcement learning to model agents or environments.Source: §7 · p. 8
8. Details on Causal Inference
The section defines causal influence as a counterfactual effect of one agent’s action on another’s action under a particular observed situation. Correct conditioning is needed to remove confounds, with different causal diagrams for same-timestep and cross-timestep influence.
8. Details on Causal Inference
- Counterfactual influence asks how another agent’s action would change if the influencer had acted differently while observed variables remained fixed.The reward conditions on the other agent’s partial state view and internal LSTM state to account for prior trajectory dependence.Source: §8 · p. 8
- In the basic same-timestep case, the influencer must act before the influencee, so influence is one-way; with more agents, influencer and influencee sets are disjoint.Source: §8 · p. 8, Figure 8 · p. 8
- Cross-timestep influence allows each agent to influence every other agent, and the MOA diagram blocks a backdoor path through the state at the current timestep.The MOA must model the environment transition and aspects of the other agent’s internal LSTM state to predict its next action.Source: §8 · p. 8
- The counterfactual calculation requires no abduction because the relevant models, inputs, and trajectory variable values are available for conditioning.Source: §8 · p. 8
9. Conclusions and Future Work
Across three experiments, social influence consistently improves collective outcomes, including communication, and can be computed without centralized control or access to other agents’ rewards. The paper proposes counterfactual reasoning as a basis for further applications, including cooperative robotics and networked systems.
Conclusions
- The influence models achieve higher collective return across varied tasks and setups, outperforming A3C and improved baselines; in some cases, influence is essential for learning.The authors report surpassing state-of-the-art performance on the studied SSDs without requiring access to other agents’ rewards.Source: §9 · pp. 8–9
- Influence improves explicit communication and can yield emergent protocols; agents can compute the reward independently using an internal model of other agents, without a centralized controller or access to their rewards.The paper reports that the influence reward can lead to emergent communication and improve communication when agents use an explicit channel.Source: §9 · p. 9
Future Work
- Counterfactual reasoning could support robot cooperation and serve as a regularizer encouraging information integration between network modules.The authors also suggest using counterfactuals to develop a form of empathy by simulating effects on another agent’s value function.Source: §9 · p. 9
10. Supplementary Material
The supplementary material formalizes influence as mutual information and reports experiments in sequential social dilemmas, communication, and decentralized modeling. Across its performance analyses, influence models achieve stronger collective outcomes, including when outcomes are weighted by equality and across hyperparameter settings.
10.1. Influence as Mutual Information
- In expectation, the social influence reward is mutual information between agents’ actions; the paper uses a directed action-specific causal effect for reward and reports robustness to divergence-measure choice.The alternatives tested include pointwise mutual information and Jensen-Shannon divergence.Source: §10.1 · pp. 11–12
10.2. Sequential Social Dilemmas
- Cleanup requires coordinating river cleaning and apple harvesting, while Harvest requires spatially distributed, sustainable harvesting; both are social dilemmas in which cooperation benefits agents.The tasks are challenging because individual incentives can favor defection even when cooperation benefits the group.Source: §10.2 · p. 12
10.3. Additional experiment - Box Trapped
- In Box Trapped, influence-trained agents open the box in 88% of trajectories, whereas converged vanilla A3C agents do so in 0%; opening produces a corresponding influence-reward spike.The purple agent’s opening action releases the trapped agent, enabling both to collect apples.Source: §10.3 · p. 12
10.4. Implementation details
- The supplementary experiments tune influence, communication, and Model of Other Agents settings, while implementation uses recurrent neural-network policies and random searches over relevant hyperparameters.The influence experiment varies influencer count, reward weight, curriculum steps, and—in its centralized setting—whether the influenced agent also receives reward.Source: §10.4 · p. 12, §10.4.1 · p. 12, §10.4.2 · p. 13
10.4.4. COMMUNICATION ANALYSIS
- A speaker-consistency metric measures how closely a speaker’s actions correspond one-to-one with its communication messages.The metric is defined using entropy relative to the maximum entropy over discrete symbols or actions.Source: §10.4.4 · pp. 14–15
10.5.1. BASIC INFLUENCE EMERGENT COMMUNICATION
- In Cleanup, an agent’s movement toward an apple patch can signal to an agent unable to see the patch whether apples have appeared.The example describes a high-influence moment in which the purple agent’s choice not to move toward the patch informs the magenta agent.Source: §10.5.1 · p. 15
10.5.2. OPTIMIZING FOR COLLECTIVE REWARD
- Collective-reward training attains higher collective reward but can leave some Cleanup agents with no reward, while influence models retain an advantage across the top five hyperparameter settings.The authors caution that collective reward alone may not indicate cooperative behavior: in Harvest, a non-learning agent can inflate the collective score, so such seeds are excluded.Source: §10.5.2 · p. 15, §10.5.3 · p. 15, §10.5.4 · p. 16
10.5.5. PERFORMANCE COMPARISON BETWEEN MODELS AND RELATED WORK
- Influence models attain the highest final performance on the equality-weighted collective reward, and Table 4 reports that they sometimes outperform prior inequity-aversion results despite not using other agents’ rewards.The equality-weighted measure multiplies total collective reward by 1 − G, where G measures inequality in individual returns.Source: Table 4 · p. 16, §10.5.5 · p. 16, §10.5.3 · p. 16
1. Influence as Mutual Information
The paper defines social influence through the change in another agent’s action distribution associated with an agent’s action and connects its expected reward to mutual information. It uses the directed action-specific measure for reward and reports that influence is robust to the choice of divergence measure.
Influence as Mutual Information
- In expectation, the social influence reward is the mutual information between agents’ actions.The paper approximates mutual information by sampling independent trajectories.Source: §1 · p. 19
- The causal influence reward measures how much agent j’s planned action distribution changes given agent k’s action, relative to j’s marginal policy.The paper describes the reward as the directed causal effect of the specific action taken by agent k.Source: §1 · p. 19
- The influence reward is reported to be robust to the choice of measure, including pointwise mutual information and Jensen-Shannon divergence.Pointwise mutual information measures the influence of one action by agent k on one action by agent j; its expectation is mutual information.Source: §1 · p. 19
2. Sequential Social Dilemmas
Cleanup and Harvest are sequential social dilemmas in which limited apples create competing individual and collective incentives. Their tasks require coordination and cooperation to sustain resource growth and harvest.
Sequential Social Dilemmas
- In both games, agents earn +1 per apple collected, but apples are limited; firing a fining beam costs −1 and penalizes a hit agent by −50.The environments are Cleanup, a public-goods game, and Harvest, a common-pool-resource game.Source: §2 · p. 19
- Cleanup requires cleaning a river before apples grow and coordinating cleaning with harvesting, while Harvest requires distributed harvesting to prevent apples from being depleted faster than they regrow.In Cleanup, agents cannot harvest while cleaning; in Harvest, apple respawn depends on nearby apples.Source: §2 · pp. 19–20
- The Schelling diagrams show that agents benefit when more cooperate, although an individual agent is motivated to defect.Even exploited agents receive higher rewards when more agents cooperate than when all defect.Source: §2 · p. 20, Figure 2 · p. 20
3. Additional experiment - Box Trapped
The Box Trapped experiment tests whether social influence rewards encourage an agent to release a trapped partner. Unlike vanilla A3C, the influence-trained agent often opens the box, and doing so produces a spike in its influence reward.
3. Additional experiment - Box Trapped
- The purple agent can either open the box to release the trapped teal agent or consume inexhaustible apples outside it.This proof-of-concept environment isolates the choice between releasing a partner and taking an individual reward.Source: §3 · p. 20, Figure 3 · p. 20
- The influence-trained agent opens the box in 88% of trajectories, while vanilla A3C does so in 0% after convergence.The influence agent usually opens it within the first few steps, giving its partner more time to collect apples.Source: §3 · p. 20
- Opening the box at the second timestep coincides with a spike in influence reward, rewarding an action that changes the other agent’s behavior.The figure tracks the influence reward across a trajectory; its spike follows the opening action.Source: §3 · pp. 20–21, Figure 5 · p. 21
4. Implementation details
The implementation uses recurrent neural network policies and searches shared and experiment-specific hyperparameters. The section also describes settings for basic influence, communication, MOA training, and a speaker-consistency metric.
4. Implementation details
- Models use a convolutional layer, two fully connected layers, and a 128-cell LSTM, with five agents and discount factor γ = .99.The convolution uses a size-3 kernel, stride 1, and six output channels; each fully connected layer has size 32.Source: §4 · p. 21
4.1. Basic influence hyperparameters
- The basic influence experiment varies the number of influencers, influence weight β, curriculum steps C, and—in centralized training—whether influencees also receive reward.The influencee reward is omitted from the other experiments because it precludes independent training; the best settings are listed in Table 1.Source: §4.1 · p. 21, Table 1 · p. 22
4.2. Communication hyperparameters
- Communication models tune symbol count, communication embedding size, communication-loss weight, and whether the extrinsic-reward weight α is zero.An additional entropy regularizer is applied to the communication output head; best settings are listed in Table 2.Source: §4.2 · p. 21, Table 2 · p. 22
4.3. Model of other agents (MOA) hyperparameters
- MOA hyperparameters vary whether training uses only visible agents’ actions and the supervised-loss weight.The best settings are listed in Table 3.Source: §4.3 · p. 21
4.4. Communication analysis
- Speaker consistency measures how much a speaker’s action corresponds one-to-one with its communication message, using entropy and maximum entropy over available symbols or actions.The maximum entropy depends on the number of discrete symbols or actions.Source: §4.4 · pp. 21–22
5. Additional results
Across the additional results, influence agents show strong collective-reward performance, including under an equality-weighted metric and across hyperparameter settings. The experiments also show a signaling example and a trade-off in directly optimizing collective reward.
5.1. Basic influence emergent communication
- In Cleanup, an influencer’s movement—or decision to stay still—can signal whether apples are available to an agent that cannot see the patch.The influencer can observe the patch while the influencee cannot; the influencer’s behavior changes the influencee’s planned action distribution.Source: §5.1 · p. 22
5.2. Optimizing for collective reward
- Directly optimizing collective reward raises collective returns in Cleanup and Harvest, but Cleanup solutions above 20 collective-return points can leave some agents with no reward.The optimal collective-reward weight is η = 0.85 in both games, while equality in individual returns is extremely low.Source: §5.2 · pp. 22–23, Figure 7 · p. 23
5.4. Collective reward and equality
- Collective reward can misrepresent cooperation in Harvest when an agent fails to learn, so those random seeds are excluded from the reported results.The resulting high collective reward can arise because one fewer agent exploits the others, making sustainable harvesting easier.Source: §5.4 · p. 23
- Weighting collective reward by 1 − G, where G measures inequality in individual returns, still leaves influence models with the highest final performance.Figure 8 reports collective reward times equality across experiments.Source: §5.4 · p. 23, Figure 8 · p. 25
5.3. Performance comparison between models and related work
- In several cases, influence agents match or exceed prior inequity-aversion results while requiring access to other agents’ actions rather than their rewards.Table 4 reports final collective reward over the last 50 agent steps; its bolded entries mark significant outperformance of prior scores.Source: §5.3 · p. 23, Table 4 · p. 24
5.5. Collective reward over multiple hyperparameters
- Influence models retain higher collective reward across the top five hyperparameter settings and five seeds, providing evidence of robustness to hyperparameter choice.Figure 9 compares the top settings for each experiment.Source: §5.5 · pp. 23–24, Figure 9 · p. 26