Source-linked AI summary
Emergent Social Learning via Multi-agent Reinforcement Learning
Kamal Ndousse, Douglas Eck, Sergey Levine, Natasha Jaques
TL;DR
This paper asks whether independent RL agents can learn from other agents’ cues, a way to acquire skills and adapt beyond individual exploration. It combines environment design with a model-based auxiliary loss, finding that social learners can learn complex skills and adapt to novel environments in ways model-free RL and imitation learning do not.
Problem
The paper asks whether independent RL agents can learn to use other agents’ cues, since imitation learning relies on curated demonstrations and cannot adapt online to novel tasks.
Source: §1 · p. 1Method
The authors design training environments to elicit social learning and add a model-based auxiliary loss to help agents learn from other agents’ trajectories.
Source: §5 · p. 8Results
Takeaways & Limitations
Mixing social and solitary training can yield individual skills that exceed those of solitary novices and remain useful when experts are absent.
Source: §5 · p. 9The experiments focus on exploratory navigation, and the paper does not thoroughly explore strategies for augmenting solitary training with social experience.
Source: §5.1 · p. 9
Abstract
from arXiv · showhide
Social learning is a key component of human and animal intelligence. By taking cues from the behavior of experts in their environment, social learners can acquire sophisticated behavior and rapidly adapt to new circumstances. This paper investigates whether independent reinforcement learning (RL) agents in a multi-agent environment can learn to use social learning to improve their performance. We find that in most circumstances, vanilla model-free RL agents do not use social learning. We analyze the reasons for this deficiency, and show that by imposing constraints on the training environment and introducing a model-based auxiliary loss we are able to obtain generalized social learning policies which enable agents to: i) discover complex skills that are not learned from single-agent training, and ii) adapt online to novel environments by taking cues from experts present in the new environment. In contrast, agents trained with model-free RL or imitation learning generalize poorly and do not succeed in the transfer tasks. By mixing multi-agent and solo training, we can obtain agents that use social learning to gain skills that they can deploy when alone, even out-performing agents trained alone from the start.
1. Introduction
The paper asks whether independent RL agents can learn to use other agents’ cues, and finds that social learning supports complex skills and adaptation to novel environments. It argues that both environment and algorithm design matter, while vanilla model-free RL does not use social learning to improve performance.
1. Introduction
- Social learning can help people and animals acquire useful behaviors that individual exploration would make difficult to discover, and adapt to new circumstances.The paper contrasts this with imitation learning’s dependence on curated demonstrations and inability to adapt online to a novel task.Source: §1 · p. 1
- The study tests independent agents pursuing their own rewards, without direct incentives to teach or learn, while experts may be visible in partial observations.Source: §1 · pp. 1–2
- Vanilla model-free RL agents fail to use social learning to improve performance, motivating environment and architecture designs intended to facilitate it.Source: §1 · p. 2
- When social learning emerges, agents learn more complex behavior than agents trained alone and even the experts themselves.Source: §1 · p. 2
- Interleaving training with experts and training alone lets agents use learned skills when alone, outperforming agents trained only solo.The agents selectively follow experts only when doing so is rewarding, unlike pure imitation learning.Source: §1 · p. 2
- Social learners use cues from new experts to adapt online in novel environments, outperforming agents trained alone, agents without social learning, and imitation learning.The paper reports high zero-shot transfer performance in these settings.Source: §1 · p. 2
2. Related Work
The paper distinguishes its approach from imitation and inverse reinforcement learning by learning in a shared multi-agent environment without curated demonstrations or privileged expert information. It also contrasts its transfer results with prior work and relates its auxiliary prediction loss to other RL methods.
2. Related Work
- Imitation learning trains agents to approximate curated expert trajectories, while the paper’s setting allows learning from experts encountered in the environment.The paper notes that imitation learning can be brittle and cites variants that combine it with RL or multiple experts.Source: §2 · p. 2
- Unlike imitation learning, the agents coexist with experts of varying skill without curated trajectories, privileged access, or an obligation to copy expert policies.Agents may eventually outperform the experts with which they learned.Source: §2 · p. 2
- Unlike inverse RL, the agents share an environment with experts and a consistent reward function, so they need not infer rewards from expert trajectories.The novices and experts do not share rewards: an expert’s reward does not benefit the novice.Source: §2 · p. 2
- Prior autonomous-driving work used expert drivers for online learning but assumed privileged information about other cars’ states or policies.Sun et al. (2019) similarly use other cars to update agents’ beliefs about occluded pedestrians.Source: §2 · p. 3
- Unlike Borsa et al. (2019), this work reports transfer to unseen environments with new experts and better generalization than solo learners.Borsa et al. (2019) found novices outperformed solitary learners only when experts had hidden information; here experts have no privileged environmental information.Source: §2 · p. 3
- The model-based auxiliary loss predicts next states to help agents learn transition dynamics from trajectories that yielded no reward.Prior auxiliary-loss and multi-agent prediction work did not study social learning, transfer, or generalization, and one such approach assumed privileged access to other agents’ actions.Source: §2 · p. 3
3. Learning Social Learning
The paper analyzes why independent agents struggle to learn from experts and proposes predictive auxiliary learning and environments designed to encourage social learning. It also describes Goal Cycle and transfer settings for studying these abilities.
3. Learning Social Learning
- Agents act simultaneously in partially observed environments, maximize individual rewards, and are trained independently without sharing parameters or directly observing others’ observations or actions.The shared state and reward function are common across agents, but each agent receives its own reward.Source: §3 · p. 3
3.1. Model-Free Sparse Reward Social Learning
- With sparse rewards, model-free agents receive no policy-gradient signal from an observed expert demonstration if the episode yields no reward, making novel states and transitions difficult to learn.The authors also explain that Q-learning can force the value of a useful novel state to zero when other Q-values are already zero; TD learning may only temporarily mitigate the problem.Source: §3.1 · pp. 3–4
3.2. Social Learning with Auxiliary Losses
- An unsupervised mean absolute error auxiliary loss trains added network layers to predict the next state from the current state and action, with gradients also improving the state representation.Because observed expert actions affect next-state prediction, the learner can improve its representation from expert cues without explicit access to the experts’ states or actions, and is not forced to imitate them.Source: §3.2 · p. 4
3.3. Social Learning Environments
- Goal Cycle makes individual exploration costly through penalties for incorrect goal orders and uses reward-dependent agent colors as prestige cues; a curriculum gradually increases the penalty.Without an exploration penalty, agents do not learn to rely on social learning.Source: §3.3 · p. 5
3.4. Social Learning in New Environments
- Transfer tests train agents in 3-Goal Cycle and evaluate them in 4-Goal Cycle, Four Rooms, and Maze, where randomized layouts and navigation challenges differ from training.Four Rooms and Maze are 17x17 and 19x19 environments, respectively, with randomly generated goals and obstacles and walls to navigate within a time limit.Source: §3.4 · pp. 5–6
4. Experiments
The experiments test whether auxiliary losses help agents learn from experts, transfer social learning to new environments, and retain skills when alone. Social learners improve on solo and imitation-learning agents across these settings, with outcomes depending on expert availability and training conditions.
4. Experiments
- The experiment compares social agents with auxiliary predictive losses against solo agents with the same architecture, vanilla PPO, reconstruction-loss agents, and imitation learning.Imitation learning receives direct access to experts’ ground-truth states and actions, unlike the other methods.Source: §4 · p. 6
4.1. H1: Learning Social Learning
- Social learners with predictive or reconstructive auxiliary losses learn more complex Goal Cycle behavior than solo or vanilla PPO agents, which fail to discover the correct goal order.The predictive loss yields higher performance than reconstruction, and many predictive-loss seeds exceed expert and imitation-learning performance by waiting for experts to explore before following their cues.Source: §4.1 · pp. 6–7
- With imperfect experts, novices still surpass solo agents, but achieve lower performance and less often exceed their experts than novices trained with optimal experts.Only 1/5 seeds learning from imperfect experts exceed those experts.Source: Figure 5 · p. 7, §4.1 · p. 7
4.2. H2: Adapting Online to New Environments
- Figure 6 reports zero-shot transfer from 3-Goal Cycle to 4-Goal Cycle, Four Rooms, and Maze, comparing social learners with solo, imitation-learning, and expert baselines.Bars show mean performance across seeds, with individual seeds and 95% confidence intervals also shown.Source: §4.2 · p. 7, Figure 6 · p. 8
- In three novel environments, social PPO agents use new expert cues to outperform agents trained alone or with imitation learning, and sometimes outperform their original experts.They adapt to the Maze experts rather than continuing to rely on 3-Goal experts; original experts transfer poorly to 4-Goal Cycle and Maze.Source: §4.2 · pp. 7–8, Figure 6 · p. 8
4.3. Social Learning Helps Solo Learning
- Agents trained only with experts can become reliant on expert cues, improving performance with experts while their solo performance drops.The observed strategy is to follow experts when present and refrain from individual exploration when alone.Source: §4.3 · p. 8
- Interleaving solo and social training helps agents retain solo performance while using expert cues, and they outperform agents trained only alone in solo transfer environments.Solo-only agents do not discover the optimal strategy in this environment because exploration is difficult.Source: §4.3 · p. 8
5. Conclusions
The paper finds that a model-based auxiliary loss can help agents use expert cues to adapt in novel environments, while mixed social and solo training supports skills that work when alone. The experiments focus on exploratory navigation and assume visible prestige cues.
5. Conclusions
- Model-based auxiliary prediction enables agents to use expert cues for online adaptation in novel environments, where agents trained alone or with imitation learning cannot adapt.The paper reports good performance on zero-shot transfer tasks.Source: §5 · p. 8
- Mixing social and solo training lets agents retain solo 3-goal performance while learning to use experts in 4-goal environments.The mixed-training agent’s solo performance exceeds that of agents trained exclusively solo, while its social and 4-goal performance is comparable to an agent trained entirely with experts.Source: §5 · p. 9
- Social learning enables agents to learn complex behavior they do not discover alone and to generalize better when transferred to novel environments.The paper also reports that social learning can yield more complex behavior than experts themselves.Source: §5 · pp. 8–9
5.1. Limitations and Future Work
- The experiments cover exploratory navigation, do not thoroughly explore ways to augment solo experience with social training, and assume learners can see prestige cues.Future work includes other domains and settings, and predicting expert performance when prestige cues are unavailable.Source: §5.1 · p. 9
7. Appendix
The appendix examines auxiliary prediction during training, the role of exploration penalties, and robustness to non-expert distractors. It reports that without the penalty, agents lack an incentive to learn from experts, while distractors do not prevent the method from outperforming baselines.
7.1. Auxiliary prediction loss
- The auxiliary next-state prediction loss reaches low mean absolute error, but it need not decrease consistently because policy changes make prediction non-stationary.New behaviors require predicting state transitions the model has not previously encountered.Source: §7.1 · p. 12
7.2. Training without an exploration penalty
- Without the exploration penalty (p = 0), agents lack an incentive to learn from experts, and their solo policies resemble the three-goal experts.Their performance in the four-goal transfer environment is similar to that of the three-goal experts.Source: §7.2 · p. 12
7.3. Identifying the experts
- With two random-action distractors alongside high-performing experts, the method remains robust and outperforms the baselines on maze transfer.The distractors are non-expert agents that take random movement actions.Source: §7.3 · p. 12, Figure 10 · p. 12
7.4. Imitation learning details
The imitation-learning agents use behavior cloning on expert trajectories. Their policy architecture resembles the social PPO plus auxiliary-recognition agents, but omits value-function and next-state-prediction layers.
7.4. Imitation learning details
- The imitation-learning agents are trained with behavior cloning on expert demonstrations.The cited implementation uses Adam to optimize policy parameters.Source: §7.4 · p. 12
- Their architecture resembles the social PPO plus auxiliary-recognition agents, with value-function and next-state-prediction layers unused.Source: §7.4 · p. 12
- The demonstration dataset contains expert trajectories comprising states and expert actions.Source: §7.4 · p. 12
7.5. Q-learning learning in sparse reward environments
The appendix describes a sparse-reward difficulty for temporal-difference learning: zero-reward experience can drive Q-values toward zero. In that situation, modeling transitions involving a novel state is not required to produce a zero output.
7.5. Q-learning learning in sparse reward environments
- Before receiving reward, a temporal-difference agent may have difficulty learning transition dynamics.The appendix illustrates the issue with a deep Q-learning agent whose Q-function is represented by a neural network.Source: §7.5 · pp. 12–13
- With zero-reward experience, observing a state transition induces correlation between the Q-values and state encodings for its two states.Source: §7.5 · p. 13
- Once all Q-values are driven toward zero, modeling transitions into or out of a novel state is unnecessary to produce a zero output.Source: §7.5 · p. 13
7.6. Environment details
The paper uses partially observed Minigrid-based environments with a limited action set. A penalty curriculum helps produce experts when individual exploration is costly.
7.6. Environment details
- Observations are 27×27 RGB images of 7×7 grid tiles; only rotate left, rotate right, and move forward are relevant actions.Agents cannot see or move through obstructive tiles, but can see and move through goals and other agents.Source: §7.6 · p. 13
- A 19×19 maze is shown as a sample environment.Source: Figure 13 · p. 13
- A penalty curriculum obtains experts for Goal Cycle tasks where costly individual exploration prevents effective strategy learning.The illustrated agent trains for 81920 episodes with penalty 0.5, then continues with penalty 1.0.Source: §7.6 · p. 13, Figure 12 · p. 13
7.7. Network architecture details
The value, policy, and auxiliary-task networks share a convolutional and recurrent core, with separate output branches. The chosen networks contain 668555 parameters, while smaller tested networks showed no significant performance difference.
7.7. Network architecture details
- The value, policy, and auxiliary-task networks share three convolution layers, a fully connected layer, and an LSTM layer.Source: §7.7 · p. 13
- Value and policy outputs use two fully connected layers, while the auxiliary prediction branch uses a fully connected layer followed by transposed convolutions.Convolutional and transposed-convolutional layers use leaky ReLU; other layers use tanh.Source: §7.7 · p. 13
- The networks have 668555 parameters; tested 357291-parameter networks showed no significant performance difference.Network size was set to roughly saturate available desktop compute resources.Source: §7.7 · p. 14
7.8. Hyperparameters
Training uses Adam with separate learning rates for novice and expert agents, and PPO updates are constrained by KL-divergence checks. The objective combines policy, value, entropy, and auxiliary prediction losses.
7.8. Hyperparameters
- Novice agents use Adam at learning rate 1e-4, while expert agents use 1e-5; specified generalization agents and imperfect experts remain frozen.Source: §7.8 · p. 14
- Each update uses 20 sequential mini-batch updates from 128-episode rollouts, stopping above target KL 0.01 and reverting parameters and optimizer state above hard limit 0.03.Hidden states and advantages are recalculated every two mini-batches.Source: §7.8 · p. 14
- The update objective combines PPO-clip policy loss, value mean-squared error, auxiliary prediction loss, and an entropy bonus.The coefficients are c_V=0.1, c_ent=1e-5, and c_aux=3.Source: §7.8 · p. 14
- Larger mini-batches were more stable in tests spanning 32 to 1025 trajectories, and training was less stable with learning rates above 1e-4.The Goal Cycle prestige decay constant α_c was 0.99; network initialization and environment-generation seeds were randomized for each trial.Source: §7.8 · p. 14
7.9. Compute
Training ran on desktop hardware and AWS instances; a single 1.5M-episode, three-agent run took about 30 hours.
7.9. Compute
- A single 1.5M-episode (375M-step) three-agent training run took about 30 hours.Runs used a desktop with an AMD Ryzen 3950x CPU and two Nvidia GTX 1080TI GPUs, or AWS g4dn.8xlarge instances.Source: §7.9 · p. 14