PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing

Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, Stefano V. Albrecht

arXiv:2102.07475v2cs.MAcs.LG

TL;DR

The paper examines how to scale MARL when parameter sharing can improve efficiency yet hurt learning for heterogeneous agents. It introduces SePS, which partitions agents by learned abilities and goals, and reports higher returns, faster training, and scaling to hundreds of agents. The main scope boundary is that pretraining-based partitions may fail when agents diverge later or when distinct roles are not reflected in early data.

  • Problem

    Indiscriminate parameter sharing can reduce training time but harm final convergence when agents differ in dynamics, rewards, or roles.

  • Method

    SePS encodes agents from trajectories and clusters them by abilities and goals so agents share parameters within selected groups.

  • Results

    SePS achieved higher converged returns than parameter-sharing baselines and no sharing, improved training speed over no sharing, and scaled to 200 agents in non-homogeneous environments.

  • Takeaways & Limitations

    Selective sharing can combine parameter-sharing efficiency with the representational capacity of multiple parameter sets for heterogeneous MARL.

  • Takeaways & Limitations

    Partitions learned from early pre-policy data may fail when agents share dynamics and rewards initially but diverge later, motivating retraining with newer experience.

Abstract

from arXiv · show

Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number of trainable parameters, shortening training times to tractable levels, and has been linked to more efficient learning. However, having all agents share the same parameters can also have a detrimental effect on learning. We demonstrate the impact of parameter sharing methods on training speed and converged returns, establishing that when applied indiscriminately, their effectiveness is highly dependent on the environment. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals. Our approach combines the increased sample efficiency of parameter sharing with the representational capacity of multiple independent networks to reduce training time and increase final returns.

1. Introduction

MARL parameter sharing helps scale training but indiscriminate sharing can harm final convergence when agents differ. SePS selectively groups agents by abilities and goals, combining shared-learning efficiency with multiple parameter sets.

  • Motivation: MARL seeks to jointly train multiple agents, but prior work typically considered only two to ten agents.Scaling to many agents has received less attention than experience sharing, agent modelling, and communication.
  • Parameter-sharing trade-offs: Naive parameter sharing reduces training time but can harm final convergence, especially when agents have distinct transition or reward functions.Distinct dynamics make shared hidden representations harder to form, reducing the effectiveness of fully shared parameters.
  • Selective Parameter Sharing: SePS partitions agents by abilities and goals after encoding their trajectories and applying unsupervised clustering.The method is designed to retain parameter sharing where useful while avoiding representational bottlenecks.
  • Parameter-sharing trade-offs: Agents with different roles can interfere with one another because a single network must represent distinct functionalities and updates for separate objectives.The paper illustrates this bottleneck with waiter and cook roles in a restaurant team.
  • Reported outcomes: SePS achieved higher converged returns than both no sharing and naive sharing, improved sample efficiency, executed faster than no sharing, and scaled to 200 agents.The scaling result was reported for environments containing non-homogeneous agents.

2. Background

MARL models partially observable multi-agent decision problems with individual observations, actions, transitions, and rewards. The background introduces policy-gradient actor-critic learning, A2C, and variational autoencoders as relevant foundations.

  • Markov Games: A partially observable Markov game specifies agents, states, local observation spaces, action spaces, transitions, and individual reward functions.Each agent observes only its local observation, while the transition distribution depends on the joint state and action.
  • Markov Games: The MARL objective is to find policies for all agents that maximize each agent’s discounted return with respect to the other policies.The formulation uses a discount factor and a finite episode horizon.
  • MARL setting: The paper does not assume identical action spaces, observation spaces, or reward functions across agents.This assumption motivates methods that can accommodate heterogeneous agents.
  • Policy Gradient and Actor-Critic: Policy-gradient methods optimize parameterized policies using return-weighted gradients, while actor-critic methods use a value function to reduce gradient-estimation variance.The paper uses A2C, which adds n-step rewards, parallel environments, and entropy regularization.
  • Variational Autoencoders: Variational autoencoders learn a density over latent variables and approximate the unknown posterior with a parameterized distribution.The associated objective is expressed through the evidence lower bound and KL-divergence terms.

3. Selective Parameter Sharing

SePS partitions agents into groups that share policies, using an encoder-decoder model to identify agents with similar observation and reward dynamics before reinforcement-learning training. This combines shared representations within groups with separate parameters across groups.

  • Selective Parameter Sharing: SePS partitions N agents into K < N groups, with each group using and updating a shared policy.A deterministic function µ maps each agent to its assigned policy, and the partition is learned before reinforcement-learning training.
  • Selective Parameter Sharing: The method seeks to group agents solving similar tasks so they share parameters without interfering with agents using other policies.The intended grouping is based on similarities in agents’ limited-perspective reward and observation functions.
  • Partitioning: The approach uses limited-perspective models of next observations and rewards to identify agents with similar dynamics rather than requiring exact environment-dynamics reconstruction.The models are intended to provide a basis for partitioning, not necessarily accurate approximations of the full dynamics.
  • Encoding Agent Identities: An encoder maps each agent identity to a latent Gaussian representation, while a decoder predicts that agent’s next observation and reward from the representation, observation, and action.The latent bottleneck forces information about the agent, including its reward or observation-transition function, through the encoding.
  • Encoding Agent Identities: The encoder-decoder is trained on experience from all agents so its latent representations capture their collection of agent-centred transition and reward functions.The agent identity reaches the decoder only through the sampled latent representation.
  • Partitioning: After pre-training, k-means clusters the encoder means, and the resulting agent groups define µ for policy training.The partition can then support generation of a static computational graph with speed advantages.

4. Experimental Evaluation

The evaluation tests SePS across four multi-agent environments with varied agent types, observation spaces, action spaces, and rewards. It compares selective sharing with no sharing and full-sharing baselines, including agent-index conditioning.

  • Evaluation Design: The evaluation tests whether SePS partitions agents correctly and improves returns, sample complexity, and training time.Reinforcement learning uses A2C, reports the sum of all agents’ returns, and searches A2C hyperparameters across baselines.
  • Multi-Agent Environments: Experiments use four environments: Blind-particle Spread, Coloured Multi-Robot Warehouse, Level-based Foraging, and StarCraft Multi-Agent Challenge.These environments include agent distinctions based on colours, levels, or unit types.
  • Multi-Agent Environments: Blind-particle Spread varies the number of agents and colours, while BPS-h additionally gives agent groups different observation spaces.The task requires agents to move toward landmarks matching their assigned colours despite not observing those colours directly.
  • Baselines: The baselines are NoPS, which gives every agent separate parameters, and FuPS, which shares one parameter set across all agents.FuPS is described as naive because it does not allow agents to develop behavioural differences.
  • Baselines: FuPS+id conditions one fully shared policy on agent identity, but its performance deteriorates sharply as the number of BPS colours increases.The reported interpretation is that multiple policies represented by the same parameters create a learning bottleneck; the figure records maximum evaluation returns and standard deviations across seeds.

4.3. An Experimental Evaluation of FuPS+id

FuPS+id gives agents the information needed to condition policies on their identities, but shared parameters still create a learning bottleneck. Across environments, SePS combines selective sharing with improved learning efficiency and returns.

  • BPS evaluation: FuPS+id supplies agent identities, yet performance deteriorates sharply even with only three colours.The agents have the information needed to learn the correct landmark policy, but overlapping policies must still be represented by shared parameters.
  • BPS evaluation: Scaling FuPS+id network width does not solve BPS tasks, suggesting shared-parameter interference rather than insufficient model capacity.The scaled baseline uses approximately #Colors ∗18K parameters across the different BPS-h tasks.
  • BPS evaluation: In BPS, independent learners can learn each agent’s target, whereas fully shared policies converge to a local minimum and NoPS becomes infeasible with 200 agents.The shared agents cannot perceive their own colours, while NoPS requires N different parameter sets.
  • Cross-environment results: In sparse-reward C-RWARE, NoPS fails to learn, while parameter sharing provides a useful learning direction and SePS outperforms naive sharing methods.The passage attributes the learning direction to combining received rewards through parameter sharing.
  • Cross-environment results: In LBF, SePS achieves higher returns with more efficient sample use, while NoPS requires considerably more samples and may eventually reach similar returns.FuPS+id approaches but does not achieve the optimal return of 1.0.
  • Cross-environment results: In MMM2, shared-parameter methods outperform NoPS, with only minimal improvement from SePS over FuPS+id.The authors hypothesize that similar agent behavior and shared-reward decomposition explain this result.

4.5. A Peek into the Embedding Space of SePS

SePS produces embeddings that generally separate agents according to meaningful environmental and behavioral differences, enabling clustering that often matches agent types. The number of clusters remains consequential: too few clusters reduces returns, while overestimating it has little effect.

  • Embedding structure: The learned embeddings produced clearly visible clusters corresponding to different SMAC unit types and their distinct environmental properties.The encoder captured differences such as movement speed, health, and damage.
  • Embedding structure: Agent starting positions can further separate otherwise similar agents in the embedding space.In MMM2, initial observations are sampled from different sets for different agents.
  • Clustering validity: Across environments and seeds, clustering matched agent types, including colour-based groups that were not present in the observation space.The grouping was inferred from transitions and rewards rather than explicit colour information.
  • Choosing K: Overestimating the number of clusters had little significance, whereas using fewer clusters than needed lowered achieved returns and K = 1 collapsed to NoPS.The results were obtained on C-RWARE with varied cluster counts.
  • Choosing K: The Davies–Bouldin index coupled with k-means reliably found the same clusters as an expert in the tested environments.The paper also identifies domain knowledge and treating K as a tunable hyperparameter as alternatives.

4.7. Computational Benefits

SePS improves computational scaling relative to fully independent networks while adding some complexity compared with fully shared networks. In BPS, its training time was substantially lower than NoPS because it used fewer trainable parameters.

  • Measurement: The computational comparison reports median timestep time during training across all environments and methods.The implementation used parallel environment sampling and an AMD Epyc 7702 system for Figure 8.
  • Training-time scaling: SePS scales significantly better than NoPS despite adding computational complexity over fully shared networks.Figure 8 compares median timestep running times across environments and methods.
  • Training-time scaling: With 30 agents in BPS, SePS required almost half the training time of NoPS.The reduction was attributed to substantially fewer trainable parameters.
  • Training-time scaling: NoPS training was infeasible in BPS-h(3) because it required 200 parameter sets, 50 more times than SePS.This comparison illustrates the scaling advantage of clustered parameter sharing.

5. Related Work

Prior MARL work addresses coordination, experience sharing, agent modelling, and scaling through several forms of shared information or parameters. SePS instead statically partitions agents into shared networks to improve the efficiency and effectiveness of parameter sharing.

  • Coordination and sharing: CTDE methods give agents access to other agents’ data during training but restrict them to local observations after training.Examples include MADDPG, Q-MIX, and SEAC.
  • Parameter sharing: Existing parameter-sharing methods such as COMA, Q-Mix, and Mean Field RL share neural-network parameters across agents.ROMA learns dynamic roles, whereas SePS performs static partitioning to maximise computational efficiency.
  • Parameter sharing: SePS creates neural-network architectures in advance, allowing more efficient and effective sharing than indiscriminate parameter sharing.This distinguishes its architecture-level partitioning from related sharing approaches.
  • Experience sharing: SEAC maintains separate policy and value networks while sharing experience, requiring one network per agent and larger stacked batches.SePS gives up SEAC’s exploration benefits in exchange for potential scaling to hundreds of agents.
  • Scaling MARL: Mean Field RL approximates interactions with the population average, while SePS uses pre-training to find a network configuration and can combine with other MARL algorithms.The passage contrasts population approximation with architecture discovery.

6. Limitations and Future Work

SePS can mispartition agents when their dynamics or rewards change after the initial data-collection phase, and identical dynamics and rewards may still conceal different intended roles. The authors leave adaptive repartitioning and trained-policy role discovery for future work.

  • Limitations: Initial pre-policy samples may fail to partition agents whose dynamics or rewards diverge later in training.In such cases, SePS may behave like full parameter sharing; retraining the encoder-decoder and redistributing agents could improve it.
  • Limitations: Agents with identical dynamics and rewards but different intended roles create a setting where the benefit of sharing depends strongly on the environment.Role discovery using trained policies is identified as future work.

7. Conclusion

The experiments found that indiscriminate parameter sharing can make learning harder because agents interfere with one another. Selective Parameter Sharing instead groups agents by abilities and goals, scaling training to hundreds of agents and outperforming the reported baselines.

  • Indiscriminate parameter sharing made learning harder because agents interfered with one another.The paper attributes this effect to interference among agents during learning.
  • Selective Parameter Sharing identifies groups of agents that may benefit from sharing parameters.SePS partitions agents based on their abilities and goals.
  • SePS scaled MARL training to hundreds of heterogeneous agents by assigning them to different parameter sets.The method successfully recognised heterogeneous agents rather than requiring one shared parameter set.
  • SePS outperformed other parameter-sharing baselines in converged returns and a non-parameter-sharing baseline in both converged returns and training speed.The comparison covers both final performance and training speed.

8. Funding Disclosure

The research received financial support from the UK EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems and the University of Edinburgh Enlightenment Scholarship.

  • The research was financially supported by the UK EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems and the University of Edinburgh Enlightenment Scholarship.
Loading 2102.07475v2…