Source-linked AI summary
Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, Stefano V. Albrecht
TL;DR
Multi-agent deep reinforcement learning faces non-stationarity because concurrently learning agents continually change their policies. This survey consolidates recent approaches for addressing the problem, categorizes five approaches, and outlines open research problems.
Problem
Changing agent policies create non-stationarity in multi-agent environments, alongside challenges including credit assignment, partial observability, and heterogeneity.
Method
The survey consolidates multi-agent deep RL progress, categorizes five approaches for handling non-stationarity, and discusses open problems and future directions.
Results
The survey provides a detailed categorization of five approaches, including centralized critic techniques, opponent modelling, meta-learning, and transfer-related methods.
Takeaways & Limitations
Open questions concern transferring knowledge for quicker adaptation, handling changing numbers of agents and heterogeneity, and avoiding reliance on opponent observations and actions during testing.
Takeaways & Limitations
Opponent-modelling methods often assume access to opponents’ observations and actions during testing, an assumption that is too strong when agents lack established communication.
Abstract
from arXiv · showhide
Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed methods range from modifications in the training procedure, such as centralized training, to learning representations of the opponent's policy, meta-learning, communication, and decentralized learning. The survey concludes with a list of open problems and possible lines of future research.
1 Introduction
Deep reinforcement learning extends autonomous decision-making to complex, high-dimensional environments, while multi-agent settings introduce non-stationarity as concurrently learning agents change their policies. This survey consolidates recent approaches for handling that problem in multi-agent deep RL and identifies open research directions.
- Motivation: Deep learning helps reinforcement learning agents act in complex environments with larger state spaces, including image-based environments.Traditional tabular and linear RL methods are difficult to scale to high-dimensional environments.
- Multi-Agent Reinforcement Learning: Multi-agent systems model multiple agents acting and learning in a shared environment, including autonomous vehicles, resource allocation, robot swarms, and human-robot interaction.These systems support decision-making problems involving multiple interacting agents.
- Non-Stationarity: Changing agent policies make multi-agent environments non-stationary because transitions and rewards depend on actions from policies that evolve during training.This breaks the Markov assumption used by most single-agent RL algorithms and can lead agents into endless adaptation cycles.
- Additional Challenges: Multi-agent learning also faces credit assignment, partial observability, and heterogeneity in agents’ sensing, acting, and learning capabilities.Partial observability limits agents to local observations, while credit assignment identifies which agents contributed to a reward.
- Survey Scope: The survey consolidates progress on non-stationarity in multi-agent deep RL and concludes with open problems and possible future research directions.It addresses a recent focus on multi-agent deep RL not covered by earlier surveys of related multi-agent RL topics.
2 Background
The background formulates reinforcement learning through MDPs and Markov games, then contrasts centralized and decentralized multi-agent architectures. It explains that changing policies alter perceived transitions and rewards, creating the non-stationarity problem illustrated by repeated Rock-Paper-Scissors.
- Markov Decision Processes: An MDP is defined by states, actions, rewards, and transition probabilities, with a policy selecting actions from states.Policies may be stochastic or deterministic.
- Markov Decision Processes: The optimal MDP policy maximizes expected discounted rewards, while reinforcement learning estimates it when reward and transition functions are unknown.Dynamic programming can solve the problem when those functions are available.
- Reinforcement Learning Methods: Temporal-difference methods estimate value functions from environmental interaction, whereas policy-gradient methods optimize parameterized policies for expected discounted rewards.Deep networks address the scaling limits of tabular and linear approaches in large state and action spaces.
- Markov Games: Markov games generalize MDPs to multiple agents whose joint actions determine transitions and whose rewards may be individual or shared.In cooperative settings, all agents can have the same reward function.
- Partial Observability: Agents in the survey observe locally and condition policies on observation histories rather than the environment’s full state.Conditioning on observation history experimentally performs better in partially observable environments.
- Centralized and Decentralized Architectures: Centralized learning jointly models agents but faces input and output growth, whereas decentralized learning trains independent local policies but introduces non-stationarity, credit assignment, and coordination problems.Centralized inputs grow linearly and outputs exponentially with the number of agents.
- The Non-Stationarity Problem: Changing policies alter each agent’s perceived transition and reward functions, so single-agent procedures assuming stationarity may not adapt quickly.Repeated Rock-Paper-Scissors illustrates agents continually changing responses to one another’s action preferences.
3 Dealing with Non-Stationarity
The survey reviews centralized and decentralized methods for addressing non-stationarity in multi-agent deep RL, including centralized critics, self-play, replay stabilization, opponent modelling, and meta-learning.
- Centralized Critic Techniques: Centralized critics use all agents’ observations and actions during training while retaining decentralized actors for execution.This conditions learning on other agents’ policies and can address both non-stationarity and credit assignment.
- Centralized Critic Techniques: MADDPG gives each agent a centralized critic and decentralized actor, making the environment appear stationary during training.
- Decentralized Learning Techniques: Self-play trains agents against current or previous policy versions so decentralized policies can generalize across opponents.Using previous versions helps recent approaches train effectively under deterministic dynamics.
- Decentralized Learning Techniques: Experience replay can become outdated under non-stationarity, motivating importance-sampling corrections that adjust the weight of prior experience.
- Opponent Modelling: Opponent modelling seeks to stabilize training by representing other agents’ intentions or policies and conditioning the agent’s policy on those representations.Separate prediction networks can model changing opponent behavior, while empirical results report improved performance against learning or stationary opponents.
- Meta-Learning: Meta-learning optimizes initial parameters for expected post-learning performance and significantly outperforms tracking and other adaptation strategies in iterated adaptation games.
4 Open Problems
The survey identifies unresolved issues involving transfer across agent populations, changing numbers of agents, limited opponent information, theoretical convergence, and credit assignment.
- 4.1 Transfer Learning for Non-Stationarity: Transfer-learning methods may accelerate adaptation, but the form of transferable knowledge and how to exploit it remain open questions.
- 4.2 Open Multi-Agent Systems: Changing agent populations create open multi-agent systems in which the same action can have different consequences as agents enter or leave.None of the surveyed techniques were tested with changing numbers of agents.
- 4.3 Limited Access to Opponent Information: Opponent-modelling methods often require opponents’ observations and actions, an assumption that is especially limiting during decentralized testing.Developing models that do not rely on this access is an open problem.
- 4.4 Theoretical Understanding: Current multi-agent deep RL methods lack theoretical understanding of their convergence properties and the outcomes they tend to achieve.Game-theoretic equilibria offer one possible direction, but require equilibrium computation and may be non-unique.
- 4.5 Credit Assignment: Credit assignment remains improvable even with centralized training and opponent representations, motivating reward-decomposition architectures for smaller agent groups.
5 Conclusion
The survey identifies five approaches for handling non-stationarity in multi-agent deep RL, categorizes their characteristics, and outlines open problems and future research directions.
- Five approaches for handling non-stationarity in multi-agent deep RL are identified.
- The survey outlines open problems concerning non-stationarity and possible directions for future research.
- The surveyed algorithms are categorized by environment setting, training and execution method, modelling type, opponent information, learning algorithm, and number of agents handled.