A Survey and Critique of Multiagent Deep Reinforcement Learning

Pablo Hernandez-Leal; Bilal Kartal; Matthew E. Taylor

Source-linked AI summary

A Survey and Critique of Multiagent Deep Reinforcement Learning

Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

arXiv:1810.05587v3Oct 12, 2018cs.MAcs.AIcs.LG

TL;DR

MDRL research has produced successes in complex multiagent domains, but multiagent pathologies and limited convergence guarantees make the area difficult to assess and advance. This survey organizes recent work, connects MDRL to RL and MAL foundations, and synthesizes lessons, benchmarks, open questions, and practical challenges. It concludes that progress must account for adaptation of foundational methods as well as computational and reproducibility constraints.

Problem
Multiagent learning is more difficult than single-agent learning because of non-stationarity, dimensionality, credit assignment, exploration, and relative overgeneralization, while convergence guarantees remain limited outside restricted settings.
Method
The survey categorizes recent MDRL work into four topics, reviews RL and MAL components adapted to MDRL, and discusses lessons, benchmarks, open problems, and practical challenges.
Results
The survey provides a broad overview of MDRL, explains adaptations of components such as experience replay and difference rewards, and identifies lessons, benchmarks, open problems, computational demands, and reproducibility concerns.
Takeaways & Limitations
MDRL research can draw on abundant RL and MAL literature while requiring methods and evaluations suited to multiagent settings.
Takeaways & Limitations
MDRL results are difficult to reproduce because baselines, hyperparameters, architectures, random seeds, and statistical-testing practices vary, while tuning is computationally expensive.

Abstract

from arXiv · show

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.

1. Introduction

MDRL has emerged as multiagent learning combines with deep reinforcement learning, but its greater complexity creates distinctive challenges. This survey organizes recent MDRL work, relates it to RL and MAL foundations, and offers guidance, benchmarks, practical critiques, and open questions.

Motivation: MDRL extends deep reinforcement learning into multiagent settings, where learning is harder because agents create non-stationarity, dimensionality, credit-assignment, exploration, and overgeneralization challenges.Despite these difficulties, recent works report successes in MDRL.
Scope: The survey complements existing work by providing a brief overview of current MDRL literature alongside surveys of multiagent learning, cooperation, agent modeling, knowledge reuse, and deep reinforcement learning.Its stated goal is to connect related literatures rather than treat MDRL as an isolated area.
Organization: It reviews foundational RL and DRL algorithms, introduces the multiagent setting, and groups recent MDRL research into emergent behaviors, communication, cooperation, and agents modeling agents.The four categories cover evaluation of single-agent algorithms in multiagent scenarios, learned messaging, action-based cooperation, and reasoning about other agents.
Synthesis and guidance: The survey relates MDRL methods to prior RL and MAL literature, including how established methods and algorithms have been scaled to MDRL.It also describes lessons learned, points to multiagent benchmarks, and identifies open research questions.
Practical challenges: The article critically examines practical MDRL challenges, including reproducibility, hyperparameter tuning, computational demands, and fragmented interaction among sub-communities.These concerns accompany the survey’s aim of developing a common understanding across researchers with DRL or MAL experience.

2. Single-agent learning

Single-agent reinforcement learning formalizes decision-making through MDPs, while deep methods use neural networks to approximate policies or value functions. The section reviews value-based, policy-gradient, and actor-critic methods, alongside challenges including non-i.i.d. data, approximation instability, and implementation pitfalls.

Reinforcement learning: An MDP models single-agent fully observable decision-making with states, actions, rewards, transitions, and a discount factor.Solving an MDP yields a policy mapping states to actions, while reinforcement-learning algorithms can learn from environment interactions when the model is unavailable.
Reinforcement learning: Q-learning estimates discounted future payoff for state-action pairs and is proven to converge under finite discrete spaces and specific learning-rate and visitation conditions.The guarantees require every state-action pair to be visited infinitely often, with learning rates whose sum diverges while the sum of squared learning rates remains finite.
Reinforcement learning: Policy-gradient methods directly optimize parameterized policies, while actor-critic methods use a learned value function to reduce variance and assign action-selection credit.The actor represents the policy, and the critic estimates values; an advantage function compares state-action values with state values.
Deep reinforcement learning: Deep neural function approximators improve generalization across large state spaces and reduce reliance on manually designed features, but introduce non-i.i.d. data and instability challenges.Sequential interactions create correlated data, changing policies make the distribution non-stationary, and the combination of function approximation, bootstrapping, and off-policy learning can cause divergence.
Deep reinforcement learning: Practical reliability remains a concern because poor representational choices can produce divergent value estimates, and reported performance gains have sometimes resulted from implementation errors.A cited study found that bugs or coding errors, rather than the proposed method, explained good performance in some works.
Deep reinforcement learning: DQN combines neural-network Q-value approximation with experience replay and a target network to stabilize value-based learning.Earlier neural-network approaches also represented Q-values, while Double Q-learning later motivated Double DQN to reduce overestimation bias.

3. Multiagent Deep Reinforcement Learning (MDRL)

MDRL extends reinforcement learning to settings where agents interact with one another and the environment, creating non-stationarity, scaling, credit-assignment, and exploration challenges. The survey organizes recent work into categories and reviews convergence results, adaptations from RL and MAL, and practical considerations.

3.1. Multiagent Learning: Multiagent learning is more complex than single-agent learning because agents interact simultaneously with the environment and with one another.Independent learners apply single-agent algorithms while treating other agents as part of the environment, despite violated assumptions.
3.1. Multiagent Learning: Agents’ policies can be non-stationary, so an agent’s optimal policy depends on the changing policies of other agents.In Markov games, transition and reward functions depend on joint actions across agents.
3.1. Multiagent Learning: Convergence guarantees vary by environment: minimax Q-learning can guarantee optimal play against arbitrary opponents in zero-sum games, whereas coordination requires strong assumptions and other settings lack guarantees.The survey distinguishes adversarial, coordination, and other environments when reviewing value-based convergence results.
3.2. MDRL Categories: Recent MDRL research is grouped into emergent-behavior analysis, learned communication, learned cooperation, and agents modeling agents.The categories are not mutually exclusive, and some works appear in more than one category.
3.2. MDRL Categories: The survey relates MDRL methods to earlier RL and MAL ideas, including self-play, population-based training, counterfactual policy gradients, and minimax learning.These adaptations address cooperation, credit assignment, robustness, and multiagent interaction.

4. Bridging RL, MAL and MDRL

The survey bridges RL, MAL, and MDRL by tracing how earlier ideas extend to multiagent deep reinforcement learning, while offering lessons, benchmarks, and open questions. It also highlights practical constraints including reproducibility, hyperparameter tuning, computational demands, and the need for clearer experimental reporting.

Guidance for practitioners: It organizes guidance for practitioners around lessons learned, standardized environments, benchmarks, and practical evaluation concerns.Open-source simulators and benchmarks are presented as an emerging resource for MDRL research.
Bridging prior literature: The survey uses earlier RL and MAL literature to explain how established ideas and algorithms have been extended to MDRL.It explicitly frames this as avoiding “deep learning amnesia” and preserving credit to original contributions.
Practical challenges: Reproducibility is aggravated in DRL by stochasticity from baselines, hyperparameters, architectures, and random seeds, alongside weak statistical-testing practices.The survey specifically criticizes selective reporting and calls for carefully designed negative results.
Practical challenges: Hyperparameters can affect algorithms differently across environments, making complete parameter reporting important for accurate MDRL comparisons.The survey also links tuning to cherry picking and notes that extensive tuning is computationally expensive.
Practical challenges: Computational infrastructure is a major MDRL bottleneck because deep RL generally requires millions of interactions and large-scale training resources.The survey recommends reporting CPU/GPU use, memory demands, and wall-clock computation, while prioritizing algorithmic contributions over compute scale.
Open questions: Open MDRL questions include scaling search, improving abstraction and factorization, and making diverse self-play more computationally efficient.Centralized search faces exponential growth in joint action spaces, while diverse self-play can require parallel training or additional memory.

5. Conclusions

MDRL extends deep reinforcement learning into multiagent settings, where non-stationarity, increased dimensionality, and credit assignment make learning more difficult. The survey organizes recent work, adapts RL and MAL components, offers practitioner guidance, and identifies computational and reproducibility challenges.

MDRL learning is harder than single-agent deep reinforcement learning because of non-stationarity, increased dimensionality, and credit assignment.
The survey categorizes MDRL research into emergent behaviors, communication, cooperation, and agents modeling agents.
It explains how RL and MAL components, including experience replay and difference rewards, must be adapted for MDRL.
The survey provides lessons for practitioners, identifies multiagent benchmarks, and highlights open research problems.
The authors identify high compute requirements, difficult reproducibility, hyperparameter tuning, and limited encouragement for negative results as practical barriers.