Source-linked AI summary
Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
Thanh Thi Nguyen, Ngoc Duy Nguyen, Saeid Nahavandi
TL;DR
High-dimensional environments challenge conventional reinforcement learning, and complex tasks may require multiple agents to communicate, compete, or cooperate. This paper surveys deep reinforcement learning methods for multi-agent challenges and reviews their applications, finding that integrating deep learning with traditional MARL has addressed many complicated problems. The review also identifies limitations and directions for further MADRL research.
Problem
High-dimensional environments and complex tasks involving multiple interacting agents motivate methods that extend deep reinforcement learning beyond single-agent settings.
Method
The paper surveys MADRL solutions across non-stationarity, partial observability, training schemes, transfer learning, and continuous state and action spaces, together with their applications.
Results
Integrating deep learning with traditional MARL has been able to solve many complicated problems, including traffic control, energy sharing, fleet management, and task allocation.
Takeaways & Limitations
The review provides insights into MADRL methods and highlights applications spanning robotics, autonomous vehicles, traffic, energy, fleet management, and resource allocation.
Takeaways & Limitations
Model-based deep RL extensions have not been investigated widely in the multi-agent domain, leaving a research gap for model-based MADRL.
Abstract
from arXiv · showhide
Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision-making problems. These algorithms however have faced great challenges when dealing with high-dimensional environments. The recent development of deep learning has enabled RL methods to drive optimal policies for sophisticated and capable agents, which can perform efficiently in these challenging environments. This paper addresses an important aspect of deep RL related to situations that require multiple agents to communicate and cooperate to solve complex tasks. A survey of different approaches to problems related to multi-agent deep RL (MADRL) is presented, including non-stationarity, partial observability, continuous state and action spaces, multi-agent training schemes, multi-agent transfer learning. The merits and demerits of the reviewed methods will be analyzed and discussed, with their corresponding applications explored. It is envisaged that this review provides insights about various MADRL methods and can lead to future development of more robust and highly useful multi-agent learning methods for solving real-world problems.
1 Introduction
Deep reinforcement learning addresses the limitations of conventional reinforcement learning in high-dimensional environments, while multi-agent systems extend the setting to tasks requiring agents to compete or cooperate. This survey reviews the resulting MADRL challenges, methods, and applications.
- Reinforcement Learning Background: Conventional Q-learning struggles with high-dimensional problems because computational requirements increase drastically with the number of inputs.This limitation is known as the curse of dimensionality.
- Deep Reinforcement Learning: Deep RL marked a modern transition by enabling agents to perform successfully in challenging environments, including Atari games, Go, physics problems, and 3D maze games.DQN agents outperformed professional players in 49 classic Atari games, while later systems achieved high-level performance in Go and other domains.
- Reinforcement Learning Background: RL differs from supervised and unsupervised learning because agents learn through environment interaction toward goals that maximize long-term profit.Supervised learning uses externally labelled input-output data, whereas unsupervised learning explores hidden data structure without specified outputs.
- Multi-Agent Deep Reinforcement Learning: Multi-agent systems become necessary when a single deep RL agent cannot handle increasingly complicated real-world problems.Agents in these systems may compete or cooperate to obtain the best overall results.
- Multi-Agent Deep Reinforcement Learning: The survey organizes MADRL research around non-stationarity, partial observability, training schemes, transfer learning, and continuous state and action spaces.It also reviews and analyzes applications of MADRL across multiple fields.
2 Background: Reinforcement Learning
Reinforcement learning models sequential decision-making through agent–environment interaction, using policies, returns, and value functions to improve behavior. Classical methods include model-free Monte-Carlo and temporal-difference learning, while dynamic programming requires complete environment dynamics and becomes infeasible in large state spaces.
- RL formulation: RL represents decision-making as an agent interacting with an environment through states, actions, and rewards until a terminal state or episode endpoint.An episode is a sequence from an initial state to a terminal state; in pole balancing, termination can depend on pole angle or cart position.
- Policies: A policy maps perceived states to actions and may be deterministic or stochastic according to the probabilities assigned to candidate actions.A deterministic policy selects one action with probability 1, whereas a stochastic policy assigns probabilities below 1 to at least some actions.
- Policy improvement: Policy improvement starts from a random policy and iteratively replaces it with better policies until an optimal policy cannot be improved further.A naive improvement changes an action when its Q-value exceeds that of the current action, while leaving other state-action pairs unchanged.
- Returns and value functions: Returns aggregate immediate rewards with discounting, and value functions estimate expected returns for states or state-action pairs under a policy.The discount factor γ controls farsightedness: values near 1 emphasize longer-term rewards, while values near 0 emphasize short-term rewards.
- Bellman equations: Bellman equations support policy evaluation and improvement, but dynamic programming requires complete transition dynamics and is infeasible for large state spaces.Model-free methods avoid requiring transition probabilities and can therefore address larger state-space problems.
- Model-free learning: Monte-Carlo learning averages returns across sampled episodes, whereas temporal-difference learning bootstraps from previous value estimates and generally learns faster in most cases.Monte-Carlo methods are model-free but require many episodes and repeated visits to states and actions; both Monte-Carlo and temporal-difference methods have on-policy and off-policy variants.
3 Deep RL: Single Agent
Deep Q-networks apply deep learning to reinforcement learning in high-dimensional environments, but their instability and other limitations motivate several variants. Double, prioritized-replay, dueling, and recurrent modifications address distinct problems and can substantially improve Atari performance.
- Deep Q-Network: DQN combines convolutional neural networks with reinforcement learning to interpret graphical states and output Q-values for possible actions.A target network and experience replay are used during training to reduce problems from correlated samples.
- Deep Q-Network: Neural-network value approximation can be unstable and may diverge because of bias from correlated samples.DQN addresses this with a separately updated target network and randomly sampled experience replay.
- DQN Variants: DDQN separates greedy-action selection from action evaluation to reduce overestimation of Q-values during training.Its normalized performance was two times greater than DQN without tuning and three times greater with tuning on 57 Atari games.
- DQN Variants: Prioritized experience replay samples rare and goal-related experiences more frequently using their absolute TD errors as priorities.Combined with DDQN, it provided stable policy-network convergence and achieved up to five times DQN’s normalized mean score on 57 Atari games.
- DQN Variants: Dueling networks separately estimate state value and action advantage before aggregating them to approximate the Q-value.This architecture targets situations with multiple actions producing no negative result and, with DDQN and prioritized replay, reached up to six times standard DQN performance on Atari.
- DQN Variants: DQN’s four-frame input is inefficient when the current state depends on substantially longer history, motivating recurrent variants for partially observable tasks.DRQN outperformed standard DQN by up to 700 percent in Double Dunk and Frostbite.
4 Deep RL: Multi-Agent
Multi-agent deep reinforcement learning extends deep RL to settings where agents must cooperate or compete under non-stationarity, partial observability, continuous spaces, and scaling constraints. The reviewed methods address these challenges through specialized value learning, recurrent architectures, curriculum learning, centralized training, communication, and hierarchical control.
- Multi-agent challenges: Multi-agent systems solve complex tasks through agent cooperation, but introduce heterogeneity, collective-goal, scalability, and non-stationarity challenges.An agent’s environment includes other learning agents, so policies that are effective at one point may not remain effective later.
- Non-stationarity: Lenient-DQN uses decaying temperature values to adjust replay-based policy updates and converges to optimal policies more effectively than hysteretic-DQN in stochastic-reward transportation tasks.Weighted double DQN with leniency and scheduled replay also outperformed double DQN in two stochastic-reward multi-agent environments.
- Partial observability: Recurrent architectures address partial observability by estimating Q(o, a) from observations and hidden states; distributed recurrent Q-networks add last-action inputs, shared weights, and no experience replay.Inter-agent weight sharing reduces learning time but assumes agents have identical action sets, limiting use with heterogeneous agents such as UAVs and ground robots.
- Scaling multi-agent learning: Curriculum learning scales multi-agent deep RL by training fewer agents on simpler tasks before progressively increasing the number of agents and task difficulty.The reviewed experiments report its usefulness for complex multi-agent problems in partially observable domains.
- Training schemes and continuous spaces: Independent learning treats other agents as part of the environment but is vulnerable to overfitting and computational expense, while centralized training with decentralized execution offers an alternative scheme.Continuous action spaces remain difficult because discretization causes an exponential increase in action numbers with degrees of freedom.
- Applications: Applications include action trading that increases social welfare, privacy-oriented hierarchical control, and communication-based task allocation among agents.Hierarchical control uses decentralized task controllers and a meta-controller, while communication methods support resource allocation but may struggle with heterogeneous agents.
5 Conclusions and Research Directions
The review identifies major challenges in multi-agent deep reinforcement learning and surveys methods, applications, and research gaps. It concludes that deep RL-based approaches are viable for complex multi-agent tasks while highlighting unresolved scalability, human-machine interaction, and model-based learning issues.
- Conclusions: The review organizes multi-agent learning challenges into five categories and evaluates the advantages and disadvantages of deep RL solutions.The categories are non-stationarity, partial observability, multi-agent training schemes, multi-agent transfer learning, and continuous state and action spaces.
- Applications: Deep RL integration with traditional multi-agent reinforcement learning has addressed complex applications including traffic control, energy sharing, fleet management, resource allocation, swarm robotics, and social phenomena.The review reports applications across urban traffic light control, zero-energy communities, large-scale fleet management, task and resource allocation, swarm robotics, and social science.
- Research Directions: Multi-agent learning from demonstration remains insufficiently explored because collaborative tasks require multiple experts and difficult-to-model communication and reasoning capabilities.The review notes that imitation learning and inverse reinforcement learning have shown effectiveness in single-agent deep RL but are not fully explored in multi-agent environments.
- Research Directions: Human-machine teaming remains important because humans and machines have complementary limitations in complex and adversarial environments.The review states that humans cannot sustain the volume of activity alone, while machines may lack creative responses to novel situations.
- Research Directions: Model-based MADRL remains under-explored despite model-based methods’ reported advantages in sample efficiency, transferability, and generality.The paper specifically identifies deep-learning extensions of model-based methods and their combination with planning or model-free policies as research directions.
- Research Directions: Scaling remains difficult for large systems with many heterogeneous agents, although partially centralized training and decentralized execution can support scalability for homogeneous agents.The review identifies heterogeneity, shared behaviors, domain knowledge, and goals as central considerations in multi-agent scalability.