Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey

I. INTRODUCTION

• Earlier autonomous-driving research separated perception, planning, and control, while RL later enabled end-to-end control from sensed information to vehicle actions.The paper situates these shifts alongside the growth of deep learning, datasets, and simulators.Source: §I · p. 1, Figure 2 · p. 2
• Individual-vehicle RL can produce self-centric, potentially aggressive driving, whereas real traffic systems seek to optimize overall efficiency across interacting agents.Agents' states depend on others' behavior, making environmental dynamics non-stationary and time-variant.Source: §I · p. 1
• The review examines over 200 publications, software, and code bases to synthesize MARL advances, benchmark criteria, challenges, and prospects for autonomous driving and intelligent transportation.Its contributions include criteria for simulators, datasets, and competitions, plus a categorization of state-of-the-art MARL methodologies.Source: §I · p. 2
• Prior surveys variously emphasize single-agent RL, motion planning, trajectory prediction, or earlier MARL work, motivating an updated review of recent theory, applications, and simulators.The paper highlights simulators' role in determining the sim-to-real gap.Source: §I · p. 2
• The paper also maintains a repository for ongoing updates to MARL-based autonomous-driving, intelligent-transportation, and related studies.It presents its review as spanning the development of datasets, simulators, hardware, and software.Source: §I · pp. 2–3

1) Realism and Fidelity:

• High realism and fidelity help reproduce real driving conditions and reduce distributional bias before deployment.Realistic simulator data is intended to represent real-world scenarios for deep-learning and deep-RL models.Source: 1) Realism and Fidelity · p. 3

2) Scalability:

• Scalability concerns whether simulators and MARL algorithms can handle numerous entities, variables, and changing numbers of agents.The aim is to mimic complex real-world environments and traffic interactions.Source: 2) Scalability · p. 3

3) Diversity:

• Scenario diversity covers variations in traffic, weather, roads, and heterogeneous vehicles, reflecting complex interactions in real traffic.The passage links this diversity to developing more robust systems for unpredictable conditions.Source: 3) Diversity · p. 3

4) Efficiency:

• Simulator efficiency balances computational cost and parallel experimentation against the computational demands of high-fidelity, especially 3D, simulation.Lightweight simulators can run on smaller hardware, while parallel environments can accelerate MARL training.Source: 4) Efficiency · p. 3

5) Transferability:

• Transferability requires sensor characteristics and data formats to match commercial devices, alongside compatibility with vehicle programming, communication, and computing platforms.The stated sensor examples include LiDAR, IMU, and cameras.Source: 5) Transferability · p. 3

A. What is important for a good benchmark?

• Verified baselines, usable APIs, annotations, tutorials, and continued dataset and code maintenance support reproducible comparisons and keep resources current.The passage notes that changing hardware and software can make older data and code outdated.Source: 6) Features, Maintenance and Supports · p. 3

1) The Open Racing Car Simulator:

• TORCS offers editable vehicle dynamics, multi-perspective visual information, and simulations up to 500Hz; Gym-TORCS adds Gym-compatible APIs, while MADRaS supports single- and multi-agent environments.Gym-TORCS lacks MARL and parallel-environment support, which MADRaS addresses.Source: 1) The Open Racing Car Simulator · pp. 3–4

B. Advanced Simulators

• SUMO supports customizable large-road-network scenarios; Flow adds high-frequency traffic simulation and distributed-RL interfaces, while CityFlow reports simulation speeds 20 times higher than SUMO.CityFlow also supports MARL interfaces and external data import.Source: 2) Simulation of Urban Mobility · p. 4

3) Scalable Multi-Agent Reinforcement Learning Training School:

• SMARTS combines heterogeneous agent assets, Gym-standardized APIs, MARL libraries, distributed computation, and web-based visualization.It uses SUMO as a background provider and a Bullet-based physical engine for vehicle dynamics.Source: 3) Scalable Multi-Agent Reinforcement… · p. 4

4) MetaDrive:

• MetaDrive supports customizable maps, traffic flows, and agent mixtures, with state-vector observations and benchmarks for model-free RL, imitation learning, and offline RL.It sacrifices fine-grained visual information for faster, more efficient simulation and can theoretically create infinitely varied traffic scenarios.Source: 4) MetaDrive · p. 4

B. Advanced Simulators

• CARLA provides realistic 3D scenes, varied traffic participants and conditions, and configurable sensors; its MACAD supplement offers a faster MARL implementation with Gym-like APIs.The passage says CARLA reduces the sim-to-real gap and supports vision-based driving research.Source: 5) CAR Learning to Act · p. 4

6) Virtual Image Synthesis and Transformation for Autonomy:

• VISTA reconstructs perceptual input sequences from real-world data, and later versions add LiDAR and event data plus basic multi-agent interaction support.Its sim-to-real gap is described as minimized, while large-scale complex scenarios remain unexplored; ISAAC-Sim is briefly identified as using PhysX5 and GPU-based photorealism.Source: 6) Virtual Image Synthesis and… · p. 4, 7) NVIDIA ISAAC-Sim · p. 4

7) NVIDIA ISAAC-Sim:

• Fidelity-oriented simulators improve optical rendering and dynamic accuracy but use more compute; Isaac Sim offers ray-traced scenes, while Isaac Lab targets high-throughput RL training.The survey contrasts these with traffic flow-oriented simulators, which prioritize parallelization and computational efficiency for large-scale vehicle simulations.Source: 7) NVIDIA ISAAC-Sim · p. 5

C. Datasets

• Most driving datasets lack real-time driver actions and multi-vehicle collaboration data, while INTERACTION and AD4RL provide interactive or policy-trajectory resources for decision-making.AD4RL packages 19 aligned datasets and seven offline RL algorithms across three driving scenarios.Source: C. Datasets · p. 6

D. Competitions

• Competitions use simulator-based and real-vehicle tracks to compare MARL methods, including SMARTS, DuckieTown, nuPlan offline planning, and CARLA policy learning.The survey describes these competitions as standardized benchmarks and a unified framework for comparing performance.Source: D. Competitions · p. 6

III. REINFORCEMENT LEARNING PRELIMINARIES

• The preliminaries aim to give readers a unified mathematical language for understanding the rest of the paper.Source: §III · p. 6

A. Deep Reinforcement Learning

• An MDP models sequential decisions through states, actions, rewards, transitions, and discounting; Bellman value functions estimate expected subsequent returns.The Q-function estimates expected return for a given state-action pair, and deep RL represents value functions and policies with neural networks.Source: A. Deep Reinforcement Learning · pp. 6–7
• On-policy methods learn from the policy being improved, whereas off-policy methods can reuse data from other policies; RL also includes value-based, policy-gradient, and actor-critic approaches.The survey notes that off-policy methods are more sample efficient, policy gradients handle discrete and continuous action spaces, and actor-critic methods learn policy and value together.Source: A. Deep Reinforcement Learning · p. 7

• Markov games extend MDPs to interacting agents, representing a global state alongside individual actions and rewards.The joint action space is formed from the action spaces of all agents.Source: B. Multi-Agent Deep Reinforcement… · p. 7
• A Nash equilibrium is a joint policy no agent can unilaterally change to improve performance; such equilibria are generally not unique.The paper also distinguishes Pareto-optimal policies by whether another policy improves value without reducing any agent’s value.Source: B. Multi-Agent Deep Reinforcement… · p. 8
• Because vehicles observe only within sensor field-of-view and geographic limits, the paper uses a decentralized partially observable MDP formulation.Source: B. Multi-Agent Deep Reinforcement… · p. 8
• MARL settings may be competitive with zero-sum rewards, cooperative with shared rewards, or mixed with trade-offs between individual and group incentives.Source: B. Multi-Agent Deep Reinforcement… · p. 8

• State and action spaces grow exponentially with agent count, creating a computational challenge for policy learning in multi-agent systems.Source: C. Learning Schemes · p. 8
• CTDE uses centralized training with decentralized execution, while DTDE trains and executes agents independently; decentralized execution avoids requiring instantaneous joint information exchange.Centralized execution relies on a joint policy and unconstrained, instantaneous information exchange, an assumption the paper says is difficult to achieve in real multi-agent systems.Source: C. Learning Schemes · p. 8, Figure 3 · p. 8

1) CTDE Schemes:

• DTDE suits large-scale robotic decision-making, with communication networks or sensing used to compensate for limited access to other agents’ information.The cited approaches include online evolution and advanced value learning to address classical DTDE limitations of non-stationarity and low efficiency.Source: 2) DTDE Schemes · p. 9

D. Issues of MARL

• Synchronous policy updates in shared environments create non-stationarity; independent learners may overfit to others’ policies, while shared rewards, learning-rate changes, and corrected replay are proposed remedies.Source: 1) Non-Stationarity · p. 9

1) Non-Stationarity:

• Partial observations leave agents with incomplete or noisy state information, making inference and coordination harder when agents hold different information.Recurrent networks and communication are among the approaches discussed for addressing these constraints.Source: 2) Partial Observability · p. 9

3) Credit Assignment:

• Shared team rewards can produce lazy agents, while value decomposition and policy-gradient methods seek to assign global rewards to individual contributions.The survey notes that credit assignment remains difficult in large, heterogeneous robotic-control problems.Source: 3) Credit Assignment · p. 9

4) Scalability:

• Growing agent counts expand joint action spaces and can strain value networks; centralized strategies also demand substantial computation, memory, and bandwidth.Decentralized Q-learning and scalable actor-critic methods instead use local information, while communication designs can exchange nearby agents’ messages or policies.Source: 4) Scalability · pp. 9–10, 5) Communication Mechanism · p. 10

IV. STATE-OF-THE-ART METHODOLOGIES

• The survey selects representative recent MARL methods for multi-vehicle motion planning and control rather than attempting to cover every related study.It encourages researchers to report additional work to the survey’s website.Source: §IV · p. 10

A. Centralized Multi-Agent RL

• CTDE combines independent vehicle policies with a central computer that merges information, evaluates a global reward, and trains policies after credit assignment.PRIMAL uses parameter-sharing actor-critic networks and combines reinforcement learning with imitation learning; its authors also implemented it on vehicles in a factory mockup.Source: A. Centralized Multi-Agent RL · p. 10
• MADDPG established a generalizable CTDE approach, while value decomposition with CTDE has improved scalability and reduced non-stationarity effects in large multi-agent systems.The survey also describes MAPPO deployment on real robots using Duckietown-Gym training and domain adaptation.Source: A. Centralized Multi-Agent RL · pp. 10–11

B. Independent Policy Optimization

• Decentralized independent policy optimization reduces communication overhead and latency, and its growing scenario complexity and agent scale align with large-scale driving needs.The survey also notes deployment advantages where communication infrastructure is limited or unreliable.Source: B. Independent Policy Optimization · p. 11
• MAPPER learns faster than PRIMAL and handles dynamic obstacles more effectively in large-scale scenarios; G2RL requires no inter-agent communication and can extend to arbitrary agent counts.MAPPER uses decentralized actor-critic learning, while G2RL combines A* global guidance with a local DDQN planner.Source: B. Independent Policy Optimization · p. 11
• At free intersections, mixed egoistic–altruistic pairs are described as healthy, whereas two altruistic vehicles wait for one another and two egoistic vehicles crash.The survey cautions that occupancy-grid methods use discrete actions and structured settings that differ from real traffic systems.Source: Figure 4 · p. 11, B. Independent Policy Optimization · p. 11
• PRIMAL2 uses finer multi-channel observations to address deadlocks in dense, complex occupancy grids and generates shorter paths than its predecessor.Its decentralized training retains an A* planner with A3C and imitation learning.Source: B. Independent Policy Optimization · p. 11

C. Learning with Social Preference

• CoPO uses decentralized independent policies and a local coordination factor to optimize each agent’s average reward from nearby agents while balancing it with ego reward.Its global coordination learns the factor as a Gaussian distribution and seeks maximum accumulated rewards.Source: C. Learning with Social Preference · p. 12
• A learnable factor representing social preference outperforms independent learning and mean-field policy optimization.CoPO uses local coordination factors to balance egoistic and altruistic policy behavior.Source: C. Learning with Social Preference · p. 12

D. Safe and Trust-Worthy Learning

• Soft guarantees use reward penalties, probabilistic guarantees constrain optimization, and hard boundaries apply instantaneous corrections to unsafe actions.Soft guarantees depend on rewards accurately capturing safety; probabilistic policies lack a clear, stable safety boundary, while real-driving constraints can be instantaneous and deterministic.Source: D. Safe and Trust-Worthy Learning · p. 12
• A multi-agent model predictive shielding method is reported to provably guarantee safety for any MARL-learned policy.The survey places it among hard safety-boundary approaches that correct actions at risky times.Source: D. Safe and Trust-Worthy Learning · p. 12
• Centralized shielding methods can guarantee safety but do not scale to massive agent counts; hand-crafted control barrier functions become infeasible with complex nonlinear driving dynamics and agent interactions.Neural barrier approaches are presented as alternatives, including MDBC, reported to scale to 1024 vehicles and drones in particle environments.Source: D. Safe and Trust-Worthy Learning · p. 13

E. Methodological Summary

• Recent studies explore decentralized approaches with more complex continuous observations, but real-world experiments remain few and use discrete simulators and few agents.Newer studies also integrate methods from fields such as data compression and machine vision.Source: E. Methodological Summary · p. 14

V. OPEN QUESTIONS AND CHALLENGES

• The survey identifies multimodal sensing, long-tail scenario coverage, generalization across environments and agents, sim-to-real transfer, and action explanation as open challenges.These issues are illustrated as distinct challenges in Fig. 6.Source: §V · p. 14, Figure 6 · p. 14

A. Multi-modal Information

• Autonomous-driving policies must combine sensor modalities with different formats and properties; direct early fusion is simple but can lose geometric information and complicate feature extraction.Middle fusion uses separate modality encoders and concatenates their latent features; sensor fusion can also provide safety redundancy if one sensor fails.Source: 1) Multi-Sensor Fusion and Integration · p. 14

1) Multi-Sensor Fusion and Integration:

• Transformer-based attention is being applied to visual tasks to model contextual relationships across modalities and sequential or crossmodal data.The survey describes this as a recent direction building on advances in sequential modeling and audiovisual fusion.Source: 1) Multi-Sensor Fusion and Integration · p. 14
• Representations range from sensor outputs and 2D or 3D occupancy grids to shared bird’s-eye-view features, but the most effective MARL representation remains unresolved.High-resolution 3D grids can require unacceptable computation; representation design also involves balancing hardware costs and algorithmic performance.Source: 2) Learning Representations · p. 14

1) Long-tailed Problem:

• Rare, safety-critical scenarios may remain unseen during training, leaving autonomous vehicles vulnerable to long-tailed conditions.Common cases may be covered while policies fail to make safety-critical decisions in rare scenarios.Source: 1) Long-tailed Problem · p. 15
• Hand-crafted tests require expertise and may miss real-world complexity; dataset-driven scenarios, continual learning, and transfer learning are proposed to broaden coverage.Realistic long-tailed scenario coverage remains a long-term challenge.Source: 1) Long-tailed Problem · p. 15

2) Sim-to-Real Gap:

• Simulators may omit real-world variability, imperfect sensors and actuators, communication and processing delays, and difficult collision mechanics.These mismatches can leave systems performing well in simulation but struggling with real driving nuances.Source: 2) Sim-to-Real Gap · p. 15
• The section proposes richer scenarios, realistic physics, sensor noise and failure models, domain adaptation, and real-world data to narrow the sim-to-real gap.Domain adaptation adjusts a simulation-trained model for the real-world target domain.Source: 2) Sim-to-Real Gap · p. 15

1) Safety in Real World:

• Real-world safety validation remains inadequate: risky scenarios cannot all be sampled, specialized facilities are scarce, and many demonstrations use simplified platforms.The section calls for expanded testing infrastructure and regional and international academic collaboration.Source: 1) Safety in Real World · pp. 15–16
• Frequent agent data exchange exposes sensitive information to unauthorized access and tampering, motivating authentication, anomaly detection, and decentralized storage.The passage identifies location history as an example of vulnerable personal data.Source: 2) Private Information Concern · p. 16

D. Explainability

• Black-box neural controllers obscure decision reasoning, while interpretable controllers, state representations, post-hoc analyses, attention, and LIME offer explanation routes.Decision trees and rule-based systems require expert knowledge for fine system-level design.Source: D. Explainability · p. 16

A. Model-based MARL

• Model-based MARL can improve performance and explainability, but adds computational demands; centralized methods face scalability limits, while decentralized methods struggle with consensus.The section points to simplified models and efficient network architectures as research directions.Source: A. Model-based MARL · p. 16

B. Development of Offline Multi-Agent Datasets

• Offline MARL for autonomous driving needs system-level datasets, since individual-policy learning is self-centric and does not validate social preference or altruism.Data augmentation may expand these datasets across daily and long-tailed scenarios.Source: B. Development of Offline Multi-Agent… · pp. 16–17

C. Human-in-the-Loop Learning

• Human-in-the-loop learning incorporates operator feedback; future work must reduce intervention needs, extract useful strategy changes, and prevent catastrophic forgetting.The passage also describes a benchmark combining RLHF baselines with over 1,000 hybrid agents and expert trajectories.Source: C. Human-in-the-Loop Learning · p. 17

D. Language Models for Autonomous Driving

• LLMs support language-based inputs but are mostly used for simple decisions without visual information, whereas VLMs can interpret scenes and contextualize driving decisions.Language models are also proposed for simplifying interventions and compressing observations and communication for edge platforms.Source: D. Language Models for Autonomous… · p. 17