Source-linked AI summary
Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning
Praveen Palanisamy
TL;DR
Autonomous driving requires adaptive learning in environments with multiple intelligent agents, where standard Markov assumptions may fail. The paper formulates connected driving as Partially Observable Markov Games, introduces a taxonomy and MACAD-Gym platform, and reports agents learning vehicle-control policies from raw camera observations in a partially observable intersection. The formulation remains computationally difficult except under simplified structures and assumptions.
Problem
Driving involves complex interactions among intelligent agents in non-stationary environments, challenging single-agent reinforcement-learning formulations and exhaustive rule-based programming.
Method
The paper uses Partially Observable Markov Games, a taxonomy of multi-agent environments, and the extensible MACAD-Gym platform for connected autonomous-driving research.
Results
MACAD-Agents learned reasonably good policies for three vehicle agents to cross a stop-sign-controlled urban intersection without collisions within the imposed time limit.
Takeaways & Limitations
The platform supports research on integrated sensing, perception, planning, and control for connected autonomous driving under realistic partially observable multi-agent settings.
Takeaways & Limitations
The POSG formulation is not computationally tractable except under simplified special structures and assumptions such as two-player zero-sum POSGs.
Abstract
from arXiv · showhide
The capability to learn and adapt to changes in the driving environment is crucial for developing autonomous driving systems that are scalable beyond geo-fenced operational design domains. Deep Reinforcement Learning (RL) provides a promising and scalable framework for developing adaptive learning based solutions. Deep RL methods usually model the problem as a (Partially Observable) Markov Decision Process in which an agent acts in a stationary environment to learn an optimal behavior policy. However, driving involves complex interaction between multiple, intelligent (artificial or human) agents in a highly non-stationary environment. In this paper, we propose the use of Partially Observable Markov Games(POSG) for formulating the connected autonomous driving problems with realistic assumptions. We provide a taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation. As our main contributions, we provide MACAD-Gym, a Multi-Agent Connected, Autonomous Driving agent learning platform for furthering research in this direction. Our MACAD-Gym platform provides an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL- based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings. We also share the MACAD-Agents that were trained successfully using the MACAD-Gym platform to learn control policies for multiple vehicle agents in a partially observable, stop-sign controlled, 3-way urban intersection environment with raw (camera) sensor observations.
1 Introduction
Autonomous driving must handle complex interactions among intelligent agents that cannot be exhaustively encoded as rules. The paper addresses this challenge with a multi-agent, connected learning framework and the MACAD-Gym platform.
- Deep RL offers a scalable framework for adaptive driving solutions learned through interaction with the environment.
- Single-agent learning often ignores other agents, whereas multi-agent learning explicitly models their interactions.
- Connectivity is increasingly important for safe, efficient, scalable autonomous driving beyond geo-fenced areas.
- The paper formulates connected autonomous driving as Partially Observable Markov Games and proposes a taxonomy of multi-agent learning environments.
- MACAD-Gym provides extensible connected-driving simulation environments for integrated sensing, perception, planning, and control research.
- MACAD-Agents are starter agents for learning experiments in realistic multi-agent driving settings.
2 Connected Autonomous Driving as Partially Observable Markov Games
The paper models connected autonomous driving as a partially observable multi-agent problem because adaptive agents make the environment non-stationary. Its formulation represents joint states, actions, observations, transitions, rewards, communication, and local decision-making.
- Treating other adaptive agents as part of a single-agent environment violates stationary and Markov assumptions because the environment becomes non-stationary.
- Partially Observable Markov Games extend Markov Games to driving settings where agents cannot realistically observe the complete environment state.
- The formulation defines finite agents, states, joint actions, joint observations, transitions, and agent-specific rewards.
- At each time step, agents receive local observations, choose actions through policies, and receive rewards from their reward functions.
- Communication can share local information and make solutions more tractable as the number of agents increases, while introducing integrity and trust concerns.
- The driving world is assumed Markov given all actor states, actions, and environmental state, although it is non-Markov from each agent’s viewpoint.
- Agents act from local observations augmented by partial information about other agents and otherwise unobservable environment state.
3 Multi-Agent Connected Autonomous Driving Platform
The platform organizes connected autonomous driving environments by task, actor type, communication, observability, timing, and adversarial conditions. This taxonomy supports varied multi-agent driving research and systematic environment naming.
- CAD tasks span sensing, perception, planning, and control, either in integrated end-to-end systems or as isolated tasks.
- The platform classifies environments by task nature, agent nature, and environment nature to support varied CAD problems.
- Task nature: Independent environments model self-interested actors with separate objectives and support non-communicating agents.
- Task nature: Cooperative environments support agents that communicate and learn joint policies intended to reduce congestion, collisions, and inefficient traffic flow.
- Task nature: Competitive and mixed environments cover adversarial driving and strategies that alternate between cooperation and competition.
- Agent nature: Agent categories include homogeneous or heterogeneous actors and communicating or non-communicating actors.
- Environment nature: Realistic driving environments are partially observable, and connectivity can make them more tractable by increasing available information.
- Environment nature: The platform also distinguishes synchronous, asynchronous, and adversarial environments, including communication perturbations and adverse weather.
4 Multi-Agent Deep Reinforcement Learning For Connected Autonomous Driving
The section formulates multi-agent driving objectives around other agents’ policies and contrasts centralized, decentralized, and shared-policy learning architectures. It also describes connectivity mechanisms that can provide shared parameters, observations, and experiences.
- Multi-agent objectives: Each agent maximizes expected long-term reward, but its objective depends on the policies of the other agents.This dependence distinguishes the multi-agent formulation from the single-agent setting.
- Multi-agent objectives: The optimal policy is a best response conditioned on the policies of the other agents.
- Global objectives: A globally optimal policy for N homogeneous driving agents maximizes the combined objectives of all agents.
- Learner architectures: Centralized learners use a decoupled actor-learner architecture to learn a globally optimal driving policy.
- Learner architectures: Decentralized learners let each driving agent independently optimize its local objective, including settings with human-driven vehicles.
- Connectivity mechanisms: Shared policies reduce non-stationarity when all vehicles follow the same policy, while shared parameters, observations, and experiences provide additional connectivity mechanisms.The paper relates shared-policy learning to controlled autonomous fleet operations and describes observation sharing as reducing partial observability.
5 Experiments and Conclusion
The paper reports a starter multi-agent experiment in MACAD-Gym and concludes that the platform supports deep-RL research for partially observed connected autonomous driving.
- Conclusion: MACAD-Gym is presented as an open-source platform for studying connected autonomous driving with a POSG formulation and categorized multi-agent environments.The paper also describes starter MACAD-Agents and a future benchmark for evaluating learning-based CAD algorithms.
- Experiments: The experiment trained three agents to learn independent vehicle-control policies from high-dimensional raw sensory data in a partially observed, multi-agent driving environment.The agents used camera observations in a stop-sign-controlled urban intersection.
- Method: DQN uses a neural network to represent the action-value function and optimizes its parameters by minimizing mean squared error against a Q-learning target.The replay memory stores experience tuples used in training.
- Method: The multi-agent setting motivates modeling other intelligent agents explicitly rather than treating them as part of a stationary environment.The supplied method passages contrast single-agent and multi-agent value-function formulations and describe independent learning extensions.
C.1 Environment description
The environment description defines a three-actor, homogeneous, non-communicating, independent, partially observable intersection scenario in Town3.
- Environment identity: HomoNcomIndePOIntrxMASS3CTWN3-v0 denotes a homogeneous, non-communicating, independent, partially observable multi-agent intersection environment in Town3.The naming convention encodes the environment and agent characteristics together with the scenario and version.
- Actors: The environment contains three actors, each assigned a vehicle scenario and associated control configuration.The configuration includes vehicle type, planner settings, collision termination, reward function, sensors, and camera settings.
C.1.2 Goals
The three actors have distinct intersection-crossing goals, and success requires completing the assigned route without collisions or road infractions within the episode time limit.
- Actor goals: Car3 goes straight, car1 turns left, and car2 goes straight across the intersection.The actors are identified as a maroon sedan, red cola van, and blue minivan, respectively.
- Success criteria: All agents must avoid collisions and road infractions while reaching their goal state within the one-episode time limit.The goal coordinates are specified for actors in the CARLA Town03 map.
- Control interface: Vehicle control is trained through a discrete action space mapped to normalized steering, throttle, and brake commands.The mapping is provided as a table for training vehicle-control policies.
C.2.1 Reward Function
The reward function combines progress, speed, safety, and lane-position terms while allowing optional shaping for cooperation, competitiveness, and stochastic environmental conditions.
- Reward definition: Each agent receives a reward conditioned on its state, environment state, and action.The environment state captures stochastic driving conditions such as weather and noisy communication channels.
- Reward terms: The reward is a weighted sum of distance traveled toward the goal, speed, collision damage, sidewalk intersection, and opposing-lane intersection.Distance and speed are expressed in kilometers and kilometers per hour, while sidewalk and opposing-lane terms lie in [0, 1].
- Reward shaping: Optional coefficients can encourage or discourage cooperation and competitiveness and shape rewards under stochastic changes in the world state.The cooperation or competition coefficient is α, while β shapes rewards under environmental stochasticity.
C.3 Agent algorithm
The MACAD-Agents use an IMPALA-based architecture with convolutional policy networks, and the experiment also evaluates a shared-weights configuration. Training performance is visualized through aggregate and per-car episodic rewards.
- The MACAD-Agents use the IMPALA architecture implemented with RLLib.
- The policy networks use a standard deep CNN followed by a fully connected layer.The convolutional configuration is [[32, [8], 4], [64, [4], 2], [64, [3], 1]].
- Figure 4 plots cumulative mean episodic rewards together with mean episodic rewards for car1, car2, and car3.The figure marks sampled states at training iterations 1.1M and 4.6M.
- In the shared-weights configuration, agents share a 128-dimensional fully connected layer before the final action-logits layer.
C.4 Results
The multi-agent system’s performance is presented in Figure 4. The section directs evaluation of system performance to that figure.
- The multi-agent system’s performance is shown in Figure 4.
- Figure 4 is the section’s referenced performance visualization.
- The reported results are presented through the multi-agent system’s performance figure.