Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey

Ammar Haydari; Yasin Yilmaz

Source-linked AI summary

Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey

Ammar Haydari, Yasin Yilmaz

arXiv:2005.00935v1May 2, 2020cs.LGcs.MAeess.SPeess.SYstat.ML

TL;DR

Deep RL-based transportation applications require a consolidated view of methods for traffic signal control and other ITS tasks. The paper surveys these applications, formulations, algorithms, and environments, finding broad activity across traffic control and autonomous driving while identifying gaps in real-world validation, comparisons, and robustness.

Problem
Deep RL applications across ITS, particularly traffic signal control, are numerous but need to be gathered and discussed together for researchers in both fields.
Method
The paper comprehensively surveys RL and deep RL applications for ITS, emphasizing traffic signal control and categorizing studies by formulations, algorithms, control models, application types, and environments.
Results
The survey covers traffic signal control most extensively, alongside autonomous driving and other ITS applications, and reports findings on states, rewards, neural networks, and simulation-based evaluation.
Takeaways & Limitations
Deep RL-based ITS research spans adaptive traffic signals, autonomous driving, traffic flow control, and vehicle energy management, with traffic signal control as the most popular application.

Abstract

from arXiv · show

Latest technological improvements increased the quality of transportation. New data-driven approaches bring out a new research direction for all control-based systems, e.g., in transportation, robotics, IoT and power systems. Combining data-driven applications with transportation systems plays a key role in recent transportation applications. In this paper, the latest deep reinforcement learning (RL) based traffic control applications are surveyed. Specifically, traffic signal control (TSC) applications based on (deep) RL, which have been studied extensively in the literature, are discussed in detail. Different problem formulations, RL parameters, and simulation environments for TSC are discussed comprehensively. In the literature, there are also several autonomous driving applications studied with deep RL models. Our survey extensively summarizes existing works in this field by categorizing them with respect to application types, control models and studied algorithms. In the end, we discuss the challenges and open questions regarding deep RL-based transportation applications.

I. INTRODUCTION

The paper surveys deep reinforcement learning applications in intelligent transportation systems, especially traffic signal control, while also covering autonomous driving and related control settings. It organizes prior work by RL formulations, algorithms, application types, and evaluation environments, and identifies open deployment and benchmarking challenges.

Intelligent transportation systems combine artificial intelligence with transportation control to pursue safe, effective, and reliable mobility with minimal human intervention.
Urban congestion motivates adaptive traffic signal control, but the best approach for optimizing traffic lights remains an open research question.The cited introduction reports 97 hours and $1,348 in annual congestion cost for a US driver in 2018.
The paper is intended to inform both ITS researchers learning deep RL and deep RL researchers studying ITS applications.
The survey reviews deep RL applications for ITS, predominantly traffic signal control, and includes standard RL and multi-agent dynamic control because intersections are connected and distributed.
It provides theoretical background on RL and deep RL, summarizes traffic signal control studies in comparison tables, and covers deep RL applications in autonomous driving and other ITS areas.

II. RELATED WORK

Earlier work established RL and AI-based approaches for traffic signal control, while this survey addresses the previously identified absence of a dedicated deep RL survey for ITS. It reviews RL foundations, MDP formulations, learning paradigms, and related transportation research.

Related surveys: Prior surveys covered AI and RL for traffic signal control, broader ITS applications, and transportation management, public transportation, and autonomous vehicles.The cited literature includes surveys of TSC methods, multi-agent RL, broad ITS AI methods, and AI-based transportation applications.
Research gap: This paper identifies no literature survey dedicated to deep RL applications for ITS.The authors position the survey as filling a gap for both ITS and deep RL researchers.
RL foundations: RL control loops repeatedly observe system states, select actions, receive rewards, and update knowledge through environment interactions.The process supports trial-and-error learning toward an optimal control policy.
RL foundations: RL models sequential decision making as an MDP with states, actions, transitions, rewards, and a discount factor, seeking a policy that maximizes cumulative reward.Transportation MDPs are mostly episodic, with terminal points based on time or terminal state.
RL parameters: The discount parameter γ controls how strongly future rewards influence actions, while TD step size determines how many experience steps contribute to the target.Larger γ values increase dependence on future rewards; smaller values emphasize instantaneous reward.
RL foundations: Model-free RL divides into value-based and policy-based methods, with Q-learning and SARSA representing off-policy and on-policy value-based approaches.Q-learning maximizes Q-values during updates, whereas SARSA updates according to the policy derived from the Q-function.

B. Deep Reinforcement Learning

Deep RL uses neural networks to address large or continuous state-action spaces, with DQN, actor-critic, DDPG, and asynchronous methods representing major approaches. Stabilization mechanisms include target networks, experience replay, and related DQN improvements.

Deep Q-Network: Deep Q-Networks approximate Q-functions with neural networks because tabular methods cannot feasibly visit all states and actions in large or continuous problems.Original DQN processes raw image states with convolutional neural networks.
Deep Q-Network: Target networks and experience replay stabilize DQN learning with deep neural networks.Target networks reduce feedback-loop instability, while replay stores experiences and samples batches for training.
Deep Q-Network: Prioritized experience replay samples experiences according to assigned probabilities, giving higher ranking to samples with larger TD errors.The approach changes uniform sampling to proportional or rank-based prioritization.
Deep Q-Network: Double and dueling DQN separately address action overestimation and state-value or advantage estimation for discrete actions.Dueling networks estimate V π(s) and Aπ(s, a) separately before aggregating them into Q-values.
Actor-Critic Methods: Actor-critic methods combine an actor that controls behavior through a policy with a critic that evaluates actions using value estimation.Deep actor-critic variants use neural-network function approximators for both components.
Actor-Critic Methods: DDPG targets high-dimensional continuous control using a deterministic policy gradient and state-space value estimation.It differs from standard DQN, which is suited to low-dimensional discrete action spaces.
Asynchronous Methods: Asynchronous methods use multiple actor-learners and parallel computation to increase learning pace, while A2C provides a synchronous simpler variant.Separate networks update within an update period, and a shared main network is updated using actor and critic parameters.

C. Summary of Deep RL

Deep RL algorithms are selected according to application specifications, and DQN- and DDPG-based controllers continue to dominate RL-based ITS applications.

Summary: Q-learning-based DQN and actor-critic-based DDPG continue to dominate RL-based ITS controllers.The survey notes that algorithm preferences depend on application specifications.

IV. DEEP RL SETTINGS FOR TSC

Deep RL traffic signal control applications differ in intersection models, monitoring technologies, state-action representations, and other configurations. The survey organizes these settings around the control loop and simulator requirements.

TSC applications: Traffic signal control applications vary across intersection models, monitoring technologies, and RL state-action configurations.These differences reflect the application-oriented nature of existing work.
TSC control loop: A TSC control unit collects traffic state information, selects an action using the current policy, and receives a reward for that action.States may include queue length, vehicle position, and vehicle speed.
TSC settings: RL-based TSC simulator studies require formulations for state, action, reward, neural-network structure, and the selected traffic simulator.The survey discusses these configurations together with simulation environments used in the literature.

A. State

Traffic-signal RL studies use image-like, vehicle-based, and feature-based state representations, with multi-intersection models also incorporating neighboring signal and traffic information.

Image-like states represent intersections using raw RGB images or snapshots encoding vehicle positions and speeds.
Feature-based vectors summarize lane information such as queue length, waiting time, speed, phase duration, and vehicle counts.
Multi-intersection state definitions can include neighboring signals’ phases, vehicle counts, and average speeds.

B. Action

Traffic-signal RL actions commonly select discrete signal phases, while some multi-intersection approaches predict continuous phase durations within predefined bounds.

Single-intersection controllers commonly choose among green phases, retain or change the current phase, or use other action models.
Four common green-phase actions are NSG, EWG, NSLG, and EWLG for a single intersection.
Some approaches output a continuous duration for the next phase, constrained by predefined minimum and maximum phase durations.

C. Reward

Traffic-signal RL rewards are scalar functions of traffic data that evaluate actions, most commonly through waiting time, delay, or queue length.

Rewards evaluate action quality using scalar traffic-data functions that penalize or reward the agent.
Waiting time sums the durations vehicles are stopped, while delay compares waiting times across continuous green phases.
Queue length is calculated for each lane, and some studies use the U.S. Bureau of Public Roads congestion function.
Traffic-signal studies may use absolute, negative, or average traffic-data values as rewards.

E. Simulation environments

RL-based traffic-signal research is predominantly evaluated in simulation, spanning single-agent and cooperative multi-agent approaches for increasingly networked intersection control.

E. Simulation environments: Traffic-signal RL studies mostly use simulators because real-world experiments can be life-threatening, sometimes building simulations from real datasets.
E. Simulation environments: GLD was an early RL traffic simulator, while SUMO became the most popular open-source platform and allows users to modify environments.
E. Simulation environments: Single-agent RL uses one agent for an entire traffic-signal network, whereas multi-agent RL uses multiple agents across the network.
A. Standard RL applications: Single-intersection RL studies include SARSA and Q-learning models using queue length, delay, and variable-phase representations.
A. Standard RL applications: Large intersection networks motivate cooperative multi-agent learning because independently applying single-agent algorithms has limits.
A. Standard RL applications: Multi-agent studies cover coordination, multiobjective rewards, continuous-state actor-critic control, and neural-network-based approaches.

B. Deep RL applications

The survey organizes deep RL traffic signal control into single-agent and multi-agent approaches, covering architectures, environments, rewards, and benchmark comparisons. Reported methods include DQN variants, actor-critic models, recurrent networks, and graph-based coordination.

Overview: Table III outlines deep RL traffic signal control studies by algorithms, network structures, simulation environments, and benchmark comparisons.
Single agent deep RL: DQN-based TSC models use image-like or lane-level traffic representations with CNNs, while actor-critic methods target smoother and more stable convergence.Reported state representations include occupancy, speed, queue length, density, and DTSE vehicle-position encoding.
Single agent deep RL: 40%: asynchronous n-step Q-learning improves performance over fixed-time and actuated traffic controllers.
Multi-agent deep RL: Multi-agent TSC methods coordinate intersection agents through reward design, shared policies, recurrent networks, prioritized replay, fingerprints, and graph convolutions.Examples include 2-by-2 grids, heterogeneous intersections, and distributed agents representing network intersections.
Multi-agent deep RL: RNN-based coordination produces lower average delay than Q-learning and fixed-time control under both low and high traffic demand.
Single agent deep RL: DQN achieves a higher average reward per episode than A2C when deep learning is applied to multiple-intersection control.

VI. DEEP RL FOR OTHER ITS APPLICATIONS

Deep RL is applied beyond traffic signal control to autonomous vehicles, traffic-flow interventions, and vehicle energy management. The surveyed examples cover several control functions and application settings.

Deep RL applications in other ITS areas include ramp metering, lane changing, speed control, acceleration and deceleration, and intersection maneuvering.

A. Autonomous Driving

The survey covers deep RL for autonomous driving across discrete and continuous control, planning, mixed autonomy, traffic-wave control, and sim-to-real transfer. Applications are evaluated in environments including TORCS, VISSIM, SUMO, and a scaled physical city map.

Control models: Autonomous-driving controllers use DQN with recurrent networks for discrete actions and DDPG or actor-critic methods for continuous actions.Sallab et al. tested these approaches without replay memory on TORCS, reporting faster convergence.
Control models: Filtered-experience DQFE outperforms neural fitted Q-learning for autonomous driving on the TORCS simulator.
Control models: Deep RL is combined with safety control, Monte Carlo tree search, and planning-learning frameworks for autonomous vehicle control.
Driving behaviors: Mixed-autonomy studies address car following, lane changing, and driving policies using DDDP, DDQN, and prioritized experience replay.Evaluations include VISSIM and SUMO under different traffic densities.
Traffic-flow control: Multiple autonomous vehicles controlled by individual deep RL agents are used to target stop-and-go waves and increase traffic flow.Three action modes are evaluated across five cases: stop/go, sequential speed actions, and combined modes.
Sim-to-real transfer: The first learned-policy transfer from simulation to real-world experiments uses a scaled city map, multiple vehicles, and adversarial noise in state and action spaces.

C. Road Control

The survey covers deep RL applications for road control, including speed limits, pricing, ramp metering, simulator integration, and traffic signal control. It also identifies unresolved deployment, realism, benchmarking, sensing, and subsystem-integration challenges for ITS.

Road-control applications: Deep RL has been applied to road-control tasks including speed-limit control, toll pricing, and ramp metering.Reported applications target traffic flow, emissions, revenue, and performance objectives.
Simulation tools: Flow integrates RLlib with SUMO and Aimsun, enabling custom Python networks for testing complex ITS control problems.The interface supports deep RL experimentation across multiple traffic-control settings.
Deployment challenges: Real-world deployment remains an open challenge because existing RL-based ITS control outcomes are primarily evaluated in simulators.The survey notes a substantial gap between simulator-based applications and deployment, despite a city-scale policy-transfer example.
Open research directions: Open directions include more realistic and failure-tolerant simulations, clearer state definitions, and unified control across autonomous-vehicle subsystems.Suggested improvements include extreme scenarios, pedestrians, public transportation, noisy or partially observable states, and coordinated subsystem management.
Evaluation challenges: TSC studies lack satisfactory comparisons with multi-agent methods and actuated controllers, limiting evaluation against common real-world practice.Fixed-time controllers, standard RL, and DQN are commonly used as benchmarks for multiple intersections.