Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network

Juntao Gao; Yulong Shen; Jia Liu; Minoru Ito; Norio Shiratori

Source-linked AI summary

Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network

Juntao Gao, Yulong Shen, Jia Liu, Minoru Ito, Norio Shiratori

arXiv:1705.02755v1May 8, 2017cs.NI

TL;DR

Traffic signal control must respond to changing demand, while existing adaptive methods rely on human-crafted features that omit information in raw vehicle data. The paper uses deep reinforcement learning with a convolutional network, experience replay, and a target network to learn adaptive control from real-time observations. Simulations report up to 47% and 86% lower vehicle delay than longest queue first and fixed-time control, respectively.

Problem
Existing adaptive signal-control methods use human-crafted features that abstract raw traffic data and can omit useful information for control.
Method
A deep reinforcement-learning algorithm uses a convolutional neural network to extract features from real-time traffic data and applies experience replay and a target network for stability.
Results
Up to 47% lower vehicle delay than longest queue first and up to 86% lower than fixed-time control were reported in simulations.
Takeaways & Limitations
The algorithm learns a fair traffic signal-control policy in which no road's vehicles wait too long for passage.

Abstract

from arXiv · show

Adaptive traffic signal control, which adjusts traffic signal timing according to real-time traffic, has been shown to be an effective method to reduce traffic congestion. Available works on adaptive traffic signal control make responsive traffic signal control decisions based on human-crafted features (e.g. vehicle queue length). However, human-crafted features are abstractions of raw traffic data (e.g., position and speed of vehicles), which ignore some useful traffic information and lead to suboptimal traffic signal controls. In this paper, we propose a deep reinforcement learning algorithm that automatically extracts all useful features (machine-crafted features) from raw real-time traffic data and learns the optimal policy for adaptive traffic signal control. To improve algorithm stability, we adopt experience replay and target network mechanisms. Simulation results show that our algorithm reduces vehicle delay by up to 47% and 86% when compared to another two popular traffic signal control algorithms, longest queue first algorithm and fixed time control algorithm, respectively.

I. INTRODUCTION

Adaptive signal control responds to changing traffic demand, but existing approaches rely on human-crafted features that omit useful raw traffic information. The paper proposes deep reinforcement learning to extract features directly from real-time data for improved signal decisions.

Motivation: Fixed-time signal control can become outdated because its offline timing settings do not reflect changing traffic demands.Improving signal-control efficiency is presented as a relatively easy alternative to expanding infrastructure.
Motivation: Adaptive traffic signal control adjusts signal timing according to real-time traffic demand and has been shown to reduce congestion.Prior approaches include back-pressure and reinforcement-learning methods.
Limitations of prior work: Existing adaptive methods base decisions on human-crafted features such as vehicle queue length and average vehicle delay.These features abstract raw data such as vehicle positions and speeds.
Limitations of prior work: Human-crafted features can ignore useful information, such as vehicles approaching soon but not yet in a queue, leading to suboptimal signal control.The paper specifically contrasts queue length with richer raw traffic observations.
Proposed approach: The proposed method uses deep reinforcement learning to extract useful features from raw real-time traffic data and learn an adaptive signal-control policy.A deep convolutional neural network processes vehicle position, speed, and traffic signal state; experience replay and a target network address instability.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The paper formulates adaptive traffic signal control as a reinforcement learning problem in which an agent observes intersection traffic, selects signal actions, and receives rewards based on vehicle staying time. The intersection state captures vehicle position, vehicle speed, and signal status, while the long-run objective is to maximize cumulative discounted reward.

Reinforcement learning formulation: The agent interacts with a four-way intersection in discrete time by observing state St, selecting and actuating action At, then receiving reward Rt and a new state St+1.The interaction sequence is St, At, Rt, St+1, with rewards guiding the agent toward reducing vehicle staying time.
Intersection state: The intersection state consists of vehicle-position and vehicle-speed matrices for four roads plus a two-element traffic-signal vector.Position and speed are represented over discrete lane cells; L=[1,0] denotes west-east green and L=[0,1] denotes north-south green.
Agent action: The agent chooses between west-east and north-south green phases, each lasting a fixed interval, with yellow and left-turn transition phases when switching actions.Consecutive identical actions keep the current signal settings unchanged.
Reward: The reward compares total vehicle staying time at the beginning and end of each green-light interval, rewarding decreases and penalizing increases.Staying time is measured for vehicles remaining at the intersection, and aggregate values Wt and W′t support the comparison.
Agent goal: The agent seeks a policy that maximizes cumulative future reward rather than only the immediate reward, with γ controlling the weight assigned to future rewards.γ=0 represents a shortsighted policy, while values approaching 1 place greater weight on future rewards.

III. DEEP REINFORCEMENT LEARNING ALGORITHM FOR TRAFFIC SIGNAL CONTROL

The paper approximates optimal traffic-signal Q-values with a deep neural network that extracts features from raw traffic data. Experience replay and a slowly updated target network support training stability.

III. DEEP REINFORCEMENT LEARNING ALGORITHM FOR TRAFFIC SIGNAL CONTROL: The algorithm approximates optimal Q-values with a parameterized DNN whose learned parameters are trained from raw traffic data.The network output estimates Q(s, a; θ) ≈ Q∗(s, a).
DNN Structure: The DNN receives intersection state S_t and outputs estimated Q-values for all available actions.Its input combines vehicle-position, vehicle-speed, and traffic-signal-state information, processed through convolutional and fully connected layers.
DNN Training: Training uses replayed interaction experiences to form input-target pairs for minimizing mean squared error.The target uses the immediate reward plus discounted next-state Q-values from a separate target network.
DNN Training: The agent randomly samples minibatches of 32 experiences from finite replay memory and updates DNN parameters with RMSProp.When replay memory is full, the oldest data are discarded.
Stability Mechanisms: After updating the main network, the agent soft-updates target-network parameters using update rate β, with β ≪ 1.The paper adopts experience replay and target network mechanisms to improve algorithm stability.
Optimal Action Policy: The policy must balance exploiting estimated Q-values against exploring actions because the agent has observed only limited and continuously changing states.The paper frames this as an exploration–exploitation trade-off.

IV. SIMULATION EVALUATION

The simulation evaluation follows an episodic deep-reinforcement-learning training procedure in which the agent observes states, selects mostly greedy actions with ε-greedy exploration, and updates both networks from stored experiences.

IV. SIMULATION EVALUATION: Each episode begins from an initialized intersection state and action, after which the agent advances through simulated time steps.The algorithm runs for episodes indexed from 1 to N.
Agent Interaction: At each new time step, the agent observes the current intersection state and selects an action using ε-greedy policy.It chooses arg max_a Q(S_t, a; θ) with probability 1 − ε and a random action with probability ε.
Agent Interaction: If the selected action repeats the previous action, the current traffic-signal settings remain unchanged.Otherwise, the algorithm proceeds through transition-signal handling before executing the selected action.
DNN Training: At the end of each time step, the agent observes the reward and next state, stores the experience, and samples 32 experiences from replay memory.The sampled data are used to form training inputs and targets.
DNN Training: The main DNN parameters are updated with RMSProp, followed by an update of target-network parameters.These updates occur after the training data are formed.

A. Simulation Settings

The evaluation uses SUMO to simulate a four-way intersection with heterogeneous traffic arrivals and specified road, vehicle, signal, and training settings.

Simulation Settings: The experiments simulate intersection traffic and signal control with the open-source Simulation of Urban MObility (SUMO) simulator.The paper provides detailed simulation settings for the modeled intersection.
Intersection: The modeled intersection has four roads with four lanes each, 500-meter roads, 160-meter road segments, and 8-meter cells.The speed limit is 19.444 m/s, with 5-meter vehicles and a 2.5-meter minimum gap.
Traffic Arrival Process: Vehicles arrive randomly, select routes in advance, and follow route-specific Bernoulli arrival rates P_ij.Roads 0 and 2 are designated busy, while roads 1 and 3 are less busy to represent heterogeneous demand.
Traffic Signal Timing: Traffic-signal timing uses a 10-second green-light interval and a 6-second yellow-light interval.The signal-actuation rules are defined in the agent-action description and illustrated in the paper’s figures.
Agent Parameters: The agent trains for 2,000 episodes, each representing 1.5 hours of traffic, with γ = 0.95, β = 0.001, and replay capacity covering 200 episodes.The experiment also fixes ε at 0.1 and the RMSProp learning rate at 0.0002.

B. Simulation Results

Simulations show that training reduces vehicle staying time and delay, while experience replay and target network mechanisms stabilize the learned policy. Compared with fixed-time and longest-queue-first control, the algorithm achieves lower delay under varying traffic demand.

Training stability: Average vehicle staying time decreases rapidly during training and remains stable at small values after 800 episodes.This indicates convergence to a good action policy and effective stabilization from experience replay and the target network.
Per-road delay: Average vehicle delay falls to around 90.5, 107.2, 91.5, and 109.4 seconds for roads 0, 1, 2, and 3, respectively.The stable values indicate adaptive control and a fair policy across roads with different vehicle arrival rates.
Algorithm comparison: On busy roads 0 and 2, delay is reduced by up to 86% versus fixed-time control and up to 47% versus longest-queue-first control.These comparisons use the same simulation settings while varying vehicle arrival rates through parameter ρ.
Demand sensitivity: As traffic demand increases, the proposed algorithm’s delay rises only slightly on busy roads but increases slightly on less busy roads 1 and 3.Fixed-time delay increases exponentially with demand, whereas longest-queue-first control ignores vehicles that are not yet queued.

V. RELATED WORK

Related work applies deep reinforcement learning and target networks to adaptive traffic signal control, but differs in feature representation, stability, fairness, and intersection modeling.

Prior deep reinforcement learning methods: Li et al. used a deep stacked autoencoder neural network to estimate optimal Q-values from queued-vehicle counts.Their reward was based on the queue difference between west-east and north-south traffic.
Prior deep reinforcement learning methods: Genders et al. used convolutional neural networks to approximate optimal Q-values from vehicle position, speed, and recent signal information.Their method used cumulative vehicle delay as reward and a target network to estimate target Q-values.
Positioning of this paper: The proposed work distinguishes itself by improving stability and learning a fair policy for common intersections with left-turn waiting areas.The related-work discussion identifies missing fairness and absent left-turn waiting areas as drawbacks of earlier work.
Stability mechanisms: Prior work addressed instability with a separate target network whose parameters are updated only periodically during training.The hard-update scheme copies DNN parameters into the target network every M time steps.

VI. CONCLUSION

The paper proposes deep reinforcement learning for adaptive traffic signals, using raw real-time traffic data and stabilization mechanisms to reduce congestion and delay fairly across roads.

Conclusion: The proposed algorithm extracts useful features from raw real-time traffic data with a deep convolutional neural network and learns an optimal signal policy.Experience replay and a target network improve stability by helping the algorithm converge to a good policy.
Conclusion: Simulation results show significantly lower vehicle delay than longest-queue-first and fixed-time control, while no road has vehicles waiting too long.The conclusion characterizes the learned policy as fair across roads.