Source-linked AI summary
Using a Deep Reinforcement Learning Agent for Traffic Signal Control
Wade Genders, Saiedeh Razavi
TL;DR
Traffic signal control must improve congested transportation systems while making fuller use of increasingly available traffic data. The paper proposes a deep reinforcement-learning agent using the information-dense DTSE with a convolutional network, and reports lower delay, queue length, and travel time than a comparison agent. Its policy is not evaluated for fairness, which the authors identify as a limitation and future research topic.
Problem
Existing traffic signal control systems using new sensors and reinforcement learning do not fully exploit available traffic data, while congestion strains transportation infrastructure.
Method
The paper develops DQTSCA, which uses the DTSE as input to a deep convolutional neural network trained with reinforcement learning in a traffic microsimulator.
Results
The DQTSCA reduces average cumulative delay by 82%, average queue length by 66%, and average travel time by 20% compared with the STSCA.
Takeaways & Limitations
Deep learning can be applied to traffic signal control with improved performance compared with traditional methods.
Takeaways & Limitations
The research does not consider fairness, which may conflict with optimizing delay or throughput and requires an appropriate reward-function balance.
Abstract
from arXiv · showhide
Ensuring transportation systems are efficient is a priority for modern society. Technological advances have made it possible for transportation systems to collect large volumes of varied data on an unprecedented scale. We propose a traffic signal control system which takes advantage of this new, high quality data, with minimal abstraction compared to other proposed systems. We apply modern deep reinforcement learning methods to build a truly adaptive traffic signal control agent in the traffic microsimulator SUMO. We propose a new state space, the discrete traffic state encoding, which is information dense. The discrete traffic state encoding is used as input to a deep convolutional neural network, trained using Q-learning with experience replay. Our agent was compared against a one hidden layer neural network traffic signal control agent and reduces average cumulative delay by 82%, average queue length by 66% and average travel time by 20%.
I. INTRODUCTION
The paper addresses congestion by proposing a deep reinforcement-learning traffic signal control agent that uses available traffic data with less abstraction. It introduces DQTSCA and situates the work within reinforcement-learning-based signal control.
- I. INTRODUCTION: Traffic congestion, delays, and emissions increase as vehicle ownership raises demand beyond road infrastructure capacity.The paper contrasts costly infrastructure expansion with improving existing infrastructure and traffic signal controllers.
- I. INTRODUCTION: The traffic signal control problem is choosing the optimal signal phase and sequence for a given intersection traffic state.
- I. INTRODUCTION: The proposed TSCA is a deep artificial neural network trained with reinforcement learning to develop an optimal control policy.
- I. INTRODUCTION: The DQTSCA models the action-value function with a deep convolutional neural network trained through reinforcement learning in the SUMO traffic microsimulator.
- I. INTRODUCTION: The DTSE provides an information-dense traffic representation intended to let the convolutional network extract useful features and higher-level state representations.
II. LITERATURE REVIEW
The literature review identifies limitations in prior traffic-signal reinforcement-learning systems: abstracted state representations, constrained action spaces, and simplified vehicle-generation assumptions. The paper responds by pursuing richer state information and greater action autonomy.
- II. LITERATURE REVIEW: Prior state definitions often omit relevant information such as moving vehicles, lane, queue position, location, and speed.
- II. LITERATURE REVIEW: Fixed sequences of signal phases limit the controller’s action space by imposing assumptions about the optimal phase order.
- II. LITERATURE REVIEW: The proposed action-space definition seeks to give the traffic signal control agent greater autonomy for learning an optimal policy.
- II. LITERATURE REVIEW: Previous studies rely on computer simulations, and most assume vehicle generation follows a Poisson process based on the negative exponential distribution.
- II. LITERATURE REVIEW: The paper argues that empirical research supports other distributions as more accurate models for different real-traffic vehicle-generation flow rates.
III. PROPOSED SYSTEM
The proposed traffic signal control problem is formulated in reinforcement-learning terms by defining traffic states, signal actions, and rewards.
- III. PROPOSED SYSTEM: Solving traffic signal control with reinforcement learning requires defining a state space S, action space A, and reward R.
A. State Space
The DTSE discretizes approaching lanes into cells and combines vehicle presence, normalized speed, and current signal phase. It is designed to retain more traffic information than highly abstract state features.
- A. State Space: The DTSE discretizes a lane length l from the stop line into cells of length c, with c affecting the system’s behavior.Very large cells can lose individual vehicle dynamics; the paper selects c simply for evaluation.
- A. State Space: The DTSE contains vectors for vehicle presence, vehicle speed, and the current traffic signal phase.
- A. State Space: Boolean DTSE elements encode vehicle presence, while real-valued elements encode speed normalized by the speed limit.
- A. State Space: The DTSE is intended to retain approaching vehicles’ speed and position information rather than only queue counts or vehicle flow.
- A. State Space: The phase vector uses one for the current phase and zero for all other phases.
- A. State Space: The DTSE is sensor agnostic: its state information may be gathered through vision, wireless communication, or other means.
B. Action Space
The agent chooses among four traffic-signal phase configurations, while safety requires intermediate yellow and all-red phases before some transitions are enacted.
- Each action names the approaching lanes’ signal phases using a compass direction for human comprehension.
- The action space contains four phase configurations: NSG, EWG, NSLG, and EWLG.
- Safety transitions may insert intermediate signal configurations between the current phase and the selected action.These transitions depend on both the current phase and chosen action.
C. Reward
The agent receives rewards after actions, with the reward defined as the change in cumulative vehicle delay between actions.
- The agent seeks a state-action policy that maximizes cumulative long-term reward after observing traffic state and performing an action.
- The reward is the change in cumulative vehicle delay between actions.An increase in delay produces a negative reward, while a decrease can produce a positive reward.
- Traffic signal control rewards can also be based on queued vehicles, cumulative delay, or vehicle throughput.
D. Agent
The proposed agent combines the information-rich DTSE with a deep convolutional Q-network trained by Q-learning, experience replay, and decreasing epsilon-greedy exploration.
- Agent design: The DTSE represents traffic with less abstraction and more information than queue length or average flow.This representation is intended to preserve information that expert-crafted features may discard.
- Agent design: The DQTSCA combines separate convolutional networks for real-valued and Boolean DTSE inputs with the current traffic signal phase.
- Q-learning: Q-learning trains the convolutional network to approximate action values and select the highest-valued action for the current state.
- Training: Decreasing epsilon-greedy exploration shifts from random actions toward value-maximizing actions as training progresses.The exploration probability begins at 1.0 and declines across training epochs.
IV. EXPERIMENTAL SETUP AND TRAINING
Experiments used SUMO to train and compare a deep traffic-signal agent under simulated traffic generated with flow-rate-specific distributions and experience replay.
- Experimental setup: SUMO v0.22 provided the traffic microsimulation environment and Python API used to implement the DQTSCA.Keras, Theano, NumPy, and SciPy supported the neural-network and simulation implementation.
- Experimental setup: The simulated intersection has four 750-meter approaches from north, south, east, and west, connected to four outgoing lanes.
- Traffic generation: Vehicle headways use probability distributions selected for different flow rates, with distinct treatment for turning and through traffic.The supplied passages state that empirical research supports different distributions for different traffic flow rates.
- Training: Experience replay collects transitions from parallel traffic simulations, stores them in memory, and periodically samples batches for training.The replay memory uses batch size 16, maximum size 500,000, and minimum size 50,000.
- Baseline: The shallow comparison agent uses one hidden layer with 64 sigmoid neurons and four linear output neurons, with queued-vehicle and phase-vector state inputs.
V. RESULTS AND DISCUSSION
During training, the DQTSCA improves across traffic metrics as it shifts from exploratory to exploitative actions and exhibits more convergent behavior. Compared with the STSCA, it reduces cumulative delay, queue length, and travel time, while throughput remains unchanged; fairness is not considered.
- Training behavior: As training progresses, exploratory actions give way to exploitative actions, improving throughput, queue length, travel time, and cumulative delay while reducing performance variance.Early exploration produces low throughput, queues, delays, and negative reward; later exploitation improves all four metrics.
- Training behavior: After training, rewards become stable and smaller in magnitude than during initial random exploration.The average and standard deviation change from (−347, 2 220) before training to (−0.485, 59.6) after training.
- Agent comparison: 82% lower average cumulative delay, 66% lower average queue length, and 20% lower average travel time are achieved by DQTSCA versus STSCA.The comparison uses the last 100 training epochs, when both agents take exploitative actions more than 93% of the time.
- Agent comparison: Throughput is unchanged between DQTSCA and STSCA, making it the only evaluated traffic metric where the agents perform equally.The paper identifies a throughput reward function as a direction for future investigation.
- Limitations: The study does not consider fairness of the learned traffic-signal policy, although fairness may conflict with minimizing delay or maximizing throughput.The authors suggest investigating reward functions that balance fairness and optimality.
VI. CONCLUSION
The DQTSCA was developed and tested in a traffic microsimulator, with results indicating improved performance compared to traditional methods. The conclusion also identifies dynamic yellow and red phase control, broader network complexity, and multiple agents as future directions.
- VI. CONCLUSION: The DQTSCA was developed and tested in a traffic microsimulator.The reported evaluation used traffic-simulation training and testing.
- VI. CONCLUSION: The results show deep learning can be applied to traffic signal control with improved performance compared to traditional methods.The supplied conclusion states the comparison at a high level without providing metric values here.
- VI. CONCLUSION: Future work could extend the agent’s control to yellow and red phases by changing the reward function to reward vehicle deceleration and non-traversal.The current agent lacks direct control of these phases; the proposed reward change is stated as a hypothesis.
- VI. CONCLUSION: The DTSE may support controlling intersections with different lane configurations without retraining.The passage gives a four-lane-to-two-lane example by setting elements for missing lanes to zero.
- VI. CONCLUSION: Additional research should increase traffic-network complexity and apply the DTSE and deep architecture to multiple TSCAs.These extensions are identified as future work.