Source-linked AI summary
Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning
Seyed Sajad Mousavi, Michael Schukat, Enda Howley
TL;DR
Traffic signal control must handle congestion and complex, high-dimensional observations. The paper develops deep policy-gradient and value-function agents that use graphical simulator snapshots to choose signals. In SUMO simulations, both approaches produced promising, stable control policies and reduced queueing and delay relative to a shallow neural-network baseline.
Problem
Traffic signal control requires adaptive decisions in complex environments with large or continuous state spaces, limiting classical reinforcement-learning methods.
Method
The paper combines deep convolutional networks with policy-gradient and action-value reinforcement-learning agents that process visual intersection observations to produce control signals.
Results
The proposed models significantly outperform the SNN baseline, reducing average cumulative delay by 67% and queue length by 72% with policy gradients, and by 68% and 73% with value-function learning.
Takeaways & Limitations
The methods find stable control policies from raw visual simulator input and reduce queue length and cumulative delay in the evaluated intersection setting.
Takeaways & Limitations
The methods were developed and tested in a small application; extending them to many intersections would require multi-agent learning to handle coordination.
Abstract
from arXiv · showhide
Recent advances in combining deep neural network architectures with reinforcement learning techniques have shown promising potential results in solving complex control problems with high dimensional state and action spaces. Inspired by these successes, in this paper, we build two kinds of reinforcement learning algorithms: deep policy-gradient and value-function based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The policy-gradient based agent maps its observation directly to the control signal, however the value-function based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Our methods show promising results in a traffic network simulated in the SUMO traffic simulator, without suffering from instability issues during the training process.
1. INTRODUCTION
The paper targets adaptive traffic signal control using deep reinforcement learning, motivated by growing congestion and complex state spaces. It proposes policy-gradient and value-function agents using visual simulator observations.
- Urban growth makes effective traffic management important, with congestion reduction also linked to time, financial, and CO2 savings.
- Adaptive signal control updates timing automatically from current traffic demand, queue lengths, and flow fluctuations.
- Large state and action spaces make real-world reinforcement-learning applications such as traffic management increasingly complex.
- The study proposes deep policy-gradient and action-value methods to optimize traffic flow and address training oscillations and instabilities.
- The proposed controllers combine reinforcement learning with a deep convolutional network that processes camera-like observations at an isolated intersection.
2. RELATED WORK
Prior traffic-signal research includes reinforcement-learning and deep-learning approaches, but the cited simulation testbeds were not mature enough for realistic comparison.
- Earlier traffic signal studies used reinforcement learning and reported promising results.
- Previous work used stacked autoencoders to estimate signal-phase Q-values from vehicle speed and queue-length measures.
3. BACKGROUND
The background frames traffic control as reinforcement learning over large or continuous spaces and explains how deep networks approximate policies and value functions. It introduces policy-gradient methods as a direct alternative to value estimation.
- 3.1 Reinforcement Learning: Reinforcement learning models an agent interacting with an environment through states, actions, transitions, rewards, and a discount factor.
- 3.1 Reinforcement Learning: The learning objective is an optimal policy that maximizes expected cumulative discounted reward over time.
- 3.1 Reinforcement Learning: Traffic-light optimization has a continuous state space, motivating function approximation or hierarchical reinforcement learning instead of tabular methods.
- 3.2 Deep learning and Deep Q-learning: Deep neural networks learn hierarchical representations directly from raw high-dimensional data through automated feature extraction.
- 3.2 Deep learning and Deep Q-learning: DQN combines a neural-network approximator with Q-learning to estimate action values and derive a control policy.
- 3.2 Deep learning and Deep Q-learning: Experience replay addresses correlated states by storing transitions and sampling uniformly random minibatches for network updates.
- 3.2 Deep learning and Deep Q-learning: Policy-gradient methods optimize a parameterized policy directly rather than estimating state-value or action-value functions.
4. SYSTEM DESCRIPTION
The system formulates traffic-light control as deep reinforcement learning using image-based intersection states, two signal actions, and delay-based rewards. Policy-gradient and value-function agents are trained with neural networks using policy updates or Q-value estimation.
- The traffic-light problem is formulated as a reinforcement-learning task by defining states, actions, rewards, a deep policy, and network training.
- 4.1 State Representation: Each state is an image snapshot of the simulator’s current intersection view, represented as a vector of pixel values.
- 4.1 State Representation: Stacked image observations let convolutional layers infer vehicle locations, queues, velocity, and travel direction without discretizing lanes into fixed-length cells.
- 4.2 Action Set: The action set contains North/South Green and East/West Green, each permitting traffic on one route while stopping traffic on the other.
- 4.3 Reward Function: The reward is the difference between total cumulative delays at consecutive time steps, so positive values indicate reduced delay and negative values increased delay.
- 4.4 Agent’s Policy: The policy-based agent learns an action distribution with a neural network, whereas the value-function agent estimates action values and greedily selects the highest-valued legal action.
5. EXPERIMENT AND RESULTS
The experiments use SUMO-GUI and deep neural networks to evaluate traffic-signal controllers from visual observations. The proposed methods outperform the baseline and shallow network on reward, delay, and queue-length measures.
- Experimental setup: Experiments use SUMO-GUI to simulate traffic and provide snapshots of an intersection with four incoming and four outgoing lanes.Snapshots are preprocessed as stacked visual observations for the agent.
- System architecture: The visual input is processed by a DQN-like convolutional architecture, while the comparison SNN uses one hidden layer and lane-queue and signal-phase vectors.The deep network uses two convolutional layers and a fully connected layer; the SNN has 64 hidden nodes.
- Evaluation: The proposed controllers are compared with a fixed-time baseline that allocates equal time to each intersection phase.Evaluation compares average reward, average total cumulative delay, and average queue length.
- Results: The learned policy minimizes queue length and total cumulative delay, and reducing cumulative delay in the reward function also reduces intersection queue length.The proposed methods achieve higher average rewards than the SNN.
- Results: The proposed methods significantly outperform the SNN, reducing average cumulative delay by 67% and 68% and average queue length by 72% and 73% for policy-gradient and value-function methods, respectively.The reductions are calculated from the last 100 training epochs.
6. CONCLUSION
The paper applies policy-gradient and value-function deep reinforcement learning to traffic-signal control using raw simulator images. It reports stable, promising control policies in a small simulated application and identifies larger multi-intersection settings as future work.
- Conclusion: The proposed agents learn traffic-signal control policies from raw visual snapshots using policy-gradient and value-function deep reinforcement learning.The methods were developed and tested in simulation.
- Conclusion: The approaches show promising results and more stable control policies than previous deep-reinforcement-learning work in traffic-light optimization.The reported application is small in scale.
- Future work: Future research should extend the methods to more complex simulations with many intersections and multiple agents coordinating through multi-agent learning.The stated scope boundary is the small application used in this work.