PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Traffic Light Control in Vehicular Networks

Xiaoyuan Liang, Xunsheng Du, Guiling Wang, Zhu Han

arXiv:1803.11115v1cs.LGcs.AIstat.ML

TL;DR

Inefficient and weakly adaptive traffic-light control motivates learning signal durations from real-time traffic information. The paper uses grid-based deep reinforcement learning with a high-dimensional phase-duration MDP, and SUMO simulations report lower waiting time and stronger performance than fixed-time strategies.

  • Problem

    Existing traffic-light control uses equal or limited-information timing, motivating adaptive duration decisions from real-time vehicular and sensor data.

  • Method

    The paper represents intersections as grid-based position and speed states, models phase-duration changes as a high-dimensional MDP, and learns control with deep reinforcement learning.

  • Results

    The model outperforms fixed-time strategies in SUMO simulations and reduces average waiting time by at least 25.7% in the evaluated equal-flow scenario.

  • Takeaways & Limitations

    The proposed 3DQN learns effective traffic-light policies under both normal and rush-hour traffic-flow rates.

Abstract

from arXiv · show

Existing inefficient traffic light control causes numerous problems, such as long delay and waste of energy. To improve efficiency, taking real-time traffic information as an input and dynamically adjusting the traffic light duration accordingly is a must. In terms of how to dynamically adjust traffic signals' duration, existing works either split the traffic signal into equal duration or extract limited traffic information from the real data. In this paper, we study how to decide the traffic signals' duration based on the collected data from different sensors and vehicular networks. We propose a deep reinforcement learning model to control the traffic light. In the model, we quantify the complex traffic scenario as states by collecting data and dividing the whole intersection into small grids. The timing changes of a traffic light are the actions, which are modeled as a high-dimension Markov decision process. The reward is the cumulative waiting time difference between two cycles. To solve the model, a convolutional neural network is employed to map the states to rewards. The proposed model is composed of several components to improve the performance, such as dueling network, target network, double Q-learning network, and prioritized experience replay. We evaluate our model via simulation in the Simulation of Urban MObility (SUMO) in a vehicular network, and the simulation results show the efficiency of our model in controlling traffic lights.

I. INTRODUCTION

The paper targets inefficient traffic-light control that lacks real-time traffic awareness, especially under heavy or uneven demand. It proposes learning signal durations from vehicular-network and sensor information using deep reinforcement learning.

  • Motivation: Fixed-time or historically scheduled signals do not adequately account for real-time traffic conditions.These systems can cause long travel delays, energy waste, worsening air quality, and sometimes contribute to accidents.
  • Motivation: During events and rush hours, conventional traffic-light systems can become paralyzed, whereas experienced human operators adapt durations to observed traffic.This motivates a system that learns intersection management from real-time conditions.
  • Prior work and approach: Vehicular networks and cameras provide vehicle counts, positions, speeds, and waiting times as inputs for adaptive control.The paper frames sensing as the system’s “eyes” and reinforcement learning as its decision-making “brain.”
  • Prior work and approach: Existing reinforcement-learning traffic studies often use limited state representations, while adding richer information increases state-space complexity exponentially.Deep neural networks are introduced to handle the resulting large state spaces.
  • Contribution: The proposed model represents intersections as grid-based states, models phase-duration changes as a high-dimensional MDP, and learns control by trial and error.A CNN maps states to expected future rewards; the paper also combines dueling, target, double-Q, and prioritized-replay components.

III. MODEL AND PROBLEM STATEMENT

The model treats traffic-light control as a reinforcement-learning problem in which network-collected traffic states guide phase-duration actions. Its objective is to learn timing policies that improve intersection efficiency while keeping neighboring-cycle changes smooth.

  • System model: The system gathers traffic information through a vehicular network, converts it into states and rewards, and uses a deep neural network to select actions.The reinforcement-learning and deep-learning components together form the traffic-light control model.
  • Problem statement: The optimization problem is to dynamically change every phase’s duration by learning from historical experience.The intended strategy is to allocate more time to directions with more vehicles without manually training a human operator.
  • MDP formulation: Reinforcement learning represents the environment with states, actions, rewards, and state-transition probabilities, and seeks a policy maximizing cumulative expected reward.In deterministic models, the transition function can be omitted.
  • Value estimation: The learned Q-value represents cumulative discounted future reward from taking an action in a given state.The discount factor places greater value on nearer rewards than rewards farther in the future.

V. REINFORCEMENT LEARNING MODEL

This section establishes the three reinforcement-learning elements needed for traffic-light control: states, actions, and rewards. The subsequent model defines each element for the intersection setting.

  • Model elements: Traffic-light reinforcement learning requires explicit definitions of the states, actions, and rewards.These elements specify what the agent observes, what it controls, and how each decision is evaluated.
  • Model elements: The paper presents the definitions of these three elements as the basis for constructing its traffic-light control system.

A. States

The state representation converts an intersection snapshot into grid-based position and speed matrices. Actions then adjust phase durations through a high-dimensional MDP while constraining changes and preserving signal safety.

  • A. States: The intersection is divided into equal square grids, each encoding whether a vehicle is present and, if so, its speed.Grid size is chosen so one vehicle fits in a cell and no cell contains two vehicles.
  • A. States: Fig. 2 builds the state matrix by mapping each intersection grid to a matrix cell, using 0 for empty cells and 1.0 for occupied cells in the position matrix.The corresponding speed value is the vehicle’s speed, or 0 when the cell is empty.
  • B. Action Space: The action selects every phase’s duration in the next cycle, with legal changes constrained to avoid large jumps between neighboring cycles.This creates a high-dimensional MDP over phase-duration changes.
  • B. Action Space: In the illustrated four-phase intersection, each MDP circle represents one cycle’s phase-duration tuple and neighboring actions change one phase duration incrementally.The example includes north-south, east-north/west-south, east-west, and east-south/west-north green phases.
  • B. Action Space: The MDP extends to irregular intersections by increasing its action dimension with the number of phases.
  • Signal safety: Yellow intervals are inserted between neighboring phases and are set from the road’s maximum speed divided by its commonly observed deceleration.This interval allows moving vehicles to stop before the signal turns red.

C. Rewards

The model defines rewards from changes in cumulative vehicle waiting time and uses a CNN to approximate Q values in the large traffic-control state space.

  • Rewards provide feedback on the performance of previous traffic-light actions and guide the learned action policy.
  • The reward is the change in cumulative vehicle waiting time between two neighboring traffic cycles.The system uses intersection efficiency and vehicle waiting time as the basis for reward design.
  • The reward equals the increment in cumulative waiting time after an action relative to before it, and is always nonpositive because delay does not decrease over time.
  • A CNN approximates Q values because the vehicular-network traffic-control system has too many states for direct solution.The resulting network combines CNN approximation with state-of-the-art techniques and is called 3DQN.
  • The CNN receives 60×60 grids containing vehicle position and speed information as a 60×60×2 input.It uses three convolutional layers followed by fully connected layers.
  • The convolutional architecture uses convolution, pooling, and activation to extract features while reducing less-important information and dimensionality.The model uses leaky ReLU activation, whose negative-side slope avoids zero gradients.

B. Dueling DQN

The dueling DQN estimates action values by separating the value of the current state from each action’s advantage, while target-network training stabilizes updates.

  • The dueling DQN estimates Q values from the current-state value and each action’s advantage relative to other actions.The state value represents expected future rewards, while the advantage is action-specific.
  • An action has positive advantage when its numerical reward exceeds the average across possible actions, and negative advantage otherwise.
  • The primary network is updated using mean squared error between predicted and target Q values.The target value guides parameter updates.
  • A separate target network with the same architecture but different parameters generates target values for more stable updates.The target network is used alongside the primary network during training.
  • The target network parameters are updated from the primary network parameters using an update rate α.α controls how strongly the newest primary-network parameters affect the target network.

D. Double DQN

Double DQN generates target values with the target network while selecting actions with the primary network, reducing overestimation according to the cited prior work.

  • Double DQN uses the target network to generate the target Q value and the primary network to generate the action.
  • Double DQN effectively mitigates overestimation and improves performance according to the cited study.
  • The system also uses ε-greedy action selection to balance exploration and exploitation.ε decreases linearly toward a specified ending value during training.

E. Prioritized Experience Replay

Prioritized experience replay samples training experiences according to their temporal-difference errors, while Adam adaptively updates the neural-network parameters.

  • Prioritized experience replay selects memory samples according to priority, potentially accelerating learning and improving the final policy.The priority increases the replay probability of experiences with high temporal-difference error.
  • Rank-based prioritization ranks experiences by error and assigns each experience a priority equal to the reciprocal of its rank.
  • The prioritization parameter τ controls sampling emphasis, with τ = 0 corresponding to random sampling.
  • Adam optimizes the neural networks using an adaptive learning rate based on first- and second-order moments.The paper describes Adam as providing fast convergence and satisfactory overall performance in cited comparisons.
  • Adam updates first- and second-order biased moments through exponential moving averages before correcting them using the time step.
  • The parameter update uses an initial learning rate ε_r and a small positive constant δ for numerical stability.

G. Overall Architecture

The architecture trains an adaptive traffic-light controller with a dueling double deep Q network and prioritized experience replay. It stores state transitions, samples prioritized minibatches, and updates primary and target networks iteratively.

  • G. Overall Architecture: The optimal policy chooses the action with the maximum Q value calculated for each state-action pair.This Q-value maximization defines the controller’s action-selection rule.
  • G. Overall Architecture: The agent selects actions using an ε-greedy policy, observes rewards and successor states, and stores each transition as ⟨s, a, r, s′⟩.Older experiences are removed when memory exceeds its capacity.
  • G. Overall Architecture: Prioritized experience replay selects B samples from memory after sufficient experience has accumulated for minibatch training.Sampling priorities are updated from the temporal-difference error δ.
  • G. Overall Architecture: The primary network parameters are optimized with Adam backpropagation, while the target network is softly updated using θ−= αθ−+ (1 −α)θ.The target network and primary network use separate parameter sets, θ− and θ.
  • G. Overall Architecture: The 3DQN with prioritized experience replay trains a traffic-light policy that changes phase durations across traffic scenarios.The algorithm initializes networks and replay memory, then learns through trial-and-error interaction with the intersection.

VII. EVALUATION

The evaluation uses SUMO to test the proposed controller at a four-road, three-lane intersection. Performance is measured by cumulative reward and average waiting time against fixed-time signals and across network configurations.

  • A. Evaluation Methodology and Parameters: The simulation evaluates cumulative delay reduction and vehicle average waiting time as the two primary objectives.The proposed model is compared with pre-scheduled traffic signals whose durations remain fixed.
  • A. Evaluation Methodology and Parameters: SUMO simulates a 300m × 300m intersection with four perpendicular roads and three lanes per road.Lane roles distinguish right-turn and through traffic, through-only traffic, and left-turn traffic.
  • A. Evaluation Methodology and Parameters: The simulation models vehicle motion with a maximum speed of 13.9 m/s and sets yellow-signal duration to 4 seconds.The stated maximum speed equals 50 km/h.
  • A. Evaluation Methodology and Parameters: Each training iteration represents one hour of traffic, and reported simulation results average the nearest 100 iterations.Training maximizes the accumulated reward by modifying traffic-signal durations.
  • A. Evaluation Methodology and Parameters: The controller is compared with fixed signal durations of 30 seconds and 40 seconds.It is also compared with deep reinforcement-learning architectures using different parameters.

B. Experimental Results

Under equal traffic flow from all lanes, the proposed 3DQN achieves higher cumulative reward than fixed-time traffic lights. Its reward improves as training proceeds and exceeds the fixed-time strategies.

  • B. Experimental Results: The 3DQN learns a higher-reward policy with more iterations, whereas fixed-time signals remain at low reward despite additional training episodes.The adaptive controller changes signal timing according to the traffic scenario and expected future traffic.
  • B. Experimental Results: After 1000 iterations, the protocol has learned how to handle different traffic scenarios to obtain the most rewards.The evaluation uses cumulative reward per episode under equal traffic flow from all lanes.

2) Average waiting time:

The proposed model reduces average waiting time under normal and rush-hour traffic and learns faster than ablated network variants. Its performance remains favorable as traffic becomes more unbalanced and complex.

  • 2) Average waiting time:: About 26 seconds average waiting time is reached after 1200 iterations, versus over 35 seconds for fixed-time signals.This corresponds to at least 25.7% less waiting time than the two fixed-time strategies.
  • 3) Comparison with different parameters and algorithms:: The full model learns fastest among variants without double-network, dueling-network, or prioritized-replay components.After 500 iterations, it exceeds -47000 rewards while the other variants remain below -50000.
  • 4) Average waiting time under rush hours:: During rush hours, one direction receives twice the normal vehicle arrival rate, making the best policy harder to learn.The west-to-east arrival rate is 2/10 per second, while other lanes remain at 1/10 per second.
  • 4) Average waiting time under rush hours:: The model still learns a good policy that reduces average waiting time under the more complex rush-hour traffic scenario.The reported rush-hour average waiting time for 3DQN is about 33 seconds, as stated in the supplied passage.
  • VIII. CONCLUSION: The conclusion reports over 20% reduction in average waiting time from the starting training across normal and rush-hour traffic.It also reports faster learning than other models in SUMO and TensorFlow simulations.
Loading 1803.11115v1…