PaperlayerView exact revision on arXiv

Source-linked AI summary

AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control

Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, Jorge Silva

arXiv:2010.05772v1cs.LGcs.AI

TL;DR

AttendLight addresses the need to retrain traffic-signal RL models for intersections with different structures or traffic flows. It uses two attention mechanisms to handle variable intersection inputs and phase choices, and outperforms benchmark algorithms across the reported cases.

  • Problem

    Many traffic-signal RL methods require redesign and retraining for intersections with different topologies or traffic distributions, incurring expert, data-collection, and computational costs.

    Source: §1 · p. 2

  • Method

    AttendLight is a reinforcement-learning policy with two attention mechanisms for representing variable intersection states and selecting among variable numbers of phases.

    Source: §1 · p. 2

  • Results

    Across all 112 cases, AttendLight outperforms benchmark algorithms in ATT, yielding 46% improvement over FixedTime.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    After training on a collection of intersections, AttendLight can be deployed without retraining when a new intersection's structure follows a pattern represented in the training set.

    Source: Broader Impact · p. 10

    AttendLight may fail to produce an efficient policy for very complex, unusual intersection topologies absent from the training set.

    Source: Broader Impact · p. 10

Abstract

from arXiv · show

We propose AttendLight, an end-to-end Reinforcement Learning (RL) algorithm for the problem of traffic signal control. Previous approaches for this problem have the shortcoming that they require training for each new intersection with a different structure or traffic flow distribution. AttendLight solves this issue by training a single, universal model for intersections with any number of roads, lanes, phases (possible signals), and traffic flow. To this end, we propose a deep RL model which incorporates two attention models. The first attention model is introduced to handle different numbers of roads-lanes; and the second attention model is intended for enabling decision-making with any number of phases in an intersection. As a result, our proposed model works for any intersection configuration, as long as a similar configuration is represented in the training set. Experiments were conducted with both synthetic and real-world standard benchmark data-sets. The results we show cover intersections with three or four approaching roads; one-directional/bi-directional roads with one, two, and three lanes; different number of phases; and different traffic flows. We consider two regimes: (i) single-environment training, single-deployment, and (ii) multi-environment training, multi-deployment. AttendLight outperforms both classical and other RL-based approaches on all cases in both regimes.

1 Introduction

AttendLight addresses the cost and inflexibility of training separate traffic-control policies for intersections with different structures or traffic flows. It uses attention to support a universal policy, which the paper reports outperforms conventional controls and FRAP.

1 Introduction

  • Separate policies for each intersection require repeated expert setup, data collection, and computational resources, making city-scale deployment burdensome.Prior transfer-learning approaches still require model changes and retraining for different structures.Source: §1 · p. 2
  • AttendLight uses attention to train one policy for variable intersection configurations and traffic distributions represented in its training set.The model is intended to reuse an abstract representation of intersection status for deployment on unseen intersections that follow training-set patterns.Source: §1 · p. 2
  • AttendLight substantially outperforms conventional controls and FRAP, a state-of-the-art RL-based method, in the reported comparisons.The introduction states this result without specifying numerical values.Source: §1 · p. 2

2 Related Work

Prior traffic-signal RL work explores varied states, rewards, and control strategies, while transfer approaches still face structural constraints. The section describes AttendLight's context as a need for broader compatibility across intersections.

2 Related Work

  • Traffic-signal RL studies use varied state and reward designs, including vehicle queues, waiting times, lane chunks, and phase selection.Examples include DQN variants and approaches targeting interpretable policies.Source: §2 · p. 2
  • PressLight uses pressure as a reward, while CoLight uses a graph-attentional network to model neighboring intersections.The cited work considers multi-intersection traffic control.Source: §2 · p. 3
  • FRAP requires structural modifications to transfer between four-way and three-way intersections, while MetaLight still retrains parameters for new intersections.A separate transfer-learning approach builds models incrementally for arterial intersection structures and is described as not robust to varying roads, lanes, and phases.Source: §2 · p. 3

3 Traffic Signal Control Problem

The paper formulates traffic signal control at a single intersection with variable roads, lanes, traffic movements, and phases. The task is to minimize average vehicle travel time, using intersection pressure as the reward measure.

3 Traffic Signal Control Problem

  • The task concerns a single intersection whose roads may have multiple directions and lanes, with topology and traffic data known for each intersection.Intersections may have three, four, or more approaching roads.Source: §3 · p. 3
  • A phase is a set of traffic movements permitted during a green light, and the next phase is chosen after a minimum phase duration.Different phases can include different numbers of traffic movements and participating lanes.Source: §3 · p. 3
  • Different phases can have different numbers of movements and participating lanes, producing variable input and output sizes that complicate universal modeling.The section illustrates the issue with a three-way intersection example.Source: §3 · p. 3, Figure 1 · p. 4
  • The objective is to minimize average travel time, while the model uses intersection pressure as a reward measure.The paper reports average travel times in its experiments and cites prior work equating pressure minimization with average travel-time minimization.Source: §3 · p. 4

4 Reinforcement Learning Design

AttendLight formulates signal control as reinforcement learning and uses two attention mechanisms to handle variable intersection inputs and phase choices. It trains with REINFORCE across either one environment or multiple environments.

4 Reinforcement Learning Design

  • The RL state is lane traffic characteristics, the action is the next active phase, and the reward is negative intersection pressure.The policy maps an intersection state to a phase for the next time step to optimize long-term cumulative reward.Source: §4 · p. 4

4.1 Attention Mechanism

  • Attention weights elements in a variable-size input set to produce a fixed-length encoding.AttendLight adapts this mechanism to traffic-signal control.Source: §4.1 · p. 4

4.2 AttendLight Algorithm

  • State-attention and action-attention handle phase representation from raw traffic state and selection of the next phase, respectively.These mechanisms make the model's input and output dimensions independent of intersection configuration.Source: §4.2 · p. 5
  • AttendLight uses attention over participating lanes to construct a phase representation that is independent of the number of lanes in that phase.The average embedded lane traffic is used as the query, allowing the model to weight each lane relative to average traffic.Source: §4.2 · p. 5
  • An LSTM encodes the sequence of active phases, and action-attention produces probabilities for switching to the available phases.The policy is invariant to lane and phase ordering, so enumeration order does not change its control decisions.Source: §4.2 · pp. 5–6

4.3 RL Training

  • AttendLight is trained with variance-reduced REINFORCE in single-environment or multi-environment regimes.In the described multi-environment regime, each episode samples all environments in the set.Source: §4.3 · p. 6

5 Numerical Experiments

AttendLight is evaluated across varied intersection configurations in single- and multi-environment regimes, with multi-environment training tested on unseen intersections. The experiments also examine whether brief calibration can specialize the universal policy for an individual intersection.

5 Numerical Experiments

  • The evaluation covers 11 intersection topologies and 112 instances, combining real-world Hangzhou and Atlanta data with synthetic data for three-lane intersections.Topologies vary in road count, lane count, and phase count; the traffic-data and topology combinations define the instances.Source: §5 · p. 6, Table 1 · p. 6

5.1 Results of the Single-Environment Regime

  • Across 112 cases, AttendLight improves ATT by 46%, 39%, 34%, 16%, and 9% over FixedTime, MaxPressure, SOTL, DQTSC-M, and FRAP, respectively.The single-environment comparison includes the intersection configurations shown below the ATT results.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 Results of Multi-Environment Regime

  • One AttendLight model is trained across 42 intersection instances and tested on 70, including unseen topologies and traffic data; five random-seed models are averaged.The multi-environment regime is compared with single-environment AttendLight and benchmark algorithms, although the authors note this comparison is unfair because the former uses one model across intersections.Source: §5.2 · p. 7
  • Across training and unseen test instances, multi-environment training yields 15% average ATT degradation on training instances and a 13% ATT gap on test instances.The ATT ratios compare multi-environment with single-environment policies; error bars show 95% confidence intervals.Source: §5.2 · p. 8, Figure 4 · p. 8
  • Multi-environment AttendLight obtains smaller ATT than MaxPressure, FixedTime, and SOTL in most cases, beats DQTSC-M on average, and is competitive with FRAP.A few cases favor DQTSC-M; the comparison is shown as densities of ρ_m across intersections.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 Few-Shot Training for Calibration

  • After 200 calibration episodes, the average gap to single-environment performance falls to 5%, and after 1,000 steps it falls to 3%.The universal policy is used as the starting point, without changing the policy network's input or output structure.Source: §5.3 · p. 9

6 Conclusion and Discussion

AttendLight uses two attention mechanisms to make a reinforcement-learning traffic-signal model independent of intersection structure. Experiments report strong results in single-environment settings and generalization to new intersections after multi-environment training.

6 Conclusion and Discussion

  • AttendLight uses two attention mechanisms to make its model inputs and outputs independent of intersection structure.The paper identifies the mechanisms as handling input-state representation and next-phase selection.Source: §6 · p. 9
  • Experiments across varied scenarios report that AttendLight outperforms existing methods in single-environment settings and generalizes to new intersections after multi-environment training.The conclusion describes the scenarios as a variety of settings, without specifying numerical comparisons here.Source: §6 · p. 9
  • Future work includes connected networks of intersections, alternative reinforcement-learning algorithms and attention mechanisms, and applications to other variable-input/output problems.The paper names Assemble-to-Order Systems, Dynamic Matching, and Wireless Resource Allocation as possible applications.Source: §6 · p. 9

Broader Impact

AttendLight is presented as a way to reuse a policy across intersections with similar structures, reducing repeated training costs. The discussion also identifies limits on unusual topologies and coordinated multi-intersection control.

Broader Impact

  • The authors argue that more efficient traffic control could reduce time in congestion, fuel consumption, and air pollution.The discussion frames these as potential societal impacts of AttendLight.Source: Broader Impact · p. 10
  • AttendLight can be deployed on new intersections without retraining when their structures are similar to those represented in its training set.The authors associate this reuse with savings in expert work, computation, and data collection.Source: Broader Impact · p. 10
  • AttendLight may fail to produce an efficient policy for very complex, unusual topologies absent from the training set.The authors suggest greater training-set diversity and more complex policy models as possible remedies.Source: Broader Impact · p. 10
  • Extending AttendLight to coordinated control of multiple intersections remains challenging because of scalability and coordination in decision-making.The paper notes that signals in a network affect one another.Source: Broader Impact · p. 10

A Numerical Experiments Details

A.1 Details of Real-World Data and the Synthetic Data Generation

The appendix describes the traffic data and synthetic-data generation, AttendLight’s model and training, and the implementation of its baseline comparisons. It also records baseline-specific settings and limitations.

A.1 Details of Real-World Data and the Synthetic Data Generation

  • Real-world data come from Hangzhou and Atlanta; adapting four-way data to three-way intersections reallocates unavailable movements uniformly within the same movement set.Synthetic cases use Poisson arrivals with λ of 3 or 4 seconds, movement probabilities of 70%, 20%, and 10%, and additional-arrival probabilities of 0.05, 0.1, or 0.3.Source: §A.1 · p. 13, Table 2 · p. 13

A.2 Details of AttendLight Policy Model

  • AttendLight embeds lane characteristics, applies state-attention and an LSTM, then uses action-attention to return probabilities for selecting each phase.The single-environment and multi-environment models use dimensions 128 and 256, respectively; the authors report no structured hyperparameter tuning.Source: §A.2 · pp. 13–14

A.3 Details of the RL Training

  • Training uses variance-reduced REINFORCE with actor and critic networks, Monte Carlo trajectories, and cumulative rewards based on negative intersection pressure.The appendix reports 100,000 episodes for single-environment training and 30,000 for multi-environment training, with parallel instances in each regime.Source: §A.3 · p. 14

A.4.1 SOTL

  • SOTL switches phases cyclically when its red-count and green-count thresholds are met after a minimum active time; the authors grid-search its parameters.They report the minimum ATT among 1,183 parameter combinations.Source: §A.4.1 · p. 14

A.4.2 Fixed-Time

  • Fixed-Time cycles through phases, assigning each phase a predetermined active duration of 15 seconds.Source: §A.4.2 · p. 15

A.4.3 Max-Pressure

  • Max-Pressure activates, at each time step, the phase with the highest pressure among its participating lanes.Source: §A.4.3 · p. 15

A.4.4 FRAP

  • FRAP assumes two traffic movements per phase and equal movement counts across phases, making it inapplicable to some tested intersections.The authors add dummy movements to handle one limitation, but differing lane counts across phases still prevent use in cases such as INT3 and INT11.Source: §A.4.4 · p. 15

A.4.5 DQTSC-M

  • The authors implemented DQTSC-M because publicly available code was unavailable, using convolutional layers followed by fully connected layers to produce action Q-values.Source: §A.4.5 · p. 15

A.5 Extended Comparison Results

AttendLight generally has lower average travel time (ATT) than the benchmark algorithms in both training regimes, with reported exceptions in its comparison with FRAP.

A.5 Extended Comparison Results

  • 107 of 112 cases: single-environment AttendLight outperforms the other algorithms, with average ATT improvements of 46%, 39%, 34%, 16%, and 9% over FixedTime, MaxPressure, SOTL, DQTSC-M, and FRAP, respectively.Source: §A.5 · p. 15
  • 57 cases: multi-environment AttendLight outperforms the other benchmarks, with average ATT improvements of 39%, 32%, 26%, and 5% over FixedTime, MaxPressure, SOTL, and DQTSC-M, respectively.Source: §A.5 · p. 15
  • 3% larger average ATT: multi-environment AttendLight performs worse than FRAP on average.Source: §A.5 · p. 15

A.6 Mean State Results

State-attention forms a unified representation from lane-traffic information, and the reported comparison finds lower ATT with attention than with mean pooling.

A.6 Mean State Results

  • State-attention produces a unified state representation from a varying number of lane-traffic inputs, whose weighted sum forms a phase-state representation.Source: §A.6 · pp. 15, 17
  • The mean-state alternative assigns equal weight to the participating lane-traffic inputs instead of using state-attention.Source: §A.6 · p. 17
  • Using state-attention rather than mean-state pooling yields smaller ATT in the reported single-environment comparison.The mean-state ATT ratios are reported as positive relative to attention-state ATT.Source: §A.6 · p. 17

A.7 Few-Shot Training for Multi-Env Regime

Few-shot fine-tuning improves the multi-environment policy’s comparisons with baselines, while the section also describes calibrating that policy for individual intersections.

A.7 Few-Shot Training for Multi-Env Regime

  • Few-shot training starts from the multi-env policy and calibrates it for a specific intersection through a few training steps.Figure 8 compares fine-tuning for 200 and 1000 episodes with no fine-tuning.Source: §A.7 · p. 17, Figure 8 · p. 20

A.8 Separated Train/Test ATT ratio distribution

  • After fine-tuning, comparisons with all baselines improved, with distributions shifted left and showing less variance than before fine-tuning.The analysis separates training and test instances.Source: §A.8 · p. 20

B Stochastic Training Regime

The stochastic multi-env regime trains on a small mini-batch of environment instances to reduce per-iteration cost. Its results are statistically equal to the multi-env regime in most cases, with reported ATT gaps against the single-env regime.

B Stochastic Training Regime

C Environment Details

C.1 Intersection and Phase Visualizations

C.2 Description of Each Environment Instance

D.1 Assemble-to-Order Systems

  • For assemble-to-order, variable part and order counts create variable-sized inputs and outputs; AttendLight represents products and selects production to minimize makespan, tardiness, and delay.Source: §D.1 · p. 23

D.2 Dynamic Matching Problem

  • For dynamic matching, queued entities leave once paired; AttendLight extracts each option’s state, then selects matchings with the highest value.Source: §D.2 · p. 23

D.3 Wireless Resource Allocation

  • For wireless allocation, state-attention encodes packet timing, priority, size, and type; action-attention selects the next packet against queue-length or response-time costs.Source: §D.3 · p. 26
Loading 2010.05772v1…