CoLight: Learning Network-level Cooperation for Traffic Signal Control

Hua Wei; Nan Xu; Huichu Zhang; Guanjie Zheng; Xinshi Zang; Chacha Chen; Weinan Zhang; Yanmin Zhu; Kai Xu; Zhenhui Li

Source-linked AI summary

CoLight: Learning Network-level Cooperation for Traffic Signal Control

Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, Zhenhui Li

arXiv:1905.05717v2May 11, 2019cs.MAcs.GTcs.LG

TL;DR

Traffic-signal coordination must handle dynamic traffic rather than rely on fixed offsets or restrictive assumptions. CoLight uses graph attention to learn changing neighbor influences and index-free representations, with experiments reporting superior performance across synthetic and real-world networks.

Problem
Conventional offset-based coordination and optimization methods rely on assumptions that do not fit dynamic real-world traffic, while existing RL communication is inefficient.
Method
CoLight uses graph attentional networks to learn dynamic communication among neighboring intersections and construct an index-free model with parameter sharing.
Results
The proposed reinforcement-learning approach demonstrates superior performance over state-of-the-art methods on synthetic and real-world traffic data.
Takeaways & Limitations
Learning dynamic communication and index-free neighbor influences supports network-level traffic-signal control at large scale, including a Manhattan network with 196 intersections.
Takeaways & Limitations
Future applicability may require more flexible neighborhood selection and exterior observation data such as road and weather conditions.

Abstract

from arXiv · show

Cooperation among the traffic signals enables vehicles to move through intersections more quickly. Conventional transportation approaches implement cooperation by pre-calculating the offsets between two intersections. Such pre-calculated offsets are not suitable for dynamic traffic environments. To enable cooperation of traffic signals, in this paper, we propose a model, CoLight, which uses graph attentional networks to facilitate communication. Specifically, for a target intersection in a network, CoLight can not only incorporate the temporal and spatial influences of neighboring intersections to the target intersection, but also build up index-free modeling of neighboring intersections. To the best of our knowledge, we are the first to use graph attentional networks in the setting of reinforcement learning for traffic signal control and to conduct experiments on the large-scale road network with hundreds of traffic signals. In experiments, we demonstrate that by learning the communication, the proposed model can achieve superior performance against the state-of-the-art methods.

1 INTRODUCTION

Traffic-signal cooperation matters because neighboring intersections affect one another, while conventional assumptions and fixed neighbor indexing fit dynamic networks poorly. CoLight learns dynamic neighbor communication with graph attention and index-free modeling, and is evaluated on networks including 196 intersections.

Neighboring traffic signals can affect one another, so effective cooperation can help vehicles move through intersections more quickly.
Optimization-based coordination often assumes uniform arrivals or unlimited lane capacity, conditions that do not hold reliably in real-world traffic.
CoLight uses graph attentional networks to learn temporally and spatially changing influences among neighboring intersections.The method addresses cases where one neighbor is more relevant than another and influence directions vary across the day.
CoLight replaces fixed neighbor indexing with attention-weighted aggregation, enabling index-free modeling while supporting parameter sharing.Fixed indexing can make shared agents associate the same positions with different physical road roles.
The experiments include synthetic and real-world data, including a Manhattan network with 196 intersections, and report performance superior to state-of-the-art methods.

2 RELATED WORK

Prior traffic-signal coordination uses offsets, centralized optimization, or independently trained communicating agents, but these approaches face restrictive assumptions, joint-action dimensionality, or indexing requirements. CoLight uses attention to learn neighbor weights and targets larger network settings.

Conventional coordination modifies offsets between consecutive intersections and requires intersections to share a cycle length.
Centralized reinforcement learning can suffer from the curse of dimensionality when one agent selects joint actions for all intersections.
Independent RL agents communicate through neighboring information, but simple concatenation requires an extra indexing mechanism that is often unrealistic and heuristic.
CoLight uses attention to learn different weights for neighboring intersections and directly models their overall influence on a target intersection.
Earlier large-scale evaluations were limited: joint-modeling studies used at most 20 intersections, while individual-modeling studies used no more than 70 signals.CoLight evaluates a real-world network with about 200 intersections.

3 PROBLEM DEFINITION

The paper formulates network-level traffic signal control as a Markov Game with one agent per intersection, partial observations, phase-selection actions, transitions, rewards, and discounted policies. The objective is to reduce queue length and vehicle travel time.

Each intersection is controlled by an agent that observes part of the system and jointly selects signal phases to minimize average queue length.
An agent’s observation contains its current traffic-signal phase and vehicle counts on lanes connected to the intersection.
At each decision time, an agent selects a phase from its predefined action set for the next Δt period.
Joint actions determine state transitions through a transition probability, while each agent receives an immediate reward from the environment.
The learning objective uses discounted returns to represent long-term effects, with rewards defined to make travel time minimization tractable through queue length.

4 METHOD

CoLight is organized as a cooperated reinforcement-learning network with observation embedding, neighborhood cooperation through GAT layers, and final q-value prediction. The section also compares computational complexity across methods.

The proposed network processes observations through an embedding layer before neighborhood cooperation.
Interior neighborhood cooperation is implemented with graph attention network layers.
A final q-value prediction layer produces the network’s reinforcement-learning output.

4.1 Observation Embedding

CoLight embeds each intersection’s local traffic observation into a learned latent representation. The input includes lane vehicle counts and the current signal phase.

Local observations contain the number of vehicles on each lane and the current signal phase.
A Multi-Layer Perceptron maps the k-dimensional observation into an m-dimensional latent space.
The embedding uses a learned weight matrix and bias, followed by a ReLU activation.
The resulting hidden state h_i represents the current traffic condition of intersection i.

4.2 Graph Attention Networks for Cooperation

CoLight uses graph attention to communicate between intersections by weighting neighboring representations according to source-target interactions and neighborhood scope. Multi-head attention summarizes different cooperation patterns, with five heads achieving the best reported performance.

Observation Interaction: The agent attends to neighboring intersection representations to model their influence on the target policy.The neighborhood summary is formed before the agent models neighborhood influence.
Observation Interaction: Source-target interaction scores can be asymmetric, so upstream traffic may receive more attention than downstream traffic.For one-way traffic from Inter 9-50 to Inter 9-49, the upstream source is more useful for the target’s future actions.
Attention Distribution within Neighborhood Scope: Neighborhood attention normalizes interaction scores over a selected set of nearby intersections using a temperature factor.Neighborhoods may be defined by road distance or node distance.
Attention Distribution within Neighborhood Scope: The neighborhood can include the target itself and nonadjacent intersections, allowing flexible cooperation scopes across road-network structures.The target intersection is included so the agent can determine how much attention to place on its own traffic condition.
Neighborhood Cooperation: Weighted neighborhood representations produce an index-free model that avoids aligning neighboring-intersection indices across agents.The summary combines source representations using attention weights.
Neighborhood Cooperation: Graph-level attention adjusts focus to dynamic traffic, exemplified by higher attention to upstream Inter 9-50 than downstream Inter 9-48.The first GAT layer directly provides Inter 9-49 with information from four adjacent intersections.
Multi-head Attention: Multi-head attention computes neighborhood cooperation from different representation subspaces, and five attention heads achieve the best performance.The parallel attention mechanisms can be averaged to summarize multiple cooperation types.

4.3 Q-value Prediction

CoLight applies graph-attention-based neighborhood cooperation in each hidden layer before predicting action values. The final prediction layer outputs q-values for the available signal phases.

Each hidden layer learns a neighborhood representation through the graph-attention procedure.
The model’s forward propagation consists of repeated GAT-based cooperation followed by q-value prediction.The predicted q-value is denoted e_q, and p is the number of phases in the action space.
The loss function optimizes CoLight’s current policy across time steps and intersections using all trainable variables.T denotes contributing time steps, N the number of intersections, and θ all trainable variables.
The attention mechanism helps the target agent distinguish complex scenarios by accounting for changing spatial and temporal neighborhood importance.

4.4 Complexity Analysis

CoLight’s parameter-sharing and index-free design make its computational demands largely independent of network size. Its space complexity is approximately O(m^2L), while computation is approximately O(m^2L).

O(m^2L) time and space demands are approximately independent of the number of intersections, supporting scalability to hundreds or thousands of intersections.This conclusion assumes concurrent processing of agents, attention heads, embeddings, and each target’s neighbors.
The learnable-parameter count is O(m(4mL + L + k + 1 + p)+p), approximately O(m^2L) under typical dimension relationships.The approximation assumes the hidden-layer size m is larger than L and p and comparable to input dimension k.
Without parameter sharing, N separate RL models require approximately O(m^2L · N) space, which is infeasible for extremely large networks.
A single shared RL model reduces space complexity to O(m^2L), identical to CoLight’s space complexity.
Individual and shared RL models each require approximately O(m^2L) computation, approaching CoLight’s computational cost.

5 EXPERIMENTS

The experiments evaluate CoLight on synthetic and real-world traffic datasets, including networks ranging from 9 intersections to a 196-intersection Manhattan network. CoLight consistently outperforms conventional and reinforcement-learning baselines in average travel time, with faster convergence and larger gains on dynamic real-world traffic.

5.1 Settings: The evaluation uses two synthetic and three real-world datasets, including a 196-intersection Manhattan network and smaller city networks.The synthetic settings include Arterial1×3, Grid3×3, and Grid6×6; real-world data come from New York, Hangzhou, and Jinan.
5.3 Compared Methods: The compared methods include fixed-time control, MaxPressure, CGRL, Individual RL, and other reinforcement-learning models, with RL models trained without pre-trained parameters.Average travel time measures the seconds vehicles spend between entering and leaving the modeled area.
5.5.1 Overall Analysis: CoLight improves over MaxPressure and Individual RL by 6.98% on synthetic data and 11.69% on real-world data.These are average improvements across the evaluated synthetic and real-world settings.
5.5.1 Overall Analysis: The average performance gap over MaxPressure grows from 8.08% on synthetic regular traffic to 19.89% on real-world dynamic traffic.The paper attributes this divergence to MaxPressure’s inability to learn from environmental feedback.
5.5.1 Overall Analysis: CoLight outperforms CGRL while using decentralized communication, avoiding CGRL’s centralized maximization over a combinatorially large joint action space.The paper characterizes CoLight’s decentralized approach as having a smaller action space.
5.5.2 Convergence Comparison: Across the listed datasets, CoLight has better jumpstart, time-to-threshold, and asymptotic performance than five RL baselines, without slowing convergence.The convergence curves are smoothed with a moving average of five points.

5.6 Scalability Comparison

CoLight is evaluated for scalability across network sizes, training efficiency, neighborhood definitions, neighbor counts, and attention-head counts. It performs consistently well across scales, while moderate neighborhood and head counts offer better efficiency and performance.

Scalability across network sizes: CoLight performs consistently better than other reinforcement-learning methods on networks ranging from 9-intersection grids to 196-intersection real-world networks.
Training time: CoLight’s training time for 100 episodes is comparable to OneModel and GCN and more efficient than CGRL, Individual RL, and Neighbor RL.Individual RL is non-scalable on the 196-intersection New York setting because separate agents cannot be trained and updated simultaneously under processor and memory limitations.
Neighborhood definition: Geographic-distance neighborhoods achieve similar performance to node-distance neighborhoods on synthetic data but largely outperform them under real-world traffic.The authors attribute the synthetic-data similarity to equal lane lengths making the two neighborhood definitions identical there.
Number of neighbors: Performance is optimal with 2–5 neighbors, while adding more neighbors requires at least 200 episodes to learn and provides no stated performance advantage.Using four nearby intersections and the target intersection is described as adequate for cooperation with time and performance guarantees.
Attention heads: Moderate numbers of attention heads improve control efficiency, but benefits disappear when the number of heads exceeds 5.Table 3 reports that more attention types with H ≤5 enhance efficiency, whereas H >5 can distract learning and deteriorate overall performance.

6 ATTENTION STUDY

The attention study examines how CoLight allocates attention across neighboring intersections and over time. The learned distributions emphasize structurally and temporally relevant traffic information.

Spatial attention: In New York, attention concentrates mainly on the target intersection and upstream arterial intersections rather than downstream intersections.The spatial distribution is interpreted as indicating the relative importance of different neighbors to the target agent.
Spatial attention: In Hangzhou, attention concentrates mainly on arterial intersections and the target intersection.
Temporal attention: The temporal distribution of attention reflects temporally changing traffic in the network.
Temporal attention: The target intersection’s own attention remains the largest component throughout the analyzed period.This indicates that the target intersection’s traffic condition receives substantial weight in the learned representation.
Temporal attention: Attention scores change with traffic direction: neighbors associated with increased South–North traffic receive more attention, while those associated with reduced West–East traffic receive less.For Inter #4, the temporal changes in neighboring attention match changes in traffic flow from the four directions.

7 CONCLUSION

CoLight uses reinforcement learning with graph attention to learn dynamic, index-free communication among traffic-signal agents. Experiments report superior performance, while the authors identify additional data and neighborhood flexibility as future directions.

CoLight learns dynamic communication between agents and constructs an index-free model using a graph attention network.
Extensive experiments on synthetic and real-world data demonstrate superior performance over state-of-the-art methods.
The authors identify flexible neighborhood determination and additional road or weather data as directions for making CoLight more applicable to real-world conditions.The current observations use only traffic-signal phase and per-lane vehicle counts.