PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Phase Competition for Traffic Signal Control

Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, Zhenhui Li

arXiv:1905.04722v1cs.LGcs.AIstat.ML

TL;DR

Traffic signal control must handle dynamic flows, large RL exploration spaces, and symmetric states that can appear different to existing models. FRAP models pairwise competition between phase demands to enforce symmetry invariance. The paper reports better solutions, faster convergence, and stronger generalizability across traffic-control settings, while identifying pedestrians, non-motorized vehicles, and field validation as future needs.

  • Problem

    Existing RL traffic-signal methods can converge slowly, fail to transfer across symmetric traffic states, and face extremely large exploration spaces in eight-phase control.

  • Method

    FRAP predicts demand for each phase and models pairwise competition between phases to achieve invariance to traffic-flow rotations and flips.

  • Results

    FRAP converges faster, achieves better performance than state-of-the-art methods, and generalizes across intersection structures, traffic flows, complex phase settings, and multi-intersection environments.

  • Takeaways & Limitations

    Competition-based symmetry modeling lets FRAP share knowledge across symmetric states and efficiently address complex all-phase traffic signal control.

  • Takeaways & Limitations

    The paper calls for modeling pedestrians and non-motorized vehicles and for field studies to obtain real-world feedback and validate the approach.

Abstract

from arXiv · show

Increasingly available city data and advanced learning techniques have empowered people to improve the efficiency of our city functions. Among them, improving the urban transportation efficiency is one of the most prominent topics. Recent studies have proposed to use reinforcement learning (RL) for traffic signal control. Different from traditional transportation approaches which rely heavily on prior knowledge, RL can learn directly from the feedback. On the other side, without a careful model design, existing RL methods typically take a long time to converge and the learned models may not be able to adapt to new scenarios. For example, a model that is trained well for morning traffic may not work for the afternoon traffic because the traffic flow could be reversed, resulting in a very different state representation. In this paper, we propose a novel design called FRAP, which is based on the intuitive principle of phase competition in traffic signal control: when two traffic signals conflict, priority should be given to one with larger traffic movement (i.e., higher demand). Through the phase competition modeling, our model achieves invariance to symmetrical cases such as flipping and rotation in traffic flow. By conducting comprehensive experiments, we demonstrate that our model finds better solutions than existing RL methods in the complicated all-phase selection problem, converges much faster during training, and achieves superior generalizability for different road structures and traffic conditions.

1 INTRODUCTION

Traffic signal control is difficult because existing plans and RL methods struggle with dynamic flows, large exploration spaces, and symmetric traffic states. FRAP models phase competition to share knowledge across symmetries, improving learning efficiency and generalization.

  • Traffic congestion creates substantial economic and time costs, while traffic signal control is a direct target for data-driven improvement.
  • Existing signal plans are not adaptive enough for complex, dynamic traffic flows.
  • Eight-phase control greatly expands the exploration space, making optimal-policy learning substantially harder than in two-phase settings.For four-approach intersections, the exploration space increases to 8 × n8 when all eight phases are considered.
  • Existing RL methods can treat flipped traffic as a different state, preventing knowledge learned in one symmetric case from transferring to another.Rotation and flipping can generate seven additional cases from one traffic state.
  • FRAP predicts phase demand and models pairwise competition, producing symmetry invariance and reducing the exploration space from 64 × n8 to 16 × n4 samples.The design prioritizes the higher-demand movement when conflicting phases compete.
  • Comprehensive experiments report better solutions, faster convergence, and superior generalizability for difficult all-phase control.The reported generalizability covers different road structures, traffic flows, complex phase settings, and multi-intersection environments.

2 RELATED WORK

Traffic signal control research spans manually specified, optimization-based, and learning-based approaches. The paper positions FRAP as an invariant learning design addressing the large exploration space that limits complete eight-phase control.

  • Traditional traffic signal control includes fixed-time, actuated, selection-based adaptive, and optimization-based approaches.
  • Traditional methods rely on manually designed plans, rules, or strong traffic-flow assumptions.
  • Reinforcement learning learns signal policies through interaction with intersections rather than predefined plans or traffic-flow models.An intersection is modeled as an agent with states, actions, and transportation-efficiency rewards.
  • Existing learning methods vary in state descriptions, action definitions, and reward designs, including vehicle positions, queue length, waiting time, phase changes, and delay.
  • Prior methods had not achieved satisfactory results for complete eight-phase control at one intersection because of the large exploration space.FRAP applies competition and invariance principles and can be adapted beyond its Ape-X DQN base framework.

3 PROBLEM DEFINITION

The paper formulates isolated-intersection signal control as a reinforcement-learning problem in which an agent observes traffic, selects phases, and optimizes travel-related efficiency. The framework uses vehicle counts, signal state, and queue length while allowing broader feature designs.

  • The problem is introduced using a four-approach intersection, while the concepts are intended to generalize to other intersection structures.
  • A traffic movement is directional traffic—left, through, or right—and a signal controls a movement rather than an individual lane.The four-approach example contains eight traffic movements.
  • The agent observes an isolated intersection and changes traffic signals to learn a policy that optimizes travel time within a Markov Decision Process.
  • The RL formulation defines a reward over state-action pairs and seeks a policy that maximizes expected discounted return.
  • The state contains vehicle counts on each movement and the current signal phase, the action selects the next phase, and the reward is average queue length.
  • The paper uses simple state features and rewards to focus on model design, while allowing more complex alternatives for performance improvements.

4 METHOD

FRAP designs traffic-signal control around phase demand and pairwise competition, using shared representations to exploit symmetry and reduce exploration. Built on DQN and Ape-X DQN, it targets faster learning and adaptation across intersection settings.

  • Model Overview: FRAP uses DQN to predict a Q-value for each traffic-signal phase from traffic-movement state features.Its design is organized into phase demand modeling, phase pair embedding, and phase pair competition.
  • Model Overview: FRAP models traffic control through competition and invariance: higher-demand movements receive priority, while symmetric rotations and flips share equivalent structure.The network focuses on relative relations between traffic movements rather than treating each movement independently.
  • Phase Demand Modeling: The demand module encodes each movement’s vehicle count and signal phase, combines hidden representations, and shares parameters across movements.Phase demand is formed by adding the demands of the two non-conflicting movements in each phase.
  • Phase Pair Competition: FRAP represents each phase pair with relation and demand embeddings, then predicts each phase’s score from its competition with opposing phases.The pairwise competition output represents relative priorities of a phase over its opponents.
  • Phase Pair Competition: 1 × 1 convolutional filters process phase-pair volumes while sharing parameters across pairs, and element-wise multiplication combines relation and demand representations.The resulting competition matrix contains one row of relative priorities for each phase.
  • Discussions: 16 × n4 samples replace 64 × n8 for the eight-phase setting, reducing the exploration space through shared pairwise competition and symmetry.The paper reports that this sample-efficiency improvement supports faster convergence and better solutions than existing RL methods.
  • Discussions: FRAP is designed for different traffic volumes, signal settings, road structures, and multi-intersection environments, including transfer without additional training.The paper reports applications to 3-, 4-, and 5-approach intersections and multi-intersection control without explicit coordination.

5 EXPERIMENT

Experiments use SUMO simulations with real-world traffic datasets and compare FRAP against conventional, adaptive, and reinforcement-learning controllers.

  • Simulation platform: Experiments run in SUMO, where traffic data drives vehicle movement and signal-control methods receive states and execute signal actions.Each green signal is followed by three seconds of yellow and two seconds of all-red time.
  • Traffic data: Traffic records describe each vehicle by its origin, arrival time, and destination, and these records are supplied to the simulator.
  • Road settings: Single-intersection experiments generally use four-way intersections with four 300-meter road segments, while multi-intersection experiments use real road networks.
  • Datasets: The evaluation uses private Jinan and Hangzhou datasets from China and a public Atlanta dataset from the United States.The Jinan data include seven intersections with relatively complete camera records for single-intersection control.
  • Compared methods: FRAP is compared with Fixedtime, SOTL, Formula, DRL, and IntelliLight, with each method separately tuned and its best performance reported.The baselines include fixed-time, threshold-based, formula-based, DQN, and more elaborate deep-reinforcement-learning controllers.

5.4 Evaluation Metrics

The experiments evaluate traffic-signal controllers using travel time, a standard measure of transportation performance.

  • Travel time: Travel time is the average time vehicles spend on approaching lanes, measured in seconds.The paper identifies it as the most frequently used transportation-performance measure.

5.5 Overall Performance

In the 8-phase setting, FRAP significantly outperforms the other methods across all datasets and converges faster than the compared reinforcement-learning approaches.

  • Overall performance: FRAP significantly outperforms all other methods on every dataset in the reported 8-phase travel-time experiments.Tables 1 and 2 report travel times, with improvement defined as the percentage by which FRAP surpasses the best baseline.
  • Convergence speed: FRAP achieves the fastest convergence among the evaluated reinforcement-learning methods.The paper attributes this to exploiting traffic-signal symmetries together with the Ape-X DQN framework, improving sampling efficiency.

5.6 Model Characteristics

FRAP is evaluated for convergence, symmetry-based transfer, traffic-volume adaptation, and the flexibility of 8-phase control under unbalanced demand.

  • Invariance to flipping and rotation: Traffic flows can reverse between morning and afternoon, making symmetry-based handling relevant to signal-control transfer.At Jinan intersection 4, west-approach volume exceeds east-approach volume around 8 am, while the relation reverses at 5 pm.
  • Invariance to flipping and rotation: The transferred FRAP model achieves almost identical travel-time performance to retrained models on flipped and rotated traffic flows.This avoids the extra training costs required by prior methods when traffic flow changes drastically.
  • Adaptation to different traffic volumes: FRAP transfers from heavy to light traffic nearly as well as retraining, whereas IntelliLight shows a distinct gap between transferred and retrained models.The models are trained on Jinan intersection 2 and evaluated on lighter traffic from intersection 1.
  • Flexibility of the 8-phase setting: Under unbalanced west-east demand at Jinan intersection 1, 8-phase control allocates green time according to movement volume more flexibly than 4-phase control.Travel time is 66.40s under 8-phase control versus 81.97s under 4-phase control.

5.7 Experiment on Different Environments

FRAP is evaluated across different intersection structures and multi-intersection environments, where it consistently outperforms other methods and requires little structural modification.

  • Different Intersection Structures: For 3-approach intersections, FRAP disables selected neurons and zero-pads inputs for nonexistent movements.These modifications make the model compatible with the reduced intersection structure.
  • Different Intersection Structures: For 5-approach intersections, FRAP adds a phase and applies phase demand modeling, phase pair embedding, and phase pair competition.The same process predicts Q-values for all phases after the additional phase is incorporated.
  • Different Intersection Structures: FRAP performs consistently better than other methods across 3-, 4-, and 5-approach intersection structures.The experiments synthesize representative traffic flows for each structure from Jinan data statistics.
  • Multi-Intersection Environments: FRAP is evaluated on 3 × 4, 4 × 4, and 1 × 5 intersection grids in Jinan, Hangzhou, and Atlanta.The experiments use real-world data, with missing data filled during preprocessing; Atlanta’s grid is shown in Figure 11.
  • Multi-Intersection Environments: The multi-intersection comparison reports FRAP’s performance against other methods in Table 4.The cited passage states that FRAP stands out in these environments, while the table caption identifies the evaluation setting.

5.8 Interpretation of Learned Policies

The learned policy is interpreted by comparing each movement’s normalized green-time percentage with its traffic-volume ratio from 8 am to 8 pm.

  • Policy Interpretation: FRAP’s green-time ratio synchronizes well with the traffic-volume percentage across the evaluated hours.The analysis computes green-time allocations from the learned policy and normalizes them for comparison with movement volumes.
  • Policy Interpretation: SOTL and DRL allocate green time more randomly and irregularly than FRAP in the comparison.The comparison uses traffic volume ratio and green time ratio for each movement between 8 am and 8 pm.
  • Policy Interpretation: Because four left-turn movements have relatively light volume, a good policy should assign more green time to the four through movements.Figure 12 uses this traffic-volume imbalance to illustrate the expected allocation pattern.

6 CONCLUSION

The paper proposes FRAP for traffic signal control and reports faster convergence, better performance, and potential application across complex environments. Future work includes modeling pedestrians and non-motorized vehicles and conducting field studies.

  • Conclusion: FRAP is proposed as a reinforcement learning model inspired by competition and invariance principles.The conclusion frames phase competition and symmetry-related design as the model’s conceptual basis.
  • Conclusion: Experiments on three datasets show that FRAP converges faster and achieves better performance than state-of-the-art methods.The conclusion also reports potential for handling different intersection structures and multi-intersection environments.
  • Conclusion: Future work should consider pedestrian and non-motorized vehicle patterns and use field studies to obtain real-world feedback.The paper identifies field validation as an important next step for evaluating the proposed RL approach.
Loading 1905.04722v1…