Source-linked AI summary
UniTSA: A Universal Reinforcement Learning Framework for V2X Traffic Signal Control
Maonan Wang, Xi Xiong, Yuheng Kan, Chengcheng Xu, Man-On Pun
TL;DR
Existing RL traffic-signal methods are difficult to generalize across different intersection structures. UniTSA uses a junction matrix and traffic-state augmentation in a universal RL framework; experiments report advantages over conventional universal models and efficient fine-tuning on new intersections.
Problem
Existing RL-based traffic-signal methods are challenging to generalize across intersection structures, and models for designated structures may need redesign and retraining for other configurations.
Source: §1 · p. 2Method
UniTSA trains a universal RL traffic-signal agent using a junction matrix to represent intersection states and traffic-state augmentation to handle diverse configurations.
Source: §1 · p. 2, §4.2 · p. 4Results
Across 12 diverse intersections, UniTSA substantially outperforms conventional universal models and can reach comparable or better performance through fine-tuning in significantly less training time than training from scratch.
Source: §1 · p. 2Takeaways & Limitations
UniTSA is presented as avoiding extensive customization and redevelopment for individual intersections while supporting generalization to unseen configurations.
Source: §6 · p. 16
Abstract
from arXiv · showhide
Traffic congestion is a persistent problem in urban areas, which calls for the development of effective traffic signal control (TSC) systems. While existing Reinforcement Learning (RL)-based methods have shown promising performance in optimizing TSC, it is challenging to generalize these methods across intersections of different structures. In this work, a universal RL-based TSC framework is proposed for Vehicle-to-Everything (V2X) environments. The proposed framework introduces a novel agent design that incorporates a junction matrix to characterize intersection states, making the proposed model applicable to diverse intersections. To equip the proposed RL-based framework with enhanced capability of handling various intersection structures, novel traffic state augmentation methods are tailor-made for signal light control systems. Finally, extensive experimental results derived from multiple intersection configurations confirm the effectiveness of the proposed framework. The source code in this work is available at https://github.com/wmn7/Universal_Light
1 Introduction
Existing RL traffic-signal controllers often require redesign and retraining for different intersection structures, while prior generalized models can degrade on unseen configurations. UniTSA combines a universal agent, traffic-state augmentation, and fine-tuning, and its experiments report advantages over conventional universal models.
1 Introduction
- Urban congestion wastes time, increases fuel consumption, and raises greenhouse-gas emissions, while V2X can provide real-time data to adapt intersection control to changing traffic conditions.Traditional traffic management approaches may be insufficient for increasing volumes and dynamic road conditions.Source: §1 · p. 1
- RL signal control can choose the next phase, keep or change the current phase, or select a duration; next-phase selection offers flexibility but may confuse drivers and increase accident risk.The introduction also describes agents learning through direct interaction with traffic environments.Source: §1 · p. 1
- Existing RL models are often designed for particular intersection structures and may need redesign and retraining for different roads, lanes, and phases; generalized models can also degrade on unseen configurations.These changes require resources for data collection, development, and testing.Source: §1 · p. 2
- UniTSA uses a junction matrix to represent diverse intersection structures in a common-sized form, five traffic-state augmentation methods, and a consistent keep-or-change phase action design.The framework also uses LoRA to fine-tune the model for crucial intersections.Source: §1 · p. 2
- Across 12 intersections of diverse structures, UniTSA substantially outperforms conventional universal models and can fine-tune for comparable or better performance with significantly less training time than training from scratch.The contribution summary describes these results across 12 intersections.Source: §1 · p. 2
2 Related Work
Traffic-signal methods range from fixed-time plans to adaptive control and RL. Existing RL approaches include methods for specific configurations and generalized models with limitations in adapting to new intersections or preserving signal structure.
2 Related Work
- Fixed-time methods use predetermined timings and can perform suboptimally as traffic changes, while actuated approaches adapt to real-time demand but require expert settings for each intersection and may degrade in complex scenarios.The passage names Webster, SCATS, Max-pressure control, and SOTL as examples.Source: §2 · p. 2
- RL-based TSC methods include value-based, policy-based, and actor-critic approaches, but most focus on specific intersection configurations or scenarios.The cited passage lists examples of each category.Source: §2 · pp. 2–3
- MetaLight uses meta-learning for a more universal model but requires retraining its parameters for each new intersection; parameter-sharing approaches avoid that but do not preserve the original traffic-light phase structure.The passage identifies for MetaLight and [18] for parameter-sharing methods.Source: §2 · p. 3
3 Preliminary
The preliminary section defines intersection concepts using a standard four-way junction as an example, with the concepts stated to extend to intersections of different structures.
3 Preliminary
- A lane is an incoming roadway, where vehicles enter, or an outgoing roadway, where vehicles exit.The paper denotes these as lin and lout, respectively.Source: §3 · p. 3
- A traffic movement connects an incoming lane to an outgoing lane; the illustrated four-way intersection has 12 movements, including right turns, left turns, and through movements in each direction.The passage identifies the 12-movement count for the common four-way example.Source: §3 · p. 3
- A movement signal indicates whether a movement is allowed or prohibited, and the four-way example uses eight signals because right-turn traffic can move regardless of the signal.The eight signals correspond to northbound, northbound left-turn, eastbound, eastbound left-turn, westbound, westbound left-turn, southbound, and southbound left-turn movements.Source: §3 · p. 3
- A phase combines movement signals so a specific group of traffic movements can proceed while others are restricted.In the example, Phase-1 permits m1 and m5 for north-south travel while prohibiting other movements.Source: §3 · p. 3
- A signal plan is a prearranged sequence and duration of phases, usually arranged cyclically.The example assigns durations t1 = 50, t2 = 20, t3 = 50, and t4 = 20.Source: §3 · p. 3
4 Methodology
The methodology describes a universal traffic signal control agent trained and fine-tuned to adapt across intersections with varying structures. It covers fixed-size junction-matrix states, traffic-state augmentation, feature extractors, and training and testing on intersection topologies with their available phases.
4.1 Framework
- UniTSA first trains a universal agent across intersections with different topologies and signal schemes, then fine-tunes it for key intersections.The agent uses movement and action features alongside five traffic-state augmentation methods.Source: §4.1 · p. 4
4.2 Agent Design and Junction Matrix
- Because a 4-way intersection has eight valid movement signals regardless of lane count, the agent represents it as a junction matrix and stacks K latest observations into a K×8×8 state, zero-padding unused movements.Actions keep the current phase or change to the next sequential phase; reward is the negative average queue length in each movement.Source: §4.2 · pp. 4–5
4.3 Traffic State Augmentation
- Traffic-state augmentation applies movement shuffling, lane-number changes, flow scaling, Gaussian noise, and masking to improve adaptation across traffic scenarios and intersection structures.Shuffling permutes movement rows; lane changes adjust traffic characteristics, while flow scaling changes flow and occupancy values using a shared factor across movements.Source: §4.3 · pp. 6–7
- The method also applies masking, which zeros selected junction-matrix values at a time step so the agent must infer traffic dynamics from surrounding observations.Augmentation is stochastic across minibatches but consistent across stacked frames.Source: §4.3 · pp. 6, 8
4.4 Intersection Feature Extraction
- For intersection feature extraction, UniTSA provides CNN-, RNN-, and transformer-based structures to represent junction information across the observation history.The CNN processes movement and junction information, the RNN summarizes sequential features, and the transformer encoder captures temporal dependencies.Source: §4.4 · pp. 8–9
4.5 RL Training and Fine-tuning
- PPO trains the universal model from trajectories gathered across diverse traffic scenarios, while LoRA fine-tunes Actor and Critic dense layers for critical intersections.The pretrained weights remain fixed during fine-tuning, and only LoRA modules are updated to adapt to intersection-specific features with limited additional parameters.Source: §4.5 · pp. 9–10
- Training and testing cover the intersection topologies shown, with their available phases.Source: Figure 7 · p. 10
- PPO training combined with LoRA fine-tuning enables the universal model to address intersections with varying structures.LoRA uses a low-rank module with rank r; its matrices are initialized as a zero matrix and a zero-mean Gaussian matrix, with α as a constant scale hyperparameter.Source: §4.5 · p. 10
- Table 1 presents the hyperparameter settings.Source: Table 1 · p. 11
5 Experiments
Experiments assess UniTSA across diverse training and unseen intersections, comparing universal RL models with conventional controls and evaluating fine-tuning and traffic-state augmentation. UniTSA variants perform strongly on test intersections, while fine-tuning reduces training effort and augmentation is most beneficial on unseen structures.
5.1 Experiment Settings
- The experiments used SUMO and PPO, measuring vehicle waiting time while modeling vehicles within 150 m of each intersection and a 3 s yellow interval.Training used 30 parallel processes and 10M environment steps; the state included the previous K = 8 junction-matrix snapshots, with actions every 5 s. Actor and critic networks were two-layer fully connected networks.Source: §5.1 · p. 11
5.2 Datasets
- The dataset comprised 12 intersections with diverse road, lane, and phase configurations: eight were used for training and four for testing.The 100 routes per intersection each lasted 30,000 seconds, with three-quarters used for training and one-quarter for evaluation.Source: §5.2 · pp. 11–12
5.3 Compared Methods
- Comparators included fixed-time control, Webster, SOTL, MPLight, and AttendLight, alongside UniTSA single- and multi-environment variants.Multi-environment variants used CNN, RNN, or Transformer structures, with additional versions incorporating traffic-state augmentation.Source: §5.3 · p. 12
5.4 Results of the training intersections
- UniTSA improved over MPLight and AttendLight by an average of 15% and 12%, respectively, across the eight training intersections.RL-based approaches generally outperformed conventional controls, although SOTL had the shortest waiting time on INT-2 and relied on manually defined environment-specific thresholds.Source: §5.4 · pp. 12–13, Table 3 · p. 13
- Without augmentation, CNN- and RNN-based multi-environment models outperformed the Transformer variant across most intersections; with augmentation, the Transformer-plus-TSA variant outperformed RNN variants in many scenarios.The paper attributes the Transformer’s unaugmented results to its need for more training data and reports that augmentation exposes it to a broader range of intersection structures.Source: §5.4 · p. 13
5.5 Results of the test intersections
- 32.9% and 41.3% lower average travel time: UniTSA (Multi+RNN+TSA) outperformed MPLight and AttendLight, respectively, across the test intersections.UniTSA models demonstrated superior performance among RL-based universal methods across all test intersections.Source: §5.5 · pp. 13–14, Table 4 · p. 13
5.6 Results of fine-tuning in test intersections
- At 1M training steps, fine-tuning improved average performance by 36% over UniTSA (Single) across four test intersections; after 10M steps, fine-tuned models still outperformed scratch-trained models by 3%.The fine-tuning model achieved comparable performance to the single-environment model with around 1M rather than 3M steps, an approximately 66% reduction in computation time.Source: Table 5 · p. 14, §5.6 · p. 15
5.7 Comparative analysis on traffic state augmentations
- On unseen intersections, most augmentation methods improved the base policy; movement shuffle and traffic flow scale were particularly effective.Training-intersection augmentation yielded around 2% average improvement, while noise and mask degraded performance on INT-7; Figure 9 compares augmentation combinations on INT-1, INT-7, INT-9, and INT-11, with 95% confidence intervals.Source: Figure 9 · p. 15, §5.7 · p. 16
6 Conclusion
UniTSA uses a junction matrix and traffic-state augmentation to train a universal RL agent for diverse intersection structures in V2X environments. The paper reports the shortest average waiting time across varied configurations and improved results with fine-tuning.
6 Conclusion
- UniTSA is reported to achieve the shortest average waiting time across diverse intersection configurations and to outperform models trained from scratch with fine-tuning.The conclusion describes it as an open-source framework for future data-efficient and generalizable RL-based traffic signal control research in V2X.Source: §6 · p. 16