PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeLight: A Reinforcement Learning Method toward Collision-free Traffic Signal Control

Wenlu Du, Junyi Ye, Jingyi Gu, Jing Li, Hua Wei, Guiling Wang

arXiv:2211.10871v3cs.LGcs.AI

TL;DR

Some RL-based TSC methods incorporate safety into rewards alongside mobility, but this can make performance sensitive to reward settings and require prolonged learning. SafeLight uses residual learning to integrate a domain safety model as modules into the underlying RL model. Over 99 percent reduction in collisions than the backbone RL model and about 30 percent lower average waiting time than the Fixed-time control were reported.

  • Problem

    Some RL-based TSC methods incorporate safety into rewards alongside mobility, but this can make performance sensitive to reward settings and require prolonged learning.

    Source: Introduction · p. 1

  • Method

    SafeLight uses residual learning to integrate a domain safety model as modules into the underlying RL model.

    Source: Introduction · p. 2

  • Results

    Over 99 percent reduction in collisions than the backbone RL model and about 30 percent lower average waiting time than the Fixed-time control were reported.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors report that safety modules can be integrated into existing RL methods and achieve superior performance under both safety and mobility measures.

    Source: Introduction · p. 2

    SafeLight-Act cannot eliminate the described collision cases involving a yellow phase.

    Source: Results of Evaluation Performance (RQ1) · p. 7

Abstract

from arXiv · show

Traffic signal control is safety-critical for our daily life. Roughly one-quarter of road accidents in the U.S. happen at intersections due to problematic signal timing, urging the development of safety-oriented intersection control. However, existing studies on adaptive traffic signal control using reinforcement learning technologies have focused mainly on minimizing traffic delay but neglecting the potential exposure to unsafe conditions. We, for the first time, incorporate road safety standards as enforcement to ensure the safety of existing reinforcement learning methods, aiming toward operating intersections with zero collisions. We have proposed a safety-enhanced residual reinforcement learning method (SafeLight) and employed multiple optimization techniques, such as multi-objective loss function and reward shaping for better knowledge integration. Extensive experiments are conducted using both synthetic and real-world benchmark datasets. Results show that our method can significantly reduce collisions while increasing traffic mobility.

Introduction

Urban traffic congestion makes signal control important, but RL-based traffic signal control has largely prioritized mobility over safety. SafeLight incorporates domain safety rules into RL and reports substantial collision reduction alongside improved mobility.

Introduction

  • Signalized intersections are common urban bottlenecks, while RL can adapt signal actions from environmental feedback and mobility-related rewards to improve traffic movement.Traffic congestion was estimated to cost the U.S. freight sector $74.1 billion annually, including $66.1 billion in urban areas.Source: Introduction · p. 1
  • More than 50 percent of combined fatal and injury crashes occur at or near intersections, underscoring the stakes of traffic-signal safety.Crashes at signalized intersections account for about one-third of intersection fatalities, including red-light running and improper left turns.Source: Introduction · p. 1
  • Safety-aware RL reward design can require ad hoc factor tuning and long exploration, while inverse reward learning still requires extensive data covering difficult-to-obtain unsafe cases.Traffic-signal-caused accidents are rare during training, and even a well-learned model might not obey safety rules in practice.Source: Introduction · pp. 1–2
  • SafeLight uses domain safety rules to identify unsafe traffic-signal actions and integrates safety modules into RL design components including state, action, reward, and loss.The approach uses residual learning and analyzes the advantages and disadvantages of design choices.Source: Introduction · p. 2
  • Current RL-based methods had higher collision occurrences than conventional non-RL methods on both synthetic and real-world benchmark datasets.Source: Introduction · p. 2
  • Over 99 percent fewer collisions than the backbone RL model and about 30 percent lower average waiting time than the Fixed-time control were reported for SafeLight.The reported performance holds under different environment settings.Source: Introduction · p. 2

Related Works

Prior RL-based traffic-signal-control work has emphasized mobility, while safety-oriented approaches face reward-design or data requirements. Safe RL offers strategies for incorporating safety into objectives or exploration.

RL-based TSC Methods

  • DQN-based and actor-critic traffic-signal methods improve mobility, but the described methods do not consider safety concerns.The passage discusses DDQN variants, Dueling DDQN, and PPO.Source: RL-based TSC Methods · p. 2
  • Multi-objective traffic-signal RL approaches combine mobility and safety, but synthetic-reward methods require expert-set weights and may lack convergence guarantees, while risk estimation can require unavailable crash data.MORL methods learn separate value functions and make decisions from synthetic Q-values; the cited work describes them as guaranteeing convergence.Source: RL-based TSC Methods · p. 2

Safe Reinforcement Learning

  • Safe RL modifies either the optimality criterion or exploration to avoid unsafe situations, but the field had not yet been applied to traffic signal control.Other approaches use external knowledge or short-horizon planning to anticipate safety violations.Source: Safe Reinforcement Learning · p. 2

Background

Reinforcement learning models traffic signal control as sequential decisions over intersection states, actions, and rewards. Transportation safety guidance supplies rules for assessing unsafe left-turn phasing.

Reinforcement Learning

  • An RL agent learns a policy through state-action-reward interactions in an MDP, with the objective of maximizing cumulative reward.The MDP is defined by state and action spaces, transition probabilities, a reward function, and a discount factor.Source: Reinforcement Learning · p. 3

TSC as an MDP

  • Traffic signal control represents the intersection state using sensed traffic conditions, with queue length commonly used among candidate factors.Other factors include stopped and approaching vehicles, speed, maximum waiting time, and vehicle positions.Source: State Space · p. 3
  • Cyclic control adjusts phase duration within a complete ordered signal cycle, whereas acyclic control selects an individual phase for a fixed execution interval.The cyclic example uses four phases in a specified order.Source: Action Space · p. 3, Figure 1 · p. 3

Domain Strategy on Safety

  • Traffic signal control can use mobility-related rewards, while domain safety strategies draw on authoritative logical rules, guidelines, and timing equations.The paper illustrates safety concerns using left-turn phasing guidelines and also points to yellow-change and red-clearance standards.Source: Reward Function · p. 3, Domain Strategy on Safety · p. 3
  • Permitted-only and protective-permitted left turns can reduce delay while negatively affecting safety; the cited guideline treats permitted phasing as unsafe when opposing-traffic 85th-percentile speed exceeds 45 mph.Protected-only gives left-turn drivers exclusive right of way; permitted-only requires yielding to opposing vehicles; protective-permitted combines the two.Source: Logical Rules for Left-turn Phasing · p. 3, Figure 2 · p. 4

Proposed Approach

SafeLight adds a safety model to an RL controller, using domain rules to identify unsafe actions and integrating safety through action, state, loss, or reward mechanisms. The paper implements these integrations with 3DQN and tests cyclic and acyclic action designs.

RL Model

  • The implementation uses 3DQN with cyclic actions from Liang et al. (2019) and acyclic actions from Ault, Hanna, and Sharon (2019), with cumulative vehicle waiting time as reward.Vehicle-tracking sensors are assumed to provide vehicle positions and speeds within a sensing radius.Source: RL Model · p. 4

Safety Module Integration: SafeLight

  • SafeLight uses a safety model to assess current traffic conditions and flag unsafe actions using logical rules abstracted from left-turn phasing guidelines.For example, a permitted but unprotected left turn is switched to protected mode when opposing vehicles approach at higher speed.Source: Safety Module Integration: SafeLight · p. 4
  • SafeLight-Act filters an RL action only when safety rules detect a potential issue, but frequent corrections can destabilize training and hinder convergence.This approach separates mobility learning from proactive safety intervention and requires minimal changes to the RL model.Source: SafeLight-Act · p. 4
  • SafeLight-Loss adds collision minimization as a second optimization objective, using KL divergence between normalized action advantages and the desired safe-advantage distribution.The safety term is zero when the RL action is safe, but its coefficient must be tuned for each intersection-specific scenario.Source: SafeLight-Loss · pp. 4–5
  • SafeLight-R adds a safety penalty to the reward using KL divergence from the desired safe-advantage distribution, but its performance depends on reward design and lacks a convergence proof.The added penalty is used when the safety model confirms an unsafe action.Source: SafeLight-R · p. 5
  • SafeLight-S&R embeds the safety model as an additional state feature, but requires SafeLight-Loss or SafeLight-R so training includes a safety-aware performance measure.The embedding is learned through an MLP and concatenated into the Q network.Source: SafeLight-S&R and SafeLight-S&Loss · p. 5

Experiments

Experiments evaluate SafeLight across synthetic and real-world traffic settings, different action spaces, and different RL backbones. The reported results show fewer collisions, with mobility and training performance varying across SafeLight variants and settings.

Evaluation Environment

  • Experiments used SUMO under time-variant traffic flow, with a synthetic four-approach intersection and a Cologne real-world intersection with eight approaching lanes.The setup assumed accurate vehicle detection, a common deceleration rate, and permitted left turns; collision cases were generated by increasing conflicting-vehicle noncompliance.Source: Intersection Modeling · p. 6, Evaluation Environment · p. 6, Collision Data · p. 6

Comparison Methods

  • Comparisons included Fixed-time control, the delay-minimizing 3DQN backbone, Synthetic Reward, and Synthetic Value methods.Synthetic Reward combines weighted rewards, while Synthetic Value combines weighted Q-values.Source: Comparison Methods · p. 6

Experiment Result

  • SafeLight-R achieved a 99.32% safety improvement on the synthetic setting, while SafeLight-Act had 93.94% and 86% fewer collisions on synthetic and real-world datasets, respectively.At least three SafeLight variants fell within the best-performance area in all three settings.Source: Figure 5 · p. 7
  • SafeLight-Act cannot eliminate two observed collision cases involving vehicles entering or remaining in the intersection during a yellow phase.One case involved an opposing through vehicle entering during yellow as a left-turning vehicle entered on green; another involved a vehicle still in the intersection when yellow changed to red.Source: Results of Evaluation Performance (RQ1) · p. 7
  • SafeLight-Act converged quickly to lower average waiting times than Backbone RL(3DQN) while maintaining strong safety performance during training.Figure 6 identifies SafeLight-R, SafeLight-Act, and SafeLight-Act as the best joint safety-and-mobility performers across the three settings, and those same variants except the second setting's SafeLight-Loss were fastest to optimum.Source: Convergence during Training · p. 7, Figure 6 · p. 8
  • SafeLight-Loss and SafeLight-Act performed relatively well across cyclic and acyclic action spaces, while Synthetic Reward failed to minimize collisions in the acyclic setting.In the IPPO backbone test with an acyclic action space, SafeLight also outperformed the other baselines in minimizing collisions.Source: Study of Action Generalizability (RQ2) · p. 8, Study of Model Generalizability (RQ3) · p. 8

Conclusion

SafeLight integrates a safety module into an underlying reinforcement-learning model and is evaluated against baseline models across benchmark datasets. The authors report generalizability across settings and describe their approach as a pathway for future safety-enforcement research.

Conclusion

  • SafeLight integrates a safety module into an underlying reinforcement-learning model and is compared with baseline models across benchmark datasets.The authors report that these comparisons demonstrate the method’s effectiveness.Source: Conclusion · p. 8
  • The authors report good generalizability across different settings, which they identify as critical for real-world deployment amid changing traffic demands.Source: Conclusion · p. 8
  • The authors present their pathway for enforcing safety as a possible guideline for future study.Source: Conclusion · p. 8
  • Future work will extend the method to more transportation rules, including specific yellow- and red-light timing rules.The authors state that the model could be used with such more specific domain safety rules.Source: Conclusion · p. 8
Loading 2211.10871v3…