Source-linked AI summary
A Survey on Traffic Signal Control Methods
Hua Wei, Guanjie Zheng, Vikash Gayah, Zhenhui Li
TL;DR
Traffic signal control must reduce urban intersection delays despite limited sensing and the complexity of real-world traffic. This survey synthesizes classical transportation methods and recent reinforcement-learning approaches using richer data and computational perspectives. It also identifies training efficiency and physical safety as important boundaries for RL deployment.
Problem
Existing traffic signal systems rely on oversimplified information and manually designed plans, while real-world traffic is complex and available mobility data are richer.
Method
The survey reviews classical transportation approaches and recent RL-based traffic signal control methods, including their state, reward, action, coordination, and experimental settings.
Results
The survey provides a comprehensive view of transportation-engineering and reinforcement-learning methods for traffic signal control.
Takeaways & Limitations
The survey can serve as a reference for comparing and exploring RL-based traffic signal control methods in intelligent transportation.
Takeaways & Limitations
High-dimensional image states can require many training samples and long training schedules without necessarily producing significant performance gains.
Abstract
from arXiv · showhide
Traffic signal control is an important and challenging real-world problem, which aims to minimize the travel time of vehicles by coordinating their movements at the road intersections. Current traffic signal control systems in use still rely heavily on oversimplified information and rule-based methods, although we now have richer data, more computing power and advanced methods to drive the development of intelligent transportation. With the growing interest in intelligent transportation using machine learning methods like reinforcement learning, this survey covers the widely acknowledged transportation approaches and a comprehensive list of recent literature on reinforcement for traffic signal control. We hope this survey can foster interdisciplinary research on this important topic.
1 INTRODUCTION
Traffic signal control is a major urban bottleneck problem, while existing systems rely on limited sensing and manually designed plans. The survey frames richer mobility data, stronger computing, and reinforcement learning as opportunities for broader traffic-control research.
- Motivation: Traffic congestion creates economic, environmental, and societal costs, making signalized intersections vital targets for urban traffic management.Signalized intersections are among the most prevalent bottleneck types in urban environments.
- Current Situation: SCATS and SCOOT adaptively select manually designed signal plans using traffic volumes detected by loop sensors.Many intersections lack loop sensors or have poorly maintained sensors, limiting the available information.
- Current Situation: Loop sensors provide only partial vehicle information, so signal systems cannot perceive and react to real-time traffic patterns.Engineers therefore need to manually change signal timings under certain traffic conditions.
- Opportunities: New cameras and trajectory sources can describe vehicle counts, turns, locations, speeds, and arrivals more comprehensively.Examples include surveillance cameras, navigation applications, ride-sharing platforms, and connected vehicles.
- Opportunities: Traditional optimization requires assumptions such as uniform, constant vehicle arrival rates, which can deviate substantially from evolving real-world conditions.Driver preferences, vulnerable road users, weather, and road conditions complicate traffic dynamics.
- Opportunities: Reinforcement learning addresses the lack of labeled good and bad signal strategies through trial-and-error interaction with traffic outcomes.The survey positions RL as an alternative to supervised learning for traffic signal control.
- Scope and Contribution: The survey covers classical transportation approaches alongside recent reinforcement-learning and deep-reinforcement-learning methods.It seeks to provide a comprehensive view spanning transportation engineering and machine learning.
2 PRELIMINARY
The preliminary section defines intersection structure, traffic movements, signal phases, signal plans, efficiency objectives, and practical timing constraints. These definitions establish the terminology used to describe traffic signal control methods.
- Term Definition: An approach is a roadway meeting an intersection, with incoming approaches admitting vehicles and outgoing approaches carrying vehicles away.An approach consists of incoming or outgoing lanes.
- Term Definition: A traffic movement carries vehicles from an incoming approach to an outgoing approach and is categorized as left, through, or right.The movement is denoted as (r_i → r_o), where r_i and r_o are incoming and outgoing lanes.
- Term Definition: A movement signal permits or prohibits a traffic movement, while a phase combines movement signals.The example four-leg intersection uses eight movement signals and a conflict matrix to represent their combinations.
- Term Definition: A phase sequence specifies the phases and their order, whereas a signal plan specifies repeated phase sequences and their starting times.Existing traffic signal control methods usually repeat similar phase sequences throughout the day.
- Objective: Traffic signal control seeks safe and efficient vehicle movement by separating conflicting movements in time.Efficiency can be measured using travel time, queue length, number of stops, and throughput.
- Special Considerations: Yellow and all-red intervals provide transition and clearance time before conflicting traffic movements receive green.The combined interval can last from 3 to 6 seconds and supports safe stopping or passage.
- Special Considerations: Minimum green time supports pedestrian passage, and a left-turn phase is usually added when left-turn volume exceeds a threshold.These timing rules are practical constraints in signal control design.
3 METHODS IN TRANSPORTATION ENGINEERING
Transportation-engineering methods range from isolated-intersection timing to coordinated arterial control and adaptive, demand-responsive strategies. Webster sets cycle length and green splits from traffic demand, while GreenWave and Maxband coordinate offsets under different directional objectives and constraints.
- 3.1 Webster (single intersection): Webster’s method calculates a single intersection’s pre-timed cycle length and phase split under an assumption of uniform traffic flow.Its equations are typically applied to aggregated data to develop fixed-time plans.
- 3.1 Webster (single intersection): Green splits are allocated proportionally to the critical lane volumes served by each phase.Critical lane volumes identify the busiest approaching lanes in the signal-timing context.
- 3.1 Webster (single intersection): The desired cycle length Cdes is the minimum cycle satisfying traffic demand when capacity equals demand.A shorter cycle causes queues to keep increasing, while a longer cycle increases mean vehicle delay linearly with cycle length.
- 3.2 GreenWave: GreenWave coordinates adjacent signals by optimizing offsets to reduce stops for vehicles traveling in one designated direction.It requires a common cycle length equal to the maximum individual-intersection cycle length, with offsets based on road length and expected speed.
- 3.3 Maxband: Maxband optimizes offsets and a maximal, symmetric, uniform-width bandwidth for two opposite traffic directions using a mixed integer linear program.Its physical constraints cover individual-intersection bandwidths plus temporal and spatial relationships between intersections.
4 REINFORCEMENT LEARNING BASED TRAFFIC SIGNAL CONTROL
Recent traffic signal control research has explored fuzzy logic, swarm intelligence, and reinforcement learning, with reinforcement learning becoming increasingly prominent.
- Researchers have proposed fuzzy logic, swarm intelligence, and reinforcement learning for traffic signal control.
- Reinforcement learning has become more prominent in recent years.
4.1 Preliminaries
The survey formulates traffic signal control with reinforcement learning as single-agent MDPs and multi-agent stochastic games, where controllers observe traffic, choose signal phases, and receive rewards. It also identifies reward, state, action, and coordination design as central modeling choices.
- Single-agent RL: A single-agent RL problem is modeled as a Markov Decision Process with states, actions, transitions, rewards, and a discount factor.
- Single-agent RL: The discount factor controls the importance of immediate rewards relative to future rewards in continuing agent-environment interactions.
- Single-agent RL: Optimal policies can be obtained through optimal action-value or state-value functions satisfying Bellman optimality equations.
- Single-intersection RL: Traffic signal control is mapped to reinforcement learning by representing traffic conditions as states, signal decisions as actions, and outcomes as rewards.For isolated intersections, states may include signal phase, vehicle waiting time, queue length, and vehicle positions.
- Multi-agent RL: In multi-intersection control, each intersection acts as an agent with local observations and phase actions, while rewards may target individual or grouped intersections.
- Design choices: RL traffic-signal formulations depend on reward design, state design, action schemes, and coordination strategy.Action definitions affect phase-sequence flexibility, while nearby signals interact through vehicles departing one intersection and arriving at another.
4.2 RL Formulation
RL traffic-signal formulations specify state representations, reward functions, and action definitions. The survey reviews common choices and highlights difficulties in representing traffic comprehensively, assigning principled rewards, and translating actions into effective control.
- 4.2 RL Formulation: Typical RL formulations combine queue length, waiting time, and delay in rewards, while states include queues, vehicle counts, waiting time, and signal phase.
- 4.2.1 State Definitions.: State representations can describe traffic at vehicle level through images or at lane level through sums or averages across vehicles.
- 4.2.1 State Definitions.: High-dimensional image states may require many training samples and longer training, without necessarily producing significant performance gains.
- 4.2.2 Reward Functions.: Travel time is difficult to use directly as an RL reward because it depends on factors beyond signals and may require unknown vehicle destinations.
- 4.2.2 Reward Functions.: Common reward factors include queue length, waiting time, speed, number of stops, and throughput.The survey states that minimizing queue length is equivalent to minimizing total travel time.
- 4.2.2 Reward Functions.: Reward factors may be represented as immediate values or differences in cumulative values, but the appropriate choice remains under discussion.
- 4.2.2 Reward Functions.: Weighted reward designs lack a guaranteed equivalence to travel-time optimization, and small weight changes can produce dramatically different results.The survey notes that no principled method yet exists for selecting the weights.
- 4.2.3 Action Definitions.: Action formulations include setting current phase duration, setting cycle-based phase ratios, and deciding whether to keep or change the current phase.
4.3 Learning Approaches
RL methods for traffic signal control are classified along several algorithmic dimensions, including model learning, value or policy learning, and tabular versus parameterized representations. The survey organizes these approaches for traffic-signal applications.
- Learning Approaches: RL methods are classified as model-based or model-free according to whether they learn the state-transition function.
- Learning Approaches: Methods are classified as value-based, policy-gradient, or actor-critic according to whether they learn value functions, policy parameters, or both.
- Learning Approaches: RL methods may use tabular representations or parameterized function representations for learned functions, policies, and models.
- Learning Approaches: The survey summarizes model-based and model-free methods used in RL-based traffic signal control.
4.3.3 Tabular methods and approximation methods.
The survey contrasts tabular and approximation-based reinforcement-learning methods, then reviews coordination, experimental settings, and challenges affecting real-world traffic-signal control.
- Tabular methods and approximation methods: Tabular Q-learning cannot scale to large state spaces and treats similar-feature samples as completely different states.Approximation methods address these limitations through parameterized function representations.
- Coordination strategies: Multi-intersection control can use global single agents, joint action modeling, or coordination strategies involving independent agents and communication.A global agent directly learns joint actions but faces exponential state-action growth.
- Coordination strategies: Without communication or coordination, neighboring agents create non-stationary environments in complex traffic settings, preventing convergence to stationary policies.Local observations can work in simple arterial networks by forming mini green waves, but dynamic environments expose coordination limits.
- Experimental settings: Traffic-signal-control experiments commonly rely on microscopic simulation because reinforcement learning requires detailed vehicle-level state representations.The survey discusses simulation environments, road networks, traffic flows, and tools including GLD and SUMO.
- Experimental settings: Traffic-flow studies represent each vehicle using origin, time, and destination, with synthetic data generating uniform or dynamically changing flow.The survey categorizes traffic-flow settings by network and arrival characteristics.
- Challenges: Current RL methods face challenges in reward and state formulation, learning efficiency, credit assignment, safety, and connection to transportation theory.Ad-hoc reward and state definitions can be sensitive to weighting, high-dimensional representations, and convergence requirements.
5 CONCLUSION
The survey presents a broad overview of traffic signal control, combining classical transportation approaches with reinforcement-learning methods and their experimental settings. It concludes by discussing challenges and future research directions for RL-based control.
- The article provides an overview of the traffic signal control problem, its terms, and its objectives.
- The survey reviews classical transportation approaches and state-of-the-art RL methods through agent formulation, coordination, and experiment settings.
- The article discusses challenges and future research directions intended to advance RL-based traffic signal control.
A.1 Summary of RL methods
The survey summarizes reinforcement-learning traffic signal control methods through overall comparison tables. These comparisons provide a consolidated view of the methods investigated across the reviewed literature.
- Table 14 summarizes the RL-based traffic signal control methods investigated in the survey.
- Together, the tables organize the survey’s reviewed RL methods for comparison.
- Table 15 provides another overall comparison of the RL-based traffic signal control methods investigated in the survey.