PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Reinforcement Learning with Partial Vehicle Detection for Intelligent Traffic Signal Control

Rusheng Zhang, Akihiro Ishikawa, Wenli Wang, Benjamin Striner, Ozan Tonguz

arXiv:1807.01628v3cs.AIcs.MA

TL;DR

Wireless vehicle detection creates a traffic-control problem because only vehicles with communication devices are observed, and deployment may begin with low penetration. This paper applies reinforcement learning to partially detected traffic control and reports reduced waiting times even at low penetration.

  • Problem

    Wireless-based traffic control observes only vehicles equipped with communication devices, creating a need for algorithms that work at low detection rates.

    Source: §I · p. 1

  • Method

    The paper applies deep Q-learning to traffic-signal control under partial vehicle detection, using DSRC as an example.

    Source: §I · p. 1, §VII · p. 10

  • Results

    The RL approach reduces average waiting time at intersections even at low penetration, and the study reports effectiveness across sparse, medium, and dense traffic flows.

    Source: §I · p. 1, §VII · p. 10

  • Takeaways & Limitations

    The authors conclude that RL is a promising approach for traffic control under partial detection, including systems using technologies beyond DSRC.

    Source: §I · p. 1, §VII · p. 10

Abstract

from arXiv · show

Intelligent Transportation Systems (ITS) have attracted the attention of researchers and the general public alike as a means to alleviate traffic congestion. Recently, the maturity of wireless technology has enabled a cost-efficient way to achieve ITS by detecting vehicles using Vehicle to Infrastructure (V2I) communications. Traditional ITS algorithms, in most cases, assume that every vehicle is observed, such as by a camera or a loop detector, but a V2I implementation would detect only those vehicles with wireless communications capability. We examine a family of transportation systems, which we will refer to as `Partially Detected Intelligent Transportation Systems'. An algorithm that can act well under a small detection rate is highly desirable due to gradual penetration rates of the underlying wireless technologies such as Dedicated Short Range Communications (DSRC) technology. Artificial Intelligence (AI) techniques for Reinforcement Learning (RL) are suitable tools for finding such an algorithm due to utilizing varied inputs and not requiring explicit analytic understanding or modeling of the underlying system dynamics. In this paper, we report a RL algorithm for partially observable ITS based on DSRC. The performance of this system is studied under different car flows, detection rates, and topologies of the road network. Our system is able to efficiently reduce the average waiting time of vehicles at an intersection, even with a low detection rate.

I. INTRODUCTION

The paper addresses traffic-signal control when wireless systems detect only equipped vehicles, and investigates reinforcement learning as a way to manage traffic under this partial detection. It positions this approach as a lower-cost alternative to traditional sensing and as a possible transition toward VTL.

I. INTRODUCTION

  • Wireless sensing is less costly than traditional cameras or loop detectors, but it cannot detect vehicles without compatible communication devices.The paper identifies DSRC, RFID, Bluetooth, UWB, Zigbee, and cellphone apps as examples of such sensing technologies.Source: §I · p. 1
  • The study explores DRL for partial-detection control and reports that its scheme can extend beyond DSRC to RFID, BLE 5.0, and cellular detection.The authors analyze performance through simulations.Source: §I · p. 1

II. RELATED WORKS

  • Earlier RL traffic-signal studies include approaches for parameter tuning and adaptive control, while deep Q-learning studies report optimization in fully observable settings.The cited prior work includes Q-learning, fuzzy-neural controllers, CMAC, and DQN-based approaches.Source: §I · p. 1, §II · p. 2
  • Existing partial-detection systems have different traffic-flow strengths: COLOMBO may miss optimal responses at low-to-medium flow, while DSRC-actuated lights are less suited to high flow.The paper also notes that traditional ITSC systems are generally expensive and that earlier RL systems performed comparably to them.Source: §II · p. 2
  • The paper proposes a transition scheme toward VTL, where detected vehicles have lower commute times than undetected vehicles, potentially encouraging device adoption.VTL requires all vehicles to carry DSRC devices, motivating a transition from current systems.Source: §II · p. 2

I. INTRODUCTION

  • Under low detection rates, RL can reduce commuter waiting and perform almost as well as full-detection ITSC.The paper also reports performance improvement as DSRC penetration increases.Source: §I · p. 1, §II · p. 3

III. PROBLEM STATEMENT

A partially detected traffic-signal system contains both observable and unobservable vehicles. It selects intersection phases using information from detected vehicles while seeking to reduce delay for both groups.

A. What is a Partial Detection based ITSC System ?

B. Example PD-ITSC System Design based on DSRC

The example PD-ITSC design uses DSRC to send information from detected vehicles to an RL decision unit, while the broader algorithm is intended to handle partial detection. The paper uses deep Q-learning and stabilizing training methods for this control problem.

B. Example PD-ITSC System Design based on DSRC

  • The DSRC roadside unit parses vehicle safety messages and sends their information to the RL decision-making unit.The roadside unit receives Basic Safety Messages broadcast by onboard units.Source: B. Example PD-ITSC System Design based… · p. 3
  • Although DSRC does not detect every vehicle, it can provide detected vehicles’ speed, distance, trajectory, and destination.The authors state that similar information is available in many partial-detection systems using wireless technologies.Source: B. Example PD-ITSC System Design based… · p. 3
  • The proposed PD-ITSC scheme is intended to perform well at low detection rates.The authors also identify accelerating transition to higher adoption and detection rates as an ideal-system goal.Source: B. Example PD-ITSC System Design based… · p. 3

A. Q-Learning Algorithm

  • Target and online Q-networks plus replay-buffer sampling are used to stabilize training and reduce time correlation between training samples.The online network is periodically synchronized with the target network.Source: A. Q-Learning Algorithm · p. 4
  • The traffic-light agents are trained using a Deep Q-network (DQN).The paper focuses on defining the agents’ actions, states, and rewards.Source: A. Q-Learning Algorithm · p. 4

B. Parameter Modeling

The parameter-modeling subsection frames the traffic-light controller as an agent that receives state and reward observations and chooses actions.

B. Parameter Modeling

  • The controller receives state observations and reward information from the environment and chooses an action.The subsection introduces the design of actions, rewards, and states for the PD-ITSC problem.Source: B. Parameter Modeling · p. 4
  • The agent’s control loop is described through its inputs—state and reward—and its action choice.The passage characterizes the controller as a traffic-light controller interacting with the environment.Source: B. Parameter Modeling · p. 4
  • The subsection sets up the parameter choices by introducing action, reward, and state design for the PD-ITSC system.Source: B. Parameter Modeling · p. 4

1) Agent action:

At each time step, the agent can keep the current traffic-light phase or switch to the next. The reward design targets lower traffic delay using vehicle speed, with reward shaping identified as a possible future direction.

1) Agent action:

  • The agent either keeps the current traffic-light phase or switches to the next phase at each time step.Source: 1) Agent action · p. 4

2) Reward:

  • The objective is to minimize average travel-time delay, defined as tS − tmin, by choosing a traffic-light phasing strategy.Here, tS is average travel time under strategy S and tmin is the physically possible lowest average travel time.Source: 2) Reward · p. 4
  • When vehicles share the same vmax, minimizing delay is equivalent to maximizing vS(t); with differing vmax values, rewards average the per-vehicle function.Source: 2) Reward · p. 4
  • Reward shaping to guide actions, such as avoiding large traffic jams under heavy flow, is proposed as future research.Source: 2) Reward · p. 4

3) State representation:

The state representation combines information about detected vehicles with signal-phase and timing information. The paper encodes green versus red through signed lane counts and distances, reporting slightly faster convergence in initial experiments than with a separate indicator.

3) State representation:

  • Partial detection limits observations to a portion of vehicles, while wireless technologies may provide richer information about those detected vehicles than traditional occupancy detection.Source: 3) State representation · p. 5
  • The selected state includes nearest-vehicle distance and vehicle count at each approach, an amber-phase indicator, elapsed phase time, and current time.The authors note that further work is needed to compare input and representation choices.Source: 3) State representation · p. 5
  • Green versus red is encoded by the sign of per-lane detected-car counts and distances rather than a separate indicator.Source: 3) State representation · p. 5
  • Initial experiments found slightly faster convergence with the distributed sign representation than with a separate phase indicator.The authors hypothesize that the encoding, together with ReLU activation, may encourage different neuron combinations for different phases.Source: 3) State representation · p. 5

C. System

The control unit uses detected traffic states and Q-values to retain or change signal phases, with minimum and maximum phase-duration checks. The agent is trained in simulation before deployment, using traffic patterns informed by historical arrivals.

C. System

  • The controller periodically evaluates Q-values for keeping the current phase or switching to the next phase, subject to minimum and maximum phase-duration rules.It receives a state representation before selecting an action.Source: C. System · p. 5

D. Implementation

  • The agent is trained in a simulator, then ported to an intersection and connected to its real traffic signal.Training and deployment form two implementation phases.Source: D. Implementation · p. 5, Figure 4 · p. 5

1) Training phase:

  • The simulator generates vehicle arrivals and samples detection using a Bernoulli distribution parameterized by the detection rate; the agent and simulator repeatedly exchange states, rewards, and actions until convergence.For DSRC, the detection rate corresponds to the DSRC penetration rate.Source: 1) Training phase · pp. 5–6
  • Historical arrival rates inform simulated car flow, while time of day is included in the state so the trained agent can adapt to daily traffic variation.Day of week could also be parameterized in the state representation.Source: 1) Training phase · p. 6
  • Training seeks a control scheme that minimizes commuters’ average commute time.The agent tries different schemes and converges toward one yielding a minimum average commute time.Source: 1) Training phase · p. 6

2) Deployment phase:

In simulation, reinforcement-learning control outperformed a pre-timed signal even at low detection rates, and its waiting-time benefit grew as detection increased. The study also reports substantial early-transition benefit and adaptive behavior across traffic-flow conditions.

2) Deployment phase:

  • After training, the deployed agent uses the trained Q-network to select whether to keep or switch the signal phase in real time, without updating that network.The detector supplies the current detected traffic state.Source: 2) Deployment phase · p. 6

A. Performance for different detection rates

  • Under medium car flow, AI-controlled signal timing outperformed a 24-second pre-timed signal even at low detection rates, while overall waiting time declined as detection increased.The simulation used Poisson arrivals and specified approach flows.Source: A. Performance for different detection… · pp. 6–7, Figure 6 · p. 6
  • Approximately 80% of the benefit occurs in the first 20% of the detection-rate transition.The authors describe the benefit as accruing early as detection increases.Source: A. Performance for different detection… · p. 7
  • During the detection transition, detected vehicles had lower commute times than undetected vehicles, although the agent was rewarded for optimizing average commute time across both groups.The authors describe this difference as a potential incentive for drivers to equip vehicles with IoT devices.Source: A. Performance for different detection… · p. 7
  • Reinforcement-learning agents performed better than DSRC-ATL throughout the 0-to-1 detection-rate transition.DSRC-ATL was designed for partial detection under sparse-to-medium car flow.Source: A. Performance for different detection… · p. 7
  • At dense flow, performance changed little with detection rate; the trained agent adapted between individual-arrival and macroscopic-flow strategies, while sparse-flow results followed the medium-flow trend.The authors characterize this as adaptive behavior across car-arrival rates.Source: A. Performance for different detection… · p. 7

B. Performance of a whole day

In the simulated day, 20% detection performed close to 100% detection for most daytime hours, with a visible full-detection advantage at midnight. The authors caution that the traffic profile is an example rather than a universal intersection pattern.

B. Performance of a whole day

  • The simulated daily flow was based on reported traffic, scaled to match road saturation at peak hours; peak flow was 1.2 vehicles/s and regular-hour flow about 0.7 vehicles/s.The profile peaked at 8 a.m. and 6 p.m.; the authors note that real intersections can differ substantially.Source: B. Performance of a whole day · p. 7, Figure 8 · p. 7
  • From 5 a.m. to 9 p.m., performance with 20% detection was close to performance with 100% detection; the full-detection system performed visibly better at midnight.The authors describe daytime performance as more critical than midnight performance.Source: B. Performance of a whole day · pp. 7–8, Figure 9 · p. 8
  • The authors attribute the small daytime performance gap to low-variance flow estimates at high car volumes, while few cars and low detection can produce high-variance estimates.They contrast this with late-night conditions where a single detected car could receive an immediate green, unlike an undetected car.Source: B. Performance of a whole day · p. 8

C. Sensitivity Analysis

The sensitivity analysis tests agents outside their training flow and detection-rate settings. Performance remains near optimal over some ranges, but the response is asymmetric for agents trained at low detection rates.

C. Sensitivity Analysis

  • The analysis evaluates sensitivity to car-flow and detection-rate changes after agents were trained and evaluated under matching environmental parameters.The authors motivate the tests by possible differences between training and operating conditions.Source: C. Sensitivity Analysis · p. 8

1) Sensitivity to car flow:

  • An agent trained at 0.1 veh/s remained near optimal from 0 to 0.15 veh/s, while one trained at 0.5 veh/s performed reasonably from 0.25 to 0.5 veh/s but substantially worse below 0.25 veh/s.The comparison is between trained agents and agents trained and tested under the flow being evaluated.Source: 1) Sensitivity to car flow · p. 8, Figure 10 · p. 8

2) Sensitivity to detection rate:

  • An agent trained at a 0.2 detection rate performed optimally from 0.1 to 0.4, with better sensitivity upward than downward; the authors suggest underestimating detection early in deployment.The comparison tests performance as actual detection rate differs from the training rate.Source: 2) Sensitivity to detection rate · p. 8, Figure 11 · p. 8
  • An agent trained at a 0.8 detection rate performed optimally from 0.5 to 1, and both tested agents remained stable when training and actual detection rates were not too different.The low- and high-training-rate cases had different sensitivity patterns.Source: 2) Sensitivity to detection rate · p. 8, Figure 11 · p. 8

D. Robustness between training and deployment scenario

The agent is trained in a simplified simulator and evaluated in the realistic LuST scenario, with deliberate differences between training and evaluation. Its performance improves asymptotically as detection rate grows, even across sparse, medium, and dense traffic periods.

D. Robustness between training and deployment scenario

  • The agent is trained in a separate simplified simulator using the hourly traffic volume measured at LuST intersection -12408, then evaluated at that intersection in the original scenario.The evaluation substitutes the trained agent into the original LuST scenario rather than training directly on it.Source: D. Robustness between training and… · p. 9
  • Training and evaluation intentionally differ in scenario details to estimate performance when simulation and real-world conditions do not match.The authors frame the LuST evaluation as a reference for real-world implementation under such differences.Source: D. Robustness between training and… · p. 9
  • Evaluation covers midnight sparse flow, 8 AM rush-hour dense flow, and 2 PM medium flow.LuST is presented as a realistic scenario, generated on Luxembourg's real map and compared with observed speed distributions.Source: D. Robustness between training and… · pp. 8–9
  • Performance improves asymptotically as detection rate grows in the LuST scenario, following the same trend observed in V-A.Figure 12 presents the agent's LuST performance.Source: D. Robustness between training and… · pp. 9–10, Figure 12 · p. 9

VI. DISCUSSION

The discussion argues that partial-detection traffic control can create commercial incentives beyond the Department of Transportation, while also supporting a form of dynamic pricing. The conclusion presents deep Q-learning as promising across traffic flows, but notes that the system must be fully trained in simulation before deployment.

VI. DISCUSSION

  • Detected vehicles have shorter commute times than undetected vehicles, while all vehicles experience shorter waiting times under RL control.The authors argue that this difference can give hardware, software, and vehicle companies an incentive to support deployment.Source: §VI · p. 10
  • The authors argue that distributing deployment incentives across companies could make installation financially feasible at most city intersections, unlike current limited ITSC deployment.They contrast this with traditional ITSC, whose deployment burden falls on the Department of Transportation alone.Source: §VI · p. 10
  • The system could support dynamic pricing by offering vehicle detection as a service, with users able to pay for a prioritized signal phase.The proposed approach does not require reserving roads for paid users.Source: §VI · p. 10
  • The system must be fully trained in a simulator because partial observation prevents it from observing reward and continuing incremental training after deployment.The authors identify this as a drawback and call for further work to improve practicality.Source: §VI · p. 10

VII. CONCLUSION

  • The paper proposes deep Q-learning for partial vehicle detection and presents it as promising given gradual DSRC penetration forecasts.The conclusion frames partial detection as applicable to systems such as DSRC.Source: §VII · p. 10
  • Results across sparse, medium, and dense arrival rates suggest reinforcement learning can handle all kinds of traffic flow by leveraging both particle and liquid flow properties.The conclusion describes this as an overall optimization scheme despite the general differences between sparse- and dense-arrival optimization.Source: §VII · p. 10
Loading 1807.01628v3…