PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Deterministic Policy Gradient for Urban Traffic Light Control

Noe Casas

arXiv:1703.09035v2cs.NE

TL;DR

Urban traffic-light control remains difficult because realistic networks expose large, continually sampled traffic state spaces, while prior methods often focus on localized intersections. The paper applies DDPG with detector-based inputs and stabilizing techniques across increasingly complex networks, finding strong small-network performance but insufficient control of the real-world scenario.

  • Problem

    Realistic traffic-light control involves large state spaces from network-wide detector data, while prior approaches often focus on localized intersections and offline strategies that do not adapt to changing demand.

  • Method

    The paper applies DDPG to traffic-light timing using detector counts and average speeds, with replay memory, gradient clipping, discount scheduling, and cycle-preserving actions.

  • Results

    DDPG performs well on small and medium-sized networks, is more stable than Q-learning in the simplest scenario, but does not control the real-world network better than doing nothing.

  • Takeaways & Limitations

    Simple deep-learning architectures can control small and medium-sized traffic networks, while larger networks require further study involving longer training, network geometry, and improved data efficiency.

  • Takeaways & Limitations

    Real-world applicability remains limited because performance varies with traffic-demand conditions and detector placement, with observability loosely related to algorithm performance.

Abstract

from arXiv · show

Traffic light timing optimization is still an active line of research despite the wealth of scientific literature on the topic, and the problem remains unsolved for any non-toy scenario. One of the key issues with traffic light optimization is the large scale of the input information that is available for the controlling agent, namely all the traffic data that is continually sampled by the traffic detectors that cover the urban network. This issue has in the past forced researchers to focus on agents that work on localized parts of the traffic network, typically on individual intersections, and to coordinate every individual agent in a multi-agent setup. In order to overcome the large scale of the available state information, we propose to rely on the ability of deep Learning approaches to handle large input spaces, in the form of Deep Deterministic Policy Gradient (DDPG) algorithm. We performed several experiments with a range of models, from the very simple one (one intersection) to the more complex one (a big city section).

1. Introduction

Urban traffic-light control must handle network-wide traffic information and coordinate decisions at scale. Existing fixed, actuated, and many learning-based approaches remain local or poorly scalable, motivating a large-scale traffic-data-driven agent.

  • Fixed programs use predetermined phase durations, whereas actuated signals adapt locally to detector data but do not coordinate nearby intersections.
  • Prior machine-learning research applied reinforcement learning to traffic-light control, including the Green Light District simulator.
  • Most previous approaches simplify the problem to one intersection or a small group, sometimes coordinating separate intersection agents.
  • Thousands of detectors and hundreds of traffic lights create huge state and action spaces that classical approaches struggle to handle.
  • The paper therefore develops an agent that uses traffic data to control traffic lights and improve traffic flow at large scale.

2. Traffic Simulation

The evaluation uses traffic simulation to represent roads, intersections, and vehicle movement under varying origin-destination demands. Simulators may model individual vehicles microscopically or aggregate traffic macroscopically, with mesoscopic models between them.

  • A traffic simulation network represents roads and intersections, while centroids generate or absorb vehicles according to origin-destination matrices.
  • Different origin-destination matrices can be applied during separate simulation periods to represent changing traffic demand.
  • Microscopic simulators compute individual vehicle positions, macroscopic simulators aggregate traffic flow, and mesoscopic simulators occupy the intermediate range.

3. Preliminary Analysis

The paper frames urban traffic control as a large-state-and-action-space problem and proposes deep reinforcement learning to address it. Its design must represent traffic information, preserve feasible signal actions, stabilize training, and use realistic testing environments.

  • Deep reinforcement learning is proposed for urban traffic control because deep learning can handle the large state and action spaces of realistic experiments.
  • The work’s contribution includes applying techniques intended to make large-scale deep-reinforcement-learning traffic control feasible and effective.
  • A sensible state space requires representing traffic information, which may be better modeled as a labelled graph than as an image-like matrix.
  • The action space must preserve synchronized intersection routing rules rather than independently setting every signal color.
  • Because convergence is not guaranteed, the study examines training stability and introduces measures to mitigate divergence.
  • The evaluation requires a realistic test bed that simulates big-city traffic and city structure.

4. Related Work

Related work spans fixed-timing, model-based, simulation-based, actuated, classical reinforcement-learning, and deep-reinforcement-learning approaches. Common limitations include weak adaptation, local control, unrealistic information or environments, and restricted test scenarios.

  • Offline Approaches: Offline fixed-timing methods derive phase plans in advance, often from models or iterative simulation-based optimization.
  • Offline Approaches: Historical time-band strategies do not adapt to changing traffic demand or progressive changes in driver behavior.
  • Local and Network Control: Classical local approaches use nearby detector data with thresholds, rules, or optimization to reduce intersection waiting times.
  • Local and Network Control: More complex network methods tune offsets and splits across multiple intersections using real-time detector data and traffic-network models.
  • Reinforcement Learning: Classical reinforcement-learning studies commonly control single intersections, while multi-agent methods address high-dimensional state and action spaces with varying coordination.
  • Reinforcement Learning: Many existing reinforcement-learning approaches rely on information unavailable in real environments and lack realistic testing environments.
  • Deep Reinforcement Learning: Deep-reinforcement-learning studies include single-intersection DQN systems using lane queues, vehicle-location images, or engineered traffic features as inputs.

5. Theoretical Background

Reinforcement learning trains an agent to choose actions from observable states using reward feedback, while deep reinforcement learning uses neural networks to approximate value functions or policies. DDPG extends actor–critic methods to continuous actions and uses mechanisms such as replay, target networks, and exploration noise to improve stability and exploration.

  • Reinforcement Learning: Reinforcement learning learns an optimal policy from observable environment states and reinforcement signals that evaluate action effects.The agent seeks to maximize accumulated rewards over time.
  • Deep Reinforcement Learning: Deep reinforcement learning uses deep neural networks as value-function approximators but historically faced instability from correlated data, policy oscillations, and large optimization steps.These issues motivated stabilizing techniques for neural reinforcement learning.
  • Deep Q-Networks: DQN stabilizes learning through experience replay, reward clipping, and a separate target network.Replay reduces temporal autocorrelation, clipping limits reward scale, and target networks reduce policy oscillations.
  • DDPG: DDPG is introduced for continuous action spaces by combining actor–critic learning with deterministic policy gradients.A neural-network actor represents the policy and a separate critic evaluates state–action choices.
  • DDPG: DDPG applies replay buffers and separate target actor and critic networks, while random policy noise supports exploration.The target networks compute target Q-values while the main networks are updated during learning.

6. Proposed Approach

The proposed approach applies DDPG to urban traffic-light control using traffic-detector information and a simulator-defined algorithmic pipeline. The method initializes actor–critic and target networks, explores with noise, and learns from replayed transitions.

  • Input and Representation: The approach selects traffic information from the simulator and defines representations for the traffic state and rewards.The authors constrain inputs to information intended to be available in typical urban traffic setups.
  • Algorithm: The algorithm initializes actor, critic, target-network weights, and a replay buffer before processing episodes.Each episode begins with a random exploration process and an initial observation state.
  • Algorithm: At each step, the actor selects a noisy action, observes a reward and next state, stores the transition, and samples replay data for critic updates.The target value combines the observed reward with a discounted target-critic estimate.
  • Input and Representation: Traffic detectors provide real-time measurements such as vehicle counts and speeds that can supply the algorithm’s input data.The detector data is intended to connect the simulated method with real urban traffic instrumentation.

6.2. Congestion Measurement

The proposed congestion measurement summarizes detector-level traffic using vehicle counts, average speeds, and occupancy, centered on a normalized speed score. This score ranges from 0 to 1 and supports both state and reward construction.

  • Congestion Measurement: The traffic summary uses vehicle counts, average speeds, and occupancy from real-world-available detector data.The authors define these measurements under a constraint to use data available in realistic deployments.
  • Congestion Measurement: The speed score compares each detector’s average speed with the maximum speed on its road.For detector i, it is based on avg speed_i relative to max speed_i.
  • Congestion Measurement: The speed score ranges in [0, 1] and is used to construct both the environment state and reinforcement-learning rewards.The measure also incorporates each road’s maximum-speed reference.

6.3. Data Aggregation Period

The method aggregates detector information to maintain a traffic state while constraining traffic-light actions to synchronized phases. The speed score is used because it reflects both relative vehicle speed and road-specific maximum speed.

  • Data Aggregation Period: The simulator advances in 0.75-second steps, but detector vehicle counts do not change sufficiently at every step.The method therefore requires an aggregation period longer than the simulator’s default step.
  • Data Aggregation Period: The state vector contains one speed-score component per detector.This representation summarizes network congestion while incorporating each road’s maximum speed.
  • Data Aggregation Period: Higher speed scores indicate vehicle speeds closer to the road maximum and therefore higher traffic flow.The speed score links detector-level observations to the traffic-flow representation.
  • Action Space: Actions are restricted to traffic-light timing phases because freely setting each light’s color could create chaotic intersection behavior.Intersection signals are synchronized so perpendicular traffic is stopped when another direction receives green.
  • Action Space: Phase-duration scaling is applied to only 80% of the duration to ensure a minimum phase duration.This constraint limits how aggressively the controller can modify timing.

6.6. Rewards

The reward compares detector speed scores with a no-intervention baseline, weights the difference by vehicle counts, and scales it for stable optimization. Rewards are computed per detector and used in DDPG.

  • 6.6. Rewards: The reward uses the difference between a detector’s speed score and its same-timestep no-intervention baseline.The baseline comes from a hypothetical simulation otherwise identical to the evaluated simulation.
  • 6.6. Rewards: Vehicle counts scale each detector reward so scores involving more vehicles receive greater weight.
  • 6.6. Rewards: A factor α scales rewards to keep them in a narrow range and control the scale of resulting gradients.The authors choose α empirically rather than clipping rewards, preserving information about reward magnitude.
  • 6.6. Rewards: Baseline simulations use the same random seed as the evaluated simulation because the microsimulator is stochastic.
  • 6.7. Deep Network Architecture: The actor network receives the simulation state and outputs traffic-light actions, while the critic evaluates the state-action pair with Q-values.

6.8. Disaggregated Rewards

The paper uses disaggregated detector-level rewards rather than scalarizing them, extending DDPG to an N-dimensional reward vector while addressing convergence through discount-factor scheduling and gradient monitoring.

  • Disaggregated Rewards: The method preserves an N-dimensional reward vector, where N is the number of detectors, instead of scalarizing rewards and losing location-specific effects.The extension treats the reward as r : S ×A →RN and leverages DDPG's critic-gradient updates without imposing reward-dimensionality constraints.
  • Disaggregated Rewards: Disaggregated rewards effectively implement multiobjective reinforcement learning through shared actor and critic networks trained over N unidimensional reward functions.The authors describe this as analogous to N agents sharing network weights θπ and θQ.
  • Disaggregated Rewards: The approach is reported as useful in the experiments, but the authors identify full characterization of its effects on benchmark problems as future work.They also suggest weighting rewards using traffic-control expert knowledge in future policy-gradient computations.
  • Convergence: Discount-factor scheduling reduces coupling in the closed-loop value-function updates, where training-induced drift can otherwise amplify and ruin learning.The schedule is applied at the experiment level rather than within an episode, and its oscillation is intended to keep the critic out of a divergent feedback regime.
  • Convergence: Monitoring the actor's gradient norm provides a convergence signal, while gradient norm clipping limits excessively large updates; the clipping threshold was empirically set to 0.5.A decreasing and persistently low gradient norm indicates a stable point where further improvement may stop.

6.10. Summary

The paper applies DDPG to urban traffic-light timing using neural actor and critic networks that process detector data and retain detector-level rewards. The design adds replay memory, gradient clipping, and discount-rate scheduling to improve convergence while preserving cycle duration.

  • 6.10. Summary: The proposal applies DDPG to traffic-light timing with multilayer-perceptron actor and critic networks.The actor modifies traffic-light timings while preserving cycle duration, and the networks are optimized with stochastic gradient descent.
  • 6.10. Summary: The state combines detector vehicle counts and average speeds into a single speed score, while rewards measure improvements over an uncontrolled baseline.Rewards are fed directly as expected critic values rather than aggregated.
  • 6.10. Summary: Replay memory, gradient norm clipping, and discount-rate γ scheduling are used to improve convergence.

7. Experiments

The experiments evaluate DDPG, multi-agent Q-learning, and random timing across increasingly complex traffic networks. DDPG is stable and competitive on simple networks, but real-world network performance remains below zero and convergence is unresolved.

  • Experimental design: The study compares DDPG, multi-agent Q-learning, and random timing across traffic networks of increasing complexity.DDPG uses all detector information to generate timings for all traffic-light phases, while the reference methods provide comparison baselines.
  • Network scenarios: Network A contains one intersection with two traffic-light phases and deliberately unbalanced phase durations that allow timing improvements.The network has eight detectors and constant vehicle demand over a one-hour simulation.
  • Network scenarios: Network B contains six signalized intersections, 17 detectors, and 30 phases in a congested grid where all turning movements are allowed.Its phase timings and demand were defined algorithmically or by hand using the same classical approach as network A.
  • Network scenarios: Network C reproduces Barcelona’s Sants area with 43 junctions, 102 traffic-light phases, and 29 detectors under peak-hour congestion.The detector locations match the real network, making this the largest and most realistic tested scenario.
  • Results: On network A, DDPG and Q-learning reach the same reward levels, but DDPG remains stable after peak performance while Q-learning is unstable.Performance is compared using the episode average reward of the best experiment trial.
  • Results: On network B, DDPG begins converging while Q-learning maintains a similar variation band, whereas on network C both methods perform equally below zero.The authors attribute unresolved medium- and large-network convergence to insufficient training time or divergence, requiring further study.

8. Conclusions

DDPG scaled better than tabular Q-learning as traffic networks became more complex, but real-world and larger-network performance remained limited. The study also introduced disaggregated rewards while identifying simulation, detector coverage, and network-geometry limitations.

  • Results: DDPG improved consistently as network complexity increased, whereas Q-learning no longer scaled beyond smaller networks.On the real-world scenario, however, DDPG did not control traffic better than doing nothing.
  • Limitations: Real-world applicability requires broader simulation variation, including changing traffic demand and road incidents, plus further study of detector amount and placement.Networks A and B used detectors at every network section, limiting the tested detector configurations.
  • New directions: The study introduced disaggregated rewards as a new deep reinforcement learning concept, but their contribution requires isolated benchmark evaluation.This would separate the technique’s effect from other factors influencing algorithm performance.
  • Limitations: The method did not exploit geometric information about the traffic network, motivating future work with graph-based convolutional approaches.The authors identify network geometry as a concrete limitation and future research direction.
  • Results: The approach performed well on small and medium-sized traffic networks, while larger networks require further study and substantially longer training.The authors also point to geometric information and data-efficiency improvements as directions for larger networks.
Loading 1703.09035v2…