Source-linked AI summary
Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning
Zhenning Li, Hao Yu, Guohui Zhang, Shangjia Dong, Cheng-Zhong Xu
TL;DR
Network-wide adaptive traffic signal control must coordinate interacting intersections under changing traffic conditions, while existing multi-agent approaches face communication and scalability challenges. The paper proposes KS-DDPG, which shares a compact collective representation of the traffic environment among agents. Across synthetic and real-world evaluations, KS-DDPG outperforms the reported baselines, with more stable training and acceptable computational costs.
Problem
Network-wide traffic signal control involves interacting intersections, but centralized or non-communicating multi-agent methods face scalability, non-stationarity, and coordination challenges.
Method
KS-DDPG extends multi-agent reinforcement learning with a knowledge-sharing protocol that stores collective environmental knowledge in a compact container for agents to interpret alongside private observations.
Results
KS-DDPG achieves superior performance relative to the reported conventional and reinforcement-learning baselines, including at least 10% control improvements over MADDPG.
Takeaways & Limitations
The results support explicit communication as a useful mechanism for coordinating agents in network-wide traffic signal control, with acceptable computational costs.
Takeaways & Limitations
All agents communicate throughout modeling, limiting overall communication efficiency and potentially constraining scalability in large systems.
Abstract
from arXiv · showhide
Inefficient traffic control may cause numerous problems such as traffic congestion and energy waste. This paper proposes a novel multi-agent reinforcement learning method, named KS-DDPG (Knowledge Sharing Deep Deterministic Policy Gradient) to achieve optimal control by enhancing the cooperation between traffic signals. By introducing the knowledge-sharing enabled communication protocol, each agent can access to the collective representation of the traffic environment collected by all agents. The proposed method is evaluated through two experiments respectively using synthetic and real-world datasets. The comparison with state-of-the-art reinforcement learning-based and conventional transportation methods demonstrate the proposed KS-DDPG has significant efficiency in controlling large-scale transportation networks and coping with fluctuations in traffic flow. In addition, the introduced communication mechanism has also been proven to speed up the convergence of the model without significantly increasing the computational burden.
1. Introduction
Adaptive traffic signal control uses reinforcement learning to respond to changing traffic conditions, but scaling cooperation across networked intersections remains difficult. The paper addresses this with KS-DDPG, which enables agents to share collective environmental knowledge.
- 1. Introduction: Adaptive traffic signal control responds to real-time traffic conditions, including traffic volume, queue length, vehicle speed, and travel time.Compared with fixed-time control, ATSC has been reported to reduce travel time by more than 10% on average.
- 1. Introduction: The paper frames reinforcement learning as a promising alternative to model-driven traffic control because it learns from interaction without requiring strong assumptions about traffic dynamics.RL learns policies through trial and error using rewards or penalties.
- 1. Introduction: Network-wide traffic control is difficult because centralized multi-agent policies grow exponentially with the number of intersections.Centralized schemes also face stability and exploration challenges as agents interact in larger networks.
- 1. Introduction: Independent agents that rely only on private observations may perform acceptably in simple networks but struggle with non-stationarity and prolonged learning in complex networks.Prior work identifies communication and coordination as important for improving multi-intersection control.
- 1. Introduction: KS-DDPG extends MADDPG with a knowledge-sharing mechanism that lets agents communicate through a compact representation of collective environmental understanding.Each agent combines its private observation with its own interpretation of the shared knowledge when making decisions.
2. Related Work
The related-work section introduces reinforcement-learning foundations for traffic signal control, from Q-learning and policy gradients to deterministic and multi-agent actor-critic methods. These methods motivate centralized critics and policy learning for interacting agents.
- 2. Related Work: The traffic-control problem is formulated as a partially observable multi-agent decision process with private observations, joint actions, transitions, rewards, and policies.Each agent observes only a partial characterization of the current state.
- 2. Related Work: Q-learning fits a parametric Q-function to estimate action values, while DQN learns the optimal action-value function using a stabilized target network.The target Q function is updated intermittently to stabilize learning.
- 2. Related Work: Policy-gradient methods directly adjust policy parameters to maximize expected return, with actor-critic methods approximating the action-value function.The policy gradient uses the state distribution and action-value estimates to update the policy.
- 2. Related Work: Deterministic policy gradients target continuous action spaces and require integration over states rather than both states and actions.This can reduce sample requirements in problems with large action spaces.
- 2. Related Work: MADDPG extends actor-critic policy gradients to multiple agents by augmenting each critic with other agents’ policies and centralized state information.Its centralized-learning and decentralized-execution structure addresses environment non-stationarity in multi-agent settings.
3. Methodology
KS-DDPG equips each intersection agent with a shared, compact representation of network knowledge while retaining agent-specific interpretations. Its actor-critic design combines local observations, obtained knowledge, and updated shared knowledge to select practical traffic-signal actions.
- Knowledge-sharing architecture: KS-DDPG uses a centralized knowledge container that progressively collects agents’ observations and makes collective knowledge accessible across intersections.Agents encode private observations, obtain and update shared knowledge, then use it during decisions.
- Knowledge-sharing architecture: Each agent embeds local traffic observations, interprets stored knowledge, and updates the shared content through sequential knowledge-obtaining and knowledge-updating operations.The observation includes traffic volume on entrance lanes and the current intersection phase.
- Knowledge-sharing architecture: GRU-inspired gates regulate knowledge retention and replacement, allowing agents to balance previously stored information with candidate knowledge derived from current observations.The update gate controls the mixture of existing and candidate knowledge, while the reset gate controls how much past knowledge is forgotten.
- Actor-critic policy: The actor selects actions from the embedded observation, the agent’s interpretation of shared knowledge, and the updated knowledge representation.The resulting policy parameters include components from the encoding, obtaining, and updating phases.
- Actor-critic policy: KS-DDPG adopts centralized learning with decentralized execution, using actor-critic networks in which actors choose actions and critics evaluate them.The learning procedure uses replay-buffer transitions and target networks within the actor-critic framework.
- Agent design: The traffic-signal agent can extend the current phase or switch phases, with bounded green, yellow, red, and cycle settings supporting practical acyclic control.Variable phasing permits unnecessary phases to be skipped in response to traffic fluctuations.
4. Experiments
KS-DDPG is evaluated in synthetic Grid and real-world MoCo networks against conventional and reinforcement-learning baselines. It converges more reliably, reduces queues more effectively across traffic conditions, and adds acceptable computational cost.
- General Setups: The experiments use a 10×10 Manhattan-style Grid and a real-world MoCo network with heterogeneous signalized and non-signalized intersections.SUMO and its TraCI interface provide the scalable simulation platform, while comparisons include fixed-time control, MaxPressure, DQN, DDPG, and MADDPG.
- Convergence Comparison: KS-DDPG converges in the large-scale Grid experiment, whereas independently trained DQN and DDPG fail to learn stable policies within 1,200 episodes.MADDPG also converges, but KS-DDPG has the more favorable learning curve.
- Convergence Comparison: KS-DDPG performs better than communication-free MADDPG on jumpstart performance, asymptotic performance, and time to threshold in both experiments.The results indicate that knowledge sharing speeds the search for an optimal policy rather than retarding convergence.
- Control Performance Comparison: In the MoCo congestion period, KS-DDPG keeps maximum average queue length below 10 veh/ln, while fixed-time control exceeds 25 veh/ln.Other RL methods outperform fixed-time control but remain worse than KS-DDPG during the sharp traffic-volume increase.
- Control Performance Comparison: KS-DDPG achieves consistent performance improvements over conventional transportation and RL-based baselines across synthetic and real-world networks and traffic conditions.Its advantage becomes larger when moving from the regular Grid network to the irregular, dynamic MoCo environment.
- Control Performance Comparison: KS-DDPG improves each measured control metric by at least 10% over MADDPG, while its training time remains comparable to MADDPG.The comparable training time indicates that the communication channel adds acceptable computational burden.
5. Conclusions
KS-DDPG optimizes network-wide adaptive traffic signal control through knowledge-sharing communication among agents. It outperforms the evaluated conventional and reinforcement-learning baselines, while its communication process remains a stated limitation for overall efficiency.
- KS-DDPG achieves superior traffic-control performance relative to conventional and reinforcement-learning baselines with acceptable computational costs.
- The knowledge container represents collective traffic-environment learning and helps agents form individual policies for improved coordination.
- The method is evaluated in synthetic and real-world scenarios to assess network-wide adaptive traffic signal control.
- Continuous communication by all agents limits overall communication efficiency during modeling.