PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning

Xiaoqiang Wang, Liangjun Ke, Zhimin Qiao, Xinghua Chai

arXiv:1908.03761v2cs.LGcs.MAstat.ML

TL;DR

Large-scale traffic signal control needs scalable multi-agent coordination while modeling interactions among signal agents. The paper proposes Co-DQL, combining independent double Q-learning, UCB exploration, mean field approximation, reward allocation, and local state sharing. Across traffic scenarios, Co-DQL outperforms several decentralized MARL baselines and shortens system-wide vehicle waiting time.

  • Problem

    Large-scale TSC remains difficult because MARL must scale to many intersections and model the behavior of interacting agents.

  • Method

    Co-DQL combines independent double Q-learning with UCB exploration, mean field approximation, reward allocation, and local state sharing.

  • Results

    Co-DQL outperforms several state-of-the-art decentralized MARL algorithms across traffic scenarios and effectively shortens vehicles’ average waiting time.

  • Takeaways & Limitations

    The results support Co-DQL as a scalable cooperative approach for decentralized traffic signal control in the tested traffic scenarios.

  • Takeaways & Limitations

    The authors have not yet tested Co-DQL on a real city road network, and the linear reward weighting may not capture nonlinear neighboring-throughput relationships.

Abstract

from arXiv · show

Finding the optimal signal timing strategy is a difficult task for the problem of large-scale traffic signal control (TSC). Multi-Agent Reinforcement Learning (MARL) is a promising method to solve this problem. However, there is still room for improvement in extending to large-scale problems and modeling the behaviors of other agents for each individual agent. In this paper, a new MARL, called Cooperative double Q-learning (Co-DQL), is proposed, which has several prominent features. It uses a highly scalable independent double Q-learning method based on double estimators and the UCB policy, which can eliminate the over-estimation problem existing in traditional independent Q-learning while ensuring exploration. It uses mean field approximation to model the interaction among agents, thereby making agents learn a better cooperative strategy. In order to improve the stability and robustness of the learning process, we introduce a new reward allocation mechanism and a local state sharing method. In addition, we analyze the convergence properties of the proposed algorithm. Co-DQL is applied on TSC and tested on a multi-traffic signal simulator. According to the results obtained on several traffic scenarios, Co- DQL outperforms several state-of-the-art decentralized MARL algorithms. It can effectively shorten the average waiting time of the vehicles in the whole road system.

I. INTRODUCTION

Large-scale traffic signal control requires adaptive coordination across many intersections, but centralized and existing MARL approaches face scalability and interaction-modeling challenges. The paper introduces decentralized Co-DQL to address these challenges in traffic signal control.

  • I. INTRODUCTION: Reinforcement learning is suited to adaptive traffic control because it can make real-time decisions from traffic flow and predict future flow.Deep neural networks allow value functions to handle large state or action spaces.
  • I. INTRODUCTION: Large-scale TSC is difficult because centralized methods incur global-state latency and exponentially expanding joint state and action spaces.These costs create communication, computation, and curse-of-dimensionality problems as intersections increase.
  • I. INTRODUCTION: MARL assigns each signalized intersection to an agent, but agents must handle dynamic interactions and coordinate across large urban networks.Existing MARL studies often evaluate only limited-size traffic networks.
  • I. INTRODUCTION: Co-DQL is a decentralized, scalable MARL method that applies independent double Q-learning with UCB exploration to traffic signal control.The design targets over-estimation in traditional independent Q-learning while preserving exploration.
  • I. INTRODUCTION: The paper combines mean field theory, reward allocation, and local state sharing to support cooperative, stable, and robust learning.It also provides convergence analysis and evaluates the method across traffic-flow scenarios.

III. DESCRIPTION OF THE PROPOSED METHOD

The proposed method is developed from independent double Q-learning, with the paper presenting that component before introducing Co-DQL and analyzing convergence.

  • III. DESCRIPTION OF THE PROPOSED METHOD: Co-DQL is developed from a new independent double Q-learning method introduced in this paper.The method description proceeds from independent double Q-learning to Co-DQL and then convergence analysis.

A. Independent Double Q-learning Method

The independent double Q-learning component uses two action-value functions and UCB exploration to reduce over-estimation while balancing exploration and exploitation. Its limitations motivate the cooperative extensions of Co-DQL.

  • A. Independent Double Q-learning Method: Double Q-learning uses two estimators to reduce the positive bias and over-estimation caused by maximizing estimated action values.The independent variant updates one action-value function using the other estimator’s value for the next state.
  • A. Independent Double Q-learning Method: Soft target updates change target-network weights gradually, which can enhance learning stability compared with directly copying weights.The update uses a small τ rather than an immediate parameter copy.
  • A. Independent Double Q-learning Method: UCB selects actions using learned Q-values and action-visit counts, favoring rarely explored actions early and becoming greedier with experience.This provides an exploration–exploitation mechanism based on both value estimates and visitation history.
  • A. Independent Double Q-learning Method: Independent learning treats other agents as part of the environment, ignoring their changing actions and lacking a guaranteed convergence property.Co-DQL addresses this instability with mean field approximation, reward allocation, and local state sharing.

B. Cooperative Double Q-learning Method

Co-DQL combines mean-field approximation, reward reallocation, and local state sharing with independent double Q-learning to support scalable cooperative MARL. Its design reduces action-value input dimensionality while stabilizing training through shared information and reallocated rewards.

  • Mean-field approximation: Mean-field approximation replaces interactions with neighboring agents by their average action, reducing the joint action dimension from C^N_k to constant C^2.Neighbor actions are represented as one-hot vectors and averaged under a uniform distribution over the neighborhood.
  • Reward allocation: The reward allocation mechanism combines each agent’s immediate reward with neighboring agents’ rewards, balancing selfishness and cooperation through α.α=0 uses only the local reward, α=1 uses the global reward, and the method specifies 0<α<1.
  • Local state sharing: Local state sharing adds the average neighboring state to each agent’s input, yielding a joint state dimension of |s|^2 independent of the number of agents.This implicitly communicates neighboring state information without constructing the full global state.
  • Scalability: Co-DQL reduces the centralized action-value input dimension from C^{N_k}·|s|^{N_k} to C^2·|s|^2 with limited communication and computation.The reduction is intended to avoid the curse of dimensionality in large-scale problems.
  • Training procedure: Each agent selects actions with UCB, executes synchronously, computes mean actions and reallocated rewards, stores transitions, and updates online and target networks from replay samples.The training loop includes replay-buffer sampling, loss minimization, gradient updates, and soft target-network updates.
  • Training properties: Mean-field collaboration, reward allocation, and local state sharing are reported to improve training stability and robustness relative to independent-agent learning.The convergence analysis is introduced as theoretical support for Co-DQL’s effectiveness.

C. Convergence Analysis

The convergence analysis establishes that Co-DQL’s two Q-functions converge to the Nash Q-value under stated finite-state, bounded-reward, visitation, policy, and technical assumptions. The proof follows a stochastic-approximation framework and treats the two updates symmetrically.

  • Assumptions: The theorem assumes every state-action pair is visited infinitely often and rewards are bounded by a constant.These requirements provide the visitation and bounded-variance conditions used in the proof.
  • Assumptions: The policy must be greedy in the limit with infinite exploration, so Boltzmann exploration becomes greedy as temperature decays to zero.This is stated as the second assumption for the convergence result.
  • Technical conditions: The analysis also requires finite state space and conditional drift and variance bounds involving γ<1 and a vanishing error term.These are the technical conditions used to apply the convergence lemma.
  • Convergence theorem: Under Assumptions 1–3 and Lemma 1’s conditions, both Q^a and Q^b converge to the Nash Q-value with probability one.The theorem applies to a finite-state stochastic game and requires the specified assumptions and lemma conditions.
  • Proof strategy: The proof targets convergence of both double estimators by showing that the update conditions required by the stochastic-approximation lemma hold.Because the two updates are symmetrical, establishing convergence for one supports convergence of the other.

IV. APPLICATION OF CO-DQL TO TSC

The paper formulates traffic signal control as a setting in which MARL can represent the key elements of the problem. It also introduces traffic-signal simulators for training and evaluation of the applied model.

  • TSC application: The TSC application uses MDP notation to represent the problem’s key elements so that MARL can be applied.The section also introduces simulators to facilitate model training and evaluation.

A. Description of TSC Based on MDP Notations

The traffic network is modeled as a decentralized multi-agent system while retaining a Markov global traffic state. The section considers multiple MDP formulations because TSC settings differ in their action, state, and reward definitions.

  • Markov formulation: The global traffic-system state is assumed to be Markov, so the next state depends only on the current state and joint action.This supports modeling the decentralized traffic network with a MARL framework.
  • MDP settings: The paper focuses on two MDP settings because TSC formulations can differ in action space, state space, and reward function.The authors note that the method may extend to other settings.

1) A simplified MDP setting for TSC problem:

The simplified TSC formulation models each intersection as an agent with binary signal actions, local queue-based observations, and rewards based on waiting vehicles. Its objective is to minimize total network waiting time while sharing local information and neighboring action summaries for cooperative control.

  • 1) A simplified MDP setting for TSC problem:: Each signal agent chooses between two opposing traffic-light configurations, giving the joint action space {0, 1}^N.The two configurations alternate green phases for north–south and east–west incoming traffic.
  • 1) A simplified MDP setting for TSC problem:: Local state records queue density on the one-way lanes entering each intersection.The observation contains four directional queue values, bounded by each lane’s maximum vehicle capacity.
  • 1) A simplified MDP setting for TSC problem:: Rewards are based on vehicles waiting on lanes leading toward each intersection and are accumulated over Δt time steps between signal decisions.The action interval reduces how frequently signals change while aggregating rewards across the interval.
  • 1) A simplified MDP setting for TSC problem:: The control objective is to minimize total vehicle waiting time across the traffic network.The objective sums waiting-related rewards over the full sequence of state transitions.
  • 1) A simplified MDP setting for TSC problem:: Each agent receives shared local-state information, neighboring mean-action information, and a reallocated reward to support cooperative learning.These inputs are used by Co-DQL to coordinate decentralized agents.

2) A more realistic MDP setting for TSC problem:

The realistic formulation defines signal phases and lane-level traffic states using congestion and delay measurements. Its reward emphasizes both traffic congestion and travel delay, with a regularization term for the first vehicle’s cumulative delay.

  • 2) A more realistic MDP setting for TSC problem:: The realistic simulator uses five feasible signal phases, including east–west straight, east–west left-turn, and directional straight-left combinations.The formulation follows the phase-based action definition from prior TSC work.
  • 2) A more realistic MDP setting for TSC problem:: Each local state combines cumulative delay of the first vehicle with the total number of approaching vehicles on every incoming lane.The state is obtained with SUMO lane-area detectors or near-intersection induction-loop detectors.
  • 2) A more realistic MDP setting for TSC problem:: The simplified grid simulator represents intersections as rectangles connected by pairs of one-way lanes, with lane colors indicating congestion and rectangle numbers indicating immediate reward.The simulator includes global random, double-ring, and four-ring traffic flows.
  • 2) A more realistic MDP setting for TSC problem:: The reward emphasizes traffic congestion and travel delay and is measured only at t+Δt.A regularization rate β scales the cumulative-delay term with the other reward components.

B. Description of the Simulation Platform

The experiments use grid and realistic SUMO-based traffic simulators spanning multiple flow patterns and a 49-intersection Xi’an road network. Co-DQL is compared with decentralized and independent RL baselines under intensive stochastic traffic conditions.

  • B. Description of the Simulation Platform: The grid platform evaluates global random, double-ring, and four-ring traffic-flow scenarios.These scenarios differ in route-generation rules and traffic intensity, producing different congestion patterns across intersections.
  • B. Description of the Simulation Platform: The realistic SUMO network is based on Xi’an roads and contains 49 signalized intersections.Figures show the overall asymmetric network and a local view of adjacent intersections.
  • B. Description of the Simulation Platform: The experiments use intensive, stochastic, time-variant flows to represent peak-hour traffic, with 60-minute episodes and four traffic-flow groups.The flow groups are generated as multiples of unit rates including 1100veh/hr, 660veh/hr, 920veh/hr, and 552veh/hr.
  • B. Description of the Simulation Platform: Co-DQL uses a shared neural Q-network with agent embeddings, mean-action input, and shared joint-state input.Its listed settings include Adam with learning rate 0.0001, γ=0.95, replay buffer size 5 × 10^5, and τ=0.01.
  • B. Description of the Simulation Platform: The comparison includes MA2C, IQL, IDQL, and DDPG, with IQL and IDQL using architectures closely matched to Co-DQL.IQL omits mean-action and shared-state inputs, while IDQL uses double estimators for target calculation.
  • B. Description of the Simulation Platform: Algorithm performance depends partly on the selected hyper-parameter settings.This qualification applies across the evaluated methods.

B. Experiments in The Simplified TSC Simulator

In the simplified simulator, Co-DQL achieves the strongest training and evaluation performance across the tested traffic scenarios. Double estimators improve performance over standard independent Q-learning, while Co-DQL’s cooperative modeling yields lower delay and more robust results than the compared decentralized methods.

  • 1) global random traffic flow:: Random-strategy traffic reaches a stable congestion range after vehicle inflow and outflow balance, enabling stabilized simulator states to seed training and evaluation.Ten discrete states are recorded after stabilization for the global random-flow scenario.
  • 1) global random traffic flow:: Co-DQL greatly outperforms all other methods in global random-flow training, while IQL has the lowest training performance.IDQL is only slightly better than IQL, suggesting that double estimators improve performance to some extent.
  • 1) global random traffic flow:: Co-DQL greatly reduces average delay time compared with the other methods in the global random traffic-flow evaluation.The evaluation uses the best training model for each algorithm over 100 episodes.
  • 2) double-ring traffic flow :: Co-DQL achieves shorter average delay time and smaller standard deviations than the other methods in the double-ring scenario.Its training performance also remains higher than the alternatives in this scenario.

3) four-ring traffic flow:

In the four-ring and related traffic scenarios, Co-DQL combines double estimators, mean-field opponent modeling, and local information sharing to improve decentralized signal control. Across the reported experiments, it achieves stronger evaluation performance and shorter delay than the compared MARL methods, while some findings remain simulator- and setting-dependent.

  • Four-ring traffic flow: Co-DQL achieves the shortest average delay time in the four-ring scenario, while its learning process is relatively stable with smaller evaluation variance than several baselines.The authors attribute this result to mean field approximation for opponent modeling and local information sharing.
  • Scenario comparison: Traffic complexity affects performance: the four-ring scenario requires coordination among four agent groups, whereas the double-ring scenario requires coordination between two groups.The authors suggest that the more complex coordination task may contribute to the four-ring scenario’s differences from the double-ring scenario.
  • Overall results: Co-DQL achieves better evaluation performance than state-of-the-art decentralized MARL algorithms across multiple traffic scenarios, including the 7×7 grid experiment.The reported comparisons include IQL, IDQL, DDPG, and MA2C.
  • Estimator comparison: Double-estimator methods consistently outperform methods without double estimators, and IDQL generally has a slight advantage over IQL across most measurements.This comparison is reported for the evaluated traffic scenarios.
  • More realistic simulator: In the more realistic simulator, Co-DQL retains higher training reward and better evaluation performance than the decentralized MARL baselines, whereas MA2C does not achieve the desired performance.The realistic evaluation reports multiple traffic measures and uses ten random seeds.
Loading 1908.03761v2…