PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control

Tianshu Chu, Jie Wang, Lara Codecà, Zhaojian Li

arXiv:1903.04527v1cs.LGstat.ML

TL;DR

Large-scale ATSC cannot use centralized RL because joint actions become infeasible, while decentralized MARL introduces partial observability. The paper proposes MA2C, an A2C-based decentralized algorithm that adds neighboring information and spatial discounting, and evaluates it against IA2C and IQL. Across synthetic and Monaco traffic networks, MA2C demonstrates robustness, optimality, scalability, and stronger evaluation performance than the compared controllers.

  • Problem

    Centralized RL is infeasible for large-scale ATSC, while decentralized MARL faces partial observability and most existing work focuses on Q-learning.

  • Method

    MA2C extends independent A2C with neighboring observations and policy fingerprints, spatially discounting neighboring states and rewards to stabilize decentralized learning.

  • Results

    MA2C outperforms IA2C, Greedy, and IQL-LR on the reported evaluation objective, with average return values of -414, -845, -972, and -1409, respectively.

  • Takeaways & Limitations

    Experiments on a synthetic grid and Monaco network support MA2C’s robustness, optimality, and scalability among the evaluated decentralized MARL algorithms.

Abstract

from arXiv · show

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. Multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent: advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. Results demonstrate its optimality, robustness, and sample efficiency over other state-of-the-art decentralized MARL algorithms.

I. INTRODUCTION

ATSC uses RL to learn control from traffic experience, while centralized deep RL becomes infeasible at large scale because global processing and joint actions grow prohibitively. MARL distributes control among local agents, but limited communication creates partial observability and nonstationary learning; MA2C addresses these challenges with two stabilizing mechanisms.

  • Motivation: RL learns ATSC control directly from real-world traffic measurements without relying on heuristic assumptions and equations.Deep neural networks improve RL’s learning capacity for complex tasks, whereas traditional table and linear models have limited scalability or optimality.
  • Motivation: Centralized RL is infeasible for large-scale ATSC because global state processing causes latency and failure risks, while the joint action space grows exponentially with intersections.Centralized processing can also lose traffic-network topology.
  • Multi-agent formulation: MARL distributes global traffic control across local agents using local observations and limited communication, but independent Q-learning models other agents as changing environment dynamics.This makes IQL scalable but creates convergence issues under partial observability.
  • Contribution: MA2C extends independent A2C for ATSC and stabilizes learning by adding neighboring observations and fingerprints while spatially discounting neighboring states and rewards.The first method improves information and fitting power; the second reduces local fitting difficulty.
  • Contribution: MA2C is evaluated on a synthetic traffic grid and a real-world Monaco network, where experiments report robustness, optimality, and scalability over decentralized MARL baselines.The evaluation uses simulated traffic dynamics designed to maintain a certain difficulty level of the MDP.

1) Q-learning:

The paper contrasts Q-learning and policy-gradient approaches for deep RL. A2C adds value estimation to policy gradients to reduce return bias and variance while retaining robustness to nonstationary transitions within trajectories.

  • Q-learning: Q-learning fits a parametric Q-function from experience, using models such as tables, linear regression, or deep neural networks.The fitted function estimates state-action values used to choose actions.
  • Q-learning: Q-learning estimates targets with a frozen recent model and updates parameters using temporal-difference error, while exploration and replay reduce regression variance.Experience replay samples less-correlated experiences in deep Q-learning.
  • Policy gradient: Policy gradient directly fits a parameterized policy, increasing the likelihood of actions with high sampled total return without requiring a separate behavior policy.Its sampled returns support exploration and exploitation but have high variance.
  • A2C: A2C introduces a value regressor to reduce sampled-return bias and variance through bootstrapping and an advantage estimate.The advantage is the difference between the action value and state value.

B. Multi-agent Reinforcement Learning

MARL makes cooperative ATSC scalable by distributing actions across intersections, but local agents face partial observability and nonstationarity. The paper extends this setting from predominantly Q-learning methods to A2C with neighborhood policy information and spatial discounting.

  • Formulation: In cooperative ATSC, centralized RL is infeasible because the joint action space is the Cartesian product of all agents’ action spaces.MARL distributes global action selection among local intersection agents sharing a global reward.
  • Q-learning MARL: Coordinated Q-learning uses message passing or control synchronization to trade off optimality and scalability, but requires extra execution computation and decomposition heuristics.These methods approximate each local Q-function using neighborhood messages.
  • Q-learning MARL: IQL is scalable because each local Q-function depends only on its local action, but changing policies of other agents create partial observability and nonstationary dynamics.Prior methods add policy parameters or low-dimensional fingerprints and may use importance sampling for replayed experience.
  • Research gap: Prior ATSC MARL studies mostly used Q-learning, with limitations including oversimplified environments, infeasible states, and sparse realistic benchmarks for fair comparison.The paper identifies a lack of comprehensive realistic studies of deep RL in practical ATSC.
  • MA2C: MA2C extends independent A2C by informing local agents about recent neighbor policies and applying spatial discounting to signals from distant agents.The framework is designed for more stable convergence under limited communication and partial observation.

A. Independent A2C

IA2C gives each agent its own policy and value function using local-region observations, but local critics remain partially observable and policy updates remain nonstationary. These problems can make gradients inconsistent and slow convergence.

  • Independent A2C: In IA2C, each agent learns an individual policy and value function over a local region defined by neighboring agents in the communication graph.The local region includes the agent itself and its immediate neighborhood.
  • Independent A2C: IA2C begins from a setting with shared global reward and state, then extends centralized A2C updating by estimating local returns.The formulation later restricts communication to local regions while retaining the global reward for offline training.
  • Independent A2C: When other agents’ parameters are fixed, an agent can converge toward its best response, but actively changing neighboring policies make policy gradients inconsistent across minibatches.The inconsistency arises because the advantage depends on a changing neighboring policy.
  • Independent A2C: Local value estimation is partially observable because each agent sees only a subset of the global state while expected returns depend on the full state.The same limitation affects the local policy’s ability to capture the impact of the return.
  • Independent A2C: Smoother local policy updates can mitigate inconsistency, but they slow convergence, leaving IA2C with a trade-off between stability and learning speed.The paper motivates additional methods to improve observability and reduce fitting difficulty.

B. Multi-agent A2C

The method stabilizes decentralized IA2C by improving local observability with neighboring policies and by spatially discounting signals to make local learning more relevant. It then trains local actor and critic networks through synchronous minibatch updates.

  • Neighborhood observability: Neighboring agents’ latest sampled policies are added to each local agent’s DNN inputs alongside its current state.The approach uses recent policies rather than long-term neighborhood behavior because traffic changes slowly over short windows and dynamics are Markovian given current state and policy.
  • Spatial discounting: Spatial discounting weakens state and reward signals from distant agents, balancing greedy control at α = 0 against cooperative control at α = 1.The distance limit Di determines the maximum spatial range considered for each agent.
  • Learning stabilization: Additional neighborhood policy fingerprints make value updating more stable by helping the critic fit the changing policy context.The fingerprints are explicitly supplied to the value function during fitting.
  • Spatial discounting: The spatially discounted return is more correlated with local observations, making the local advantage more relevant to the marginal impact of each policy.This adjustment addresses the difficulty of fitting a global return with a local value regressor.
  • Training procedure: Synchronous MA2C collects local experience until minibatches are available, applies actor and critic gradients, and stops at a maximum step or specified condition.The algorithm uses actor and critic learning rates ηθ and ηw and can use constant or adaptive learning rates.

V. MA2C FOR TRAFFIC SIGNAL CONTROL

The implementation section specifies the components used to realize MA2C for ATSC in the SUMO microscopic traffic simulator. It covers control definitions, network design, training, normalization, and evaluation.

  • Implementation scope: MA2C is implemented for ATSC under the microscopic traffic simulator SUMO.The implementation covers action, state, reward, A2C network structures and normalizations, training tips, and evaluation metrics.
  • Implementation scope: The implementation defines the traffic-control action, state, and reward used by the multi-agent system.These definitions are presented together with the A2C network and training configuration.
  • Implementation scope: Evaluation metrics are included as part of the MA2C implementation description.The passage frames evaluation alongside simulator and training details.

A. MDP Settings

The MDP setting chooses an interaction period that balances responsiveness, computational and communication constraints, and safety concerns. The period determines how often RL agents issue traffic-control decisions.

  • Interaction period: The interaction period Δt specifies how long the traffic environment is simulated after each MDP step.RL agents interact with the traffic environment once per Δt seconds.
  • Interaction period: Longer Δt reduces adaptability, whereas shorter Δt can prevent timely control because of computation and communication latency.The choice must account for the delivery time of RL decisions.
  • Interaction period: Switching RL control too frequently raises safety concerns, adding a practical constraint on the interaction period.The period therefore balances control responsiveness with operational feasibility.

1) Action definition:

MA2C defines local traffic-light actions as feasible intersection phases and uses detector-based traffic measurements for local state information. Its reward combines queue length and waiting time after each decision.

  • Action definition: Each local action selects one feasible traffic-light phase, represented by a red-green combination at an intersection, for duration Δt.The feasible phase set Ui is predefined for each intersection.
  • State definition: Local state measurements include cumulative delay of the first vehicle and approaching vehicles within 50 m on each incoming lane.SUMO laneAreaDetector supplies these measurements for real-time ATSC.
  • Reward definition: The post-decision reward combines measured queue length and waiting time using a tradeoff coefficient a.Both quantities are measured at time t + Δt, emphasizing congestion and trip delay.

B. DNN Settings

MA2C uses recurrent, separately trained actor–critic networks with normalized inputs and evaluates decentralized controllers in synthetic and real-world SUMO traffic environments. The synthetic-grid experiments compare training behavior and traffic outcomes across MARL algorithms.

  • DNN architecture: Separate fully connected streams process wave, wait, and neighbor-policy states before combining them in an LSTM representation.The actor outputs a softmax policy, while the critic uses a linear output layer; actor and critic DNNs are trained separately.
  • Training stabilization: State normalization clips normalized inputs to [0, 2], while reward normalization clips rewards to [-2, 2] and caps gradients at 40.These settings are used to stabilize minibatch updating and prevent gradient explosion.
  • Experimental environments: Evaluation covers a 5 × 5 synthetic grid and a real-world 30-intersection Monaco network under time-variant traffic flows.Both environments are simulated with SUMO for comparisons across controllers.
  • Controller comparisons: All controllers share the same action space, state space, and interaction frequency, while IA2C differs from MA2C by omitting the proposed stabilizing methods.The benchmarks include IA2C, IQL-LR, IQL-DNN, and a decentralized greedy policy.
  • Synthetic-grid setup: The synthetic grid contains 25 intersections and uses five possible phases per intersection, making the centralized joint action space 5^25.Four time-variant traffic-flow groups are simulated to make the MDP challenging.
  • Synthetic-grid results: MA2C training increases steadily and stabilizes with a narrow variance band, whereas IQL-DNN fails to learn and IQL-LR reaches performance comparable to MA2C.The training curves report average reward per episode with standard-deviation shading.

3) Evaluation results:

In the Monaco network, MA2C produced the strongest evaluation performance and maintained lower congestion and delay during simulated peak-hour traffic. The experiment used a 30-intersection network with intensive, stochastic, time-varying flows and modified rewards and states.

  • Controller comparison: -414 average evaluation reward made MA2C the best controller, ahead of IA2C, Greedy, and IQL-LR.The corresponding averages were -845, -972, and -1409 for the other listed controllers.
  • Delay: MA2C maintained low intersection delay even at the traffic peak, whereas IQL-LR and IA2C showed increasing delays while failing to recover congestion.IA2C performed better than Greedy when the grid was less saturated, but MA2C maintained the lowest delay profile.
  • Experiment setting: The Monaco experiment covered 30 signalized intersections with two- through six-phase signal configurations.The network included 11 two-phase, 4 three-phase, 10 four-phase, 1 five-phase, and 4 six-phase intersections.
  • Experiment setting: Peak-hour traffic was simulated using intensive, stochastic, time-varying flows generated from four groups of 325 veh/hr unit flows.Origin-destination pairs were randomly sampled for the traffic-flow groups.
  • Experimental assumption: The experiment removed wait terms from both reward and state to make the value function easier to fit, so MARL did not explicitly optimize delay.Wave and reward normalization factors were also specified for the experiment.

2) Training results:

Across training and evaluation analyses, MA2C converged to stable, sustainable control and coordinated traffic more effectively than the compared decentralized alternatives. Its design combines neighborhood policy information with a spatial discount factor, while real-world deployment remains future work.

  • Training curves: Both IA2C and MA2C converged, while IQL-DNN learned nothing and IQL-LR failed to converge.IQL-LR nevertheless showed good performance during the middle of training.
  • Network dynamics: MA2C achieved lower and more sustainable intersection delays by distributing traffic more homogeneously through shared neighborhood fingerprints.IA2C and Greedy reduced queue lengths after peak values but failed to maintain sustainable intersection delays.
  • Network dynamics: MA2C kept most output-flow and vehicle-accumulation points near the macroscopic fundamental diagram’s sweet spot, maximizing network-capacity utilization.The scatter aggregates each point over 5 minutes.
  • Method: The proposed MA2C uses neighbor fingerprints to improve observability and a spatial discount factor to reduce learning difficulty.These mechanisms target decentralized learning under limited communication.
  • Deployment boundary: Real-world deployment still requires more realistic traffic simulation, robustness to noisy and delayed measurements, deployable training pipelines, and lower inference, memory, and communication latency.These are identified as future-work requirements for deploying the proposed MARL algorithm.
Loading 1903.04527v1…