PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent Reinforcement Learning for Networked System Control

Tianshu Chu, Sandeep Chinchali, Sachin Katti

arXiv:2004.01339v2cs.LGstat.ML

TL;DR

Networked MARL must learn decentralized control under local observations, neighborhood communication, partial observability, and non-stationarity. The paper formulates this setting as a spatiotemporal MDP, introduces spatial discounting, and proposes NeurComm; experiments report improved non-communicative learning with suitable discounting and stronger communication performance from NeurComm.

  • Problem

    Networked MARL faces partial observability and non-stationarity when agents must control systems using local observations and neighborhood communication.

  • Method

    The paper formulates NMARL as a decentralized spatiotemporal MDP, introduces a spatial discount factor, and proposes the differentiable NeurComm communication protocol.

  • Results

    Suitable spatial discount factors improve non-communicative MARL learning, while NeurComm outperforms existing communication protocols in learning efficiency and control performance across the evaluated scenarios.

  • Takeaways & Limitations

    Effective networked control combines practical neighborhood communication constraints with spatial discounting and adaptive communication rather than directly reusing existing MARL algorithms.

  • Takeaways & Limitations

    The approach assumes limited-neighborhood communication, offline training with global rollout information, and local transitions that depend on neighboring agents.

Abstract

from arXiv · show

This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

1 INTRODUCTION

The paper studies networked MARL for decentralized control under local observations and neighborhood communication. It addresses scalability, partial observability, non-stationarity, and practical constraints through a spatiotemporal formulation, spatial discounting, and NeurComm.

  • MARL improves scalability for real-world control but introduces partial observability and non-stationarity during decentralized training and execution.
  • NMARL lets agents learn decentralized control policies from local observations and messages exchanged with connected neighbors.
  • Networked control requires neighborhood-limited communication because global observations can increase delay and failure risk during distributed execution.
  • The paper formulates NMARL as a decentralized spatiotemporal MDP, introduces a spatial discount factor, and proposes NeurComm for adaptive information sharing.

2 RELATED WORK

Prior MARL communication methods range from centralized critics and heuristic information sharing to learnable protocols and selective attention. NeurComm differs by encoding and concatenating signals while communicating policy fingerprints to reduce information loss and non-stationarity.

  • Non-communicative methods stabilize training with centralized value estimation, while heuristic protocols share policy fingerprints or mean-field summaries.
  • NeurComm encodes and concatenates communication signals instead of aggregating them, while including policy fingerprints to reduce information loss and non-stationarity.
  • Learnable protocols such as DIAL and CommNet generate messages during learning, but aggregation can lose information.
  • Attention-based protocols learn which agents should receive messages, although restricted neighborhood communication makes target prioritization less central in NMARL.

3 SPATIOTEMPORAL RL

The paper models networked control as a decentralized spatiotemporal MDP in which local dynamics depend on neighborhood states and policies. A spatial discount factor scales rewards by distance to reduce learning difficulty while preserving decentralized execution.

  • 3.1 NETWORKED MARL: Networked MARL is challenging because each agent has partial local information and other agents’ evolving policies make its transition process non-stationary.
  • 3.1 NETWORKED MARL: Each agent observes its local state together with messages from neighbors, while the global reward is shared across agents.
  • 3.2 SPATIOTEMPORAL RL: The spatiotemporal MDP assumes local transitions depend on neighboring agents, extending the Markov property across both time and network space.
  • 3.2 SPATIOTEMPORAL RL: The spatial discount factor α scales down rewards from farther agents, with α approaching 0 favoring local greedy control and α approaching 1 favoring global coordination.
  • 3.2 SPATIOTEMPORAL RL: Offline training can use global rollout minibatch information, whereas online execution restricts policies to local observations and available neighborhood messages.

4 SPATIOTEMPORAL RL WITH NEURAL COMMUNICATION

NeurComm uses differentiable, neighborhood-based messages containing state, policy, and belief information to improve observability and reduce non-stationarity while retaining decentralized execution. Its communication and propagation design supports multi-pass belief updates and connects to a single meta-DNN, while avoiding the aggregation losses of earlier protocols.

  • NeurComm protocol: NeurComm messages explicitly include state, prior policy, and prior belief to improve observability and reduce non-stationarity.Communication occurs before the decision, so only prior belief and prior policy are available from neighbors.
  • NeurComm protocol: Each agent learns messages through differentiable encoding and extraction functions, with communication attention selectable at the sender or receiver.The protocol supports multi-pass communication for updating agent beliefs.
  • Spatiotemporal propagation: NeurComm can be represented as a single meta-DNN whose decentralized gradient updates expand spatially through cascaded neighborhood communication.The intermediate outputs after communication layers support multi-step spatiotemporal propagation.
  • Decentralized execution: Execution remains fully decentralized because each policy depends only on the locally received communicated input.Training updates can involve neighboring parameters, but execution uses the agent’s own extracted communication input.
  • Relation to prior protocols: Unlike CommNet and DIAL, NeurComm avoids their input-aggregation losses and includes policy fingerprints in messages.CommNet averages received messages, while DIAL and CommNet sum encoded inputs without including policy fingerprints.

5 NUMERICAL EXPERIMENTS

Experiments evaluate NMARL methods in adaptive traffic signal control and cooperative adaptive cruise control under realistic networked settings. Tuned spatial discount factors improve training, while NeurComm achieves strong training and execution performance, especially in challenging ATSC scenarios.

  • Experimental environments: The experiments cover a 5×5 synthetic traffic grid, a 28-intersection Monaco network, and two physical-dynamics CACC scenarios.ATSC uses SUMO-based peak-hour traffic, while CACC evaluates Catch-up and Slow-down tasks with eight vehicles.
  • Training results: NeurComm clearly outperforms other policies in sample efficiency and learning stability on challenging ATSC scenarios, while remaining competitive in CACC.In CACC, episode-return variability is high because collisions incur a large penalty.
  • Training results: Lower spatial discount factors are preferred by almost all policies in challenging ATSC Monaco, demonstrating that α can enhance performance on difficult tasks.The spatial discount factor also acts as an indicator of problem difficulty and algorithm coordination level.
  • Execution results: NeurComm achieves the best execution performance for ATSC, whereas α-enhanced FPrint performs best in CACC scenarios.NeurComm still outperforms other communicative algorithms in CACC, but delayed information sharing may be less helpful for easy, safety-critical tasks.
  • Execution results: Communicative policies reduce intersection delay in homogeneous ATSC Grid but can increase it rapidly in heterogeneous ATSC Monaco, indicating a risk of overfitting to queue reduction.The comparison uses network-averaged queue length and intersection delay over time.
  • Execution results: In CACC, MARL controllers reach the target velocity and headway for the platoon’s first vehicle but have difficulty eliminating perturbations through the platoon.The authors suggest the heuristic low-level controller and delayed information sharing may contribute.

6 CONCLUSIONS

The paper formulates decentralized NSC as a spatiotemporal MDP, introduces spatial discounting to improve non-communicative MARL, and proposes NeurComm for adaptive communication.

  • The paper combines a spatiotemporal MDP, spatial discounting, and NeurComm to develop scalable and robust MARL controllers for networked system control.The conclusion emphasizes practical engineering assumptions and combining suitable learning and communication methods.

A.1 PROOF OF PROPOSITION 3.1

The proof adapts actor–critic learning to neighborhood-restricted observations and communications by using spatially discounted rewards and local actor–critic quantities.

  • The proposition replaces global policies, values, and rewards with neighborhood-local counterparts and a spatially discounted reward for decentralized learning.The resulting local actor and critic approximate their globally observed counterparts under neighborhood restrictions.
  • Delayed communication propagates information from increasingly distant agents through neighborhood messages, yielding delayed global observations.The propagation expands across graph distance while retaining temporal delays.
  • The spatiotemporal formulation mitigates non-stationarity arising from limited information about future actions.The proof explicitly identifies partial observability and non-stationarity in the local actor and critic.

A.2 PROOF OF PROPOSITION 4.1

The proof shows that NeurComm propagates spatial information through hidden states and propagates learning signals across agents through differentiable messages.

  • NeurComm uses delayed global information to estimate each agent’s hidden state through recurrent message propagation.The hidden-state construction incorporates local states, neighboring policies, and neighboring messages across time and graph distance.
  • NeurComm messages receive gradients from other agents’ actor and critic losses, enabling communication parameters to optimize multi-agent performance.Except for the initial distance-dependent samples, messages receive gradients from the loss signals of other agents.
  • Training alternates message updates, hidden-state and action updates, value estimation, execution, and actor–critic–communication gradient updates.The synchronous algorithm performs four iterations at each step.

C EXPERIMENT DETAILS

The experiments instantiate several recurrent MARL communication policies by varying how local states, neighbor policies, and neighbor hidden states enter the hidden-state update.

  • NeurComm updates its hidden state from local states, neighboring policies, and neighboring hidden states, unlike the simpler communication baselines.IA2C uses only local state input, FPrint adds neighbor policies, and CommNet aggregates neighbor hidden states by a mean.

C.2 EXPERIMENTS IN ATSC ENVIRONMENT

In ATSC environments, NeurComm outperforms all baselines in minimizing queue length and intersection delay, with communicative controllers showing better intersection-delay performance.

  • ATSC uses heterogeneous signal-phase action spaces, including intersections with two through six phases.The Grid scenario has five phases per intersection, while Monaco contains intersections with multiple phase counts.
  • NeurComm outperforms all baselines on minimizing queue length and intersection delay in ATSC.Table 3 covers both the synthetic traffic grid and Monaco traffic network.
  • Communicative MARL controllers reduce intersection delay more effectively than non-communicative controllers, with NeurComm and CommNet performing best overall.The comparison comes from one execution episode visualized across simulation steps.
Loading 2004.01339v2…