PaperlayerView exact revision on arXiv

Source-linked AI summary

IntNet: A Communication-Driven Multi-Agent Reinforcement Learning Framework for Cooperative Autonomous Driving

Leandro Parada, Kevin Yu, Panagiotis Angeloudis

arXiv:2610.34466v1cs.ROcs.LG

TL;DR

Safety and coordination in autonomous driving remain challenging under partial observability and dynamic urban conditions. IntNet shares predicted vehicle intentions through a communicative MARL framework; across urban scenarios, it reports higher rewards and success, with faster highway learning than baselines.

  • Problem

    State-sharing strategies are insufficient for safety-critical driving, where partial observability and dynamic urban conditions complicate coordination.

    Source: §I · p. 1

  • Method

    IntNet adds predicted vehicle-intent sharing to communicative MARL, using an observation predictor, Graph Attention Network message processing, and a communication scheduler.

    Source: §I · p. 2, 3) Communication Scheduling and Graph… · p. 4

  • Results

    Across three scenarios, IntNet consistently achieves higher rewards and success rates; on highways it reaches high success with 100k episodes versus baselines' around 200k.

    Source: A. Baseline comparison · pp. 5–6

  • Takeaways & Limitations

    Intention-sharing is particularly beneficial in lane merging, while its impact is smaller at intersections; the study also reports safer, more efficient policies with lower communication graph density.

    Source: A. Baseline comparison · p. 5, §V · p. 7

Abstract

from arXiv · show

Achieving safety in autonomous driving through Multi-Agent Reinforcement Learning (MARL) is a critical yet challenging task due to non-stationarity, partial observability, and the need for effective coordination among agents. Although earlier cooperative MARL methods have aimed to improve coordination by sharing encoded state observations, we find these strategies are insufficient in safety-critical scenarios. To address this gap, we present IntNet, a novel MARL framework that enhances safety by incorporating the transmission of vehicle intent and adaptive communication scheduling into a unified end-to-end learning paradigm, jointly optimising all components for safe coordination. Central to our approach is an observation prediction module that enables agents to forecast subsequent policy outputs, which they can then share across a vehicle-to-vehicle network. Our model-free architecture employs a Graph Attention Network to process incoming messages and a scheduler component to dynamically optimise the communication graph for bandwidth efficiency. We compare IntNet against state-of-the-art communicative MARL methods in complex urban environments with both autonomous and human-operated vehicles, achieving improved coordination, lower collision rates, and reducing communication efforts by up to 60\%. Through extensive experiments, we assess the framework's learning efficiency, scalability, and the balance between information sharing and bandwidth usage for collision-free trajectories.

I. INTRODUCTION

Safe coordination in urban autonomous driving is complicated by partial observability and unpredictable human behavior. IntNet addresses limits of state-only communication by sharing predicted vehicle intentions alongside observations, with adaptive communication scheduling.

I. INTRODUCTION

  • Dynamic urban settings, partial observability, and unpredictable human drivers make safety and coordination difficult for autonomous-driving MARL.Model-free MARL offers trial-and-error learning for complex configurations but does not inherently provide safety guarantees.Source: §I · p. 1
  • Sharing state features alone is insufficient for safety-critical driving, where future intentions add context that can support safer, more efficient decisions.Earlier intention-sharing studies were limited to grid-like settings without high-dimensional observations or autonomous traffic dynamics.Source: §I · p. 1
  • IntNet combines encoded observations and recent policy context with predicted intentions over horizon H, using a scheduler and graph attention to support coordinated communication.The observation predictor is trained from environmental interaction without pretraining.Source: §I · p. 2
  • The paper evaluates communicative MARL with high-dimensional observations and realistic vehicle kinematics, moving beyond prior grid-based environments.Source: §I · p. 2

II. RELATED WORK

Prior work applies MARL to autonomous-driving tasks and develops selective, attention-based, and graph-based communication. IntNet builds on intention-sharing research with a framework aimed at high-dimensional driving observations and bandwidth constraints.

A. MARL for Autonomous Driving

  • MARL has been used for collaborative navigation, traffic control, merging, and lane-changing, but safety and coordination in dynamic urban environments remain unresolved challenges.Source: A. MARL for Autonomous Driving · p. 2

B. Multi-Agent Communicative Reinforcement Learning

  • Communicative MARL spans gating, attention, and communication-graph methods; IntNet extends intention-sharing toward autonomous driving with GATs and scheduled messaging for high-dimensional observations and bandwidth constraints.Source: B. Multi-Agent Communicative… · p. 2

III. METHODOLOGY

The methodology formulates communication-based MARL as a Dec-POMDP and equips agents to exchange current information and estimated future intentions. Agents encode local observations and share policy fingerprints that summarize recent decision-making context.

A. Communication-based Multi-Agent Reinforcement Learning

  • The communication-based MARL problem is formulated as a Dec-POMDP with agents, states, actions, observations, transitions, rewards, and a discount factor.Messages and their aggregation protocol enter the agents' policies, which aim to maximize expected cumulative reward.Source: A. Communication-based Multi-Agent… · p. 3

B. Intention-sharing MARL framework

1) Observation Encoding and Policy Fingerprint:

  • Each agent encodes its local observation as a message and shares a policy fingerprint, a probability distribution over actions summarizing recent decision-making context.The context includes the agent's prior local observation and messages received from other agents.Source: 1) Observation Encoding and Policy… · p. 3

2) Intention Prediction and Sharing:

IntNet predicts future observations recursively and shares agents’ projected policy fingerprints over a future horizon. The prediction begins from each agent’s current observation and other agents’ recent policy fingerprints, then feeds predicted observations into later steps.

2) Intention Prediction and Sharing:

  • IntNet shares a sequence of future policy fingerprints summarizing each agent’s projected decisions over horizon H.These fingerprints represent agents’ estimated intentions.Source: 2) Intention Prediction and Sharing · p. 3
  • Each agent uses the predicted observations to compute future policy fingerprints for sharing over the specified horizon.For later steps, the predictor takes the preceding predicted observation as input.Source: 2) Intention Prediction and Sharing · pp. 3–4
  • The observation predictor recursively forecasts future observations from the current observation and other agents’ policy fingerprints.At the first step, it uses the current observation and other agents’ previous-step fingerprints; later steps use the previous predicted observation.Source: 2) Intention Prediction and Sharing · p. 4
  • The recursive predictions incorporate other agents’ evolving intentions to support forward-looking, coordinated decisions.The passage describes this as a mechanism for forming a coherent plan of anticipated observations over H.Source: 2) Intention Prediction and Sharing · p. 4

3) Communication Scheduling and Graph Attention:

A scheduler constructs communication links to reduce unnecessary exchanges, while a Graph Attention Network processes received messages. The attention weights prioritize relevant neighbors when producing updated agent features.

3) Communication Scheduling and Graph Attention:

  • The scheduler constructs an adjacency matrix Gt to determine which agents communicate at each time step, reducing unnecessary data exchange.Gt_ij is 1 when agent i communicates with agent j at time t, and 0 otherwise.Source: 3) Communication Scheduling and Graph… · p. 4
  • The Graph Attention Network computes attention coefficients between each node and its neighbors, then normalizes them across neighbors with softmax.The attention mechanism uses node features, a weight matrix, and a weight vector.Source: 3) Communication Scheduling and Graph… · p. 4
  • The GAT updates each node’s feature representation as a weighted sum of neighbor information, adaptively prioritizing relevant messages.After message processing, independent linear layers produce policy and value outputs.Source: 3) Communication Scheduling and Graph… · p. 4

C. Training

The observation predictor and policy are trained concurrently using episode-rollout data, rather than pretraining the predictor. The predictor learns next observations from transitions, while the policy uses an actor-critic loss.

C. Training

  • The observation predictor and policy are updated concurrently from episode rollouts, without a separate predictor pretraining phase.The paper contrasts this with approaches such as Nagabandi et al. [15], which pre-train the predictor.Source: C. Training · p. 4
  • The predictor is trained on rollout transitions using current observations and actions as inputs and next observations as targets.Observations are normalized, and variables are categorized as continuous or categorical; the predictor uses MSE loss.Source: C. Training · p. 4
  • In parallel, the policy is trained with an actor-critic loss incorporating both actor and critic losses.The critic’s value function is identified in the training description.Source: C. Training · p. 4

D. Environment Description

The experiments use a modified Highway-Env for multi-agent urban navigation, with randomized vehicle conditions and communication limited to 60 meters. Scenario-specific rewards prioritize safety, and scenarios include different mixes of autonomous and human-driven vehicles.

D. Environment Description

  • The modified Highway-Env adds real-time communication and dynamic graph learning to multi-agent urban navigation.Vehicle speeds, locations, and destinations are randomized, and communication is limited to 60 meters to align with DSRC protocol constraints.Source: D. Environment Description · p. 5
  • Agents have limited visibility and exchange messages through a vehicle-to-vehicle network; observations use 64 raycasts over 360° to detect vehicles within 80 meters.Observations are updated every timestep.Source: D. Environment Description · p. 5
  • Actions are discrete meta-actions for acceleration, deceleration, maintaining speed, and changing lanes.Scenario-specific speed values are included in the Appendix.Source: D. Environment Description · p. 5
  • The cooperative reward combines speed, collision avoidance, arrival, and cooperative lane-merging terms, with safety prioritized during weight tuning.Reward weights are tailored to scenario objectives; for example, arrival is less relevant in highway environments.Source: D. Environment Description · p. 5

IV. EXPERIMENTS

The experiments compare IntNet with communication-based MARL baselines in three driving scenarios, including settings with both autonomous and human-driven vehicles. They assess arrival, collision, and team-return performance across repeated training and testing runs.

IV. EXPERIMENTS

  • The Highway and Lane Merging scenarios include autonomous and human-driven vehicles, while the Intersection scenario contains only CAVs.Human-driven vehicles are modeled with IDM/MOBIL.Source: §IV · p. 5
  • Evaluation uses average arrival rate, collision rate, and team return to measure task completion, safety, and overall performance.Arrival requires all CAVs to reach their destinations; collision rate counts episodes with at least one CAV collision.Source: §IV · p. 5
  • Learning curves use three independent seeds, results are validated over 100 test episodes, and experiments use prediction horizon H = 2.H = 2 was chosen based on a preliminary experiment.Source: §IV · p. 5

A. Baseline comparison

IntNet achieves higher rewards and success rates across the three scenarios, with faster learning in Highway and the strongest baseline difficulty in Lane Merging. As intersection traffic grows, IntNet has lower collision rates than IC3Net.

A. Baseline comparison

  • IntNet achieves higher rewards and success rates across the three scenarios than the compared communication-based MARL methods.The comparison includes No Communication, CommNet, IC3Net, and MAGIC.Source: A. Baseline comparison · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
  • In Highway, IntNet reaches high success levels in 100k training episodes, while baselines need around 200k for similar success and safety.In Intersection, IC3Net has comparable success, but IntNet's learning curve is marginally quicker and more consistent; in Lane Merging, baselines learn inconsistently and often favor speed at the expense of safety, causing more collisions.Source: A. Baseline comparison · p. 6

B. Scalability

  • With ten agents, IC3Net's collision rate exceeds 15%, while IntNet produces significantly lower collision rates as agent count increases.The intersection evaluation progressively increases the number of agents and reports test collision rates.Source: B. Scalability · p. 6, Figure 6 · p. 6, Table II · p. 6

C. Communication Efficiency

The communication scheduler reduces network load while maintaining the same success level, and agents communicate at key moments in the intersection. The conclusion reports safer, more efficient policies, reduced graph density, and faster learning convergence.

C. Communication Efficiency

V. CONCLUSIONS

  • The conclusion reports safer and more efficient policies, reduced communication-graph density, and faster learning convergence in complex multi-agent urban settings.Future work will extend scenarios and prediction horizons and consider communication delays and packet loss.Source: §V · p. 7

APPENDIX

The appendix identifies the scope of the scenario-parameter and IntNet-hyperparameter tables, and reports a preliminary experiment used to select the prediction horizon.

APPENDIX

  • Reward weights are ordered as [rspeed, rcol, rarr, rmerge], with values assigned by scenario.These components correspond to speed, collision, arrival, and cooperative lane-merging rewards.Source: 1) Environment · p. 7
  • The appendix includes separate tables for simulation parameters across scenarios and for IntNet hyperparameters.The supplied text does not state the table entries.Source: Table IV · p. 7, Table V · p. 7
  • In Lane Merging, test success performance decreased for H > 2, with greater variability and diminishing returns, so experiments use H = 2.The preliminary experiment tested H values from 1 to 6 and selected H = 2 to balance predictive accuracy and practical utility.Source: 3) Prediction Horizon H · p. 7, Figure 8 · p. 7

Related papers

Loading 2610.34466v1…