Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination

Esmaeil Seraj; Rohan Paleja; Luis Pimentel; Kin Man Lee; Zheyuan Wang; Daniel Martin; Matthew Sklar; John Zhang; Zahi Kakish; Matthew Gombolay

Source-linked AI summary

Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination

Esmaeil Seraj, Rohan Paleja, Luis Pimentel, Kin Man Lee, Zheyuan Wang, Daniel Martin, Matthew Sklar, John Zhang, Zahi Kakish, Matthew Gombolay

arXiv:2606.20962v1Jun 18, 2026cs.ROcs.MA

TL;DR

Heterogeneous robot teams need communication strategies that account for differing state, action, and observation spaces while scaling across team configurations. The paper extends HetNet with scalable heterogeneous graph attention, MAH-PPO, transfer-oriented preprocessing, binarized messaging, and noisy-channel handling. Across small, medium, and large configurations, HetNet outperforms baselines while substantially reducing communication bandwidth.

Problem
MARL has struggled to learn effective communication for heterogeneous teams with differing state, action, and observation spaces, especially under scaling, partial observability, and communication constraints.
Method
The paper extends HetNet with MAH-PPO, scalable heterogeneous graph-attention preprocessing, binarized encoder-decoder communication, transfer across configurations, and noisy-channel modeling.
Results
HetNet achieves 5.91% to 1131.00% improvement over baselines in small configurations and 32.5% to 1166.10% improvement in medium and large configurations.
Takeaways & Limitations
HetNet supports heterogeneous coordination across varied team sizes and communication conditions while learning policies that outperform the evaluated baselines.

Abstract

from arXiv · show

High-performing human-human teams learn intelligent and efficient communication and coordination strategies to maximize their joint utility. These teams implicitly understand the different roles of heterogeneous team members and adapt their communication protocols accordingly. Multi-Agent Reinforcement Learning (MARL) has attempted to develop computational methods for synthesizing such joint coordination-communication strategies, but emulating heterogeneous communication patterns across agents with different state, action, and observation spaces has remained a challenge. Without properly modeling agent heterogeneity, as in prior MARL work that leverages homogeneous graph networks, communication becomes less helpful and can even deteriorate the team's performance. In the past, we proposed Heterogeneous Policy Networks (HetNet) to learn efficient and diverse communication models for coordinating cooperative heterogeneous teams. In this extended work, we extend Heterogeneous Policy Networks (HetNet) to support scaling heterogeneous robot teams. Building on heterogeneous graph-attention networks, we show that HetNet not only facilitates learning heterogeneous collaborative policies but also enables end-to-end training for learning highly efficient binarized messaging. Our empirical evaluation shows that HetNet sets a new state of the art in learning coordination and communication strategies for heterogeneous multi-agent teams by achieving an 5.84% to 707.65% performance improvement over the next-best baseline across multiple domains while simultaneously achieving a 200x reduction in the required communication bandwidth.

I. INTRODUCTION

The paper extends MARL communication for heterogeneous robot teams, where differing capabilities and interpretations make coordination difficult. It introduces a scalable HetNet-based framework targeting transfer, efficiency, and robustness across team compositions and noisy channels.

Motivation: Prior MARL approaches have struggled to model heterogeneity and communication overhead, limiting their applicability to composite teams.Related communication methods include discrete, continuous, attentional, and graph-based approaches, but are not readily applicable to all heterogeneous settings.
Motivation: Heterogeneous robot teams combine agents with different state, action, and observation spaces, making shared communication protocols difficult to design.Agents may also lack usable sensory inputs, creating dependencies on better-sensing teammates.
Prior Work: HetNet learns heterogeneous communication protocols by modeling sender-receiver relationships within cooperative robot teams.Prior work reported a 200× reduction in communicated bits per round over baselines.
Extension Goals: The extended framework targets large task configurations, sparse global rewards, transfer to novel team compositions, and robustness to noise-degraded communication.The paper also addresses the many possible composite-team permutations that make agent-count-specific MARL policies less useful.
Contributions: The paper introduces MAH-PPO, a scalable architecture invariant to agent count, environment size, and observation-space size.Its contributions include inter- and intra-class coordination, FCN-based transferability, and policies for noisy communication channels.

III. PRELIMINARIES

The paper formulates heterogeneous cooperative robot coordination as a Multi-Agent Heterogeneous POMDP. Agents are organized by class, may have distinct state, action, and observation spaces, and receive a shared team reward.

MAH-POMDP Formulation: The Multi-Agent Heterogeneous POMDP extends a standard POMDP with agent classes and class-specific state, action, and observation spaces.The formulation is represented as a 9-tuple containing classes, agent counts, spaces, reward, transitions, and discount factor.
Observations and Actions: Agents can receive class-specific partial observations or no environmental observation when sensory input is unavailable.Regardless of observation access, each agent selects an action according to its class-dependent policy.
Shared Objective: All agents receive the immediate reward r(s̄, ā), which is shared across classes as the objective for learning cooperative policies.The objective maximizes total expected discounted reward over an infinite horizon.

A. Policy Gradient Methods

Policy-gradient methods optimize parameterized action policies using estimated expected returns. PPO improves stability and sample efficiency by constraining policy updates, adding exploration, and applying repeated minibatch updates.

Policy Gradients: A policy-gradient agent uses a parameterized policy πθ(a|s) to choose actions that maximize expected future discounted reward.Gradient ascent updates the actor parameters using an estimate of the expected return.
Policy Gradients: The advantage function measures how much better an action is than the current policy’s default action.The state-action value is approximated from discounted rewards, while the state value is estimated by a critic network.
Proximal Policy Optimization: PPO constrains policy-gradient updates so the current policy does not change too much from the previous policy.The clipped surrogate objective uses the probability ratio between current and old policies.
Proximal Policy Optimization: PPO adds an entropy term for exploration and performs multiple minibatch gradient updates to improve sample efficiency.The entropy coefficient controls the exploration contribution.

C. Graph Neural Networks

Graph neural networks represent agent interactions through message passing, while HetNet extends this idea with heterogeneous attention and scalable preprocessing. The method also uses class-specific binarized communication for heterogeneous teams.

Graph Neural Networks: GNNs update node representations by aggregating feature vectors from neighboring nodes through message passing.After L layers, a node captures structural information from nodes reachable within L hops.
Graph Attention: Attention coefficients replace structure-dependent normalization to improve generalizability when graph size changes.Softmax normalizes feature-dependent coefficients across neighbors, avoiding dependence on a fixed graph structure.
HetNet: HetNet combines heterogeneous graph-attention layers with preprocessing that is invariant to agent count, environment size, and observation-space size.Its method includes class-specific communication, binarized encoder-decoder messages, and arbitrary-depth HetNet assembly.
Communication Architecture: The communication architecture produces local embeddings, converts them into binarized messages, and decodes and aggregates them with learned attention.The design supports direct communication among heterogeneous agents with different capabilities and information inputs.

B. Heterogeneous Communication Model

HetNet represents heterogeneous agents as typed nodes and communication links, then uses class-specific preprocessing to convert variable-sized state and observation tensors into fixed-length features. This design supports heterogeneous communication while accommodating changes in team composition and environment size.

Homogeneous graph methods use universal feature updates, whereas HetNet learns diverse communication strategies from agent classes.
Each agent class defines a node type with class-specific state and action spaces, while nearby agents communicate through directed bidirectional edges.
Separate state and observation preprocessors use FCN and LSTM units to encode spatial and temporal information before HetGAT communication.
Variable-sized tensorized representations encode agent and class dimensions, allowing preprocessing across different environment sizes and team compositions.The tensor representation indexes agents by i and classes by c; state tensors use one channel, while observation channels correspond to heterogeneous classes.
CNN processing and adaptive average pooling produce fixed-length embeddings from varying tensor dimensions, supporting policy transfer without additional parametrization.
The tensor-based formulation supports partial observability with message communication but is limited to domains whose state and observations can be represented as tensors.

D. Binarized Communication Channels

HetGAT communication uses class-aware sender and receiver phases with edge-type attention and binarized messages. Class-specific encoders and decoders preserve heterogeneous communication structure while reducing messages to an efficient intermediate language.

Each HetGAT feature update first performs per-edge-type message passing and then per-node-type feature reduction through sender and receiver phases.
During sending, agents apply class-specific transformations and encoders before transmitting messages to neighboring agents.
Gumbel-Softmax converts encoded messages into 0s and 1s, producing binarized communication across classes.The encoder maps features to a communication dimension n, which represents channel bandwidth.
During receiving, class-specific decoders reconstruct messages, and normalized attention coefficients aggregate them separately for each communication edge type.
Multi-head HetGAT computes independent parallel subchannels and merges their outputs by concatenation, using averaging in the final layer.

E. Heterogeneous Policy Network (HetNet)

HetNet stacks heterogeneous graph-attention layers to perform multi-round communication and produce class-specific action features. MAH-PPO extends PPO with class-wise objectives to improve learning in heterogeneous, shared-reward settings.

Each HetGAT layer represents one message-exchange round, and stacked layers generate high-level agent embeddings for decision-making.The final layer outputs features sized to each agent class’s action space.
MAH-PPO modifies PPO to learn class-wise coordination policies for heterogeneous multi-agent scenarios.
The per-class surrogate objective uses class-specific probability ratios, advantage estimates, and clipped updates.
The policy incorporates joint actions, joint observations, and received message vectors in its class-specific probability ratio.
A centralized critic with global state input supplies advantage estimates, while an entropy term encourages exploration.
HetNet can be trained with shared rewards, although the formulation also permits individual and class-specific rewards.

B. Scalability via Transfer

Scaling MARL is difficult because larger domains expand exploration, credit assignment, computation, and policy-gradient challenges. HetNet addresses this by training on a small configuration and transferring the converged policy to larger configurations.

Large-scale MARL increases the joint state-action space, complicates credit assignment, raises policy-gradient variance, and prolongs training.
HetNet is first trained to convergence on a small environment with few agents, then directly transferred to a larger environment with more agents.

C. Lossy Communication

HetNet models lossy inter-agent communication by perturbing binarized messages during transmission, addressing the usual assumption of reliable channels. The framework evaluates static, distance-dependent, and agent-type-dependent noise models parameterized by bit error rate.

Communication-channel modification: HetNet introduces artificial bit perturbation between message encoding and receiver decoding to model transmission-channel loss.Sender features are class-encoded and binarized before noise is applied prior to decoding.
Noise modeling: The noisy channel ρ outputs a noisy message and is non-differentiable because communication noise is stochastic.The channel is evaluated over HetNet-Binary with tensor-based agent observations.
Noise-model variants: HetNet evaluates static white Gaussian, distance-dependent, and type-based communication noise models.These models respectively vary noise intensity globally, with inter-agent distance, or according to communicating agents’ classes.
Noise parameterization: All three noisy-channel implementations are parameterized by the bit error rate (BER) p_b.The BER controls the likelihood of bit perturbation in the encoded message.
Structured channel effects: The range-based model scales signal strength with inter-agent distance, while type-based noise uses edge-specific attenuation constants γ_e.In the type-based model, γ_e = 1 for every edge recovers the base AWGN noise model.

VI. EMPIRICAL EVALUATION

The evaluation compares HetNet with communicative MARL baselines across two heterogeneous domains, PCP and FC, using small, medium, and large configurations. HetNet achieves more efficient coordination, stronger convergence performance, and improved results over baselines and prior MAHAC training.

Evaluation Environments: The evaluation covers two cooperative heterogeneous domains, PCP and FireCommander, with perception-action agent classes and varying environment and team sizes.PCP uses perception agents with limited vision and action agents without observations; FC requires perception and action agents to discover, track, and extinguish propagating fires.
Performance Comparison To Baselines: HetNet outperforms baselines in both PCP and FC by converging to coordination policies requiring fewer steps, regardless of shared or individual reward training.Learning curves report average steps across episodes and three random seeds.
Performance Comparison To Baselines: HetNet outperforms all baselines in 100-trial evaluations across both heterogeneous domains, and it alone learns a high-performing policy in the more complex small FC configuration.Evaluation uses identical randomly generated initial conditions and final policies at convergence.
Performance Comparison To Baselines: Shared rewards outperform individual rewards among baseline methods because they support policies aimed at the composite team’s goal rather than one agent class’s skill.The results associate individual rewards with suboptimal policies in heterogeneous teaming.
Performance Comparison To Baselines: 5.91% to 1131.00% performance improvement is achieved over baselines in small PCP and FC configurations, establishing state-of-the-art heterogeneous policies for composite teams.The comparison includes baselines trained with shared or individual rewards.
Performance Comparison To Baselines: 23.83% to 1166.10% performance improvement over MAHAC is achieved by MAH-PPO in small configurations, rising to 305.13% to 1166.10% in medium configurations.MAH-PPO outperforms MAHAC across both PCP and FC and across both configuration scales.

2) Performance Comparison to HetNet under MAHAC:

MAH-PPO improves HetNet’s ability to learn coordination policies in complex heterogeneous domains compared with MAHAC and makes large-scale training feasible. The framework also benefits from transfer across configurations, improving sample efficiency and runtime.

MAH-PPO versus MAHAC: HetNet trained with MAH-PPO outperforms HetNet trained with MAHAC across PCP and FireCommander domain configurations and scales.The comparison covers Small and Medium configurations under shared reward conditions.
Training feasibility: 98.68% decrease in experiment runtime makes training HetNet with MAH-PPO feasible where MAHAC training in PCP Large is intractable.The result is attributed to implementation improvements in the MAH-PPO training procedure.
Scalability: 32.5% to 1166.10% performance improvement over other baselines is achieved by HetNet PPO in medium and large PCP and FireCommander configurations.HetNet PPO outperforms baselines trained with either shared or individual rewards.
Transferability: 389.51% improvement in sample efficiency and 80.63% decrease in runtime result when a transferred HetNet policy reaches PCP medium performance.The transferred policy reaches under 20 steps per episode in approximately 3 million samples, versus approximately 15 million for training from scratch.
Transferability: HetNet policies can transfer from small to medium and from medium to large PCP configurations, including configurations where baselines trained from scratch perform worse.Transfer increases agent count and observation dimensionality in the medium configuration.

5) Robustness:

The evaluation examines communication-noise robustness, binarized-message bandwidth, and a physical-robot demonstration. HetNet’s performance degrades under noise but remains above no-communication performance in supported conditions, while binarized communication reduces bandwidth.

Communication noise: HetNet performance decreases as communication noise increases, with AWGN performance decaying exponentially as SNR decreases and approaching no-communication performance at negative SNR.Higher SNR lowers the bound on average steps, while negative SNR approaches approximately 20 steps.
Communication noise: HetNet under range-based and type-based noise remains above no-communication performance at some SNR levels, with type-based noise producing performance similar to AWGN.The range-based noise condition causes a significant decrease relative to noise-free communication.
Communication bandwidth: Binarized HetNet-PPO messages require significantly less Communication Bandwidth than real-valued baselines, measured as bits communicated per round.The bandwidth-performance relationship follows a Pareto-efficiency curve in the small FireCommander domain, where baselines did not converge.
Physical-robot demonstration: The physical Robotarium demonstration uses two perception agents and one action agent coordinating wildfire detection and suppression across eight trajectory frames.Perception agents are marked in blue and the action agent in red; the demonstration is not a training evaluation on real-world robots.
Scope boundary: Training MARL algorithms on real-world heterogeneous robots remains future work because it can require a large number of samples.The paper demonstrates a deployed policy but does not report training MARL algorithms directly on physical robots.