Source-linked AI summary
Who2com: Collaborative Perception via Learnable Handshake Communication
Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, Zsolt Kira
TL;DR
The paper asks how robots can combine observations to improve perception for agents with degraded sensors while respecting bandwidth limits. It introduces an end-to-end three-stage handshake that compresses requests, matches neighboring agents, and selects information to receive. For semantic segmentation, the method improves accuracy over decentralized baselines and approaches centralized performance with one quarter of the bandwidth.
Problem
Collaborative perception requires combining neighboring observations to improve degraded agents’ scene understanding while learning communication under bandwidth constraints.
Method
An end-to-end handshake framework uses compressed requests, learned matching scores, and selected-agent information transfer without labels identifying the best communicating agent.
Results
16.51% relative accuracy improvement over CatAll is reported with one quarter of its bandwidth usage, while the method remains comparable to centralized approaches.
Takeaways & Limitations
The framework effectively combines neighboring information while using significantly less bandwidth than centralized approaches, and AirSim-CP enables photorealistic evaluation.
Abstract
from arXiv · showhide
In this paper, we propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task. Unlike existing work in robotics and multi-agent reinforcement learning, we formulate the problem as one where learned information must be shared across a set of agents in a bandwidth-sensitive manner to optimize for scene understanding tasks such as semantic segmentation. Inspired by networking communication protocols, we propose a multi-stage handshake communication mechanism where the neural network can learn to compress relevant information needed for each stage. Specifically, a target agent with degraded sensor data sends a compressed request, the other agents respond with matching scores, and the target agent determines who to connect with (i.e., receive information from). We additionally develop the AirSim-CP dataset and metrics based on the AirSim simulator where a group of aerial robots perceive diverse landscapes, such as roads, grasslands, buildings, etc. We show that for the semantic segmentation task, our handshake communication method significantly improves accuracy by approximately 20% over decentralized baselines, and is comparable to centralized ones using a quarter of the bandwidth.
I. INTRODUCTION
The paper formulates collaborative perception as bandwidth-limited information sharing that helps agents with degraded observations. It introduces learnable handshake communication and evaluates it with AirSim-CP for multi-agent scene understanding.
- Collaborative perception combines neighboring robots’ observations to improve a degraded agent’s perception accuracy.The motivating examples include occlusion, sensor degradation, and sensor failure.
- The proposed problem requires learning what information to communicate while respecting bandwidth constraints.The framework targets scene-understanding tasks such as semantic segmentation.
- Handshake communication uses request, match, and connect stages to select useful information from other agents.The degraded agent broadcasts a compressed request, normal agents return matching scores, and the target connects to a selected agent.
- AirSim-CP provides photo-realistic aerial-robot scenarios with diverse landscapes for evaluating multi-agent perception under varied conditions.The scenarios vary trajectories, field-of-view overlap, and pose-information quality.
- 20% improvement in accuracy over decentralized baselines is reported for semantic segmentation, with performance comparable to centralized methods using one quarter of the bandwidth.The method is trained end-to-end without supervision identifying the best communicating agent.
II. RELATED WORK
Prior communication research largely addresses multi-agent decision-making or simplified observations, while this paper studies learnable, bandwidth-aware communication for photorealistic perception.
- Multi-agent communication has been extensively studied for collaborative perception and decision-making, especially in multi-agent reinforcement learning.Earlier systems used predefined protocols, followed by learnable communication methods for changing environments and agent counts.
- Simplistic 2D grid environments with low-dimensional observations limit analysis of the interaction between perception and communication.The paper instead evaluates collaboration in a more complicated, photorealistic environment.
- Existing targeted or group-based communication methods can broadcast broadly or scale bandwidth with the number of agents.The proposed framework instead selects which agents to communicate with to reduce bandwidth while maintaining perception performance.
III. PROPOSED METHOD
The method lets a degraded target agent fuse neighboring observations under bandwidth limits through a learned three-stage handshake. It compresses requests, matches agents using learned keys, and retrieves selected feature maps for prediction.
- A. Problem Definition and Motivation: The task optimizes target-agent prediction accuracy and transmission bandwidth when the target observation is degraded.Semantic segmentation is used as one instantiation of the broader perception framework.
- B. Communication via Three-Stage Handshake: The degraded agent broadcasts a compressed request message, normal agents compute matching scores, and the target selects agents with the highest scores.The implementation selects top n = 1, while the mechanism can generalize to top-n selection.
- B. Communication via Three-Stage Handshake: The framework decouples message, key, and value representations, allowing different compression rates for each communication stage.This design is intended to improve the bandwidth–accuracy trade-off.
- B. Communication via Three-Stage Handshake: Matching scores compare the degraded agent’s request with keys generated from other agents’ observations.The method uses a learned matching function, with general attention selected empirically as the default.
- B. Communication via Three-Stage Handshake: The selected agent transmits a feature map, which the degraded agent concatenates with its own encoded feature map before final prediction.The transmitted feature map supplies high-dimensional information for semantic segmentation.
C. Learning to Communicate with Weak Supervision
The model uses centralized training with decentralized execution: training accesses all normal-agent observations, while inference connects only to selected agents under bandwidth limits. Softmax enables differentiable training, argmax selects the highest-scoring agent at inference, and target-view labels provide the only supervision.
- Centralized training with decentralized execution: During training, the target agent accesses all other agents’ observations, but inference uses only information from selected agent(s).This follows the centralized-training, decentralized-execution strategy.
- Centralized training with decentralized execution: Training weights normal-agent observations by matching scores before the task decoder computes the final prediction.
- Decentralized execution: Argmax inference connects the target agent to the normal agent with the highest computed matching score.
- Differentiable training: Softmax is used during training and argmax during inference because argmax selection is non-differentiable during training.The authors report similar results to more complex schemes such as sparsemax.
- Weak supervision: The end-to-end objective uses only target-view ground-truth annotations, without labels identifying the best communication agent.The loss is cross-entropy between the target-view labels and prediction.
A. AirSim-CP Dataset
AirSim-CP is a simulator-based dataset for benchmarking collaborative perception with aerial robots observing diverse landscapes. It records multimodal observations and provides a semantic segmentation mask for one agent.
- Dataset construction: AirSim-CP uses AirSim to simulate five drones flying over roads, grasslands, buildings, lakes, and other diverse landscapes.
- Recorded data: For each drone, the dataset records RGB images, depth images, and poses.
- Benchmark task: Semantic segmentation is the downstream task used to benchmark collaborative perception methods.
- Annotations: The dataset provides the semantic segmentation mask of one agent.
B. Proposed Experimental Settings.
The experiments vary trajectories, view overlap, pose accuracy, and target-view degradation across four AirSim-CP settings. They test collaboration during waypoint following and random exploration, including cases with aligned or misaligned warped observations.
- Experimental settings: The dataset includes four settings: hidden target view, accurate pose with waypoint following, inaccurate pose with waypoint following, and accurate pose with random exploration.
- Target degradation: All settings include a degraded target agent whose view is perturbed with Gaussian blur and Gaussian noise.
- View overlap: Waypoint-following experiments vary whether the target and normal agents’ fields of view overlap partially or completely.
- Accurate pose: Accurate-pose waypoint following uses depth and relative pose transformations to warp normal-agent observations into the target view.
- Inaccurate pose: Inaccurate-pose experiments add positional noise, producing warped images that are not well-aligned with the target view.
- Random exploration: Random-exploration experiments examine changing relative positions and overlapping fields of view as agents disperse and wander.
C. Baselines
The baselines span single-agent, centralized feature-fusion, random distributed selection, and the proposed learned distributed method. Centralized methods transmit all normal-agent feature maps, whereas distributed methods transmit one selected feature map.
- Single-agent baselines: Single normal and Single degraded provide upper- and lower-bound models trained on non-degraded and degraded target images, respectively.
- Centralized baselines: CatAll centrally concatenates features from degraded and normal agents for semantic segmentation.
- Centralized baselines: Attention centrally weights and sums feature maps instead of concatenating them.
- Centralized baselines: Compression centrally applies two convolutional layers and uniformly compresses all observations at rate 25% before concatenation.
- Distributed baselines: Random selection chooses a feature map from a random normal agent, while ours learns which agent to communicate with.
- Distributed baselines: The proposed method includes a message variant and a constant-request variant, without training labels identifying the best agent.
- Bandwidth: Centralized baselines require all normal-agent feature maps, while Random selection and ours transmit only a single image feature map.Thus, centralized bandwidth scales linearly with the number of agents in the system.
D. Evaluation metrics.
The evaluation measures semantic-segmentation performance with overall accuracy and communication cost with kilobytes per frame, while BIS summarizes accuracy improvement relative to bandwidth usage.
- Overall accuracy measures semantic-segmentation performance, while Kbytes per frame measures bandwidth usage for communication and selection.
- Bandwidth-Improvement Score (BIS) measures relative improvement in overall accuracy divided by bandwidth usage.Higher BIS results from smaller bandwidth usage and larger accuracy improvements.
V. RESULTS AND ABLATION STUDIES
Across waypoint-following and random-exploration settings, the handshake models improve accuracy and bandwidth efficiency, while ablations show the value of message conditioning and asymmetric compression.
- Multi-agent Waypoint Following: The model improves overall accuracy by a relative 16.51% versus CatAll using only one quarter of the bandwidth.The comparison indicates that concatenating all observations does not guarantee meaningful information combination.
- Multi-agent Waypoint Following: The model with the message improves overall accuracy by a relative 20.06% versus the Compression model.The authors attribute this to the need for high-dimensional feature maps for pixel-wise and fine-grained predictions.
- Message Ablation: 29.49% relative mIoU improvement from adding the message demonstrates its necessity for communication.
- Pose Variation: With accurate or inaccurate pose, the models achieve results similar to centralized methods using one quarter of the bandwidth and the highest BIS among methods.Inaccurate pose lowers BIS somewhat, but it remains significantly above the baselines.
- Random Exploration: The models perform favorably against baselines during random exploration, indicating robustness across changing environments and tasks.Relative positions and overlapping fields of view change frequently in this setting.
- Message and Key Sizes: A message size of 4 is sufficient for comparable selection and segmentation performance, after which larger sizes plateau.Message sizes range from 1 to 64 in the ablation, while key sizes range from 4 to 1024.
- Message and Key Sizes: A small message such as 8 paired with a large key such as 1024 achieves amenable performance while reducing transmitted information.The key is used locally to compute scores and therefore does not need transmission.
VI. CONCLUSION
The paper formulates collaborative perception as bandwidth-constrained information fusion and proposes handshake communication with asymmetric compression to improve accuracy efficiently.
- Handshake communication learns compressed representations and decouples message, key, and value elements to support asymmetric compression and bandwidth savings.
- AirSim-CP and benchmarking metrics evaluate whether neighboring-agent information can improve scene-understanding accuracy with less bandwidth than centralized approaches.