Source-linked AI summary
When2com: Multi-Agent Perception via Communication Graph Grouping
Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, Zsolt Kira
TL;DR
Collaborative perception needs distributed agents to share information despite bandwidth limits and potentially degraded or uninformative sensors. The paper learns communication groups and when to communicate, generalizing across two vision tasks. It reports superior performance with lower bandwidth than compared methods, including 0.98 MBpf versus Who2com’s 2 MBpf in one setting.
Problem
Multi-agent collaborative perception remains under-studied because distributed agents must improve perception through bandwidth-limited communication rather than centralized access.
Method
The framework learns which agents communicate and when, using only downstream task supervision and supporting decentralized inference after centralized training.
Results
The method achieves superior performance with lower bandwidth across compared methods and reports 0.98 MBpf versus Who2com’s 2 MBpf in the SRMS setting.
Takeaways & Limitations
Learned communication groups and timing provide a bandwidth-efficient framework applicable to collaborative semantic segmentation and multi-agent 3D shape recognition.
Takeaways & Limitations
Inference assumes each agent observes only its local data and communicates through bandwidth-limited channels, including scenarios with uninformative or degraded sensors.
Abstract
from arXiv · showhide
While significant advances have been made for single-agent perception, many applications require multiple sensing agents and cross-agent communication due to benefits such as coverage and robustness. It is therefore critical to develop frameworks which support multi-agent collaborative perception in a distributed and bandwidth-efficient manner. In this paper, we address the collaborative perception problem, where one agent is required to perform a perception task and can communicate and share information with other agents on the same task. Specifically, we propose a communication framework by learning both to construct communication groups and decide when to communicate. We demonstrate the generalizability of our framework on two different perception tasks and show that it significantly reduces communication bandwidth while maintaining superior performance.
1. Introduction
The paper frames collaborative perception as bandwidth-limited information exchange among distributed sensing agents. It proposes learning both communication groups and communication timing, achieving favorable performance across tasks with less bandwidth.
- Multi-agent perception can improve coverage and robustness, but distributed agents must exchange information under bandwidth-limited channels.The setting includes uninformative or degraded sensor data.
- The framework learns which agents should communicate and when, without explicit ground-truth communication labels.It decouples communication stages to support asymmetric message and key sizes, reducing transmitted data.
- The model supports collaborative semantic segmentation and 3D shape recognition through end-to-end training from downstream task supervision.The perception decoder is not restricted to a specific vision task.
- Across different tasks, the method performs favorably against prior learned-communication methods while using less bandwidth.The paper also analyzes message-query trade-offs, learned groupings, and message-key correlations.
2. Related works
Prior communication methods rely on rigid protocols, fully connected graphs, or assumptions that limit scalability and adaptability. The paper motivates learned group communication that prunes irrelevant links and decides whether communication is needed.
- Pre-defined communication protocols and heuristic camera-grouping rules do not readily adapt to dynamic or complex environments.Recent multi-agent reinforcement-learning methods instead explore learnable interactions.
- Fully connected communication uses bandwidth growing on the order of O(N^2), making scaling difficult as the number of agents increases.Group communication prunes irrelevant connections to reduce network complexity.
- Who2com reduces bandwidth through handshake communication but assumes every agent always communicates with another agent.That assumption wastes bandwidth and does not prevent detrimental messages.
- The proposed framework addresses these shortcomings by learning both communication timing and communication-group construction.Its communication design is inspired by attention mechanisms that can use asymmetric queries and keys.
3. Method
The method learns bandwidth-limited communication groups and when agents should communicate, then integrates selected supporter features into each requester’s downstream perception. It uses compact asymmetric query-key matching, prunes connections, and trains end-to-end from task labels without communication annotations.
- Communication objective: The framework learns both which agents should communicate and when communication is necessary under bandwidth limits.It aims to avoid communication when an agent already has sufficient information and to activate transmission when outside information may improve perception.
- Communication groups: A two-step procedure constructs groups by computing connection weights and pruning less important connections with an activation function.The activation function zeros out matrix elements smaller than the threshold δ.
- Communication groups: Agents compress local observations into compact queries and keys, broadcast queries, and compute learned requester-supporter matching scores.The query is transmitted broadly, while the key remains larger and is not transmitted, enabling asymmetric message and key sizes.
- Communication groups: The pruned matching matrix forms a directed communication graph whose rows describe received information and columns describe supporter transmissions.Blue intra-agent transmission does not consume bandwidth, whereas red inter-agent transmission does.
- Perception integration: Each requester integrates its local feature map with weighted supporter features before decoding the combined representation for downstream visual tasks.The decoder is not limited to a specific vision task, and the framework is demonstrated across different visual tasks.
- Learning procedure: Training uses only downstream task labels and updates the query, key, encoder, and decoder parameters end-to-end, while inference is decentralized and bandwidth-limited.During training, agents can access all local observations; during inference, each agent observes only its own local observation.
4. Experiment
The experiments evaluate bandwidth-efficient collaborative perception across semantic segmentation and multi-agent 3D shape recognition. The proposed method learns communication groups and communication timing, achieving competitive or superior performance with substantially lower bandwidth than baselines.
- Experimental setup: The framework is evaluated on collaborative semantic segmentation and multi-view 3D shape recognition.The segmentation experiments use multi-agent observations to produce a mask for each agent; the recognition experiments use a multi-agent ModelNet40 variant.
- Experimental setup: Performance is measured with mean IoU for segmentation, prediction accuracy for recognition, and MBpf plus average links per agent for bandwidth.Bandwidth includes transmitted feature vectors and broadcast keys, scaled by their storage size.
- Semantic segmentation: 0.98 MBpf versus Who2com’s 2 MBpf: the proposed model achieves higher prediction accuracy while using less bandwidth in SRMS.The SRMS setting tests whether one requesting agent can learn when and whom to contact.
- Semantic segmentation: In MRMS, selecting noisy supporting views can make CatAll and RandCom perform worse than NoCom, whereas attention-based AuxAttend and Who2com avoid this deterioration.These results show that supporting information is not universally beneficial and must be selected conditionally.
- Semantic segmentation: In MRMPS, the proposed model maintains accuracy similar to fully connected models while using one-fourth of their communication bandwidth.With partially overlapping observations, all methods show smaller gains over NoCom than in the previous scenario.
- 3D shape recognition: For multi-agent 3D shape recognition, the model performs competitively with TarMac while using approximately one-eighth of its bandwidth.Figure 6 qualitatively illustrates communication with correct and informative agents.
- Analysis: Removing handshake communication significantly reduces performance relative to the full model, supporting communication for deciding when to communicate and whom to select.The ablation compares the full model with RandCom and an otherwise similar model without handshake communication.
5. Conclusion
The paper presents a general bandwidth-efficient framework for collaborative perception that learns both communication-group construction and communication timing, achieving superior performance with lower bandwidth requirements.
- The framework learns how to construct communication groups and when to communicate for collaborative perception.
- It generalizes across downstream tasks, including multi-agent semantic segmentation and multi-agent 3D shape recognition.
- The framework achieves superior performance with lower bandwidth requirements across all compared methods.