PaperlayerView exact revision on arXiv

Source-linked AI summary

Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps

Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, Siheng Chen

arXiv:2209.12836v1cs.CV

TL;DR

Collaborative perception must balance improved perception against communication bandwidth. Where2comm addresses this by using spatial confidence maps to share sparse, critical features and adjust communicated areas. Across diverse datasets, modalities, and agent types, it consistently achieves a superior perception–bandwidth trade-off.

  • Problem

    Collaborative perception faces an inevitable trade-off between perception performance and communication bandwidth, while prior methods share all spatial areas equally after collaboration begins.

  • Method

    Where2comm uses spatial confidence maps to identify critical areas, pack sparse messages, construct communication graphs, and guide multi-head message fusion.

  • Results

    Where2comm consistently and significantly outperforms previous works in the perception–bandwidth trade-off across multiple datasets and modalities, including real-world and simulation scenarios.

  • Takeaways & Limitations

    Focusing communication on perceptually critical spatial areas enables more compact messages, sparser communication graphs, and more comprehensive fusion.

  • Takeaways & Limitations

    The current work focuses on spatially critical areas and leaves analogous temporal selection, such as determining critical timestamps, for future work.

Abstract

from arXiv · show

Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication. It inevitably results in a fundamental trade-off between perception performance and communication bandwidth. To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information. It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate. Based on this novel spatial confidence map, we propose Where2comm, a communication-efficient collaborative perception framework. Where2comm has two distinct advantages: i) it considers pragmatic compression and uses less communication to achieve higher perception performance by focusing on perceptually critical areas; and ii) it can handle varying communication bandwidth by dynamically adjusting spatial areas involved in communication. To evaluate Where2comm, we consider 3D object detection in both real-world and simulation scenarios with two modalities (camera/LiDAR) and two agent types (cars/drones) on four datasets: OPV2V, V2X-Sim, DAIR-V2X, and our original CoPerception-UAVs. Where2comm consistently outperforms previous methods; for example, it achieves more than $100,000 \times$ lower communication volume and still outperforms DiscoNet and V2X-ViT on OPV2V. Our code is available at https://github.com/MediaBrain-SJTU/where2comm.

1 Introduction

Collaborative perception addresses occlusion and long-range limitations by sharing complementary information, but communication bandwidth creates a central performance–cost trade-off. Where2comm uses spatial confidence maps to communicate sparse, perceptually critical areas and adapts communication across conditions.

  • Collaborative perception shares complementary information to improve holistic perception and address occlusion and long-range limitations.
  • Communication bandwidth must be evaluated alongside perception performance because sharing all spatial areas can waste bandwidth on irrelevant information.Previous methods compress or select collaborators, but assume that collaborating agents share all spatial areas equally.
  • A spatial confidence map identifies perceptually critical areas so agents can exchange spatially sparse features and request complementary information over multiple rounds.
  • Where2comm combines spatial confidence generation, confidence-aware communication, and confidence-aware message fusion to decide what, where, and with whom to communicate.
  • Where2comm consistently improves the perception–bandwidth trade-off across four datasets, two modalities, and car- and drone-based agents while adapting to bandwidth and communication rounds.The evaluation covers real-world and simulation scenarios, LiDAR and cameras, and collaborative 3D object detection.

2 Related Works

Collaborative perception extends multi-agent communication to perception tasks, where research targets a better performance–bandwidth trade-off. Prior work includes predefined, learning-based, and handshake-based communication strategies, alongside growing benchmark datasets and comparative system analyses.

  • Multi-agent communication: Predefined protocols and heuristics have difficulty generalizing to complex communication tasks.Recent work therefore explores learning-based communication methods.
  • Multi-agent communication: CommNet learns continuous communication, while Vain uses attention to selectively fuse information from other agents.These examples come from prior multi-agent communication research.
  • Multi-agent communication: Most prior communication studies address decision-making with reinforcement learning, whereas this work focuses on perception with direct supervision.The paper applies supervised learning to optimize communication strategy for perception tasks.
  • Collaborative perception: Collaborative perception remains immature despite a surge of datasets and methods targeting the performance–bandwidth trade-off.Examples include V2X-Sim, OpenV2V, Comap, and DAIR-V2X; Table 1 compares major system components.
  • Collaborative perception: When2com introduces a handshake communication mechanism to decide when agents communicate.The supplied passage presents this as an example of collaborative-perception methods.

3 Problem Formulation

The formulation defines collaborative perception as maximizing all agents’ perception performance under a total communication budget and communication-round constraint. It also requires robust multi-round collaboration across bandwidth conditions, with agents serving as both message providers and recipients.

  • Objective: Collaborative perception maximizes all agents’ perception performance as a function of communication budget B and communication rounds K.Each agent has an observation X_i and perception supervision Y_i.
  • Objective: The perception metric g and trainable perception network Φ define how collaborative perception performance is evaluated.The formulation also includes transmitted messages P_i→j^(k) between agents at communication round k.
  • Boundary condition: With B = K = 0, the formulation reduces to single-agent perception without collaboration.The paper uses ξ_Φ(0, 0) to denote single-agent perception performance.
  • Requirements: The communication strategy and network parameters should support consistent performance across communication bandwidths and rounds.The formulation explicitly considers multi-round communication.
  • Design goals: For 3D object detection, the paper targets compact messages, sparse communication graphs, comprehensive message fusion, and dynamic communication adaptation.Adaptation adjusts where and who agents communicate under varying communication conditions.

4 Where2comm: Spatial Confidence-Aware Collaborative Perception System

Where2comm uses spatial confidence maps to identify perceptually critical areas, guide sparse communication, and support targeted multi-agent feature fusion. Its modules jointly produce compact messages, adaptive communication graphs, and location-specific attention.

  • Observation encoder: Where2comm projects sensor features into a shared bird’s-eye-view coordinate system before communication.It supports RGB images and 3D point clouds, with image features warped from front view to BEV and point clouds discretized as BEV maps.
  • Spatial confidence generator: The spatial confidence generator converts feature maps into detection confidence maps that mark object-containing, perceptually critical areas.High-confidence object regions are prioritized, while background regions can be omitted to save bandwidth.
  • Spatial confidence-aware communication: Spatial confidence-aware communication selects where to communicate, which partners to use, and what sparse features to transmit.Messages contain request maps and spatially sparse feature maps, while communication graphs avoid unnecessary fully connected exchange.
  • Spatial confidence-aware communication: Binary selection dynamically allocates communication budget across spatial areas according to perceptual criticality and bandwidth conditions.Only non-zero features and their indices are transmitted, reducing communication cost while adapting to varying communication conditions.
  • Spatial confidence-aware message fusion: The fusion module applies multi-head attention independently at each spatial location, using confidence maps and sensing distances as priors.These priors encourage aggregation of higher-quality, more critical features and make collaboration strength spatially flexible.

5 Experimental Results

Experiments span four datasets, real-world and simulated settings, camera and LiDAR modalities, and car and drone agents. Where2comm improves the performance–bandwidth trade-off, remains robust across communication rounds and localization noise, and benefits from its spatially selective fusion designs.

  • Datasets and experimental settings: The evaluation covers four datasets, including real-world and simulation scenarios, car and drone agents, and camera-only and LiDAR-based 3D detection.The datasets are DAIR-V2X, V2X-Sim, OPV2V, and CoPerception-UAVs.
  • Quantitative evaluation: Where2comm achieves a superior perception–communication trade-off across camera-based and LiDAR-based collaborative 3D detection tasks.It improves previous state-of-the-art performance by 7.7% on DAIR-V2X, 6.62% on CoPerception-UAVs, 25.81% on OPV2V, and 1.9% on V2X-Sim.
  • Quantitative evaluation: 5128 times less communication on CoPerception-UAVs and more than 100K times less on OPV2V achieve the same detection performance as previous state-of-the-art methods.The corresponding reductions are 55 times on V2X-Sim and 105 times on DAIR-V2X.
  • Multi-round evaluation: Multi-round communication steadily improves the performance–bandwidth trade-off across three datasets while supporting varying bandwidths and communication rounds.The evaluation compares one to three communication rounds.
  • Robustness to localization noise: Where2comm remains superior under localization noise, while V2VNet and DiscoNet fail at higher noise levels on CoPerception-UAVs.The tested Gaussian noise has zero mean and standard deviation from 0m to 0.6m; V2VNet fails above 0.4m and DiscoNet above 0.5m.
  • Ablation studies: On OPV2V, multi-head attention outperforms vanilla attention by 10.84% on AP@0.50, while all three fusion designs improve performance by 22.06%.The three designs are multi-head attention, sensing position encoding, and spatial confidence map.

6 Conclusion and limitation

Where2comm uses spatial confidence maps to communicate sparse, perceptually critical features and adapt collaboration across communication conditions. The authors identify temporal criticality and communication timing as directions for future work.

  • Conclusion: Where2comm uses spatial confidence maps to promote pragmatic compression, select what and whom to communicate with, and aggregate complementary information over multiple rounds.Each agent offers sparse critical features and requests complementary information from other agents.
  • Conclusion: Experiments across multiple agent types and modalities show a superior perception–communication bandwidth trade-off.The conclusion describes the evidence as comprehensive across multi-type agents and multi-modality inputs.
  • Limitation and future work: The current work focuses on perceptually critical spatial areas and leaves critical timestamps and when to communicate for future investigation.The authors also propose exploring further pragmatic compression and emergent communication methods.

Checklist

The checklist documents the study’s reported contributions, experimental reproducibility materials, and ethical-data status, while noting that repeated experiments for error bars were not conducted.

  • Research reporting: The paper reports that its main claims accurately reflect its contributions and scope.The checklist marks this item affirmatively.
  • Reproducibility and assets: The authors report code, data, instructions, and training details are available in the paper or supplemental material.They also report citing existing assets, describing licenses, and providing new assets via supplemental material or a URL.
  • Experimental reporting: The experiments do not report error bars because the large-scale 3D object-detection experiments were not repeated many times.The authors report that compute details are provided elsewhere.
  • Data and ethics: The datasets are synthesized with open-source tools and are reported to contain no real-world personally identifiable information or offensive content.The checklist states that the asset owners consented to public research use.
  • Ethics and applicability: The checklist reports no theoretical results, human-subject research, crowdsourcing, or potential negative societal-impact analysis as applicable items.These entries are marked N/A, while ethics-guideline compliance is marked Yes.

7 Appendix

Where2comm decomposes collaborative perception into spatial selection and perception optimization, using confidence-guided sparse communication to allocate bandwidth toward critical regions.

  • Contributions: Where2comm introduces a fine-grained spatial-aware strategy that selects and packs messages from perceptually critical spatial areas.The strategy also supports targeted requests from other agents during multi-round communication.
  • Contributions: Where2comm combines spatial confidence guidance with message packing, communication graph learning, and confidence-aware multi-head attention.These components target lower bandwidth and better perception performance.
  • Optimization: The optimization introduces an auxiliary variable and decomposes the original constrained problem into two easier sub-optimization problems.The decomposition addresses hard constraints and binary-variable nondifferentiability.
  • Optimization: The binary selection matrix determines where to communicate by imposing spatial sparsity on messages sent between agents.For one communication round, the transmitted message is P_i→j = M_i→j ⊙ F_i.
  • Optimization: Feasible selection matrices allocate bandwidth by prioritizing spatial regions with the highest confidence values.The proxy solution selects the top-b1 regions under the communication budget.
  • Module design: The framework includes spatial confidence-aware message packing and spatial-decouple partially connected communication to prune irrelevant per-location connections.These design choices make communication more bandwidth-efficient.

7.4 Detailed information about the module design

The module design encodes observations into BEV features, generates spatial confidence, selectively communicates features, and fuses received messages at each location.

  • Observation encoder: The camera observation encoder lifts image features into 3D voxel space before collapsing them into BEV features.Estimated categorical depth and known camera parameters support the projection.
  • Communication: Spatial confidence-aware message packing takes a confidence map as input and outputs a transmitted message and a request map.This module operates at each communication round.
  • Communication: Spatial-decouple partially connected communication prunes irrelevant connections separately by location, reducing bandwidth compared with fully connected communication.Fully connected communication grows on the order of O(N^2) with the number of agents.
  • Fusion: Message fusion decodes received messages into confidence maps and applies per-location multi-head attention to produce augmented features.The attention uses features and corresponding confidence scores at each location.
  • Observation encoder: Sensor positional encoding uses physical distance from sensor coordinates to BEV-grid locations as a spatial prior.The encoding applies sine and cosine functions to the distance.

7.5 Detailed information about experimental settings

Experiments evaluate Where2comm across multiple datasets, modalities, and settings, including communication volume, multi-round allocation, and qualitative collaboration outcomes.

  • Experimental settings: The experiments cover camera-only and LiDAR-based 3D object detection with detector implementations based on CADDN, MotionNet, CenterNet, and PointPillar.Training configurations differ across OPV2V, CoPerception-UAVs, and LiDAR-based tasks.
  • Inference strategy: In two-round inference, about 20% of bandwidth activates collaboration and about 80% transmits targeted information in the next round.The strategy allocates remaining bandwidth to meet agents’ requests.
  • Metrics: Communication volume is measured in bytes on a base-2 logarithmic scale, with nonzero selected spatial elements determining transmitted feature-map size.The calculation accounts for channel dimension and float32 representation.
  • Benchmarks: The benchmark evaluation reports overall performance on CoPerception-UAVs, OPV2V, V2X-Sim, DAIR-V2X, and V2X-Sim2.0.Communication volume is reported using Equation (5), and V2X-Sim2.0 uses AP@(0.50/0.70).
  • Results: Where2comm consistently achieves significant improvements over previous methods across the evaluated benchmarks.The reported evaluation includes both the four-dataset benchmark and the updated V2X-Sim2.0 benchmark.

7.7 Visualization

Visualizations show how confidence-guided sparse communication supports collaboration, while ablations examine multi-round bandwidth allocation and qualitative detection performance.

  • OPV2V and V2X-Sim: Vehicle 2’s sparse message helps Vehicle 1 detect objects missed from its single view on OPV2V.The visualization links confidence maps, selection masks, request maps, sparse features, attention weights, and detections.
  • Qualitative results: Where2comm qualitatively outperforms state-of-the-art methods on CoPerception-UAVs, V2X-Sim, and OPV2V.Green boxes denote ground truth and red boxes denote detections.
  • UAV collaboration: Collaborative perception is presented as a way to overcome reception-field restrictions in traditional single-agent UAV perception.The point is framed in the context of UAV swarms.
  • Bandwidth allocation: Allocating more bandwidth to second and later rounds yields a better performance-bandwidth trade-off than allocating all bandwidth initially.The gain is stable across bandwidth allocation strategies because request maps target areas needing more information.

7.9 Discussion on the realistic limitations

The discussion centers on communication bandwidth as a central constraint in collaborative perception and outlines how Where2comm addresses several realistic deployment issues. It also describes the simulated UAV dataset used to evaluate collaboration under varied formations, sensors, and scenes.

  • Realistic limitations: Communication bandwidth is identified as the biggest challenge because collaborative perception is enabled and limited by highly dynamic, limited communication capacity.
  • Realistic limitations: Where2comm communicates strategically only when necessary or in informative spatial regions to reduce possible latency-related communication problems and estimation difficulty.
  • Realistic limitations: For time synchronization, the transformer-based fusion module can attentively augment features with asynchronous information, while delay-conditioned positional encoding is proposed as an extension.
  • Realistic limitations: For noisy localization, exchanging intermediate features at relatively low spatial resolution is described as relatively robust to noisy pose.
  • Evaluation setting: The CoPerception-UAVs evaluation uses simulated UAV swarms flying over diverse scenes and altitudes, with data collected through AirSim and Carla co-simulation.
  • Evaluation setting: The dataset includes discipline and dynamic swarm formations, five UAVs per swarm, multi-camera observations, semantic labels, 2D and 3D boxes, and BEV semantic maps.
Loading 2209.12836v1…