Source-linked AI summary
CERF: Communication-Efficient and Retraining-Free Collaborative Perception
Jiuwu Hao, Ziyi Ni, Liguo Sun, Yuting Wan, Yueyang Wu, Ti Xiang, Haolin Song, Pin Lv
TL;DR
Collaborative perception must overcome the communication cost and heterogeneity challenges of dense feature-map exchange. CERF exchanges compact perception results, forms the Poture virtual modality, and uses tracking and forecasting for current predictions; experiments report competitive performance with 95% lower bandwidth and retraining-free integration of unknown agents.
Problem
Dense feature-map exchange creates communication overhead and heterogeneity challenges that limit practical collaborative perception deployment.
Method
CERF constructs Poture from received perception outputs, fuses it with local BEV features, and uses Kalman-filter tracking plus motion forecasting for current predictions.
Results
CERF achieves competitive performance with a 95% reduction in communication bandwidth and accommodates unknown heterogeneous agents without additional retraining costs.
Takeaways & Limitations
CERF supports practical multi-agent collaborative perception by exchanging compact perception results and integrating unknown agents into collaboration.
Takeaways & Limitations
The framework assumes that only Nk agents participate in collaborative training, while Nu unknown agents appear only during inference.
Abstract
from arXiv · showhide
Collaborative perception shares information among multiple agents to obtain a comprehensive scene representation, enhancing the perceptual capability of individual agents. However, most existing methods rely on transmitting and fusing dense feature maps for collaboration, which incurs inevitable communication overhead and heterogeneity challenges, limiting their practicality for real-world deployment. To address these challenges, we propose CERF, a novel Communication-Efficient and Retraining-Free framework for open heterogeneous collaborative perception. In CERF, we introduce a new virtual modality (termed Poture), which is generated from the perception outputs of other agents, to augment the extracted Bird's Eye View (BEV) features of the ego agent. To mitigate transmission delays, we employ a Kalman-filter based tracker and a motion forecasting model to derive the current predictions from historical perception results. Extensive experiments demonstrate that CERF achieves performance comparable to mainstream intermediate-collaboration methods while reducing communication overhead by 95% across various downstream tasks. Furthermore, CERF enables seamless integration of unknown heterogeneous agents into the existing collaborative framework without additional retraining costs. Code is available at https://github.com/uestchjw/CERF.
1. INTRODUCTION
Collaborative perception improves individual agents’ situational awareness, but intermediate collaboration commonly transmits high-dimensional feature maps that create communication and heterogeneity challenges. CERF instead exchanges compact perception results, constructs the Poture virtual modality, and reports competitive performance with reduced bandwidth and retraining-free integration of unknown agents.
- 1. INTRODUCTION: Intermediate collaboration transmits implicit feature maps, creating substantial memory and transmission costs because these representations are high-dimensional.Prior efforts compress features or use sparse queries, but neural features remain more resource-intensive than compact perception outputs.
- 1. INTRODUCTION: Unknown agents are difficult to accommodate in existing intermediate-collaboration systems because feature maps from different agents have a semantic gap.
- 1. INTRODUCTION: CERF constructs Poture from received perception outputs and fuses it with the ego agent’s local BEV features for downstream tasks.The framework treats other agents’ perception results as complementary information rather than relying on high-dimensional neural features.
- 1. INTRODUCTION: CERF uses a Kalman-filter based tracker and motion forecasting to derive current predictions from historical messages, mitigating transmission delays.
- 1. INTRODUCTION: 95% reduction in communication bandwidth accompanies competitive performance across UAV3D and Air-Co-Pred experiments, while unknown heterogeneous agents require no additional retraining.
2. PROBLEM FORMULATION
The problem formulation considers heterogeneous agents that send and receive messages while optimizing collaborative perceptual performance under a communication budget. Only a subset participates in training, whereas unknown agents may appear during inference.
- 2. PROBLEM FORMULATION: Agents may differ in input modalities, model architectures, and downstream tasks, and each can send and receive messages.
- 2. PROBLEM FORMULATION: Only Nk agents participate in collaborative training, while Nu unknown agents appear exclusively during inference.
- 2. PROBLEM FORMULATION: The collaborative perception system optimizes perceptual performance under communication bandwidth constraints.The objective is expressed through an evaluation metric, a parameterized collaboration model, and a communication budget B.
3. METHODOLOGY
CERF replaces dense feature-map exchange with compact perception outputs that are transformed, temporally predicted, and converted into Poture to augment the ego agent’s BEV features. Confidence-aware filtering and fusion integrate this virtual modality for collaborative downstream perception.
- Each agent transmits compact perception outputs and pose information instead of dense intermediate feature maps.The receiving agent transforms detections into its own coordinate system before further processing.
- Poture is a virtual modality generated from received detections to enhance the ego agent’s local BEV features.Directly merging detections through late fusion can lose information and degrade performance.
- Historical detections are associated with a Kalman-filter tracker and projected to the current moment using constant-velocity motion forecasting.The resulting predictions provide the detections used to derive Poture and mitigate transmission delays.
- Poture encodes selected bounding-box attributes on a discretized H × W BEV grid before fusion with local BEV features.The modality uses attributes including box size, heading angle, and confidence score; the supplied passages specify the discretization and fusion role.
- Confidence-aware NMS retains high-confidence features at overlapping locations, after which Poture is concatenated with the original BEV feature along the channel dimension.The resulting fused feature map is passed onward for collaborative perception.
4. EXPERIMENTS
CERF is evaluated across 3D detection, tracking, and trajectory prediction, including open heterogeneous settings. It maintains competitive perception while using compact communication and supports unknown agents without retraining.
- Experimental settings: Experiments cover 3D object detection and tracking on UAV3D and trajectory prediction on Air-Co-Pred, using mAP, NDS, tracking metrics, IoU, and VPQ.The virtual modality uses three historical frames, K = 3.
- Performance comparison: CERF outperforms most intermediate-collaboration methods on UAV3D 3D object detection while requiring only about 1/1000 of the communication bandwidth.This bandwidth is comparable to late-fusion.
- Performance comparison: CERF achieves acceptable object tracking accuracy despite transmitting and fusing only compact perception results.
- Performance comparison: CERF reaches 0.530 IoU and 0.458 VPQ on Air-Co-Pred trajectory prediction, on par with existing state-of-the-art methods.
- Open heterogeneity evaluation: CERF incorporates informative data from unknown agents without retraining, independent of input resolution, backbone architecture, or detection threshold.Collaborative performance is primarily determined by individual perception capability.
- Ablation studies: Box size, heading angle, and confidence score improve Poture, whereas adding center location degrades performance because positions are represented by discretized BEV grids.
5. CONCLUSION
CERF is a communication-efficient and retraining-free framework for multi-UAV collaboration. It uses Poture and historical-message prediction to support compact communication, competitive performance, and integration of unknown agents.
- CERF generates Poture from compact perception results to enhance local BEV features instead of sharing dense neural features.
- Kalman filtering and motion forecasting derive current predictions from historical messages to mitigate transmission delays.
- CERF achieves competitive performance with minimal bandwidth and integrates unknown agents into collaboration without additional retraining.