PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

Yushan Han, Hui Zhang, Huifang Li, Yi Jin, Congyan Lang, Yidong Li

arXiv:2301.06262v4cs.CV

TL;DR

Individual perception faces occlusion, distant-object sensing limits, and sensor noise, while collaborative perception uses multiple agents to address these issues. This paper surveys collaboration schemes, methods, datasets, benchmark results, and practical challenges. It organizes methods around ideal-scenario efficiency and performance and real-world robustness and safety, while comparing large-scale datasets and existing results.

  • Problem

    Existing reviews often focus on specific application issues or perception tasks and do not fully cover recent methods and large public collaborative perception datasets.

  • Method

    The paper systematically surveys collaboration schemes, methods for ideal and real-world scenarios, large-scale datasets, benchmark comparisons, and practical challenges.

  • Results

    The survey organizes methods around collaboration efficiency and performance in ideal scenarios and robustness and safety in real-world applications.

  • Takeaways & Limitations

    The review provides a consolidated view of collaborative perception methods, datasets, performance comparisons, and practical implementation issues.

Abstract

from arXiv · show

Collaborative perception is essential to address occlusion and sensor failure issues in autonomous driving. In recent years, theoretical and experimental investigations of novel works for collaborative perception have increased tremendously. So far, however, few reviews have focused on systematical collaboration modules and large-scale collaborative perception datasets. This work reviews recent achievements in this field to bridge this gap and motivate future research. We start with a brief overview of collaboration schemes. After that, we systematically summarize the collaborative perception methods for ideal scenarios and real-world issues. The former focuses on collaboration modules and efficiency, and the latter is devoted to addressing the problems in actual application. Furthermore, we present large-scale public datasets and summarize quantitative results on these benchmarks. Finally, we highlight gaps and overlook challenges between current academic research and real-world applications. The project page is https://github.com/CatOneTwo/Collaborative-Perception-in-Autonomous-Driving

I. INTRODUCTION

Individual perception is limited by occlusion, distant sensing, and sensor noise, motivating collaborative perception. This survey systematically reviews collaboration methods, datasets, quantitative benchmark results, and challenges for real-world deployment.

  • Motivation: Individual perception is limited by occlusion, distant objects, and sensor noise when vehicles sense their surroundings.Collaborative perception uses information shared among multiple agents to address these limitations.
  • Motivation: Collaborative perception expands an ego vehicle’s field of view and can detect distant and occluded objects while improving dense-area detection accuracy.
  • Survey scope: The survey systematically covers collaboration modules for ideal scenarios and solutions for real-world issues, emphasizing efficiency and performance versus robustness and safety.
  • Survey scope: The paper tracks newer application problems, state-of-the-art frameworks, and large public datasets that earlier reviews did not fully cover.
  • Survey scope: The review compares large-scale collaborative perception datasets and existing benchmark performance, then discusses challenges and future directions.

II. COLLABORATION SCHEME

Collaborative perception is organized by where information is shared in the perception pipeline: early raw-data fusion, intermediate feature aggregation, or late output fusion. These schemes trade perception performance, bandwidth, computation, and simplicity differently.

  • Collaboration scheme: The perception pipeline comprises input, base network, feature, perception head, and output stages, defining three collaboration locations.
  • Early Collaboration: Early collaboration transmits and fuses raw sensor data at the network input, providing comprehensive information but requiring high bandwidth.Its bandwidth demand makes real-time edge computing challenging.
  • Intermediate Collaboration: Intermediate collaboration transfers deep semantic features for ego-vehicle fusion and is popular because it balances performance and bandwidth with greater flexibility.Feature extraction can cause information loss and unnecessary redundancy.
  • Late Collaboration: Late collaboration merges individually generated outputs after spatial transformation and postprocessing, making it simpler and more bandwidth-economic.Its outputs may be noisy and incomplete.
  • Collaboration scheme: A systematic taxonomy is needed because collaborative perception methods use different frameworks and objectives, including ideal-scenario and real-application settings.

A. Methods for Ideal Scenarios

Ideal-scenario methods organize collaboration across early, intermediate, and late stages, using modules for data, communication, feature, loss, and output fusion. Intermediate methods emphasize bandwidth-efficient feature exchange and increasingly structured fusion strategies.

  • Collaboration schemes: Three collaboration schemes place raw data fusion at input, feature fusion during processing, and output fusion at the network output.Communication mechanisms reduce transmission bandwidth, while customized loss functions guide the network toward helpful information.
  • Intermediate collaboration: Intermediate communication methods reduce transmission by selecting agents or spatial features instead of sharing complete feature maps.Who2com selects needed agents, When2com determines when to communicate, FPV-RCNN selects proposal features, and Where2comm uses confidence and agent requests.
  • Early collaboration: Early collaboration directly fuses point clouds but creates high bandwidth pressure, motivating intermediate collaboration.Cooper compresses and reconstructs received LiDAR data, whereas Coop3D spatially transforms sensor data in a central system.
  • Feature fusion: Feature fusion methods are categorized as traditional, graph-based, or attention-based approaches for aggregating received collaborative perception messages.Traditional fusion uses permutation-invariant operations such as concatenation, summation, and linear weighting.
  • Feature fusion: F-Cooper uses voxel and spatial feature fusion, CoFF weights overlapping features by similarity and area, and V2VNet applies spatial-aware graph message passing.CoFF improves F-Cooper experimentally, while V2VNet compresses, aligns, and aggregates neighboring features in overlapping fields of view.
  • Feature fusion: CoCa3D combines collaborative depth estimation with confidence-based point-wise maximum feature fusion for camera-only 3D detection.Its reported experiments indicate that collaboration enables camera-based detection to overtake LiDAR in 3D object detection.

4) Customized Loss Function:

Collaborative information can be redundant or uncertain, so customized loss functions and guidance modules are used to improve how models learn from shared features.

  • Customized loss function: Redundancy and uncertainty in shared information motivate guidance and customized loss functions for collaborative model training.Neighbor agents may provide similar information, while uncertainty reflects perception inaccuracies or sensor noise.
  • Customized loss function: CRCNet uses contrastive learning for complementarity enhancement and a separate module for redundancy minimization.The two modules guide the network to increase information gain while reducing duplicated collaborative content.
  • Customized loss function: Uncertainty estimation uses moving block bootstrap for model and data uncertainty together with a loss function that directly captures data uncertainty.Experiments report that uncertainty estimation could reduce uncertainty and improve perception.

5) Output Fusion:

Late collaboration merges multi-agent perception outputs during postprocessing, commonly using NMS to remove redundant and low-confidence predictions. Its main challenges are spatial and temporal misalignment, motivating more robust refinement strategies.

  • Output fusion: Late fusion combines multi-agent outputs at postprocessing, with NMS removing redundant and low-confidence 3D detection predictions.Late collaboration methods also face spatial and temporal misalignment.

B. Methods for Real-world Issues

Real-world collaborative perception must address localization errors, communication disruptions, and other application issues that can damage robustness and safety. Proposed methods correct pose errors, compensate for delay, and recover information lost through interruption or packet loss.

  • Real-world challenges: Real-world systems may perform worse than individual perception under localization error, communication latency or interruption, model or task discrepancies, and privacy or security issues.These issues threaten collaborative perception robustness and safety.
  • Localization errors: GPS noise and asynchronous sensing cause spatial misalignment during aggregation, with V2VNet reported as vulnerable to pose noise.Pose regression and consistency modules learn to correct noisy relative transformations before feature aggregation.
  • Localization errors: Object-level matching and pose-graph optimization provide alternative localization-error corrections for late collaboration.OptiMatch uses optimal transport for fine object matching, while CoAlign estimates poses without pose supervision.
  • Communication latency: Latency-aware methods estimate or align current information using tracking, positional encoding, historical features, and feature-attention synchronization.TCLF interpolates current infrastructure predictions, V2X-ViT uses delay-aware positional encoding, and SyncNet uses historical collaboration information.
  • Communication interruption and loss: Interruption- and packet-loss methods recover missing collaborative features from historical information or learned repair networks.V2X-INCOP adds spatial attention and curriculum learning, while LCRN uses an encoder-decoder architecture with repair loss.

3) Model & Task Discrepancies:

Collaborative perception must address model heterogeneity and task-specific representations because real deployments involve agents with different models and multiple perception tasks. Reviewed solutions include confidence- and feature-alignment methods, self-supervised scene completion, and security protections.

  • Model Discrepancies: Different agent architectures and parameters can produce unreliable fusion results, while homogeneous models are impractical for real-world deployment.
  • Model Discrepancies: Model-agnostic collaboration combines offline confidence calibration with online Promotion-Suppression Aggregation to improve robustness under model and parameter discrepancies.
  • Model Discrepancies: MPDA addresses intermediate-feature domain gaps by resizing received features and using a sparse crossdomain transformer to generate domain-invariant features.
  • Task Discrepancies: Task-specific training learns task-specific features, motivating representations that remain general and robust across multiple collaborative perception tasks.
  • Task Discrepancies: Multi-robot scene completion uses a spatiotemporal autoencoder to balance reconstruction performance and communication volume while learning more robust multi-task representations.
  • Security: The review also identifies privacy leakage during collaboration training and adversarial attacks during data transmission as security issues.

IV. DATASETS AND EVALUATION

The review surveys large-scale collaborative perception datasets and evaluates common benchmarks because conventional autonomous-driving datasets focus on individual perception. It emphasizes dataset characteristics, selected tasks, and imperfect-condition evaluation settings.

  • Dataset Motivation: KITTI, nuScenes, and Waymo focus on individual perception, whereas large-scale collaborative benchmarks have accelerated collaborative autonomous-driving research.
  • Evaluation Scope: The evaluation selects 3D object detection, 3D object tracking, and BEV semantic segmentation because collaborative datasets lack unified motion-prediction metrics and results.
  • Dataset Comparison: The review compares datasets by collection source, frames, agent type and number, modality, and supported perception tasks.
  • Dataset Characteristics: Most collaborative datasets are simulated, while only a few are collected from the real world; included datasets are open-source, large-scale, and benchmarked.
  • V2X-Sim: V2X-Sim contains 10,000 frames from 100 simulated scenes with 2–5 vehicles per scene and supports BEV detection, tracking, and segmentation.
  • OPV2V: OPV2V contains 11,464 frames and provides a realistic Culver City test set for evaluating model generalization.
  • V2XSet: V2XSet contains 11,447 frames and evaluates 3D detection and BEV segmentation under perfect and noisy test settings.

4) DAIR-V2X:

DAIR-V2X is a real-world V2I benchmark, while the surrounding detection results show that collaboration can help across datasets but remains sensitive to asynchrony and heterogeneous conditions. Specialized fusion and temporal methods address these challenges.

  • DAIR-V2X: DAIR-V2X is the first large-scale real-world V2I collaborative perception dataset, with VIC3D supporting image- and LiDAR-based object detection.
  • 3D Object Detection: On V2X-Sim, adaptive fusion methods such as V2VNet, DiscoNet, and CRCNet outperform late collaboration, whereas simple attention-based fusion can perform worse than late collaboration.
  • 3D Object Detection: Intermediate collaboration surpasses early collaboration on OPV2V and V2XSet, although V2VNet and AttFusion are weaker than F-Cooper on V2XSet.
  • 3D Object Detection: V2X-ViT performs better in heterogeneous V2X collaboration through its specially designed framework.
  • DAIR-V2X Results: On DAIR-V2X, both early and late collaboration improve vehicle detection performance.
  • DAIR-V2X Results: Asynchrony degrades collaborative perception on DAIR-V2X, while TCLF mitigates the impact by predicting vehicle positions from historical information.
  • 3D Object Detection: On V2V4Real, collaboration improves detection under synchronous and asynchronous settings, but asynchronous performance degrades; V2X-ViT and CoBEVT remain relatively stable.

2) 3D Object Tracking:

The review defines collaborative tracking around maintaining object identities and locations across frames, then compares tracking results on V2V4Real and V2X-Sim using established multi-object tracking metrics.

  • Problem Definition: Multiple object tracking maintains object identities and locations across frames and supports autonomous-vehicle decision-making.
  • Problem Definition: The review focuses primarily on detection-based tracking rather than joint detection and tracking.
  • Evaluation Metrics: Tracking evaluation uses MOTA, MOTP, AMOTA, and AMOTP, with AMOTA and AMOTP averaging MOTA and MOTP across recall thresholds.
  • Quantitative Results: The review summarizes multi-object tracking results from the V2V4Real and V2X-Sim benchmarks.
  • Quantitative Results: On V2V4Real, CoBEVT achieves the best tracking performance using its fused axial attention module.
  • Quantitative Results: On V2X-Sim, DiscoNet outperforms other intermediate methods with matrix-valued weights and knowledge distillation.

3) BEV Semantic Segmentation:

BEV semantic segmentation uses multi-agent sensing to support semantic scene understanding, with IoU as its common evaluation metric. On reported benchmarks, collaboration is effective, while performance varies by method, object category, and dataset.

  • BEV semantic segmentation predicts a rasterized map of surrounding semantics from LiDAR points and multiple cameras under the BEV view.
  • Intersection over Union (IoU) measures overlap between predicted maps and ground-truth map-view labels.
  • Collaboration improves BEV semantic segmentation, but CoBEVT outperforms novel methods on OPV2V by targeting multi-view multi-agent fusion.
  • On V2X-Sim, regularly shaped objects such as vehicles are easier to identify than other categories.
  • V2VNet and DiscoNet outperform early collaboration for pedestrians, sidewalks, and terrain on V2X-Sim.
  • Real-world collaborative perception must address transmission and computing efficiency because both contribute substantially to inference time.

B. Collaborative Perception in Complex Scenes

Collaborative perception still faces difficult real-world conditions, including complex scenes, privacy constraints, and costly annotation requirements. The survey identifies data collection, multimodal and spatiotemporal fusion, federated learning, and weaker supervision as directions for addressing these issues.

  • B. Collaborative Perception in Complex Scenes: Existing large-scale datasets mainly cover common scenarios, leaving bad weather, highways, and distant or small objects underrepresented.
  • B. Collaborative Perception in Complex Scenes: Complex scenes can degrade sensor data through lighting or distance effects and introduce severe spatiotemporal inconsistencies.
  • B. Collaborative Perception in Complex Scenes: The survey calls for collaborative data collection in complex environments and methods tailored to varied challenging scenarios.
  • B. Collaborative Perception in Complex Scenes: Multi-sensor fusion can compensate for weather- and distance-related data-quality effects, while virtual point-cloud generation supports long-range object prediction.
  • B. Collaborative Perception in Complex Scenes: Federated learning enables vehicles to share perception models without direct data exchange, preserving data privacy while supporting distributed learning.
  • B. Collaborative Perception in Complex Scenes: Existing federated-learning methods for connected automated vehicles focus on individual perception, motivating combinations with privacy-preserving data exchange.
  • B. Collaborative Perception in Complex Scenes: Collaborative perception training depends heavily on labor-intensive, time-consuming full annotations, motivating weakly supervised learning and domain adaptation.
  • The paper surveys methods, large-scale datasets, benchmark comparisons, and practical implementation challenges across collaborative perception.
Loading 2301.06262v4…