Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds

Qi Chen; Sihai Tang; Qing Yang; Song Fu

Source-linked AI summary

Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds

Qi Chen, Sihai Tang, Qing Yang, Song Fu

arXiv:1905.05265v1May 13, 2019cs.CV

TL;DR

Inaccurate detection can cause autonomous vehicles to make wrong decisions, while raw-data exchange across vehicles challenges existing network capacity. Cooper fuses aligned LiDAR point clouds from different viewpoints and applies point-cloud 3D detection. Experiments report expanded sensing, improved detection, and feasible transmission through existing vehicular networks.

Problem
Inaccurate object detection and recognition can cause autonomous vehicles to make wrong decisions, while multi-vehicle raw-data exchange creates bandwidth challenges.
Method
Cooper fuses LiDAR point clouds collected from different vehicle positions and viewpoints, using aligned data with a point-cloud 3D object detector.
Results
Cooper outperforms individual perception by extending sensing area, improving detection accuracy, and complementing object detection, while DSRC bandwidth can support point-cloud transmission.
Takeaways & Limitations
Connected vehicles can combine LiDAR data to provide information unavailable to individual vehicles and support cooperative perception through existing vehicular networks.
Takeaways & Limitations
High-level fusion is limited for this goal because previously undetected objects remain undetected after fusion.

Abstract

from arXiv · show

Autonomous vehicles may make wrong decisions due to inaccurate detection and recognition. Therefore, an intelligent vehicle can combine its own data with that of other vehicles to enhance perceptive ability, and thus improve detection accuracy and driving safety. However, multi-vehicle cooperative perception requires the integration of real world scenes and the traffic of raw sensor data exchange far exceeds the bandwidth of existing vehicular networks. To the best our knowledge, we are the first to conduct a study on raw-data level cooperative perception for enhancing the detection ability of self-driving systems. In this work, relying on LiDAR 3D point clouds, we fuse the sensor data collected from different positions and angles of connected vehicles. A point cloud based 3D object detection method is proposed to work on a diversity of aligned point clouds. Experimental results on KITTI and our collected dataset show that the proposed system outperforms perception by extending sensing area, improving detection accuracy and promoting augmented results. Most importantly, we demonstrate it is possible to transmit point clouds data for cooperative perception via existing vehicular network technologies.

I. INTRODUCTION

Isolated autonomous vehicles can make unsafe decisions when detection or recognition fails, especially with blocked, ambiguous, or weakly sensed objects. Cooperative data sharing is proposed to add redundancy, but prior fusion approaches and connectivity introduce practical limitations.

Detection risks: Detection and recognition failures can cause autonomous vehicles to make wrong decisions with potentially disastrous impacts.Single vehicles lack information redundancy from nearby vehicles, so sensor or technical errors can directly affect decisions.
Detection risks: Failures arise when objects are blocked, lie in sensor blind zones, or produce weak or missing signals because of malfunction.These conditions can lead to missed detection or poor recognition.
Cooperative response: Connected autonomous vehicles can share data to compensate for data scarcity and provide more information than isolated vehicles.The proposed motivation is constructive redundancy across multiple vehicles.
Connectivity challenges: Connectivity introduces security, privacy, and data analytics or aggregation challenges because vehicles access and share large information volumes.These are identified as challenges of adding connectivity to vehicles.
Fusion approaches: Low-level fusion combines raw sensor data, feature-level fusion combines extracted features, and high-level fusion combines detected objects.Each category has distinct advantages and disadvantages.
Fusion approaches: High-level fusion is unsuitable for previously undetected objects because it depends heavily on individual sensors and shared reference objects.An object missed by both vehicles’ detection outputs remains unavailable for object-level fusion.

C. Proposed Solution

Cooper addresses sparse point-cloud perception by sharing and fusing raw LiDAR data from connected vehicles. It aligns data across viewpoints, applies SPOD for 3D detection, and reports broader sensing and improved detection with existing network technology.

Proposed Solution: Cooper uses low-level raw-data fusion to support robust perception across autonomous vehicles.Raw sensing data can be transferred between vehicles without being affected by heterogeneous downstream processing algorithms.
Proposed Solution: The system must select shareable regional data because each autonomous vehicle generates more than 1000GB of data daily.The design addresses both the data type to share and the amount transferred to the recipient vehicle.
Data substrate: LiDAR point clouds provide spatial dimensions, native obfuscation of faces and license plates, and robust fusion across differing vehicle viewpoints.Unlike image or video fusion, point-cloud fusion does not require a clear overlapping zone.
Proposed Solution: Cooper combines sensing data from connected vehicles, maps point clouds to object positions, and uses SPOD to detect 3D objects in aligned LiDAR data.The approach is intended to discover objects that individual perception may miss.
Reported benefits: Cooper outperforms individual perception by extending sensing area and improving detection accuracy.This is stated as a principal system contribution.
Networking: Existing vehicular network technology can transmit region-of-interest LiDAR data among vehicles for cooperative perception.The contribution concerns transmitting selected LiDAR regions rather than all collected data.

II. COOPERATIVE SENSING

Cooperative sensing lets connected vehicles combine observations to improve coverage and detection, including information from regions hidden from one vehicle. Its feasibility depends on network bandwidth and latency.

Benefits of Sharing: Cooperative sensing is framed as a set of challenges and benefits that accompanies progress toward connected autonomous vehicles.The paper positions the concept within broader CAV development.
Benefits of Sharing: Cooperative sensing combines a vehicle’s sensor data with data from cooperators to enhance perceptive ability, detection accuracy, and driving safety.The framework is motivated by the possibility that multiple cars can improve detection accuracy.
Benefits of Sharing: Nearby vehicles can sense blocked regions behind obstacles that one vehicle cannot observe and provide that information to it.This creates complementary coverage across different vehicle positions.
Sharing constraints: Vehicles prefer raw data because detections from other cars are difficult to authenticate, while sharing all collected data is impractical.Bandwidth and latency must satisfy the requirements of cooperative perception.

C. Data Choice

The paper selects LiDAR point clouds for spatially informative, viewpoint-flexible cooperative fusion and reconstructs a shared scene by transforming transmitter data into the receiver’s coordinate state.

C. Data Choice: Image data supports object classification and recognition, whereas LiDAR provides location information; SPOD addresses sparse point-cloud detection.The data choice prioritizes LiDAR-based sensing for cooperative perception.
C. Data Choice: LiDAR point clouds are fused as points rather than pixels, avoiding the clear overlap required by image or video fusion.This supports fusion across vehicles observing scenes from different viewpoints.
D. Data Reconstruction: The vehicle IMU records offset information during driving, including yaw, pitch, and roll angles α, β, and γ.These angles are used to construct the rotation needed for reconstruction.
D. Data Reconstruction: R = Rz(α)Ry(β)Rx(γ) constructs the rotation matrix from rotations about the z-, y-, and x-axes.Rz(α), Ry(β), and Rx(γ) are the three basic rotation matrices used in the transform.
D. Data Reconstruction: Cooperative perception forms a new frame by unioning receiver and transmitter coordinates after transforming transmitter points into the receiver’s state.The transmitter transform uses the IMU-value difference between transmitter and receiver.

III. COOPERATIVE PERCEPTION

Cooperative perception addresses sparse, variable-density LiDAR and single-vehicle detection failures by combining sensing data from multiple vehicles.

LiDAR point clouds are difficult for accurate object detection because they are sparse and have highly variable point density.
On KITTI, VoxelNet reported average precision of 89.60% for cars, but performance dropped to 65.95% for pedestrians and 74.41% for cyclists in easy conditions.
Under hard conditions, detection average precision fell to 78.57% for cars, 56.98% for pedestrians, and 50.49% for cyclists.
These failures motivate cooperative sensing methods that use data from multiple vehicles to improve detection accuracy.
After fusion, differing LiDAR resolutions and point densities can still challenge deep neural-network detectors.

C. Architecture of SPOD

SPOD processes aligned dense and sparse LiDAR point clouds through a unified 3D detection architecture, enabling cooperative sensing across vehicles and datasets.

Architecture of SPOD: SPOD represents Cartesian LiDAR coordinates with reflection values and projects sparse, irregular point clouds onto a sphere for denser representation.
Architecture of SPOD: Its voxel feature extractor feeds voxel-wise features into voxel feature encoding, Sparse CNN, and an SSD-based prediction architecture.
Architecture of SPOD: Feature maps from Sparse CNN and the region proposal network are concatenated into one feature map for prediction.
Architecture of SPOD: The same end-to-end trainable network produces 3D detections from dense local LiDAR and low-resolution LiDAR data received from nearby vehicles.
Evaluation setup: Cooper is evaluated on KITTI and the collected T&J dataset, which provide dense and sparse point clouds, respectively.
Evaluation setup: The evaluation compares single-shot point clouds from individual vehicles with cooperative sensing that merges point clouds from nearby vehicles.

B. Evaluations on KITTI Dataset

KITTI experiments compare individual 64-beam LiDAR frames with merged frames across multiple vehicle positions, showing broader and more complete detection after cooperation.

Experimental setup: The KITTI evaluation uses consecutive raw 64-beam Velodyne point clouds from folder 2011/09/26/0009 and evaluates a 120° front-view area.
Experimental setup: Two single-shot frames collected at times t1 and t2 are merged to emulate cooperative sensing between two vehicles.
Cooperative detection: The merged frame detects 9 cars, compared with 6 cars at t1 and 6 cars at t2 individually.
Cooperative detection: Four KITTI scenarios—T-junction, stop sign, left turn, and curve—compare paired single shots and their corresponding cooperative point-cloud combinations against image-based ground truth.
Result analysis: Detection-result visualizations encode scores in cells, missing detections as X, out-of-area objects as blank cells, and near, medium, and far ranges as white, gray, and black.
Result analysis: Across the scenarios, cooperative data detect at least as many cars as individual single shots.
Result analysis: Cooper detects more cars and reports better detection accuracy because the cooperative point clouds contain no missing detections.
Dataset scope: KITTI lacks sufficient multi-vehicle cooperation scenarios, motivating construction of the T&J dataset for collaboration experiments.

D. Evaluation on T&J Dataset

Evaluation on the T&J dataset shows that fusing two vehicles’ LiDAR point clouds expands sensing coverage and detects objects missed in individual views across varied scenarios.

Cooperative sensing: Cooperative sensing detects objects present in either vehicle’s single shot, expanding the sensing area through data sharing.The fused frame detects all objects appearing in the individual shots and also includes three additional vehicles.
Scenario evaluation: Two vehicles positioned relatively far apart produce an even larger expanded detection area across all tested scenarios.The four scenarios use fusion data sampled at different distances to expose regional information differences.
Detection range: Cooperative perception detects far-, medium-, and near-distance objects, including some missed by single-shot sensing.The results indicate that data fusion complements detections limited by point-cloud scarcity, blockage, or distance.
Detection results: Fused data yields substantially more detected cars than either vehicle alone, although some cars remain undetected.This pattern is reported for Scenario 1 and is accompanied by a persistent residual detection gap.
Detection results: In congested environments, many cars difficult to detect individually appear after fusion, while performance remains robust across environments, times, and congestion levels.The authors connect this pattern to parking lots and congested junctions where surrounding vehicles constrain each car’s view.
Safety relevance: Fusion can mitigate missed detections for nearby vehicles when a speeding car ignores stop signs or runs a red light.

E. Statistical Analysis

Statistical analysis classifies detection difficulty by how many vehicles detect an object and finds modest gains for easier objects but a larger improvement for objects initially missed by both.

Difficulty categories: Objects are classified as easy, moderate, or hard according to whether both, one, or neither vehicle detects them.Easy means at least one or more vehicles detect the same object; moderate means only one clearly detects it; hard means neither detects it.
Performance improvement: 10% detection-score improvement occurs for easy objects 80% of the time.
Performance improvement: Easy and moderate objects show marginal but consistent detection-rate gains, mainly within 10% detection-score improvement.The authors attribute the smaller gains to already detailed and saturated point clouds in single scenes.
Performance improvement: 50% raw detection-score improvement occurs at worst for hard objects detected by neither vehicle.The authors state that detecting an object’s presence can support avoidance even without full recognition.

F. Fusion Robustness

The fusion method is tested under artificial GPS drift, including abnormal drift beyond the known boundary, and retains detection-score clustering similar to baseline readings.

Drift setup: GPS and IMU integration produces positional errors below 10 cm before the artificial drift tests.
Drift setup: The robustness evaluation skews both coordinates, one coordinate, and twice the maximum known GPS drift.These conditions represent normal-axis drift limits and abnormal drift beyond the boundary.
Robustness result: Skewed detection scores cluster similarly to baseline scores, with most vehicles still successfully detected except those already known to be undetected.

G. Networking Requirements

Cooper reduces raw LiDAR exchange by transmitting regions of interest and evaluates three exchange scenarios, finding that their simulated data demands remain within DSRC bandwidth capacity.

ROI exchange: ROI extraction reduces exchanged point-cloud data to hundreds of kilobytes per frame by prioritizing traffic lights, blocked areas, nearby vehicles, and driving-path free space.The strategy addresses the gap between vehicle-generated data and limited DSRC throughput.
ROI exchange: The networking analysis compares three ROI categories to balance exchanged data size against detection accuracy.The scenarios are illustrated through ROI exchange and LiDAR-volume figures at a 1 Hz sample rate.
Exchange scenarios: The most costly scenario exchanges the full LiDAR frame between laterally separated vehicles, requiring around 1.8 Mbit per frame for each car.This setting corresponds to opposing traffic without a physical buffer between vehicles.
Exchange scenarios: At junctions, vehicles exchange a minimum 120-degree driver-perspective field of view, with transaction cost additive across both vehicles.
Exchange scenarios: When a trailing car requests the leading car’s view, one-way exchange consumes the least bandwidth among the three scenarios.
Bandwidth feasibility: All three simulated exchange cases fall within DSRC bandwidth capacity in a real-world test.

H. Summary of Experiment Results

Cooper addresses the gap in multi-vehicle raw-sensor fusion for 3D object detection by aligning LiDAR point clouds from different vehicles. Experiments show improved sensing coverage and detection performance compared with individual perception.

Results: Cooper outperforms individual perception by extending sensing area, improving detection accuracy, and complementing object detection.These improvements are reported in the summary of experimental results.
Motivation: Existing state-of-the-art CNN-based detection still leaves substantial room for improvement, motivating cooperative perception.The paper positions Cooper as a response to persistent detection limitations in individual-vehicle systems.
Research gap: The work targets a gap in prior research, which had not implemented multi-vehicle raw sensor data for object detection.Earlier approaches used feature- or object-level fusion, or fused image and point-cloud data from the same vehicle.
Method: Cooper aligns shared LiDAR point clouds from different vehicle positions and angles for cooperative 3D object detection.The framework maps point clouds to corresponding object positions and applies deep-learning-based SPOD to the aligned data.
Results: Evaluations on KITTI and a collected dataset show expanded effective sensing area, critical information capture, and improved detection performance.The reported gains include scenarios involving information unavailable to individual vehicles.