Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds

Detection risks

• Detection and recognition failures can cause autonomous vehicles to make wrong decisions with potentially disastrous impacts.Single vehicles lack information redundancy from nearby vehicles, so sensor or technical errors can directly affect decisions.Source: §I · p. 1
• Failures arise when objects are blocked, lie in sensor blind zones, or produce weak or missing signals because of malfunction.These conditions can lead to missed detection or poor recognition.Source: A. Motivations · p. 1

Cooperative response

• Connected autonomous vehicles can share data to compensate for data scarcity and provide more information than isolated vehicles.The proposed motivation is constructive redundancy across multiple vehicles.Source: A. Motivations · p. 1

Connectivity challenges

• Connectivity introduces security, privacy, and data analytics or aggregation challenges because vehicles access and share large information volumes.These are identified as challenges of adding connectivity to vehicles.Source: B. Limitations of Prior Work · p. 1

Fusion approaches

• Low-level fusion combines raw sensor data, feature-level fusion combines extracted features, and high-level fusion combines detected objects.Each category has distinct advantages and disadvantages.Source: B. Limitations of Prior Work · pp. 1–2
• High-level fusion is unsuitable for previously undetected objects because it depends heavily on individual sensors and shared reference objects.An object missed by both vehicles’ detection outputs remains unavailable for object-level fusion.Source: B. Limitations of Prior Work · p. 2

Proposed Solution

• Cooper uses low-level raw-data fusion to support robust perception across autonomous vehicles.Raw sensing data can be transferred between vehicles without being affected by heterogeneous downstream processing algorithms.Source: C. Proposed Solution · p. 2
• The system must select shareable regional data because each autonomous vehicle generates more than 1000GB of data daily.The design addresses both the data type to share and the amount transferred to the recipient vehicle.Source: C. Proposed Solution · p. 2

Data substrate

• LiDAR point clouds provide spatial dimensions, native obfuscation of faces and license plates, and robust fusion across differing vehicle viewpoints.Unlike image or video fusion, point-cloud fusion does not require a clear overlapping zone.Source: C. Proposed Solution · p. 2

Proposed Solution

• Cooper combines sensing data from connected vehicles, maps point clouds to object positions, and uses SPOD to detect 3D objects in aligned LiDAR data.The approach is intended to discover objects that individual perception may miss.Source: C. Proposed Solution · p. 2, D. Contributions · p. 2

Reported benefits

• Cooper outperforms individual perception by extending sensing area and improving detection accuracy.This is stated as a principal system contribution.Source: D. Contributions · p. 2

Networking

• Existing vehicular network technology can transmit region-of-interest LiDAR data among vehicles for cooperative perception.The contribution concerns transmitting selected LiDAR regions rather than all collected data.Source: D. Contributions · p. 2

Benefits of Sharing

• Cooperative sensing is framed as a set of challenges and benefits that accompanies progress toward connected autonomous vehicles.The paper positions the concept within broader CAV development.Source: §II · p. 2
• Cooperative sensing combines a vehicle’s sensor data with data from cooperators to enhance perceptive ability, detection accuracy, and driving safety.The framework is motivated by the possibility that multiple cars can improve detection accuracy.Source: A. Benefits of Sharing · pp. 2–3
• Nearby vehicles can sense blocked regions behind obstacles that one vehicle cannot observe and provide that information to it.This creates complementary coverage across different vehicle positions.Source: A. Benefits of Sharing · p. 2

Sharing constraints

• Vehicles prefer raw data because detections from other cars are difficult to authenticate, while sharing all collected data is impractical.Bandwidth and latency must satisfy the requirements of cooperative perception.Source: B. Difficulty of Sharing · p. 3

C. Data Choice

• Image data supports object classification and recognition, whereas LiDAR provides location information; SPOD addresses sparse point-cloud detection.The data choice prioritizes LiDAR-based sensing for cooperative perception.Source: C. Data Choice · p. 3
• LiDAR point clouds are fused as points rather than pixels, avoiding the clear overlap required by image or video fusion.This supports fusion across vehicles observing scenes from different viewpoints.Source: D. Data Reconstruction · p. 3

D. Data Reconstruction

• The vehicle IMU records offset information during driving, including yaw, pitch, and roll angles α, β, and γ.These angles are used to construct the rotation needed for reconstruction.Source: D. Data Reconstruction · p. 3
• R = Rz(α)Ry(β)Rx(γ) constructs the rotation matrix from rotations about the z-, y-, and x-axes.Rz(α), Ry(β), and Rx(γ) are the three basic rotation matrices used in the transform.Source: D. Data Reconstruction · p. 3
• Cooperative perception forms a new frame by unioning receiver and transmitter coordinates after transforming transmitter points into the receiver’s state.The transmitter transform uses the IMU-value difference between transmitter and receiver.Source: D. Data Reconstruction · p. 3

• LiDAR point clouds are difficult for accurate object detection because they are sparse and have highly variable point density.Source: A. Object Detection based on Point… · p. 3
• On KITTI, VoxelNet reported average precision of 89.60% for cars, but performance dropped to 65.95% for pedestrians and 74.41% for cyclists in easy conditions.Source: A. Object Detection based on Point… · pp. 3–4
• Under hard conditions, detection average precision fell to 78.57% for cars, 56.98% for pedestrians, and 50.49% for cyclists.Source: A. Object Detection based on Point… · p. 4
• These failures motivate cooperative sensing methods that use data from multiple vehicles to improve detection accuracy.Source: A. Object Detection based on Point… · p. 4
• After fusion, differing LiDAR resolutions and point densities can still challenge deep neural-network detectors.Source: B. Sparse Point-cloud Object Detection… · p. 4

Architecture of SPOD

• SPOD represents Cartesian LiDAR coordinates with reflection values and projects sparse, irregular point clouds onto a sphere for denser representation.Source: C. Architecture of SPOD · p. 4
• Its voxel feature extractor feeds voxel-wise features into voxel feature encoding, Sparse CNN, and an SSD-based prediction architecture.Source: C. Architecture of SPOD · p. 4
• Feature maps from Sparse CNN and the region proposal network are concatenated into one feature map for prediction.Source: C. Architecture of SPOD · p. 4
• The same end-to-end trainable network produces 3D detections from dense local LiDAR and low-resolution LiDAR data received from nearby vehicles.Source: C. Architecture of SPOD · p. 4

Evaluation setup

• Cooper is evaluated on KITTI and the collected T&J dataset, which provide dense and sparse point clouds, respectively.Source: §IV · p. 4, A. Datasets · p. 4
• The evaluation compares single-shot point clouds from individual vehicles with cooperative sensing that merges point clouds from nearby vehicles.Source: A. Datasets · p. 4

Experimental setup

• The KITTI evaluation uses consecutive raw 64-beam Velodyne point clouds from folder 2011/09/26/0009 and evaluates a 120° front-view area.Source: B. Evaluations on KITTI Dataset · pp. 4–5
• Two single-shot frames collected at times t1 and t2 are merged to emulate cooperative sensing between two vehicles.Source: B. Evaluations on KITTI Dataset · p. 5

Cooperative detection

• The merged frame detects 9 cars, compared with 6 cars at t1 and 6 cars at t2 individually.Source: B. Evaluations on KITTI Dataset · p. 5
• Four KITTI scenarios—T-junction, stop sign, left turn, and curve—compare paired single shots and their corresponding cooperative point-cloud combinations against image-based ground truth.Source: B. Evaluations on KITTI Dataset · p. 5

Result analysis

• Detection-result visualizations encode scores in cells, missing detections as X, out-of-area objects as blank cells, and near, medium, and far ranges as white, gray, and black.Source: B. Evaluations on KITTI Dataset · p. 5
• Across the scenarios, cooperative data detect at least as many cars as individual single shots.Source: B. Evaluations on KITTI Dataset · p. 5
• Cooper detects more cars and reports better detection accuracy because the cooperative point clouds contain no missing detections.Source: B. Evaluations on KITTI Dataset · p. 5

Dataset scope

• KITTI lacks sufficient multi-vehicle cooperation scenarios, motivating construction of the T&J dataset for collaboration experiments.Source: C. T&J Dataset · p. 6

Cooperative sensing

• Cooperative sensing detects objects present in either vehicle’s single shot, expanding the sensing area through data sharing.The fused frame detects all objects appearing in the individual shots and also includes three additional vehicles.Source: D. Evaluation on T&J Dataset · p. 6

Scenario evaluation

• Two vehicles positioned relatively far apart produce an even larger expanded detection area across all tested scenarios.The four scenarios use fusion data sampled at different distances to expose regional information differences.Source: D. Evaluation on T&J Dataset · p. 6

Detection range

• Cooperative perception detects far-, medium-, and near-distance objects, including some missed by single-shot sensing.The results indicate that data fusion complements detections limited by point-cloud scarcity, blockage, or distance.Source: D. Evaluation on T&J Dataset · p. 7

Detection results

• Fused data yields substantially more detected cars than either vehicle alone, although some cars remain undetected.This pattern is reported for Scenario 1 and is accompanied by a persistent residual detection gap.Source: D. Evaluation on T&J Dataset · p. 7
• In congested environments, many cars difficult to detect individually appear after fusion, while performance remains robust across environments, times, and congestion levels.The authors connect this pattern to parking lots and congested junctions where surrounding vehicles constrain each car’s view.Source: D. Evaluation on T&J Dataset · pp. 7–8

Safety relevance

• Fusion can mitigate missed detections for nearby vehicles when a speeding car ignores stop signs or runs a red light.Source: D. Evaluation on T&J Dataset · p. 8

Difficulty categories

• Objects are classified as easy, moderate, or hard according to whether both, one, or neither vehicle detects them.Easy means at least one or more vehicles detect the same object; moderate means only one clearly detects it; hard means neither detects it.Source: E. Statistical Analysis · p. 8

Performance improvement

• 10% detection-score improvement occurs for easy objects 80% of the time.Source: E. Statistical Analysis · p. 8
• Easy and moderate objects show marginal but consistent detection-rate gains, mainly within 10% detection-score improvement.The authors attribute the smaller gains to already detailed and saturated point clouds in single scenes.Source: E. Statistical Analysis · p. 8
• 50% raw detection-score improvement occurs at worst for hard objects detected by neither vehicle.The authors state that detecting an object’s presence can support avoidance even without full recognition.Source: E. Statistical Analysis · p. 8

Drift setup

• GPS and IMU integration produces positional errors below 10 cm before the artificial drift tests.Source: F. Fusion Robustness · p. 9
• The robustness evaluation skews both coordinates, one coordinate, and twice the maximum known GPS drift.These conditions represent normal-axis drift limits and abnormal drift beyond the boundary.Source: F. Fusion Robustness · p. 9

Robustness result

• Skewed detection scores cluster similarly to baseline scores, with most vehicles still successfully detected except those already known to be undetected.Source: F. Fusion Robustness · p. 9

ROI exchange

• ROI extraction reduces exchanged point-cloud data to hundreds of kilobytes per frame by prioritizing traffic lights, blocked areas, nearby vehicles, and driving-path free space.The strategy addresses the gap between vehicle-generated data and limited DSRC throughput.Source: G. Networking Requirements · p. 9
• The networking analysis compares three ROI categories to balance exchanged data size against detection accuracy.The scenarios are illustrated through ROI exchange and LiDAR-volume figures at a 1 Hz sample rate.Source: G. Networking Requirements · p. 9, Figure 11 · p. 9, Figure 12 · p. 10

Exchange scenarios

• The most costly scenario exchanges the full LiDAR frame between laterally separated vehicles, requiring around 1.8 Mbit per frame for each car.This setting corresponds to opposing traffic without a physical buffer between vehicles.Source: G. Networking Requirements · pp. 9–10
• At junctions, vehicles exchange a minimum 120-degree driver-perspective field of view, with transaction cost additive across both vehicles.Source: G. Networking Requirements · p. 10
• When a trailing car requests the leading car’s view, one-way exchange consumes the least bandwidth among the three scenarios.Source: G. Networking Requirements · p. 10

Bandwidth feasibility

• All three simulated exchange cases fall within DSRC bandwidth capacity in a real-world test.Source: G. Networking Requirements · p. 10

Results

• Cooper outperforms individual perception by extending sensing area, improving detection accuracy, and complementing object detection.These improvements are reported in the summary of experimental results.Source: H. Summary of Experiment Results · p. 10

Motivation

• Existing state-of-the-art CNN-based detection still leaves substantial room for improvement, motivating cooperative perception.The paper positions Cooper as a response to persistent detection limitations in individual-vehicle systems.Source: §V · p. 10

Research gap

• The work targets a gap in prior research, which had not implemented multi-vehicle raw sensor data for object detection.Earlier approaches used feature- or object-level fusion, or fused image and point-cloud data from the same vehicle.Source: §V · p. 10

Method

• Cooper aligns shared LiDAR point clouds from different vehicle positions and angles for cooperative 3D object detection.The framework maps point clouds to corresponding object positions and applies deep-learning-based SPOD to the aligned data.Source: §VI · p. 10

Results

• Evaluations on KITTI and a collected dataset show expanded effective sensing area, critical information capture, and improved detection performance.The reported gains include scenarios involving information unavailable to individual vehicles.Source: §VI · p. 10, H. Summary of Experiment Results · p. 10