PaperlayerView exact revision on arXiv

Source-linked AI summary

Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors

Eduardo Arnold, Mehrdad Dianati, Robert de Temple, Saber Fallah

arXiv:1912.12147v2cs.CVcs.LGcs.MAcs.ROstat.ML

TL;DR

Single-view sensing remains vulnerable to occlusion, restricted field of view, and low point density, motivating cooperative 3D object detection. The paper proposes early, late, and hybrid fusion using infrastructure sensors and evaluates them with a cooperative dataset for T-junctions and roundabouts. Results indicate that adding sensors improves detection, while early fusion requires more computation because it processes denser fused point clouds.

  • Problem

    Single-view multimodal sensing remains vulnerable to occlusion, restricted perception horizon, and low point density, motivating cooperative 3D object detection.

  • Method

    The paper proposes early, late, and hybrid fusion schemes using fixed-position infrastructure sensors and evaluates them with a synthesized cooperative dataset covering T-junction and roundabout scenarios.

  • Results

    Increasing the number of sensors improves detection performance, while early fusion requires more computation because fused point clouds contain more points than separately processed clouds.

  • Takeaways & Limitations

    Spatially diverse infrastructure observations increase fused point-cloud density, reduce false negatives, and support more accurate 3D bounding-box estimation.

Abstract

from arXiv · show

3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the environment as a way to mitigate these limitations. This article proposes two schemes for cooperative 3D object detection using single modality sensors. The early fusion scheme combines point clouds from multiple spatially diverse sensing points of view before detection. In contrast, the late fusion scheme fuses the independently detected bounding boxes from multiple spatially diverse sensors. We evaluate the performance of both schemes, and their hybrid combination, using a synthetic cooperative dataset created in two complex driving scenarios, a T-junction and a roundabout. The evaluation shows that the early fusion approach outperforms late fusion by a significant margin at the cost of higher communication bandwidth. The results demonstrate that cooperative perception can recall more than 95% of the objects as opposed to 30% for single-point sensing in the most challenging scenario. To provide practical insights into the deployment of such system, we report how the number of sensors and their configuration impact the detection performance of the system.

I. INTRODUCTION

The paper motivates cooperative 3D object detection as a response to occlusion, limited field-of-view, and low point density that persist with single-view sensing. It proposes early, late, and hybrid fusion schemes, a cooperative dataset, and evaluations focused on detection and deployment configuration.

  • Accurate environmental representation is central to autonomous-vehicle perception and safe operation in complex environments.
  • Single-view multimodal sensing remains vulnerable to occlusion, restricted field-of-view, and low point density at distant regions.
  • Early fusion combines raw sensor observations before detection, whereas late fusion combines independently detected 3D bounding boxes after detection.
  • The paper proposes early, late, and hybrid cooperative 3D object-detection schemes using spatially distributed infrastructure sensors.
  • A synthetic cooperative dataset using up to eight infrastructure sensors supports multi-view simultaneous 3D object detection.
  • The evaluations compare detection performance and communication costs while analyzing how sensor number and configuration affect practical deployment.

II. RELATED WORKS

The related work reviews single-sensor 3D detection models and positions this paper’s point-cloud-based cooperative approach against existing multimodal and cooperative methods.

  • A. 3D Object Detection Models: 3D detection models are categorized by input modality as monocular colour images, point clouds, or combinations of both.
  • A. 3D Object Detection Models: Monocular images lack direct depth information, whereas point clouds support more accurate object-pose estimation.
  • A. 3D Object Detection Models: Point-cloud models commonly project points into Bird-Eye-View or cylindrical coordinates before convolutional detection.
  • A. 3D Object Detection Models: Multimodal models fuse image features with lidar or projected point-cloud representations through proposal, frustum, or feature-processing strategies.
  • A. 3D Object Detection Models: The paper uses Voxelnet with point clouds to support cooperative detection while reducing transmission bandwidth and avoiding potential privacy issues.

B. Cooperative 3D Object Detection Schemes

The paper frames cooperative detection around infrastructure-based fusion and distinguishes its evaluation from prior work through urban scenarios and practical sensor-configuration analysis.

  • B. Cooperative 3D Object Detection Schemes: Prior cooperative studies examined raw point-cloud or feature-level fusion while considering detection performance, communication costs, and localization robustness.
  • B. Cooperative 3D Object Detection Schemes: This study uses a central system to fuse data from multiple infrastructure sensors and amortize sensor and processing costs through shared resources.
  • B. Cooperative 3D Object Detection Schemes: Unlike prior evaluations on KITTI scenes and a parking lot, it evaluates complex T-junction and roundabout scenarios where occlusion is most severe.
  • B. Cooperative 3D Object Detection Schemes: The study analyzes sensor number, pose, and overlapping field-of-view as practical configuration factors for deployment.
  • B. Cooperative 3D Object Detection Schemes: Early fusion concatenates point clouds from calibrated infrastructure sensors at a central fusion system before producing and disseminating 3D bounding boxes.

B. Data Preprocessing

Each sensor’s point cloud is transformed into a shared global coordinate system, filtered to the detection area, and either concatenated for centralized detection or processed through the cooperative pipeline.

  • Coordinate transformation: Sensor points are mapped from local coordinates to a global coordinate system using each sensor’s inverse extrinsic matrix.The extrinsic matrix decomposes into a rotation matrix and translation vector.
  • Coordinate transformation: Calibration accuracy limits point-cloud alignment because sensor position and orientation errors can produce false positives and missed detections.Fixed infrastructure sensors avoid localization errors associated with mobile nodes in this system model.
  • Point-cloud filtering: Points outside the specified detection area and above 4m are removed before detection.The filtering step excludes points considered irrelevant to the target detection region.
  • Early fusion pipeline: In early fusion, processed point clouds from n sensors are transmitted to a central system, concatenated, and fed into the 3D detector.The resulting 3D bounding boxes are disseminated to nearby vehicles.

D. Late Fusion Scheme

Late fusion independently detects objects at each sensor and combines the resulting bounding boxes centrally, reducing raw-data sharing but retaining failures from individual views.

  • Late fusion pipeline: Each sensor independently processes its point cloud and transmits detected 3D bounding boxes to a central fusion system.The central system combines the sensor-level object lists into one list.
  • Late fusion pipeline: Non-Maximum Suppression removes the lower-confidence box when overlapping detections exceed an Intersection Over Union threshold.This addresses duplicate detections when an object appears in multiple sensor fields of view.
  • Late fusion limitation: Late fusion cannot recover an object that no individual sensor detects because of occlusion or low point density.The overall system depends on at least one sensor producing a detection for that object.
  • Hybrid and detection architecture: The hybrid scheme combines late- and early-fusion information as an intermediate design intended to improve detection likelihood without a drastic communication-cost increase.The paper’s detector is based on Voxelnet, with feature learning, convolutional middle layers, and a Region Proposal Network.
  • Hybrid and detection architecture: Voxelnet’s feature-learning network converts point clouds into fixed-sized representations for convolutional processing.Its convolutional middle layers add spatial context, while the Region Proposal Network produces confidence and bounding-box regression outputs.

IV. DATASET

The study introduces a synthetic cooperative dataset for infrastructure-camera perception in T-junction and roundabout scenarios, with multi-camera depth data, ground truth, and controlled scene coverage.

  • Dataset motivation: The dataset addresses the lack of a publicly available cooperative 3D object-detection dataset suitable for dynamic multi-sensor evaluation.Earlier simulated approaches used two time-separated views from the same vehicle and were limited to mostly static objects.
  • Dataset construction: The dataset is generated in CARLA to simulate complex driving scenarios and provide accurate ground-truth data for training and evaluation.The authors plan to compare this synthetic data with realistic outdoor-track datasets in future work.
  • Dataset construction: Fixed roadside cameras provide 400 x 300 RGB and depth images with a 90-degree horizontal field of view in the T-junction and roundabout scenes.The T-junction uses six infrastructure cameras, while the roundabout uses seven cameras positioned to cover the scene.
  • Dataset contents: The dataset contains four collections with 4000 training and 1000 test frames for the T-junction, plus equal training and test counts for the roundabout.Each frame synchronizes all camera images at one instant and includes object position, orientation, size, and class annotations.
  • Dataset contents: Vehicles, cyclists or motorcyclists, and pedestrians are represented, with a maximum of 30 objects per frame and spawning probabilities of 0.6, 0.2, and 0.2.Cyclists and motorcyclists are not distinguished in this paper.
  • Point-cloud generation: Depth images are converted into 3D point clouds using a pinhole camera model, with added AWGN depth noise having σ = 0.015m.The image resolution is downsampled to 200 x 150 pixels, yielding 30000 3D points per camera.

V. TRAINING PROCESS

The training process uses scenario-specific Voxelnet-based detectors trained on fused multi-sensor point clouds, with augmentation and adjusted spatial resolution for the larger roundabout scene.

  • Training setup: One 3D object-detection model is trained for each scenario using fused point clouds from multiple sensors.Training uses SGD for 30 epochs with learning rate 10^-3 and momentum 0.9.
  • Scenario-specific configuration: The T-junction model uses voxel size (0.2, 0.2, 0.4)m and anchor stride 0.4m along the X and Y dimensions.These settings match the referenced Voxelnet configuration.
  • Scenario-specific configuration: The roundabout model doubles X-Y voxel dimensions to (0.4, 0.4, 0.4)m and uses anchor stride 0.8m because the larger scene would otherwise exceed GPU memory.The roundabout area is approximately three times larger than the T-junction area.
  • Detection targets: Models are trained to detect vehicles, while pedestrian and cyclist samples are retained to encourage learning of distinct vehicle features and reduce over-fitting.Cyclists and motorcyclists are treated as one dataset category.
  • Data augmentation: Ground-truth boxes are randomly rotated uniformly between −18 and 18 degrees to improve angle-estimation generalization.Rotating the whole point cloud did not produce a significant performance gain.

VI. PERFORMANCE EVALUATION

The evaluation compares cooperative 3D object-detection schemes in T-junction and roundabout scenarios using detection, communication-cost, and computation-time measures. Detection quality is characterized with IOU, precision, recall, and AP3D.

  • Evaluation setup: The system is evaluated on an independent test dataset for both T-junction and roundabout scenarios.The comparison covers detection performance, communication cost, and computation time.
  • Evaluation metrics: Four object-detection metrics are used: IOU, precision, recall, and average precision.Communication cost is additionally measured as average data exchanged per sensor per frame in kilobits.
  • Evaluation metrics: IOU measures spatial similarity between estimated and ground-truth 3D bounding boxes, incorporating location, size, and orientation.Its value ranges from 0 for non-intersecting boxes to 1 for equal boxes.
  • Evaluation metrics: Precision is the ratio of matched estimated boxes to all estimated boxes, while recall is the ratio of matched estimated boxes to all ground-truth boxes.Both metrics depend on the matching and confidence thresholds and exhibit a precision-recall trade-off.
  • Evaluation metrics: AP3D summarizes precision across multiple recall levels and is reported for varying IOU thresholds as AP3D @ IOU κ.The interpolated precision curve is used to obtain the averaged value.

B. Comparative evaluation of fusion schemes

The comparative experiment evaluates early, late, and hybrid fusion across two driving scenarios. Early fusion provides the strongest detection performance, while its higher communication and computation demands create deployment trade-offs.

  • Comparative evaluation of fusion schemes: 20% higher detection performance in the T-junction and 18% in the roundabout distinguish early fusion from late fusion at AP3D with IOU threshold 0.7.Early fusion performs especially better at higher IOU thresholds of 0.8 and 0.9.
  • Comparative evaluation of fusion schemes: Early fusion’s detection advantage requires higher communication cost because sensors transmit raw point clouds rather than estimated objects.Hybrid fusion reduces communication cost relative to early fusion but underperforms it because omitted points are lost.
  • Comparative evaluation of fusion schemes: Early fusion requires more computation time per frame because the fused point cloud contains more points than separately processed late-fusion point clouds.The reported computation times depend on the hardware used, here an Nvidia Quadro M4000 GPU.

C. Impact of sensors pose and number on detection performance

Detection performance improves with more sensors and favorable spatial configurations, while early fusion consistently exceeds late fusion. Gains are strongest when added sensors increase coverage or point-cloud diversity.

  • Sensor number: More engaged sensors improve detection performance, with gains saturating as sensor count increases and larger areas requiring more sensors.The roundabout requires eight sensors to match the performance of six sensors in the smaller T-junction area.
  • Fusion scheme: Early fusion consistently outperforms late fusion, with the gap widening as more sensors contribute information before detection.The comparison uses AP3D at κ = 0.7 across sensor combinations.
  • Detection performance: More than 95% of ground-truth objects are detected with precision above 95% when all six T-junction or eight roundabout sensors are engaged.A single sensor detects only 30% of vehicles in the T-junction and slightly more than 30% in the roundabout.
  • Spatial diversity: Adding sensor 1 to the T-junction set (0,2,5) increases AP3D by 20% without increasing field-of-view, demonstrating a spatial-diversity benefit.The added sensor improves input information despite unchanged coverage.
  • Spatial diversity: Combining point clouds increases point density, reduces false negatives, and improves estimated box quality when sensors have overlapping coverage.The spatial-diversity experiment evaluates objects within regions jointly observed by selected sensor pairs.

E. Impact of point density on estimated bounding boxes accuracy

The analysis links object point density to bounding-box accuracy and uses this relationship to estimate the sensing required for a target accuracy in a scenario.

  • Point-density analysis: Point density is defined as the number of points representing an object and is analyzed as a factor affecting estimated-box accuracy.The experiment examines how sensor count changes fused-cloud density and detection accuracy.
  • Point-density analysis: The CDF compares object point-density distributions for the best T-junction sensor sets selected from Table II.The vertical CDF coordinate gives the fraction of objects with density at or below a specified value.
  • Bounding-box accuracy: Objects with point density below 70 points have poor bounding-box accuracy, measured by IOU.Accuracy is computed by averaging IOU within 200 equal-sized point-density bins.
  • Practical implication: Point density can predict estimated-box accuracy and identify the minimum density and sensor count needed for a specified accuracy requirement.The paper presents this as a practical implication of the density–IOU analysis.

F. Comparison with existing benchmarks

The benchmark comparison is qualified because the proposed system uses a distinct sensing strategy and evaluates all objects within its detection area. Within that scope, increasing sensor count is highly beneficial.

  • Comparison scope: Direct comparison with Voxelnet, Cooper, and F-Cooper is not fully meaningful because the proposed system uses a unique sensing strategy.The authors nevertheless compare selected single-sensor and two-sensor AP3D results.
  • Voxelnet comparison: Single-sensor AP3D is around 28% in both scenarios, below Voxelnet’s KITTI results of 81.97%, 65.46%, and 62.85% for easy, moderate, and hard categories.The authors attribute the gap to evaluating all ground-truth objects within the detection area, unlike the referenced studies.
  • Overall finding: Increasing the number of sensors is highly beneficial to detection performance in the proposed system.The conclusion connects sensor cooperation with overcoming occlusion and restricted field-of-view.
  • Cooperative perception: The system increases fused point-cloud density, reducing false negatives and improving bounding-box estimation.These benefits are reported for spatially diverse observations with overlapping fields-of-view.
  • Scope boundary: Future work must address localization estimation and bandwidth requirements when incorporating vehicle sensor data.The authors also identify more efficient fusion as a way to reduce transferred data while maintaining detection performance.
Loading 1912.12147v2…