PaperlayerView exact revision on arXiv

Source-linked AI summary

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

Su Pang, Daniel Morris, Hayder Radha

arXiv:2009.00784v1cs.CV

TL;DR

Single-modality detectors have not been reliably improved by combining camera and LiDAR information. CLOCs performs low-complexity late fusion over pre-NMS 2D and 3D candidates using learned geometric and semantic consistencies, and improves KITTI detection performance, especially at long distance.

  • Problem

    Effective camera-LiDAR fusion that improves state-of-the-art single-modality detectors has been difficult to achieve.

  • Method

    CLOCs combines pre-NMS candidates from 2D and 3D detectors and learns fusion from their geometric and semantic consistencies.

  • Results

    CLOCs improves single-modality detectors and outperforms previous state-of-the-art fusion methods on KITTI, especially for long-distance detection.

  • Takeaways & Limitations

    CLOCs provides a fast, simple fusion approach applicable to broadly chosen pre-trained 2D and 3D detectors.

Abstract

from arXiv · show

There have been significant advances in neural networks for both 3D object detection using LiDAR and 2D object detection using video. However, it has been surprisingly difficult to train networks to effectively use both modalities in a way that demonstrates gain over single-modality networks. In this paper, we propose a novel Camera-LiDAR Object Candidates (CLOCs) fusion network. CLOCs fusion provides a low-complexity multi-modal fusion framework that significantly improves the performance of single-modality detectors. CLOCs operates on the combined output candidates before Non-Maximum Suppression (NMS) of any 2D and any 3D detector, and is trained to leverage their geometric and semantic consistencies to produce more accurate final 3D and 2D detection results. Our experimental evaluation on the challenging KITTI object detection benchmark, including 3D and bird's eye view metrics, shows significant improvements, especially at long distance, over the state-of-the-art fusion based methods. At time of submission, CLOCs ranks the highest among all the fusion-based methods in the official KITTI leaderboard. We will release our code upon acceptance.

I. INTRODUCTION

CLOCs addresses the difficulty of improving single-modality detection with sensor fusion by combining camera and LiDAR detection candidates. It uses geometric and semantic consistencies in a modular, low-complexity framework that improves detection performance.

  • Motivation: 3D object detection supports spatial path planning, but LiDAR's lower input resolution harms accuracy at longer ranges.The paper contrasts LiDAR-based 3D detection with 2D visual detection and notes long-range difficulty caused by sparse input.
  • Motivation: Effective sensor fusion has been difficult because LiDAR-only methods often outperform fusion methods on the official KITTI 3D benchmark.
  • Approach: CLOCs performs late fusion by combining low-threshold 2D and 3D detection candidates before NMS.This preserves candidates that single-modality methods might mistakenly suppress.
  • Properties: CLOCs uses any pair of pre-trained 2D and 3D detectors without retraining, while adding less than 3ms latency per frame on a desktop-level GPU.
  • Approach: CLOCs exploits geometric and semantic consistencies between modalities and automatically learns probabilistic dependencies from training data.
  • Results: CLOCs improves single-modality detectors and ranked highest among fusion-based methods on the official KITTI leaderboard at submission time.

II. RELATED WORK

The related work reviews image-based, point-cloud, and multimodal approaches to 3D detection. Point-cloud methods lead in popularity, while existing multimodal methods face accuracy and representation challenges.

  • Method categories: 3D object detection methods use 2D images, 3D point clouds, or both modalities, with this paper focusing on point-cloud and multimodal categories.
  • Image-based methods: Image-based methods avoid LiDAR but generate much less accurate 3D bounding boxes than LiDAR-based techniques.
  • Point-cloud methods: Point-cloud techniques are popular because single-sensor setups avoid calibration and synchronization issues, although long-distance detection remains relatively poor.

C. 3D Detection Using Multi-modal Fusion

Multimodal 3D detection combines camera and LiDAR information through several fusion strategies, but existing approaches can lose spatial structure or underperform LiDAR-only methods. Late fusion instead combines compact detection outputs using aligned candidates and learned scoring.

  • Existing multimodal methods: Camera-LiDAR fusion is common in self-driving systems, but 2D-driven proposals can miss objects identifiable only in 3D space.
  • Existing multimodal methods: Existing deep-fusion methods may underperform LiDAR-only detectors because BEV projection loses spatial information and feature crop-and-resize can destroy modality-specific structure.
  • Existing multimodal methods: MMF remains 2∼4% worse in moderate difficulty than the best LiDAR-only detectors on the KITTI leaderboard.
  • Detection representations: 2D systems output classified axis-aligned boxes, whereas 3D systems output classified oriented boxes; KITTI simplifies 3D rotation to yaw.
  • Fusion categories: Early, deep, and late fusion combine modalities at the input, intermediate-feature, and decision levels, respectively.
  • Fusion limitations: Early fusion must handle alignment, representation, and sparsity differences that are not necessarily addressed by a shared network.
  • Fusion limitations: Deep fusion uses separate modality channels but is more complicated, and complexity alone does not establish improvement over single-modality methods.
  • Late fusion: Late fusion trains single-modality algorithms independently and requires jointly aligned and labeled data only for the final fusion step.

IV. CAMERA-LIDAR OBJECT CANDIDATES FUSION

CLOCs associates camera and LiDAR detection candidates using geometric overlap and semantic consistency. These two consistencies form the fundamental basis of its fusion network.

  • Candidate association: CLOCs seeks a single set of 3D detections and scores by associating candidates from the image and LiDAR modalities.
  • Candidate association: The association uses a geometric score together with semantic consistency to evaluate cross-modal detection candidates.
  • Candidate association: Geometric and semantic consistencies are the fundamental concepts used in the fusion network.

B. Network Architecture

CLOCs converts pre-NMS 2D and 3D detection candidates into sparse, geometrically consistent joint candidates, then processes them to produce probability scores. Its tensor encoding combines detector confidence, geometric overlap, and distance information while preserving unmatched 3D detections.

  • B. Network Architecture: CLOCs converts individual 2D and 3D candidates into a sparse tensor of consistent joint detection candidates.The architecture then processes the sparse tensor and maps it to a probability score map through maxpooling.
  • B. Network Architecture: 2D candidates encode image-plane box coordinates and confidence scores, while 3D candidates use seven box parameters and a confidence score.The 3D vector contains height, width, length, location, and yaw.
  • B. Network Architecture: For each 2D–3D pair, CLOCs encodes IoU, both detector scores, and normalized LiDAR distance, eliminating zero-IoU pairs as geometrically inconsistent.The resulting tensor has four channels per candidate pair.
  • B. Network Architecture: CLOCs uses raw 2D and 3D predictions before NMS, allowing the fusion network to reevaluate candidates that single-modality suppression might discard.This design forms a k × n × 4 tensor for k 2D and n 3D detections.
  • B. Network Architecture: The sparse representation is necessary because only a few 2D detections intersect each projected 3D detection, while pre-NMS detectors can produce many candidates.For SECOND, the paper reports 70,400 predictions per frame, making dense 1 × 1 convolution impractical.
  • B. Network Architecture: Unmatched 3D detections are retained by encoding IoU and the 2D score as −1 rather than zero.This distinguishes missing 2D detections from pairs with very small overlap or score.

C. Network Details

The fusion network applies four sequential 1 × 1 convolution layers to the non-empty elements of the sparse tensor and outputs one score per element.

  • C. Network Details: Four sequential 1 × 1 convolution layers transform 4 input channels into a single output channel.The layers are Conv2D(4,18), Conv2D(18,36), Conv2D(36,36), and Conv2D(36,1), each with stride 1.

D. Loss

CLOCs trains target classification with cross-entropy loss modified by focal loss to address severe target–background imbalance.

  • D. Loss: CLOCs uses focal-loss-modified cross entropy with α = 0.25 and γ = 2 for target classification.The modification addresses the large class imbalance between targets and background.

E. Training

The experiments evaluate CLOCs on KITTI using multiple open-source 2D and 3D detectors, focusing on the car class and comparing fusion combinations with baseline methods.

  • E. Training: Experiments focus on the car class because it has the most training and testing samples in KITTI.The evaluation uses the KITTI 3D object detection benchmark containing camera images and LiDAR point clouds.
  • E. Training: The evaluation combines RRC, MS-CNN, or Cascade R-CNN with SECOND, PointPillars, PointRCNN, or PV-RCNN.These were selected as currently available open-source detectors, although they were not the top KITTI leaderboard performers.
  • E. Training: CLOCs fusion methods outperform their baseline methods across 3D and bird’s eye view evaluation on the KITTI validation set.The comparison uses the new 40 recall positions metric for the car class.

C. Evaluation Results

On KITTI, CLOCs consistently improves corresponding single-modality baselines across 3D, BEV, pedestrian, cyclist, and distance-based evaluation, with especially large gains at 40–50m.

  • KITTI test-set results: CLOCs fusion outperforms all multimodal fusion methods on KITTI moderate and hard test-set levels at submission time.The test evaluation includes three combinations using SECOND, PointRCNN, or PV-RCNN with Cascade R-CNN; the open-source PV-RCNN baseline is distinguished from the authors’ private implementation.
  • KITTI test-set results: Fusion with Cascade R-CNN increases 3D and BEV performance substantially over SECOND, PointRCNN, and PV-RCNN baselines.This comparison is reported for the corresponding baseline methods in Table I.
  • Car validation results: CLOCs improves every evaluated 2D–3D detector combination over its corresponding 3D baseline on the KITTI car validation benchmark.The results are reported for both 3D and BEV detection, demonstrating the flexibility of the fusion approach.
  • Pedestrian and cyclist results: CLOCs improves pedestrian and cyclist detection performance by a large margin on KITTI validation across the reported 3D and BEV evaluations.The pedestrian and cyclist results use SECOND, PointPillars, and MSCNN configurations where training settings were available.
  • Distance-based analysis: CLOCs achieves higher AP than corresponding SECOND and PointRCNN baselines across all distance ranges for both 3D and BEV detection.The largest improvement occurs at 40–50m, where LiDAR point clouds are especially sparse and 2D detections provide additional information.
  • Qualitative results: Qualitative KITTI test examples show CLOCs removing false SECOND detections and recovering missed detections while retaining correct detections.The figure distinguishes false, missed, and correct detections with red, blue, and green bounding boxes respectively.

D. Score Scales

CLOCs performs better when fusion uses detector log-likelihood scores than poorly calibrated sigmoid scores, while fitted sigmoid outputs can restore comparable performance.

  • Score-scale comparison: CLOCs improves performance with detector log-likelihood scores compared with normalized sigmoid scores.The normalized score may poorly approximate target probability or precision, whereas score discrepancies in log-likelihood space can be corrected by a simple offset.
  • Score calibration: Fitted sigmoid outputs from PointRCNN make fusion work equally well with either score scale.The authors therefore generally favor using log-likelihood outputs from each single-modality detector for fusion.

E. Ablation Study

The ablation evaluates four fusion channels and focal loss, identifying geometric overlap and the original 3D confidence as especially important. Overall, CLOCs is presented as a fast, simple fusion method that improves detection, particularly at long distances.

  • Channel and loss ablation: CLOCs evaluates IoU, 2D confidence, 3D confidence, normalized LiDAR distance, and focal loss in its fusion pipeline.The ablation uses SECOND and Cascade RCNN on the moderate car class of the KITTI validation split, with AP calculated at 40 recall positions.
  • Channel and loss ablation: IoU provides geometric consistency, while 3D confidence is the most important channel because fused scores depend heavily on original 3D scores.The 2D confidence and normalized distance channels provide additional semantic and distance-related cues.
  • Channel and loss ablation: Focal loss addresses the strong positive-negative imbalance among detection candidates and improves detection accuracy.
  • Fusion design: CLOCs automatically learns fusion parameters from geometric and semantic consistencies between 2D and 3D detections.
  • Overall outcome: CLOCs outperforms previous state-of-the-art fusion methods on KITTI, with especially strong improvements for long-distance detection.
Loading 2009.00784v1…