PaperlayerView exact revision on arXiv

Source-linked AI summary

PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module

Liang Xie, Chao Xiang, Zhengxu Yu, Guodong Xu, Zheng Yang, Deng Cai, Xiaofei He

arXiv:1911.06084v3cs.CV

TL;DR

LIDAR point clouds lack semantic richness and become sparse, while existing BEV- or voxel-based fusion can lose precision. The paper introduces PACF for direct point-wise fusion and builds PI-RCNN around joint image segmentation and 3D detection, achieving state-of-the-art results on KITTI’s 3D AP benchmark.

  • Problem

    LIDAR lacks semantic information and suffers sparsity, while BEV-based multi-sensor fusion loses precision during neighbor search and fusion.

  • Method

    PACF applies continuous convolution directly to raw 3D points with Point-Pooling and Attentive Aggregation, while PI-RCNN combines image segmentation and point-based 3D detection.

  • Results

    PI-RCNN achieves state-of-the-art results on the KITTI 3D Detection benchmark, including state-of-the-art 3D AP.

  • Takeaways & Limitations

    The framework provides an effective multi-sensor, multi-task approach for supplementing LIDAR point features with image semantics in 3D detection.

  • Takeaways & Limitations

    Training omits GT-AUG to preserve correct correspondence between LIDAR points and image pixels, unlike most LIDAR-only 3D detection algorithms.

Abstract

from arXiv · show

LIDAR point clouds and RGB-images are both extremely essential for 3D object detection. So many state-of-the-art 3D detection algorithms dedicate in fusing these two types of data effectively. However, their fusion methods based on Birds Eye View (BEV) or voxel format are not accurate. In this paper, we propose a novel fusion approach named Point-based Attentive Cont-conv Fusion(PACF) module, which fuses multi-sensor features directly on 3D points. Except for continuous convolution, we additionally add a Point-Pooling and an Attentive Aggregation to make the fused features more expressive. Moreover, based on the PACF module, we propose a 3D multi-sensor multi-task network called Pointcloud-Image RCNN(PI-RCNN as brief), which handles the image segmentation and 3D object detection tasks. PI-RCNN employs a segmentation sub-network to extract full-resolution semantic feature maps from images and then fuses the multi-sensor features via powerful PACF module. Beneficial from the effectiveness of the PACF module and the expressive semantic features from the segmentation module, PI-RCNN can improve much in 3D object detection. We demonstrate the effectiveness of the PACF module and PI-RCNN on the KITTI 3D Detection benchmark, and our method can achieve state-of-the-art on the metric of 3D AP.

Introduction

LIDAR provides 3D structure but lacks semantic information and becomes sparse, especially for distant objects. PI-RCNN addresses these limitations with point-based multi-sensor fusion and joint image segmentation and 3D detection.

  • Motivation: LIDAR lacks sufficient semantic information and suffers point sparsity, making confusing scenes and distant objects difficult to recognize.These challenges are illustrated by vehicles blending into background structure and receiving only a few points when far away.
  • Prior approaches: Existing multi-sensor methods combine image and point-cloud data through BEV maps, frustums, or continuous convolution.The cited approaches merge image and BEV features, use 2D-detection frustums, or apply continuous convolution for fusion.
  • Research gap: BEV-based fusion loses precision because it quantizes the 3D world into a pseudo-image, making neighbor search and feature fusion less accurate.This limitation is also associated with mismatched RGB-image and bird’s-eye-view perspectives.
  • Proposed approach: PI-RCNN combines an image segmentation sub-network with a point-based 3D detection sub-network, bridged by PACF.The segmentation network produces full-resolution image outputs, while the detector processes raw LIDAR points.
  • Results: PI-RCNN achieves state-of-the-art 3D AP on the KITTI 3D Detection benchmark.The paper reports extensive KITTI experiments demonstrating the approach’s effectiveness.
  • Proposed approach: PACF fuses multi-sensor features directly on raw 3D points using continuous convolution, Point-Pooling, and Attentive Aggregation.The module is designed to address shortcomings of BEV-based fusion.

3D Object Detection from Single Sensor

Single-sensor 3D detection methods primarily process LIDAR point clouds by converting them into CNN-compatible representations or operating directly on raw points.

  • Representation-based methods: Traditional CNNs cannot be applied directly to LIDAR points, so many methods convert point clouds into fixed-size pseudo-images.Common representations include BEV maps and voxels.
  • Representation-based methods: BEV and voxel representations enable standard CNN processing for 3D object detection.The passage identifies BEV and voxel formats as common paradigms for point-cloud detection.
  • Raw-point methods: Other methods use raw 3D points for point-cloud classification, point segmentation, proposal generation, and local-range refinement.PointNet++ and point-based RCNN are used in the cited raw-point detection pipeline.

3D Object Detection from Multi Sensors

Multi-sensor detectors combine RGB images with point-cloud representations, but prior fusion can be coarse, perspective-mismatched, or imprecise. PI-RCNN instead fuses image semantics directly with 3D points through PACF.

  • Prior fusion methods: Prior multi-sensor methods merge RGB-image features with BEV features, use frustums from 2D detections, or apply continuous convolution.These approaches use feature merging, frustum-based prediction, or BEV-based continuous convolution.
  • Limitations of prior methods: Direct fusion methods are coarse because image RoIs contain background noise and overlook differences between bird’s-eye-view and image perspectives.Frustum methods can also include background or other instances because of occlusion.
  • Limitations of prior methods: BEV-based continuous-convolution fusion loses precision, while claimed point-wise fusion may still operate on BEV maps rather than raw LIDAR points.The cited critique distinguishes genuine point-wise fusion from fusion performed on a BEV representation.
  • PI-RCNN architecture: PI-RCNN’s architecture extracts image segmentation features, generates 3D proposals from raw LIDAR points, fuses them point-wise with PACF, and predicts final 3D boxes.The two-stage detector uses PACF-augmented point features for final bounding-box prediction.
  • PACF module: PACF fuses raw 3D points with higher-resolution, more semantic image features and adds Point-Pooling and Attentive Aggregation over neighboring points.Attentive Aggregation uses learnable parameters to aggregate neighbor features.

PI-RCNN

PI-RCNN fuses RGB-image semantics with raw LIDAR points through the point-based PACF module, while jointly learning image segmentation and 3D detection. PACF avoids BEV quantization and strengthens fusion with point pooling and attentive aggregation.

  • PI-RCNN: PI-RCNN combines an image segmentation sub-network with a point-based 3D detection sub-network.The segmentation network produces semantic features, while the detection network generates and refines 3D proposals from raw LIDAR points.
  • PACF module: PACF fuses semantic image features with LIDAR features directly on raw 3D points rather than BEV or voxel pseudo-images.The module searches point neighbors, projects them onto image feature maps, retrieves semantic features, and applies attentive continuous convolution.
  • PACF module: Point-Pooling and Attentive Aggregation make PACF fused features more expressive by extracting and aggregating information from K neighboring points.Point-Pooling uses max-pooling in practice, while another MLP aggregates neighbor features.
  • PACF module: PACF addresses BEV fusion imprecision by performing neighbor search, continuous convolution, and final fusion directly on 3D points.BEV quantizes 3D space into a pseudo-image, causing precision loss during neighbor search and feature fusion.
  • Semantic segmentation: Segmentation features provide pixel-level semantics and higher-resolution image maps for more accurate correspondence between LIDAR points and image pixels.PI-RCNN uses segmentation outputs as RGB semantic features and does not require instance-level segmentation supervision.
  • Fusion strategies: PI-RCNN provides two fusion strategies, V1 and V2, whose difference is the location of the fusion module.In V1, image semantics supplement 3D point features after the first detection stage.

Experiments

PI-RCNN is evaluated on KITTI using a segmentation-and-detection architecture with raw LIDAR points, RGB features, and PACF fusion. Experiments compare fusion settings, training choices, and ablations, showing gains over relevant baselines and benefits from PACF components.

  • Implementation and Training Details: PI-RCNN uses an image segmentation sub-network and a point-based 3D detection sub-network connected by PACF.The segmentation network produces full-resolution masks, while the detector takes raw LIDAR points as input.
  • Implementation and Training Details: Raw LIDAR points are used instead of BEV or voxel inputs, with 16,384 points sampled from the camera-visible region.The LIDAR region of concern is [0, 70.4] × [−40, 40] × [−1, 3], and RGB images are resized to 376 × 1248.
  • Results on KITTI Dataset: PI-RCNN surpasses previous state-of-the-art methods on testing-split 3D AP and outperforms multi-sensor detectors and PointRCNN on the validation split.The PointRCNN comparison uses a re-implementation without GT-AUG because multi-sensor fusion requires preserving point-image correspondence.
  • Ablation Study: Adding Point-Pooling and Attentive Aggregation improves feature fusion over the re-implemented PointRCNN baseline.The ablation also finds K = 3 performs best, while larger K values can introduce noise from distant points.
  • Ablation Study: PI-RCNN V1 slightly outperforms V2, suggesting fusion in the middle of the detection sub-network is better than fusion at its beginning.The proposed explanation is that early proposal generation relies mainly on LIDAR geometry, making appended image features less useful at that stage.
  • Ablation Study: Multi-class semantic segmentation produces the best image-feature results, and pretraining the segmentation sub-network improves detection performance.Pretraining uses a single foreground Car class and background supervision because the available 3D annotations provide binary segmentation labels.

Conclusion

The paper proposes PACF and PI-RCNN, a multi-sensor multi-task detector combining image segmentation with 3D detection. PI-RCNN achieves state-of-the-art results on the KITTI 3D Detection benchmark.

  • Conclusion: The paper proposes PACF and PI-RCNN, which combines image segmentation with 3D object detection.PACF is the proposed fusion module, while PI-RCNN is the corresponding multi-sensor multi-task detection network.
  • Conclusion: PI-RCNN achieves state-of-the-art results on the KITTI 3D Detection benchmark.
Loading 1911.06084v3…