Source-linked AI summary
IPOD: Intensive Point-based Object Detector for Point Cloud
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, Jiaya Jia
TL;DR
3D object detection from LiDAR must handle sparse, unordered point clouds without losing precise spatial information. IPOD generates proposals from individual raw points and extracts context-aware proposal features, achieving state-of-the-art KITTI performance, especially on difficult cases, while its reported scope includes 2D, BEV, and 3D detection.
Problem
LiDAR point clouds are sparse and unordered, while projection, voxelization, and 2D-detector-dependent pipelines can discard information or struggle with occlusion and clutter.
Method
IPOD seeds object proposals from each raw point and uses proposal features combining context information with precise point-cloud coordinates for final bounding-box inference.
Results
IPOD achieves new state-of-the-art KITTI detection performance across 2D, BEV, and 3D tasks, especially for highly occluded and crowded objects.
Takeaways & Limitations
The raw-point proposal paradigm preserves localization information and is reported to provide high recall while handling difficult cluttered and occluded scenes.
Takeaways & Limitations
Performance for cyclists is relatively weak when foreground points are difficult to select because the referenced dataset lacks a cyclist class.
Abstract
from arXiv · showhide
We present a novel 3D object detection framework, named IPOD, based on raw point cloud. It seeds object proposal for each point, which is the basic element. This paradigm provides us with high recall and high fidelity of information, leading to a suitable way to process point cloud data. We design an end-to-end trainable architecture, where features of all points within a proposal are extracted from the backbone network and achieve a proposal feature for final bounding inference. These features with both context information and precise point cloud coordinates yield improved performance. We conduct experiments on KITTI dataset, evaluating our performance in terms of 3D object detection, Bird's Eye View (BEV) detection and 2D object detection. Our method accomplishes new state-of-the-art , showing great advantage on the hard set.
1. Introduction
IPOD addresses the sparsity, disorder, and occlusion challenges of LiDAR point clouds by generating proposals directly from raw points. Its point-based design preserves information and achieves strong KITTI performance, especially on difficult cases.
- LiDAR point clouds provide precise spatial information but are sparse, unordered, and nonuniform, complicating 3D recognition.
- Existing approaches project point clouds or voxelize them, while frustum-based detection depends heavily on 2D detector results and is vulnerable to occlusion and clutter.
- IPOD seeds an object proposal from each raw point, avoiding approximation and retaining precise point-cloud information for cluttered and occluded scenes.
- The proposal-generation module addresses redundant proposals and ambiguous ground-truth assignment, then combines context and local information through proposal features for final inference.
- IPOD evaluates 2D, BEV, and 3D detection on KITTI and reports new state-of-the-art performance, particularly for highly occluded and crowded objects.
2. Related Work
Prior 3D object detectors mainly use voxel grids, multiple views, or raw-point methods. These approaches introduce representation, small-object, depth-overlap, or 2D-detector-dependence limitations.
- 3D object detection methods are broadly organized into voxel-grid, multi-view, and PointNet-based approaches.
- Voxel-grid methods encode points within fixed voxels using hand-crafted or learned representations.
- Multi-view methods combine BEV, image, and front-view features, but can struggle with small objects and multiple objects aligned in depth.
- F-PointNet uses 2D detections to form frustum proposals, crops the point cloud, segments instances, and regresses 3D boxes from positive points.
- F-PointNet’s final performance heavily relies on its 2D detector, motivating a more general raw-point design.
3. Our Framework
IPOD generates point-based proposals directly from raw point clouds, reducing redundancy and label ambiguity before extracting context-aware proposal features for bounding-box prediction.
- 3.1. Point-based Proposal Generation: Each positive point seeds proposals across multiple scales, angles, and shifts after semantic foreground filtering.Projected semantic segmentation identifies positive points, and proposals are generated around them to cover object points.
- 3.1. Point-based Proposal Generation: Around 60K initial proposals are reduced to approximately 500 effective proposals using NMS while maintaining high recall.Proposal scores sum semantic segmentation scores over interior points, and IoU is computed from BEV projections.
- 3.1. Point-based Proposal Generation: Proposal alignment resolves cases where different boxes contain identical points and therefore produce identical feature representations.The method replaces proposal sizes and centers with class-specific anchor sizes and the center of the interior-point set.
- 3.1. Point-based Proposal Generation: PointsIoU addresses label ambiguity by counting points in the intersection and union of proposal and ground-truth boxes.This criterion handles proposals that contain many ground-truth points despite having low conventional box IoU.
- 3.2. Network Architecture: The backbone processes the entire raw point cloud with PointNet++ SA and FP layers, sharing point features across proposals.For N × 4 input points, it outputs an N × C feature map containing local and global information.
- 3.2. Network Architecture: Each proposal combines sampled context features with canonized point coordinates, then predicts classification, size, center, and orientation.The proposal feature uses M = 512 sampled points, while the prediction network applies a small PointNet++ model.
4. Experiments
Experiments on KITTI evaluate IPOD across detection settings and ablations. The method improves difficult-case detection, while semantic foreground selection, PointsIoU sampling, context features, and combined proposal features materially affect performance.
- Evaluation setup: KITTI evaluation covers Car, Pedestrian, and Cyclist detection using AP, with IoU thresholds of 0.7 for Car and 0.5 for the other classes.
- Test-set results: 6.75%, 2.52%, and 4.14% improvements over F-PointNet occur on the hard set for 2D, BEV, and 3D detection, respectively.
- Test-set results: 6.12%, 1.87%, and 1.51% margins over multi-view methods occur for Pedestrian prediction on easy, moderate, and hard levels, respectively.
- Failure cases: The absence of cyclist labels in Cityscapes weakens cyclist foreground selection and produces relatively weak performance for that class.
- Ablation studies: 2D semantic segmentation is empirically the best subsampling choice, while the no-mask baseline performs notably worse than the full design.
- Ablation studies: PointsIoU sampling improves hard-case performance by 7.3%, while adding PointNet++ context features increases AP by around 15% on easy and 20% on moderate and hard sets.
- Ablation studies: Combining high-level abstraction features with canonized coordinates greatly enhances model capability and is adopted in the final structure.
5. Concluding Remarks
The method operates directly on raw points, preserving localization while combining contextual and precise shape information for proposal prediction. Experiments report strong gains on difficult, highly occluded or crowded scenes.
- The method seeds each raw point with an object proposal, preserving localization information from the point cloud.
- Proposal features combine context from a large receptive field with point coordinates that retain accurate shape information.
- The model outperforms state-of-the-art 3D detection methods in the hard set, especially in highly occluded or crowded scenes.