Source-linked AI summary
STD: Sparse-to-Dense 3D Object Detector for Point Cloud
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, Jiaya Jia
TL;DR
Point-cloud 3D detection must handle sparse, unordered data while retaining localization accuracy and computational efficiency. STD addresses this with spherical-anchor proposals, PointsPool feature compaction, and a 3D IoU branch, then reports improved KITTI BEV and 3D detection performance, especially on hard examples, at 10 FPS.
Problem
Point-cloud detectors need to preserve localization information despite sparse, unordered inputs while avoiding the computational cost of processing many point-based proposals.
Method
STD uses a two-stage detector with point-based spherical-anchor proposal generation, PointsPool compact proposal features, and a 3D IoU prediction branch.
Results
STD outperforms other state-of-the-art methods on KITTI BEV and 3D detection, especially for difficult examples, with 10 FPS inference speed.
Takeaways & Limitations
The framework combines point-based localization with compact proposal representations and IoU-aware prediction for efficient KITTI 3D detection.
Abstract
from arXiv · showhide
We present a new two-stage 3D object detection framework, named sparse-to-dense 3D Object Detector (STD). The first stage is a bottom-up proposal generation network that uses raw point cloud as input to generate accurate proposals by seeding each point with a new spherical anchor. It achieves a high recall with less computation compared with prior works. Then, PointsPool is applied for generating proposal features by transforming their interior point features from sparse expression to compact representation, which saves even more computation time. In box prediction, which is the second stage, we implement a parallel intersection-over-union (IoU) branch to increase awareness of localization accuracy, resulting in further improved performance. We conduct experiments on KITTI dataset, and evaluate our method in terms of 3D object and Bird's Eye View (BEV) detection. Our method outperforms other state-of-the-arts by a large margin, especially on the hard set, with inference speed more than 10 FPS.
1. Introduction
STD is a two-stage point-cloud 3D detector that preserves point-level localization during proposal generation, compacts proposal features with PointsPool, and improves final selection with a 3D IoU branch. On KITTI, it reports stronger BEV and 3D detection performance, particularly on difficult examples, while operating at 10 FPS.
- Framework: STD uses a two-stage framework that generates proposals from point-based spherical anchors and performs final box prediction.The proposal generation module seeds points with spherical anchors; the box prediction network classifies and regresses proposals.
- Framework: The proposal stage preserves accurate location information by treating each point as an element and assigning it a spherical anchor.A PointNet++ backbone extracts semantic context and produces objectness scores for filtering anchors.
- Framework: PointsPool converts sparse proposal features into compact representations while retaining localization and context information.The layer gathers canonical coordinates and semantic features from interior points, enabling efficient CNN or fully connected processing.
- Framework: A parallel 3D IoU branch predicts overlap between proposals and ground-truth boxes to reduce inappropriate removal during post-processing.This branch improves alignment between classification scores and localization quality.
2. Related Work
Prior point-cloud 3D detection methods use multi-view, voxel, or raw-point representations, each with distinct dependencies and representation trade-offs. The related work contrasts view fusion and voxelization with point-based approaches that operate more directly on LiDAR data.
- Method Taxonomy: 3D object detection methods are grouped into multi-view, voxel-based, and point-based approaches.This taxonomy organizes the related work by how each method represents and processes point-cloud data.
- Multi-view Methods: Multi-view methods project LiDAR into views and fuse BEV, image, and front-view features for proposal generation or box refinement.MV3D and AVOD differ in whether multi-view fusion is also used during the region proposal stage.
- Voxel-based Methods: Voxel-based methods encode point clouds into voxel grids and apply learned or sparse-convolution representations for detection.VoxelNet learns voxel features with VFE layers, while SECOND uses sparse convolutions on compact representations.
- Point-based Methods: F-PointNet uses 2D detector frustums and interior points, making its performance dependent on the 2D object detector.PointRCNN instead uses the whole point cloud for proposal generation and classifies proposals from their centric-point segmentation scores.
- Point-based Methods: Point-based approaches operate directly on raw LiDAR data but must address proposal location, size, orientation, and representation challenges.The cited discussion notes that some point-based features neglect size and orientation while seeking to exploit point-cloud representation power.
3. Our Framework
STD is a two-stage point-cloud detector that generates proposals from spherical point-centered anchors, converts sparse proposal features into compact representations with PointsPool, and predicts boxes with a parallel IoU branch.
- Proposal Generation: STD uses each point to seed spherical anchors, preserving location information while producing proposals from interior point features.Spherical receptive fields use class-specific radii, including 2 meters for cars and 1 meter for pedestrians and cyclists.
- Proposal Generation: PointNet++ extracts per-point semantic context and objectness scores, while PointNet predicts anchor classification, regression offsets, and orientations.Proposal-network inputs include anchor-normalized coordinates and semantic features.
- Proposal Generation: PointsIoU assigns anchor labels by comparing points in intersecting and union regions rather than applying traditional box IoU to spherical receptive fields.This criterion is designed specifically for spherical anchors.
- Proposal Feature Generation: PointsPool gathers canonical coordinates and semantic features, voxelizes each proposal, and produces compact features for efficient fully connected box prediction.The layer is gradient-conductive, enabling end-to-end training; proposals are partitioned into 6 × 6 × 6 voxels with 35 sampled points per voxel.
- Evaluation: On KITTI, the framework is evaluated through 3D object detection AP and inference-time comparisons across box-regression architectures and post-processing methods.Table 1 compares inference time and AP on the moderate validation set, while Table 2 examines Score-NMS and IoU-NMS.
- Box Prediction: The box prediction network uses separate box-estimation and IoU-estimation branches, multiplying classification scores by predicted 3D IoU for NMS ranking.This sorting criterion is intended to reduce the discrepancy between classification confidence and localization quality.
4. Experiments
Experiments on KITTI evaluate STD across 3D and BEV detection, proposal recall, component ablations, and inference speed. STD achieves strong performance, particularly on difficult cases, while retaining fast inference.
- Evaluation setup: STD is evaluated on KITTI using AP, with IoU thresholds of 0.7 for Cars and 0.5 for Pedestrians and Cyclists.The benchmark contains Car, Pedestrian, and Cyclist categories and follows the official evaluation protocol.
- Main results: STD outperforms competing methods on Car and Cyclist test detection, especially on the hard set, using only raw point-cloud input.Against UberATG-MMF, STD improves moderate Car 3D detection by 0.88% and hard-set performance by 7.65%.
- Main results: STD remains the best LiDAR-only detector for Pedestrians, while RGB information helps distinguish sparse small objects such as pedestrians and indicators.Pedestrians contain very few 3D points, making them difficult to distinguish from other small objects in LiDAR data.
- Main results: 1.87%, 2.64%, and 3.97% are STD’s moderate Car AP gains over PointRCNN, PointPillars, and SECOND, respectively.On the hard set, the corresponding gains are 7.74%, 7.76%, and 9.86%.
- Ablation studies: Spherical receptive fields avoid the extra orientations and computation required by cuboid fields, while one-orientation cuboids reduce average recall by 1.5%.The ablation uses Average Recall at IoU threshold 0.7 and reports additional context from spherical fields as beneficial for anchor classification and regression.
- Ablation studies: Canonical coordinates increase moderate-set AP by 11.1%, while 3D semantic features provide an approximately 36.5% performance boost.These results quantify the contribution of proposal-feature design.
- Inference time: 80 ms total inference time is reported on a TitanV GPU, including 54 ms for PointNet++, 10 ms for proposal generation, 6 ms for PointsPool, and 10 ms for the second stage.The reported total corresponds to more than 10 FPS.
5. Conclusion
STD combines point-based spherical-anchor proposals, compact PointsPool representations, and a second-stage prediction design. The model performs well on 3D detection, especially for hard examples.
- Conclusion: STD uses spherical point-based anchors to generate accurate proposals without losing localization information.The first stage refines point-seeded proposals before final prediction.
- Conclusion: PointsPool converts proposal features into compact representations, combining voxel- and point-based advantages while reducing inference time.The layer supports efficient processing of proposal features.
- Conclusion: The second stage reduces incorrect removal during post-processing and further improves performance.The conclusion reports particularly strong behavior on the hard set.