Source-linked AI summary
3DSSD: Point-based 3D Single Stage Object Detector
Zetong Yang, Yanan Sun, Shu Liu, Jiaya Jia
TL;DR
Point-based single-stage 3D detection remains underexplored, while point-cloud sparsity and costly point-based processing make efficient detection difficult. 3DSSD removes upsampling and refinement, adds fusion sampling and a specialized box-prediction network, and achieves strong benchmark performance with fast inference. The method surpasses voxel-based single-stage detectors, is comparable to two-stage point-based methods, and runs at more than 25 FPS.
Problem
Point-based single-stage 3D detectors are underexplored, while point clouds’ sparsity, unordered structure, and locality sensitivity complicate direct detection.
Method
3DSSD is a lightweight point-based single-stage detector that removes FP layers and refinement, uses fusion sampling, and adds candidate generation with anchor-free center-ness prediction.
Results
3DSSD outperforms all state-of-the-art voxel-based single-stage methods, matches two-stage point-based methods, and achieves more than 25 FPS.
Takeaways & Limitations
The framework provides a point-based single-stage design balancing detection performance and inference efficiency across KITTI and nuScenes.
Takeaways & Limitations
Fusion sampling preserves foreground points but discards many negative points, which hampers classification by limiting receptive-field enlargement.
Abstract
from arXiv · showhide
Currently, there have been many kinds of voxel-based 3D single stage detectors, while point-based single stage methods are still underexplored. In this paper, we first present a lightweight and effective point-based 3D single stage object detector, named 3DSSD, achieving a good balance between accuracy and efficiency. In this paradigm, all upsampling layers and refinement stage, which are indispensable in all existing point-based methods, are abandoned to reduce the large computation cost. We novelly propose a fusion sampling strategy in downsampling process to make detection on less representative points feasible. A delicate box prediction network including a candidate generation layer, an anchor-free regression head with a 3D center-ness assignment strategy is designed to meet with our demand of accuracy and speed. Our paradigm is an elegant single stage anchor-free framework, showing great superiority to other existing methods. We evaluate 3DSSD on widely used KITTI dataset and more challenging nuScenes dataset. Our method outperforms all state-of-the-art voxel-based single stage methods by a large margin, and has comparable performance to two stage point-based methods as well, with inference speed more than 25 FPS, 2x faster than former state-of-the-art point-based methods.
1. Introduction
3DSSD addresses the difficulty of detecting objects in sparse, unordered point clouds with a lightweight point-based single-stage design. It removes costly upsampling and refinement while using fusion sampling and a specialized prediction network to retain accuracy and efficiency.
- Motivation: Point clouds are sparse, unordered, and locality sensitive, so conventional 2D convolutional detection methods cannot be directly applied.
- Related limitations: Voxel-based methods compact point clouds efficiently but lose information during voxelization and encounter performance bottlenecks.
- Motivation: Existing point-based detectors use downsampling, upsampling, proposal generation, and refinement, with FP layers and refinement consuming half of inference time.
- 3DSSD: Fusion sampling combines feature- and distance-based sampling to preserve interior foreground points, making detection on less representative points feasible.
- 3DSSD: 3DSSD removes FP layers and refinement, reducing inference cost while requiring detection directly on downsampled representative points.
- Results: 38ms per scene: 3DSSD outperforms all single-stage methods and matches state-of-the-art two-stage methods in performance at much faster speed.
2. Related Work
Prior LiDAR-only 3D detectors mainly follow voxel-based or point-based paradigms. Voxel methods compact the cloud for efficient processing, whereas point-based methods retain raw points and build predictions around them.
- Multiple-sensor detection: Multi-sensor methods such as MV3D and AVOD fuse LiDAR-derived views with image information for proposal or box prediction.
- LiDAR-only detection: Voxel-based LiDAR detectors voxelize the entire point cloud, differing mainly in how voxel features are initialized.
- LiDAR-only detection: Point-based methods take raw point clouds as input and generate predictions from individual points.
- Point-based methods: PointRCNN uses set abstraction and feature propagation, followed by region proposals and refinement for bounding-box prediction.
3. Our Framework
3DSSD removes feature propagation and refinement from point-based detection, then uses fusion sampling and a candidate-based prediction network to retain useful points while limiting computation.
- Framework motivation: 3DSSD removes FP layers and refinement, addressing major inference-time costs in existing point-based detectors.These components account for substantial computation in prior architectures, motivating a single-stage design.
- Framework motivation: D-FPS can discard all interior points from remote or sparse foreground instances, making those objects undetectable after downsampling.The problem worsens on complex datasets such as nuScenes, where background points may dominate the representative-point budget.
- Fusion sampling: F-FPS combines spatial and feature distances to preserve informative foreground points while reducing redundant sampling within instances.The balance factor λ weights the L2 XYZ distance against the L2 feature distance.
- Box prediction network: The box prediction network uses candidate generation followed by an anchor-free prediction head for regression and classification.This design uses candidate points as centers and exploits the full representative-point set for local feature extraction.
- Fusion sampling: 89.2% of nuScenes instances are preserved with 1024 representative points and λ=1, 23.3% higher than D-FPS.This result is reported as points recall for F-FPS sampling.
- Fusion sampling: Fusion sampling combines F-FPS and D-FPS to retain positive points for localization while preserving enough negative points for classification.The two point sets are sampled separately and fed together into the subsequent grouping operation.
- Box prediction network: The candidate generation layer shifts F-FPS points toward instance centers, then aggregates features from surrounding F-FPS and D-FPS representative points.The resulting candidate-point features feed the prediction head, avoiding an additional conventional SA layer.
4. Experiments
Experiments evaluate 3DSSD on KITTI and nuScenes, including benchmark comparisons, sampling and assignment ablations, and inference speed. The method achieves strong detection performance while maintaining fast inference.
- Main Results: 3DSSD outperforms voxel-based single-stage detectors on KITTI and nuScenes, with 3.61% and 5.26% gains over SECOND and PointPillars on KITTI moderate AP.On KITTI moderate AP, it also exceeds PointRCNN and Part-A^2 net by 3.93% and 1.08%, respectively.
- Sampling Ablation: Fusion sampling improves AP by 2.7% over F-FPS alone because D-FPS contributes enough negative points for larger receptive fields and more accurate classification.Sampling strategies containing F-FPS also achieve higher point recall than D-FPS alone.
- Prediction Ablation: Candidate-point shifting improves AP under different assignment strategies, while 3D center-ness assignment performs best when shifting is enabled.Shifting moves representative points toward instance centers, making corresponding instances easier to retrieve.
- Inference Time: 3DSSD requires 38ms per KITTI scene, close to SECOND at 40ms and faster than existing point-based methods.The reported timing was measured on a Titan V GPU; PointPillars remains faster with implementation optimizations such as TensorRT.
5. Conclusion
The conclusion presents 3DSSD as a lightweight point-based single-stage detector that removes costly upsampling and refinement while preserving detection performance. Its fusion sampling and prediction designs support efficiency and accuracy.
- 5. Conclusion: 3DSSD removes FP layers and refinement, using fusion sampling, candidate generation, anchor-free regression, and 3D center-ness labeling.These designs are intended to reduce computation while fully using downsampled representative points.
- 5. Conclusion: The combined designs make 3DSSD superior to existing single-stage 3D detectors in both performance and inference time.