Source-linked AI summary
Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion
Xiaopei Wu, Liang Peng, Honghui Yang, Liang Xie, Chenxi Huang, Chengqi Deng, Haifeng Liu, Deng Cai
TL;DR
LiDAR point-cloud sparsity limits 3D detection, while differing image and point-cloud representations make multimodal fusion difficult. SFD uses depth-completion pseudo clouds with 3D-GAF, SynAugment, and CPConv, and ranks first on the KITTI car 3D object detection leaderboard.
Problem
LiDAR point-cloud sparsity limits information in distant and occluded regions, while differing image and point-cloud representations make multimodal fusion difficult.
Method
SFD uses depth-completion pseudo point clouds, fine-grained 3D-GAF fusion, SynAugment, and CPConv for extracting 2D and 3D pseudo-cloud features.
Results
SFD ranks 1st on the KITTI car 3D object detection leaderboard.
Takeaways & Limitations
The experiments demonstrate that SFD significantly improves 3D detection accuracy.
Abstract
from arXiv · showhide
Current LiDAR-only 3D detection methods inevitably suffer from the sparsity of point clouds. Many multi-modal methods are proposed to alleviate this issue, while different representations of images and point clouds make it difficult to fuse them, resulting in suboptimal performance. In this paper, we present a novel multi-modal framework SFD (Sparse Fuse Dense), which utilizes pseudo point clouds generated from depth completion to tackle the issues mentioned above. Different from prior works, we propose a new RoI fusion strategy 3D-GAF (3D Grid-wise Attentive Fusion) to make fuller use of information from different types of point clouds. Specifically, 3D-GAF fuses 3D RoI features from the couple of point clouds in a grid-wise attentive way, which is more fine-grained and more precise. In addition, we propose a SynAugment (Synchronized Augmentation) to enable our multi-modal framework to utilize all data augmentation approaches tailored to LiDAR-only methods. Lastly, we customize an effective and efficient feature extractor CPConv (Color Point Convolution) for pseudo point clouds. It can explore 2D image features and 3D geometric features of pseudo point clouds simultaneously. Our method holds the highest entry on the KITTI car 3D object detection leaderboard, demonstrating the effectiveness of our SFD. Codes are available at https://github.com/LittlePey/SFD.
1. Introduction
SFD addresses LiDAR sparsity and representation mismatch in multimodal 3D detection with dense pseudo point clouds, fine-grained fusion, synchronized augmentation, and specialized feature extraction. It reports first place on the KITTI car 3D object detection leaderboard.
- LiDAR sparsity leaves distant and occluded regions poorly represented, making precise 3D bounding boxes difficult.
- Previous RoI fusion concatenates 2D image and LiDAR features, mixing object and background information while ignoring 2D–3D part correspondences.
- 3D-GAF fuses raw-point-cloud and pseudo-point-cloud RoI features in a more fine-grained manner.
- SynAugment addresses insufficient multimodal augmentation by enabling image and point-cloud representations to undergo synchronized augmentation.
- CPConv extracts both 2D image features and 3D geometric features from pseudo point clouds, using image-domain neighbor search.
- 1st place is reported on the KITTI car 3D object detection leaderboard.
2. Related Work
Related work spans single-modal LiDAR detectors, multimodal image–point-cloud detectors, and depth-completion methods that produce dense depth from sparse depth with color-image guidance.
- Single-modal 3D detectors primarily use LiDAR data and include voxel-, auxiliary-network-, and point/voxel-based approaches.
- Multimodal detectors combine images and point clouds through cascading, region-of-interest, continuous, multitask, or multisensor fusion strategies.
- Depth completion predicts a dense depth map from sparse depth guided by a color image, but few methods use it for 3D detection.
3. Sparse Fuse Dense
SFD builds pseudo point clouds from depth-completed images and combines them with raw LiDAR through synchronized augmentation, CPConv processing, and fine-grained 3D-GAF fusion.
- Preliminaries: Raw LiDAR and images are converted into pseudo clouds through sparse-depth projection, depth completion, and reprojection into LiDAR coordinates.Each pseudo point can carry 3D coordinates, RGB values, and image coordinates.
- Framework: The framework contains a LiDAR Stream for 3D RoI proposals, a Pseudo Stream for CPConv and sparse-convolution features, and a fusion head for final predictions.Two auxiliary heads regularize training and can be detached during inference.
- 3D-GAF: 3D-GAF crops both cloud types within the same 3D RoI, aligns corresponding grids, and adaptively weights each raw–pseudo grid pair.This avoids 2D image RoIs that may include other objects or background features.
- 3D-GAF: The fusion computation predicts scalar weights from concatenated raw and pseudo grid features before weighting and combining them.All RoI grid-feature pairs in a batch can be processed in parallel.
- SynAugment: SynAugment converts images to RGB-painted pseudo clouds so multi-modal training can apply LiDAR-style augmentation, while extracting image features in 3D avoids FOV-occlusion restrictions.The method addresses augmentation challenges including ground-truth sampling and local noising.
- CPConv: CPConv searches pseudo-point neighbors on the image domain and uses RoI-aware projection to extract 2D semantic and 3D geometric information efficiently.RoI-aware search projects points to their original image spaces, preventing occluded points from interfering as neighbors.
4. Experiments
Experiments on KITTI evaluate SFD against state-of-the-art methods and isolate the contributions of 3D-GAF, CPConv, and SynAugment. Results also examine detector generality, difficult object conditions, inference speed, and qualitative detection behavior.
- Comparison with State-of-the-Arts: SFD ranks first on the KITTI car 3D detection benchmark and improves Voxel-RCNN by 3.14% AP on the moderate metric.It also outperforms published SE-SSD by 2.22% and 1.07% AP on the moderate and mAP metrics, respectively.
- Comparison with State-of-the-Arts: SFD improves Voxel-RCNN by 3.02% AP on the KITTI val-set moderate metric and remains leading in BEV detection.The test and validation evaluations use KITTI car detection benchmarks with average precision computed at specified recall positions.
- Ablation Study: 3D-GAF improves AP by 0.61%, 1.10%, and 2.32% on easy, moderate, and hard levels, respectively.The ablation extracts pseudo RoI features by voxelizing pseudo clouds and applying 3D sparse convolutions.
- Ablation Study: CPConv raises moderate AP to 88.56%, a 1.99% AP improvement over the preceding configuration.The experiment uses CPConv to extract richer pseudo-cloud features after applying 3D-GAF.
- Ablation Study: SynAugment enables LiDAR-specific augmentations in the multimodal framework, and removing them causes a drastic performance drop.The evaluated augmentations include gt-sampling, local noising, and global scaling.
- Additional Analysis: SFD improves different LiDAR-only detectors significantly, with the largest gains for distant and heavily occluded objects.Qualitative examples attribute these gains to richer pseudo-cloud geometric and semantic information and improved separation of visually similar raw-cloud regions.
5. Conclusion
SFD is a multi-modal framework for high-quality 3D detection that combines fine-grained RoI fusion, synchronized augmentation, and pseudo-cloud feature extraction. Experiments show that the approach significantly improves detection accuracy.
- SFD is proposed as a multi-modal framework for high-quality 3D detection.
- 3D-GAF fuses raw and pseudo point-cloud RoI features in a more fine-grained manner.
- SynAugment enables SFD to use data augmentation methods tailored to LiDAR-only methods.
- CPConv extracts pseudo-cloud features efficiently while combining image and geometric information.
- Experimental results demonstrate that SFD significantly improves detection accuracy.