Source-linked AI summary
3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection
Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, Jun Won Choi
TL;DR
Camera-LiDAR fusion is challenging because the modalities provide features in different views and projection can lose spatial information. 3D-CVF addresses this with BEV cross-view projection, gated fusion, and 3D RoI-based proposal refinement, achieving gains on KITTI and nuScenes and comparable-to-state-of-the-art KITTI accuracy.
Problem
Camera and LiDAR features occupy different view domains, while camera projection into 3D can lose useful spatial information, making effective fusion difficult.
Method
3D-CVF projects camera features into calibrated BEV maps, selectively fuses them with LiDAR features using gated attention, and performs 3D RoI-based fusion during proposal refinement.
Results
3D-CVF improves mAP by up to 1.57% on KITTI and 2.74% on nuScenes over the baseline without sensor fusion, with accuracy comparable to KITTI state-of-the-art performance.
Takeaways & Limitations
Fusion benefits are larger for lower-resolution LiDAR and distant objects, indicating that camera information can compensate for limited LiDAR resolution.
Abstract
from arXiv · showhide
In this paper, we propose a new deep architecture for fusing camera and LiDAR sensors for 3D object detection. Because the camera and LiDAR sensor signals have different characteristics and distributions, fusing these two modalities is expected to improve both the accuracy and robustness of 3D object detection. One of the challenges presented by the fusion of cameras and LiDAR is that the spatial feature maps obtained from each modality are represented by significantly different views in the camera and world coordinates; hence, it is not an easy task to combine two heterogeneous feature maps without loss of information. To address this problem, we propose a method called 3D-CVF that combines the camera and LiDAR features using the cross-view spatial feature fusion strategy. First, the method employs auto-calibrated projection, to transform the 2D camera features to a smooth spatial feature map with the highest correspondence to the LiDAR features in the bird's eye view (BEV) domain. Then, a gated feature fusion network is applied to use the spatial attention maps to mix the camera and LiDAR features appropriately according to the region. Next, camera-LiDAR feature fusion is also achieved in the subsequent proposal refinement stage. The camera feature is used from the 2D camera-view domain via 3D RoI grid pooling and fused with the BEV feature for proposal refinement. Our evaluations, conducted on the KITTI and nuScenes 3D object detection datasets demonstrate that the camera-LiDAR fusion offers significant performance gain over single modality and that the proposed 3D-CVF achieves state-of-the-art performance in the KITTI benchmark.
1 Introduction
3D-CVF addresses the difficulty of fusing camera and LiDAR features represented in different views by combining cross-view projection, gated fusion, and proposal refinement. Evaluations on KITTI and nuScenes report gains over non-fusion baselines and strong KITTI benchmark accuracy.
- Camera-only 3D detection lacks depth, while LiDAR-only detection lacks dense fine-grained shape, color, and texture information.
- Camera-LiDAR fusion is difficult because the modalities represent features in camera-view and 3D-world coordinates, and projection can lose spatial information.
- 3D-CVF projects camera features into smooth, dense BEV maps, then applies gated attention to generate joint camera-LiDAR features.
- 1.57% and 2.74% mAP gains over the baseline without sensor fusion are reported on KITTI and nuScenes, respectively.
- 3D-CVF further fuses camera and LiDAR information during proposal refinement using 3D RoI-based feature aggregation.
- Fusion gains are higher on nuScenes and for distant objects, suggesting camera information compensates for lower-resolution LiDAR.
2 Related Work
Prior work combines camera and LiDAR through sequential detection or multimodal fusion, but performance often remains limited by camera proposals or cross-view feature mismatch. 3D-CVF instead organizes fusion around BEV projection, gated feature combination, and RoI refinement.
- Camera-LiDAR fusion: Several methods generate region proposals from camera images before processing LiDAR points within the regions, making performance dependent on camera-based detection accuracy.
- Camera-LiDAR fusion: Existing sensor-fusion networks do not easily outperform LiDAR-only detectors, partly because camera and LiDAR features occupy different view domains.
- 3D-CVF architecture: 3D-CVF transforms camera-view features into BEV and fuses them with LiDAR features through a gated feature-fusion network.
3 Proposed 3D Object Detector
3D-CVF builds a two-stage camera-LiDAR detector by mapping camera features into BEV, adaptively fusing modalities, and refining proposals with 3D RoI feature aggregation.
- Architecture: The architecture contains LiDAR and camera pipelines, cross-view spatial feature mapping, gated fusion, and proposal generation and refinement.LiDAR features are produced in BEV, while camera features are extracted in the camera-view domain before cross-view mapping.
- Gated Camera-LiDAR Feature Fusion: Adaptive gated fusion generates spatial attention maps from concatenated camera and LiDAR features, then weights each modality before forming a joint feature map.The attention maps are applied through element-wise products, and the gated features are concatenated channel-wise.
- 3D RoI Fusion-based Refinement: For proposal refinement, multi-scale LiDAR and camera features are pooled within 3D RoIs, encoded separately, and fused with the joint feature map.Camera RoI grid points are projected into the camera view and encoded with PointNet, while low-level features preserve detailed object spatial information.
- Cross-View Feature Mapping: Auto-calibrated projection maps camera-view features into BEV using voxel-center projection, learned regional offsets, and interpolation of neighboring camera pixels.The resulting camera feature map is designed to be spatially smooth and aligned with LiDAR features.
- Proposal Generation and Refinement: The region proposal network predicts initial boxes from the joint feature, retaining high-objectness proposals through non-maximum suppression at an IoU threshold of 0.7.This produces the proposals subsequently used by the refinement stage.
4 Experiments
Experiments on KITTI and nuScenes evaluate 3D-CVF’s sensor-fusion gains, component contributions, runtime, and behavior across object distances. The results show consistent benefits from cross-view mapping, gated fusion, and 3D RoI refinement, especially for distant or LiDAR-difficult objects.
- KITTI results: 3D-CVF achieves up to 2.89% gains for hard KITTI objects over UberATG-MMF, the strongest fusion-based leaderboard method.It outperforms most LiDAR-based detectors, except STD at the hard level.
- KITTI results: 75ms per frame inference time adds only 25ms over the 50ms LiDAR-only baseline.The reported runtime is measured on one NVIDIA GTX 1080 Ti.
- nuScenes results: On nuScenes, camera fusion yields higher gains than on KITTI, where lower-resolution LiDAR makes camera information particularly useful for difficult objects.The nuScenes comparison uses a baseline with the same structure but without the camera pipeline.
- Ablation study: Sensor fusion provides 1.32%, 1.57%, and 1.39% gains in APEasy, APMod., and APHard over the LiDAR-only baseline.The ablation study is conducted on the KITTI validation set for the Car category.
- Ablation study: The adaptive gated fusion network boosts APEasy, APMod., and APHard by 0.54%, 0.87%, and 0.79%, respectively.It selectively weights camera and LiDAR features according to their relevance to detection.
- Ablation study: Auto-calibrated projection improves APEasy, APMod., and APHard by 0.5%, 0.06%, and 0.15%, respectively.The projection generates smooth, dense camera features in the BEV domain.
- Ablation study: 3D RoI fusion-based refinement improves APEasy, APMod., and APHard by 0.28%, 0.63%, and 0.45%, respectively.The refinement addresses insufficient spatial information in joint camera-LiDAR features.
- Distance analysis: Fusion gains are significantly higher for distant objects, with nearby-to-distant mAP differences reaching 5%.The authors attribute this pattern to sparse distant LiDAR points and compensation from the camera modality.
5 Conclusions
The proposed 3D-CVF fuses camera and LiDAR information across two detection stages. Evaluations on KITTI and nuScenes report significant fusion gains and state-of-the-art KITTI leaderboard performance.
- 5 Conclusions: 3D-CVF uses cross-view BEV mapping, adaptive gated fusion, and 3D RoI-based fusion refinement across two detection stages.The first stage creates a joint representation; the second refines proposals using pooled camera and LiDAR features.
- 5 Conclusions: Evaluations on KITTI and nuScenes confirm significant camera-LiDAR fusion gains, with 3D-CVF outperforming state-of-the-art 3D detectors on the KITTI leaderboard.