Source-linked AI summary
EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection
Tengteng Huang, Zhe Liu, Xiwu Chen, Xiang Bai
TL;DR
3D detection must combine complementary LiDAR and camera information while resolving inconsistent classification and localization confidence. EPNet addresses these issues with point-wise LI-Fusion and a consistency-enforcing CE loss, achieving state-of-the-art results on KITTI and SUN-RGBD.
Problem
3D detection needs to exploit complementary LiDAR and camera sensors while addressing inconsistency between localization and classification confidence.
Method
EPNet uses LI-Fusion to enhance point features with semantic image features point-wise without image annotations, and CE loss to encourage confidence consistency.
Results
EPNet achieves state-of-the-art results on the KITTI and SUN-RGBD benchmark datasets.
Takeaways & Limitations
The integrated framework combines LiDAR geometry and image semantics while explicitly enforcing consistency between localization and classification confidence.
Abstract
from arXiv · showhide
In this paper, we aim at addressing two critical issues in the 3D detection task, including the exploitation of multiple sensors~(namely LiDAR point cloud and camera image), as well as the inconsistency between the localization and classification confidence. To this end, we propose a novel fusion module to enhance the point features with semantic image features in a point-wise manner without any image annotations. Besides, a consistency enforcing loss is employed to explicitly encourage the consistency of both the localization and classification confidence. We design an end-to-end learnable framework named EPNet to integrate these two components. Extensive experiments on the KITTI and SUN-RGBD datasets demonstrate the superiority of EPNet over the state-of-the-art methods. Codes and models are available at: \url{https://github.com/happinesslz/EPNet}.
1 Introduction
EPNet targets two challenges in 3D detection: fusing complementary but potentially interfering camera and LiDAR information, and aligning classification with localization confidence. It introduces LI-Fusion and CE loss within an end-to-end framework.
- Camera images provide semantic features such as color and texture, whereas LiDAR supplies depth and geometric structure but is sparse and unevenly distributed.
- Camera information can help distinguish closely packed objects with similar geometric structure, but poor-quality images may also introduce interference.
- Higher classification confidence can coincide with lower localization confidence, causing NMS to filter boxes with large overlaps but low classification confidence.
- LI-Fusion enhances LiDAR point features with corresponding semantic image features point-wise without image annotations, while suppressing interfering image features.
- CE loss encourages consistency between classification and localization confidence, and EPNet integrates it with LI-Fusion in an end-to-end framework.
2 Related Work
Prior work develops 3D detection from camera images, LiDAR, or multiple sensors, but each approach faces limitations involving depth, sparsity, annotations, or fusion complexity.
- Camera-only methods use monocular or stereo images but have difficulty generating accurate 3D bounding boxes.
- LiDAR-based detectors exploit sparse convolutions, pseudo-images, or two-stage refinement, yet sparse data challenges accurate localization.
- F-PointNet cascades 2D proposals into 3D detection, requiring extra 2D annotations and depending on the 2D detector's performance.
- Other multi-sensor methods jointly reason over camera and BEV feature maps, including ROI-level fusion approaches such as MV3D and AVOD.
3 Method
EPNet combines LiDAR geometry and camera semantics through point-wise LI-Fusion modules, then uses consistency enforcing loss to align classification and localization confidence. Its two-stream RPN and refinement network are trained jointly end-to-end.
- Framework: EPNet combines a two-stream RPN with a refinement network and trains both jointly using a multi-task objective.The two-stream RPN generates proposals, while the refinement network refines bounding boxes.
- Two-stream RPN: The geometric stream extracts LiDAR point features, while the image stream produces multi-scale semantic feature maps for point-feature enhancement.The image stream uses lightweight convolutional blocks and transposed convolutions to form a representative multi-scale feature map.
- LI-Fusion Module: LI-Fusion projects LiDAR points onto the image, samples corresponding semantic features with bilinear interpolation, and fuses them point-wise.The module contains a grid generator, image sampler, and fusion layer for establishing point-wise correspondence.
- LI-Fusion Module: A LiDAR-guided weight map adaptively controls image-feature contribution before concatenating the enhanced image and LiDAR features.The weight map is normalized with a sigmoid and is intended to suppress interfering image information caused by illumination or occlusion.
- Consistency Enforcing Loss: The consistency enforcing loss encourages classification confidence to match localization confidence so high-overlap boxes retain high classification scores during NMS.The loss jointly encourages classification confidence and localization confidence, defined by IoU, to be high.
4 Experiments
Experiments on KITTI and SUN-RGBD evaluate EPNet's fusion module, consistency loss, learned image features, and performance against alternative methods and state-of-the-art systems.
- Datasets and settings: EPNet is evaluated on KITTI and SUN-RGBD, covering outdoor autonomous-driving and indoor-scene 3D detection.KITTI results include validation and testing splits across Easy, Moderate, and Hard difficulty levels.
- Ablation study: Adding LI-Fusion improves KITTI validation 3D mAP by 1.73% over the baseline.The study also compares alternative fusion mechanisms, including simple concatenation and single-scale fusion.
- Ablation study: The learned image stream focuses on foreground objects without explicit image annotations and extracts distinct features under poor illumination.The image stream is jointly optimized using 3D-box supervision, while LI-Fusion establishes point-wise correspondence with image features.
- Ablation study: With a weight map, the method improves KITTI 3D mAP by 0.65% over the LiDAR-only baseline, while simple concatenation decreases performance by 1.08%.Removing weight-map estimation causes a 0.69% decrease, indicating adaptive selection of beneficial and harmful image features.
- Ablation study: The CE loss improves KITTI 3D mAP by 3.93% over baseline and by 1.28% over IoU loss, while retaining more positive candidates across confidence thresholds.Positive candidates are defined by overlap threshold τ, then filtered using classification-confidence threshold υ.
- Comparison with state-of-the-art methods: On KITTI test, EPNet exceeds F-PointNet, MV3D, AVOD-FPN, PC-CNN, ContFuse, and MMF by 10.37%, 17.03%, 7.71%, 6.23%, 9.85%, and 2.55% in 3D mAP.On SUN-RGBD, EPNet outperforms PointFusion, COG, F-PointNet, and VoteNet by 15.7%, 12.2%, 5.8%, and 2.1%, respectively.
5 Conclusion
EPNet combines a two-stream RPN, refinement network, LI-Fusion module, and CE loss for end-to-end 3D object detection. The authors report that extensive experiments validated both LI-Fusion and CE loss.
- EPNet consists of a two-stream RPN and a refinement network for 3D object detection.
- LI-Fusion jointly reasons over LiDAR point clouds and camera images while enhancing point features with semantic image features.
- The CE loss explicitly encourages consistency between localization and classification confidence.
- Extensive experiments validated the effectiveness of both the LI-Fusion module and the CE loss.
A More Qualitative Results
The paper presents additional qualitative results on KITTI and SUN-RGBD, followed by analyses of LI-Fusion's effect on 3D object detection.
- Additional qualitative results are presented on the KITTI and SUN-RGBD datasets.
- The qualitative analyses examine the effect of the LI-Fusion module in 3D object detection.
A.1 KITTI Dataset
On the KITTI validation set, EPNet qualitatively detects objects accurately, including crowded cars and distant cars that are difficult to recognize in camera images.
- EPNet produces precise boxes when multiple cars are crowded together.
- EPNet captures far-away cars well despite their difficulty in camera images.
- Figure 6 pairs camera images with representative LiDAR views, highlighting ground-truth boxes in green and detected boxes in blue.
A.2 SUN-RGBD Dataset
On the SUN-RGBD test set, EPNet qualitatively detects varied object categories and scales, including crowded scenes with many chairs.
- EPNet detects large and small objects across multiple categories and significant scale variations.
- EPNet outputs precise bounding boxes in crowded scenes containing many chairs.
- Figure 7 pairs camera images with representative LiDAR views, highlighting ground-truth boxes in green and detected boxes in blue.
A.3 Analysis on the LI-Fusion Module
The LI-Fusion module combines camera-image semantics with LiDAR geometry to improve 3D detections. Qualitative comparisons show more precise bounding boxes when the module is included.
- LI-Fusion combines camera-image semantic information with geometric information from LiDAR point clouds.The module is evaluated through qualitative analysis of its effect on detection.
- Removing LI-Fusion produces less precise bounding boxes than EPNet on the SUN-RGBD test set.The comparison is presented in Fig. 8.
- Edge and color cues help distinguish objects from neighboring environments, including desks, beds, and dressers.
A.4 Visualization for Images with Varying Illumination
The illumination transformation generates darkened and lightened images to simulate underexposure and overexposure. Under severe illumination conditions, LI-Fusion still enhances point features and improves detection performance by selecting beneficial features and suppressing harmful ones.
- Darkened and lightened images simulate underexposure and overexposure cases in real scenes.These examples are generated through an illumination transformation function.
- Even under severe illumination conditions, LI-Fusion enhances point features and improves detection performance.
- LI-Fusion adaptively selects beneficial image features while suppressing harmful features.