Source-linked AI summary
Multi-Task Multi-Sensor Fusion for 3D Object Detection
Ming Liang, Bin Yang, Yun Chen, Rui Hu, Raquel Urtasun
TL;DR
3D object detection must combine complementary but imperfect sensor and task information for real-time autonomous-driving perception. The paper introduces an end-to-end multi-task, multi-sensor detector, and reports leading KITTI performance across 2D, 3D, and BEV detection while running in real time.
Problem
Cameras lack fine-grained 3D information and LiDAR becomes sparse at long range, limiting robust detection of distant or occluded objects.
Method
An end-to-end model jointly reasons about 2D and 3D detection, ground estimation, and depth completion using point-wise and ROI-wise feature fusion.
Results
The approach surpasses the second-best KITTI 3D detector by over 3% AP, exceeds 10 frames per second, and improves detection on TOR4D over a previous state-of-the-art detector.
Takeaways & Limitations
Joint multi-task learning and multi-sensor fusion provide complementary geometric and cross-modality information for 3D detection across KITTI and TOR4D.
Takeaways & Limitations
The fusion process in prior ROI-level approaches is slow and imprecise because it involves thousands of ROIs and can ignore object orientation.
Abstract
from arXiv · showhide
In this paper we propose to exploit multiple related tasks for accurate multi-sensor 3D object detection. Towards this goal we present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels. Importantly, our approach leads the KITTI benchmark on 2D, 3D and BEV object detection, while being real time.
1. Introduction
The paper addresses limitations of individual and cascade sensor-fusion approaches with an end-to-end model that jointly reasons about multiple perception tasks and sensors. It reports improved KITTI detection performance while operating in real time.
- Individual cameras struggle with fine-grained 3D information, while LiDAR observations become sparse at long range, making occluded and distant objects difficult to detect.
- The proposed end-to-end detector jointly reasons about 2D and 3D object detection, ground estimation, and depth completion.
- Point-wise and ROI-wise fusion are combined to produce fully fused feature representations, while ground estimation supplies geometric cues for 3D detection.
- Depth completion is used to learn cross-modality representations and achieve denser point-wise feature fusion.
- Over 3% AP separates the proposed detector from the second-best 3D detector on KITTI, while the system runs over 10 frames per second.
2. Related Work
Prior work uses cameras, LiDAR, multi-sensor fusion, or auxiliary tasks for 3D detection, but these approaches face depth-estimation, sparsity, fusion, or optimization limitations. The paper positions its approach as addressing these limitations through denser and jointly trainable fusion.
- Camera-based 3D detectors face difficulty estimating depth, whereas LiDAR detectors provide precise depth but sparse, less information-rich observations.
- Cascade fusion methods combine image detections with LiDAR-based 3D regression, but overall performance is bounded by the stages in the cascade.
- ROI fusion operates only on selected high-level regions, while continuous-convolution fusion remains limited when LiDAR points are very sparse.
- Auxiliary tasks such as geometric ground shape, semantic road masks, depth reconstruction, and semantic segmentation have been used to improve 3D object detection.
3. Multi-Task Multi-Sensor Detector
The proposed detector combines point-wise and ROI-wise multi-sensor fusion with ground estimation and depth completion in an end-to-end multi-task architecture. These components support coarse-to-precise 2D and 3D detection while improving geometric and cross-modality representations.
- Architecture: The model combines point-wise and ROI-wise feature fusion in a two-stream LiDAR–image architecture trained end-to-end with multiple perception tasks.It reasons about 2D and 3D detection, ground estimation, and depth completion.
- Point-wise Feature Fusion: The backbone fuses multi-scale image features into LiDAR features, establishing dense correspondences through projection and geometric features.Image and LiDAR streams are fused across backbone layers using projected correspondences and an MLP-based fusion operation.
- Ground Estimation: Ground estimation regresses voxel-wise road height online, enabling LiDAR features and predicted object heights to be represented relative to the ground.The formulation estimates ground height for each BEV voxel and adds the ground height back to the predicted Z term.
- Depth Completion: Depth completion predicts dense image-resolution depth from projected LiDAR depth and RGB images, densifying observations and supporting dense point-wise fusion.The completed depth is transformed into dense pseudo-LiDAR points for cross-modality feature representation.
4. Experiments
The experiments evaluate the proposed detector on KITTI and TOR4D, using benchmark comparisons, ablations, and qualitative results to assess multi-sensor fusion and multi-task learning. The method improves detection across tasks while supporting real-time operation and high-quality detections in difficult cases.
- Experimental setup: The method is evaluated on KITTI 2D, 3D, and BEV detection and on the multi-class TOR4D BEV benchmark.KITTI evaluation focuses on cars; TOR4D evaluates vehicles, pedestrians, and bicyclists across a 100-meter range.
- KITTI results: The detector outperforms prior methods across KITTI 2D, 3D, and BEV detection tasks.It surpasses RRC by 1.1% AP in hard 2D detection and HDNET by 0.9% AP in BEV detection.
- KITTI results: 3.09% AP over SECOND and 4.87% AP over AVOD-FPN are achieved in KITTI 3D detection.The paper attributes the gains mainly to fully fused feature representations and orientation-aware ROI feature extraction for precise localization.
- Ablation study: Over 5% AP is gained in KITTI 3D detection when image features are added to a LiDAR-only baseline through point-wise fusion.Ground estimation further improves moderate-setting 3D and BEV detection by 1.9% and 1.4% AP, respectively.
- TOR4D results: Depth completion improves TOR4D BEV detection through multi-task learning and dense fusion using estimated depth.The TOR4D experiment uses higher-resolution long-focus camera images to assess depth completion and dense fusion.
- Qualitative results and discussion: The detector produces high-quality 3D detections for highly occluded or distant objects and can detect some unannotated cars.Detecting in BEV space enables high-quality 3D detections through a single-pass FCN with densely fused multi-sensor feature maps.
5. Conclusion
The conclusion presents a jointly trained multi-task, multi-sensor detector for 2D and 3D detection, ground estimation, and depth completion. It combines point-wise and ROI-wise fusion with task-derived geometric cues and reports large-margin state-of-the-art results on KITTI and TOR4D.
- Conclusion: The proposed model jointly reasons about 2D and 3D object detection, ground estimation, and depth completion.It applies point-wise and ROI-wise feature fusion to achieve full multi-sensor fusion.
- Conclusion: Multi-task learning provides map priors and geometric cues that support better representation learning and denser feature fusion.The conclusion frames these cues as complementary to the multi-sensor fusion architecture.
- Conclusion: The method surpasses the state of the art by a large margin across all detection tasks on KITTI and TOR4D.The paper identifies both benchmarks as validation settings for the proposed approach.
Supplementary Materials
Supplementary evaluations compare MMF with state-of-the-art detectors on KITTI and examine TOR4D range-wise BEV detection and qualitative depth completion. The proposed detector shows consistent precision advantages across KITTI detection settings, with larger TOR4D gains at long range.
- KITTI detection: Figure 7 compares MMF’s precision–recall curves with state-of-the-art methods for KITTI 2D, 3D, and BEV car detection.The comparison covers all three detection settings on the KITTI test set.
- KITTI detection: MMF shows a consistent precision-rate advantage across KITTI 2D, 3D, and BEV detection settings.The paper attributes this result to the effectiveness of the joint model in producing high-quality detections.
- TOR4D BEV detection: Figure 8 evaluates TOR4D multi-class BEV detection across different ranges and IoU thresholds.The evaluation is fine-grained and explicitly analyzes performance as detection range changes.
- TOR4D BEV detection: Depth completion produces larger AP gains at long range on TOR4D through dense fusion.The corresponding qualitative depth-completion results are shown for KITTI and TOR4D in Figure 9.