Source-linked AI summary
Deep Continuous Fusion for Multi-Sensor 3D Object Detection
Ming Liang, Bin Yang, Shenlong Wang, Raquel Urtasun
TL;DR
3D localization is difficult because LIDAR becomes sparse for distant or occluded objects and images lack precise depth. The paper introduces an end-to-end BEV detector that continuously fuses image features with LIDAR geometry across scales. Experiments on KITTI and a large-scale benchmark report significant improvements over the state of the art, including strong long-range detection.
Problem
LIDAR is sparse for distant or occluded objects, while images provide dense measurements but lose depth information needed for precise 3D localization.
Method
An end-to-end BEV detector uses continuous convolutions to fuse projected image features and LIDAR features at multiple resolutions while encoding geometric offsets.
Results
The approach significantly outperforms the state of the art on KITTI and a large-scale 3D object detection benchmark, with superior performance in long-range detection.
Takeaways & Limitations
The model can incorporate high-resolution images for long-range detection and requires no extra image labels because it trains using 3D/BEV detection only.
Abstract
from arXiv · showhide
In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.
1 Introduction
3D object detection remains difficult because LIDAR is sparse for distant or occluded objects, while images lack precise depth. The paper addresses this by jointly reasoning over camera and LIDAR inputs in BEV with continuous fusion.
- LIDAR-based detectors suffer at long range and with occluded objects because returns become sparse.
- Images provide dense measurements, but perspective projection makes precise 3D localization difficult by removing depth information.
- Cascaded camera-LIDAR methods limit joint reasoning because 3D localization depends on the preceding 2D image-only detection step.
- The proposed detector projects image features into BEV and fuses them with LIDAR features using continuous convolutions at multiple resolutions.
- Evaluation on KITTI and a large-scale 3D object detection benchmark shows significant improvements over the state of the art.
2 Related Work
Prior work combines cameras and 3D sensors through image-space processing, voxelization, proposal cascades, or point-based geometric operators. These approaches face output-space, efficiency, information-loss, or joint-reasoning limitations that motivate the paper’s BEV fusion design.
- LIDAR-Based Detection: LIDAR detectors use front-view, BEV, volumetric, or point-based representations to estimate 3D objects.
- Joint Camera-3D Sensor Detection: Depth-image methods augment RGB features with geometric channels, but their output space remains the camera image plane.
- Joint Camera-3D Sensor Detection: Voxelization incorporates color into 3D cells, but it is inefficient and can lose color information under perspective projection.
- Convolution on 3D Point Clouds: Point-cloud convolution methods encode geometric relationships over irregular data without rasterization-related accuracy loss.
3 Multi-sensor 3D Object Detection
The detector creates dense BEV features by continuously fusing projected image features with LIDAR geometry across multiple network scales. Its architecture uses neighborhood aggregation, geometric offsets, multi-scale fusion, and a BEV detection head.
- BEV preserves LIDAR’s native 3D structure, while sensor fusion combines it with complementary camera information.
- The method projects image features into BEV and uses continuous convolutions to interpolate discrete camera features into a dense BEV map.
- Continuous Fusion Layer: Deep parametric continuous convolution uses an MLP kernel over neighboring non-grid points, weighting neighbors by relative geometric offset.
- Continuous Fusion Layer: For each BEV target pixel, K neighboring LIDAR points provide image features and 3D offsets; an MLP aggregates them into a Do-dimensional output feature.
- Continuous Fusion Layer: The fusion layer directly outputs target features rather than weights, providing more flexible neighbor aggregation and improved memory efficiency over standard continuous convolution.
- Multi-Sensor Object Detection Network: The network has image and BEV streams, with four continuous fusion layers connecting multi-scale image features to BEV residual groups.
- Multi-Sensor Object Detection Network: A 1 × 1 BEV convolution produces anchored class confidence, box center, size, and orientation outputs before non-maximum suppression.
4 Experiments
Experiments on KITTI and TOR4D evaluate the continuous fusion detector against LIDAR-based and fusion baselines, including ablations of its fusion components. The model improves detection performance, especially at long range, while maintaining real-time efficiency.
- KITTI Evaluation: The model outperforms all other methods on KITTI BEV detection and ranks third for 3D detection, with the best AP on the easy subset.These comparisons use the KITTI benchmark and its official evaluation server.
- KITTI Evaluation: >15 frames per second: the detector runs much faster than all other LIDAR-based and fusion-based methods.The reported speed is paired with high detection accuracy.
- Ablation Study: All continuous fusion models significantly outperform the LIDAR BEV model across all six KITTI metrics, with a larger advantage for 3D detection.The authors suggest fused image features provide complementary z-axis information to BEV features.
- Ablation Study: Removing geometric features significantly reduces performance, while continuous fusion without offsets still outperforms discrete fusion, supporting the value of KNN interpolation.The ablation separates the contributions of geometric features and KNN pooling.
- Ablation Study: KNN pooling consistently outperforms its removal across distance thresholds, whereas increasing nearest neighbors from 1 to 3 worsens performance.The authors attribute the degradation to inclusion of more distant neighbors with less predictive power.
- TOR4D Evaluation: On TOR4D, the continuous fusion model significantly outperforms the BEV baseline and PIXOR across all classes, with its advantage generally increasing as range increases.Piecewise AP is computed over 10-meter ranges for vehicles and pedestrians and 20-meter ranges for bicyclists.
5 Conclusion
The proposed end-to-end detector fuses LIDAR and camera features through continuous convolutions at multiple resolutions, achieving strong 3D localization and outperforming the state of the art on KITTI and a large-scale benchmark.
- The detector projects image features into bird’s eye view and fuses them with LIDAR features using continuous convolutions at different resolutions.
- The approach performs very accurate 3D localization by exploiting both LIDAR and cameras.
- The method significantly outperforms the state of the art on KITTI and a large-scale 3D object detection benchmark.