Source-linked AI summary
Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation
Gregory P. Meyer, Jake Charland, Darshan Hegde, Ankit Laddha, Carlos Vallespi-Gonzalez
TL;DR
LiDAR-based perception must handle sparse measurements at long range while meeting real-time and compute constraints, and camera–LiDAR fusion methods can be computationally inefficient. This paper extends LaserNet with efficient image-feature fusion and joint 3D detection and segmentation; it reports state-of-the-art performance on both tasks, with gains especially at long ranges and on smaller objects.
Problem
LiDAR measurements become sparse at long range, while camera–LiDAR fusion methods are often computationally inefficient despite cameras providing dense texture information.
Source: §1 · p. 1Method
The paper extends LaserNet with efficient 2D image and 3D LiDAR fusion and adds joint 3D semantic segmentation without requiring 2D image labels.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
A single network for detection and segmentation reduces the compute and latency of running multiple independent models and provides more semantic classes to downstream self-driving components.
Source: §5 · p. 7
Abstract
from arXiv · showhide
In this paper, we present an extension to LaserNet, an efficient and state-of-the-art LiDAR based 3D object detector. We propose a method for fusing image data with the LiDAR data and show that this sensor fusion method improves the detection performance of the model especially at long ranges. The addition of image data is straightforward and does not require image labels. Furthermore, we expand the capabilities of the model to perform 3D semantic segmentation in addition to 3D object detection. On a large benchmark dataset, we demonstrate our approach achieves state-of-the-art performance on both object detection and semantic segmentation while maintaining a low runtime.
1. Introduction
Autonomous driving needs accurate 3D detection and semantic understanding under real-time and computational constraints. The paper fuses camera imagery with LiDAR in LaserNet and extends the model to jointly perform detection and segmentation, reporting state-of-the-art performance on both tasks without requiring 2D image labels.
1. Introduction
- At long range, LiDAR becomes increasingly sparse, motivating image fusion to improve performance on distant objects.The paper notes that camera images provide dense texture information missing from LiDAR data.Source: §1 · p. 1
- The method efficiently fuses 2D image data with LiDAR to improve LaserNet, an existing state-of-the-art LiDAR-based detector [18].The fusion approach is trained end-to-end without requiring additional 2D image labels.Source: §1 · pp. 1–2
- The approach achieves state-of-the-art performance on both 3D object detection and semantic segmentation on a large dataset.The paper also reports that the method maintains low runtime.Source: §1 · p. 2
- A single model performs 3D object detection and semantic segmentation, making better use of compute resources on self-driving vehicles.Figure 1 shows example results using 2D images at the top and 3D LiDAR points at the bottom.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
Prior work on 3D object detection and semantic segmentation uses different fusion strategies and data representations, each with efficiency or information trade-offs. This work places image features directly into LiDAR's native range view, while addressing the sparse camera–LiDAR correspondence found outdoors.
2.1. 3D Object Detection
- 2D-to-3D methods rely on computationally expensive image detection and require both 2D and 3D box labels, while this approach uses a lightweight image CNN and only 3D labels.Proposal-fusion methods pool features for each 3D proposal and typically have high runtime.Source: §2.1 · p. 2
- The proposed dense-fusion approach projects image features directly into LiDAR's native range view and is reported to improve detection efficiently.Earlier dense fusion projected image features into a voxelized BEV and used computationally expensive nearest-point identification.Source: §2.1 · p. 2
2.2. 3D Semantic Segmentation
- Segmentation representations trade accuracy and efficiency: discretizing 3D space can lose detail through quantization, while unstructured point clouds can slow convolutions.Prior representations include point clouds, voxelized 3D space, and spherical images.Source: §2.2 · p. 2
- Outdoor camera–LiDAR fusion has sparse pixel-to-range correspondence; the paper reports that attaching RGB values directly to points does not help, whereas CNN image features fused in range view improve long-range point segmentation.Indoor RGB-D methods can fuse per pixel because color and depth have dense correspondence.Source: §2.2 · p. 2
3. Proposed Method
The method extends LaserNet [18] by projecting camera information into the LiDAR’s native range-view representation and training one model for 3D detection and semantic segmentation.
3. Proposed Method
- The method modifies LaserNet [18] to fuse RGB image data and jointly perform 3D object detection and semantic segmentation.Source: §3 · p. 2, Figure 2 · p. 3
3.1. Input Data
- The inputs are Velodyne 64E LiDAR measurements and an RGB image, cropped to aligned 512×64×5 LiDAR and 1920×640×3 RGB images.Source: §3.1 · p. 3
3.2. Sensor Fusion
- LiDAR points are projected into RGB pixels, and CNN-extracted image features—not raw RGB values—are mapped into the LiDAR image.For lower-resolution feature maps, the pixel mapping is rescaled and coordinates are rounded to the nearest integer.Source: §3.2 · pp. 3–4
3.3. Network Architecture
- An auxiliary ResNet extracts image features for warping into the LiDAR view; expanded LiDAR channels and camera features are concatenated and processed by the primary network.Source: §3.3 · p. 4
3.4. Predictions and Training
- The predictions distinguish road from background and bicycles from motorcycles; training adds classes to the classification loss, keeps regression unchanged, and needs no 2D image labels.Source: §3.4 · p. 4
4. Experiments
On ATG4D, the experiments report strong object-detection and semantic-segmentation results, with fusion benefits varying by range and a runtime increase of 8 ms for the combined system. The evaluation covers the front 90° field of view out to 70 meters.
4. Experiments
- Evaluation uses ATG4D, with detection and segmentation assessed in the front 90° field of view and within 70 meters.The training set has 5,000 sequences and 1.2 million images; validation has 500 sequences and 5,969 images.Source: §4 · p. 4
4.1. 3D Object Detection
- Our method outperforms existing detectors in most cases, with stronger gains at long range and on pedestrians and bikes than a LiDAR-and-RGB method.The authors attribute the small-object advantage to the range-view representation retaining fine detail removed by BEV voxelization.Source: §4.1 · pp. 4–5, Table 1 · p. 5
4.2. 3D Semantic Segmentation
- Our approach considerably outperforms the segmentation baseline across all metrics, with particularly strong performance on pedestrian, bicycle, and motorcycle classes.The authors attribute the small-class results to using range view rather than the baseline's voxelized BEV representation.Source: Table 2 · p. 5, §4.2 · p. 5
- At long range, CNN-extracted image features considerably improve segmentation, while raw RGB has little effect and near-range fusion provides practically no benefit.Because segmentation metrics are dominated by near-range performance, the authors examine performance across multiple ranges.Source: Table 3 · p. 5, §4.2 · p. 5
- The segmentation confusion matrix shows that most class confusion occurs between motorcycles and bicycles.Source: Figure 4 · p. 5, §4.2 · p. 5
- Qualitative examples include detecting bikes 50–60 meters away in sparse LiDAR, while steep road elevation can cause road points to be classified as background.The examples also show correct bounding boxes despite many points on a person being classified as vehicle.Source: §4.2 · p. 5, Figure 5 · p. 6
4.3. Runtime Evaluation
- The combined image fusion and semantic-segmentation extensions add 8 ms on a TITAN Xp GPU, while the system still runs above 25 Hz.Source: Table 4 · p. 7, §4.3 · p. 7
5. Conclusion
The LaserNet extension fuses camera and LiDAR data for state-of-the-art 3D object detection and semantic segmentation, with gains especially at long ranges and on smaller objects. Joint prediction also reduces compute and latency, while identifying more semantic classes for downstream self-driving components.
5. Conclusion
- The method achieves state-of-the-art performance in 3D object detection and semantic segmentation on a large dataset.It extends LaserNet by fusing 2D camera data with 3D LiDAR data.Source: §5 · p. 7
- RGB image data improves performance especially at long ranges, where LiDAR measurements are sparse, and on smaller objects such as pedestrians and bikes.The approach is straightforward and efficient, and can be trained end-to-end without 2D labels.Source: §5 · p. 7
- The model identifies more semantic classes for downstream components, while combining detection and segmentation in one network reduces compute and latency compared with running independent models.Source: §5 · p. 7