Source-linked AI summary
LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks
Luca Caltagirone, Mauro Bellone, Lennart Svensson, Mattias Wahde
TL;DR
Road detection requires robust use of complementary camera and LIDAR sensing. The paper projects sparse LIDAR data into camera-aligned 2D images and compares fixed-depth and trainable fusion FCNs, with cross fusion reaching a 96.03% MaxF score on the KITTI road benchmark.
Problem
Camera imagery is illumination-sensitive while LIDAR is sparse and range-limited, motivating robust integration of both sensors for road segmentation.
Method
The paper projects LIDAR point clouds into dense camera-aligned 2D images and trains single-sensor, early-fusion, late-fusion, and cross-fusion FCNs.
Results
96.03% MaxF was achieved by the cross fusion FCN in the urban category of the KITTI road benchmark.
Takeaways & Limitations
Multimodal systems provide more robust and accurate road segmentations across a wider spectrum of external conditions than camera-only FCNs in challenging scenes.
Abstract
from arXiv · showhide
In this work, a deep learning approach has been developed to carry out road detection by fusing LIDAR point clouds and camera images. An unstructured and sparse point cloud is first projected onto the camera image plane and then upsampled to obtain a set of dense 2D images encoding spatial information. Several fully convolutional neural networks (FCNs) are then trained to carry out road detection, either by using data from a single sensor, or by using three fusion strategies: early, late, and the newly proposed cross fusion. Whereas in the former two fusion approaches, the integration of multimodal information is carried out at a predefined depth level, the cross fusion FCN is designed to directly learn from data where to integrate information; this is accomplished by using trainable cross connections between the LIDAR and the camera processing branches. To further highlight the benefits of using a multimodal system for road detection, a data set consisting of visually challenging scenes was extracted from driving sequences of the KITTI raw data set. It was then demonstrated that, as expected, a purely camera-based FCN severely underperforms on this data set. A multimodal system, on the other hand, is still able to provide high accuracy. Finally, the proposed cross fusion FCN was evaluated on the KITTI road benchmark where it achieved excellent performance, with a MaxF score of 96.03%, ranking it among the top-performing approaches.
I. INTRODUCTION
Road detection supports safe trajectory planning, but camera and LIDAR each have important sensing limitations. This work therefore investigates FCN-based fusion of both modalities and contributes a challenging evaluation dataset.
- Road-surface detection is a prerequisite for safe trajectory planning and decision-making in automated driving.
- Cameras provide dense, long-range data at high frame rates but are strongly affected by illumination.
- LIDAR is only marginally affected by external lighting and measures distance accurately, but typically has 10–100 meters of range and sparse data.
- The paper investigates integrating LIDAR point clouds and camera images for road segmentation using fully convolutional neural networks.
- The authors contribute a fusion FCN and a dataset of visually challenging KITTI scenes for evaluating multimodal road segmentation.
II. RELATED WORK
Prior work applies machine learning to road detection and sensor fusion using cameras, LIDAR, and combined representations. The cited approaches include CNNs, Bayesian models, graph methods, and SVMs across varied driving conditions.
- Recent state-of-the-art road-detection algorithms largely use machine-learning techniques.
- Prior camera-based methods use CNNs for road segmentation and vehicle detection or Bayesian networks for estimating road surfaces and boundaries.
- LIDAR road-detection approaches transform point clouds into 2D top-view images or project them onto the camera plane before segmentation.
- Deep-learning sensor-fusion studies combine depth or HHA representations with RGB images for pedestrian and vehicle detection.
- Other road-detection work uses SVMs for 3D cloud data, including online learning and challenging conditions such as slippery roads and adverse weather.
III. NETWORK ARCHITECTURES
The proposed base network is a fully convolutional encoder-decoder with an intermediate context module. Its design combines downsampling, dilated convolutions, and spatial dropout for efficient contextual processing and regularization.
- The base FCN is a 21-layer fully convolutional encoder-decoder containing an intermediate context module.
- The encoder uses five 4 × 4 convolutional layers with stride 2 to downsample inputs and reduce memory requirements.
- The context module uses nine 3 × 3 convolutional layers with exponentially growing dilation to expand receptive fields with few layers.
- Spatial dropout follows each context-module convolutional layer, with dropout probability p = 0.25, to regularize training.
- The network uses ELU activation after each convolutional layer, and zero-padding preserves feature-map width and height in the context module.
B. Early and late fusion
Early and late fusion combine camera and LIDAR data at fixed processing depths. Cross fusion instead lets training learn where and how strongly to integrate the modalities across the network.
- LIDAR point clouds are transformed into ZYX 2D images matching the camera images’ spatial size, enabling CNN-based fusion.
- Early fusion concatenates RGB and ZYX tensors into a six-channel input, requiring joint feature learning from the network’s first layer.
- Late fusion processes LIDAR and camera inputs through independent branches before combining them at the end for decision-level fusion.
- Cross fusion replaces manually selected fusion depth with trainable connections that learn where fusion is necessary and to what extent.
C. Cross fusion
Cross fusion lets the FCN integrate LIDAR and camera information at multiple processing depths through trainable connections between their branches.
- C. Cross fusion: Cross fusion allows information integration at any processing depth rather than at one predefined level.
- C. Cross fusion: The cross-fusion inputs combine features from the two sensor-specific branches through learned cross-modal connections.
- C. Cross fusion: Trainable parameters connect the LIDAR and camera processing branches and are adjusted during training to integrate both modalities.The parameters are initialized to zero, representing no initial information flow between branches.
IV. DATA PREPROCESSING
The preprocessing converts sparse LIDAR point clouds into dense image-aligned representations containing 3D coordinates for CNN processing.
- IV. DATA PREPROCESSING: Each LIDAR point cloud is converted into three 2D images that align color-image pixels with 3D spatial information.The resulting channels encode the X, Y, and Z coordinates of projected points.
- IV. DATA PREPROCESSING: LIDAR points are projected onto the camera plane using the transformation, rectification, and camera projection matrices.Points with negative scaling factors or projections outside the image are discarded.
- IV. DATA PREPROCESSING: The projection procedure computes each point’s image column and row coordinates from its 3D LIDAR representation.
- IV. DATA PREPROCESSING: Because many LIDAR-image pixels contain zero values where no laser beam reached the scene, the images are upsampled before CNN processing.
V. EXPERIMENTS AND DISCUSSION
The experiments compare five FCNs on KITTI data, using standard and challenging scene sets while preparing inputs to a common image size.
- V. EXPERIMENTS AND DISCUSSION: Five FCNs were evaluated: ZYX, RGB, Early fusion, Late fusion, and Cross fusion.ZYX uses only LIDAR images, RGB uses only camera images, and the remaining networks implement the corresponding fusion strategies.
- V. EXPERIMENTS AND DISCUSSION: The KITTI road data set contains 289 training images and 290 test images from city, rural, and highway settings.Most training examples were captured under relatively ideal weather and lighting conditions.
- V. EXPERIMENTS AND DISCUSSION: RGB images were zero-padded so that every training example had dimensions of 384 × 1248 pixels.
- V. EXPERIMENTS AND DISCUSSION: The challenging set was created by the authors and is separate from the standard KITTI road data set.It was intended to evaluate scenes with more difficult visual conditions.
B. Training procedure
Training used Adam optimization for 100k iterations, with validation-based checkpointing and poly learning-rate decay.
- B. Training procedure: Training ran for N = 100k iterations using the Adam optimization algorithm.
- B. Training procedure: The network weights were saved whenever validation performance improved, with validation evaluated every 1000 iterations.
- B. Training procedure: The learning rate η was decayed using the poly learning policy.The starting learning rate was η0 = 0.0005 and α = 0.9.
- B. Training procedure: Training used a batch size of 1 and random image rotations between −20° and 20° for data augmentation.
C. Comparison of fusion strategies
The cross fusion network achieved the best validation performance among the compared single-modality and fusion FCNs, while KITTI’s mostly ideal conditions can limit the visible benefits of sensor fusion.
- 96.25% MaxF was achieved by the cross fusion network on the validation set, exceeding late fusion at 96.06% and RGB-FCN at 96.00%.
- 94.82% MaxF was obtained by the LIDAR-only FCN, the lowest result among the compared networks.
- KITTI road images mostly contain ideal lighting and weather, making camera images informative for road detection.
- The challenging set was extracted to evaluate scenes with shadows, strong reflections, or unusual lighting that affect road appearance.It contained 33 images from KITTI raw driving sequences.
E. KITTI road benchmark
The cross fusion FCN was evaluated on the KITTI road benchmark’s urban category and was reported among the best published methods at submission.
- The cross fusion FCN was among the best methods in the benchmark at the time of submission.The system is called LidCamNet.
- Individual-category benchmark results include false positive rate and false negative rate metrics.
VI. CONCLUSION
The proposed cross fusion FCN learns where to combine LIDAR and camera features through trainable cross connections across processing layers. It performed best among the compared networks and supported more robust segmentation in challenging conditions.
- Trainable cross connections between the LIDAR and camera branches replace the single predefined fusion depth used by early and late fusion.
- 96.03% MaxF was achieved by the cross fusion FCN in the urban category of the KITTI road benchmark.The method was reported among the top-performing algorithms.
- Multimodal systems provided more robust and accurate segmentations across a wider spectrum of external conditions than camera-only FCNs.The challenging data set was used to highlight this difference.
- KITTI test-set examples show correct road classifications alongside false negatives and false positives.