PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive LiDAR Adaptation for Road Detection

Zhe Chen, Jing Zhang, Dacheng Tao

arXiv:1904.01206v1cs.CV

TL;DR

Visual road detection is vulnerable to visual noise, while LiDAR and visual data and features occupy mismatched spaces that hinder effective fusion. PLARD progressively adapts LiDAR data and features to visual representations through altitude-based transformation and cascaded fusion. On KITTI, it outperforms other state-of-the-art road detection models and ranks at the top of the publicly accessible leaderboard.

  • Problem

    Visual road detection can underperform under visual noise, while LiDAR data and features do not share the same spaces as visual data and features.

  • Method

    PLARD applies data-space adaptation with altitude difference-based transformation, then feature-space adaptation through transformed LiDAR features and cascaded fusion.

  • Results

    PLARD outperforms other state-of-the-art road detection models on the KITTI benchmark and ranks at the top of the publicly accessible leaderboard.

  • Takeaways & Limitations

    Progressive adaptation enables PLARD to leverage both visual and LiDAR information for robust road detection in challenging urban scenes.

Abstract

from arXiv · show

Despite rapid developments in visual image-based road detection, robustly identifying road areas in visual images remains challenging due to issues like illumination changes and blurry images. To this end, LiDAR sensor data can be incorporated to improve the visual image-based road detection, because LiDAR data is less susceptible to visual noises. However, the main difficulty in introducing LiDAR information into visual image-based road detection is that LiDAR data and its extracted features do not share the same space with the visual data and visual features. Such gaps in spaces may limit the benefits of LiDAR information for road detection. To overcome this issue, we introduce a novel Progressive LiDAR Adaptation-aided Road Detection (PLARD) approach to adapt LiDAR information into visual image-based road detection and improve detection performance. In PLARD, progressive LiDAR adaptation consists of two subsequent modules: 1) data space adaptation, which transforms the LiDAR data to the visual data space to align with the perspective view by applying altitude difference-based transformation; and 2) feature space adaptation, which adapts LiDAR features to visual features through a cascaded fusion structure. Comprehensive empirical studies on the well-known KITTI road detection benchmark demonstrate that PLARD takes advantage of both the visual and LiDAR information, achieving much more robust road detection even in challenging urban scenes. In particular, PLARD outperforms other state-of-the-art road detection models and is currently top of the publicly accessible benchmark leader-board.

I. INTRODUCTION

Visual road detection remains vulnerable to visual noise, while LiDAR offers complementary robustness but is difficult to integrate because the modalities occupy different spaces. PLARD progressively adapts LiDAR data and features to visual information and achieves strong KITTI benchmark performance.

  • Robust urban road detection supports safe autonomous navigation, but visual methods remain vulnerable to variable illumination, overexposure, ambiguous appearances, and blur.
  • LiDAR is less susceptible to visual noise, yet existing LiDAR-based road detection methods provide only limited improvements over visual image-based approaches.
  • LiDAR and visual data occupy different spaces, making effective integration difficult.
  • PLARD progressively adapts LiDAR data space to visual data space and LiDAR feature space to visual feature space before fusion.
  • On KITTI, PLARD outperforms visual, LiDAR-based, and fused-information road detection algorithms, reaching state-of-the-art leaderboard performance.

II. RELATED WORK

Prior work develops visual, LiDAR-based, and multimodal road detection methods, but integrating the two sensing modalities remains challenging. The paper formulates road detection as binary pixel labeling on the 2D image plane using adapted LiDAR and visual inputs.

  • Road detection research includes model-based, classifier-based, and DCNN-based approaches for identifying road areas.
  • DCNN road detectors improve segmentation but remain susceptible to visual noise and can require long processing times.
  • LiDAR-based methods are robust to visual noise, but prior studies did not effectively exploit both LiDAR and visual information.
  • The paper formulates road detection as assigning each 2D image-plane pixel a binary road or non-road label.

B. Overview

PLARD combines visual and LiDAR processing through progressive adaptation and fusion. It transforms LiDAR data into an aligned image-plane representation, adapts LiDAR features, and classifies fused features for road detection.

  • Overview: PLARD addresses data- and feature-space gaps by progressively adapting LiDAR information for combination with visual information.
  • Overview: The detector fuses visual and adapted LiDAR features, then applies final binary classification to identify road areas.
  • Data space adaptation: Data space adaptation transforms raw LiDAR from 3D space to the 2D image plane while preserving distinguishable road characteristics.
  • Feature space adaptation: Feature space adaptation uses a learning-based module to transform LiDAR features so they better complement visual features.
  • Overview: The adaptation function comprises data-space and feature-space functions applied to LiDAR before fusion with visual features.

C. LiDAR Adaptation

LiDAR adaptation addresses both the mismatch between 3D point clouds and 2D images and the difficulty of preserving road-discriminative structure after projection. The approach uses altitude-based transformation to improve the LiDAR representation before feature adaptation.

  • The data-space module uses altitude difference-based transformation to transform LiDAR data while preserving road-discriminative characteristics.
  • Raw LiDAR consists of 3D coordinate points, whereas visual data consists of RGB pixels on a 2D image plane.
  • Calibration enables projecting LiDAR points onto the image plane, but direct projection can make road and non-road appearances less distinguishable.

1) Data Space Adaptation:

PLARD adapts LiDAR information across data and feature spaces before fusion with visual features. Altitude differences expose geometric structure, while a learned transformation aligns LiDAR features with visual features.

  • Data Space Adaptation: Altitude difference-based transformation computes image-plane values from projected LiDAR altitudes and neighborhood positions.The method ignores neighboring pixels not associated with 3D points.
  • Data Space Adaptation: The transformation behaves like a mean absolute altitude-gradient measure over the projected image plane.Road regions typically have small intensities, whereas upright and sharp objects produce larger altitude differences.
  • Feature Space Adaptation: Feature-space adaptation addresses inconsistency between LiDAR and visual features by learning an appropriate transformation operation.This gap arises because road appearances can differ across the two data sources.
  • Feature Space Adaptation: PLARD models feature adaptation as a linear transformation with a scale vector α, offset vector β, and LiDAR feature f_lidar.A transformation network estimates α and β from concatenated LiDAR and visual features using learned network functions.
  • Feature Space Adaptation: The learned feature transformation is implemented with fully convolutional operations and optimized jointly with the overall road detection system.The feature adaptation stage has low complexity relative to the overall DCNN, using three 1x1 convolutions and element-wise operations.

D. Cascaded Fusion for Adapted LiDAR Information

PLARD fuses adapted LiDAR features with visual features through a residual-based cascaded structure across successive convolutional stages.

  • Cascaded Fusion: The fusion function takes visual features and adapted LiDAR features as inputs and improves visual representations through a residual structure.The same residual-based fusion is applied at every subsequent convolutional stage in the DCNN pipeline.
  • Cascaded Fusion: The cascaded fusion operates stage by stage, with k identifying the convolutional stage and λ denoting a scalar parameter.A ResNet-101 backbone contains five convolutional stages in this design.

E. Overall Objective

PLARD is trained end-to-end with a fused road-detection objective and auxiliary supervision for its LiDAR-based and visual branches.

  • Overall Objective: End-to-end optimization learns parameters for both the visual image-based and LiDAR-based DCNNs.The fused LiDAR-and-image road-detection objective is optimized jointly across PLARD parameters.
  • Overall Objective: The LiDAR branch receives an additional loss for directly detecting roads from altitude difference images.The visual branch also uses an auxiliary loss to facilitate convergence.
  • Overall Objective: PLARD defines its losses with multinomial cross-entropy and combines them using corresponding loss weights.The relevant weights are w_parsing, w_lidar, and w_aux.

IV. EXPERIMENT

The experiments evaluate PLARD on the KITTI road benchmark using cross-validation for component analysis and test-set comparisons with state-of-the-art methods.

  • Experiment: PLARD is evaluated on KITTI through 5-fold cross-validation and test-set comparisons against other state-of-the-art road detection algorithms.The cross-validation examines individual components before overall benchmark evaluation.

A. Dataset

The KITTI road benchmark provides multimodal sensor data, multiple road-scene categories, and standardized evaluation metrics. PLARD uses a visual PSPNet with a ResNet-101 backbone and a separate LiDAR network under specified training settings.

  • KITTI contains 289 training images and 290 testing images across UM, UMM, and UU road-scene categories.
  • KITTI evaluation includes MaxF, AP, PRE, REC, FPR, and FNR, with MaxF used to rank results.PRE, REC, FPR, and FNR are measured at the MaxF working point, and results are transformed into birds-eye-view space.
  • Altitude difference images use a 7×7 window, with a maximum of 48 noncentral values and rescaling to [0, 255].
  • PLARD employs PSPNet with a ResNet-101 backbone for visual road detection and a 101-level DCNN for LiDAR feature extraction.
  • Images are resized to 384 by 1280, models are trained for 80 epochs with SGD, and the learning rate decays from 1 × 10^-4 to 1 × 10^-6.

D. Ablation Study

Ablation and benchmark evaluations show that both LiDAR adaptation stages improve road detection, while the complete PLARD system achieves strong overall and task-specific performance on KITTI.

  • Ablation Study: Altitude difference transformation improves MaxF by around 3 points over directly projected LiDAR points.The comparison is between L-ADT and L-Proj in the ablation study.
  • Ablation Study: The final Img + L-ADT + FSA model achieves the highest performance among the compared ablation methods.FSA uses cascaded fusion for feature space adaptation.
  • Quantitative Results: A single PLARD model improves the visual PSPNet baseline and achieves a superior MaxF score to other road detection algorithms.
  • Quantitative Results: The augmented PLARD system achieves the best scores for most metrics through multi-scale testing and a 3-model ensemble.The “+” configuration denotes this ensemble with multi-scale testing.
  • Quantitative Results: On UM, augmented PLARD exceeds NF2CNN by around 1 point for MaxF, while on UU it exceeds LidCamNet by around 2.5 points for AP.

2) Qualitative Results:

Qualitative test-set results indicate that PLARD remains robust across UM, UMM, and UU scenes, including heavy shadows and over-exposed areas. Its progressive adaptation combines perspective-aligned LiDAR data with cascaded feature fusion.

  • PLARD qualitative results cover UM, UMM, and UU road-detection tasks on the benchmark test set.
  • PLARD is robust to severe illumination conditions such as heavy shadows and over-exposed areas.
  • PLARD first adapts LiDAR data to 2D image space using altitude difference-based transformation to align with the perspective view.
  • PLARD then adapts learned LiDAR features to visual features through cascaded fusion layers.
Loading 1904.01206v1…