PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review

Ying Li, Lingfei Ma, Zilong Zhong, Fei Liu, Dongpu Cao, Jonathan Li, Michael A. Chapman

arXiv:2005.09830v1cs.CV

TL;DR

Automated processing of uneven, unstructured, noisy, and massive LiDAR point clouds is difficult, while comprehensive task-focused coverage of deep learning methods for autonomous driving has been limited. The paper systematically reviews architectures, datasets, metrics, and applications for segmentation, detection, and classification, summarizing milestone models and reported performance before identifying remaining challenges and future research.

  • Problem

    Automated analysis of uneven, unstructured, noisy, and massive LiDAR point clouds is challenging, and prior surveys had not comprehensively reviewed deep learning applications for autonomous-driving tasks.

  • Method

    The paper systematically surveys milestone 3D deep models, LiDAR datasets, evaluation metrics, and deep learning applications in segmentation, detection or localization, and classification or recognition.

  • Results

    The review summarizes and evaluates state-of-the-art architectures and applications across segmentation, detection, and classification, including merits and demerits comparisons.

  • Takeaways & Limitations

    The survey identifies research challenges and opportunities intended to advance deep learning with LiDAR point clouds in autonomous driving.

Abstract

from arXiv · show

Recently, the advancement of deep learning in discriminative feature learning from 3D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3D point clouds is a challenging and tedious task. In this paper, we provide a systematic review of existing compelling deep learning architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving such as segmentation, detection, and classification. Although several published research papers focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on deep learning applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this paper is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3D deep architectures, the remarkable deep learning applications in 3D semantic segmentation, object detection, and classification; specific datasets, evaluation metrics, and the state of the art performance. Finally, we conclude the remaining challenges and future researches.

I. INTRODUCTION

The paper addresses the lack of a comprehensive review of deep learning applied to LiDAR point clouds for autonomous-driving segmentation, detection, and classification. It organizes architectures, datasets, metrics, applications, and remaining challenges across these tasks.

  • LiDAR supports autonomous-driving environment perception, object detection, and HD-map or urban-model generation for localization.
  • Existing surveys largely cover visual data, remote sensing, individual tasks, CAD models, or general LiDAR methods rather than deep learning on LiDAR point clouds for autonomous driving.
  • The review systematically covers milestone 3D deep models and deep learning methods for segmentation, object detection or localization, and classification or recognition in autonomous vehicles.
  • It surveys LiDAR datasets and introduces quantitative evaluation metrics and performance comparisons for segmentation, detection, and classification.
  • The paper also lists remaining challenges and future research directions for advancing deep learning with LiDAR point clouds in autonomous driving.

B. Challenges and Problems

LiDAR point-cloud learning must handle sensing variation, irregular and unordered data, and competing demands for accurate, efficient processing. These challenges arise from both the data itself and the requirements imposed on autonomous-driving models.

  • Challenges on LiDAR point clouds: Changing scenes, scan times, locations, weather, sensors, distances, and backgrounds create substantial intra-class and extra-class variation.
  • Challenges on LiDAR point clouds: LiDAR data exhibit varied point density and reflective intensity, noise, incompleteness from occlusion or clutter, and confusion between shape- or reflectance-similar categories.
  • Problems for DL models: Deep learning models must be invariant to point permutations, missing orientation, and rigid transformations such as rotations and translations.
  • Problems for DL models: LiDAR scale creates storage and processing difficulties, with datasets containing millions to billions of points and KITTI frames containing about 100k points.
  • Problems for DL models: Models must maintain accurate perception despite sparse, uneven, missing, and variable-quality data while remaining efficient on computationally and storage-limited autonomous-vehicle hardware.

III. DATASETS AND EVALUATION METRICS

This section surveys LiDAR datasets used for autonomous-driving research and the evaluation metrics applied to segmentation, detection, localization, and classification. It organizes datasets by task while noting their scale, acquisition setting, annotation structure, and challenges.

  • Dataset roles: Reliable datasets support algorithm comparison and motivate increasingly complex LiDAR point-cloud tasks.They also underpin progress in segmentation, 3D object detection, and classification.
  • Dataset organization: Existing datasets are grouped into segmentation-based, detection-based, classification-based, and long-term autonomy categories.This organization reflects the tasks and operating settings represented in the survey.
  • Segmentation datasets: Semantic3D contains more than 4 billion points covering around 110,000m2, with eight labeled classes for outdoor scene segmentation.Its challenges include massive point clouds, uneven point density, and severe occlusions; a reduced-8 version supports computation-constrained evaluation.
  • Segmentation datasets: Paris-Lille-3D contains 140 million points across 55,000m2 and resembles autonomous-vehicle LiDAR because it uses sparse, lower-resolution mobile scans.Its annotations cover 50 classes, also grouped into 10 coarse classes for challenging.
  • Detection datasets: KITTI provides LiDAR, imagery, GPS, and inertial-navigation data from six hours of autonomous-driving recordings for detection and birds-eye-view benchmarks.The cited passage identifies 7,481 training images and 7,518 test images for the benchmark.

B. Evaluation Metrics

This section introduces evaluation metrics and reviews milestone deep-learning architectures for LiDAR point clouds. It contrasts volumetric, point-based, and related representations while highlighting their computational and structural trade-offs.

  • Evaluation Metrics: Segmentation is commonly evaluated with Intersection over Union (IoU) and overall accuracy (OA).IoU quantifies the overlap between a target mask and the prediction output.
  • Evaluation Metrics: Detection and classification commonly use precision, recall, F1-score, and Matthews correlation coefficient (MCC).Precision measures correct detections among results, recall measures correct detections in ground truth, and F1-score balances precision and recall.
  • Evaluation Metrics: 3D localization and detection commonly use Average Precision (AP3D) and Average Orientation Similarity (AOS).AP3D averages valid bounding-box overlaps above predefined thresholds, while AOS averages orientation similarities at thresholded overlaps.
  • Volumetric models: Voxel-based representations enable direct 3D convolution and pooling but incur cubic computational and memory growth as resolution increases.Existing voxel models therefore commonly remain at low resolution, with 30^3 cited as a typical grid size.
  • Volumetric models: ShapeNet uses a convolutional deep belief network on binary voxel distributions, achieving notable low-resolution results but limiting large-scale performance through cubic cost.Its architecture uses three convolutional layers with layer-wise pretraining and generative fine-tuning.

B. Point clouds based models

Point-cloud-based models process unordered 3D points directly, using permutation-aware architectures and increasingly local geometric structure. Representative models include PointNet, PointNet++, Kd-networks, and PointCNN, but pointwise independence can miss relationships among neighboring points.

  • PointNet processes raw point clouds directly, learning each point’s spatial feature with MLPs and aggregating features through max-pooling.A spatial transform network and symmetric function improve permutation invariance.
  • PointNet++ addresses PointNet’s local feature limitations by sampling and grouping points into overlapping neighborhoods before hierarchical feature extraction.It uses Euclidean neighborhoods, farthest point sampling, and KNN or query-ball searches.
  • Kd-networks recursively partition point clouds with a balanced kd-tree to impose an input-point order instead of using symmetric aggregation.The tree is constructed top-down by splitting along the coordinate axis with maximum point-coordinate range.
  • PointCNN learns an χ-transformation that weights and permutes input features into a latent order before applying conventional convolutions.The transformation supports permutation and transformation handling while enabling hierarchical aggregation of local features.
  • Pointwise processing can neglect geometric relationships among neighboring points, causing missing local features and limiting discriminative performance.This limitation motivates models that explicitly exploit local structure.

C. Graph-based models

Graph-based models represent points as nodes connected to neighboring points, enabling convolutions and attention to exploit local geometric relationships. Related view-based models trade geometric completeness for computational efficiency and reuse of mature 2D architectures.

  • Graph-based models: Graph representations connect each point to neighboring nodes, allowing graph neural networks and graph convolutions to extract spatially local correlation features.Two central design challenges are handling dynamically sized neighborhoods with shared weights and exploiting neighbors’ spatial and geometric relationships.
  • Graph-based models: Spectral graph filters face basis dependence, high computational cost, and missing local edge features when applied to point-cloud graphs.These limitations concern transfer across domains, whole-input filtering, and underused local structural information.
  • Graph-based models: ECC conditions convolution-filter weights on edge labels, while DGCNN’s EdgeConv dynamically updates neighborhood graphs across layers to learn local geometric structures.Both methods operate on graph nodes and edge information, but DGCNN recalculates graph structure for each layer output.
  • Graph-based models: GAT assigns different attention weights to neighboring nodes, and multi-head attention concatenates independently computed representations.This focuses computation on relevant neighbor features but does not initially account for local spatial relationships in the weights.
  • View-based models: Multi-view models project 3D shapes into 2D images, enabling established CNNs and image datasets while reducing computation through dimensionality reduction.MVCNN renders multiple views and fuses their features with max-pooling, but this discards information from less representative views.
  • View-based models: View-based approaches can lose geometrically related spatial information and contain redundant information across multiple views.MVCNN-MultiRes adds multi-resolution filtering, while RotationNet jointly outputs object pose and category from multiview images.

E. 3D Data Processing and Augmentation

Because reliable 3D datasets are limited by massive data volumes and tedious labeling, point-cloud augmentation is used to enrich variation and generate additional samples. Common transformations alter geometry, visibility, position, or noise while retaining specified similarities and differences.

  • Limited reliable 3D datasets result from massive data volumes and tedious labeling processes.The paper motivates augmentation as a response to this data constraint.
  • Data augmentation enriches variation and generates new point-cloud samples through transformations in data or feature space.Common data-space transformations include translation, rotation, and scale changes.
  • Augmentation assumes original and augmented data preserve similar features such as shape while differing in features such as orientation.
  • Classical point-cloud augmentation includes mirroring x and y axes, rotating around the z-axis, and randomly jittering height or position.
  • Additional transformations randomly change scale, occlude or down-sample points, add artifacts, or inject distributed noise into coordinates and local features.

V. DEEP LEARNING IN LIDAR POINT CLOUD FOR AVS

Deep-learning applications for autonomous-driving LiDAR are organized around segmentation, detection/localization, and classification/recognition. Across these tasks, models use direct sensor features and neighborhood-derived geometric descriptors before feature embedding and network construction.

  • LiDAR deep-learning applications for autonomous vehicles comprise 3D segmentation, object detection and localization, and object classification and recognition.These tasks support environment perception, scene understanding, object detection, HD-map generation, and localization.
  • The reviewed tasks all require exploiting input point-feature representations before feature embedding and network construction.
  • Direct point features come from laser scanners and include XYZ coordinates, intensity, angle, and number of returns.XYZ gives real-world point position, while intensity reflects material-surface reflectance.
  • Local point features encode spatial relationships among points and neighborhoods, commonly searched with KNN or spherical neighborhoods.These representations support segmentation, detection, classification, and neighborhood-based operations such as CNNs.
  • Geo-local descriptors derive from neighborhood covariance decomposition and include density, normal, curvature, linearity, and planarity.Examples include curvature = η0/(η0 + η1 + η2) and linearity = (η1 − η2)/η1.

A. LiDAR point cloud semantic segmentation

LiDAR semantic segmentation assigns semantic classes to individual points, but large-scale, irregular point clouds require architectures that address permutation, orientation, spatial structure, and computational cost.

  • A. LiDAR point cloud semantic segmentation: Semantic segmentation labels each point with a class such as street, building, car, pedestrian, tree, or traffic light.The task also requires classifying small features in large-scale point clouds.
  • Point-based networks: Point-based networks combine feature embedding with network construction to handle unordered and unoriented points while learning local and global features.Conventional CNNs require discriminative feature embedding because point clouds lack fixed permutation and orientation.
  • Point-based networks: Local features commonly use neighborhood geometry, including normals and curvature, while PointNet-derived modules abstract sampled points into high-level representations.These representations support semantic feature extraction from unordered point sets.
  • Point-based networks: Encoding-decoding architectures combine same-scale features and upsample them with 3D deconvolutions before interpolating per-point labels.Downsampling and upsampling can make fine-grained edge information difficult to preserve.
  • Point-based networks: DNNSP improves feature learning for objects with weakly discriminative features by pooling spatial information through clustered point sets and distance minimum spanning trees.An MLP then performs classification using the pooled features.
  • Voxel-based networks: Voxel-based networks voxelize point clouds into grids, learn grid features, and map them to segmentation masks.Point labels may come directly from voxel predictions or from additional global optimization enforcing spatial consistency.

3) Multiview-based networks:

Multiview segmentation projects point clouds into organized 2D representations so established CNN architectures can process them, while the review contrasts this approach with point- and voxel-based alternatives.

  • 3) Multiview-based networks:: Multiview segmentation uses rendering to create structured 2D grids and a deep architecture suited to the resulting data.View rendering and network construction are identified as the two key modules.
  • 3) Multiview-based networks:: Figure 7 contrasts point-cloud, voxel-based, and view-based segmentation architectures through SPG, MSNet, and DeePr3SS.The three examples represent different data representations for the same segmentation task.
  • 3) Multiview-based networks:: Hand-designed descriptors can represent local and global information, while point splatting projects points into view images using a spread function.Spin images are cited for point-based local features, and point splatting is cited for view generation.
  • 3) Multiview-based networks:: After projection into multiview 2D images, models such as VGG16, AlexNet, GoogLeNet, and ResNet can be applied.VGG16 is described as frequently used because stacked small-receptive-field convolutional layers provide a lightweight network with limited parameters.
  • 3) Multiview-based networks:: LiDAR detection commonly proceeds from rough scene segmentation and region proposals to feature extraction and final bounding-box and class prediction.This staged procedure addresses sparse point clouds where object centers may lie in empty space.
  • 3) Multiview-based networks:: LiDAR object detection similarly compares point-cloud, voxel, and view representations through VoteNet, VoxelNet, and ComplexYOLO.These examples are presented as representative architectures for 3D object detection and localization.
  • 3) Multiview-based networks:: VoteNet uses Hough voting over PointNet++ features to cluster object centers and generate bounding-box proposals, improving robustness to sparse, large-scale clouds.The review also reports high object-center localization accuracy for VoteNet.

2) 3D objects detection (localization) from regular voxel grid:

Regular-grid approaches convert LiDAR points into voxel or image representations, enabling convolutional processing for segmentation and detection while trading spatial detail against efficiency and completeness.

  • 2) 3D objects detection (localization) from regular voxel grid:: Voxel-based networks discretize 3D space into fixed-size grids and convert occupied cells into fixed-dimensional feature vectors.Non-occupied cells are represented with zero feature vectors.
  • 2) 3D objects detection (localization) from regular voxel grid:: Sparse convolutional voxel methods can use neighboring-cell voting to accumulate scores for potential objects.The voting scheme weights voxel grids with convolution kernels over receptive-field neighbors.
  • 2) 3D objects detection (localization) from regular voxel grid:: View-based approaches project 3D points into BEV, front-view, spherical, or cylindrical images before applying object detectors.The pipeline consists of 3D-point projection followed by detection from projected images.
  • 2) 3D objects detection (localization) from regular voxel grid:: BEV representations can encode height, intensity, and density as three image channels, including maximum height, maximum intensity, and normalized density.Alternative channel choices use maximum, median, and minimum height values.
  • 2) 3D objects detection (localization) from regular voxel grid:: Front-view projection can compensate for missing information in sparse BEV images by producing denser cylindrical representations.Multiview spherical projections are also used to retain 3D spatial information during projection.

4) Evaluation on 3D objects localization and detection:

The review evaluates LiDAR detection on KITTI benchmarks and discusses classification representations and road-marking architectures, while noting that current results remain below human-level performance.

  • 4) Evaluation on 3D objects localization and detection:: KITTI birds-eye-view and 3D detection benchmarks evaluate localization and detection using stated overlap thresholds and object-selection criteria.Pedestrian and cyclist boxes require at least 50% overlap, while car boxes require 70% overlap.
  • 4) Evaluation on 3D objects localization and detection:: Detection comparisons report runtime and average precision AP3D on the KITTI validation set because real-time localization matters for autonomous vehicles.Both accuracy and execution time are compared.
  • 4) Evaluation on 3D objects localization and detection:: LiDAR classification is more challenging than CAD or RGB-D benchmarks because real scans contain noise, occlusion, background clutter, and nonuniform point distributions.The paper identifies semantic object recognition as important for safe and reliable autonomous driving.
  • 4) Evaluation on 3D objects localization and detection:: Voxel-based classification depends on spatial resolution, orientation, and origin, with LiDAR examples using occupancy, density, and hit grids.One cited approach uses a voxel resolution of (0.1m)3.
  • 4) Evaluation on 3D objects localization and detection:: A modified U-Net combined with GAN processing classifies road markings from intensity images while reshaping small, broken, or missing markings.The architecture combines multiscale CNN processing with expert context knowledge.
  • 4) Evaluation on 3D objects localization and detection:: The survey concludes that LiDAR deep architectures have achieved significant success across segmentation, detection, and classification but remain far from human-level performance.This is presented as a remaining challenge motivating further research.

1) Multi-source Data Fusion:

Deep learning for LiDAR-based autonomous driving remains constrained by data sparsity, incomplete object context, limited onboard computation, and the lack of unified models across tasks. The review summarizes these challenges alongside existing architectures and future opportunities.

  • 1) Multi-source Data Fusion:: Multi-source fusion can provide accurate, geo-referenced, information-rich cues, but sparsity causes inconsistent and missing data, while current deep-learning fusion is not end-to-end.
  • No robust, memory-efficient 3D data representation has yet been agreed upon across voxels, point clouds, graphs, and 2D views.
  • Point-cloud and graph representations are limited by permutation invariance and computational capacity, constraining the quantity of processable points and model performance.
  • Onboard memory and computation limits make efficient architectures crucial, although only some models achieve real-time segmentation, detection, and classification.
  • Point-cloud sparsity and incomplete scans prevent deep models from fully exploiting detailed object context, including semantic cues in traffic signs.
  • Few architectures jointly combine segmentation, detection, and classification, leaving information shared among these tasks underexploited.
Loading 2005.09830v1…