3D Point Cloud Processing and Learning for Autonomous Driving

Siheng Chen; Baoan Liu; Chen Feng; Carlos Vallespi-Gonzalez; Carl Wellington

Source-linked AI summary

3D Point Cloud Processing and Learning for Autonomous Driving

Siheng Chen, Baoan Liu, Chen Feng, Carlos Vallespi-Gonzalez, Carl Wellington

arXiv:2003.00601v1Mar 1, 2020cs.CVeess.SP

TL;DR

Autonomous driving requires reliable processing and learning from LiDAR point clouds, but autonomous-system modules still face technical bottlenecks and long-tail issues. This paper reviews point-cloud processing and learning methods across map creation, localization, and perception, finding substantial performance improvements while identifying important open challenges.

Problem
Autonomous systems remain imperfect because of technical bottlenecks and long-tail issues, motivating reliable 3D point-cloud methods for core driving modules.
Method
The paper surveys 3D point-cloud processing and learning techniques and their applications to map creation, localization, and perception.
Results
Overall performance in map creation, localization, and perception has significantly improved with advances in 3D point-cloud processing and learning.
Takeaways & Limitations
LiDAR and advanced 3D point-cloud techniques are critical building blocks for autonomous driving, although an ultimate solution has not yet been determined.

Abstract

from arXiv · show

We present a review of 3D point cloud processing and learning for autonomous driving. As one of the most important sensors in autonomous vehicles, light detection and ranging (LiDAR) sensors collect 3D point clouds that precisely record the external surfaces of objects and scenes. The tools for 3D point cloud processing and learning are critical to the map creation, localization, and perception modules in an autonomous vehicle. While much attention has been paid to data collected from cameras, such as images and videos, an increasing number of researchers have recognized the importance and significance of LiDAR in autonomous driving and have proposed processing and learning algorithms to exploit 3D point clouds. We review the recent progress in this research area and summarize what has been tried and what is needed for practical and safe autonomous vehicles. We also offer perspectives on open issues that are needed to be solved in the future.

I. INTRODUCTION AND MOTIVATION

Autonomous driving systems combine sensing, mapping, localization, perception, prediction, routing, planning, and control, with LiDAR point clouds supporting several core modules. The review motivates these tools by their role in handling complex dependencies and improving reliable autonomous operation.

Module imperfections and long-tail issues create dependencies in which a small upstream error can produce substantial system-level failures.
An autonomous system typically combines sensing, map creation, localization, perception, prediction, routing, motion planning, and control.
LiDAR directly provides precise 3D scene information, whereas image-based depth estimates can remain unreliable under low light and other constraints.
High-definition maps are built offline, while runtime LiDAR sensing supports localization, perception, and subsequent trajectory planning.
Real-time LiDAR sweeps and point-cloud maps serve different roles: sweeps support localization and perception, while maps provide environmental reference information.

D. Outline

The article surveys foundational point-cloud properties, representations, and processing and learning tools before examining their use in map creation, localization, and perception. Real-time sweeps and point-cloud maps differ substantially in organization, density, viewpoint, and occlusion.

The article separates 3D point-cloud processing, which optimizes transmission, storage, or quality, from learning, which interprets and understands point clouds.
Real-time LiDAR sweeps: Real-time LiDAR sweeps are approximately organized on a 2D lattice using laser IDs and timestamps, but sequential firing prevents perfect alignment.
Real-time LiDAR sweeps: Real-time sweeps provide pseudo-3D, single-view observations with occlusion and sparse coverage, especially for far-away objects.
Point-cloud maps: Point-cloud maps aggregate sweeps across time and viewpoints, producing dense, full-3D, irregular point clouds with reduced occlusion.
Point-cloud maps: Aligned semantic feature maps attach traffic-related labels such as roads, buildings, and trees to the geometric point-cloud map.

B. Matrix representations

Matrix representations convert point clouds into structured forms for storage and computation, including raw points, voxels, range views, and bird’s-eye views. Each representation trades geometric fidelity, sparsity, organization, occlusion reasoning, resolution, and computational convenience differently.

A point cloud can be represented as a set of points and mapped to a matrix or tensor for efficient storage and scientific computation.
Raw points: Raw-point representations preserve all original point information but omit geometric structure, making them suitable for high-precision map and localization modules.
3D voxelization: Voxelization provides uniform spatial cells and supports 3D convolutions, but commonly creates extreme sparsity and resolution–memory trade-offs.
Range view: Range-view representations organize real-time LiDAR by angular coordinates, selecting the nearest range in each frustum and yielding compact sensor-native inputs.
Bird’s-eye view: Bird’s-eye-view representations project voxels onto the ground plane, simplifying 2D processing and map fusion while losing height, occlusion reasoning, and fine detail.

C. Representative tools

Representative tools process 3D point clouds through discretized or raw-point representations. CNNs leverage spatial structure and mature image-processing techniques, while PointNet and graph-based methods operate directly on unordered points with different strengths and limitations.

Representations: 3D point clouds can be represented through voxelization, range views, bird’s-eye views, or raw points for processing and learning.Voxelization partitions three-dimensional space; range views use azimuth and elevation; BEV omits height.
Convolutional neural networks: CNN discretization modifies exact 3D positions and creates sparse volumes or images, causing information loss and wasted computation on empty voxels.Three-dimensional convolution is also expensive in computation and memory usage.
PointNet-based methods: PointNet directly processes unordered raw points with shared point-wise MLPs and global max-pooling, producing permutation-invariant geometric features.The MLPs test geometric patterns at each point, while max-pooling records the strongest response for each pattern.
PointNet-based methods: PointNet-based methods have achieved strong performance across recognition and segmentation tasks and become standard building blocks for 3D point-cloud learning.Their effectiveness has also been validated in various processing and learning tasks.
Graph-based methods: Graph-based methods model points as graph nodes and pairwise relationships as edges to capture local spatial relationships in raw point clouds.Graphs may be constructed using K-nearest neighbors, distance thresholds, or learnable structures.
Graph-based methods: Graph filtering applies polynomial filters of a graph shift to graph signals, with higher filter order yielding a larger receptive field.Graph filtering supports tasks including 3D point-cloud downsampling and denoising.

III. 3D POINT CLOUD PROCESSING FOR HIGH-DEFINITION MAP CREATION

HD maps encode static 3D surroundings and traffic rules, supplying priors across autonomous-driving modules. Their creation combines precise point-cloud stitching with semantic-feature extraction under sophisticated engineering and human supervision.

HD-map representation: An HD map represents the static 3D environment and traffic rules through aligned point-cloud and semantic-feature layers.Semantic features include road boundaries, lanes, signs, lights, and curb heights.
HD-map priors: HD maps provide priors that support localization, perception, prediction, and motion planning in autonomous-driving systems.These priors encode environmental information and traffic rules offline with human supervision and quality assurance.
Localization priors: Map-based localization registers real-time LiDAR sweeps to the point-cloud map to obtain high-precision vehicle ego-motion.Point-cloud and semantic features such as lane markers and poles serve as localization priors.
Perception priors: The perception module uses HD-map priors such as traffic-light positions and point-cloud maps for state estimation and real-time foreground-background separation.Background points can then be removed from real-time LiDAR sweeps.
Prediction priors: Road and lane geometries and connectivities guide predicted object trajectories to follow traffic lanes.These are described as important priors for the prediction module.
Motion-planning priors: Traffic-rule features guide motion planning toward correct lanes and compliance with stop signs, traffic signs, and lane speed limits.Relevant map features include lane geometries, connectivities, traffic lights, traffic signs, and speed limits.
Map creation: HD-map creation requires high precision and current information, using machine learning, multiple modalities, human supervision, and quality assurance.A standard creation module includes 3D point-cloud stitching and semantic-feature extraction.

B. 3D point cloud stitching

3D point cloud stitching creates centimeter-precise city-scale maps by estimating LiDAR poses and optimizing their relations in a graph. Hierarchical refinement improves global optimization by supplying initialization from efficiently optimized adjacent edges.

B. 3D point cloud stitching: City-scale stitching must produce centimeter-level local map precision while remaining highly robust and efficient across more than 100 million LiDAR poses.The map is built from sensor data collected by a vehicle fleet across time.
B. 3D point cloud stitching: A LiDAR pose is a 6-DOF transformation between the global map frame and the vehicle’s LiDAR frame, combining 3D translation and rotation.
B. 3D point cloud stitching: Graph-based SLAM models LiDAR poses as nodes and their pairwise misalignment measurements as edges, optimizing all poses jointly from measurements across time.The pose objective can use ICP to measure sweep misalignment and GPS to measure deviation from predicted map-frame positions.
B. 3D point cloud stitching: Hierarchical refinement separates adjacent and loop-closure edges, using efficient adjacent-edge optimization to initialize the global optimization.This initialization significantly reduces computational cost and improves robustness for the entire pose graph.
B. 3D point cloud stitching: Semantic feature extraction combines automatic machine-learning extraction with human supervision to produce traffic-rule-related features from the point-cloud map.Human edits also provide training data, forming a feedback loop intended to improve HD-map production precision and efficiency.

D. Real-world challenges

Real-world autonomous-driving systems must maintain precise, robust localization and mapping despite difficult geometry, weather, GPS, scale, and traffic conditions. The reviewed pipeline combines map registration with multisensor fusion, while automatic semantic extraction and globally precise map updates remain challenging.

D. Real-world challenges: Global centimeter-level precision is difficult to enforce in graph-based SLAM, especially where GPS is unavailable, including building canyons, tunnels, and underground garages.Global optimization can distribute edge errors across the graph and misalign poses even when GPS observations are accurate.
D. Real-world challenges: Automatic extraction of intersection lane connectivities and traffic-light lane-control relations remains challenging because training labels are limited and traffic conditions are complex.
D. Real-world challenges: Map-based localization estimates the vehicle’s 6-DOF pose by registering LiDAR sweeps to a point-cloud map and fusing IMU, GPS, odometer, and camera measurements.Its two components are LiDAR-to-map registration and multisensor fusion.
D. Real-world challenges: Geometry matching is effective in heavy traffic and snow, whereas laser-reflectivity matching is advantageous on highways and bridges with sparse geometry but rich ground textures.Reflectivity matching is weakened by heavy rain and snow, so both strategies can be used together.
D. Real-world challenges: LiDAR-to-map registration alone cannot guarantee perfect pose estimation because severe occlusion, such as trucks surrounding the vehicle, can cause failure.Multisensor fusion is required to handle such extreme cases and improve localization robustness.

C. Real-world challenges

Real-world perception and 3D detection systems must combine heterogeneous sensor inputs, representations, and detection stages while meeting demanding robustness and efficiency requirements. LiDAR-based methods offer detailed 3D information but face sparsity, scale, occlusion, and multimodal-fusion challenges.

Perception module: Perception modules combine cameras, LiDAR, RADAR, ultrasound, localization poses, and HD-map priors to produce traffic-light states and tracked 3D bounding boxes.
Perception module: Late fusion combines modalities in semantic space, whereas early fusion combines them in feature space; late fusion is more mature, while early fusion has greater potential.Late fusion supports modular implementation, debugging, and management; early fusion enables mutual promotion in a high-dimensional feature space.
3D object detection: 3D object detection localizes scene objects with bounding boxes from one or more sensor measurements, and its outputs feed object association and tracking.
3D object detection: Single-stage detectors directly estimate bounding boxes, while two-stage detectors first propose coarse 3D regions and then refine precise object positions.In the two-stage formulation, proposal generation produces candidate regions and bounding-box estimation extracts their points to estimate object positions.
3D object detection: LiDAR detection uses raw points, voxels, bird’s-eye view, range views, or hybrids, each trading spatial priors, sparsity, efficiency, occlusion modeling, scale variation, and training requirements.VFE combines raw-point and voxel-based representations by grouping points into voxels, learning point features, and aggregating voxel features.
Fusion-based detection: Multimodal detection remains difficult because modalities occupy different measurement spaces, are asynchronously sampled, and have distinct sensor-specific characteristics.LiDAR is sparse and instantaneous, RADAR provides motion information, and images provide dense measurements; naive detection merging has limited benefit.

C. Real-world challenges

Perception has improved substantially but remains constrained by cost, latency, data requirements, long-tail conditions, research-to-industry transfer, and evaluation metrics. These challenges limit the direct relationship between model-level performance and autonomous-system behavior.

Expensive LiDAR and computing hardware make maintaining a scaled autonomous-vehicle fleet difficult, leaving low-cost processing of real-time LiDAR sweeps unresolved.
Real-time operation requires balancing perception effectiveness against efficiency, yet research often emphasizes algorithmic effectiveness more than latency.
Large-scale training requires substantial time and computation, while selecting a representative subset of data remains unresolved.
Training data cannot cover every traffic condition, leaving corner cases and objects absent from training data as unresolved long-tail problems.
Algorithms developed on clean, small datasets may fail on noisy, large datasets, while methods effective at scale may not transfer to small datasets.
PR curves and AP weight samples uniformly and poorly reflect rare high-impact cases, so evaluation should also examine targeted subsets and system-level metrics.

VI. SUMMARY AND OPEN ISSUES

The review surveys how 3D point-cloud processing and learning supports map creation, localization, and perception in autonomous driving. It reports significant performance improvements while identifying unresolved needs for scalability, robustness, iteration speed, and system-level evaluation.

3D point-cloud processing and learning are critical building blocks for map creation, localization, and perception, with LiDAR described as indispensable for reliable autonomous vehicles.
Overall performance in map creation, localization, and perception has significantly improved, but substantial challenges remain.
Scalability and efficiency: City- or country-scale deployment requires scalable algorithms to create and update HD maps as LiDAR point clouds become denser.
Robustness: Robust algorithms must handle corner cases through representative real-world or simulated data, although training data cannot cover all possibilities.
Iteration speed: Practical progress requires faster research iteration and closer industry–academia collaboration while retaining increasingly capable but efficient algorithms.
Evaluation: Evaluation should move beyond task-specific average metrics toward system-level behavior and rare long-tail cases.

APPENDIX

The review contrasts academic and industrial priorities in 3D point cloud research and surveys reconstruction as a representative task for autonomous driving. Reconstruction compresses point clouds into compact codes while preserving recoverability, with Earth mover’s distance favoring quality and Chamfer distance favoring efficiency.

APPENDIX: Academic research often optimizes standardized datasets and model-level metrics, whereas industry emphasizes system-level performance in complex real-world settings.The comparison spans aims, datasets, methodology, and evaluation metrics.
APPENDIX: LaserNet++ uses LiDAR and camera data and outperforms the LiDAR-only LaserNet especially for objects far from the vehicle.The fusion model is trained on 5,000 sequences containing 1.2 million images.
APPENDIX: HD maps combine dense point-cloud geometry with traffic-rule-related semantic features, while localization matches real-time LiDAR sweeps against the point-cloud map.The map and localization examples connect point-cloud processing to autonomous-system operation.
APPENDIX: Reconstruction seeks a compact representation that can recover the original point cloud, addressing storage costs for HD maps and real-time LiDAR sweeps.The review notes that no mature compression standard yet handles large-scale, open-scene 3D point clouds.
APPENDIX: An encoder maps a point cloud to a low-dimensional code and a decoder reconstructs it, allowing the reconstruction to contain a different number of points.The code dimension satisfies C ≪ 3N, and optimization pushes the reconstruction close to the input.
APPENDIX: Earth mover’s distance usually yields better reconstruction than Chamfer distance, while Chamfer distance is more efficient to compute.Both metrics encourage the reconstructed and original point clouds to retain similar underlying manifolds.

2) 3D point cloud recognition:

3D point cloud recognition classifies an entire point cloud into a predefined category, supporting object classification in autonomous-vehicle perception. Standard practice uses end-to-end supervised neural networks, with ModelNet40 and classification accuracy providing a common evaluation setup.

2) 3D point cloud recognition:: Recognition classifies a complete 3D point cloud into a predefined category for object understanding in autonomous-driving perception.The classifier outputs category confidence values for the input point cloud.
2) 3D point cloud recognition:: A classifier maps a point cloud to a C-dimensional confidence vector, whose cth element gives the likelihood of class c.The output is denoted y, with y_c representing the likelihood of membership in the cth class.
2) 3D point cloud recognition:: ModelNet40 contains 12,311 meshed CAD models across 40 categories, typically split into 9,843 training and 2,468 testing models.Point clouds are sampled from mesh faces, rescaled to a unit sphere, and evaluated using classification accuracy.
2) 3D point cloud recognition:: State-of-the-art recognition methods use end-to-end supervised neural networks, with PointNet-based and graph-based methods receiving substantial attention.Earlier pipelines separated feature extraction from classification.
2) 3D point cloud recognition:: Encoder-decoder reconstruction can also support recognition by supplying an encoded representation to a separately trained classifier.This approach uses separate training phases and datasets for the encoder-decoder and classifier.

3) 3D point cloud segmentation:

3D point cloud segmentation assigns categories to individual points, enabling object and scene interpretation in autonomous-driving perception. The review distinguishes part and scene segmentation, describes common datasets and metrics, and surveys feature-based, filtering, optimization, and neural denoising approaches.

3) 3D point cloud segmentation:: Segmentation classifies each point into a predefined category, such as vehicle, tree, or road, for autonomous-driving perception.Part segmentation divides one object into parts, whereas scene segmentation divides a large scene into objects.
3) 3D point cloud segmentation:: Point-wise classifiers output a confidence vector for each point, with each class component representing that point’s class probability.The ith point’s probability for class c is denoted y_ic.
3) 3D point cloud segmentation:: ShapeNet part contains 16,881 models from 16 object categories annotated with 50 parts and is evaluated using mean IoU.Most models contain fewer than six labeled parts.
3) 3D point cloud segmentation:: S3DIS contains six indoor areas and 272 rooms, with points assigned to 13 semantic categories plus clutter; evaluation uses mean IoU and per-point accuracy.Each point is represented by a 9D vector containing XYZ, RGB, and normalized spatial coordinates.
3) 3D point cloud segmentation:: Segmentation extends recognition by combining each point’s local features with global point-cloud features before point-wise classification.This adapts the recognition pipeline’s local-feature extraction and global aggregation to per-point outputs.
3) 3D point cloud segmentation:: Denoising removes noise and recovers an original point cloud, supporting smoother downstream processing when reconstruction produces noisy points.The review describes filtering-based and optimization-based approaches, including edge-preserving and smoothness-regularized methods.
3) 3D point cloud segmentation:: Neural projection denoising estimates reference planes and projects noisy points onto them in a two-stage procedure.NPD uses a PointNet-based backbone to estimate a reference plane for each noisy point.

5) 3D point cloud downsampling:

Downsampling selects fewer representative points so large point clouds can be processed more efficiently, with potential benefits for storage and localization. The review covers farthest-point, learning-based, and nonuniformly random strategies and evaluates them through downstream tasks.

5) 3D point cloud downsampling:: Downsampling selects a representative subset of points, reducing the computational burden of large point clouds and supporting storage and map creation.Representative points from HD maps can enable faster and better localization performance.
5) 3D point cloud downsampling:: A downsampling operator selects M points from an N-point cloud, where M < N, and records their original indices.The selected index sequence is denoted M=(M_1,...,M_M).
5) 3D point cloud downsampling:: Downsampling is usually evaluated through downstream registration, classification, or reconstruction rather than directly.Metrics include localization error, classification accuracy, Earth mover’s distance, and Chamfer distance.
5) 3D point cloud downsampling:: The three common approaches are farthest point sampling, learning-based sampling, and nonuniformly random sampling.These strategies differ in how they select points and whether they adapt selection to subsequent tasks.
5) 3D point cloud downsampling:: Farthest point sampling iteratively chooses the point farthest from the current selected set, improving coverage over uniform random sampling at equal sample counts.FPS is deterministic K-means++ and is agnostic to the subsequent application.
5) 3D point cloud downsampling:: S-NET produces a fixed-size point cloud optimized for a subsequent task using reconstruction and task-specific losses, but large-scale operation remains impractical.Because reconstructed points are not necessarily original points, S-NET matches each to its nearest original neighbor.
5) 3D point cloud downsampling:: Nonuniform random downsampling chooses point indices from a designed distribution intended to make the process more efficient and task-adaptive.The distribution is obtained by solving a reconstruction-based optimization problem.

6) 3D point cloud upsampling:

3D point cloud upsampling generates dense point clouds from sparse inputs to describe scene geometry, while registration aligns multiple frames for mapping and localization. The review discusses their formulations, evaluation practices, and classical and learning-based methods.

3D point cloud upsampling: Upsampling generates a dense, high-resolution point cloud from a sparse, low-resolution input to describe an object's or scene's underlying geometry.It is described as the inverse procedure of downsampling and may reduce costs by enabling use of a low-resolution LiDAR.
3D point cloud upsampling: Downsampling and upsampling form primal and dual resolution-changing tasks: one selects representative points, while the other generates additional points.Downsampling can support data storage and map creation, whereas upsampling targets higher-density geometry.
3D point cloud upsampling: There is no standard benchmark for upsampling; studies use datasets such as VisionAir, ModelNet40, ShapeNet, or SHREC15 and commonly report Earth mover’s or Chamfer distance.
3D point cloud upsampling: Classical upsampling methods construct surfaces, add points geometrically, or use locally optimal projection to preserve sharp edges.Learning-based methods include PU-Net, progressive multi-step architectures, and EC-Net’s edge-aware loss for more precise reconstruction of sharp edges.
3D point cloud registration: Registration transforms point clouds from local sensor frames into a standardized global frame by finding correspondences and minimizing alignment error.It is important for autonomous-driving map creation and localization, with pose estimation formulated as an optimization over sensor poses.
3D point cloud registration: Registration is commonly evaluated using position and orientation differences for pairs and absolute trajectory error for sequences, with KITTI, Oxford RobotCar, and ETH ASL among standard datasets.Classical methods often alternate correspondence search and alignment; local methods such as ICP require a good initialization because of limited convergence range.