PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for 3D Point Clouds: A Survey

Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, Mohammed Bennamoun

arXiv:1912.12033v2cs.CVcs.LGcs.ROeess.IV

TL;DR

Deep learning for point clouds lacks a focused synthesis despite rapid growth across major 3D understanding tasks. This survey organizes recent methods, compares them across datasets, and reports that region proposal-based detectors outperform single-shot methods while graph-based networks attract increasing attention.

  • Problem

    Rapidly growing point-cloud methods lack a comprehensive survey focused specifically on deep learning for point-cloud understanding.

  • Method

    The paper develops a taxonomy of methods for classification, detection and tracking, and segmentation, with comparisons across publicly available datasets.

  • Results

    Region proposal-based detectors outperform single-shot methods on KITTI 3D and BEV benchmarks, while graph-based networks attract increasing attention for irregular data.

  • Takeaways & Limitations

    The survey consolidates state-of-the-art point-cloud understanding methods, their trade-offs, and potential research directions.

  • Takeaways & Limitations

    Point-based methods lack explicit neighboring information and often require expensive neighborhood construction, while projection and discretization methods suffer information loss or high computational costs.

Abstract

from arXiv · show

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.

1 INTRODUCTION

Affordable 3D sensors provide rich geometric information, making point clouds a preferred representation for scene understanding. This survey reviews deep learning methods for major 3D point-cloud understanding tasks and distinguishes its focus from broader 3D-data reviews, [20].

  • Motivation: Point clouds preserve original 3D geometry without discretization, supporting scene-understanding applications such as autonomous driving and robotics.3D sensors include scanners, LiDARs, and RGB-D cameras, while 3D data also provides shape and scale information,,.
  • Research landscape: Public datasets have accelerated research on deep learning for point clouds across classification, detection and tracking, segmentation, registration, and related problems.Examples include ModelNet, ScanObjectNN, ShapeNet [8], PartNet, S3DIS, ScanNet, Semantic3D, ApolloCar3D, and KITTI,.
  • Survey contributions: This work is presented as the first comprehensive survey of deep learning methods for 3D shape classification, 3D object detection and tracking, and 3D point-cloud segmentation.Unlike existing reviews, [20], it specifically focuses on deep learning for 3D point clouds rather than all types of 3D data.
  • Paper organization: The paper organizes its review around datasets and evaluation metrics, followed by task-specific surveys of classification, detection and tracking, and segmentation.The segmentation review includes semantic, instance, and part segmentation before the concluding section.

2 BACKGROUND

The background organizes benchmark datasets by point-cloud task and data characteristics, while introducing task-specific evaluation metrics. These resources span synthetic and real-world settings, indoor and outdoor scenes, and multiple sensing modalities.

  • Datasets: Table 1 summarizes representative datasets and their attributes for 3D shape classification, object detection and tracking, and point cloud segmentation.
  • Datasets: Shape-classification datasets include synthetic objects without occlusion or background and real-world objects affected by occlusion and background noise, [8],,.
  • Datasets: Detection and tracking datasets cover indoor scenes from depth maps or meshes and sparse, spatially separated outdoor urban scenes designed for autonomous driving,,,,,.
  • Datasets: Segmentation datasets use MLS, ALS, TLS, RGB-D cameras, and other 3D scanners to support challenges including distractors, shape incompleteness, and class imbalance,,,, [38],,,.
  • Evaluation metrics: For evaluation, overall accuracy and mean class accuracy are common for shape classification, while average precision is commonly used for 3D object detection.OA averages accuracy over test instances, mAcc averages accuracy over shape classes, and AP is calculated as the area under the precis…

3 3D SHAPE CLASSIFICATION · 3.1 Multi-view based Methods · 3.2 Volumetric-based Methods

3D shape classification methods learn global shape embeddings for classification and are organized by input representation into multi-view, volumetric, and point-based approaches. This section reviews multi-view feature fusion and volumetric voxel-based processing, including methods addressing information loss and computational scaling.

  • 3 3D SHAPE CLASSIFICATION: Shape classification methods aggregate point-level embeddings into a global shape representation, then use fully connected layers for prediction; the survey mainly emphasizes point-based methods.Methods are categorized by whether networks receive projected images, voxelized volumes, or raw point clouds.
  • 3.1 Multi-view based Methods: Multi-view methods project 3D shapes into multiple views, extract view-specific features, and fuse them into a discriminative global representation.Designing effective aggregation across views is the central challenge because view-wise information must be combined without losing useful structure.
  • 3.1 Multi-view based Methods: MVCNN max-pools multi-view features, but this can discard information by retaining only maximum elements from particular views.MHBN instead uses harmonized bilinear pooling to integrate local convolutional features into a compact descriptor.
  • 3.1 Multi-view based Methods: Relation-network approaches exploit inter-view and region relationships, while hierarchical node features can be concatenated across levels to form the global shape descriptor.These strategies target richer multi-view aggregation than simple max-pooling.
  • 3.2 Volumetric-based Methods: Volumetric methods voxelize point clouds into 3D grids and apply 3D convolutional networks for shape classification.VoxNet uses volumetric occupancy, while 3D ShapeNets models shape distributions with binary variables on voxel grids.
  • 3.2 Volumetric-based Methods: Dense volumetric methods cannot scale well to dense 3D data because their computation and memory requirements grow cubically.This limitation motivates hierarchical compact structures such as octrees.
  • 3.2 Volumetric-based Methods: OctNet reduces volumetric costs with a hybrid grid-octree representation, using efficient bit-string encoding and arithmetic voxel indexing.Compared with dense-grid baselines, OctNet uses much less memory and runtime for high-resolution point clouds.
  • 3.2 Volumetric-based Methods: PointGrid combines point and grid representations by sampling a constant number of points per volumetric cell, enabling 3D convolutions to capture geometric details.The hybrid representation is presented as an efficient alternative for point-cloud processing.

3.3 Point-based Methods

Point-based methods are organized by feature-learning architecture into pointwise MLP, convolution-based, graph-based, hierarchical data-structure, and other approaches. Pointwise methods ensure permutation invariance through shared point processing and symmetric aggregation, while other families address local geometry, irregular convolution, graph relations, or computational efficiency.

  • Pointwise MLP Methods: Pointwise methods independently process each point with shared MLPs and aggregate global features using symmetric functions, achieving permutation invariance.PointNet uses max pooling, while Deep Sets [53] sums representations before nonlinear transformations; independent learning limits local structural modeling.
  • Convolution-based Methods: Convolution-based methods address irregular point clouds with continuous kernels based on spatial relations or discrete kernels defined on regular grids.Continuous approaches include RS-Conv and PointConv, while discrete approaches transform points into grids or latent orders for convolution.
  • Convolution-based Methods: Rotation-equivariant convolution methods learn representations that remain equivariant to 3D rotations through spherical functions, spherical harmonics, or related geometric constructions.3D Spherical CNN and tensor field networks [72] use spherical harmonic representations for localized or point convolutions.
  • Graph-based Methods: Graph-based methods represent points as vertices connected to neighboring points, then learn features in spatial or spectral domains using edge geometry and neighborhood aggregation.Spatial graph methods commonly use MLP-based neighbor operations and pooling to form coarsened graphs; ECC uses a filter-generating network.
  • Hierarchical Data-Structure Methods: Grid-GCN improves computational efficiency by combining volumetric and point-based structuring, running 5× faster on average than other models for ModelNet classification.The method targets the time cost of farthest-point sampling and neighbor-point querying.

3.4 Summary

For 3D shape classification, ModelNet10/40 are the most frequently used datasets, while pointwise MLPs, convolution-based networks, and graph-based networks represent important architectural directions. Convolution-based networks perform well on irregular point clouds, whereas extending graph-based networks to diverse spectral-domain graph structures remains challenging.

  • 3.4 Summary: ModelNet10/40 are the most frequently used datasets for 3D shape classification, and Table 2 reports results from different point-based networks.
  • 3.4 Summary: Convolution-based networks can achieve superior performance on irregular 3D point clouds, motivating attention to discrete and continuous convolution networks.Pointwise MLPs commonly serve as basic building blocks for learning pointwise features in other network types.
  • 3.4 Summary: Graph-based networks have attracted increasing attention because of their inherent ability to handle irregular data, but spectral-domain extension across graph structures remains challenging.

4 3D OBJECT DETECTION AND TRACKING

This section reviews deep-learning methods for 3D object detection, tracking, and scene-flow estimation, focusing on detectors that predict oriented 3D bounding boxes from scene point clouds. Detection methods are organized into region proposal-based and single-shot approaches, with proposal-based methods further categorized by proposal-generation strategy.

  • 3D Object Detection: 3D object detectors take scene point clouds as input and output oriented 3D bounding boxes, broadly using region proposal-based or single-shot designs.The review also presents milestone methods chronologically and compares detectors on the KITTI test 3D detection benchmark.
  • Region Proposal-Based Methods: Region proposal-based detectors generate candidate regions, extract region-wise features, and classify proposals using multi-view, segmentation-based, or frustum-based generation.These categories respectively fuse view-specific features, focus proposals on foreground points, or derive 3D frustums from 2D detections.
  • Multi-view Based Methods: Multi-view methods fuse LiDAR, BEV, and image features but usually incur high computational cost; one method reached 99.1% recall at IoU 0.25 with 300 proposals but was too slow for practice.Subsequent work pursued multimodal fusion, high-recall proposals for small objects, and more robust multi-scale representations.
  • Segmentation-based Methods: Segmentation-based methods remove background points before proposal generation, achieving higher object recall than multi-view methods,, in crowded and heavily occluded scenes.Representative approaches use 2D or 3D segmentation to identify foreground points and refine proposals with criteria such as PointsIoU.
  • Frustum-based Methods: Frustum-based methods efficiently propose 3D object locations from 2D detections, but their sequential pipeline makes performance dependent on the underlying 2D image detector.F-PointNets learns frustum features with PointNet or PointNet++ [54], while a follow-up using Point-SENet and PointSIFT [141] improved robustness and performance on indoor and outdoor datasets,.

1. The ranking refers to the time of the submission: 12th June, 2020

Single-shot 3D object detectors directly classify objects and regress 3D bounding boxes without region proposals or post-processing, enabling high-speed inference. They comprise BEV-based, discretization-based, and point-based approaches, alongside other efficient designs.

  • Single-shot methods: Single-shot methods avoid region proposal generation and post-processing, directly predicting class probabilities and 3D bounding boxes for high-speed inference.They are categorized as BEV-based, discretization-based, and point-based methods.
  • BEV-based Methods: BEV-based detectors use regularized bird’s-eye-view representations and CNNs to estimate object locations and headings, while map-aware variants improve baseline performance but generalize poorly across point densities.The map-aware method significantly outperforms its baseline on TOR4D, and KITTI.
  • Discretization-based Methods: Discretization-based methods convert point clouds into regular representations for CNN-based detection; VoxelNet uses voxel features and a region proposal network, whereas 3D FCN methods extend 2D detection to 3D.Compared with [154], the 3D FCN-based method gains over 20% in accuracy.
  • Discretization-based Methods: PointPillars encodes pillar features as a pseudo-image for 2D detection, outperforming several fusion methods in Average Precision and reaching 62 fps on KITTI benchmarks.The compared methods include MV3D, RoarNet, and AVOD.
  • Point-based Methods: Point-based detectors operate directly on raw point clouds, with 3DSSD combining distance- and feature-based sampling to reduce costly feature-propagation and refinement stages.3DSSD further uses representative candidate points and an anchor-free regression head with a 3D centerness label.

2. The ranking refers to the time of the submission: 12th June, 2020

The section surveys 3D object detection, tracking, and scene-flow learning from point clouds, emphasizing geometric and spatiotemporal information for dynamic-scene understanding. It reports that region proposal-based detectors outperform single-shot methods on KITTI benchmarks while identifying long-range detection and image-texture use as unresolved challenges.

  • 3D Object Tracking: 3D object tracking estimates an object’s state across subsequent frames and can exploit point-cloud geometry to mitigate occlusion, illumination, and scale-variation issues.The task begins with object locations in the first frame and predicts later states.
  • 3D Object Tracking: The 3D Siamese tracker uses Kalman-filter candidates, shape-regularized representations, and cosine similarity to locate objects in subsequent frames.Shape completion regularization encodes the model and candidate into compact representations before similarity-based search.
  • 3D Scene Flow: FlowNet3D learns point-level and motion features to predict scene flow directly from consecutive point clouds, but struggles with direction errors and deformable non-static scenes.Scene flow maps each point x_i in one cloud to x′_i = x_i + d_i in the next cloud.
  • Dynamic Point Clouds: PointRNN, PointGRU, PointLSTM, and MeteorNet model dynamic point clouds by capturing spatial-temporal information and aggregating temporal neighbors [178].MeteorNet uses direct grouping and chained-flow grouping to determine temporal neighbors.
  • 3D Object Detection: Region proposal-based detectors outperform single-shot methods by a large margin on KITTI test 3D and bird’s-eye-view benchmarks.The comparison covers the KITTI test benchmarks, where region proposal methods are the more frequently investigated category.
  • 3D Object Detection: Existing 3D detectors have poor long-range capability, while fully exploiting image texture remains an open problem; multi-task learning is a future direction.These limitations and the future direction are stated for 3D object detection.

5 3D POINT CLOUD SEGMENTATION

3D point cloud segmentation must capture both global geometric structure and fine-grained point details, and is categorized by granularity into semantic, instance, and part segmentation.

  • Segmentation requires understanding global geometric structure alongside each point’s fine-grained details.
  • By granularity, segmentation comprises semantic segmentation at the scene level, instance segmentation at the object level, and part segmentation at the part level.

5.1 3D Semantic Segmentation

3D semantic segmentation methods are organized into projection-, discretization-, point-, and hybrid paradigms. Projection and discretization methods use intermediate regular representations, whereas point-based methods operate directly on irregular point clouds, each involving distinct efficiency and information-preservation trade-offs.

  • Overview: Semantic segmentation separates a point cloud into subsets according to point-level semantic meanings across four paradigms: projection-based, discretization-based, point-based, and hybrid methods.
  • Projection-based Methods: Projection-based methods transform point clouds into multi-view or spherical images, segment them in 2D, and project predictions back to the original points.Multi-view methods fuse reprojected pixel-wise scores across camera views, while spherical methods support fast or real-time LiDAR segmentation and can address domain shift.
  • Discretization-based Methods: Voxelization introduces discretization artifacts and information loss: high resolution increases memory and computation, whereas low resolution loses geometric detail.
  • Discretization-based Methods: Discretization-based methods convert point clouds into dense voxels or sparse lattices, enabling 3D convolutions and efficient sparse processing.Volumetric methods preserve neighborhood structure and scale across spatial sizes, while sparse convolutions reduce memory and computational costs for spatially sparse data.
  • Point-based Methods: Point-based networks directly process irregular point clouds, using shared MLPs and symmetric pooling in PointNet before extending toward local geometry and point interactions.Because point clouds are orderless and unstructured, standard CNNs cannot be applied directly; later methods add neighboring-feature pooling, attention, or local-global feature aggregation.

5.2 Instance Segmentation

3D instance segmentation must distinguish semantic categories while separating same-category instances, and existing methods fall into proposal-based and proposal-free families. Proposal-based methods detect regions before predicting masks, whereas proposal-free methods learn discriminative features and cluster points after semantic segmentation.

  • Overview: Instance segmentation is more challenging than semantic segmentation because it must distinguish semantic meanings and separate points belonging to different instances with the same meaning.Several milestone methods are summarized chronologically in Fig. 13.
  • Proposal-based methods: Proposal-based methods,,, jointly detect 3D regions and predict instance masks, producing good objectness but requiring costly multi-stage training and proposal pruning.These methods convert instance segmentation into 3D object detection and instance mask prediction.
  • Proposal-free methods: Proposal-free methods,,,, [237], [238],, omit object detection and cluster points after semantic segmentation using discriminative features and point grouping.They generally assume points from the same instance have similar features.
  • Proposal-free methods: Elich et al. achieve state-of-the-art performance on ScanNet by combining global BEV instance features with local point-cloud geometry and Mean-shift clustering.Directional predictions help determine instance boundaries, while non-maximum suppression groups voxels into instances.
  • Proposal-free methods: Proposal-free methods avoid computationally expensive region proposals but usually produce lower-objectness segments because they do not explicitly detect object boundaries.Their scalability and grouping strategies vary; SGPN [233], for example, is limited by the memory required for pairwise similarity matrices.

5.3 Part Segmentation

3D shape part segmentation is challenged by geometric variation among similarly labeled parts and varying part counts across semantically similar objects. Recent methods address these issues through multiscale voxel features, mesh-based convolution and graph refinement, and weakly supervised part representations.

  • Challenges: Part segmentation is difficult because similarly labeled parts vary geometrically, while semantically similar objects can contain different numbers of parts.
  • Voxel-based methods: VoxSegNet combines atrous residual blocks for multiscale sparse-volumetric features with attention-based reweighting and progressive feature fusion.Its Spatial Dense Extraction and Attention Feature Aggregation modules target fine-grained segmentation on 3D voxelized data under a limited solution.
  • Mesh-based methods: Shape Fully Convolutional Networks segment 3D meshes from three low-level geometric features and refine predictions using voting-based multilabel graph cuts.
  • Weakly supervised methods: Weakly supervised CoSegNet [252] learns compact part-specific representations through shape reconstruction, then decodes coordinates and branch features into binary point-membership predictions.The encoder-decoder method generalizes to large 3D shape collections of up to 5000+ shapes.

5.4 Summary

The summary situates existing methods through results on public benchmarks and highlights unresolved challenges in point-cloud representation, large-scale segmentation, and dynamic spatio-temporal learning.

  • 5.4 Summary: Table 5 reports existing methods on the S3DIS, Semantic3D, ScanNet, and SemanticKITTI public benchmarks.
  • 5.4 Summary: Projection- and discretization-based methods reuse mature 2D architectures, but projection loses 3D information while discretization incurs cubically increasing computation and memory costs.
  • 5.4 Summary: Most approaches,,,, [207] target small point clouds, motivating efficient segmentation of immense, large-scale sensor point clouds.
  • 5.4 Summary: A handful of works [178],, have begun learning spatio-temporal information from dynamic point clouds.

6 CONCLUSION

The paper surveys state-of-the-art deep learning methods for 3D understanding, organizes them taxonomically, compares their performance, and identifies method trade-offs and future research directions.

  • 6 CONCLUSION: The survey covers 3D shape classification, 3D object detection and tracking, and 3D scene and object segmentation.It presents a comprehensive taxonomy and performance comparison, discusses methods’ merits and demerits, and lists potential research directions.
Loading 1912.12033v2…