PaperlayerView exact revision on arXiv

Source-linked AI summary

Review: deep learning on 3D point clouds

Saifullahi Aminu Bello, Shangshu Yu, Cheng Wang

arXiv:2001.06280v1cs.CV

TL;DR

Deep learning is difficult to apply directly to point clouds because their irregular, unstructured, and unordered form conflicts with conventional convolutional processing. This survey reviews structured-grid and direct point-cloud methods, datasets, and applications, concluding that direct methods avoid conversion artifacts and reduce computational demands while remaining incomplete for instance segmentation, direct detection, and large scenes.

  • Problem

    Point clouds provide 3D information but are difficult for conventional deep learning because their structure does not match regular-grid convolutional networks.

  • Method

    The paper surveys point-cloud challenges, structured-grid preprocessing, direct deep-learning methods, benchmark datasets, and applications in classification, segmentation, and detection.

  • Results

    Direct point-cloud methods avoid conversion artifacts and mitigate the need for higher computational cost, while PointNet achieves state-of-the-art performance on several benchmark datasets.

  • Takeaways & Limitations

    The review presents direct point-cloud learning as an active approach for 3D visual perception across classification, segmentation, and detection tasks.

  • Takeaways & Limitations

    Instance segmentation, direct point-cloud detection, and learning on large scenes remain relatively underexplored.

Abstract

from arXiv · show

Point cloud is point sets defined in 3D metric space. Point cloud has become one of the most significant data format for 3D representation. Its gaining increased popularity as a result of increased availability of acquisition devices, such as LiDAR, as well as increased application in areas such as robotics, autonomous driving, augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision, becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes use of deep learning for its processing directly very challenging. Earlier approaches overcome this challenge by preprocessing the point cloud into a structured grid format at the cost of increased computational cost or lost of depth information. Recently, however, many state-of-the-arts deep learning techniques that directly operate on point cloud are being developed. This paper contains a survey of the recent state-of-the-art deep learning techniques that mainly focused on point cloud data. We first briefly discussed the major challenges faced when using deep learning directly on point cloud, we also briefly discussed earlier approaches which overcome the challenges by preprocessing the point cloud into a structured grid. We then give the review of the various state-of-the-art deep learning approaches that directly process point cloud in its unstructured form. We introduced the popular 3D point cloud benchmark datasets. And we also further discussed the application of deep learning in popular 3D vision tasks including classification, segmentation and detection.

1. Introduction

Point clouds provide a practical 3D representation that preserves depth and relative positions, while increasing sensing availability has enabled their use in deep-learning-based visual intelligence. The review surveys point-cloud challenges, structured-grid approaches, direct-processing methods, datasets, and vision tasks.

  • Point clouds represent 3D geometry and can preserve depth and relative positions lost by 2D images.
  • Their use is expanding with LiDAR and time-of-flight depth cameras, alongside applications in robotics, autonomous vehicles, and augmented and virtual reality.
  • Deep learning became more feasible as point-cloud data availability increased, replacing earlier reliance on handcrafted features.
  • The paper reviews point-cloud challenges, early structured-grid solutions, and recent methods that process raw point clouds directly.

2. Challenges of deep learning on point clouds

Point clouds are difficult for conventional deep learning because they are irregular, unstructured, and unordered, whereas convolutional neural networks expect ordered data on regular grids. Early methods therefore converted point clouds into structured representations, while newer methods operate directly on raw points.

  • Point clouds contain occlusion, noise, outliers, and misalignment, with irregularity, unstructuredness, and unorderedness being the pronounced challenges.
  • Irregularity: Irregular sampling produces dense regions alongside sparse regions across an object or scene.
  • Unstructured: Unstructured point clouds lack a regular grid and fixed neighboring distances, unlike image pixels.
  • Unorderedness: Point-cloud storage order does not change the represented scene because the points form a set.
  • Because CNN convolution requires ordered, regular, structured-grid data, approaches either convert point clouds or process them directly.

3. Structured grid based learning

Structured-grid methods adapt deep learning to point clouds by converting them into voxels or multiview images, enabling conventional convolutions but introducing distinct trade-offs. Voxel methods incur memory and resolution limitations, whereas multiview methods are reported to perform better because they use mature 2D CNNs and avoid voxelization artifacts.

  • Structured-grid approaches broadly comprise voxel-based and multiview-based methods that convert point clouds before learning.
  • 3.1. Voxel based: Voxel methods convert point clouds into fixed-size 3D grids and apply 3D convolutional, pooling, and fully connected layers.
  • 3.1. Voxel based: Voxelization can waste computation on unoccupied regions, consume substantial memory, limit resolution typically to 32^3–64^3, and introduce artifacts.
  • 3.1. Voxel based: Adaptive voxel representations reduce voxelization challenges but remain limited to 256^3 voxels and are more complex than regular grids.
  • 3.2. multiview based: Multiview methods project point clouds into collections of 2D images and apply established 2D CNN techniques.
  • 3.2. multiview based: Multiview methods are reported to outperform voxel methods because they use mature 2D techniques and avoid voxelization quantization artifacts.

4. Deep learning directly on raw point cloud

Direct point-cloud methods avoid structured-grid conversion by operating on unordered points, with PointNet providing the foundation and later methods modeling local structure through hierarchical operations or graphs.

  • 4.1. PointNet: PointNet applies deep learning directly to unstructured point clouds and underpins many subsequent approaches.Its input is raw point cloud data rather than a structured grid.
  • 4.1. PointNet: PointNet uses symmetric MLP and maxpooling operations to produce feature descriptors for recognition and segmentation.The maxpooling operation makes the output insensitive to point ordering.
  • 4.1. PointNet: PointNet achieves state-of-the-art performance but does not model local dependencies, limiting its capture of local structure.Its winner-takes-all global maxpooling is also reported as susceptible to targeted adversarial attack.
  • 4.2. Approaches with local structure computation: Local-structure methods commonly sample representative points, group neighboring points, and map each neighborhood into higher-level features with an MLP.Grouping uses k-nearest neighbors or radius-based ball queries, while mapping often follows PointNet-style symmetric functions.
  • 4.2. Approaches with local structure computation: PointNet++ hierarchically applies PointNet to sampled local regions, reducing point resolution as depth increases.It uses farthest-point sampling for centroids and ball queries for neighboring points.
  • 4.2.3. Graph based: Graph approaches represent points as nodes and edges, with Dynamic Graph CNN updating k-nearest-neighbor edges after each edge-convolution layer.Other graph methods reduce point resolution as networks deepen, while kd-tree designs capture hierarchical representations without overlapping receptive fields at equal depths.

5. Benchmark Datasets

Public point-cloud benchmarks cover virtual and real scenes across classification, segmentation, registration, detection, and related 3D vision tasks.

  • 5. Benchmark Datasets: Benchmark datasets provide shared evaluation resources with ground-truth labels for comparing diverse point-cloud approaches.They include both virtual and real scenes and support several task categories.
  • 5.1. 3D Model Datasets: ModelNet40 contains 12,311 CAD models across 40 man-made object categories, split into 9,843 training and 2,468 testing shapes.ModelNet10 is a 10-category subset with 3,991 training and 908 testing shapes.
  • 5.1. 3D Model Datasets: ShapeNet provides nearly 3,000,000 indexed models, including 220,000 models classified into 3,135 categories; ShapeNetCore contains nearly 51,300 unique models.Its annotations include semantic categories, parts, symmetry planes, physical sizes, and other metadata.
  • 5.2. Indoor Scene Datasets: Indoor-scene benchmarks include S3DIS, ScanNet, NYUDv2, SceneNN, SUN3D, and Matterport3D with RGB-D or laser-scanned data and semantic annotations.S3DIS contains over 215 million points with instance-level semantic labels across 13 categories, while ScanNet provides 3D object and semantic-segmentation labels.
  • 5.2. Indoor Scene Datasets: 3DMatch provides correspondence labels for point-cloud registration across 62 scenes, with 54 training and 8 testing scenes.It combines several existing RGB-D reconstruction datasets.
  • 5.3. Outdoor Scene Datasets: KITTI supports autonomous-driving research including 3D detection and tracking using cameras, a Velodyne HDL-64E scanner, and GPS/IMU.Other outdoor datasets include ASL sequences and iQmulus urban scenes with 50 semantic classes and 300 million points.

6. Application of deep learning in 3D vision tasks

Deep learning methods are applied to point-cloud classification, part and semantic segmentation, and object detection, with evaluations organized around task-specific benchmark datasets.

  • 6. Application of deep learning in 3D vision tasks: The review evaluates classification on ModelNet40, part segmentation on ShapeNet, and semantic segmentation on S3DIS.These tasks are presented as applications of the methods reviewed for direct and structured point-cloud processing.
  • 6.1. Object classification: Classification assigns an input point cloud to an object category, using learned features with fully connected networks or other classifiers such as SVM.The review presents a timeline of point-based deep-learning performance on ModelNet40.
  • 6.2. Segmentation: Part segmentation assigns each point of an object to a part category, often combining global descriptors with point features before MLP classification.Other methods propagate global descriptors to high-resolution predictions through interpolation or deconvolution.
  • 6.2. Segmentation: Semantic segmentation assigns every point to a class, with S3DIS and ScanNet serving as popular evaluation datasets.The review reports state-of-the-art performances on both datasets.
  • 6.2. Segmentation: Instance segmentation uniquely separates multiple objects of the same class and receives less attention than part and semantic segmentation.Several state-of-the-art works build instance-segmentation systems on PointNet or PointNet++ backbones.
  • 6.3. Object detection: 3D object detection recognizes and localizes multiple objects with bounding boxes, but proposal generation is harder in 3D because search and proposal regions are three-dimensional.Structured-grid methods such as Vote3D and Vote3Deep use extensive sliding-window operations that are computationally expensive.

7. Summary and Conclusion

The review surveys the shift from handcrafted and structured-grid processing toward direct point-cloud deep learning, while identifying underexplored areas in detection, instance segmentation, and large-scale scenes.

  • 7. Summary and Conclusion: Increasing point-cloud availability and applications in autonomous vehicles, robotics, AR, and VR motivate fast and efficient processing algorithms.Earlier methods relied on handcrafted features because data were scarce and deep learning was less popular.
  • 7. Summary and Conclusion: Structured-grid conversion can lose depth information, introduce conversion artifacts, and require higher computational cost.Direct point-cloud learning avoids conversion artifacts and mitigates the need for higher computational cost.
  • 7. Summary and Conclusion: PointNet is a foundational direct-processing method, but later approaches improve local-structure modeling through sampling, grouping, and learned neighborhood mappings.The review identifies these operations as common components of local-structure methods.
  • 7. Summary and Conclusion: Instance segmentation remains largely uncharted compared with classification, part segmentation, and semantic segmentation.The review also notes that these latter tasks have shown good performance.
  • 7. Summary and Conclusion: Few methods detect objects directly on point clouds, while scaling to large scenes remains underexploited because scenes are often cut into smaller pieces.At the time of the review, only a few works explored deep learning on large-scale 3D scenes.
Loading 2001.06280v1…