PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-based 3D Point Cloud Classification: A Systematic Survey and Outlook

Huang Zhang, Changshuo Wang, Shengwei Tian, Baoli Lu, Liping Zhang, Xin Ning, Xiao Bai

arXiv:2311.02608v1cs.CV

TL;DR

Deep learning-based point-cloud classification remains challenging because 3D point clouds are irregular, disordered, and sparse. This paper surveys representations, datasets, classification methods, evaluations, and future directions, concluding that original-point-cloud methods have performance advantages but often require more complex models.

  • Problem

    Point-cloud classification remains challenging because 3D point clouds are irregular, disordered, and sparse, while existing reviews need updating with recent methods such as transformers.

  • Method

    The paper comprehensively surveys point-cloud classification, organizing methods into multi-view-based, voxel-based, point-cloud-based, and polymorphic fusion-based categories, then compares their performance.

  • Results

    The survey compares existing methods and reports that original-point-cloud methods have certain algorithm-performance advantages, although their network models are more complicated.

  • Takeaways & Limitations

    The classification and comparison provide a basis for exploring methods according to practical needs, while simple point-cloud-based methods are identified as a future research trend.

Abstract

from arXiv · show

In recent years, point cloud representation has become one of the research hotspots in the field of computer vision, and has been widely used in many fields, such as autonomous driving, virtual reality, robotics, etc. Although deep learning techniques have achieved great success in processing regular structured 2D grid image data, there are still great challenges in processing irregular, unstructured point cloud data. Point cloud classification is the basis of point cloud analysis, and many deep learning-based methods have been widely used in this task. Therefore, the purpose of this paper is to provide researchers in this field with the latest research progress and future trends. First, we introduce point cloud acquisition, characteristics, and challenges. Second, we review 3D data representations, storage formats, and commonly used datasets for point cloud classification. We then summarize deep learning-based methods for point cloud classification and complement recent research work. Next, we compare and analyze the performance of the main methods. Finally, we discuss some challenges and future directions for point cloud classification.

I. INTRODUCTION

Point cloud classification is important but difficult because point clouds are irregular, disordered, and sparse. This survey organizes recent deep learning methods, compares their performance, and discusses current limitations and future directions.

  • Point cloud classification assigns markers to points to identify overall or part properties, and this paper focuses on overall point-cloud attributes.
  • Before PointNet, point-cloud processing relied on hand-designed feature rules and machine-learning classifiers because deep learning could not directly process unordered point clouds.
  • PointNet directly processes point clouds, after which direct point-cloud methods gradually became dominant.
  • Point clouds represent 3D scenes with rich information, but their irregularity, disorder, and sparsity make classification challenging.
  • The paper introduces 3D data and datasets, reviews recent methods, evaluates performance, and analyzes challenges and future trends.
  • The survey classifies methods into multi-view-based, voxel-based, point-cloud-based, and polymorphic fusion-based categories, with further subdivision of point-cloud methods.

II. 3D DATA

3D data can be represented as point clouds, meshes, voxels, or multi-view images, while point-cloud storage uses numerous ASCII, binary, and hybrid formats. Each representation and format offers specific capabilities alongside practical trade-offs.

  • A. 3D data representation: A point cloud is a collection of 3D points expressing spatial distribution and surface characteristics, with coordinates, color, and surface-normal information.
  • A. 3D data representation: A mesh represents 3D data as points with local relationships, commonly using triangular patches connected by edges.
  • A. 3D data representation: Voxels represent regularly sampled 3D space and can improve computing efficiency and reduce random-memory access after voxelization.Voxelization inevitably causes some information loss.
  • A. 3D data representation: Multi-view representation renders a 3D object from multiple viewpoints, with perspective selection and viewpoint fusion as its main challenges.
  • B. Point cloud data storage format: Point-cloud files differ primarily in ASCII versus binary storage; common formats include FLS, PCD, LAS, PLY, FBX, and E57.
  • B. Point cloud data storage format: OBJ stores geometry, normals, color, and texture in a primarily ASCII text format, while LAS mainly stores LIDAR data in binary form.
  • B. Point cloud data storage format: PLY supports ASCII and binary versions and stores polygonal 3D data with attributes such as color, transparency, normals, textures, and confidence.
  • B. Point cloud data storage format: E57 is vendor-neutral and stores point clouds, attributes, images, and metadata using both ASCII and binary codes.

C. 3D point cloud public datasets

The paper surveys public datasets used for point cloud classification, spanning synthetic CAD objects, real-world scans, urban objects, and robustness benchmarks.

  • ModelNet40 contains synthetic CAD objects in 40 categories, with 9843 training and 2468 testing shapes.
  • ModelNet-C contains 185,000 point clouds with 15 damage types and five severity levels for robustness benchmarking.
  • ModelNet10 is a ModelNet40 subset containing 10 classes, divided into 3991 training and 908 testing shapes.
  • ShapeNet provides large-scale CAD repositories, including ShapeNetCore with nearly 51,300 models across 55 annotated object categories.
  • Real-world datasets include Sydney Urban Objects, ScanNet, and ScanObjectNN, with ScanObjectNN challenging classification through background, missing parts, and deformations.

III. POINT CLOUD CLASSIFICATION METHOD BASED ON DEEP LEARNING

The survey organizes deep learning point cloud classification methods into major representation-based families and reviews developments in multi-view learning. Multi-view methods improve classification over hand-designed features but remain limited in information use and large-scale geometric handling.

  • The survey supplements recent research and divides classification methods into multi-view, voxel, point-cloud, and polymorphic fusion categories.
  • Multi-view-based methods: Multi-view methods project 3D objects into multiple views, extract view features, and fuse them for classification.
  • Multi-view-based methods: MVCNN combines information from multiple rendered views into a compact shape descriptor, improving recognition over a single-view CNN.
  • Multi-view-based methods: Later methods address efficiency, view relationships, pooling information loss, and viewpoint selection through GPU acceleration, view grouping, clustering, and differentiable viewpoint prediction.
  • Multi-view-based methods: Multi-view methods outperform traditional manual feature extraction but struggle to fully exploit information, scale to large scenes, and model inherent 3D geometric relationships.

B. Voxel-based methods

Voxel-based methods convert point clouds into volumetric structures processed by 3D CNNs, with octrees and Kd-trees improving efficiency. Their main limitations are information loss, sparsity, and boundary-related loss of local features.

  • Voxel-based methods convert point clouds into voxel grids and use 3D CNNs to classify the resulting volumetric representation.
  • VoxNet represents 3D information with a volumetric occupancy grid and is described as the earliest voxel-based 3DCNN model.
  • OctNet and Kd-network reduce memory consumption and improve computational efficiency by exploiting sparse structure through octrees or Kd-trees.
  • Octree and Kd-tree methods cannot fully use local features because voxel boundary values affect representation, which reduces accuracy.
  • Voxel-based models address point cloud disorder and unstructuredness, but sparsity and incomplete information limit efficiency and prevent full information use.

C. Point cloud-based method

Point-cloud-based methods directly process discrete points, with feature aggregation transferring information between them. PointNet pioneered global aggregation by transforming, independently encoding, and max-pooling point features.

  • Feature aggregation operators transfer information among discrete points and are divided into local and global aggregation.The survey organizes point-cloud-based methods from this feature-aggregation perspective.
  • Point-cloud-based methods process the original points directly rather than converting them into alternate 3D representations.
  • PointNet directly accepts point clouds, applies T-Net transformation, shares fully connected layers across points, and uses max pooling for global features.PointNet is described as a pioneering global feature-aggregation method.

1) Local feature aggregation:

Local feature aggregation methods model neighborhoods and geometric relationships to improve point-cloud representations. The survey covers point-based, convolutional, graph-based, and specialized local-geometry approaches.

  • Point-based methods: PointNet++ hierarchically samples neighborhood centroids, groups local subsets, and applies PointNet to learn relationships among neighboring points.Its layers consist of sampling, grouping, and PointNet operations.
  • Point-based methods: PointNeXt improves PointNet++ with revised training, separable MLPs, and inverted residual bottlenecks.
  • Point-based methods: PointWeb connects neighboring points and uses Adaptive Feature Adjustment to model information transfer between points.The module adaptively adjusts local point features.
  • Point-based methods: RandLA-Net uses random sampling and efficient local aggregation to enlarge receptive fields while reducing memory and computational costs.The approach may discard important features from sparse points.
  • Point-based methods: PointMLP omits complex local geometry extraction and instead uses a lightweight geometric affine module, improving inference speed.
  • Specialized representations: RepSurf represents point clouds with triangular or umbrella representative surfaces and can be inserted into classification or segmentation frameworks.
  • Convolution-based methods: Convolutional methods adapt CNNs to irregular point clouds through point convolutions, dynamic filters, continuous volume convolutions, and related operators.PointCNN highlights the importance of developing local structures for point-cloud classification networks.
  • Graph-based methods: Graph-based methods represent points as graph vertices and use point relationships or spatially efficient pooling to aggregate features.ECC uses distances as edge weights but has high computational cost; Grid-GCN reduces theoretical time complexity through grid-space efficiency.

2) Global feature aggregation:

Global feature aggregation methods capture broader point-cloud context using transformers, global modules, recurrent networks, and related attention mechanisms. The survey emphasizes architectures designed to connect local and global information efficiently.

  • Transformer-based methods: Point Transformer processes unordered point sets, extracts local and global features, ranks local features, and links them through local-global attention.
  • Transformer-based methods: Point TnT uses a two-stage design so a single point and a point set can attend to each other while addressing inefficient self-attention growth.
  • Transformer-based methods: PCT introduces offset attention with an implicit Laplacian operator and normalization refinement while retaining permutation invariance.
  • Transformer-based methods: Point-BERT masks local point-cloud blocks and applies a backbone Transformer, extending BERT-style learning to point clouds.
  • Transformer-based methods: Point-MAE is a masked-autoencoder approach for point-cloud self-supervised learning that addresses location-information leakage and uneven information density.
  • Transformer-based methods: VoxSeT combines voxel-set attention with Transformer processing and parallel point-cluster management at linear complexity.
  • Global module-based methods: PointASNL combines adaptive sampling with local-nonlocal processing to handle noisy point clouds and reduce learning sensitivity.
  • RNN or LSTM-based methods: Recurrent approaches use contextual sequence modeling, including PointNet extensions for large-scale scenes and CNN-DQN-RNN feature integration.

D. Polymorphic Fusion-based methods

Polymorphic fusion methods combine point, voxel, grid, or image-derived representations. The surveyed approaches seek complementary geometric information, including sparse attention, zero-shot transfer, and 2D-to-3D correspondence.

  • PointGrid mixes points and grids through convolution blocks, pooling, and fully connected layers for point-cloud representation.
  • PVT combines point- and voxel-based modeling with sparse-window and relative-attention modules, and is reported to excel in classification accuracy.
  • PointCLIP projects point clouds into multi-view depth maps without rendering and transfers pretrained CLIP knowledge for zero-shot recognition.An inter-view adapter extracts global features.
  • CrossPoint learns 2D-to-3D correspondence by maximizing invariant-space agreement between point clouds and corresponding rendered 2D images.
  • The survey identifies direct point-cloud processing as advantageous over multi-view and voxel methods because it can use point-cloud information fully.It also projects wider future use of Transformer-based methods.

IV. EVALUATION

The evaluation section defines common metrics for point cloud classification and notes that accuracy is the key performance indicator. It also introduces overall, average, and mean intersection-over-union measures for comparing methods.

  • Accuracy, space complexity, and execution time evaluate point cloud classification methods, with accuracy identified as the key indicator.
  • Accuracy, precision, recall, and intersection-over-union are commonly used to evaluate classification accuracy.
  • Accuracy is the ratio of correctly predicted samples to total predicted samples.
  • Precision measures the proportion of true positive samples predicted as positive, while recall measures predicted positive samples relative to all true positive samples.
  • Intersection-over-union measures the ratio between the intersection and union of predicted and true values.
  • Overall accuracy, average accuracy, and mean intersection-over-union are used to measure current point cloud classification accuracy.

V. FUTURE TRENDS

Future work emphasizes balancing classification accuracy with efficiency, improving methods for complex outdoor scenes, and simplifying models without discarding point-cloud information. The section also calls for more innovative approaches and network optimization.

  • Future methods should achieve high accuracy while remaining efficient, because existing approaches tend to prioritize one objective over the other.
  • Outdoor point cloud scenes have complex, changing information structures, so their classification efficiency and accuracy require further improvement.
  • Original-point-cloud methods show performance advantages but use more complicated network models because their inputs preserve information integrity.
  • Simple point-cloud-based methods are identified as a future research trend.
  • Future research should pursue more innovative methods rather than mainly incremental improvements.
  • Network architecture optimization should reduce computational complexity and memory usage while handling complex, irregular point clouds.

VI. CONCLUSION

The paper surveys deep learning-based point cloud classification, covering data characteristics, representations, datasets, method categories, performance comparisons, current problems, and future directions.

  • The paper provides a comprehensive survey and discussion of recent deep learning-based point cloud classification methods.
  • It introduces point clouds, their applications, characteristics, and processing difficulties.
  • It summarizes 3D data representations, point cloud storage formats, and commonly used classification datasets.
  • Deep learning methods are classified into multi-view-based, voxel-based, point-cloud-based, and polymorphic fusion-based categories.
  • The paper compares existing methods, identifies current problems, and prospects future research directions.
Loading 2311.02608v1…