PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey

Aoran Xiao, Jiaxing Huang, Dayan Guan, Xiaoqin Zhang, Shijian Lu, Ling Shao

arXiv:2202.13589v3cs.CVcs.AI

TL;DR

Large-scale point-cloud annotation is costly, and unsupervised point-cloud learning still lacks a systematic survey amid challenges in data, backbones, and benchmarking. This paper surveys methods and evaluates them across tasks; its benchmarks report gains over scratch training, though improvements remain limited.

  • Problem

    Point-cloud learning requires laborious large-scale annotation, and the field lacks a systematic survey while facing constraints in data, unified backbones, generalizable approaches, and benchmarks.

    Source: §1 · pp. 1–2

  • Method

    The paper surveys point-cloud URL methods, organizes them by pretext-task approach, and benchmarks representative methods across multiple 3D tasks.

    Source: §1 · p. 2, §8 · p. 16

  • Results

    Benchmarks report improving object-classification results, shrinking part-segmentation gaps, and gains over scratch across multiple scene-level tasks, with improvements still limited.

    Source: §6.2.1 · p. 14, §6.2.2 · p. 14, §6.3 · p. 15

  • Takeaways & Limitations

    The survey aims to provide a foundation for future point-cloud URL research and identifies potential directions for further work.

    Source: §8 · p. 16

    Diverse, non-unified 3D backbones impede scalable network design and efficient deployment across new tasks.

    Source: §7 · p. 16

Abstract

from arXiv · show

Point cloud data have been widely explored due to its superior accuracy and robustness under various adverse situations. Meanwhile, deep neural networks (DNNs) have achieved very impressive success in various applications such as surveillance and autonomous driving. The convergence of point cloud and DNNs has led to many deep point cloud models, largely trained under the supervision of large-scale and densely-labelled point cloud data. Unsupervised point cloud representation learning, which aims to learn general and useful point cloud representations from unlabelled point cloud data, has recently attracted increasing attention due to the constraint in large-scale point cloud labelling. This paper provides a comprehensive review of unsupervised point cloud representation learning using DNNs. It first describes the motivation, general pipelines as well as terminologies of the recent studies. Relevant background including widely adopted point cloud datasets and DNN architectures is then briefly presented. This is followed by an extensive discussion of existing unsupervised point cloud representation learning methods according to their technical approaches. We also quantitatively benchmark and discuss the reviewed methods over multiple widely adopted point cloud datasets. Finally, we share our humble opinion about several challenges and problems that could be pursued in future research in unsupervised point cloud representation learning. A project associated with this survey has been built at https://github.com/xiaoaoran/3d_url_survey.

1 INTRODUCTION

Point-cloud networks often depend on costly human annotation, while unsupervised representation learning offers a way to learn from unlabelled data. This survey reviews the field, benchmarks methods, and outlines future directions.

1 INTRODUCTION

  • URL pre-trains networks on unannotated point clouds, then transfers the learned representations as initialization for downstream fine-tuning with a small amount of labeled data.Source: Figure 1 · p. 1
  • Large-scale point-cloud annotation is laborious and time-consuming, while URL still faces shortages in data, unified backbones, generalizable approaches, and comprehensive benchmarks.Source: §1 · pp. 1–2
  • Existing methods are grouped into four pretext-task families: data generation, global and local contexts, multimodality data, and local descriptors.Source: §1 · p. 2
  • The survey reviews unsupervised point-cloud learning across datasets, architectures, methods, benchmarks, and future research directions.Source: §1 · p. 2

2 BACKGROUND

This background defines the learning and transfer terminology, then introduces object- and scene-level point-cloud tasks used to evaluate learned representations. It also positions the paper among surveys of supervised point-cloud learning and unsupervised learning in other modalities.

2.1 Basic concepts

  • Unsupervised representation learning learns meaningful representations from unlabelled data for transfer to downstream tasks; the survey also distinguishes supervised, semi-supervised, pre-training, and transfer learning.A point cloud is a set of point vectors with 3D coordinates and optional attributes such as RGB values, LiDAR intensity, or normals.Source: §2.1 · pp. 2–3

2.2.1 Object classification

  • Object classification assigns each point cloud object to a predefined category and is commonly evaluated with overall accuracy and mean class accuracy.Overall accuracy averages accuracy across test instances; mean class accuracy averages accuracy across object classes.Source: §2.2.1 · p. 3

2.2.2 Object part segmentation

  • Object part segmentation assigns a part-category label to each point, with mean Intersection over Union the most widely adopted metric.The overall IoU averages instance IoUs, while categorywise IoU averages instances within each category.Source: §2.2.2 · p. 3, Figure 3 · p. 3

2.2.3 3D object detection

  • 3D object detection localizes objects in 3D space with bounding boxes and is widely evaluated using average precision.The illustrations show boxes in ScanNet-V2 and KITTI.Source: §2.2.3 · p. 3, Figure 4 · p. 3

2.2.4 3D semantic segmentation

  • 3D semantic segmentation assigns a category to every point in complex scene-level point clouds and commonly uses OA, mIoU, and mAcc.The S3DIS illustration pairs a scene sample with its category-labeled ground truth.Source: §2.2.4 · pp. 3–4, Figure 5 · p. 3

2.2.5 3D instance segmentation

  • 3D instance segmentation delineates distinct objects in scene-level point clouds and gives each object a unique identity beyond its semantic category.Mean Average Precision is widely used for quantitative evaluation.Source: §2.2.5 · p. 4, Figure 6 · p. 4

2.3 Relevant surveys

  • The survey describes itself as the first comprehensive review of unsupervised point-cloud learning, distinct from prior surveys of supervised point-cloud learning or unsupervised learning in other modalities.It identifies reviews spanning 3D data, point-cloud segmentation and multiple point-cloud tasks, as well as 2D vision, NLP, graph learning, and small-data learning.Source: §2.3 · p. 4

3 POINT CLOUD DATASETS

Prior work draws unsupervised point-cloud representations mainly from synthetic object datasets and real-scene datasets, while using task-specific datasets for downstream evaluation. The survey identifies limited dataset size and scene variety as an outstanding constraint.

Dataset overview

  • Existing work learns representations mainly from synthetic object datasets such as ModelNet and ShapeNet or real-scene datasets such as ScanNet and KITTI.Task-specific evaluation datasets include ScanObjectNN and ModelNet40 for classification, ShapeNetPart for part segmentation, S3DIS and ScanNet for semantic segmentation, and SUN RGB-D and ONCE for object detection.Source: §3 · p. 4

Synthetic object datasets

  • ModelNet40 contains 13,834 objects in 40 categories, while ShapeNetPart contains 16,881 objects with 50 part categories across 16 categories.ModelNet is generated by uniformly sampling CAD models; ShapeNet objects are complete, aligned, and without occlusion or background.Source: §3 · p. 4

Real-scene datasets

  • ScanObjectNN contains 2,902 real indoor-scene objects across 15 categories, with noise, background points, occlusions, holes, and objects that are not axis-aligned.S3DIS covers over 215 million points across six indoor areas, while ONCE provides one million LiDAR scenes, including 560 unlabelled sequences for unsupervised learning.Source: §3 · pp. 4–5

Dataset limitations

  • Public point-cloud datasets remain limited in both size and scene variety compared with image and text datasets, and large-scale, high-quality point-cloud data are needed.The survey contrasts this scarcity with ImageNet's more than 10 million images and the billions of words available in BooksCorpus and English Wikipedia.Source: §3 · p. 5

4 COMMON DEEP ARCHITECTURES

Point-cloud URL lacks a universal backbone, partly because unordered point sets have no regular, universal structure. The survey focuses on five architecture families frequently used for URL.

4.1 Point-based deep architectures

  • Point-based networks process raw clouds directly, using point-wise MLP features and symmetric aggregation; PointNet++ adds local, hierarchical geometry and VoteNet builds on it for detection.VoteNet uses Hough voting to form box proposals.Source: §4.1 · p. 5

4.2 Graph-based deep architectures

  • Graph-based networks connect points through neighborhood edges; DGCNN uses dynamic feature-space EdgeConv within a PointNet structure to learn global shape and semantic features.Graph convolution shares weights and extracts localized features that capture neighboring-point dependencies.Source: §4.2 · pp. 5–6

4.3 Sparse voxel-based deep architectures

  • Sparse voxel methods represent clouds with sparse tensors and Rulebooks to reduce the redundancy or accuracy loss of dense voxel processing; SR-UNet encoders transfer to downstream tasks.Minkowski Engine provides generalized sparse convolution and autodifferentiation for sparse tensors.Source: §4.3 · p. 6, Figure 9 · p. 6

4.4 Spatial CNN-based deep architectures

  • Spatial CNNs extend grid CNNs to irregular points: continuous kernels weight neighbors by spatial distribution, while discrete kernels use fixed grid offsets.RS-CNN learns dynamic convolution weights from local geometric topology relations.Source: §4.4 · p. 6, Figure 10 · p. 6

4.5 Transformer-based deep architectures

  • Point-cloud Transformers stack multi-head self-attention and feed-forward blocks; pre-trained encoders can be fine-tuned for classification and semantic segmentation.The survey illustrates the Point-BERT Transformer architecture used for unsupervised pre-training.Source: §4.5 · p. 6, Figure 11 · p. 6

5 UNSUPERVISED POINT CLOUD REPRESENTA-

The survey organizes point-cloud URL methods by their pretext tasks into four broad families, providing a systematic review of existing approaches.

TION LEARNING

  • The taxonomy groups methods into generative-based, context-based, multiple modal-based, and local descriptor-based families.The grouping is based on the methods’ pretext tasks.Source: TION LEARNING · p. 7
  • The review sorts existing methods into the taxonomy and introduces them systematically in the following subsections.Source: TION LEARNING · p. 7
  • The survey’s organizing framework covers the existing point-cloud URL methods through their pretext tasks.Source: TION LEARNING · p. 7

5.1 Generation-based methods

Generation-based URL methods learn from point clouds themselves, using reconstruction, adversarial generation, up-sampling, or completion as pretext tasks without human annotations.

5.1 Generation-based methods

  • Generation-based methods use point clouds as ground truth and include self-reconstruction, GAN generation, up-sampling, and completion.The four tasks respectively reproduce the input, generate fake clouds, create denser similar shapes, or predict missing parts.Source: §5.1 · p. 7

5.1.1 Learning through point cloud self-reconstruction

  • Autoencoders compress a cloud into a codeword and decode it to the original input; permutation-invariant Chamfer or Earth Mover’s Distance losses measure reconstruction similarity.Reconstruction trains the encoding to regenerate the input, encouraging low-dimensional representations that ignore insignificant noise.Source: §5.1.1 · p. 7, Figure 12 · p. 7
  • Self-reconstruction also covers hierarchical and local-global geometry, point-distribution modeling, and multi-resolution representations.Examples include FoldingNet’s canonical-grid folding, PointFlow’s shape and conditional point distributions, and reconstruction across resolutions to learn sampling-invariant features.Source: §5.1.1 · pp. 7–8

5.1.2 Learning through point cloud GAN

  • Point-cloud GANs train a generator to produce samples and a discriminator to distinguish real from generated clouds; the discriminator’s learned features transfer to downstream tasks.The reviewed approaches include voxel, raw-point, hierarchical, and graph-based GANs,, [88],.Source: §5.1.2 · p. 8, Figure 13 · p. 8

5.1.3 Learning through point cloud up-sampling

  • Up-sampling learns shape geometry by producing denser point sets with similar geometry, and learned encoders or discriminators can initialize downstream models.PU-GAN upsamples points over surface patches; transfer-learning performance for this task has not been well studied.Source: §5.1.3 · pp. 8–9, Figure 14 · p. 9

5.1.4 Learning through point cloud completion

  • Completion predicts arbitrary missing parts from the remaining points, learning object geometry and semantics without annotations for transfer to downstream tasks.Occlusion completion with encoder-decoder models has been used to initialize classification, part segmentation, and semantic segmentation.Source: §5.1.4 · p. 9, Figure 15 · p. 9
  • Completion methods span voxel and raw-point autoencoders, including SA-Net’s skip-attention transfer of local geometry to the decoder.Point-BERT instead masks discrete patch tokens and trains a Transformer to recover the complete tokens.Source: §5.1.4 · p. 9

5.2 Context-based methods

Context-based methods use annotation-free discriminative pretext tasks to learn point-cloud representations from similarities and spatial or temporal structure. The survey covers contrastive and hybrid approaches, spatial-relation tasks, and learning from neighboring frames.

5.2 Context-based methods

  • Context-based methods learn through discriminative pretext tasks for context similarity, spatial structure, or temporal structure without human annotations.The survey groups these methods separately from generation-based approaches.Source: §5.2 · p. 9

5.2.1 Learning with context similarity

  • Contrastive learning brings augmented views of an instance together and separates different instances; point-cloud methods also combine it with local-part or multi-task objectives.Clustering assigns samples pseudo-labels, but point-cloud work described here combines it with reconstruction or contrastive learning rather than using clustering alone.Source: §5.2.1 · pp. 9–11
  • PointContrast pretraining on 3D scene partial frames boosts fine-tuned classification, semantic segmentation, and object detection, with consistent gains across these tasks.It builds point-level pairs from aligned ScanNet views and trains them with a unified SR-UNet backbone.Source: §5.2.1 · p. 10

5.2.2 Learning with spatial context structure

  • Spatial-context tasks train networks to recover voxel assignments, part relations, rotations, sequence order, or distortions from point-cloud geometry.Jigsaw3D randomly rearranges voxelized points and predicts each point's voxel label.Source: §5.2.2 · p. 11

5.2.3 Learning with temporal context structure

  • STRL treats neighboring frames as positive pairs and minimizes feature distance, while another approach contrasts temporally ordered samples from synthetic dynamic scenes.Temporal-context learning has shown effectiveness in 2D and 3D, but the survey calls for more research to better harvest temporal information.Source: §5.2.3 · p. 11

5.3 Multiple modal-based methods

Multiple modal-based methods exploit relationships between point clouds and other data modalities to learn representations. The reviewed approaches use cross-modal correspondences, while text and audio remain under-explored.

5.3 Multiple modal-based methods

  • CMCV [4] combines rendered-view image features and point-cloud features to learn intra- and cross-modal correspondences; both pretrained encoders outperform random initialization on classification.It uses a 2D CNN, a 3D GCN, and a two-layer FCN for cross-modality correspondence prediction.Source: §5.3 · p. 12, Figure 20 · p. 12
  • Other work projects point clouds into colored images and uses a frozen image-pretrained model to extract features for downstream tasks.Source: §5.3 · p. 12
  • Learning unsupervised point-cloud representations with modalities such as text descriptions and audio remains under-explored.Source: §5.3 · p. 12

5.4 Local descriptor-based methods

Local descriptor-based methods target low-level tasks by learning detailed spatial representations, but their transfer to high-level recognition remains rarely discussed.

5.4 Local descriptor-based methods

  • PPF-FoldNet learns rotation-invariant local descriptors for 3D matching, while other methods use non-rigid correspondence or unsupervised registration tasks.Registration seeks the rigid transformation that aligns source and target point clouds.Source: §5.4 · p. 12
  • Evaluation mainly targets low-level tasks; adaptation of learned descriptors to high-level tasks is rarely discussed, limiting evidence about broader transfer.Source: §5.4 · p. 12

5.5 Pros and Cons

The survey contrasts the advantages and limits of generation-, context-, multimodal-, and local descriptor-based methods. Their practical scope is constrained by data complexity, task transfer, and unresolved training challenges.

5.5 Pros and Cons

  • Generation-based methods often focus on object-level clouds, limiting applicability mainly to classification and part segmentation; complex, noisy, sparse, and occluded scenes are difficult to generate.Source: §5.5 · p. 12
  • Context-based methods suit complex real-world scene tasks, but effective hard-example mining and self-supervision that generalizes across tasks remain challenging.Source: §5.5 · p. 12
  • Multimodal methods can enrich point-cloud representations with additional modalities, but large paired datasets, cross-domain tasks, and a shared representation space remain challenges.Source: §5.5 · p. 12
  • Local descriptor methods capture detailed spatial cues but transfer poorly to high-level recognition, restricting their use in more abstract tasks.Source: §5.5 · p. 12

6 BENCHMARK PERFORMANCES

The survey benchmarks unsupervised point-cloud representations across object and scene tasks using linear classification and fine-tuning. Results show gains over training from scratch in many settings, while improvements remain limited in several comparisons.

6 BENCHMARK PERFORMANCES

  • The benchmark compiles representative methods and reported results from public point-cloud datasets using linear classification and fine-tuning.The authors extract performance values from the corresponding papers.Source: §6 · p. 12, §6.1 · pp. 12–13

6.1 Evaluation Criteria

  • Linear classification evaluates learned representations with a supervised linear classifier, while fine-tuning measures downstream performance after adapting the pre-trained model.The survey notes that reconstruction and clustering metrics can complement these protocols, but task-specific metrics may not suit all settings; generic measures of representation quality remain scarce.Source: §6.1 · pp. 12–13

6.2.1 Object classification

  • Linear-classification results have improved over time, with some unsupervised methods surpassing supervised methods on ModelNet10 or ModelNet40.Table 4 compares representations learned on ShapeNet or ModelNet40 and reports test-set classification on ModelNet10 and ModelNet40.Source: Table 4 · p. 13, §6.2.1 · p. 14
  • Fine-tuned models initialized with unsupervised pre-training outperform random initialization on ModelNet40 and ScanObjectNN, but gains remain limited.The survey attributes the limited gaps largely to pre-training dataset size and diversity and the simplicity of existing backbones.Source: §6.2.1 · p. 14, Table 5 · p. 14

6.2.2 Object part segmentation

  • On ShapeNetPart, unsupervised and supervised performance gaps are narrowing, and fine-tuning improves performance over supervised training in most cases, with limited gains.Table 6 compares linear classification, fine-tuning, and supervised performance.Source: §6.2.2 · p. 14, Table 6 · p. 14

6.3 Scene-level tasks

  • Unsupervised representations generalize across domains and improve performance over training from scratch on multiple high-level 3D tasks, though gains remain limited.The benchmark spans scene-level semantic segmentation, object detection, and instance segmentation.Source: §6.3 · p. 15, Table 7 · p. 15, Table 8 · p. 15

7 FUTURE DIRECTION

The survey identifies backbone architectures, dataset scale and diversity, and underexplored task settings as priorities for future point-cloud representation learning. It also points to multimodal and sequential data as opportunities for further study.

Unified 3D backbones are needed

  • A shared 3D backbone could support knowledge transfer across datasets and tasks; the variety of existing architectures impedes scalable design and efficient deployment.The survey calls for universal backbones comparable in ubiquity to ResNet in 2D vision.Source: §7 · p. 16

Larger datasets are needed

  • Most existing URL datasets were collected for supervised learning and are constrained in size and diversity, limiting their suitability for unsupervised point-cloud learning.The survey calls for large, high-quality unlabelled datasets spanning object- and scene-level, indoor and outdoor point clouds.Source: §7 · p. 16

Unsupervised pre-training for scene-level tasks

  • Scene-level pre-training remains nascent despite reported cross-domain and cross-task generalization, with network architectures and datasets cited as constraints.Earlier work has focused mainly on object-level processing; scene-level studies address tasks including object detection and instance segmentation.Source: §7 · p. 16

Learning representations from multi-modal data

  • Multimodal learning remains under-investigated, although sensors can provide complementary information and cross-modal correspondences have been explored as pretext tasks.Examples include point clouds, RGB images, and depth maps.Source: §7 · p. 16

Learning Spatio-temporal representations

  • Sequential point clouds provide temporal information for unsupervised learning, while most existing work focuses on static data.The survey anticipates pretext tasks for learning spatio-temporal representations from unlabelled frames.Source: §7 · p. 16

8 CONCLUSION

The paper surveys unsupervised point-cloud representation learning, reviewing datasets, architectures, methods, and benchmarks across multiple 3D tasks. It concludes by outlining future research directions.

8 CONCLUSION

  • The survey combines background on datasets and architectures, a taxonomy and review of methods, multi-task benchmarking, and proposed future research directions.The authors hope the survey will provide a foundation for future work on unsupervised point-cloud representations.Source: §8 · p. 16
Loading 2202.13589v3…