Source-linked AI summary
Self-supervised Learning of Point Clouds via Orientation Estimation
Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, Vladimir G. Kim
TL;DR
Manual annotation makes scaling point-cloud learning costly, so this paper uses rotation prediction to learn features from unlabeled 3D shapes. The approach achieves superior performance on shape classification and 3D keypoint prediction, and its features can complement other self-supervised methods.
Problem
Manually annotating large point-cloud datasets is costly and impractical to scale, motivating learning from unlabeled data.
Source: §1 · p. 1Method
The method rotates point clouds and trains a model to predict the applied rotation, using the learned features for downstream tasks.
Source: §1 · p. 1Results
The approach achieves superior performance to competing methods on shape classification and 3D keypoint prediction, and features from different self-supervised models can complement one another.
Source: §4.1 · p. 5, §5 · p. 8Takeaways & Limitations
A small subset of training data can suffice for high accuracy on shape classification, reducing the need for extensive annotation.
Source: §1 · p. 1, §5 · p. 8The method assumes consistently aligned shapes, a condition the authors note is common in online stock repositories.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Point clouds provide a compact and efficient representation of 3D shapes. While deep neural networks have achieved impressive results on point cloud learning tasks, they require massive amounts of manually labeled data, which can be costly and time-consuming to collect. In this paper, we leverage 3D self-supervision for learning downstream tasks on point clouds with fewer labels. A point cloud can be rotated in infinitely many ways, which provides a rich label-free source for self-supervision. We consider the auxiliary task of predicting rotations that in turn leads to useful features for other tasks such as shape classification and 3D keypoint prediction. Using experiments on ShapeNet and ModelNet, we demonstrate that our approach outperforms the state-of-the-art. Moreover, features learned by our model are complementary to other self-supervised methods and combining them leads to further performance improvement.
1. Introduction
Point clouds are useful 3D representations, but the cost of manual annotation motivates learning from unlabeled data. The paper uses rotation prediction as a self-supervised task to learn features for downstream shape classification and keypoint prediction.
1. Introduction
- Manual annotation of large point-cloud datasets is costly and difficult to scale, motivating methods that learn from unlabeled data.Self-supervision creates an automatic pretext-task signal to learn representations for downstream tasks.Source: §1 · p. 1
- The paper proposes a rotation-prediction approach for 3D point clouds, using their canonical orientation or known up axis as a label-free signal.The authors describe it as the first self-supervised approach based on rotations of 3D data.Source: §1 · p. 1
- Features learned from rotation prediction support classification and keypoint prediction, including when simple SVMs use only a small training subset.The paper also reports that these features complement other self-supervised features, improving performance when combined.Source: §1 · p. 1
2. Related Work
Related work spans deep learning methods for geometric data and unsupervised or self-supervised representation learning. Prior point-cloud approaches address unordered inputs, while existing self-supervised methods use varied proxy tasks across modalities and for point clouds.
2.1. Deep Learning on Geometry
- Grid-based view and voxel methods adapt geometric data to conventional architectures, but voxel memory grows cubically with resolution and octrees add implementation complexity while remaining limited in grid size.Raw geometry lacks the regular grid structure typical convolutional architectures require.Source: §2.1 · pp. 1–2
- Point-cloud networks address unordered inputs with permutation-invariant global pooling or local-structure methods, but these models still require many annotated samples.The section describes PointNet, PointNet++, PointCNN, PointConv, Relation-Shape CNN, and Dynamic Graph CNN.Source: §2.1 · p. 2
- Learning-based geometric methods also cover meshes, implicit functions, and structured models.Source: §2.1 · p. 2
2.2. Unsupervised and Self-Supervised Learning
- Unsupervised and self-supervised learning use generative models and annotation-free proxy tasks, including image transformations, spatial relations, and temporal or audiovisual cues.Prior work also predicts the spatial relation between image patches and extends that idea to 3D representations.Source: §2.2 · p. 2
- Point-cloud self-supervision includes reconstruction, part rearrangement, part contrasting, clustering, prediction, and reconstruction tasks; the paper characterizes its rotation task as requiring more holistic shape understanding.It also states that its approach can be combined with competing methods for further performance gains.Source: §2.2 · p. 2
3. Method
The method trains a point-cloud model to predict an applied rotation, then reuses its features for downstream tasks. It explores classification and regression formulations, choices of rotation set, and assumes consistently aligned shapes.
3. Method
- A randomly selected rotation is applied to a point cloud, and a classifier predicts which rotation was used; the architecture then passes learned features to downstream tasks.The model is based on PointNet or DGCNN, with modified final layers and cross-entropy training for rotation classification.Source: §3 · pp. 2–3, Figure 1 · p. 3
- The number of candidate angles K trades off proxy-task difficulty: small K makes rotation prediction easier, while larger K lowers rotation-classification accuracy as angles become closer.The passage states that downstream features are not necessarily best when the rotation task is too easy or too difficult.Source: §3 · p. 3, Table 1 · p. 3
- Candidate directions are selected to be as uniform as possible, using axis directions, polyhedron-derived points, or a golden-spiral distribution for different K values.Figure 2 depicts the object’s upward direction rotated to one of the marked points.Source: §3 · p. 3, Figure 2 · p. 4
- Rotation can instead be regressed as an axis and angle or with a continuous 6D representation; the latter is mapped to SO(3) for single-cloud orientation prediction.The axis-angle model uses equal-weight L2 losses for axis and angle prediction.Source: §3 · pp. 3–4
- For classification, rotation-model features feed a linear SVM; for keypoint prediction, the model’s final layers are modified and fine-tuned for regression.Rotation estimation is trained on ShapeNet, while target models are trained on ModelNet.Source: §3 · pp. 3–4
- The method assumes shapes are consistently aligned, though the authors describe a relative-rotation extension that could operate without canonical orientations.They caution that features learned by that unsupervised extension might be less transferable than the current features.Source: §3 · p. 4
4. Experiments
Experiments evaluate rotation-prediction features on shape classification and 3D keypoint prediction. The reported results include better performance than competing methods, benefits from combining complementary features, and useful performance with limited training data.
4.1. Shape Classification
- On ModelNet-40 shape classification, rotation-classification features outperform competing methods, with K = 18 and 32 achieving the best performance.The reported test accuracy is from a linear SVM; values of K that are too large or too small perform worse.Source: §4.1 · p. 5, Table 2 · p. 5
- Combining rotation features with context-prediction features improves performance over either individual model, consistent with their complementary information.The paper characterizes the rotation model's features as global-orientation information and the context model's as part-level information.Source: §4.1 · p. 5
4.2. 3D Keypoint Prediction
- On 3D keypoint prediction, our approach outperforms competing methods at every error threshold and performs well with training subsets.Figure 4 shows PCK curves for the keypoint task and for different training-set sizes.Source: Figure 4 · p. 6, §4.2 · p. 6
4.1. Shape Classification
- Rotation regression features perform worse than rotation-classification features for shape categorization, which the authors relate to regression being challenging and too task-specific.The regression features are evaluated with a linear SVM on ModelNet-40; the regression model targets exact rather than discretized rotations.Source: §4.1 · p. 6
- Rotation-prediction features retain relatively strong classification performance when the linear SVM is trained on a small subset of labeled ModelNet-40 data.Accuracy is evaluated on the full test set while the SVM training set varies from 1% to 100%.Source: §4.1 · p. 6
4.2. 3D Keypoint Prediction
- For keypoint prediction, a rotation-pretrained model is fine-tuned to regress ten chair keypoints, and predicted and ground-truth keypoints are often close or overlapping.The visualization shows predicted keypoints in red and ground truth in black across objects and viewpoints.Source: §4.2 · pp. 6, 8, Figure 5 · p. 7
5. Conclusion and Future Work
The approach learns point-cloud representations through rotation prediction and performs well on shape classification and 3D keypoint prediction. The authors also identify other 3D representations and downstream tasks as future research directions.
5. Conclusion and Future Work
- Rotation-prediction features outperform competing methods on shape classification and 3D keypoint prediction, and complement other self-supervised features.Combining features from different models outperforms each individual model.Source: §5 · p. 8
- A small subset of training data suffices for high accuracy on shape classification, reducing the need to annotate massive 3D datasets.Source: §5 · p. 8
- Future work includes self-supervision for meshes, voxel grids, and implicit functions, as well as evaluation on additional downstream tasks.The paper focuses on point clouds and describes self-supervision for other 3D representations as under-explored.Source: §5 · p. 8