Source-linked AI summary
Unsupervised Multi-Task Feature Learning on Point Clouds
Kaveh Hassani, Mike Haley
TL;DR
Point-cloud representation learning must handle irregular data while reducing reliance on labeled examples, and existing unsupervised approaches often target only one task. The paper jointly trains a multi-scale graph-based encoder with clustering, autoencoding, and self-supervised classification. It reports state-of-the-art unsupervised results, including 89.1% accuracy on ModelNet40 and mIoU of 77.7 on ShapeNetPart.
Problem
Deep point-cloud models require large amounts of labeled data, while existing unsupervised methods often lack supervisory signals and target a single task.
Method
The model jointly trains a multi-scale graph-based encoder using clustering, autoencoding, and self-supervised classification to learn point and shape features.
Results
The learned features outperform prior state-of-the-art unsupervised models across classification and segmentation benchmarks, achieving 89.1% accuracy on ModelNet40 and mIoU of 77.7 on ShapeNetPart.
Takeaways & Limitations
Joint unsupervised tasks can produce features competitive with supervised models on point-cloud classification and segmentation benchmarks.
Takeaways & Limitations
The methodology assumes a training set of order-invariant point sets, each containing M points.
Abstract
from arXiv · showhide
We introduce an unsupervised multi-task model to jointly learn point and shape features on point clouds. We define three unsupervised tasks including clustering, reconstruction, and self-supervised classification to train a multi-scale graph-based encoder. We evaluate our model on shape classification and segmentation benchmarks. The results suggest that it outperforms prior state-of-the-art unsupervised models: In the ModelNet40 classification task, it achieves an accuracy of 89.1% and in ShapeNet segmentation task, it achieves an mIoU of 68.2 and accuracy of 88.6%.
1. Introduction
Point clouds are irregular, permutation-invariant data for which deep feature learning usually requires substantial labeled data. The paper addresses this by jointly training an unsupervised multi-task model with clustering, autoencoding, and self-supervised classification, reporting state-of-the-art performance among unsupervised models.
- Point clouds are sparse, order-invariant sets sampled from object surfaces and used in sensing applications such as self-driving and robotics.
- Deep models can perform many point-cloud tasks, but their feature learning typically requires large amounts of labeled data.
- Existing unsupervised approaches use autoencoders or generative models, but often lack supervisory signals and target a single task.
- The proposed model jointly uses self-supervision, autoencoding, and clustering to learn point and shape features while avoiding trivial solutions.
2. Related Work
Related work develops deep architectures, generative models, autoencoders, self-supervised tasks, and multi-task learning for point-cloud representation learning. These approaches differ in how they process irregular point sets and obtain useful supervisory signals.
- Point-cloud deep models include PointNet, hierarchical PointNet++, kd-tree architectures, RNNs, and graph-based convolutional models.
- Generative approaches include multiresolution VAEs, autoregressive PointGrow, GANs, and GMMs trained on PointNet features.
- Autoencoder methods reconstruct point clouds or point-pair feature spaces using architectures such as FoldingNet, PPF-FoldNet, AtlasNet, and SO-Net.
- Self-supervised learning creates proxy tasks on unlabeled data, but its main challenge is designing tasks related to downstream objectives.
- Multi-task learning shares features across relevant tasks, and prior work has applied self-supervised multi-task learning to high-level visual representation learning.
3. Methodology
The method encodes point sets into local point features and aggregated shape features, then jointly optimizes clustering, self-supervised classification, and reconstruction objectives. Alternating pseudo-label generation with multi-task learning is designed to avoid trivial clustering solutions while learning both feature types.
- Representation: The encoder maps each point set to higher-dimensional point features and aggregates them into a shape feature.The point features encode local information, while aggregation produces a global representation.
- Training tasks: Clustering maps latent shape features to K categories and supplies hard cluster assignments as pseudo-labels for self-supervised classification.The classifier predicts the assignments generated by the clustering function.
- Training tasks: The decoder reconstructs the original point set from the latent variable, while Chamfer distance measures differences between the original and reconstructed clouds.Chamfer distance compares nearest neighbors in both directions.
- Optimization: Centroid updates occur epoch-wise rather than batch-wise, and reconstruction is introduced to prevent convergence to trivial solutions.The model can empty some clusters and converge to a fixed number when initialized with an upper bound.
- Optimization: The model minimizes a weighted linear combination of clustering, classification, and decoding objectives and trains the components jointly.The training procedure uses mini-batch stochastic gradient descent after centroid initialization.
- Architecture: A multi-scale graph-based encoder applies graph convolutions, convolutions, and pooling to learn point and shape features from jittered point clouds.The architecture uses three neighborhood scales for graph convolution.
20 end
The method builds hierarchical point and shape representations through graph-based feature extraction and pooling. Three jointly trained tasks operate on shape features, with shape learning also encouraging useful point features.
- Feature hierarchy: Intermediate features pass through convolution and pooling layers to learn another representation level.The architecture progressively builds higher-level features before producing the final shape representation.
- Feature hierarchy: Point-wise features are pooled into a final shape feature and concatenated with it to form the final point features.This connects global shape information with local point representations.
- Multi-task learning: Clustering, classification, and autoencoding are defined on shape features and trained jointly end-to-end.The classifier predicts cluster assignments, while the autoencoder reconstructs the original point cloud.
- Multi-task learning: Because shape features aggregate corresponding point features, learning a good shape feature also pushes the model toward good point features.The paper explicitly links shape-level supervision to point-level representation learning.
4. Experiments
The model is pre-trained on ShapeNet with frozen features transferred to classification and segmentation, while experiments examine clustering behavior, feature quality, and component effectiveness. It achieves strong unsupervised and semi-supervised performance, with ablations highlighting the roles of the graph encoder, multi-task learning, and reconstruction.
- Experimental Protocol: The model is pre-trained across 55 ShapeNet categories, then transferred to classification and part segmentation without fine-tuning its weights.ShapeNet pre-training uses 57,000 models, and the transferred features support downstream evaluation.
- Shape Classification: NMI reaches 0.68 on the train set and 0.62 on validation, indicating that learned clusters encode category-level information without ground-truth labels.The model converges from 500 to 88 clusters, empties over 80% of them, and avoids collapsing to one cluster.
- Shape Classification: The learned shape features achieve state-of-the-art ModelNet40 accuracy among unsupervised methods and reach 68.88% accuracy in zero-shot clustered classification.The standard transfer result is obtained with a linear SVM and no fine-tuning; random hyper-parameter search raises accuracy to 89.5%.
- Part Segmentation: The learned point features achieve state-of-the-art accuracy and mIoU on ShapeNetPart in semi-supervised evaluation using 1% and 5% of the training set.A four-layer MLP is trained on sampled labels while the pre-trained feature extractor remains frozen.
- Part Segmentation: HDBSCAN produces decent fully unsupervised segmentation, while co-clustering shows consistent learned point features within categories.The qualitative analyses include airplane co-clustering and comparisons with ground-truth labels, MLP predictions, and AHC.
- Ablation Study: The graph-based encoder reaches 89.1% ModelNet40 accuracy versus 86.2% for PointNet, while a single reconstruction decoder reaches 85.7%.Masking cross-entropy reduces accuracy to 87.6%, masking k-means to 88.3%, and using all three losses yields 89.1%; reconstruction has the highest impact.
5. Conclusion
The paper presents an unsupervised multi-task approach that jointly learns point and shape features using clustering, autoencoding, and self-supervised classification. It reports strong performance on classification and segmentation benchmarks, including competitive results with supervised models.
- The model trains a multi-scale graph-based encoder with clustering, autoencoding, and self-supervised classification tasks.
- 89.1% accuracy on ModelNet40 shape classification is reported as state-of-the-art among unsupervised models and competitive with supervised models.
- 77.7 mIoU on ShapeNetPart segmentation is reported as only 8% below the state-of-the-art supervised model.