PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Point Cloud Pre-Training via Occlusion Completion

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, Matthew J. Kusner

arXiv:2010.01089v3cs.CVcs.LG

TL;DR

Point-cloud labeling is costly and difficult, motivating unsupervised representations that transfer beyond a single dataset. OcCo masks viewpoint-occluded points, learns an encoder-decoder to complete them, and reuses the encoder for downstream initialization; it improves few-shot learning, classification, and segmentation while producing broader and more invariant representations.

  • Problem

    Point-cloud models depend on labeled datasets, but point-cloud annotation is difficult because data may be sparse, occluded, massive, time-consuming, and error-prone.

  • Method

    OcCo samples camera viewpoints to create occluded point clouds, trains an encoder-decoder to reconstruct them, and uses the learned encoder weights to initialize downstream tasks.

  • Results

    OcCo improves few-shot learning and generalization in object classification, object part segmentation, and semantic segmentation, including across datasets different from ModelNet40.

  • Takeaways & Limitations

    OcCo provides a transferable point-cloud initialization whose fine-tuned minima are wider and whose representations show semantic meaningfulness and transformation-invariant clustering.

  • Takeaways & Limitations

    The unsupervised mutual-information probe evaluates zero-shot label information only through global features extracted by OcCo-PointNet on cross-domain datasets.

Abstract

from arXiv · show

We describe a simple pre-training approach for point clouds. It works in three steps: 1. Mask all points occluded in a camera view; 2. Learn an encoder-decoder model to reconstruct the occluded points; 3. Use the encoder weights as initialisation for downstream point cloud tasks. We find that even when we construct a single pre-training dataset (from ModelNet40), this pre-training method improves accuracy across different datasets and encoders, on a wide range of downstream tasks. Specifically, we show that our method outperforms previous pre-training methods in object classification, and both part-based and semantic segmentation tasks. We study the pre-trained features and find that they lead to wide downstream minima, have high transformation invariance, and have activations that are highly correlated with part labels. Code and data are available at: https://github.com/hansen7/OcCo

1. Introduction

OcCo addresses the difficulty and expense of point-cloud annotation by using viewpoint occlusion and completion for unsupervised pre-training. A single ModelNet40 pre-training dataset improves downstream performance and yields more invariant, semantically meaningful representations across tasks.

  • Motivation: Point-cloud labeling is difficult because data can be sparse, occluded, low-resolution, massive, time-consuming, and error-prone.These challenges have impeded deployment of point-cloud models into new settings.
  • Occlusion Completion: OcCo generates masked point clouds through viewpoint occlusions and trains a completion task to reconstruct the missing points.The approach is motivated by the need for representations that capture spatial and semantic properties.
  • Results: Pre-training on ModelNet40 improves a range of downstream tasks, including few-shot learning, object classification, part segmentation, and semantic segmentation, even on different datasets.The paper compares OcCo with other initialization techniques across multiple downstream settings.
  • Representation analysis: OcCo representations are associated with wider fine-tuned minima, more semantically meaningful activations, and better clustering under jittering, translation, and rotation.These analyses characterize the learned features beyond downstream accuracy.

2. Related Work

Prior point-cloud pre-training methods largely use generative modeling or self-supervised geometric objectives. OcCo instead targets a completion-based initialization intended to transfer across few-shot learning, classification, and segmentation datasets.

  • Existing approaches: Unsupervised point-cloud pre-training commonly uses generative models or self-supervised learning to obtain model initializations.Generative approaches include GANs, autoencoders, normalizing flows, and approximate convex decomposition.
  • Existing approaches: Self-supervised rearrangement predicts original voxel locations, but its usefulness for object classification or segmentation is unclear because randomly permuted clouds differ from real objects and scenes.The cited method splits clouds into k^3 voxels, permutes them, and predicts each point's original voxel location.
  • Existing approaches: Cover-tree partition prediction may divide semantically contiguous regions arbitrarily because cover trees are designed for nearest-neighbor search.This creates a potential mismatch between the pre-training partition task and semantic structure.
  • OcCo's distinction: Earlier completion pre-training initialized 3D voxel descriptors for visual localization, whereas OcCo focuses on initialization for point-cloud models.OcCo aims to improve few-shot learning, classification, and segmentation across varied datasets.

3. Occlusion Completion

OcCo constructs occluded point clouds from sampled camera viewpoints and trains an encoder-decoder to complete them. The learned encoder weights then initialize downstream point-cloud models.

  • Generating Occlusions: A pinhole camera projection uses camera intrinsics together with rotation and translation to map world-frame points into a camera reference frame.The prose defines focal length, skewness, image width, image height, and the matrices K and [R|t].
  • Generating Occlusions: Occlusion is identified by treating projected points as pixels with depth, where farther points sharing pixel coordinates are hidden by nearer points.The procedure first uses Delaunay triangulation to reconstruct a polygon mesh before removing points.
  • Generating Occlusions: Occluded point clouds are generated by sampling viewpoints, projecting points into camera coordinates, removing occluded points, and transforming the visible points back.For V views, this produces an occluded cloud ˜P_v for each sampled viewpoint.
  • The Completion Task: The completion task learns a mapping from an occluded cloud ˜P to a completed cloud ˆP using an encoder-decoder network.The encoder maps the occluded cloud to a vector, while the decoder completes the point cloud.
  • Downstream Initialization: After pre-training, the encoder weights initialize downstream tasks such as classification and segmentation.The same overall approach is also described as improving few-shot and fully supervised generalization.

4. Experiments

Experiments pre-train encoder-decoder models on occluded ModelNet40 point clouds, then fine-tune them across few-shot learning, classification, part segmentation, and semantic segmentation tasks. OcCo consistently improves performance across datasets and encoders, with especially clear gains over competing initialisations.

  • Pre-training setup: OcCo pre-training uses ModelNet40 training objects, generating occluded point clouds from 10 randomly selected viewpoints.ModelNet40 contains 12,311 synthesised CAD objects across 40 categories, split into 9,843 training and 2,468 testing objects.
  • Pre-training setup: The completion model uses PointNet, PCN, or DGCNN encoders and a folding-based decoder that reconstructs coarse and fine point-cloud shapes.Each encoder maps an occluded point cloud to a 1024-dimensional vector; the decoder outputs 1,024 coarse points and a 16,384-point fine shape.
  • Evaluation tasks: Experiments evaluate few-shot learning, object classification, object part segmentation, and indoor and outdoor semantic segmentation.Classification uses ModelNet40, ScanNet10, and ScanObjectNN; part segmentation uses ShapeNetPart; semantic segmentation uses S3DIS and SensatUrban.
  • Few-shot learning: OcCo outperforms Jigsaw and cTree in few-shot learning both in-domain on ModelNet40 and cross-domain on ScanObjectNN.The authors attribute this to occlusions preserving geometric structure, unlike voxel permutations or cover-tree partitions that may disrupt it.
  • Object classification: OcCo-initialised models outperform random and Jigsaw initialisations on object classification across all evaluated datasets and consistently across three encoders.The reported gains include in-domain ModelNet performance and cross-domain performance on ScanNet and ScanObjectNN.
  • Segmentation: OcCo improves object part segmentation and semantic segmentation, while SensatUrban gains are strongest for categories represented in ModelNet40.For unseen SensatUrban classes, OcCo is competitive with other methods; part-segmentation improvements hold for overall accuracy and mean class IoU.

5. Analysis

The analysis examines OcCo’s learned representations through optimisation landscapes, feature visualisation, transformation robustness, clustering, and semantic concept detection. Across these probes, OcCo features are flatter, more discriminative, and more semantically informative than comparison initialisations.

  • Optimisation landscape: OcCo pre-training leads to a flatter local minimum after fine-tuning than random and Jigsaw initialisations.The loss landscape is visualised by perturbing fine-tuned parameters with two normalised random directions over α, β ∈ [−1, 1].
  • Learned features: OcCo features progress from low-level primitives such as planes, cylinders, and cones to complex shapes including wings, leaves, and upper bodies.Feature activations are visualised on ModelNet40 objects, while t-SNE embeddings on ShapeNet10 form distinguishable clusters after pre-training.
  • Feature quality: OcCo learns more discriminative and predictive feature representations than hand-crafted descriptors and Jigsaw-based pre-training.The comparison includes Jigsaw, viewpoint feature histogram, and M2DP.
  • Semantic concept detection: OcCo detects more semantic concepts than Jigsaw, with concepts identified when channel-to-label mIoU exceeds 0.5.Activation masks select each channel’s top 20% point activations, while concept masks come from ground-truth semantic annotations.

6. Discussion

The discussion concludes that OcCo learns useful point-cloud representations across few-shot learning, classification, and segmentation, and analyses relate this performance to several representation properties.

  • Discussion: OcCo representations are accurate across few-shot learning, object classification, and part and semantic segmentation tasks compared with prior work.The analyses include loss landscapes, learned features, transformation invariance, and semantic concept detection.

A. Implementation details

OcCo uses an encoder-decoder completion architecture and compares occluded pre-training datasets, finding that the generated ModelNet occlusions are more realistic and yield better transfer than ShapeNet Occluded.

  • Completion architecture: Point-cloud completion models use an encoder to map a partial point cloud to a fixed-dimensional vector and a decoder to reconstruct the full shape.OcCo follows this encoder-decoder design for pre-training.
  • Completion architecture: OcCo uses PointNet or DGCNN encoders that map partial point clouds to 1024-dimensional vectors and a two-stage PCN folding decoder.The decoder produces coarse and fine-grained output point clouds; batch normalisation is removed because it harms completion loss and convergence.
  • Occluded datasets: ModelNet Occluded is generated from ModelNet40, while ShapeNet Occluded uses PCN-provided data produced by a similar but different method.The two occluded datasets are compared through their basic statistics and cross-dataset pre-training experiments.
  • Occluded datasets: The generated ModelNet occlusions are described as more naturalistic and closer to real collected data than the ShapeNet Occluded examples.The comparison is illustrated with examples from both datasets.
  • Cross-dataset transfer: OcCo models pre-trained on ShapeNet Occluded perform worse than models pre-trained on the other occluded dataset in cross-dataset evaluation.The paper tests whether pre-training on one occluded dataset transfers when fine-tuned on the other.
  • Jigsaw re-implementation: Jigsaw pre-training reformulates shuffled 3D cubes as a 27-class semantic segmentation task.Objects or chopped indoor scenes are divided into 3^3 = 27 cubes, shuffled, and assigned pointwise cube labels.

B. Ablations

Additional experiments assess statistical significance and vary the number of object views and categories used during OcCo pre-training.

  • Statistical significance: All p-values from ten-run t-tests comparing OcCo with Jigsaw or random baselines are below α=0.05.The tests use unpaired unequal-variance t-tests, and the family-wise error rate remains controlled with Holm-Bonferroni correction.
  • Ablations: The ablations vary the number of object views and categories using the fastest-running experimental setting.Tables 11 and 12 report these ablations, with five runs per setting.
  • Statistical significance: The statistical comparison covers few-shot learning on ScanObjectNN with DGCNN, classification on ScanNet with PCN, and segmentation on SensatUrban with PointNet.These are the three settings listed in the p-value table.

C. More results

Additional experiments show that OcCo improves few-shot learning and part segmentation across settings, while also outperforming random and Jigsaw initialisation with RBF-kernel SVMs.

  • Additional results: The supplementary results include a linear SVM embedding comparison and additional few-shot learning results.The referenced tables report linear-SVM results and expanded few-shot experiments.
  • Few-shot learning: OcCo pre-training either outperforms or has overlapping standard deviations with the best method in 7 of 8 few-shot settings.The comparison uses one ModelNet40 pre-training dataset, whereas cTree pre-trained on both datasets before fine-tuning.
  • Object classification: All OcCo pre-trained models outperform random and Jigsaw initialisations by a large margin with grid-searched RBF-kernel SVMs.The SVM uses a radial basis function kernel with parameters selected by grid search.
  • Part segmentation: OcCo initialisation achieves better results for more than two thirds of the 15 ShapeNetPart object classes across all three encoders.The detailed results report scores separately for each ShapeNetPart category.

D. Algorithmic Description of OcCo

OcCo repeatedly samples camera viewpoints, projects and occludes the point cloud, transforms it back, and trains a completion model by gradient descent before transferring the encoder to downstream tasks.

  • Algorithm setup: Algorithm 1 defines the initial point cloud, camera matrix, viewpoint count, point-cloud loss, completion model, and downstream prediction model.These variables specify the inputs, loop control, optimization objective, and transfer target.
  • Occlusion generation: OcCo samples random viewpoints and transforms the initial point cloud into the camera reference frame using camera intrinsics and the sampled pose.The viewpoint combines random rotation and translation, followed by camera-coordinate projection.
  • Occlusion generation: The algorithm creates an occluded point cloud with z-buffering and maps it back to the world frame using inverse camera and pose transforms.The inverse intrinsic matrix and transposed homogeneous transformation recover the world-frame representation.
  • Completion training: The completion model is optimized by computing a point-cloud loss between the completed point cloud and the original point cloud, then applying gradient descent.The loop repeats until all sampled viewpoints have been processed.
  • Downstream transfer: After pre-training, the downstream model is initialised with the completion model's encoder parameters and then trained on the downstream task.Only the encoder parameters are transferred to the downstream prediction model.

E. Visualisation from Completion Pre-Training

Visualisations show completion quality across three encoder architectures, including difficult occluded shapes, while also displaying failures associated with complicated fine structures.

  • Qualitative results: The visualisations compare input, coarse output, fine output, and ground truth across training epochs and encoder architectures.Figures 10, 11, and 12 correspond to PCN, PointNet, and DGCNN encoders on occluded ModelNet40.
  • Qualitative results: The trained completion models can complete difficult occluded shapes such as plants and planes.This observation is reported for the visualisations using PCN, PointNet, and DGCNN encoders.
  • Failure cases: Failure examples are associated with complicated fine structures in completed shapes.Figure 13 presents these failure cases.
Loading 2010.01089v3…