PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Path Region Mining For Weakly Supervised 3D Semantic Segmentation on Point Clouds

Jiacheng Wei, Guosheng Lin, Kim-Hui Yap, Tzu-Yi Hung, Lihua Xie

arXiv:2003.13035v1cs.CV

TL;DR

Point-cloud segmentation is limited by the cost of dense point-level annotation, even though collecting large 3D datasets has become easier. The paper learns segmentation from scene- and subcloud-level weak labels using multi-path region mining to generate pseudo point-level labels, achieving compatibility with some fully supervised methods.

  • Problem

    Point-cloud segmentation requires costly and labor-intensive point-level annotation despite easier large-scale 3D data collection.

  • Method

    The method trains a weak-label classification network and uses multi-path region mining with several attention modules to generate pseudo point-level labels for segmentation.

  • Results

    The weakly supervised approach is compatible with some fully supervised methods, and subcloud-level training outperforms scene-level training by a large margin.

  • Takeaways & Limitations

    Scene- and subcloud-level labels provide a feasible way to train point-cloud segmentation while reducing annotation cost.

  • Takeaways & Limitations

    A substantial performance gap remains between the weakly supervised approach and current state-of-the-art methods.

Abstract

from arXiv · show

Point clouds provide intrinsic geometric information and surface context for scene understanding. Existing methods for point cloud segmentation require a large amount of fully labeled data. Using advanced depth sensors, collection of large scale 3D dataset is no longer a cumbersome process. However, manually producing point-level label on the large scale dataset is time and labor-intensive. In this paper, we propose a weakly supervised approach to predict point-level results using weak labels on 3D point clouds. We introduce our multi-path region mining module to generate pseudo point-level label from a classification network trained with weak labels. It mines the localization cues for each class from various aspects of the network feature using different attention modules. Then, we use the point-level pseudo labels to train a point cloud segmentation network in a fully supervised manner. To the best of our knowledge, this is the first method that uses cloud-level weak labels on raw 3D space to train a point cloud semantic segmentation network. In our setting, the 3D weak labels only indicate the classes that appeared in our input sample. We discuss both scene- and subcloud-level weakly labels on raw 3D point cloud data and perform in-depth experiments on them. On ScanNet dataset, our result trained with subcloud-level labels is compatible with some fully supervised methods.

1. Introduction

The paper addresses costly point-level annotation by learning 3D point-cloud segmentation from scene- and subcloud-level weak labels. Its multi-path region mining module generates pseudo point-level labels from localization cues, and experiments show compatibility with some fully supervised methods.

  • Weak supervision: Scene-level labels identify classes appearing in a scene, whereas subcloud-level labels identify classes appearing in sampled subclouds.The paper compares these weak-label strategies as alternatives to point-level labels.
  • Motivation: Point-cloud annotation remains labor- and time-intensive despite increasingly accessible 3D data collection.ScanNet annotation required more than 500 participants, with a median of 16.8 minutes per scan.
  • Results: Experiments use both weak-label strategies, with subcloud-level training outperforming scene-level training by a large margin.The reported results also outperform some popular fully supervised point-cloud recognition models.
  • Approach: The proposed approach learns 3D semantic segmentation using only scene- and subcloud-level labels on raw point clouds.The paper presents this as the first approach of its kind to learn a point-cloud scene segmentation network from cloud-level weak labels on raw 3D data.
  • Approach: MPRM generates pseudo point-level labels by mining localization cues through spatial, channel, and point-wise spatial attention modules.These modules target long-range spatial context, channel inter-dependencies, and global context from network features.

2. Related Work

Prior point-cloud methods use projections, voxelization, or direct point processing, while most segmentation approaches rely on full supervision. Earlier self-supervised point-cloud methods learn representations but cannot directly use those features for several downstream tasks.

  • Weakly Supervised Semantic Segmentation on 2D Images: Image-level, bounding-box, scribble, and point annotations have been studied to reduce dense-labeling costs in 2D semantic segmentation.
  • Deep Learning on Point Clouds: Projection-based point-cloud methods can suffer from occlusion and varying point densities in segmentation tasks.
  • Deep Learning on Point Clouds: Voxelization processes point clouds on 3D grids, while PointNet-like methods reduce quantization by directly processing unordered raw points.
  • Deep Learning on Point Clouds: Point convolution networks apply convolution operations directly to raw point clouds, but the reviewed approaches require fully annotated training data.
  • Point Cloud Recognition with Less Supervision: Self-supervised methods such as part reassembly and MortonNet learn point-cloud features but cannot directly use them for object classification, part segmentation, and semantic segmentation.

3. Our Weakly Supervised Setting

The paper studies scene-level and subcloud-level labels as weak supervision for 3D segmentation. Subcloud labels retain low annotation cost while providing more localized class information through overlapping spherical samples.

  • Scene-level Annotation: Scene-level labels indicate only which classes appear in an entire scene, making them the most economical weak labels.
  • Scene-level Annotation: Scene-level supervision is challenged by the extensive information in reconstructed 3D scenes and frequent classes such as walls and floors.
  • Subcloud-level Annotation: Subcloud-level labels address these challenges by labeling classes within spherical subsamples while retaining low annotation cost.
  • Subcloud-level Annotation: Subclouds are formed around uniformly placed seeding points and may overlap, so a point can belong to multiple subclouds.
  • Subcloud-level Annotation: With radius r = 2m, ScanNet contains an average of 18.4 subclouds per scene, each labeled by the classes appearing within it.

4. Our Framework

The framework converts weak cloud-level labels into point-level pseudo-labels using PCAMs and a four-path region-mining module. Attention paths capture complementary spatial, channel, and global-context cues, whose PCAMs are merged to supervise segmentation.

  • 4.1. Baseline Method: PCAM: PCAM applies class activation mapping to point convolution features to produce class-specific localization cues for point clouds.
  • 4.1. Baseline Method: PCAM: The PCAM classifier uses weak labels during training, while its pointwise feature maps provide localization evidence before global average pooling.
  • 4.2. Multi-Path Region Mining: MPRM uses four parallel classification paths: plain PCAM, spatial attention, channel attention, and point-wise attention.
  • 4.2. Multi-Path Region Mining: The module merges path-specific PCAMs by element-wise maximum and upsamples them to the original point resolution for pseudo-label generation.
  • 4.2.1 Spatial Attention Module: Spatial attention aggregates weighted features from other points, selectively adding global context to local representations.
  • 4.2.2 Attention Modules: Channel attention models interdependencies among feature channels, while the point-wise path combines local and weighted global features for region mining.

5. Implementation Details

Experiments use ScanNet with scene-level and subcloud-level weak labels, alongside KPConv-based classification and segmentation networks. The implementation adapts sampling and batching to variable-sized point-cloud subclouds.

  • Dataset and Labels: ScanNet experiments use 1,205 training scenes and 312 validation scenes under the official train-validation split, with 20 classes.
  • Dataset and Labels: Subcloud-level labels are generated by spherical sampling with query radius r = 2.0m, while scene-level labels identify classes appearing in each scene.
  • Classification Network: The classification backbone is a simplified KPConv network with five ResNet bottleneck blocks and deformable kernels in the final three blocks.
  • Classification Network: Because whole scenes exceed network capacity, scene-level training also uses randomly sampled subclouds carrying the scene labels.
  • Batching and Segmentation: Variable point counts are handled by stacking subclouds until a point-count batch limit is reached rather than using a fixed batch size.
  • Batching and Segmentation: The final segmentation model is KPConv-based and converges at around 200 epochs.

6. Experiments

Experiments show that subcloud-level supervision and multi-path region mining improve pseudo-label quality and final segmentation, while retraining further improves validation performance. The weakly supervised result remains compatible with some fully supervised methods but trails current state of the art.

  • Scene-level Versus Subcloud-level Labels: Subcloud-level labels outperform scene-level labels by a large margin, especially for small objects near dominant classes.Scene-level models tend to assign points to walls and floors, whereas subcloud sampling alleviates class imbalance.
  • Pseudo Label Evaluation: MPRM improves pseudo-label segmentation with both scene-level and subcloud-level supervision, separating small objects from dominant classes.The module improves classes such as doors, windows, and pictures and produces smoother spatial regions.
  • Ablation Study: The four-path MPRM combination yields higher pseudo-label performance than any individual path or other combination.The spatial-attention path performs best alone, while combining paths captures complementary features.
  • Ablation Study: Max fusion outperforms sum fusion because it avoids overwhelming classes with larger score amplitudes.Global average pooling can produce different score scales across classes, making summation distort fusion.
  • Segmentation Results: Retraining a full-scale segmentation network with MPRM-CRF pseudo labels improves validation results over directly using raw MPRM outputs.Retraining incorporates low-level dCRF features into an end-to-end model.
  • Segmentation Results: The weakly supervised approach remains compatible with some fully supervised methods but has a large gap to current state-of-the-art methods.The comparison uses ScanNet test-set results reported in the online benchmark.

7. Conclusions

The paper presents weakly supervised 3D point-cloud scene segmentation using subcloud labels and a Multi-path Region Mining module. The approach reduces annotation cost and achieves performance compatible with some fully supervised methods.

  • Conclusions: The proposed scheme trains 3D point-cloud scene segmentation using weak labels, including an economical subcloud labeling strategy.The authors state that subcloud labeling reduces annotation labor and time costs.
  • Conclusions: MPRM improves segmentation performance by mining localization cues from classification features.The final qualitative results compare predictions with ground truth while ignoring unclassified points during evaluation.
  • Conclusions: The weakly supervised result is compatible with some fully supervised methods.This is the paper's stated conclusion about final segmentation performance.
Loading 2003.13035v1…