PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Path Region Mining For Weakly Supervised 3D Semantic Segmentation on Point Clouds

Jiacheng Wei, Guosheng Lin, Kim-Hui Yap, Tzu-Yi Hung, Lihua Xie

arXiv:2003.13035v1cs.CV

TL;DR

Point-cloud segmentation is limited by the cost of dense point-level annotation, even though collecting large 3D datasets has become easier. The paper learns segmentation from scene- and subcloud-level weak labels using multi-path region mining to generate pseudo point-level labels, achieving compatibility with some fully supervised methods.

  • Problem

    Point-cloud segmentation requires costly and labor-intensive point-level annotation despite easier large-scale 3D data collection.

    Source: §1 · p. 1

  • Method

    The method trains a weak-label classification network and uses multi-path region mining with several attention modules to generate pseudo point-level labels for segmentation.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    The weakly supervised approach is compatible with some fully supervised methods, and subcloud-level training outperforms scene-level training by a large margin.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Scene- and subcloud-level labels provide a feasible way to train point-cloud segmentation while reducing annotation cost.

    Source: §1 · p. 2, §7 · p. 8

    A substantial performance gap remains between the weakly supervised approach and current state-of-the-art methods.

    Source: §6.4 · p. 8

Abstract

from arXiv · show

Point clouds provide intrinsic geometric information and surface context for scene understanding. Existing methods for point cloud segmentation require a large amount of fully labeled data. Using advanced depth sensors, collection of large scale 3D dataset is no longer a cumbersome process. However, manually producing point-level label on the large scale dataset is time and labor-intensive. In this paper, we propose a weakly supervised approach to predict point-level results using weak labels on 3D point clouds. We introduce our multi-path region mining module to generate pseudo point-level label from a classification network trained with weak labels. It mines the localization cues for each class from various aspects of the network feature using different attention modules. Then, we use the point-level pseudo labels to train a point cloud segmentation network in a fully supervised manner. To the best of our knowledge, this is the first method that uses cloud-level weak labels on raw 3D space to train a point cloud semantic segmentation network. In our setting, the 3D weak labels only indicate the classes that appeared in our input sample. We discuss both scene- and subcloud-level weakly labels on raw 3D point cloud data and perform in-depth experiments on them. On ScanNet dataset, our result trained with subcloud-level labels is compatible with some fully supervised methods.

1. Introduction

The paper addresses costly point-level annotation by learning 3D point-cloud segmentation from scene- and subcloud-level weak labels. Its multi-path region mining module generates pseudo point-level labels from localization cues, and experiments show compatibility with some fully supervised methods.

Weak supervision

  • Scene-level labels identify classes appearing in a scene, whereas subcloud-level labels identify classes appearing in sampled subclouds.The paper compares these weak-label strategies as alternatives to point-level labels.Source: Figure 1 · p. 1, §1 · p. 2

Motivation

  • Point-cloud annotation remains labor- and time-intensive despite increasingly accessible 3D data collection.ScanNet annotation required more than 500 participants, with a median of 16.8 minutes per scan.Source: §1 · p. 1

Results

  • Experiments use both weak-label strategies, with subcloud-level training outperforming scene-level training by a large margin.The reported results also outperform some popular fully supervised point-cloud recognition models.Source: §1 · p. 2

Approach

  • The proposed approach learns 3D semantic segmentation using only scene- and subcloud-level labels on raw point clouds.The paper presents this as the first approach of its kind to learn a point-cloud scene segmentation network from cloud-level weak labels on raw 3D data.Source: §1 · p. 2
  • MPRM generates pseudo point-level labels by mining localization cues through spatial, channel, and point-wise spatial attention modules.These modules target long-range spatial context, channel inter-dependencies, and global context from network features.Source: §1 · p. 2

2. Related Work

Prior point-cloud methods use projections, voxelization, or direct point processing, while most segmentation approaches rely on full supervision. Earlier self-supervised point-cloud methods learn representations but cannot directly use those features for several downstream tasks.

Weakly Supervised Semantic Segmentation on 2D Images

  • Image-level, bounding-box, scribble, and point annotations have been studied to reduce dense-labeling costs in 2D semantic segmentation.Source: §2 · p. 2

Deep Learning on Point Clouds

  • Projection-based point-cloud methods can suffer from occlusion and varying point densities in segmentation tasks.Source: §2 · p. 2
  • Voxelization processes point clouds on 3D grids, while PointNet-like methods reduce quantization by directly processing unordered raw points.Source: §2 · p. 2
  • Point convolution networks apply convolution operations directly to raw point clouds, but the reviewed approaches require fully annotated training data.Source: §2 · p. 3

Point Cloud Recognition with Less Supervision

  • Self-supervised methods such as part reassembly and MortonNet learn point-cloud features but cannot directly use them for object classification, part segmentation, and semantic segmentation.Source: §2 · p. 3

3. Our Weakly Supervised Setting

The paper studies scene-level and subcloud-level labels as weak supervision for 3D segmentation. Subcloud labels retain low annotation cost while providing more localized class information through overlapping spherical samples.

Scene-level Annotation

  • Scene-level labels indicate only which classes appear in an entire scene, making them the most economical weak labels.Source: §3 · p. 3
  • Scene-level supervision is challenged by the extensive information in reconstructed 3D scenes and frequent classes such as walls and floors.Source: §3 · p. 3

Subcloud-level Annotation

  • Subcloud-level labels address these challenges by labeling classes within spherical subsamples while retaining low annotation cost.Source: §3 · p. 3
  • Subclouds are formed around uniformly placed seeding points and may overlap, so a point can belong to multiple subclouds.Source: §3 · p. 3
  • With radius r = 2m, ScanNet contains an average of 18.4 subclouds per scene, each labeled by the classes appearing within it.Source: §3 · p. 3

4. Our Framework

The framework converts weak cloud-level labels into point-level pseudo-labels using PCAMs and a four-path region-mining module. Attention paths capture complementary spatial, channel, and global-context cues, whose PCAMs are merged to supervise segmentation.

4.1. Baseline Method: PCAM

  • PCAM applies class activation mapping to point convolution features to produce class-specific localization cues for point clouds.Source: §4.1 · pp. 3–4
  • The PCAM classifier uses weak labels during training, while its pointwise feature maps provide localization evidence before global average pooling.Source: §4.1 · p. 4

4.2. Multi-Path Region Mining

  • MPRM uses four parallel classification paths: plain PCAM, spatial attention, channel attention, and point-wise attention.Source: §4.2 · p. 4
  • The module merges path-specific PCAMs by element-wise maximum and upsamples them to the original point resolution for pseudo-label generation.Source: §4.2 · p. 4, §4.1 · p. 4

4.2.1 Spatial Attention Module

  • Spatial attention aggregates weighted features from other points, selectively adding global context to local representations.Source: §4.2.1 · p. 5

4.2.2 Attention Modules

  • Channel attention models interdependencies among feature channels, while the point-wise path combines local and weighted global features for region mining.Source: §4.2.2 · p. 5, §4.2.3 · pp. 5–6

5. Implementation Details

Experiments use ScanNet with scene-level and subcloud-level weak labels, alongside KPConv-based classification and segmentation networks. The implementation adapts sampling and batching to variable-sized point-cloud subclouds.

Dataset and Labels

  • ScanNet experiments use 1,205 training scenes and 312 validation scenes under the official train-validation split, with 20 classes.Source: §5.1 · p. 6
  • Subcloud-level labels are generated by spherical sampling with query radius r = 2.0m, while scene-level labels identify classes appearing in each scene.Source: §5.1 · p. 6

Classification Network

  • The classification backbone is a simplified KPConv network with five ResNet bottleneck blocks and deformable kernels in the final three blocks.Source: §5.2 · p. 6
  • Because whole scenes exceed network capacity, scene-level training also uses randomly sampled subclouds carrying the scene labels.Source: §5.2 · p. 6

Batching and Segmentation

  • Variable point counts are handled by stacking subclouds until a point-count batch limit is reached rather than using a fixed batch size.Source: §5.2 · p. 6
  • The final segmentation model is KPConv-based and converges at around 200 epochs.Source: §5.3 · p. 6

6. Experiments

Experiments show that subcloud-level supervision and multi-path region mining improve pseudo-label quality and final segmentation, while retraining further improves validation performance. The weakly supervised result remains compatible with some fully supervised methods but trails current state of the art.

Scene-level Versus Subcloud-level Labels

  • Subcloud-level labels outperform scene-level labels by a large margin, especially for small objects near dominant classes.Scene-level models tend to assign points to walls and floors, whereas subcloud sampling alleviates class imbalance.Source: §6.1 · pp. 6–7

Pseudo Label Evaluation

  • MPRM improves pseudo-label segmentation with both scene-level and subcloud-level supervision, separating small objects from dominant classes.The module improves classes such as doors, windows, and pictures and produces smoother spatial regions.Source: §6.2 · p. 7, Table 2 · p. 7

Ablation Study

  • The four-path MPRM combination yields higher pseudo-label performance than any individual path or other combination.The spatial-attention path performs best alone, while combining paths captures complementary features.Source: §6.3 · p. 7, Table 3 · p. 7
  • Max fusion outperforms sum fusion because it avoids overwhelming classes with larger score amplitudes.Global average pooling can produce different score scales across classes, making summation distort fusion.Source: §6.3 · p. 7, Table 3 · p. 7

Segmentation Results

  • Retraining a full-scale segmentation network with MPRM-CRF pseudo labels improves validation results over directly using raw MPRM outputs.Retraining incorporates low-level dCRF features into an end-to-end model.Source: §6.4 · p. 7, Table 4 · p. 8
  • The weakly supervised approach remains compatible with some fully supervised methods but has a large gap to current state-of-the-art methods.The comparison uses ScanNet test-set results reported in the online benchmark.Source: §6.4 · p. 8, Table 5 · p. 8

7. Conclusions

The paper presents weakly supervised 3D point-cloud scene segmentation using subcloud labels and a Multi-path Region Mining module. The approach reduces annotation cost and achieves performance compatible with some fully supervised methods.

Conclusions

  • The proposed scheme trains 3D point-cloud scene segmentation using weak labels, including an economical subcloud labeling strategy.The authors state that subcloud labeling reduces annotation labor and time costs.Source: §7 · p. 8
  • MPRM improves segmentation performance by mining localization cues from classification features.The final qualitative results compare predictions with ground truth while ignoring unclassified points during evaluation.Source: §7 · p. 8, Figure 6 · p. 8
  • The weakly supervised result is compatible with some fully supervised methods.This is the paper's stated conclusion about final segmentation performance.Source: §7 · p. 8

Related papers

Loading 2003.13035v1…