PaperlayerView exact revision on arXiv

Source-linked AI summary

One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation

Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu

arXiv:2104.02246v4cs.CV

TL;DR

Point-cloud semantic segmentation needs costly dense annotation, while existing weakly supervised approaches still require substantial or constrained supervision. The paper proposes One Thing One Click with iterative graph-based self-training and relation-network similarity learning, achieving results that outperform existing weakly supervised methods and approach fully supervised performance on ScanNet-v2 and S3DIS.

  • Problem

    Large-scale point-cloud annotation is costly, and extremely sparse labels risk overfitting and poor generalization when used directly for 3D semantic segmentation.

  • Method

    One Thing One Click labels one point per object, while iterative graph propagation and network training generate pseudo labels using learned super-voxel similarities and category prototypes.

  • Results

    With around 0.02% point annotations, the approach surpasses existing weakly supervised methods on ScanNet-v2 and S3DIS and achieves results comparable to a fully supervised counterpart.

  • Takeaways & Limitations

    The results support extremely sparse one-point-per-object annotation together with self-training as an effective approach for weakly supervised 3D semantic segmentation.

  • Takeaways & Limitations

    Earlier weakly supervised comparisons include methods with annotation requirements or assumptions that may be difficult to follow in practice, such as uniformly distributed labeled points.

Abstract

from arXiv · show

Point cloud semantic segmentation often requires largescale annotated training data, but clearly, point-wise labels are too tedious to prepare. While some recent methods propose to train a 3D network with small percentages of point labels, we take the approach to an extreme and propose "One Thing One Click," meaning that the annotator only needs to label one point per object. To leverage these extremely sparse labels in network training, we design a novel self-training approach, in which we iteratively conduct the training and label propagation, facilitated by a graph propagation module. Also, we adopt a relation network to generate per-category prototype and explicitly model the similarity among graph nodes to generate pseudo labels to guide the iterative training. Experimental results on both ScanNet-v2 and S3DIS show that our self-training approach, with extremely-sparse annotations, outperforms all existing weakly supervised methods for 3D semantic segmentation by a large margin, and our results are also comparable to those of the fully supervised counterparts.

1. Introduction

3D semantic segmentation depends on costly point-cloud annotation, motivating One Thing One Click, which labels one point per object and uses iterative self-training with graph-based propagation. Experiments on ScanNet-v2 and S3DIS report strong performance with extremely sparse annotations.

  • Motivation: Annotating ScanNet-v2 takes 22.3 minutes per scene on average, making large-scale point-cloud labeling costly.The dataset contains 1,513 scenes.
  • One Thing One Click: One Thing One Click labels one randomly chosen point per object and takes less than 2 minutes per ScanNet-v2 scene.This is reported as more than 10x faster than the original annotation scheme.
  • Challenge: Extremely sparse labels, below 0.02% in ScanNet-v2 and S3DIS, can cause direct network training to overfit and generalize poorly.The paper frames matching fully supervised performance under this sparsity as a central challenge.
  • Self-training framework: The method iteratively propagates pseudo labels through a graph while retraining the network, allowing improved labels and model predictions to reinforce each other.A relation network learns similarity among 3D super-voxel nodes, while category-wise prototypes support similarity measurement.
  • Results: With around 0.02% point annotations, the approach surpasses existing weakly supervised methods on ScanNet-v2 and S3DIS and reaches performance comparable to a fully supervised counterpart.The result is reported across both public datasets.

2. Related Work

Prior weakly supervised 3D segmentation methods reduce annotation effort but retain burdens involving localization, image preparation, subcloud processing, or point-distribution assumptions. This paper introduces iterative graph-based self-training for extremely sparse 3D annotations.

  • 3D semantic segmentation: 3D semantic segmentation methods include point-based networks operating on raw point clouds and voxel-based networks operating on regular voxel grids.The paper adopts 3D-UNet as its backbone architecture.
  • Self-training: The proposed method iteratively combines graph-based label propagation with network training to expand sparse annotations into pseudo labels.The approach is presented as a self-training method for weakly supervised 3D semantic segmentation.
  • Framework: The framework uses super-voxel partitioning, a 3D U-Net for semantic prediction, and a Relation Net for super-voxel similarity learning.Features from the two networks are pooled to construct a graph for label propagation.

3. Methodology

The method combines graph-based label propagation with a relation network and iterative self-training to exploit extremely sparse point annotations. It alternates pseudo-label updates with network optimization, using super-voxel similarities and high-confidence predictions to guide training.

  • Sparse annotation and graph construction: The approach starts from one point per object, over-segments the cloud into geometrically homogeneous super-voxels, and builds a smaller graph for propagation.Each super-voxel is a graph node, reducing propagation cost compared with point-level graphs.
  • Segmentation backbone: The 3D U-Net predicts point-wise semantic probabilities from coordinates and colors, providing the network predictions used throughout propagation and training.The architecture is adopted as the method’s segmentation backbone.
  • Graph propagation: The graph energy combines 3D U-Net predictions with pairwise similarities among super-voxels to propagate labels into unknown regions.The unary term is the negative log probability of a super-voxel label, while pairwise terms penalize different labels for similar super-voxels.
  • Relation network: The relation network produces super-voxel embeddings that group same-category regions and separate different categories for similarity measurement.It complements the 3D U-Net by learning category-related embeddings and uses category prototypes supported by a memory bank.
  • Iterative self-training: Self-training alternates between optimizing pseudo labels through graph propagation and optimizing the 3D U-Net and relation network with softmax and contrastive losses.High-confidence propagated predictions become pseudo labels for the next training iteration.

4. Experiments

Experiments on ScanNet-v2 and S3DIS evaluate the self-training framework against weakly and fully supervised baselines. With extremely sparse annotations, the method achieves strong segmentation performance, and ablations support both graph propagation and relation-network similarity learning.

  • Experimental setup: Experiments use ScanNet-v2 and S3DIS, with official training and evaluation splits described for both datasets.ScanNet-v2 contains 1,513 scans and 20 categories; S3DIS contains 271 rooms and 13 categories.
  • Experimental setup: The self-training algorithm expands annotated points to super-voxels, trains with pseudo labels, and iteratively updates them through graph propagation.The procedure begins from “One Thing One Click” annotations and alternates label expansion, relation-network training, propagation, and pseudo-label updates.
  • ScanNet-v2: 62.18% mIoU is achieved by the initial ScanNet-v2 pseudo-label baseline, while self-training improves it by nearly 10% mIoU and reaches within 2% of the fully supervised baseline.The reported relative improvement is 16%.
  • Ablation studies: 67.92% versus 65.91% mIoU shows a 2% gain from graph propagation, while the relation network further improves similarity measurement over handcrafted features.The relation network better groups intra-class embeddings and distinguishes inter-class embeddings in the t-SNE analysis.
  • S3DIS: On S3DIS, “One Thing One Click” reaches 50.1% mIoU and “One Thing Three Clicks” reaches 55.3% mIoU, outperforming [52] under both reported annotation settings.The same table reports a 63.7% mIoU fully supervised baseline and relative improvements of 14.6% and 13.1% over the respective baselines.
  • S3DIS: With only 0.06% 3D annotation, “One Thing Three Clicks” outperforms [47] with 100% 2D annotations by nearly 3% mIoU.The comparison accounts for the substantially different numbers of annotated 2D pixels and 3D points.

5. Conclusion

The paper introduces “One Thing One Click” annotation and an iterative self-training framework for learning from extremely sparse point labels. Across ScanNet-v2 and S3DIS, the approach outperforms existing weakly supervised methods and approaches fully supervised performance.

  • 5. Conclusion: “One Thing One Click” labels one point per object, while self-training alternates graph-based label expansion with network training on updated pseudo labels.A relation network learns feature similarity among graph nodes with complex 3D structures.
  • 5. Conclusion: Experiments on ScanNet-v2 and S3DIS show large-margin gains over existing weakly supervised methods with extremely sparse annotations.The reported results are also comparable to fully supervised counterparts.

A. Results with Fewer Annotations

The method remains effective when annotations are reduced further on ScanNet-v2. Labeling one random point on only half of the objects yields less than 0.01% annotated points and 60.62% mIoU.

  • A. Results with Fewer Annotations: 60.62% mIoU is achieved with less than 0.01% annotated points under the “Two Things One Click” scheme.A single random point is annotated on half of the objects selected randomly in each scene.

B. More Results on ScanNet-v2 and S3DIS

Additional ScanNet-v2 and S3DIS visual results show that training with one annotated point per object can produce segmentation comparable to a fully supervised baseline. Error maps provide a visual comparison of prediction mistakes.

  • B. More Results on ScanNet-v2 and S3DIS: One annotated point per object produces segmentation results comparable to the fully supervised baseline on ScanNet-v2 and S3DIS.The comparison is visualized in Figures 7 and 8, including corresponding error maps.

C. Relation to Prototypical Networks

The relation network differs from Prototypical Networks because this method handles shared train/test categories in 3D segmentation, where category prototypes must represent global means despite uneven mini-batch sampling.

  • Unlike Prototypical Networks, this method uses the same categories during training and testing, so prototypes must represent global category means.
  • A memory bank updates category prototypes with moving averages rather than a single mini-batch, stabilizing them during training and inference.
  • 3D segmentation mini-batches may contain insufficient or no samples for some categories, requiring replacement sampling to balance category examples.
  • The ablation replaces memory-bank prototypes with current-mini-batch mean embeddings to evaluate the memory bank’s effect.

D. Summary of All the Ablation Studies

The ablations isolate self-training, graph propagation, relation learning, and memory-bank components across progressively richer 3D U-Net variants. The full model uses graph propagation and relation-network information during both training and inference, and the reported summary states that each key module contributes to overall performance.

  • The baseline 3D U-Net without self-training uses the initial pseudo label, while the self-training variant iteratively replaces labels using high-confidence network predictions.
  • The figures compare model outputs with fully supervised results on ScanNet-v2 and S3DIS, marking wrong predictions in red; example super-voxels vary in shape, geometry, density, and point count.
  • Graph propagation adds pairwise similarity based on handcrafted colors and coordinates to pseudo-label generation.
  • The relation-network variant augments pairwise similarity with learned embeddings, initially deriving category prototypes from current mini-batch means without a memory bank.
  • The dagger variant uses memory-bank prototypes but restricts graph propagation and relation-network computation to training, matching the 3D U-Net inference complexity.
  • The full model additionally uses graph propagation and the relation network during inference, and the ablation summary reports that each key module contributes to overall performance.

E. Examples of the Super-Voxel

ScanNet-v2 super-voxels exhibit substantial variation in shape, geometric structure, density, and point count, motivating learned similarity representations for these irregular groups.

  • ScanNet-v2 super-voxels vary substantially in shape, geometric structure, density, and number of points.
  • Because colors and coordinates cannot fully describe these irregular structures, the method uses a relation network to learn high-level similarities.
Loading 2104.02246v4…