SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds

Qingyong Hu; Bo Yang; Guangchi Fang; Yulan Guo; Ales Leonardis; Niki Trigoni; Andrew Markham

Source-linked AI summary

SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds

Qingyong Hu, Bo Yang, Guangchi Fang, Yulan Guo, Ales Leonardis, Niki Trigoni, Andrew Markham

arXiv:2104.04891v3Apr 11, 2021cs.CVcs.AIcs.RO

TL;DR

Dense point-cloud annotation is costly, raising whether full labeling is necessary for semantic segmentation. The paper introduces SQN, which uses local neighborhood queries to exploit semantic similarity and sparse supervision, achieving close to fully supervised accuracy with 0.1% labels. Its scope includes annotation schemes whose purity and labeling ratios may not be comparable across datasets.

Problem
Dense point-cloud labeling is costly and limits practical semantic segmentation, motivating learning from a small fraction of annotations.
Method
SQN queries local neighborhoods in hierarchical point-cloud representations and summarizes them to predict semantic labels from extremely sparse point annotations.
Results
0.1% training labels yield close to fully supervised accuracy across multiple large-scale open datasets, with annotation costs reduced by up to 98% for similar accuracy.
Takeaways & Limitations
Dense 3D annotations can be redundant, and local semantic similarity provides a basis for weakly supervised segmentation of large-scale point clouds.
Takeaways & Limitations
Some compared weak-supervision methods use dataset-dependent annotation schemes and ratios, and superpoint label purity may not transfer across datasets.

Abstract

from arXiv · show

Labelling point clouds fully is highly time-consuming and costly. As larger point cloud datasets with billions of points become more common, we ask whether the full annotation is even necessary, demonstrating that existing baselines designed under a fully annotated assumption only degrade slightly even when faced with 1% random point annotations. However, beyond this point, e.g., at 0.1% annotations, segmentation accuracy is unacceptably low. We observe that, as point clouds are samples of the 3D world, the distribution of points in a local neighborhood is relatively homogeneous, exhibiting strong semantic similarity. Motivated by this, we propose a new weak supervision method to implicitly augment highly sparse supervision signals. Extensive experiments demonstrate the proposed Semantic Query Network (SQN) achieves promising performance on seven large-scale open datasets under weak supervision schemes, while requiring only 0.1% randomly annotated points for training, greatly reducing annotation cost and effort. The code is available at https://github.com/QingyongHu/SQN.

1 Introduction

SQN addresses costly dense point-cloud annotation by exploiting local semantic similarity to learn from extremely sparse labels. It achieves close to fully supervised accuracy with 0.1% training labels across large-scale datasets.

Results: SQN shows comparable or better qualitative segmentation than fully supervised RandLA-Net on S3DIS when trained with 0.1% annotations.
Motivation: Dense annotation is costly because point clouds are unordered, unstructured, and non-uniform, motivating weaker supervision.Prior annotation efforts can exceed 1700 person-hours for a typical dataset.
Motivation: Existing weak-supervision methods use varied annotation schemes and often require multistage pipelines, complicating fair comparison, tuning, and deployment.These methods also under-exploit local semantic homogeneity among neighboring points.
Method: SQN queries hierarchical latent representations within a local neighborhood and predicts labels from their summarized features.This design uses neighboring points’ semantic similarity to spread sparse training signals across a wider spatial region.
Results: 0.1% labels enable SQN to achieve close to fully supervised accuracy on multiple large-scale open datasets without self-supervised pretraining, handcrafted constraints, or complicated post-processing.The paper reports annotation-time reductions of up to 98% for similar accuracy.
Motivation: Fully supervised methods degrade slowly until 1% sparse point annotation, indicating that dense labeling is not necessary in that regime.

2 Related Work

Existing 3D semantic segmentation methods achieve strong results but depend on costly dense annotations, while weakly supervised approaches use varied annotation schemes and assumptions that complicate fair comparison and generalization.

Fully supervised 3D segmentation methods rely on large amounts of labeled data, making them costly and prohibitive in real applications.The paper motivates learning semantics from a small fraction of annotations as a cheaper and more realistic alternative.
Weakly supervised 3D methods use 2D labels, sparse 3D labels, pseudo-labels, superpoint annotations, or contrastive pretraining.
Superpoint-based comparisons can be unreliable when supervoxels are not label-pure or when labeling ratios divide clicks by raw points despite dense supervoxel labels.The paper specifically notes these concerns for 1T1C and concludes that direct comparison on ScanNet is not possible.

3 Exploring Weak Supervision

The paper evaluates random sparse annotation as a weak-supervision regime and finds that fully supervised baselines remain near their full-label performance at 1%, but degrade sharply below 0.1%, motivating SQN.

Annotation cost: Less than 2 minutes: annotating 0.1% of a standard S3DIS room, compared with more than 20 minutes for full annotation.
Annotation cost: About 18 hours: annotating 0.1% of SensatUrban, compared with more than 600 person-hours for annotating all points.
Experimental setup: The study defines weak signals by randomly annotating 10%, 1%, 0.1%, or 0.01% of points in each training room, with Area 5 reserved for testing.
Findings: Less than 4%: all baselines’ performance decreases marginally when annotations fall from 100% to 1%.This indicates dense annotations are unnecessary for comparable accuracy under simple random annotation.
Findings: Below 0.1%: baseline performance drops significantly, indicating that retaining a certain amount of training signal is essential.
Motivation: SQN is proposed to obtain close to fully supervised accuracy using only 0.1% labels, targeting the annotation-cost and accuracy trade-off identified in large-scale point clouds.

4 SQN

SQN combines hierarchical point-cloud feature extraction with local semantic querying, allowing sparse labels to supervise a wider spatial context and achieving near fully supervised performance at 0.1% annotation.

Architecture: SQN uses a local feature extractor and a point feature query network to collect relevant semantic features from sparse supervision.
Point Local Feature Extractor: The encoder applies four Local Feature Aggregation layers followed by Random Sampling to produce hierarchical point features while preserving point locations.
Point Feature Query Network: For each query point, SQN searches nearest neighboring features across four encoded levels and uses Euclidean distance to select them.
Point Feature Query Network: Trilinear interpolation compresses queried vectors at each level, and the resulting hierarchical feature vectors are concatenated into a representation for the query point.
Point Feature Query Network: The compact query representation is passed through MLPs to infer the point’s semantic category.
Training and inference: During training, sparse query labels back-propagate through a wider spatial context; during testing, all 3D points are estimated using the two sub-networks.

5 Experiments

Across seven large-scale point-cloud datasets, SQN achieves strong weakly supervised segmentation using sparse random annotations. Experiments show that hierarchical semantic querying is effective, robust, and sensitive to annotation density and neighborhood design.

Comparison with SOTA Approaches: SQN outperforms RandLA-Net by nearly 9% with the same 0.1% random annotations on S3DIS, while remaining comparable to fully supervised RandLA-Net.The comparison uses identical weak-supervision settings.
Comparison with SOTA Approaches: SQN achieves higher mIoU on ScanNet with 0.1% labels than methods trained with sub-cloud labels or 1% annotations.The authors omit comparison with the ScanNet Data-Efficient benchmark because its training settings cannot be verified.
Comparison with SOTA Approaches: With 0.1% labels, SQN achieves competitive performance with fully supervised baselines on both Semantic8 and Reduced8 of Semantic3D.The framework also achieves satisfactory accuracy with 0.01% labels, though the authors identify room for improvement.
Comparison with SOTA Approaches: With 0.1% labels, SQN is on par with fully supervised RandLA-Net across several datasets and surpasses it on SensatUrban.The evaluation covers DALES, SensatUrban, Toronto3D, and SemanticKITTI.
Ablation Study: Combining queries from different encoding layers performs better than querying only a single layer.Performance drops significantly when relevant point features are collected only from the first or last layer; the last layer alone outperforms the first.
Ablation Study: An overly large queried neighborhood can reduce performance by including points with different semantics, despite larger neighborhoods generally helping.The study varies the number of neighboring points from 1 to 25.
Ablation Study: Performance is satisfactory at 0.1% labels but drops significantly at 0.01% because the supervision signal becomes too sparse.Using 10% labels slightly improves mIoU over full supervision, with 2%-5% gains on some minority categories.

6 Conclusion

The paper presents SQN as a simple framework that learns large-scale point-cloud semantics from as few as 0.1% supplied labels. It attributes this efficiency to semantic similarity among neighboring 3D points and identifies extensions to other segmentation and annotation tasks.

Conclusion: SQN learns large-scale point-cloud semantics with as few as 0.1% supplied labels for training.The framework is based on semantic similarity among neighboring points and is described as wider label propagation.
Conclusion: The authors propose extending SQN to weakly supervised instance segmentation, panoptic segmentation, and interactive annotation with active learning.These directions are presented as future extensions rather than evaluated results.

7 Details of Sparse Annotation Tool

The sparse annotation tool randomly downsamples point clouds, enlarges sampled points for visualization, and supports polygonal labeling with refinement. Grid downsampling further reduces annotation volume before applying the reported labeling ratio.

Annotation Pipeline: The annotation pipeline downsamples raw point clouds, enlarges sampled points, visualizes both versions, labels sampled points polygonally, and refines their labels.The six-step workflow is designed for random sparse annotation.
Number of Annotated Points: Grid downsampling can reduce the Semantic3D point count to 1/50 before annotation, yielding 78,100 annotated points under the 0.1% setting.The reported 0.1% ratio is computed after grid downsampling, not against the original raw-point total.
Annotation Pipeline: The random annotation pipeline costs 600 person-hours and avoids much of the boundary-focused work required by dense labeling.The authors attribute the productivity difference to error tolerance around boundaries and flexible polygonal editing.

8 Implementation Tricks

Implementation includes standard point-cloud augmentation and optional retraining with generated pseudo labels. The reported retraining benefit is dataset-dependent and is not noticeable on DALES.

Data Augmentation: Training uses random flipping, random rotation, and random noise as data augmentation techniques.These augmentations are applied to input point clouds during training.
Re-training with Generated Pseudo Labels: On DALES, retraining with generated pseudo labels does not noticeably improve SQN beyond satisfactory performance from 0.1% annotated points.The authors leave advanced pseudo-label refining for future work.

9 Video Illustration

The paper provides a video illustrating SQN's segmentation performance.

A supplementary video illustrates the performance achieved by SQN.The video is available at https://youtu.be/Q6wICSRRw3s.

10 Additional Ablation Results

Additional ablations examine SQN with different backbones and feature-propagation layers, while sensitivity analysis identifies which categories vary most under 0.1% random annotations.

Varying backbones: SQN is further evaluated with the voxel-based MinkowskiNet backbone.The implementation uses four encoding layers, each containing a 3D convolution block and two residual blocks.
Feature propagation layers: SQN queries hierarchical representations at multiple spatial resolutions, unlike general propagation layers designed to recover dense resolution under full supervision.This design aims to propagate limited weak-supervision signals to a wider context.
Feature propagation layers: Ablative experiments compare SQN with two general feature-propagation layers across three datasets.The feature encoder, labeling ratio, and experimental setting are kept consistent for the comparison.
Sensitivity analysis: Under 0.1% random annotations, major performance variations occur in minor categories such as door, sofa, and board.The sensitivity analysis reports Overall Accuracy, mean IoU, and per-class IoU over five runs on S3DIS Area 5.

11 Additional Discussion

Additional discussion evaluates SQN near semantic boundaries, on incomplete or diverse large-scale datasets, and across several fully supervised comparisons. The results show broad weakly supervised utility, while highlighting sensitivity to underrepresented categories and boundary assumptions.

Boundary performance: Boundary points are defined where spherical neighbors within radius r have different semantics, testing the method's neighborhood-consistency assumption.The paper notes that this assumption may not hold at semantic boundaries.
SQN flexibility: SQN can infer labels for arbitrary 3D positions and achieves satisfactory performance on complete SemanticKITTI clouds after training on partial clouds.This supports the framework's stated flexibility for irregularly sampled point clouds.
Future exploration: Unsupervised discovery of 3D semantics from raw point clouds is left for future exploration.The paper explicitly identifies this as an open direction.
Large-scale datasets: On SensatUrban, SQN is evaluated on an urban-scale dataset with nearly 3 billion points and highly imbalanced class distributions.Qualitative results are also shown when SQN is trained with only 0.1% labels.
Cross-dataset results: On Toronto3D, SQN outperforms several fully supervised methods with 0.1% point annotations, while color information narrows its gap with RandLA-Net.The dataset contains 78.3 million points across eight semantic categories.
Cross-dataset results: On DALES, SQN exceeds several fully supervised methods in mIoU with 0.1% labels but trails RandLA-Net because minor categories such as trucks and cars remain weak.The paper attributes this potential weakness partly to random annotation overlooking underrepresented classes.