Source-linked AI summary
Few-shot 3D Point Cloud Semantic Segmentation
Na Zhao, Tat-Seng Chua, Gim Hee Lee
TL;DR
Fully supervised 3D point cloud segmentation needs extensive labels and cannot generalize well to new classes from only a few examples. This paper proposes attention-aware multi-prototype transductive inference with multi-level feature learning, achieving consistent improvements over baselines across few-shot settings on two benchmarks. The method’s scope remains constrained by one-shot support and the assumption that nearby points usually share labels.
Problem
Fully supervised methods require large labeled datasets and struggle to generalize to new classes when only a few labeled examples are available.
Method
The method learns attention-aware multi-level features, represents each class with multiple prototypes, and propagates labels through affinities among labeled and unlabeled points.
Results
The method consistently outperforms baselines across 2/3-way 1/5-shot settings on S3DIS and ScanNet, including 52% and 53% gains over fine-tuning in 3-way 1-shot evaluation.
Takeaways & Limitations
Discriminative geometric and semantic features, multi-prototype modeling, and transductive relationships are identified as important for few-shot point cloud segmentation.
Takeaways & Limitations
Qualitative results remain constrained in some scenarios by having only one labeled support sample, and transductive inference assumes nearby points are most likely to share labels.
Abstract
from arXiv · showhide
Many existing approaches for 3D point cloud semantic segmentation are fully supervised. These fully supervised approaches heavily rely on large amounts of labeled training data that are difficult to obtain and cannot segment new classes after training. To mitigate these limitations, we propose a novel attention-aware multi-prototype transductive few-shot point cloud semantic segmentation method to segment new classes given a few labeled examples. Specifically, each class is represented by multiple prototypes to model the complex data distribution of labeled points. Subsequently, we employ a transductive label propagation method to exploit the affinities between labeled multi-prototypes and unlabeled points, and among the unlabeled points. Furthermore, we design an attention-aware multi-level feature learning network to learn the discriminative features that capture the geometric dependencies and semantic correlations between points. Our proposed method shows significant and consistent improvements compared to baselines in different few-shot point cloud semantic segmentation settings (i.e., 2/3-way 1/5-shot) on two benchmark datasets. Our code is available at https://github.com/Na-Z/attMPTI.
1. Introduction
Few-shot point cloud segmentation addresses the need to segment new classes from scarce labeled examples, overcoming the data and closed-set limits of fully supervised methods. The proposed method combines multi-prototype modeling, transductive inference, and attention-aware feature learning, improving over baselines across few-shot settings.
- Fully supervised methods require large, costly labeled datasets and assume training and testing share the same label space.
- Few-shot segmentation learns to segment query point clouds containing new classes from a support set of a few labeled examples.
- Few-shot point cloud segmentation must represent novel-class distributions from scarce support data and use that knowledge for effective segmentation.
- The proposed method represents each class with multiple prototypes and performs transductive inference using discriminative features learned under few-shot constraints.
- 52% and 53% improvements over fine-tuning are reported for 3-way 1-shot segmentation on S3DIS and ScanNet, respectively.
2. Related Work
Prior 3D segmentation methods generally rely on full supervision, while few-shot image segmentation commonly matches support and query samples through global representations. The paper situates its point-cloud approach against these limitations and related few-shot strategies.
- 3D Semantic Segmentation: Fully supervised 3D segmentation methods use point-wise ground truths but require extensive training data and recognize only classes seen during training.
- 3D Semantic Segmentation: PointNet processes raw point clouds efficiently, whereas DGCNN adds local-structure modeling through EdgeConv and serves as this paper’s feature-extractor backbone.
- Few-shot Image Segmentation: Most few-shot image-segmentation methods use metric-based one-to-many matching with each support class represented by one global vector.
3. Our Methodology
The method represents each class with multiple prototypes, learns attention-aware multi-level embeddings, and propagates labels over a sparse graph connecting prototypes and query points. It trains and evaluates this transductive predictor in episodic N-way K-shot segmentation tasks.
- 3.1. Problem Definition: Few-shot episodes use disjoint training and testing class sets, with support examples and query point clouds forming each N-way K-shot task.The model predicts query-point label distributions from the support set, while query labels are available only during training.
- 3.2.1. Embedding Network: The embedding network combines local geometric, global semantic, and metric-adaptive features through a feature extractor, attention learner, and metric learner.DGCNN supplies local and semantic features, while self-attention explores semantic correlations in the global context.
- 3.2.2. Multi-prototype Generation: Each class receives multiple prototypes generated by sampling embedding-space seeds and assigning support points to their closest seed.The number of prototypes is kept uniform across classes for simplicity.
- 3.2.3. Transductive Inference: A sparse k-NN graph connects class prototypes with query points and encodes cross-set and intra-set relationships using Gaussian affinities and symmetric normalization.The graph contains n × (N + 1) prototypes and T × M query points; labeled prototype rows in Y use one-hot class labels, while other rows are zero.
- 3.2.3. Transductive Inference: Label propagation iteratively diffuses prototype labels through the graph, with α controlling the balance between neighboring-node information and initial labels.The method uses the converged closed-form solution to obtain query predictions, normalizes them with softmax, and computes cross-entropy loss.
- 3.2.4. Training Objective: Training gradients reach the embedding network through the parameter-less prototype-generation and transductive-inference operations, so optimization is governed by graph affinities.The loss is computed between predicted query labels and ground-truth query labels.
4. Experiments
Experiments evaluate the method on S3DIS and ScanNet under few-shot settings, comparing it with several baselines and analyzing features, hyperparameters, and qualitative predictions. The method consistently outperforms baselines across 2/3-way 1/5-shot settings, while attention-aware features and transductive multi-prototype inference are important to performance.
- Datasets and setup: Experiments use S3DIS and ScanNet, with rooms or indoor scenes divided into 1m×1m blocks and 2,048 points randomly sampled per block.S3DIS contains 272 point clouds and ScanNet contains 1,513 point clouds; preprocessing produces 7,547 and 36,350 blocks, respectively.
- Datasets and setup: Testing episodes sample novel classes with support and query sets, and mean-IoU is averaged over the testing classes.Training and testing episodes are constructed by sampling classes and point-cloud blocks according to the few-shot setup.
- Comparison with baselines: The method significantly outperforms baselines in all four 2/3-way 1/5-shot settings on both datasets.The comparison includes fine-tuning, ProtoNet, and degraded variants designed to isolate the proposed components.
- Comparison with baselines: Under 3-way 1-shot, performance improves over FT by around 52% on S3DIS and 53% on ScanNet, while one-shot gains over ProtoNet are around 10% and over 20%.The authors attribute the gains to extracting more useful knowledge from very limited data during inference.
- Ablation and analysis: MPTI outperforms ProtoNet in one-shot settings but loses that gain under five shots, indicating that naive multi-prototype generation can struggle with many support points.The authors connect this behavior to the learned embedding space and the role of self-attention in producing representative embeddings.
- Ablation and analysis: Increasing prototypes per class generally helps until overly large values overfit, k = 200 performs best on both datasets, and optimal σ differs between S3DIS and ScanNet.The reported best σ values are 1 for S3DIS and 5 for ScanNet; the feature ablation studies local geometric, global semantic, and metric-adaptive features.
- Qualitative results: Qualitative results show generally more accurate predictions than ProtoNet on challenging S3DIS scenes and diverse ScanNet room types.The S3DIS cases include similar columns and walls, barely visible door frames, and cluttered tables; ScanNet examples include bathrooms, bedrooms, hotels, living rooms, and lounges.
5. Conclusion
The paper presents attention-aware multi-prototype transductive inference for few-shot point cloud segmentation and reports consistent improvements over baselines. It highlights discriminative geometric-semantic features, multimodal point-cloud distributions, and embedding-space relationships as important.
- The proposed method achieves clear and consistent improvements over baselines across few-shot point cloud semantic segmentation tasks.
- Discriminative features encoding geometric and semantic context are central to metric-based few-shot point cloud segmentation.
- 3D point-cloud distributions are complex and cannot be sufficiently modeled by a unimodal distribution.
- Exploiting intrinsic relationships in the embedding space is necessary for point cloud segmentation.
- Future work could adaptively generate multi-prototypes to infer the required number of prototypes from data complexity.
A.1. Dataset Split
The dataset-split appendix identifies the class names assigned to each split of the S3DIS and ScanNet datasets.
- The S3DIS and ScanNet dataset splits are specified by listing their test class names.
A.2.1 EdgeConv architecture details
The feature extractor uses a dynamically constructed k-NN graph and EdgeConv to combine each point with translated neighboring-point features.
- EdgeConv dynamically computes a k-NN graph from input point-wise features as a basic feature-extractor block.
- Each point x_i is concatenated with the translated neighbor feature x_j − x_i in the local system centered at x_i.
- The experiments set the EdgeConv k-NN graph size to k = 20.
A.2.2 Hyper-parameter settings
Hyper-parameter choices vary with dataset and few-shot setting: σ differs by dataset, while the optimal prototype count n increases with available shots and task difficulty; k is fixed at 200 for label propagation.
- The optimal prototype count n increases with the number of shots and ways in different few-shot settings.More shots provide more support points and more ways require finer-grained class modeling.
- k = 200 is used for the k-NN graph in all few-shot settings on both datasets.