Source-linked AI summary
Mining Latent Classes for Few-shot Segmentation
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao
TL;DR
Few-shot segmentation must generalize to unseen classes from sparse annotations, but conventional training can treat latent novel classes as background and undermine their embeddings. The paper jointly mines latent classes with transferable sub-clusters and rectifies foreground and background prototypes, including through extra unlabeled data. Across two benchmarks, it reports substantially better performance with fewer parameters and faster inference.
Problem
Few-shot segmentation methods can treat latent novel classes as background during training, undermining their embeddings and limiting exploitation of available training data.
Method
The method adds a latent-class mining branch using transferable sub-clusters to episodic support-query training and rectifies both foreground and background prototypes.
Results
The method outperforms previous methods on two FSS benchmarks, while using fewer parameters and achieving faster inference.
Takeaways & Limitations
Mining latent novel classes and rectifying prototypes improves use of training data, with further gains possible from extra unlabeled data.
Abstract
from arXiv · showhide
Few-shot segmentation (FSS) aims to segment unseen classes given only a few annotated samples. Existing methods suffer the problem of feature undermining, i.e. potential novel classes are treated as background during training phase. Our method aims to alleviate this problem and enhance the feature embedding on latent novel classes. In our work, we propose a novel joint-training framework. Based on conventional episodic training on support-query pairs, we add an additional mining branch that exploits latent novel classes via transferable sub-clusters, and a new rectification technique on both background and foreground categories to enforce more stable prototypes. Over and above that, our transferable sub-cluster has the ability to leverage extra unlabeled data for further feature enhancement. Extensive experiments on two FSS benchmarks demonstrate that our method outperforms previous state-of-the-art by a large margin of 3.7% mIOU on PASCAL-5i and 7.0% mIOU on COCO-20i at the cost of 74% fewer parameters and 2.5x faster inference speed. The source code is available at https://github.com/LiheYoung/MiningFSS.
1. Introduction
Few-shot segmentation reduces annotation demands for unseen classes but can undermine their embeddings by treating latent novel classes as background. The paper proposes joint latent-class mining and prototype rectification to improve feature discrimination and prototype stability.
- Few-shot segmentation targets unseen classes using only a handful of annotations, avoiding the exhaustive pixel-wise labeling required by fully supervised segmentation.
- Existing FSS training can over-smooth latent novel-class embeddings because episodes focus on the current support class and label other latent classes as background.
- Few-shot supports cannot fully mimic real class-wise statistics, creating prototype bias when prototypes rely only on current support examples.
- Its auxiliary branch uses transferable semantic sub-clusters to mine latent classes from training backgrounds and can also exploit extra unlabeled data.
- Prototype rectification addresses both foreground and background bias, while experiments report fewer parameters, faster evaluation, and better performance than prior methods.
- The proposed framework jointly mines latent objects and learns pairwise metrics, enabling direct application to unseen classes without further training or fine-tuning.
2. Related Work
Related work spans semantic segmentation, few-shot learning and few-shot segmentation, alongside semi-supervised and self-supervised approaches for using limited or unlabeled data. The paper positions its method against these established directions.
- Semantic Segmentation: Semantic segmentation research has developed benchmark components such as pyramid pooling, dilated convolution, deformable convolution, and non-local modules around fully convolutional networks.
- Few-shot Learning: Few-shot learning seeks to recognize unseen classes from few examples by extracting meta knowledge shared across known and unseen classes.
- Few-shot Segmentation: Early few-shot segmentation generated segmentation parameters from supports, while later methods used embedding similarity and bidirectional support-query alignment.
- Semi-/self-supervised Learning: Semi-supervised methods use consistency regularization, entropy minimization, and pseudo labeling, whereas self-supervised methods mine unlabeled data through pretext tasks.
3. Method
The method jointly trains on conventional support-query episodes and pseudo-labeled extra images to mine latent novel classes and stabilize prototypes. It transfers representative sub-clusters from base-class prototypes, assigns dense pseudo masks by nearest-neighbor similarity, and rectifies background prototypes using dataset-wide statistics.
- 3.1. Problem Definition: Few-shot segmentation trains on base classes but must segment unseen classes from a small support set without retraining.
- 3.2. Overview: Latent novel classes are mined from training backgrounds because treating them as background undermines their feature embeddings.
- 3.2. Overview: The unified framework combines episodic support-query training with auxiliary supervision from pseudo-labeled extra images.Ground-truth masks supervise support-query pairs, while pseudo masks provide additional multi-class supervision.
- 3.3. Representative Sub-clusters: K-Means clusters foreground prototypes into K representative sub-clusters, while averaging all background prototypes produces one global background descriptor.The resulting K + 1 representatives describe recurring foreground characteristics and background features in the training set.
- 3.3.1. Annotating with Representative Sub-clusters: Each image feature is assigned to its most similar representative prototype, producing dense pseudo masks with up to K + 1 semantically consistent regions.The clustered regions do not denote concrete object classes but capture typical characteristics that support discriminative feature learning.
- 3.4. Rectifying Support Prototypes: The background support prototype is rectified with an exponential-moving-average global prototype that captures common scene characteristics.The fusion weight is set to 0.9, and the online global prototype outperforms an offline alternative attributed to training-testing inconsistency.
4. Experiments
Experiments on PASCAL-5i and COCO-20i evaluate MiningFSS across few-shot settings, component ablations, unlabeled-data sources, and efficiency. The method consistently improves segmentation performance while reducing parameters and inference time.
- Comparison with State-of-the-Arts: PASCAL-5i gains 1.3%–3.7% over prior state-of-the-art results across backbones and shot settings, while using 74% fewer parameters.The gains are 1.3% and 2.5% in 1-shot, and 3.1% and 3.7% in 5-shot, for ResNet-50 and ResNet-101 respectively.
- Comparison with State-of-the-Arts: COCO-20i gains 4.0%–7.0% over previous best results across backbones and shot settings, including 7.0% in the 5-shot ResNet-101 setting.The reported improvements are 6.7% and 4.0% in 1-shot, and 3.9% and 7.0% in 5-shot, for ResNet-50 and ResNet-101 respectively.
- Ablation Study: Mining latent classes contributes most to performance improvement, while foreground and background prototype rectification are both indispensable.The full combination of mining, foreground rectification, and background rectification achieves the best ablation performance.
- Ablation Study: Additional data improve performance: ImageNet reuse gives a 2.6% gain, labeled training images used as unlabeled data give a 2.8% gain, and combining sources improves it further.The method is also reported to exploit training and extra unlabeled data more effectively than the compared self-supervised and semi-supervised approaches.
- Efficiency: The model uses 8.7M parameters versus 34.3M for PFENet and runs 1.8x faster in 1-shot and 2.5x faster in 5-shot inference.The efficiency comparison is reported at testing time.
5. Conclusion
The method mines latent classes from backgrounds while learning meta-knowledge from groundtruth and pseudo masks, then rectifies support prototypes. Experiments on two FSS benchmarks show large-margin gains and better use of training data, including unlabeled data.
- The framework mines latent classes from backgrounds while jointly learning meta-knowledge and embeddings from groundtruth and pseudo masks.
- The method adds prototype rectification to improve support prototypes after mining latent classes.
- Experiments on two FSS benchmarks report large-margin improvements over previous methods without additional architectural enhancements.
- Ablation and comparisons with self-supervised and semi-supervised techniques indicate stronger exploitation of training and extra unlabeled data through latent-class mining.