Source-linked AI summary
Self-Support Few-Shot Semantic Segmentation
Qi Fan, Wenjie Pei, Yu-Wing Tai, Chi-Keung Tang
TL;DR
Few-shot semantic segmentation lacks support coverage for the appearance variations present in query objects. The paper introduces query-derived self-support matching, adaptive background prototypes, and self-support loss, achieving state-of-the-art performance across multiple benchmarks while benefiting from stronger backbones and more supports.
Problem
Few-shot supports are limited and often fail to cover the underlying appearance variations of target classes in query images.
Method
The method generates self-support prototypes from high-confidence query predictions, matches them to query features, and adds adaptive background prototypes and self-support loss.
Results
The self-support network achieves state-of-the-art results on multiple benchmarks and reaches 87.3/88.6 mIoU in 1/5-shot FSS-1000 settings with refinement.
Takeaways & Limitations
Query-derived prototypes narrow the gap between support prototypes and query features while improving prototype quality and scaling with stronger backbones and more supports.
Abstract
from arXiv · showhide
Existing few-shot segmentation methods have achieved great progress based on the support-query matching framework. But they still heavily suffer from the limited coverage of intra-class variations from the few-shot supports provided. Motivated by the simple Gestalt principle that pixels belonging to the same object are more similar than those to different objects of same class, we propose a novel self-support matching strategy to alleviate this problem, which uses query prototypes to match query features, where the query prototypes are collected from high-confidence query predictions. This strategy can effectively capture the consistent underlying characteristics of the query objects, and thus fittingly match query features. We also propose an adaptive self-support background prototype generation module and self-support loss to further facilitate the self-support matching procedure. Our self-support network substantially improves the prototype quality, benefits more improvement from stronger backbones and more supports, and achieves SOTA on multiple datasets. Codes are at \url{https://github.com/fanq15/SSP}.
1 Introduction
Few-shot semantic segmentation is limited by supports that cannot cover target-class appearance variation. The paper addresses this gap with query-derived self-support matching and complementary background and loss modules, reporting stronger performance and favorable scaling properties.
- Motivation: Few-shot semantic segmentation uses only a few support samples to segment arbitrary novel classes, limiting coverage of target-class appearance variation.Typical support counts are 1, 3, 5, or 10 images per class.
- Method: Self-support matching uses query prototypes generated from confident query features to match query features and reduce support-query appearance discrepancy.Initial mask predictions identify confident regions whose features form the self-support prototype.
- Method: The method applies self-support matching to foreground and background, generating adaptive background prototypes from locally similar query pixels rather than aggregating cluttered background globally.A self-support loss further facilitates the procedure.
- Novelty: The approach is distinguished from prior query-prototype methods because it matches query prototypes to query features during the main matching process, rather than using query prototypes only for auxiliary training.Earlier approaches also improve support prototypes, use feature priors, or rely on extra unlabeled images.
- Results: The self-support method improves prototype quality and benefits more from stronger backbones and additional supports, while offering high-confidence predictions and robustness to weak support labels.The paper reports these properties from experiments across multiple datasets.
2 Related Works
Prior semantic segmentation methods rely on abundant pixel-level annotations, while few-shot learning studies recognition from very few samples. The paper is positioned within metric-based support-query learning and prototype-generation approaches for few-shot segmentation.
- Semantic Segmentation: Semantic segmentation produces dense pixel-wise predictions, but conventional methods rely heavily on abundant pixel-level annotations.The related-work discussion describes encoder-decoder, pyramid, context, and convolutional modules developed within this setting.
- Few-Shot Learning: Few-shot learning approaches include transfer learning, optimization-based meta-learning, and metric-based methods that evaluate relevance between support-query pairs.The paper follows the metric-based few-shot learning setting.
- Few-Shot Segmentation: Existing few-shot segmentation methods improve limited supports through matching mechanisms, representative support prototypes, extra unlabeled data, feature priors, and attention or adaptive prototype generation.Prior methods have also explored query prototypes, but not for the same main self-support matching role.
3 Self-Support Few-Shot Semantic Segmentation
The method addresses appearance gaps caused by scarce support examples by generating query-derived self-support prototypes for matching. It combines foreground and adaptive background prototypes with support matching and adds self-support supervision.
- Self-Support Matching: Query pixels are aggregated into self-support prototypes that match query features to reduce the feature gap between support and query objects.The initial mask prediction selects confident query features, which form prototypes used for subsequent matching.
- Self-Support Matching: The network first predicts an initial mask from support prototypes, then generates self-support foreground and adaptive background prototypes before combining them for final matching.This architecture uses the initial query mask to aggregate query features and incorporates the resulting prototypes with the original support prototype.
- Evaluation: Partial and noisy self-support prototypes outperform traditional support prototypes in the reported 1-shot matching evaluation.The comparison is described for prototypes aggregated from full or partial objects, including partial prototypes with 20% noise.
- Prototype Generation: Self-support foreground prototypes use masked average pooling, whereas adaptive background prototypes aggregate similar background pixels separately for each query pixel.The distinction reflects stronger global semantic commonality for foreground regions and mainly local similarity in cluttered backgrounds.
- Optimization: The method adds a query self-support loss to further facilitate matching between query features and their generated prototype.The loss applies training supervision to the query feature and its prototype.
4 Experiments
Experiments evaluate the method across standard few-shot segmentation benchmarks, ablations, and robustness settings. The results show strong performance, gains from self-support components, and benefits from stronger backbones and more supports.
- Comparison with State-of-the-Arts: 73.1% overall performance on PASCAL-5i is achieved with self-support refinement, surpassing the previous SOTA by 2.7%.The method also improves over MLC by 2.7% to 3.7% in the 5-shot setting with ResNet-50 and ResNet-101, respectively.
- Comparison with State-of-the-Arts: 1.3/1.8% improvements over MLC are obtained on COCO-20i in 1/5-shot settings with the ResNet-101 backbone.The method achieves SOTA under the HSNet evaluation protocol and performs comparably or best with ResNet-50.
- Ablation Studies: 68.8% performance is obtained after integrating self-support matching, self-support loss, and adaptive self-support background prototypes, up from 64.8%.The self-support module contributes 2.5%, the loss further raises performance to 68.1%, and ASBP adds 0.9%.
- Self-Support Advantages: 3.7% improvement comes from switching ResNet-50 to ResNet-101, while 8.5% is gained by increasing support images from 1-shot to 5-shot.The authors attribute these gains to better self-support prototypes.
- Self-Support Advantages: ~28 FPS is achieved on a Tesla V100 with ResNet-50 in the 1-shot setting, while the method remains robust to weak support annotations and applicable to other models.The method is non-parametric and improves both PANet and PPNet when added to them.
5 Conclusion
The paper addresses intra-class appearance discrepancy in few-shot segmentation with query-derived self-support prototypes and self-support matching. Adaptive background prototypes and self-support loss further facilitate the procedure, which achieves SOTA on multiple benchmarks.
- Conclusion: The method generates self-support prototypes from query features and matches them with query features to address intra-class appearance discrepancy.It additionally introduces adaptive self-support background prototypes and a self-support loss.
- Conclusion: The self-support network achieves SOTA on multiple benchmarks and is examined through extensive experiments and analysis.
6 More Implementation Details
The implementation combines original support, self-support, and refined self-support prototypes to produce the final segmentation output. Refinement repeats self-support prototype generation from the predicted mask, with ASBP handling background features adaptively.
- Model Setup: The baseline uses MLC’s metric-learning framework with only an encoder.
- Self-Support Refinement: Refined foreground and background self-support prototypes are generated from the predicted mask using MAP and ASBP over query features.Foreground and background mask thresholds are set to τfg = 0.7 and τbg = 0.6.
- Prototype Combination: The refined support prototype combines support, self-support, and refined self-support prototypes with weights α1 = 0.5, α2 = 0.2, and α3 = 0.3.
- Prototype Matching: The matching output M3 is computed by applying softmax to cosine similarity between the augmented support prototype and query features.
- Final Prediction: The final output combines M2 and M3 with β1 = 0.3 and β2 = 0.7.
7 More Quantitative Results
Additional evaluations confirm the method’s stability and generalization across FSS-1000 and Pascal VOC. Performance improvements remain consistent across baseline models, backbones, and support-shot settings.
- Evaluation Protocol: Five evaluations with different random seeds are repeated to obtain stable experimental results.
- FSS-1000: 87.3/88.6 mIoU is achieved on FSS-1000 in 1/5-shot settings after self-support refinement.The dataset contains 1,000 classes and evaluation uses 2,400 randomly sampled support-query pairs.
- Pascal VOC: Large performance improvements are consistently obtained on Pascal VOC across different baseline models, backbones, and support shots.The gains are larger with stronger backbones and more support shots.
8 More Qualitative Results
The 1-shot visualizations show that support-query appearance discrepancies limit initial masks to small object regions. Adding self-support produces substantially improved qualitative segmentation results.
- 1-shot visualizations compare the model’s initial and final outputs across Figures 6–9.The qualitative results use a ResNet-50 backbone.
- Large appearance discrepancies between same-class support and query objects cause initial predictions to cover only small target-object regions.The initial masks use foreground and background thresholds on the original mask prediction M1.
- The self-support method produces satisfactory results with substantial qualitative improvement over the initial predictions.