PaperlayerView exact revision on arXiv

Source-linked AI summary

Part-aware Prototype Network for Few-shot Semantic Segmentation

Yongfei Liu, Xiangyi Zhang, Songyang Zhang, Xuming He

arXiv:2007.06309v3cs.CV

TL;DR

Few-shot semantic segmentation must adapt to new classes from limited annotations, while prior methods may be restricted to one-way settings or incompletely represent objects. The paper introduces PPNet, which learns graph-enhanced part-aware prototypes from labeled and unlabeled images. Evaluations on few-shot segmentation benchmarks report large-margin, state-of-the-art performance, with reduced prototype representation power remaining in the 1-shot case.

  • Problem

    Existing methods face one-way-setting restrictions, holistic representations, and limited support data for modeling diverse object appearances and intra-class variation.

  • Method

    PPNet decomposes class representations into part-aware prototypes and uses a graph neural network to enhance them from labeled and unlabeled support images.

  • Results

    The method outperforms prior work by a large margin and achieves state-of-the-art performance on few-shot segmentation benchmarks.

  • Takeaways & Limitations

    Part-aware prototypes and unlabeled-data learning provide a framework for capturing diverse class appearances in one-way and multi-way few-shot segmentation.

  • Takeaways & Limitations

    1-shot performance is affected by the limited representation power of prototypes learned from a single support image.

Abstract

from arXiv · show

Few-shot semantic segmentation aims to learn to segment new object classes with only a few annotated examples, which has a wide range of real-world applications. Most existing methods either focus on the restrictive setting of one-way few-shot segmentation or suffer from incomplete coverage of object regions. In this paper, we propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes, capable of capturing diverse and fine-grained object features. In addition, we propose to leverage unlabeled data to enrich our part-aware prototypes, resulting in better modeling of intra-class variations of semantic objects. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes based on labeled and unlabeled images. Extensive experimental evaluations on two benchmarks show that our method outperforms the prior art with a sizable margin.

1 Introduction

Few-shot semantic segmentation reduces annotation demands but existing methods struggle with multi-way tasks, holistic class representations, and limited support data. PPNet addresses these issues with part-aware prototypes, unlabeled-data enrichment, and graph-based prototype learning.

  • Motivation: Few-shot semantic segmentation targets new object classes using only a few annotated examples, reducing reliance on costly pixel-wise labeling.Conventional supervised segmentation often requires thousands of labeled images.
  • Limitations of Prior Work: Prior matching-based methods may be restricted to one-way segmentation or computationally expensive to generalize to multi-way settings.These methods use dense pair-wise matching and specialized decoding networks.
  • Limitations of Prior Work: Holistic prototypes struggle with diverse object parts, poses, subcategories, and rich intra-class variation from small support sets.The limitation concerns both incomplete object representation and restricted support information.
  • Proposed Framework: PPNet decomposes holistic class representations into part-aware prototypes that capture fine-grained features and improve spatial coverage of semantic regions.The framework also uses a flexible matching strategy applicable to one-way and multi-way settings.
  • Proposed Framework: Unlabeled images augment the support set, enabling prototype learning from labeled and unlabeled data to better model intra-class variation.The paper refers to this setting as semi-supervised few-shot semantic segmentation.
  • Evaluation: Experiments on PASCAL-5i and COCO-20i report that part-aware prototype learning outperforms the state of the art by a large margin.The evaluation includes an ablation study of the method.
  • Proposed Framework: A graph attention network supports semi-supervised prototype learning, while the framework combines embedding, prototype generation, and mask generation modules.An auxiliary semantic branch and augmented loss are also used for network learning.

2 Related Work

Prior few-shot segmentation work includes metric-, optimization-, prototype-, and graph-based approaches. Prototype methods are efficient but commonly rely on holistic class representations, motivating part-level modeling.

  • Few-shot Learning: Few-shot learning methods are commonly categorized as metric-learning, optimization-learning, or graph-neural-network approaches.The paper positions its method within metric-learning-based work.
  • Semi-supervised Learning: Semi-supervised few-shot learning incorporates unlabeled data through prototype refinement or consistency losses, mainly in few-shot classification.These approaches motivate using unlabeled data for segmentation prototypes.
  • Few-shot Segmentation: Few-shot semantic segmentation includes parametric matching-based and prototype-based methods, with MetaSegNet as an optimization-based exception.MetaSegNet formulates segmentation as a pixel-classification problem.
  • Parametric Matching: Parametric matching methods use mechanisms such as weight imprinting, dense comparison, or graph attention to compare support and query features.These methods differ in how they construct comparisons and predictions.
  • Prototype Methods: Prototype-based methods perform pixel-wise query matching with holistic semantic-class prototypes, whose representations are limited for diverse object features.The paper addresses this limitation by decomposing object representations into part-level features.
  • Graph Neural Networks: Graph neural networks learn representations on graph-structured data through message passing and graph convolutions.This graph-learning literature provides the basis for the paper’s prototype enhancement model.

3 Problem Setting

The paper formulates few-shot segmentation as episodic meta-learning over support and query sets, with test classes disjoint from training classes. Its semi-supervised setting adds unlabeled support images.

  • Task Formulation: Few-shot semantic segmentation learns to segment semantic objects from only a few annotated training images per class.A meta learner is trained across a distribution of segmentation tasks.
  • Task Formulation: Each episode contains support data with ground-truth masks and query images whose annotations are available during training but unknown at test time.The meta learner maps a support set and query image to a query segmentation.
  • Model Pipeline: The model overview processes support, unlabeled, and query images through embedding, prototype generation, and part-aware mask generation networks.A semantic branch additionally predicts masks over the global training class space Ctr.
  • Semi-supervised Setting: In the semi-supervised setting, support data combines annotated image-label pairs with unlabeled images.The unlabeled support images have their class labels removed during training and inference.
  • Task Formulation: For C-way K-shot tasks, annotated support provides K image-label pairs from each of C task classes.The task class set is a subset of the broader semantic class set.
  • Meta-learning Protocol: Training uses episodically sampled tasks from Ctr, while evaluation uses tasks from a non-overlapping class set Cte.This separates meta-training classes from test classes.

4 Our Approach

PPNet represents each semantic class with part-aware prototypes generated from labeled support data and refined using unlabeled features. A graph-based pipeline then matches these prototypes to query features for segmentation.

  • PPNet captures fine-grained and intra-class variation by replacing holistic class representations with multiple part-aware prototypes and using unlabeled data.
  • The framework contains embedding, prototype generation, and part-aware mask generation networks for processing task images and producing query segmentation.
  • Part Generation with Labeled Data: The Part Generation Module clusters labeled object features into Np part prototypes and augments each with global class context through attention.
  • Part Refinement with Unlabeled Data: The Part Refinement Module groups and prunes noisy unlabeled features, applies graph attention for contextual smoothing, and aggregates relevant features into refined prototypes.
  • Part-aware Mask Generation Network: For query prediction, the model computes prototype-pixel similarity maps, max-pools part scores within each class, concatenates class maps, and selects the highest-scoring label after upsampling.

5 Experiments

Experiments evaluate the method on PASCAL-5i and COCO-20i across few-shot settings, reporting strong comparisons, qualitative behavior, and component ablations.

  • Experimental configuration: The method is evaluated on PASCAL-5i and COCO-20i using mean-IoU, with comparisons across one-way, multi-way, and ablation settings.PASCAL-5i uses four-fold cross-validation; mean-IoU is the primary metric.
  • PASCAL-5i results: 52.84% mean-IoU in 1-shot and 62.97% in 5-shot are achieved after refining part prototypes with unlabeled data on PASCAL-5i.With ResNet-50, the method also reports 61.96% in 5-shot, while ResNet-101 reaches 65.10% in 5-shot.
  • Visualization analysis: Qualitative visualizations associate part prototypes with different body parts and show improved handling of appearance and scale variation using unlabeled images.Heatmap brightness denotes higher prototype-query similarity; masks distinguish predictions from support and query ground truth.
  • COCO-20i results: On COCO-20i, the method achieves state-of-the-art performance across different splits and generally improves over PANet* by a sizable margin.In split-B, it is superior to FWB in nearly every fold except fold-4 in 1-shot.
  • Ablation study: Ablations improve averaged mean-IoU from 22.95% to 23.35% with part-aware prototypes and to 25.02% after global-context augmentation.The full PPNet reaches 27.16% averaged mean-IoU when the graph attention network exploits unlabeled data.

6 Conclusion

The conclusion presents a flexible prototype-based few-shot segmentation method that models fine-grained object features and intra-class variation with unlabeled data. Across few-shot benchmarks, it reports large-margin improvements and state-of-the-art performance.

  • Conclusion: The method represents semantic classes with part-aware prototypes to capture diverse appearances and fine-grained object features.It is designed as a flexible prototype-based approach for few-shot semantic segmentation.
  • Conclusion: Unlabeled data are leveraged to capture intra-class prototype variations through a graph neural network operating on support images with and without annotations.The paper characterizes this as a semi-supervised few-shot semantic segmentation framework.
  • Conclusion: Evaluations on few-shot segmentation benchmarks report large-margin improvements over prior work and state-of-the-art performance.The conclusion summarizes results across several benchmarks without giving a single aggregate score.

Part-aware Prototype Network for Few-shot Semantic Segmentation Supplementary Material

The supplementary material adds binary-IoU results, complexity details, multi-way experiments, graph-attention analysis, and additional visualizations.

  • Supplementary analyses: The supplement reports binary-IoU results for PASCAL-5i 1-way experiments and details model complexity.These analyses supplement the main experimental results.
  • Supplementary analyses: Additional experiments examine model versatility in multi-way settings and the effectiveness of graph attention for using unlabeled data.The supplement also provides further analysis related to the main method.
  • Supplementary analyses: More qualitative visualizations are provided for both PASCAL-5i and COCO-20i.These visualizations extend the examples presented in the main paper.

1 Binary-IoU for PASCAL-5i

On PASCAL-5i, the model is evaluated using averaged binary-IoU across four folds in the 1-way setting and outperforms PGNet at both shot levels.

  • 70.90% binary-IoU at 1-shot, a 1.0% gain over PGNet.The result is reported for the 1-way PASCAL-5i setting.
  • 77.45% binary-IoU at 5-shot, a 6.95% gain over PGNet.The comparison is reported in terms of binary-IoU.
  • The authors associate these gains with improved discrimination of complex backgrounds.

2 Model Complexity of 1-way 1-shot Setting

The method incurs extra prototype-generation computation from unlabeled data, while maintaining inference cost similar to PANet*. Because prototypes are generated once per task, query inference dominates average cost with enough queries.

  • Model complexity: Prototype generation uses more FLOPs than PANet* because the method incorporates extra unlabeled data.The comparison separates prototype-generation cost, Cg, from query inference cost, Cq.
  • Model complexity: Query-image inference cost is similar to PANet*.
  • Model complexity: With sufficient queries, one-time prototype generation is amortized and inference dominates average computation cost.

3 More Quantitative Results on COCO-20i

On COCO-20i, the model outperforms PANet* in 2-way and 5-way settings with and without unlabeled data. The graph attention network also improves performance over a non-parametric alternative.

  • Quantitative comparison: The model outperforms PANet* by a sizeable margin in 2-way and 5-way COCO-20i settings, with and without unlabeled data.In the 5-way 1-shot setting, mean-IoU improves consistently across every fold.
  • Quantitative comparison: 5-way 1-shot mean-IoU improves consistently in each COCO-20i fold.

4 More Qualitative Visualization for PASCAL-5i and COCO-20i

Qualitative results show that the model handles appearance and scale variation between support and query images while modeling classes at a fine-grained, part-aware level across PASCAL-5i and COCO-20i.

  • Visualization for PASCAL-5i: The model copes with large appearance and scale variations in both 1-way 1-shot and 2-way 1-shot PASCAL-5i settings.The visualizations distinguish appearance-variation and scale-variation scenarios.
  • Visualization for PASCAL-5i: The qualitative examples attribute robustness to utilizing unlabeled data.
  • Visualization for COCO-20i: On COCO-20i, part-aware prototypes model one semantic class at a fine-grained level despite support-query variation.The visualizations include part-prototype prediction heatmaps and appearance- and scale-variation scenarios.
Loading 2007.06309v3…