PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation

Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, Joongkyu Kim

arXiv:2104.01893v2cs.CV

TL;DR

Few-shot segmentation must handle unseen categories and substantial support-query variation, while single prototypes may inadequately represent object information. ASGNet addresses this with superpixel-guided clustering and guided prototype allocation, achieving mIoUs of 64.36%/42.48% on Pascal-5i/COCO-20i and exceeding the state-of-the-art by 2.40%/5.08%.

  • Problem

    Few-shot segmentation must handle unseen categories and large appearance and shape variations between support and query images, challenging single-prototype representation.

  • Method

    ASGNet combines superpixel-guided clustering for adaptive multi-prototype extraction with guided prototype allocation for matching prototypes to query features.

  • Results

    64.36%/42.48% mIoUs on Pascal-5i/COCO-20i exceed the state-of-the-art by 2.40%/5.08% in the 5-shot setting.

  • Takeaways & Limitations

    ASGNet delivers top-performing few-shot segmentation with fewer parameters and less computation while adapting to varying object scales, shapes, and occlusions.

Abstract

from arXiv · show

Prototype learning is extensively used for few-shot segmentation. Typically, a single prototype is obtained from the support feature by averaging the global object information. However, using one prototype to represent all the information may lead to ambiguities. In this paper, we propose two novel modules, named superpixel-guided clustering (SGC) and guided prototype allocation (GPA), for multiple prototype extraction and allocation. Specifically, SGC is a parameter-free and training-free approach, which extracts more representative prototypes by aggregating similar feature vectors, while GPA is able to select matched prototypes to provide more accurate guidance. By integrating the SGC and GPA together, we propose the Adaptive Superpixel-guided Network (ASGNet), which is a lightweight model and adapts to object scale and shape variation. In addition, our network can easily generalize to k-shot segmentation with substantial improvement and no additional computational cost. In particular, our evaluations on COCO demonstrate that ASGNet surpasses the state-of-the-art method by 5% in 5-shot segmentation.

1. Introduction

The introduction frames few-shot segmentation as challenging because novel categories and support-query appearance and shape variation complicate mask transfer. ASGNet addresses these issues with adaptive, multi-prototype learning and allocation through SGC and GPA, achieving strong 5-shot results with fewer parameters and less computation.

  • Problem: Few-shot segmentation learns to segment query objects from only a few support images with ground-truth masks, despite novel test categories and support-query appearance and shape variation.The problem is difficult because test categories are absent from training data and support and query images can differ substantially.
  • Limitations: Existing prototypical methods often condense masked support features into a single prototype, losing information and discriminability.Single-prototype learning is typically performed with masked average pooling.
  • Approach: ASGNet adapts the number and spatial extent of prototypes to image content, addressing variation in object scales, shapes, and occlusions.Its adaptive multi-prototype strategy is designed to remain content-adaptive and spatially aware across images.
  • Approach: SGC performs adaptive prototype extraction, while GPA allocates the learned prototypes adaptively to query features.The modules are also presented as plug-and-play components for feature matching.
  • Results: 64.36%/42.48% mIoUs are achieved in 5-shot Pascal-5i/COCO-20i, exceeding the state-of-the-art by 2.40%/5.08%.The method is reported to use fewer parameters and less computation.

2. Related Work

Related work spans conventional semantic segmentation, few-shot learning and segmentation, and superpixel-based representations. The paper is specifically inspired by MaskSLIC and SSN to cluster similar features within a masked region in feature space.

  • Semantic Segmentation: Semantic segmentation commonly uses FCNs for pixel-level prediction, with recent advances emphasizing multi-scale feature aggregation and attention mechanisms.These methods often use dilated convolution kernels and encoder-decoder structures.
  • Few-shot Learning: Few-shot learning targets generalization to new classes from only a few annotated examples, primarily through metric-learning and meta-learning approaches.Prototypical networks have also been extensively adopted in few-shot learning.
  • Few-shot Segmentation: Few-shot segmentation extends few-shot classification by predicting a label for every query-image pixel rather than one label for the entire image.The problem was introduced by Shaban et al. with a classical two-branch network, followed by prototype-based prediction in PL.
  • Superpixel Segmentation: Superpixels group pixels with similar color, texture, or category characteristics and provide compact representations used as basic units in few-shot segmentation.They carry more information than individual pixels and support downstream vision tasks.
  • Superpixel Segmentation: Inspired by MaskSLIC and SSN, the paper proposes masked superpixel clustering in feature space to gather similar features within a defined region of interest.MaskSLIC adapts SLIC to a region of interest, while SSN introduced an end-to-end trainable differentiable SLIC algorithm.

3. Problem Definition

Few-shot segmentation evaluates on classes unseen during training, with disjoint training and testing categories. Given support and query images from the same class, the task estimates the query mask to approximate its ground truth.

  • Problem setting: Training and testing categories are non-overlapping, so inference involves classes completely unseen during training.The formulation states Strain ∩ Stest = ∅.
  • Problem setting: Support and query images share the same class c, and the estimated query mask ˜MQ approximates the ground-truth mask MQ.The support set provides K examples, while the query mask is estimated from them.

4. Proposed Method

ASGNet integrates superpixel-guided clustering (SGC) for extracting multiple compact prototypes with guided prototype allocation (GPA) for adaptive query guidance. The method adapts prototype learning to object scale and shape and supports efficient k-shot segmentation by collecting prototypes across support images.

  • Superpixel-Guided Clustering: SGC clusters similar masked support feature vectors into multiple superpixel centroids that serve as compact prototypes.It incorporates positional information, filters features with the support mask, and iteratively updates centroid assignments and values.
  • Superpixel-Guided Clustering: SGC initializes seeds uniformly inside the foreground mask rather than across the entire image.This adapts superpixel initialization to the masked support region.
  • Guided Prototype Allocation: GPA selects the most similar prototype at each query location and combines the resulting guide feature and probability map with the query feature.Prototype-query similarity is computed with cosine distance, while the two branches provide pixel-wise and aggregated guidance.
  • Adaptive Ability: SGC adapts the number of prototypes to foreground scale, while GPA adapts prototype matching to object shape.For small foregrounds, SGC can reduce to masked average pooling; Nmax limits the maximum number of prototypes.
  • ASGNet Architecture: ASGNet extracts support and query features with a shared ImageNet-pretrained CNN, then applies SGC and GPA before multi-scale segmentation prediction.The architecture integrates both proposed modules for few-shot semantic segmentation.
  • k-Shot Setting: For k-shot segmentation, ASGNet applies SGC independently to each support image-mask pair and collects all resulting centroids for GPA.This gives GPA a larger selection range across shots and enables more accurate query guidance without collapsing support features.

5. Experiments

Experiments on Pascal-5i and COCO-20i evaluate ASGNet using mIoU and FB-IoU, with ablations examining prototype count, adaptive superpixel centroids, SGC, and GPA. ASGNet improves state-of-the-art performance, including a 2.40% 5-shot margin on Pascal-5i and a 4.82% gain from its k-shot fusion strategy without additional computation.

  • Experimental Setup: ASGNet is evaluated on Pascal-5i and COCO-20i, two widely used few-shot semantic-segmentation datasets.Pascal-5i uses four cross-validation splits, with three for training and one for testing.
  • Evaluation Metrics: mIoU is the primary ablation metric, while FB-IoU is additionally reported for consistent comparisons.The experiments also use FLOPs to quantify computation, counting both additions and multiplications.
  • Prototype Ablations: Nmax = 5 gives the best 1-shot ablation performance, while excessive prototypes slightly reduce results and may cause over-fitting.The adaptive superpixel-centroid setting reduces redundant computation compared with a fixed number of five superpixels.
  • SGC and GPA: 4.82% is the largest increment over the 1-shot baseline achieved by the proposed k-shot fusion strategy without additional computation.Attention-based fusion requires substantially more computation and provides limited performance improvement, whereas GPA is effective with many selections.
  • State-of-the-Art Comparisons: 2.40% is ASGNet's 5-shot margin over recent methods on Pascal-5i with ResNet-101, while its 1-shot results remain on par with state-of-the-art methods.ASGNet also achieves competitive 1-shot and top-performing 5-shot FB-IoU results, including on the more challenging COCO-20i dataset.

6. Conclusions

The paper proposes ASGNet for few-shot image segmentation, addressing limitations of existing single-prototype models through adaptive prototype learning and allocation. It introduces Superpixel-guided Clustering (SGC) and Guided Prototype Allocation (GPA) as its core modules.

  • 6. Conclusions: ASGNet targets the limitations of existing single-prototype-based few-shot segmentation models.The model is proposed for few-shot image segmentation.
  • 6. Conclusions: SGC aggregates similar feature vectors using feature-based superpixel clustering for adaptive prototype learning.Superpixel-guided Clustering is one of the two proposed modules.
  • 6. Conclusions: GPA allocates the most relevant prototype to each query feature element for adaptive prototype allocation.It does so by measuring similarity with the query feature element.

A. Additional Experimental Details · A.1. Detailed mean IoU results on COCO-20i · A.2. Calculation of FLOPs

The additional experimental details report per-split mean IoU results on COCO-20i and describe the FLOPs calculation used to evaluate computation and model complexity. ASGNet achieves the best performance in every split, while 256 1×1 convolution filters require 0.9G FLOPs for the merged feature F.

  • A.1. Detailed mean IoU results on COCO-20i: The per-split mean IoU results are presented in Table 8 to demonstrate the method’s superiority.
  • A.2. Calculation of FLOPs: FLOPs are used in the ablation study to evaluate computation and model complexity.The paper provides a detailed calculation for a general convolution layer.
  • A.2. Calculation of FLOPs: For one output pixel, the convolution operation count depends on Cin and K, with separate expressions for bias=False and bias=True.The expressions are given as (Cin · K2) + (Cin · K2 −1) for bias=False and (Cin · K2) + (Cin · K2) for bias=True.
  • A.2. Calculation of FLOPs: The first item denotes multiplication, the second denotes addition, Cin is the input-channel count, and K is the kernel size.
  • A.2. Calculation of FLOPs: 0.9G FLOPs are required when 256 1×1 convolution filters process the merged feature F.For the whole feature map, the calculation uses output-feature dimensions H and W and output-channel count Cout.

A.3. Ablation Study on Iteration Number

The ablation study finds that increasing the iteration number consistently improves both FB-IoU and mIoU, with convergence reached at around five iterations in the 1-shot Pascal-50 setting.

  • Ablation Study on Iteration Number: Both FB-IoU and mIoU increase monotonically as the number of iterations grows.The experiments use the 1-shot setting on Pascal-50.
  • Ablation Study on Iteration Number: Around 5 iterations are sufficient to obtain the converged result.

B. Additional Qualitative Results · B.1. Visual Results on Pascal-5i and COCO-20i · B.2. Visualizations of Simiarity Map

ASGNet qualitatively handles substantial appearance, scale, and shape variation between support and query images, with particular gains in occluded cases. Its similarity maps show that multiple prototypes represent object parts with similar characteristics, improving adaptability and discrimination.

  • B.1. Visual Results on Pascal-5i and COCO-20i: ASGNet handles large appearance, scale, and shape variations between support and query images.The comparison is against a single-prototype baseline.
  • B.1. Visual Results on Pascal-5i and COCO-20i: ASGNet performs particularly better than the single-prototype baseline in occluded cases.The cited examples are columns 3–6 of Figure 9.
  • B. Additional Qualitative Results: The section also includes an ablation study on evaluation iterations.This study is presented in Figure 8.
  • B.2. Visualizations of Simiarity Map: Each similarity map is computed as the cosine distance between a prototype and a query feature.The visualization is used to inspect how the proposed method guides segmentation.
  • B.2. Visualizations of Simiarity Map: The prototypes represent object parts with similar characteristics, making the network more adaptive and discriminative.This interpretation is drawn from the similarity-map visualizations in Figure 10.
  • B. Additional Qualitative Results: Qualitative comparisons include examples from both Pascal-5i and COCO-20i.Figure 9 places Pascal-5i examples on the left and COCO-20i examples on the right.
  • B.2. Visualizations of Simiarity Map: The number of prototypes in the Pascal-5i similarity-map visualization is determined by the support-object size.Figure 10 illustrates this prototype-count relationship.
Loading 2104.01893v2…