PaperlayerView exact revision on arXiv

Source-linked AI summary

PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment

Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, Jiashi Feng

arXiv:1908.06391v2cs.CV

TL;DR

Few-shot segmentation seeks to reduce the annotation burden and improve generalization to unseen categories. PANet learns class-specific support prototypes in an embedding space, segments query pixels by non-parametric matching, and aligns support and query prototypes during training. On PASCAL-5i, it reports 48.1% mIoU for 1-shot and 55.7% for 5-shot segmentation, surpassing state-of-the-art methods by up to 8.6%.

  • Problem

    Deep CNN segmentation models require many densely annotated images and generalize poorly to unseen object categories, motivating few-shot segmentation from limited annotated examples.

  • Method

    PANet extracts class-specific prototypes from support images, segments query pixels through non-parametric nearest-prototype matching, and applies prototype alignment regularization during training.

  • Results

    48.1% and 55.7% mIoU are achieved on PASCAL-5i for 1-shot and 5-shot settings, respectively, outperforming state-of-the-art methods by up to 8.6%.

  • Takeaways & Limitations

    PANet exploits support information through robust, discriminative prototypes and prototype alignment while avoiding decoder and post-processing modules.

  • Takeaways & Limitations

    PANet can produce unnatural patches and confuse chairs with tables when their prototypes are similar in the embedding space.

Abstract

from arXiv · show

Despite the great progress made by deep CNNs in image semantic segmentation, they typically require a large number of densely-annotated images for training and are difficult to generalize to unseen object categories. Few-shot segmentation has thus been developed to learn to perform segmentation from only a few annotated examples. In this paper, we tackle the challenging few-shot segmentation problem from a metric learning perspective and present PANet, a novel prototype alignment network to better utilize the information of the support set. Our PANet learns class-specific prototype representations from a few support images within an embedding space and then performs segmentation over the query images through matching each pixel to the learned prototypes. With non-parametric metric learning, PANet offers high-quality prototypes that are representative for each semantic class and meanwhile discriminative for different classes. Moreover, PANet introduces a prototype alignment regularization between support and query. With this, PANet fully exploits knowledge from the support and provides better generalization on few-shot segmentation. Significantly, our model achieves the mIoU score of 48.1% and 55.7% on PASCAL-5i for 1-shot and 5-shot settings respectively, surpassing the state-of-the-art method by 1.8% and 8.6%.

1. Introduction

PANet addresses few-shot segmentation by separating prototype extraction from non-parametric metric-based prediction and adding prototype alignment to exploit support information. It achieves strong PASCAL-5i performance without decoder or post-processing modules.

  • Few-shot segmentation reduces reliance on densely annotated images but still faces poor generalization to unseen object categories.
  • Prototype alignment regularization reverses segmentation from query to support, encouraging consistent prototypes across support and query images.The query image and predicted mask form a new support set during training.
  • PANet extracts class-specific prototypes from support images and labels each query pixel by matching its embedding to the nearest prototype.The model uses a shared feature extractor and metric learning rather than a parametric classification module.
  • PANet introduces no extra learnable parameters, requires no additional inference passes, and does not increase inference computation because alignment is training-only.
  • PANet can learn from weak annotations such as bounding boxes or scribbles and is directly applicable to learning from a few weakly annotated examples.
  • 48.1% and 55.7% mIoU are achieved on PASCAL-5i in the 1-shot and 5-shot settings, respectively, with improvement up to 8.6%.

2. Related work

Prior few-shot segmentation methods use conditioning, feature concatenation, decoders, or prototype-guided architectures. PANet instead uses a simpler metric-learning design that directly segments through prototype matching and incorporates annotation masks through late fusion.

  • Few-shot segmentation methods commonly condition query segmentation on support-derived parameters, concatenate support and query features, or use decoders.
  • Earlier prototypical approaches can involve complex multi-stage training and use prototypes to guide segmentation rather than directly obtain masks through metric learning.
  • In a 2-way 1-shot pipeline, masked average pooling forms prototypes, cosine distances classify query locations, and prototype alignment predicts masks for support images.
  • PANet adopts a simpler Prototypical Network-like design that performs segmentation directly by matching query features to support-derived prototypes.
  • PANet uses late fusion to incorporate annotation masks, supporting generalization to sparse or changing annotations.

3. Method

PANet formulates few-shot segmentation as prototype extraction followed by non-parametric metric learning in an embedding space. Prototype alignment reverses the support-query direction during training to encourage consistent representations, while inference adds no extra computation.

  • Problem setting: Few-shot episodes train on support image-mask pairs and evaluate segmentation on query pairs from the same classes, including unseen test classes.Training uses Cseen, while testing uses non-overlapping Cunseen classes.
  • Prototype-based segmentation: PANet separates support knowledge extraction from segmentation by learning compact class prototypes and matching query pixels to their nearest prototypes.A shared backbone embeds support and query images; masked average pooling extracts prototypes from support features.
  • Efficiency and annotation flexibility: PANet introduces no extra learnable parameters, performs prototype embedding and prediction on existing feature maps, and adds no inference-time cost for alignment regularization.The regularization strength is set to λ = 1 in the reported experiments.
  • Prototype-based segmentation: Foreground and background prototypes are computed separately from masked support feature maps, producing representative and discriminative class representations.The prototypes are optimized end-to-end through non-parametric metric learning.
  • Non-parametric metric learning: Query segmentation applies a distance-based softmax over prototypes to produce probabilities for each semantic class, including background.The distance function may use cosine or squared Euclidean distance, with cosine scaling affecting softmax behavior.
  • Prototype alignment regularization: Prototype alignment regularization swaps query and support roles: predicted query masks generate prototypes that segment the original support images.This reverse-direction loss encourages mutual alignment between support and query prototypes.
  • Efficiency and annotation flexibility: The model also accepts weaker support annotations such as scribbles and bounding boxes while retaining comparably good query segmentation results.These annotations are easier to obtain than dense pixel-level annotations.

4. Experiments

PANet is evaluated across PASCAL-5i and MS COCO using one- and two-way few-shot settings, multiple metrics, and weak support annotations. It consistently outperforms prior methods, benefits more from additional support images, and uses prototype alignment to improve training.

  • 4.1. Setup: PANet outperforms state-of-the-art methods on PASCAL-5i in both 1-shot and 5-shot mean-IoU settings.It uses fewer parameters and no decoder or post-processing module.
  • 4.1. Setup: Up to 7.6% mean-IoU improvement separates PANet’s 1-shot and 5-shot results, whereas other methods improve by less than 3.1%.PANet-init gains more than 11% in mean-IoU with additional support images despite no further training.
  • 4.2. Comparison with state-of-the-arts: More than 20% improvement over previous works is reported for PANet on 2-way PASCAL-5i segmentation.Qualitative results show segmentation on unseen classes with one annotated support image, without decoder structure or post-processing.
  • 4.2. Comparison with state-of-the-arts: The model can produce unnatural segmentation patches and confuse chairs with tables when their embedding prototypes are similar.Post-processing can alleviate the patch artifacts, but prototype similarity remains a reported failure mode.
  • 4.3. Ablation study: Prototype alignment regularization aligns support and query prototypes, improving results and accelerating convergence.The regularizer transfers information from the query set back to the support set during training.
  • 4.4. Test with weak annotations: In 5-shot learning, scribble annotations outperform bounding boxes by 2% in mean-IoU.The authors attribute this to scribbles producing more representative prototypes and bounding boxes introducing more noise.

5. Conclusion

PANet concludes that metric learning over robust prototypes enables effective few-shot segmentation without decoder or post-processing steps. Prototype alignment further exploits support information and supports strong performance across the task.

  • 5. Conclusion: PANet performs few-shot segmentation by extracting robust support prototypes and matching query pixels through non-parametric distance calculation.Prototype alignment regularization further exploits support information during training.
  • 5. Conclusion: PANet outperforms previous work by a large margin without a decoder structure or post-processing step.
Loading 1908.06391v2…