PaperlayerView exact revision on arXiv

Source-linked AI summary

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer

Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, Tao Xiang

arXiv:2108.03032v3cs.CV

TL;DR

Few-shot semantic segmentation must adapt a complex encoder, decoder, and classifier from as few as one annotated support image. The paper pre-trains and freezes the encoder and decoder, then uses a Classifier Weight Transformer to adapt classifier weights per query image. Across two benchmarks, it reports state-of-the-art performance, with larger gains in 5-shot and cross-domain settings.

  • Problem

    Existing methods meta-learn all three segmentation components, making adaptation to a new class challenging under very limited support supervision.

  • Method

    The method pre-trains and freezes the encoder and decoder, meta-learns the classifier, and uses CWT to adapt support-trained classifier weights independently for each query image.

  • Results

    The method achieves state-of-the-art performance on two standard benchmarks, including 2.3% mIoU over the best ResNet-50 competitor in 1-shot and 5.1%/4.6% gains with ResNet-50/ResNet-101 in 5-shot.

  • Takeaways & Limitations

    Pre-trained feature representations can support few-shot segmentation, while query-specific classifier transformation addresses cases where target objects differ substantially from support objects.

  • Takeaways & Limitations

    The model fails when target instances exhibit extreme appearance changes between support and query images, and its pre-training assumption is weaker with fewer training classes and images.

Abstract

from arXiv · show

A few-shot semantic segmentation model is typically composed of a CNN encoder, a CNN decoder and a simple classifier (separating foreground and background pixels). Most existing methods meta-learn all three model components for fast adaptation to a new class. However, given that as few as a single support set image is available, effective model adaption of all three components to the new class is extremely challenging. In this work we propose to simplify the meta-learning task by focusing solely on the simplest component, the classifier, whilst leaving the encoder and decoder to pre-training. We hypothesize that if we pre-train an off-the-shelf segmentation model over a set of diverse training classes with sufficient annotations, the encoder and decoder can capture rich discriminative features applicable for any unseen classes, rendering the subsequent meta-learning stage unnecessary. For the classifier meta-learning, we introduce a Classifier Weight Transformer (CWT) designed to dynamically adapt the supportset trained classifier's weights to each query image in an inductive way. Extensive experiments on two standard benchmarks show that despite its simplicity, our method outperforms the state-of-the-art alternatives, often by a large margin.Code is available on https://github.com/zhiheLu/CWT-for-FSS.

1. Introduction

Few-shot semantic segmentation reduces annotation demands but must adapt a complex encoder–decoder–classifier model from very limited support data. This work instead pre-trains and freezes the encoder and decoder, meta-learning only the classifier for simpler new-class adaptation.

  • Few-shot segmentation reduces the need for exhaustive pixel-level annotations by adapting to a new class from a support set as small as one image.This addresses the scalability and annotation-cost limitations of conventional semantic segmentation.
  • Adapting a segmentation model from few support images is difficult because modern systems contain interconnected CNN encoder, decoder, and classifier components.The classifier is much simpler than the encoder and decoder and commonly uses a 1 × 1 convolutional layer.
  • Existing methods typically meta-learn all three components so they can adapt jointly to a novel class using annotated support images and query images.This episodic strategy imitates testing by sampling a class, support set, and query set during training.
  • The proposed training paradigm pre-trains and freezes the encoder and decoder while meta-learning only the simpler classifier component.The goal is to make few-shot adaptation to new classes more tractable by reducing the adaptation complexity.
  • The method assumes diverse pre-training can produce class-agnostic encoder and decoder features that transfer to unseen test classes.The paper reports that its classifier-focused approach achieves state-of-the-art performance across two backbones and often surpasses alternatives by a large margin.

2. Related Work

Few-shot semantic segmentation extends meta-learning beyond image classification, with prior methods relating support and query features before updating the full segmentation model. This work uses self-attention differently, transforming classifier weights to address intra-class variation.

  • Few-shot learning has been applied to semantic segmentation to address learning new segmentation tasks from limited labeled examples.
  • A segmentation system generally comprises an encoder, decoder, and classifier, while existing meta-learning methods commonly update all three using query prediction loss.Support and query features are typically related through prototypical learning or feature concatenation.
  • Prototypical methods predict foreground and background through prototype similarity, with later work adding prototype alignment and part-aware prototypes.
  • The Classifier Weight Transformer uses self-attention to adapt a support-set-trained classifier to each query image, targeting intra-class appearance variation.Unlike FEAT, which calibrates relationships among class prototypes for image classification, CWT applies the Transformer to classifier adaptation in segmentation.

3. Methodology

The method simplifies few-shot semantic segmentation by pre-training and freezing the encoder and decoder, then meta-learning only a classifier and a query-conditioned Classifier Weight Transformer. During episodic training and meta-testing, support-set classifier weights are adapted to each query image before segmentation.

  • Task Definition: Few-shot segmentation samples a class and forms disjoint support and query sets for episodic task training.The support set contains K labeled images, while the query set contains Q samples; the standard setting uses Q = 1.
  • Motivation: Existing methods meta-learn the encoder, decoder, and classifier, requiring all three components to adapt from a small annotated support set.The classifier separates foreground and background pixels, whereas the encoder and decoder form the more complex feature representation pipeline.
  • Two-Stage Training: The proposed two-stage paradigm pre-trains the encoder and decoder with supervised learning, freezes them, and meta-trains only the classifier.The method uses PSPNet as the backbone and treats the pre-trained representation as sufficiently generalizable to unseen classes.
  • Feature Reuse and Limitation: Directly reusing pre-trained features with a newly trained classifier is already a strong baseline, but it cannot handle query objects that differ substantially from support objects.The CWT addresses this intra-class variation by adapting classifier weights to individual query images.
  • Classifier Weight Transformer: A Classifier Weight Transformer adapts support-initialized classifier weights to each query image using classifier-to-query attention and residual learning.Classifier weights and query features are projected into a latent space; attention-weighted Value aggregation produces the adapted weights, which are trained with query cross-entropy loss.
  • Meta-Testing: At meta-testing, the classifier is trained on the support set, updated by the frozen CWT for each query image, and then applied for segmentation.Both support and query images condition the transformer, while the transformer parameters remain fixed after meta-training.

4.1. Datasets and Settings

Experiments use COCO-20i and PASCAL-5i, each divided into four class splits for base-class training and novel-class testing under a similar protocol.

  • COCO-20i: COCO-20i contains 82,081 training and 40,137 validation images across 80 classes, divided into four 20-class splits.Each experiment trains on three splits and tests on the remaining split, yielding four experiments.
  • PASCAL-5i: PASCAL-5i contains 5,953 training and 1,449 validation images across 20 categories, divided into four five-class splits.Its experiments use a protocol similar to COCO-20i, with class splits serving as base and test classes.

4.2. Implementation Details and Metrics

Implementation pre-trains PSPNet on base classes with supervised cross-entropy learning, evaluates ResNet-50 and ResNet-101 backbones, and uses fixed optimization and augmentation settings.

  • Model and Backbones: The model uses PSPNet with ResNet-50 and ResNet-101 backbones, while VGG-16 results are reported in the supplementary material.The encoder and decoder are pre-trained on 16/61 base classes, including background, for PASCAL-5i/COCO-20i splits.
  • Optimization: Pre-training runs for 100 epochs on PASCAL-5i and 20 epochs on COCO-20i using cross-entropy loss and SGD with momentum 0.9.The batch size is 12, image size is 417, weight decay is 1e-4, and the initial learning rate is 2.5e-3 with cosine scheduling.
  • Regularization and Augmentation: Training uses label smoothing ϵ = 0.1 and random horizontal flipping as the only data augmentation.These settings are part of the supervised pre-training configuration described for the segmentation model.
  • Episodic Training: After pre-training, the encoder and decoder are frozen while episodic training forms support-query episodes from randomly selected base classes.A new classifier is trained for the selected class before subsequent episodic adaptation.

4.3. Single Domain Evaluation

On COCO-20i and PASCAL-5i, the method is evaluated across shot settings and backbones, achieving strong overall performance while exposing a dataset-dependent limitation.

  • COCO-20i Results: 2.3% mIoU gain over the best competitor is achieved in the 1-shot COCO-20i case with a ResNet-50 backbone.The gains increase to 5.1% and 4.6% in the 5-shot case with ResNet-50 and ResNet-101, respectively.
  • COCO-20i Results: 5.1%/4.6% gains are reported in the 5-shot COCO-20i case with ResNet-50/ResNet-101, respectively.The method also benefits consistently from expanding the support set.
  • COCO-20i Results: Some existing methods are clearly inferior at leveraging extra labeled samples, whereas the proposed method benefits consistently from support-set expansion.This contrast is reported for PPNet, RPMMs, FWB, and PFENet in the evaluated comparisons.
  • PASCAL-5i Results: The method achieves the best overall performance on PASCAL-5i, although its advantage over competitors is less pronounced than on COCO-20i.It performs worse in the 1-shot case, suggesting the pre-trained encoder/decoder assumption is weaker with fewer training classes and images.

4.4. Cross Domain Evaluation

The cross-domain evaluation tests direct transfer between COCO-20i and PASCAL-5i without domain-specific retraining or fine-tuning. The proposed method outperforms RPMMs on almost all splits, with larger gains from COCO to PASCAL.

  • Cross-Domain Setting: The cross-domain setting targets generalization across unseen domains with different data distributions, in addition to new classes.This setting is described as more difficult yet more practical for real-world segmentation tasks.
  • Cross-Domain Setting: Direct transfer between COCO-20i and PASCAL-5i is evaluated without domain-specific model retraining or fine-tuning.The two datasets provide domain shifts in instance size, instance number, and categories per image.
  • Cross-Domain Results: 9.9% and 12.7% average gains are obtained over RPMMs in the COCO-to-PASCAL setting for 1-shot and 5-shot cases, respectively.The comparison uses a ResNet-50 backbone.
  • Cross-Domain Results: 2.8%/4.8% improvements are reported from PASCAL to COCO for 1-shot/5-shot cases, respectively.The method is significantly superior on almost all evaluated splits.

4.5. Ablation Study

The ablations examine whether adapting only the classifier is preferable to updating the full segmentation model, and assess query-image adaptation for intra-class variation. Results support classifier-only meta-learning and show that the Classifier Weight Transformer corrects baseline failures, while extreme appearance and viewpoint changes remain challenging.

  • What should be meta-learned, the whole model or the classifier only?: 11.4% lower mIoU results from meta-learning the whole model compared with meta-learning only the classifier.The comparison updates the encoder, decoder, and classifier in episodic training versus adapting classifier weights initialized from support-set prototypes.
  • Component analysis: The method pre-trains the encoder and decoder, then adapts only a support-set-trained classifier for each meta-test task.The baseline omits meta-learning and directly trains a classifier on the support set while freezing the feature representation components.
  • Component analysis: Figure 4 compares support images with masks, query images with masks, baseline results, and the proposed results from left to right.The examples are evaluated under 1-shot segmentation on COCO-20i.
  • Component analysis: CWT adapts support-initialized classifier weights to each query image to address intra-class variation.The query-dependent adaptation improves over a transformer variant that does not use the query image, according to the ablation discussion.
  • Component analysis: The baseline fails to detect an airplane and a person, whereas query-image adaptation with CWT rectifies these failures.These examples are shown in the first and second rows of Figure 4.
  • Failure Cases: The model fails when support and query instances exhibit extreme appearance changes or viewpoint differences.Examples include hands in the support image versus a whole person in the query image; the paper identifies view, pose, and occlusion changes as directions requiring better modeling.

5. Conclusion

The paper presents a few-shot segmentation method that meta-learns only the classifier while freezing a pre-trained encoder and decoder, with CWT adapting classifier weights to each query image. Experiments report superiority over existing state-of-the-art methods on two standard benchmarks and advantages in cross-domain few-shot segmentation.

  • 5. Conclusion: The method meta-learns only the classifier while freezing the pre-trained encoder and decoder.This design reduces adaptation to the classifier component of the segmentation model.
  • 5. Conclusion: CWT adapts classifier weights initialized on a support set to every query image to address intra-class variation.The transformer provides query-specific classifier adaptation.
  • 5. Conclusion: Experiments report performance superiority over existing state-of-the-art few-shot segmentation methods on two standard benchmarks.The conclusion also reports advantages in the more challenging cross-domain few-shot segmentation setting.
Loading 2108.03032v3…