PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Few-shot Semantic Segmentation

Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, Jiaya Jia

arXiv:2010.05210v4cs.CV

TL;DR

Few-shot segmentation adapts models to novel classes with limited labels, but its constraints leave open whether models can also predict base classes without query-matched supports. The paper introduces GFS-Seg and CAPL; experiments report CAPL gains in GFS-Seg and FS-Seg. CAPL’s comparison with HSNet differs between two FS-Seg benchmarks.

  • Problem

    Classic FS-Seg evaluates novel classes with query-matched supports, leaving the ability to predict base and novel classes together without those supports as an open challenge.

    Source: §1 · p. 2

  • Method

    The paper defines GFS-Seg and proposes CAPL, which uses support-sample co-occurrence information and dynamically adapts contextual information to each query image.

    Source: §1 · p. 2

  • Results

    CAPL significantly improves the GFS-Seg baseline and also improves FS-Seg performance, reaching state-of-the-art performance in that setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CAPL applies to normal semantic segmentation models without specific structural constraints and generalizes to FS-Seg.

    Source: §1 · p. 2, §6 · p. 8

    CAPL does not add dense spatial reasoning between query and support features, and it does not outperform HSNet on Pascal-5i in FS-Seg.

    Source: §6 · p. 8

Abstract

from arXiv · show

Training semantic segmentation models requires a large amount of finely annotated data, making it hard to quickly adapt to novel classes not satisfying this condition. Few-Shot Segmentation (FS-Seg) tackles this problem with many constraints. In this paper, we introduce a new benchmark, called Generalized Few-Shot Semantic Segmentation (GFS-Seg), to analyze the generalization ability of simultaneously segmenting the novel categories with very few examples and the base categories with sufficient examples. It is the first study showing that previous representative state-of-the-art FS-Seg methods fall short in GFS-Seg and the performance discrepancy mainly comes from the constrained setting of FS-Seg. To make GFS-Seg tractable, we set up a GFS-Seg baseline that achieves decent performance without structural change on the original model. Then, since context is essential for semantic segmentation, we propose the Context-Aware Prototype Learning (CAPL) that significantly improves performance by 1) leveraging the co-occurrence prior knowledge from support samples, and 2) dynamically enriching contextual information to the classifier, conditioned on the content of each query image. Both two contributions are experimentally shown to have substantial practical merit. Extensive experiments on Pascal-VOC and COCO manifest the effectiveness of CAPL, and CAPL generalizes well to FS-Seg by achieving competitive performance. Code is available at https://github.com/dvlab-research/GFS-Seg.

1. Introduction

Semantic segmentation models can struggle to adapt to unseen classes without sufficient fully labeled data, while fine-tuning costs time and resources. The paper introduces GFS-Seg and CAPL to evaluate and improve joint segmentation of base and novel classes.

1. Introduction

  • Without sufficient fully labeled data, trained segmentation frameworks cannot handle unseen classes, and fine-tuning on novel-class data costs additional time and resources.The paper motivates methods for adapting to novel classes with limited labeled data.Source: §1 · p. 1
  • FS-Seg requires supports matched to query classes and evaluates only novel classes, constraints that limit its fit to practical segmentation.The paper reports that exemplar FS-Seg models struggle to evaluate both base and novel classes together.Source: §1 · p. 1
  • GFS-Seg evaluates whether models can segment base and novel classes together without knowing which classes appear in each query image.Unlike FS-Seg, it does not require forwarding support samples matched to the query’s target classes during evaluation.Source: §1 · p. 2
  • The paper proposes CAPL, which uses support-sample co-occurrence information and adapts contextual information to query images, improving the GFS-Seg baseline.CAPL is presented as applicable to normal segmentation models without specific structural constraints and as improving the baseline in both GFS-Seg and FS-Seg.Source: §1 · p. 2
  • The paper analyzes the performance gap between FS-Seg and GFS-Seg, reporting that recent popular FS-Seg models cannot handle the generalized setting well.The paper identifies constraints in the classic FS-Seg setting as the main source of the discrepancy.Source: §1 · pp. 1–2

2. Related Work

Related work covers semantic segmentation architectures, few-shot learning strategies, and few-shot segmentation methods. It also distinguishes dense pixel labeling from image-level recognition by its contextual information.

Semantic segmentation

  • Semantic segmentation assigns a label to each pixel; prior approaches use encoder-decoder refinement, dilated convolution, context modeling, and attention.The cited examples include global pooling and pyramid pooling for context, and attention models for long-range scene relationships.Source: §2 · p. 2

Few-shot learning

  • Few-shot learning methods include meta-learning, metric learning, and data augmentation, while generalized few-shot learning allows queries from base or novel categories.Prior work combines supervised base-class models with prototype-based novel-class methods, but image-level classification lacks segmentation’s contextual information for each target.Source: §2 · p. 2

Few-shot segmentation

  • Few-shot segmentation predicts dense labels for new classes from a few support samples, using methods based on generated classifier weights, prototypes, alignment, or convolutions.PFENet uses pretrained-backbone prior knowledge to locate regions of interest and a Feature Enrichment Module to alleviate spatial inconsistency.Source: §2 · p. 2
  • Despite their performance on novel classes with corresponding supports, FS-Seg models struggle when query images contain both base and novel classes and target-class supports are unavailable.The paper points to this practical setting as a limitation of state-of-the-art FS-Seg models.Source: §2 · pp. 2–3

3. Towards Generalized Few-shot Semantic Segmentation

GFS-Seg extends few-shot segmentation to predict base and novel classes together without query-matched support samples at evaluation. The section defines its phases and explains why classic FS-Seg does not meet this setting.

Revisit the classic setting

  • Classic FS-Seg trains on base classes and evaluates novel classes using support samples that share the query’s target class.In a K-shot task, the support set contains K labeled samples of that class.Source: §3 · p. 3

The proposed generalized setting

  • Because users may not know which classes occur in a test image, selecting supports that match the query makes classic FS-Seg impractical in many cases.The paper also notes that FS-Seg may require processing additional manually selected supports to predict across possible classes.Source: §3 · p. 3
  • In GFS-Seg, models learn from base classes, register novel classes from limited support samples, then predict both class groups from query images alone.Queries may contain base classes, novel classes, or both; the major metric is total mIoU averaged over all classes.Source: §3 · p. 3
  • FS-Seg predicts masks only for classes in its support set, so base classes absent from that set can be missed even after base-class training.The paper’s Figure 2 example also notes that redundant novel classes in the supports may adversely affect performance.Source: §3 · p. 3
  • GFS-Seg evaluates all base and novel classes, whereas FS-Seg models can degrade because they learn to predict only the provided novel-class masks.GFS-Seg performs inference without prior knowledge of the classes in the query image.Source: §3 · p. 3

4. Our Method

The method represents base classes with a learned segmentation classifier and forms novel-class prototypes from few labeled support samples. CAPL enriches base-class weights with support- and query-derived context, while retaining the novel prototypes for joint base-and-novel prediction.

4.1. Prototype Learning

  • Novel prototypes are formed by mask-pooling support features and averaging across K shots, then concatenated with learned base prototypes for joint prediction.Predictions are assigned according to the most similar prototype.Source: §4.1 · pp. 3–4
  • The baseline treats the learned base classifier as base prototypes, avoiding the impractical step of forwarding all base-class training samples to form them.It learns the classifier by backpropagation on base classes.Source: §4.1 · p. 4

4.2. Context-Aware Prototype Learning (CAPL)

  • CAPL targets GFS-Seg’s richer class co-occurrence: a novel class may appear with base classes in support samples, so preserving contextual relationships can matter for predictions.The paper illustrates this with a novel Sofa class appearing alongside the base Dog class.Source: §4.2 · p. 4
  • Support Contextual Enrichment (SCE) combines original base-class weights with support-derived base prototypes, using an adaptive weight conditioned on old and new prototypes.SCE enriches base classes present in the novel-class support samples.Source: §4.2 · p. 5
  • Dynamic Query Contextual Enrichment (DQCE) derives contextual representatives from each query’s temporary predictions and uses them to enrich all base-class weights dynamically.Because query labels are unavailable, the method estimates the reliability of query-derived prototypes; unlike SCE, it considers all base classes.Source: §4.2 · p. 5
  • CAPL training splits batches into fake support and query samples, designates some base classes as fake novel and fake context classes, and trains with the context-aware classifier.These base-class roles mimic novel and base classes in support samples at test time.Source: §4.2 · p. 6

5. Experiments

In GFS-Seg, representative FS-Seg models fall short when asked to identify base and novel classes together, and the paper examines why and how CAPL improves performance. Ablations assess CAPL’s contextual-enrichment and training strategies, with further experiments applying it to FS-Seg and visualizing predictions.

5.1. Comparison with FS-Seg Models in GFS-Seg

  • Under joint base- and novel-class evaluation, CANet, SCL, PFENet, and PANet perform poorly compared with models implemented with CAPL.Table 1 reports mIoU for base, novel, and total classes.Source: §5.1 · p. 6, Table 1 · p. 7
  • FS-Seg models fall short because episodic training emphasizes foreground-versus-background discrimination, while GFS-Seg requires simultaneous multiclass prediction without knowing query classes in advance.Fixing the backbones in CANet, SCL, and PFENet also limits adaptation to GFS-Seg’s multiclass setting.Source: §5.1 · p. 7

5.2. Ablation Study

  • SCE with MLP weighting and DQCE with cosine weighting are the most conducive tested choices for enriching the baseline.SCE mines support co-occurrence cues, while DQCE adapts enrichment to each query image; cosine similarity estimates the reliability of query-derived prototypes.Source: §5.2 · p. 7, Table 2 · p. 7
  • SCE (MLP) outperforms SCE-Sw (Cos) by producing moderate, data-conditioned weights that balance novel context with the original classifier information.The reported γsup values from the MLP fall within 0.4–0.7 rather than collapsing to 0 or 1.Source: §5.2 · p. 7
  • CAPL’s training and contextual-enrichment strategies complement each other: training alignment alone brings minor improvement, while enrichment without that training strategy is suboptimal.The latter is attributed to misalignment between the original classifier and the features.Source: Table 3 · p. 7, §5.2 · p. 8

5.3. Apply CAPL to FS-Seg

  • Applied to PANet and PFENet, CAPL significantly improves their FS-Seg baselines while making no significant structural changes.The reported improved models have nearly the same inference speed as the originals.Source: §5.3 · p. 8

5.4. Visual Examples

  • Figure 4 compares baseline predictions with SCE and SCE+DQCE for both GFS-Seg and FS-Seg; the text says the enrichment stages refine baseline predictions.The figure labels the rows by setting and the panels as input, ground truth, baseline, SCE, and SCE+DQCE.Source: §5.4 · p. 8, Figure 4 · p. 8

6. Concluding Remarks

GFS-Seg evaluates base and novel classes together, and CAPL improves performance by dynamically enriching context with adapted features. CAPL generalizes to FS-Seg, but does not outperform HSNet on Pascal-5i.

6. Concluding Remarks

  • CAPL significantly improves performance by dynamically enriching context information with adapted features.It has no structural constraints on the base model and generalizes well to FS-Seg.Source: §6 · p. 8
  • GFS-Seg evaluates both base and novel classes, a setting that classic FS-Seg models fall short in.The benchmark is presented with CAPL as a solution.Source: §6 · p. 8
  • CAPL does not outperform HSNet on Pascal-5i in FS-Seg, where it introduces no new dense spatial reasoning between query and support features.On COCO-20i, CAPL achieves new SOTA performance, where semantic cues are better exploited.Source: §6 · p. 8

Related papers

Loading 2010.05210v4…