PaperlayerView exact revision on arXiv

Source-linked AI summary

Prior Guided Feature Enrichment Network for Few-Shot Segmentation

Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, Jiaya Jia

arXiv:2008.01449v1cs.CV

TL;DR

Few-shot segmentation must adapt to unseen classes with few labeled support samples, while existing methods face generalization loss and spatial inconsistency. PFENet addresses these issues with training-free prior masks and FEM, achieving state-of-the-art results on PASCAL-5i and COCO without efficiency loss, and also demonstrating performance in a zero-shot setting.

  • Problem

    Few-shot segmentation seeks to adapt to unseen classes from few labeled support samples, but existing methods suffer generalization loss from high-level features and spatial inconsistency between query and support samples.

  • Method

    PFENet combines training-free prior mask generation from high-level features with FEM, which adaptively enriches query features using support features and conditioned inter-scale interaction.

  • Results

    PFENet achieves new state-of-the-art results on PASCAL-5i and COCO without compromising efficiency, using 10.8 M parameters and 15.9 and 5.1 FPS in 1-shot and 5-shot settings.

  • Takeaways & Limitations

    PFENet’s prior generation and FEM both improve the baseline, while the model also achieves decent performance in a zero-shot setting without major structural modification.

Abstract

from arXiv · show

State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results and hardly work on unseen classes without fine-tuning. Few-shot segmentation is thus proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information of training classes and spatial inconsistency between query and support targets. To alleviate these issues, we propose the Prior Guided Feature Enrichment Network (PFENet). It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks. Extensive experiments on PASCAL-5$^i$ and COCO prove that the proposed prior generation method and FEM both improve the baseline method significantly. Our PFENet also outperforms state-of-the-art methods by a large margin without efficiency loss. It is surprising that our model even generalizes to cases without labeled support samples. Our code is available at https://github.com/Jia-Research-Lab/PFENet/.

1 INTRODUCTION

Few-shot segmentation addresses the difficulty of segmenting unseen classes with limited labeled data, but existing methods suffer from generalization loss and spatial inconsistency. PFENet introduces training-free priors and FEM, achieving state-of-the-art results without efficiency loss.

  • Semantic segmentation performance worsens with insufficient fully labeled data or unseen classes, while fine-tuning remains time- and resource-consuming.
  • Few-shot segmentation adapts to unseen query classes using a few labeled support samples, but existing methods misuse high-level features and mismatch query-support spatial structure.
  • The paper presents prior generation, FEM, and PFENet as its threefold contribution.
  • Training-free priors from high-level features improve prediction accuracy while retaining generalization to unseen classes.
  • FEM adaptively enriches query features with support features and conditioned inter-scale information to address spatial inconsistency.
  • PFENet achieves new state-of-the-art results on PASCAL-5i and COCO without compromising efficiency.

2 RELATED WORK

Related work spans semantic segmentation, few-shot learning, and few-shot segmentation. PFENet differs by targeting generalization loss and spatial inconsistency with a comparatively lightweight multi-scale design.

  • 2.1 Semantic Segmentation: Semantic segmentation methods use convolutional, contextual, attention, and encoder-decoder structures to predict labels for each pixel.
  • 2.1 Semantic Segmentation: Context and attention methods work well on large-sample classes but are not designed for rare or unseen classes and require fine-tuning for adaptation.
  • 2.2 Few-shot Learning: Few-shot learning includes meta-learning and metric-learning methods, with some approaches synthesizing training samples to address limited data.
  • 2.3 Few-shot Segmentation: Few-shot segmentation performs dense pixel labeling on new classes from few support samples, using approaches such as classifier-weight generation, prototypes, and cosine similarity.
  • 2.3 Few-shot Segmentation: PFENet replaces cosine similarity with convolution and combines basic convolutions, prior masks, and FEM in a multi-scale structure.

3 OUR METHOD

The method section introduces the prior generation method, FEM, and the PFENet architecture. These components are presented as the network’s core design.

  • 3 OUR METHOD: The method section presents the training-free prior generation method and Feature Enrichment Module.
  • 3.2 Prior Generation: Figure 2 illustrates prior generation using masked support images, query images, and highlighted regions in the resulting query prior masks.
  • 3 OUR METHOD: It then discusses the Prior Guided Feature Enrichment Network, which combines the proposed prior generation method and FEM.

3.1 Task Description

Few-shot segmentation uses support and query sets to segment an unseen class in query images from a small number of labeled support samples.

  • A few-shot segmentation system contains a query set Q and support set S, with K support samples provided for segmenting an unseen class in each query image.
  • Models train on disjoint base classes Ctrain and test on previously unseen novel classes Ctest in episodic tasks.
  • Each episode pairs same-class support samples containing images and labels with a query image whose ground-truth mask evaluates the prediction.

3.2 Prior for Few-Shot Segmentation

The prior converts fixed high-level features into a class-insensitive mask, using query-support pixel correspondences to guide target localization without training-class bias.

  • High-level features can reduce few-shot segmentation performance by favoring training classes over unseen test classes.This motivates using high-level semantic information in a training-class-insensitive way.
  • The prior mask transforms ImageNet-pretrained high-level query and support features into pixel-level target-class correspondence scores.The support mask removes background support features before correspondence computation.
  • The training-free generation process uses fixed backbone features and therefore upholds class-insensitivity during evaluation on unseen classes.The method is reported as simple and effective in ablation comparisons with alternative prior-generation methods.
  • A high prior value indicates strong correspondence with at least one foreground support pixel, making the query pixel likely to belong to the target area.Background removal ensures correspondence is computed against foreground support features rather than support background.
  • For each query pixel, the method takes the maximum cosine similarity over all support pixels, reshapes the scores, and min-max normalizes them to [0, 1].The similarity matrix has size hw × hw, and ε is set to 1e-7.

3.3 Feature Enrichment Module

FEM addresses spatial inconsistency by enriching query features with support features and prior masks across multiple scales, then selectively transferring information along a top-down path.

  • Global pooling of support images creates spatial information inconsistency because query targets and support objects can differ substantially in scale.PPM and ASPP add multi-level information but lack per-scale refinement and ignore hierarchical relations across scales.
  • FEM horizontally interacts query features with support features and prior masks at each scale, then vertically enriches coarse maps using finer-scale information.The resulting multi-scale features are collected to form the refined query feature.
  • The top-down path passes useful information from finer auxiliary features to coarser main features, helping preserve information for tiny objects absent from down-sampled maps.Alternative bottom-up and bidirectional paths are also considered, while the top-down path shows superiority in Section 4.3.1.
  • The inter-scale merging module resizes auxiliary features, extracts conditioned information with a 1×1 convolution, applies two 3×3 convolutions, and preserves main-feature integrity through a residual link.When no auxiliary feature exists, refinement uses only the main feature.
  • FEM improves over alternative feature-enhancement designs because multi-resolution query-support fusion and selective inter-scale transfer provide conditioned information for varied target scales.The paper reports greater performance gains than PPM, ASPP, and GAU; removing the vertical top-down path worsens results.

3.4 Prior Guided Feature Enrichment Network

PFENet combines shared support-query feature extraction with FEM, which enriches query features using support features and prior masks across scales. Training uses cross-entropy with intermediate and final prediction losses.

  • Network Architecture: FEM enriches query features with support features and prior masks after shared backbone extraction and channel reduction.The module operates on reduced 256-channel middle-level features.
  • Network Architecture: The backbone remains shared for support and query images while FEM performs the feature enrichment before subsequent convolutional processing.This preserves a common feature space for support-query matching.
  • Network Architecture: In 5-shot evaluation, pooled support features and prior masks are averaged across the five support samples before FEM concatenation.The query feature is paired with an averaged support representation and averaged prior mask.
  • Loss Function: The model uses cross-entropy loss with intermediate supervision for FEM outputs and a second loss for the final prediction.The total loss is a weighted sum of these losses, with intermediate terms indexed over FEM spatial sizes.

4 EXPERIMENTS

Experiments evaluate PFENet on PASCAL-5^i and COCO using cross-validation, class-level IoU metrics, and multiple feature-enrichment comparisons. PFENet reaches state-of-the-art performance, while ablations support top-down, conditioned, and multi-scale design choices.

  • Datasets: COCO evaluation samples 20,000 query-support pairs per fold because its 40,137-image validation set makes 1,000 pairs unreliable.PASCAL-5^i uses 1,000 query-support pairs per test.
  • Evaluation Metrics: The evaluation uses class mIoU as the major ablation metric and also reports FB-IoU averaged across folds.Class mIoU averages class IoU values, while FB-IoU considers foreground and background only.
  • Results: PFENet reaches new state-of-the-art results on PASCAL-5^i and COCO, with more than 10 class mIoU points over other methods on COCO.The best performance uses the fewest learnable parameters among compared models: 10.4M for VGG and 10.8M for ResNet.
  • Inter-Scale Interaction Strategies: Top-down inter-scale paths outperform bottom-up alternatives, while TD+BU performs comparably to TD despite using 16.0M rather than 10.8M parameters.The experiments compare TD, BU, TD+BU, and BU+TD.
  • Multi-Scale Structures: FEM with spatial sizes {60, 30, 15, 8} outperforms PPM and ASPP, and the baseline with GAU obtains class mIoU 55.4 and 56.1 in 1- and 5-shot evaluation.GAU+ with the FEM scales obtains lower values: 54.9 in 1-shot and 55.4 in 5-shot.
  • Feature Fusion Strategies: Conditioned feature selection improves HRB-TD, and HRB-TD-Cond approaches FEM performance with 7.5M learnable parameters.The results identify conditioned inter-scale selection as important for multi-resolution structures.

4.4 Ablation Study of the Prior Generation

The prior ablation shows that training-free fixed high-level features and maximum correspondence produce more effective, generalizable prior masks than learnable or middle-level alternatives. Prior masks improve the baseline and retain utility on totally unseen objects.

  • Motivation: The prior improves models both with and without FEM, while trainable mask generation can bias priors toward base classes.Existing trainable approaches can also lose discriminative support information through masked average pooling.
  • Feature Selection: Fixed high-level features are selected because they provide semantic information without sacrificing generalization, and their training-free generation avoids training-class bias.The resulting prior supplies bias-free information for seen and unseen data during evaluation.
  • Quantitative Analysis: Learnable or fixed middle-level priors improve less than PriorFH, while learnable high-level priors perform significantly worse because they overfit base classes.Middle-level features are less effective at revealing semantic correspondence between query and support features.
  • Qualitative Analysis: Fixed high-level priors preserve target-region integrity on unseen classes, whereas learnable high-level priors fail to reveal potential regions clearly and middle-level priors favor background.These qualitative comparisons are shown using normalized prior values interpreted as target-region probabilities.
  • Prior Generation: The maximum correspondence strategy outperforms averaging similarities and mask-pooled support features for prior generation.Maximum correspondence indicates whether at least one support pixel has a close semantic relation to each query pixel.
  • Generalization on Totally Unseen Objects: The fixed high-level prior remains effective for objects unseen by ImageNet, improving the Person baseline from 14.38 to 15.81 IoU.On the broader evaluation, foreground IoU reaches 80.8 and 81.4 in 1- and 5-shot settings versus 79.7 and 80.1 without the prior.

4.5 Backbone Training

Training the backbone causes performance reductions attributed to overfitting and nearly doubles training time per batch. FEM and prior masks remain beneficial, with the combined model still outperforming the baseline.

  • Backbone Training: Training additional backbone parameters causes significant performance reduction due to overfitting on training classes.This contrasts with the fixed-backbone setup used in recent few-shot segmentation work.
  • Backbone Training: Backbone training nearly doubles each batch's training time but does not affect inference speed.The added cost comes from requiring an additional parameter update.
  • Ablation Results: With trainable backbones, FEM and the prior provide less significant gains than with fixed backbones, yet their combination still outperforms the baseline.In this setting, priors use learnable high-level features because the entire backbone is trainable.
  • Ablation Results: In 5-shot evaluation, the prior yields greater gains than FEM because averaging five support samples provides a more accurate prior mask.The averaged prior helps combat overfitting.

4.6 Model Efficiency

PFENet remains efficient while achieving stable evaluation results and extending to zero-shot segmentation without labeled support samples. Its best model has 10.8M trainable parameters and runs at 15.9 FPS in 1-shot and 5.1 FPS in 5-shot settings.

  • Model efficiency: 10.8M trainable parameters are used by the best Baseline + FEM + Prior model.The prior generation method adds no learnable parameters.
  • Inference speed: 15.9 FPS and 5.1 FPS are achieved by ResNet-50 PFENet in 1-shot and 5-shot settings, respectively.Evaluation uses 473 × 473 test images on an NVIDIA Titan V GPU.
  • Result stability: Standard deviations below 0.5 show stable PASCAL-5i results with 1,000 evaluation pairs in both 1-shot and 5-shot settings.The analysis averages five test results with different random seeds for comparison.
  • Result stability: 20,000 randomly sampled COCO query-support pairs produce more stable results than 1,000 pairs.The larger sample is used because COCO validation contains 40,137 images across 80 classes.
  • Zero-shot extension: 53.2 class mIoU is achieved on unseen classes without support samples, increasing to 54.2 with FEM.The zero-shot experiment replaces pooled support features with class-label embeddings; the prior is not applied.

5 CONCLUSION

PFENet combines a prior generation method with a Feature Enrichment Module to improve query-target localization and address spatial inconsistency. The prior uses pre-trained high-level features, while FEM adaptively merges query and support features.

  • Conclusion: PFENet consists of a prior generation method and a Feature Enrichment Module.These are the two proposed components highlighted in the conclusion.
  • Conclusion: The prior generation method uses cosine similarity on pre-trained high-level features to improve query-target localization while retaining generalization power.The prior mask encourages better localization without losing generalization power.
  • Conclusion: FEM addresses spatial inconsistency by adaptively merging query and support features at multiple levels.The supplied conclusion passage truncates the remainder of the multilevel description.
Loading 2008.01449v1…