PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Non-target Knowledge for Few-shot Semantic Segmentation

Yuanwei Liu, Nian Liu, Qinglong Cao, Xiwen Yao, Junwei Han, Ling Shao

arXiv:2205.04903v1cs.CV

TL;DR

Few-shot segmentation methods struggle with ambiguous background and distracting-object regions because they mainly mine target information. NTRE mines and eliminates those non-target regions through background and distractor modules, and experiments on PASCAL-5i and COCO-20i show performance superiority, especially in 1-shot settings. The authors note that its averaged mIoU advantage is not obvious in 5-shot settings.

  • Problem

    Existing few-shot segmentation methods focus on target information but struggle to distinguish ambiguous background and distracting-object regions.

  • Method

    NTRE learns a general background prototype, eliminates background and distracting-object information from query features, and uses prototypical contrastive learning to distinguish targets from distractors.

  • Results

    The framework achieves performance superiority over previous methods on PASCAL-5i and COCO-20i, especially in the 1-shot setting.

  • Takeaways & Limitations

    Explicitly mining and eliminating non-target regions provides a complementary perspective to traditional target-focused few-shot segmentation.

  • Takeaways & Limitations

    Averaged mIoU results do not show obvious advantages in the 5-shot setting because the method focuses on eliminating non-target regions rather than exploiting additional support samples.

Abstract

from arXiv · show

Existing studies in few-shot semantic segmentation only focus on mining the target object information, however, often are hard to tell ambiguous regions, especially in non-target regions, which include background (BG) and Distracting Objects (DOs). To alleviate this problem, we propose a novel framework, namely Non-Target Region Eliminating (NTRE) network, to explicitly mine and eliminate BG and DO regions in the query. First, a BG Mining Module (BGMM) is proposed to extract the BG region via learning a general BG prototype. To this end, we design a BG loss to supervise the learning of BGMM only using the known target object segmentation ground truth. Then, a BG Eliminating Module and a DO Eliminating Module are proposed to successively filter out the BG and DO information from the query feature, based on which we can obtain a BG and DO-free target object segmentation result. Furthermore, we propose a prototypical contrastive learning algorithm to improve the model ability of distinguishing the target object from DOs. Extensive experiments on both PASCAL-5i and COCO-20i datasets show that our approach is effective despite its simplicity.

1. Introduction

Few-shot semantic segmentation reduces annotation demands but existing methods often confuse ambiguous background and distracting-object regions because they focus primarily on target information. NTRE instead mines and eliminates these non-target regions, using dedicated modules and prototypical contrastive learning, and reports new state-of-the-art performance, especially in 1-shot settings.

  • Few-shot semantic segmentation targets unseen classes with only a few annotated support samples, reducing the annotation burden of fully supervised methods.
  • Existing methods primarily mine target information from supports, but often produce false positives in backgrounds and distracting objects.
  • NTRE mines and excludes background and distracting-object regions rather than directly segmenting the target object alone.
  • BGEM and DOEM filter background and distracting-object information, while prototypical contrastive learning improves target-versus-distractor discrimination.
  • BGMM learns a general background prototype and uses a background loss based only on known target segmentation masks, without background ground truth.
  • The framework achieves new state-of-the-art performance on PASCAL-5i and COCO-20i, especially under the 1-shot setting.

2. Related Works

Related work in few-shot semantic segmentation centers on pixel-level matching or embedding-distance measurement, while contrastive learning improves feature representations through positive and negative comparisons.

  • Fully convolutional networks enable pixel-level prediction and underpin many modern semantic segmentation architectures.
  • Few-shot semantic segmentation methods commonly generate target prototypes from supports for dense query matching or measure query-support embedding similarity.
  • Pixel-level matching methods refine query predictions using support prototypes, similarity-based prior masks, multi-part prototypes, or feature reconstruction.
  • Pixel-level measurement methods embed object classes into prototypes and label query pixels according to distances between query and support representations.
  • Contrastive learning focuses on improving deep feature representations by contrasting positive and negative samples.

3.1. Problem Definition

Few-shot semantic segmentation trains on episodes containing base classes and evaluates on disjoint novel classes, using annotated supports to segment a query image.

  • The model trains on base classes and segments unseen novel objects without retraining, using a few annotated support samples.
  • The base and novel class sets are disjoint, with C_base ∩ C_novel = ∅.
  • Each training episode samples K + 1 image-mask pairs from one base class, assigning K to support and one to the query.
  • The query ground-truth mask is available during training, while evaluation uses analogous episodes sampled from novel classes.

3.2. Overview

NTRE addresses ambiguous non-target regions by mining and removing background and distracting objects from query features before producing the final segmentation.

  • NTRE targets background and distracting-object regions because few-shot models often fail in ambiguous non-target areas.
  • A pretrained backbone extracts query and support feature maps for subsequent non-target-region processing.
  • BGMM learns a general background prototype and a background loss without accurate background segmentation ground truth.
  • BGEM filters background information from the query feature before target-prototype matching.
  • DOEM mines distracting objects from the background map and initial target prediction, then uses a query-derived distractor prototype for elimination.
  • PCL improves target-versus-distractor discrimination, followed by segmentation that produces a background- and distracting-object-free prediction.

3.3. Background Mining and Eliminating

The Background Mining Module learns a general background prototype from support and query images, predicts background regions using target masks as supervision, and filters background information from query features.

  • Background Mining Module: A general BG prototype encodes background knowledge common across images and is learned from support and query images during training.The prototype is randomly initialized and expanded before being combined with feature maps.
  • Background Mining Module: BGMM concatenates the expanded BG prototype with support and query features and applies a shared two-layer convolutional network to produce BG probability maps.The output map has spatial dimensions H×W×1.
  • Background Mining Module: The BG mining loss uses known target masks to penalize BG predictions on target pixels and adds regularization to produce valid BG regions.The target-pixel constraint alone could yield an all-zero BG prediction, so a second term is included.
  • Background Eliminating Module: BGEM concatenates the expanded BG prototype with the query feature and applies a 1×1 convolution to obtain a BG-filtered query feature.The resulting feature is denoted X_BG^q.

3.4. Support Feature Matching

Support feature matching uses masked average pooling to form a support target prototype, compares it densely with the BG-filtered query, and produces an activated query feature and initial target prediction.

  • Support Feature Matching: Dense feature matching activates the target object region in the query feature using the support prototype and a pixel-level prior confidence map.The confidence map is computed from maximum pixel-level similarity scores.
  • Support Feature Matching: Masked average pooling extracts the support target prototype from the support feature map using the support target region.The prototype is expanded spatially and concatenated with the BG-filtered query feature.
  • Support Feature Matching: The matching stage produces an activated query feature and an initial target-object prediction through convolutional layers.The initial prediction is generated after feature activation.

3.5. Distracting Objects Eliminating

DOEM identifies distracting-object regions as complementary to background and initial target regions, then removes their information before final segmentation; PCL refines prototype separability.

  • Distracting Objects Eliminating Module: After background removal, the query may still contain distracting objects, so DOEM filters DO information from the activated query feature.The DO region is mined using the known background region and the initial target prediction.
  • Distracting Objects Eliminating Module: The DO mask is defined as the complement of the union of the background and initial target binary maps.This identifies regions outside both known background and predicted target areas.
  • Distracting Objects Eliminating Module: Masked average pooling over the DO mask produces a query DO prototype, whose element-wise feature aggregation covers spatial locations.The prototype is computed by weighted summation and normalization over the mask.
  • Distracting Objects Eliminating Module: The DO prototype is expanded and combined with the activated query feature before a segmentation network generates the final target prediction.The final output y^q is the whole model’s target-object segmentation result.
  • Prototypical Contrastive Learning: PCL treats the corresponding support target prototype as positive and query and support DO prototypes as negatives to make target and DO embeddings more discriminative.The method also encourages greater similarity between query and support target prototypes and uses a DO prototype bank for negative samples.

3.6. Total Training Loss

Training combines target segmentation supervision for initial and final predictions with the BG mining loss and prototypical contrastive learning loss.

  • Total Training Loss: The target segmentation loss sums binary cross-entropy for the initial and final target predictions against the target mask.Both predictions are supervised with the same target mask.
  • Total Training Loss: The total loss is a weighted combination of target segmentation, BG mining, and prototypical contrastive learning losses.The weights β, λ, and γ are adjustable.

4. Experiments

Experiments on PASCAL-5i and COCO-20i show that NTRENet improves few-shot segmentation by eliminating background and distracting-object regions, with strongest gains in 1-shot settings and progressively effective modules.

  • Datasets and Evaluation Metrics: The study evaluates class mIoU and FB-IoU on four-fold PASCAL-5i and COCO-20i benchmarks, using fixed pretrained VGG-16, ResNet-50, and ResNet-101 backbones.PASCAL-5i contains 20 categories across four folds, while COCO-20i contains 80 categories across four folds.
  • Comparison with State-of-the-art Methods: NTRENet outperforms prior methods across PASCAL-5i backbones in 1-shot segmentation, with gains in both mIoU and FB-IoU.Averaged mIoU gains are 1.2%, 3.4%, and 1.8% for VGG-16, ResNet-50, and ResNet-101; FB-IoU gains are 1.1%, 5.0%, and 3.3%.
  • Comparison with State-of-the-art Methods: On COCO-20i, NTRENet achieves 1-shot mIoU scores of 39.3 and 39.1 with ResNet-50 and ResNet-101, surpassing previous best results by 4.8% and 5.7%.The 5-shot FB-IoU results also support superiority in challenging realistic scenes.
  • Limitations: NTRENet’s averaged mIoU advantage is less pronounced in 5-shot settings because the method focuses on non-target elimination rather than additional target information.The authors identify this as a limitation of the current approach.
  • Qualitative Comparison: Qualitative results show NTRENet produces more accurate masks with fewer false positives in background and distracting-object regions than PFENet.The visualizations compare support images, query images, PFENet predictions, and NTRENet predictions.
  • Ablation Study: BGEM improves baseline performance by up to 4%, DOEM adds 2%, and PCL contributes a further 1% in PASCAL-5i 1-shot ablations.The modules progressively improve segmentation performance when added to the baseline.
  • Ablation Study: The full method improves precision by 3% over baseline, while BGEM, DOEM, and PCL progressively reduce false positives.Precision is used to evaluate performance on decreasing false-positive predictions.

5. Conclusion

NTRE addresses few-shot segmentation by explicitly focusing on background and distracting-object regions. Its BG/DO elimination design and PCL are reported to outperform previous methods on benchmark datasets.

  • NTRE reframes few-shot segmentation around attention to background and distracting-object regions.
  • The channel dimension of the BG prototype is evaluated through a 1-shot PASCAL-5i ablation study using averaged class mIoU.The figure marks the best averaged class mIoU result with a star.
  • The framework combines BGMM, BGEM, and DOEM to mine and eliminate background and distracting-object information.A BG mining loss trains the BG prototype without background ground truth.
  • PCL improves the model’s ability to distinguish target objects from distracting objects.
  • Experiments on two benchmark datasets demonstrate performance superiority over previous methods.
Loading 2205.04903v1…