Source-linked AI summary
Self-Guided and Cross-Guided Learning for Few-Shot Segmentation
Bingfeng Zhang, Jimin Xiao, Terry Qin
TL;DR
Few-shot segmentation must transfer from a few annotated examples to unseen classes, but masked GAP can discard discriminative support information. SCL mines the lost information with self-guided support vectors and fuses multiple-shot predictions using cross-guided quality weighting, attaining state-of-the-art results on PASCAL-5i and COCO-20i.
Problem
Masked GAP-based support representations inevitably lose useful critical information, limiting precise segmentation of query images.
Method
SCL uses self-guided predictions to create primary and auxiliary support vectors, then cross-guided inference to weight multiple support predictions by quality without retraining.
Results
SCL achieves new state-of-the-art performance on PASCAL-5i and COCO-20i, including 61.8% and 62.9% mIoU for PASCAL-5i 1-shot and 5-shot tasks.
Takeaways & Limitations
The approach improves support information and multiple-shot prediction fusion while applying directly to different baselines during inference.
Takeaways & Limitations
The method uses foreground pixels to produce support vectors because background features are more complicated and are not guaranteed to separate from foreground features.
Abstract
from arXiv · showhide
Few-shot segmentation has been attracting a lot of attention due to its effectiveness to segment unseen object classes with a few annotated samples. Most existing approaches use masked Global Average Pooling (GAP) to encode an annotated support image to a feature vector to facilitate query image segmentation. However, this pipeline unavoidably loses some discriminative information due to the average operation. In this paper, we propose a simple but effective self-guided learning approach, where the lost critical information is mined. Specifically, through making an initial prediction for the annotated support image, the covered and uncovered foreground regions are encoded to the primary and auxiliary support vectors using masked GAP, respectively. By aggregating both primary and auxiliary support vectors, better segmentation performances are obtained on query images. Enlightened by our self-guided module for 1-shot segmentation, we propose a cross-guided module for multiple shot segmentation, where the final mask is fused using predictions from multiple annotated samples with high-quality support vectors contributing more and vice versa. This module improves the final prediction in the inference stage without re-training. Extensive experiments show that our approach achieves new state-of-the-art performances on both PASCAL-5i and COCO-20i datasets.
1. Introduction
Few-shot segmentation addresses unseen classes with limited annotations, but masked GAP can discard discriminative support information. SCL mines this information through self-guided and cross-guided modules, achieving state-of-the-art results on PASCAL-5i and COCO-20i.
- Motivation: Few-shot segmentation uses one or a few annotated support samples to segment unseen classes in query images.Most approaches encode support and query images with Siamese networks and summarize support features into prototypes.
- Motivation: Masked GAP can lose useful support information, causing inaccurate query segmentation even when support and query images are identical.The average operation may remove information needed for precise pixel predictions.
- Approach: SCL’s self-guided module predicts the annotated support image, then encodes covered and uncovered foreground regions as primary and auxiliary support vectors.Aggregating these vectors provides more comprehensive support information for query segmentation.
- Approach: SCL’s cross-guided module evaluates prediction quality across multiple support images and weights higher-quality supports more heavily during fusion without retraining.The module directly improves multiple-shot inference using the existing segmentation model.
- Results: 61.8% and 62.9% mIoU are reported for 1-shot and 5-shot PASCAL-5i, while COCO-20i reaches 37.0% and 39.9%.These results are reported as new state-of-the-art performances for the task.
2. Related Work
Related few-shot segmentation methods commonly use metric learning and masked feature pooling to compare support and query images. The paper identifies information loss from collapsing foreground or background features into a single vector as a limitation of this paradigm.
- Segmentation settings: Fully supervised segmentation methods rely on extensive pixel-level annotations and perform poorly when classes are unseen or annotations are insufficient.Few-shot segmentation is presented as an alternative setting for this challenge.
- Metric-learning approaches: Most few-shot segmentation approaches use metric learning with support and query images processed by Siamese convolutional networks.Support information guides query segmentation through cosine similarity or dense feature comparison.
- Support representation: Masked GAP and related feature-fusion methods summarize foreground or background features as a single vector, which can discard useful information.The proposed method addresses this limitation with self-guided support information.
3. Problem Setting
Few-shot segmentation evaluates a model on unseen classes using episodic support and query sets. Each episode specifies a class and provides annotated support images alongside query images for segmentation.
- Task definition: The task trains on Dtrain and evaluates on Dtest, whose class sets are disjoint: Ctrain ∩ Ctest = ∅.This formalizes evaluation on unseen object classes.
- 1-shot framework: The 1-shot framework encodes support and query images into feature maps, applies masked GAP to obtain an initial support vector, and generates refined vectors through SGM.Support and query encoders share weights.
- Episode construction: A K-shot episode contains a support set of K annotated images and a query set for the same target class.Support masks identify the class, while query images are segmented using support information.
- Train-test protocol: The model learns from training support and query sets, then is evaluated with test support and query sets.Query masks are used only for training in the described setup.
4. Methodology
The framework improves few-shot segmentation by recovering support information lost during masked GAP, then uses these richer support representations for query prediction and multi-shot fusion. Self-guided learning generates primary and auxiliary support vectors from support-image predictions, while cross-guided learning weights multiple support predictions by support quality without retraining.
- 1-shot segmentation: The 1-shot framework encodes support and query images with a shared encoder, then generates an initial support vector using masked GAP over foreground pixels.The support image mask identifies foreground regions, and the mask is downsampled to the feature-map resolution.
- Self-guided learning: SGM predicts the support mask and uses correctly predicted foreground pixels to form a primary vector while collecting missing foreground pixels in an auxiliary vector.The primary vector preserves the main support information; the auxiliary vector targets critical information omitted by the initial support vector.
- Design assumption: Support vectors use foreground pixels only because background content is more complicated and may not remain well separated from foreground features.This is an explicit design assumption of the support-vector construction.
- Query prediction: The primary and auxiliary vectors are expanded and concatenated with query features, then processed by a query FPM and decoder to produce the query mask.The support and query branches use shared encoders and decoders, while support FPM variants provide single-scale or multi-scale decoder inputs.
- Cross-guided learning: Cross-guided learning fuses predictions from multiple annotated support images, assigning greater contribution to support images with higher-quality support vectors.For each support image, the model predicts other support images and evaluates confidence using intersection over union with their ground-truth masks.
- Cross-guided learning: CGM improves inference without retraining by directly reusing the segmentation model trained for the 1-shot task.The method therefore applies cross-guided fusion during inference rather than training a separate multi-shot model.
5. Experiments
Experiments evaluate SCL with CANet and PFENet on PASCAL-5i and COCO-20i, using mIoU and FB-IoU across 1-shot and 5-shot settings. Results show consistent gains, with SGM improving 1-shot segmentation and CGM further improving 5-shot inference without retraining.
- Experimental Setup: SCL is evaluated with CANet and PFENet baselines on PASCAL-5i, and with PFENet on COCO-20i.PASCAL-5i uses four class splits, while COCO-20i divides 80 classes into four splits; evaluation uses mIoU and FB-IoU.
- State-of-the-Art Comparisons: 2.1% and 1.0% mIoU increases are obtained over CANet and PFENet, respectively, for 1-shot PASCAL-5i segmentation.For 5-shot segmentation, SCL reaches 59.2% mIoU with CANet and 62.9% with PFENet, directly improving both baselines without retraining.
- State-of-the-Art Comparisons: 4.6% and 1.4% mIoU gains are reported on COCO-20i for 1-shot and 5-shot segmentation, respectively.On PASCAL-5i, CANet also gains 4.1% and 1.1% in FB-IoU for 1-shot and 5-shot settings.
- Ablation Study: 2.1% mIoU and 4.1% FB-IoU gains result from adding SGM to the baseline in 1-shot PASCAL-5i ablations.With SGM and CGM together, 5-shot mIoU reaches 59.2%, while CGM adds 0.5% mIoU during inference without retraining.
- Ablation Study: 57.5% mIoU and 70.3% FB-IoU are achieved when primary and auxiliary support vectors are used jointly, compared with 55.6% and 67.3% for the initial vector alone.Using either primary or auxiliary vectors individually performs worse than using them jointly.
6. Conclusion
The approach extracts more comprehensive support information through self-guided learning and improves multiple-support fusion with cross-guided learning. Experiments support the effectiveness of both modules, while future work will incorporate background information as extra support.
- Self-guided learning extracts comprehensive support information for few-shot segmentation.
- Cross-guided fusion gives higher-quality support images greater influence on the final prediction.
- Extensive experiments show the effectiveness of the proposed self-guided and cross-guided modules.
- Future work will use background information as extra support to improve the approach.