Source-linked AI summary
SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation
Xiaolin Zhang, Yunchao Wei, Yi Yang, Thomas Huang
TL;DR
One-shot semantic segmentation must identify object regions from unseen categories using only one annotated example. SG-One uses masked support-feature pooling and cosine-similarity guidance in a unified network, reporting 46.3% mIoU on PASCAL-5i and surpassing baselines.
Problem
One-shot segmentation seeks to predict query-image masks for unseen classes from one densely annotated support example.
Method
SG-One uses masked average pooling to represent support objects, cosine similarities to guide query segmentation, and a unified support-query network.
Results
46.3% cross-validate mIoU is reported on PASCAL-5i for one-shot segmentation, surpassing baseline methods.
Takeaways & Limitations
SG-One effectively segments semantic pixels of unseen categories using only one annotated example.
Abstract
from arXiv · showhide
One-shot image semantic segmentation poses a challenging task of recognizing the object regions from unseen categories with only one annotated example as supervision. In this paper, we propose a simple yet effective Similarity Guidance network to tackle the One-shot (SG-One) segmentation problem. We aim at predicting the segmentation mask of a query image with the reference to one densely labeled support image of the same category. To obtain the robust representative feature of the support image, we firstly adopt a masked average pooling strategy for producing the guidance features by only taking the pixels belonging to the support image into account. We then leverage the cosine similarity to build the relationship between the guidance features and features of pixels from the query image. In this way, the possibilities embedded in the produced similarity maps can be adapted to guide the process of segmenting objects. Furthermore, our SG-One is a unified framework which can efficiently process both support and query images within one network and be learned in an end-to-end manner. We conduct extensive experiments on Pascal VOC 2012. In particular, our SGOne achieves the mIoU score of 46.3%, surpassing the baseline methods.
I. INTRODUCTION
SG-One addresses one-shot semantic segmentation by predicting unseen-category object regions from one densely annotated support example. It uses masked support-feature pooling and pixel-wise cosine similarities to guide a unified segmentation network.
- Problem: One-shot segmentation predicts query-image object pixels for unseen classes using one densely annotated support image.The support and query categories are disjoint from training classes in the stated problem setting.
- Method: SG-One extracts robust object-related representative vectors through masked average pooling over support-image object regions.The operation avoids changing the network input structure while excluding background pixels from the average.
- Method: Pixel-wise cosine similarities between support representative vectors and query features form guidance maps for segmentation.The similarity maps provide higher-scoring target-region guidance to the segmentation process.
- Method: SG-One uses a unified network to process support and query images and learn end-to-end.Its guidance and segmentation branches produce similarity maps and query segmentation masks within one framework.
- Results: 46.3% cross-validate mIoU is reported on PASCAL-5i for one-shot segmentation, surpassing baseline methods.This result is reported as a contribution of the unified network.
II. RELATED WORK
Related work situates one-shot segmentation alongside weakly supervised segmentation, video object segmentation, and few-shot learning. These areas differ in supervision, adaptation requirements, and feature-learning strategies.
- Weakly supervised segmentation: Weakly supervised segmentation reduces labeling costs using image-level labels, scribbles, bounding boxes, or points.These methods generally lack the ability to generalize learned segmentation models to unseen classes.
- One-shot segmentation: Existing one-shot segmentation methods use Siamese networks to separately extract support and query features before fusion into probability maps.SG-One is presented as an alternative similarity-guidance framework.
- Video object segmentation: Video object segmentation targets a specific object using annotation from the first video frame, with methods using finetuning, feature embeddings, cropping, or optical flow.OSVOS is noted for testing latency because it finetunes on new video clips.
- Few-shot learning: Few-shot learning seeks recognition of new classes from few labeled samples using base models trained on disjoint classes.Embedding-based methods place same-category vectors close and different-category vectors apart.
III. METHODOLOGY
The problem is formulated as predicting query-image segmentation masks from annotated support examples, with training and testing categories disjoint. The model learns a function mapping a support image-mask pair and a query image to a predicted mask.
- The support and training sets contain annotated images and segmentation masks, while support and testing categories are disjoint from training categories.
- In the one-shot setting, one annotated support image guides segmentation of a query image from an unseen category.
- The predicted query mask assigns 1 to pixels matching the support object's category and 0 to background pixels.
- The prediction is represented as ˆYj = fθ((Ii, Yi), Ij), using a support image-mask pair and a query image as inputs.
B. Proposed Model
SG-One extracts an object representative vector from masked support features and compares it with query features using cosine similarity. The resulting similarity map guides query segmentation while preserving the network's input structure.
- Masked average pooling extracts support-object representative vectors by averaging feature pixels within the annotated object regions.Feature maps are resized to the mask resolution before pooling.
- The operation disregards background contents while retaining contextual information and leaving the network input structure unchanged.
- SG-One measures cosine similarity between the support representative vector and each query-image feature pixel to construct a similarity map.Each similarity value is defined over the query feature-map pixels.
- The similarity map is multiplied element-wise with query segmentation features, whose guided representations are optimized against ground-truth masks.The cosine similarity itself is not explicitly optimized.
C. Similarity Guidance Method
SG-One uses a stem, similarity guidance branch, and segmentation branch to process support and query images. Similarity maps guide segmentation, and multiple support images can be handled without retraining.
- The stem is a fully convolutional network that extracts intermediate features from both support and query images.
- The similarity guidance branch combines abstract support features and query features, using masked average pooling to produce representative support vectors and similarity maps.
- The segmentation branch discovers query-object regions using similarity maps and concatenated features from the parallel guidance branch.
- The four-fold evaluation uses disjoint training and testing class partitions in Pascal-5i.
- At test time, one annotated support image guides segmentation of each unseen category without fine-tuning or changing network parameters.
- For K-shot testing, SG-One either ensembles masks from K support images or averages their representative vectors without retraining.
A. Dataset and Metric
The method is evaluated on Pascal-5i, constructed from Pascal VOC 2012 and extended SDS, using four-fold cross-validation with disjoint class splits. Performance is reported with mean Intersection over Union across testing classes.
- Pascal-5i is constructed from Pascal VOC 2012 and the extended SDS dataset for evaluation.
- The 20 Pascal VOC categories are divided into four folds, each testing five classes while the remaining 15 classes train the model.
- Training and testing label sets are disjoint in the cross-validation protocol.
- For class l, IoU is computed from true positives, false positives, and false negatives as TP_l/(TP_l+FP_l+FN_l).
- mIoU averages class IoUs over the testing classes, and the reported value is averaged across the four cross-validation datasets.
B. Implementation details
SG-One processes support and query images through a VGG-16-based network for one-shot segmentation on PASCAL-5i. It reports 46.3% mIoU across four divisions and qualitative success despite limited appearance similarity, with failures linked to ambiguous or uncommon target features.
- B. Implementation details: SG-One uses VGG-16 components, pretrained ILSVRC weights, and simultaneous support-query inputs without data augmentation.The support image passes through the guidance branch, while the query image passes through both guidance and segmentation branches.
- C. Comparison: 46.3% mIoU is achieved across four PASCAL-5i divisions in one-shot segmentation, surpassing co-FCN by 5.2% and OSLSM by 5.5%.The largest gain is 7.8% on PASCAL-5i1.
- C. Comparison: SG-One can distinguish object regions from background on unseen classes even when support and query images have limited appearance similarity.Figure 3 also presents failure pairs with ground truth on the left and predictions on the right.
- C. Comparison: Reported failures occur when target regions resemble background pixels or contain uncommon features distant from representative support-object features.Examples include a bus side or car and a dog's vest.
- C. Comparison: Similarity maps highlight query regions matching support-object categories while depressing distracting regions and background.The maps are computed from cosine distance between support-object features and query feature maps.
- C. Comparison: Five-shot testing improves mIoU by only 0.8% over one-shot prediction under the reported settings.The paper attributes the marginal increase to already-close high-level features for objects sharing category labels.
D. Multi-Class Segmentation
The paper evaluates SG-One on query images containing multiple classes by using support images from different testing classes. It combines the resulting class-specific masks for multi-class prediction.
- D. Multi-Class Segmentation: The multi-class experiment randomly selects 1000 query-support entries, with each query potentially containing objects from multiple classes.Five annotated support images are sampled from the five testing classes for each entry.
- D. Multi-Class Segmentation: For each query, SG-One predicts segmentation masks separately using support images from different classes, then fuses the five class masks.Fusion compares the class-specific predictions.
E. Ablation Study
Ablations compare support-feature extraction, similarity distance, branch structure, and model size. The reported results favor masked average pooling, cosine similarity without post-third-layer ReLU, and the unified network.
- Masked Average Pooling: 46.3% mIoU is obtained with masked average pooling, versus 45.0% with masking, 41.75% with concatenation, and 42.2% with general GAP.The comparison uses the four PASCAL-5i datasets.
- Similarity Guidance: Cosine similarity without post-third-layer ReLU reaches 46.3% mIoU, while adding ReLU reduces the result to 45.5%.The non-ReLU design allows cosine measurements to vary in [−1, 1].
- Similarity Guidance: The 2-norm guidance variant achieves 30.7% mIoU across the four datasets, below the proposed cosine-similarity method.The paper therefore reports 2-norm as an inferior guidance choice in this experiment.
- The Unified Structure: The unified SG-One network has 19.0M parameters and achieves 46.3% mIoU, compared with 44.8% for the separate variant.The separate variant has 36.1M parameters, while co-FCN has 34.2M.
- The Unified Structure: The unified structure reuses the network for support and query features, which the paper attributes to a 1.5% gain over the separate variant.The paper also associates reuse with reduced computational demand and over-fitting risk.
- Task Comparison: Few-shot image segmentation differs from few-shot video segmentation because objects and background vary greatly across images but remain consistent between video frames.This comparison motivates treating image segmentation as a more variable setting.
F. Relationship with Video Object Segmentation
One-shot image segmentation differs from video segmentation because support and query images lack sequential consistency in objects and backgrounds. Fair video comparisons therefore disable sequential cues and online fine-tuning, while SG-One is evaluated under these constraints.
- Task Differences: Video segmentation benefits from consistent target objects and backgrounds across frames, unlike one-shot image segmentation.In image segmentation, support and query content can differ substantially despite sharing a category.
- Task Differences: Sequential cues and online updating available in video cannot be applied when support and query images lack temporal continuity.The paper contrasts frame-to-frame similarity and online updating with the absence of sequential information in image segmentation.
- Evaluation: DAVIS2016 comparisons disable sequential cues and online fine-tuning to provide a fair evaluation for SG-One.The table caption explicitly states that these cues are unavailable in the image segmentation task.
- Evaluation: SG-One is compared with OSVOS, VideoMatch, and RGMP after their background features and successive frame consistencies are excluded.This comparison uses versions of the baselines trained only on DAVIS2016 training data under the stated restrictions.
V. CONCLUSION AND FUTURE WORK
The paper concludes that SG-One segments unseen-category pixels from one annotated example and improves one-shot segmentation through a unified, end-to-end design. It also identifies feature and mask-quality limitations in difficult visual conditions.
- Conclusion: SG-One segments semantic pixels of unseen categories using only one annotated example.The conclusion presents this as the central capability of the proposed network.
- Conclusion: Masked average pooling extracts robust object-related representative features while incorporating contextual information without changing network inputs.The conclusion contrasts this strategy with previous input-modification approaches.
- Conclusion: A unified network reduces overfitting risks by avoiding extra parameters and supports direct multi-class image segmentation after single-class training.The reported framework is pure end-to-end and requires no pre-processing or post-processing.
- Limitations: Predicted masks may cover only part of the target or include background noise when the object closely resembles its background.This is identified as a limitation of the method in challenging visual conditions.
- Limitations: Misaligned training and test classes can prevent the network from obtaining better features for input images.The conclusion identifies class alignment as another constraint on feature quality.