Source-linked AI summary
Feature Weighting and Boosting for Few-Shot Segmentation
Khoi Nguyen, Sinisa Todorovic
TL;DR
Few-shot segmentation must segment query objects from only a few masked support examples despite variation in object appearance and pose. The paper improves feature discriminativeness and adds gradient-guided ensemble boosting at test time, achieving significant gains over prior work on PASCAL-5^i and COCO-20^i in one- and five-shot settings.
Problem
Few-shot object segmentation must segment a query image from a few support images and masks despite large variation in target instances and poses.
Method
The paper encourages foreground-selective features with a closed-form relevance optimization and uses gradient-guided support-feature ensembles for test-time boosting, jointly handling K-shot supports.
Results
The method significantly outperforms the state of the art on PASCAL-5^i and COCO-20^i in both one-shot and five-shot settings.
Takeaways & Limitations
Combining feature relevance weighting with gradient boosting gives a significant performance gain over either contribution alone.
Takeaways & Limitations
The K-shot extension jointly analyzes all support images and assumes this is more robust than independently averaging feature relevances.
Abstract
from arXiv · showhide
This paper is about few-shot segmentation of foreground objects in images. We train a CNN on small subsets of training images, each mimicking the few-shot setting. In each subset, one image serves as the query and the other(s) as support image(s) with ground-truth segmentation. The CNN first extracts feature maps from the query and support images. Then, a class feature vector is computed as an average of the support's feature maps over the known foreground. Finally, the target object is segmented in the query image by using a cosine similarity between the class feature vector and the query's feature map. We make two contributions by: (1) Improving discriminativeness of features so their activations are high on the foreground and low elsewhere; and (2) Boosting inference with an ensemble of experts guided with the gradient of loss incurred when segmenting the support images in testing. Our evaluations on the PASCAL-$5^i$ and COCO-$20^i$ datasets demonstrate that we significantly outperform existing approaches.
1. Introduction
Few-shot segmentation uses masked support features and query features to segment a target class despite changes in instance count and pose. The paper addresses feature nondiscriminativeness and few-shot overfitting with relevance weighting and gradient-guided boosted inference.
- Few-shot segmentation predicts the target class in a query image from a few support images with known foreground masks, despite differences in instance count and 3D pose.
- Prior methods pool support features over the known foreground, compute a cosine similarity map with query features, and use both for query segmentation.
- Feature relevance weighting encourages high activations inside target foreground locations and low activations elsewhere, using a closed-form solution to an optimization problem.
- Gradient-guided boosted inference adapts support features into an ensemble whose weighted segmentation predictions form the final query prediction during testing.
- For K-shot segmentation, both contributions jointly analyze the support images and their ground truths rather than treating supports independently.
- The method significantly outperforms prior work on PASCAL-5^i and is also evaluated on the larger COCO-20^i dataset.
2. Related Work
Few-shot segmentation adapts classification and semantic-segmentation ideas to label query pixels using masks from only a few support images. Prior methods condition query segmentation through pooled support features, prototypes, or similarity estimates.
- Few-shot classification methods use transfer learning, meta-learning, or generative data augmentation to learn from limited examples.
- Standard semantic-segmentation networks label every image pixel but typically require very large training sets.
- Few-shot semantic segmentation labels query pixels for a target class conditioned on ground-truth masks from a few support images.
- The paper’s boosted inference forms an ensemble of support features and averages the resulting query similarity maps, extending beyond prior conditioning mechanisms.
- Existing approaches condition query segmentation using concatenated pooled support features, predicted prototypes with nearest-neighbor labeling, or pooled-support and query-feature similarity.
3. Our Approach
The approach trains a few-shot segmentation network episodically, using masked support features and query similarity maps, then improves feature discriminativeness and test-time robustness through feature weighting and guided ensembles.
- Episodic Training: The method samples support-query image pairs with binary masks and trains a CNN end to end using cross-entropy on the predicted query mask.The support mask conditions the target class, while the query mask supplies the training target.
- Support-Query Matching: Masked averaging of support feature maps produces a class feature vector whose cosine similarities with query features form the segmentation cue.High similarity values indicate likely target-class locations in the query image.
- Feature Weighting: Feature weighting estimates relevance from foreground-background activation differences, encouraging high foreground and low background activations through a closed-form optimization solution.The relevance vector is incorporated into cosine similarity during both training and testing with low computational overhead.
- Feature Boosting: Guided ensemble inference adapts support features using gradients of support-segmentation loss while keeping the CNN parameters fixed during testing.Each adapted expert predicts a query mask, and experts are fused using confidence estimates based on support segmentation quality.
- K-shot Setting: For K-shot segmentation, the contributions are extended by jointly estimating feature relevance and constructing one ensemble over all support images rather than independent ensembles.The paper describes this joint treatment as more robust than independently estimated feature relevances and as improving robustness beyond standard query-mask averaging.
4. Implementation Details and Complexity
The implementation uses modified VGG-16 or ResNet-101 feature extractors and an 8-stride feature representation. Boosted inference adds an ensemble-dependent testing cost while leaving training complexity unchanged.
- Implementation Details: The CNN modifies the final two convolutional layers, uses dilation rates 2 and 4, and produces feature maps with stride 8.The segmentation head uses 3 × 3 and 1 × 1 convolutions, with 128 channels followed by background and foreground outputs.
- Implementation Details: Training uses ImageNet-pretrained CNNs, 512 × 512 resized images, SGD with learning rate 7e−3, batch size 8, and 10,000 iterations.Test images retain their original size, and the experiments analyze the number of experts for contribution 2.
- Complexity: Prior-work training complexity combines CNN feature extraction, O(d w h) similarity computation, and, for some methods, O(Conv) segmentation.The convolutional segmentation cost is O(Conv) = O(d w h) because both operate on d×w ×h feature maps.
- Complexity: Testing complexity becomes O(Test) = O(CNN)+O(N d w h), adding N query segmentations while increasing only the smaller complexity term.For small N, O(CNN) remains dominant; with N = 10, the paper reports significantly better state-of-the-art performance.
5. Experiments
Experiments evaluate the method with four-fold cross-validation on PASCAL-5i and COCO-20i, using mIoU and ablations across one-shot and five-shot settings. The full method outperforms prior approaches, while joint multi-support analysis improves five-shot performance and boosted inference trades additional test time for larger gains.
- Datasets: PASCAL-5i uses four folds, with five test classes and 15 training classes per fold; COCO-20i uses 20 test classes and 60 training classes.Each fold samples 1000 support-query pairs, while COCO-20i is designed as a more challenging benchmark because MSCOCO has more classes and lower-quality masks.
- Metrics: mIoU averages class IoUs across the four cross-validation folds, with IoU computed from true-positive, false-positive, and false-negative pixels.The experiments use mean intersection-over-union as their quantitative metric.
- Training/testing time: Contribution 1 adds very little computational overhead, whereas contribution 2 increases testing time by about 40% with VGG and 35% with ResNet but yields larger performance gains.Using C2 increases only test time, consistent with its boosted inference design.
- One-shot Segmentation: The full one-shot method B+C1+C2 achieves the best PASCAL-5i performance with both VGG-16 and ResNet-101, and ResNet-101 improves average mIoU over state of the art by 13.49%.Combining both contributions improves results over either contribution alone, while contribution 2 produces larger gains than contribution 1 individually.
- Five-shot Segmentation: Our-K-shot achieves the best five-shot PASCAL-5i performance with both backbones, with ResNet-101 increasing average mIoU over state of the art by 15.97%.Jointly analyzing the K support images outperforms averaging them in every cross-validation fold.
- COCO-20i: On COCO-20i, lower mIoU values indicate greater difficulty than PASCAL-5i, although ResNet-101 gives higher average results.In fold COCO-200, VGG-16 outperforms ResNet-101 in both one-shot B+C1+C2 and five-shot Our-K-shot settings.
- Qualitative Results: Qualitative examples show that five-shot Our-K-shot better handles pose changes, partial occlusion, and incomplete object predictions than one-shot B+C1+C2.Examples include improved segmentation of bikes, bottles, and airplanes.
6. Conclusion
The paper addresses one-shot and few-shot object segmentation with two contributions: discriminative feature weighting and gradient-boosted fine-tuning for new classes. On PASCAL-5i and COCO-20i, the method significantly outperforms the state of the art in both one-shot and five-shot settings.
- The method encourages high foreground responses and low background activations to improve object segmentation.This contribution is formulated as an optimization problem.
- The method applies gradient boosting to fine-tune the model for new classes during testing.In the few-shot setting, support images and their ground truths are jointly analyzed.
- Significantly outperforms the state of the art on both PASCAL-5i and COCO-20i in one-shot and five-shot settings.
- Using only the second contribution performs better than using only the first, while combining both yields a significant gain over either contribution alone.