Source-linked AI summary
Learning What Not to Segment: A New Perspective on Few-Shot Segmentation
Chunbo Lang, Gong Cheng, Binfei Tu, Junwei Han
TL;DR
Few-shot segmentation models can be biased toward seen classes, hindering recognition of novel concepts. BAM adds a base learner to identify base-class regions, adaptively ensembles it with a meta learner using scene-difference adjustment, and achieves state-of-the-art benchmark performance while extending to generalized FSS.
Problem
Meta-learning-based FSS models are biased toward seen classes rather than being class-agnostic, hindering recognition of new concepts.
Method
BAM combines complementary base and meta learners with an ensemble module, using query-support scene differences to adjust the meta learner's contribution.
Results
BAM sets new state-of-the-art results across FSS benchmark settings, including 4.71%p and 4.66%p mIoU gains on PASCAL-5i and 7.03%p and 4.26%p gains on COCO-20i.
Takeaways & Limitations
The scheme addresses FSS bias with two plain learners and extends segmentation to generalized FSS, identifying both base- and novel-class pixels.
Abstract
from arXiv · showhide
Recently few-shot segmentation (FSS) has been extensively developed. Most previous works strive to achieve generalization through the meta-learning framework derived from classification tasks; however, the trained models are biased towards the seen classes instead of being ideally class-agnostic, thus hindering the recognition of new concepts. This paper proposes a fresh and straightforward insight to alleviate the problem. Specifically, we apply an additional branch (base learner) to the conventional FSS model (meta learner) to explicitly identify the targets of base classes, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to yield precise segmentation prediction. Considering the sensitivity of meta learner, we further introduce an adjustment factor to estimate the scene differences between the input image pairs for facilitating the model ensemble forecasting. The substantial performance gains on PASCAL-5i and COCO-20i verify the effectiveness, and surprisingly, our versatile scheme sets a new state-of-the-art even with two plain learners. Moreover, in light of the unique nature of the proposed approach, we also extend it to a more realistic but challenging setting, i.e., generalized FSS, where the pixels of both base and novel classes are required to be determined. The source code is available at github.com/chunbolang/BAM.
1. Introduction
Few-shot segmentation models trained with meta-learning can remain biased toward base classes, limiting novel-class recognition. BAM adds a base learner and adaptive ensemble, estimates query-support scene differences, achieves state-of-the-art results, and extends to generalized FSS.
- Few-shot segmentation uses scarce labeled data to segment a specific semantic category from a raw image.
- Meta-training on base classes introduces seen-class bias, hindering novel-concept recognition and potentially collapsing on hard, similar query samples.
- BAM adds a base learner that predicts base-class regions, then adaptively integrates its coarse output with the meta learner to refine novel-object segmentation.
- BAM estimates query-support scene differences through a Gram matrix to mitigate adverse effects from meta-learner sensitivity.
- BAM sets new state-of-the-art results across FSS benchmark settings, even with two plain learners.
- The approach extends to generalized FSS, where pixels belonging to both base and novel classes must be identified.
2. Related Works
Related work covers semantic segmentation, few-shot learning, and few-shot segmentation. Existing FSS methods commonly rely on meta-learning and can suffer from base-class bias, domain shift, and fragile generalization.
- Semantic Segmentation: Semantic segmentation assigns every image pixel to a predefined semantic category, but conventional models require extensive annotation and generalize poorly to unseen categories.
- Few-Shot Learning: Few-shot learning commonly samples episodic tasks from base data to mimic few-shot scenarios, using metric-, optimization-, or augmentation-based approaches.
- Few-Shot Learning: The paper extends low-data segmentation to generalized FSS by predicting base-class regions while recognizing new targets.
- Few-Shot Segmentation: Few-shot segmentation transfers support annotations through support and query branches to segment novel categories with limited labeled data.
- Few-Shot Segmentation: FSS generalization can be fragile because unbalanced data and domain shift bias trained models toward base classes.
3. Problem Definition
Few-shot segmentation trains on disjoint base and test categories, using episodic support-query tasks to learn transferable knowledge. Evaluation tests query predictions on unseen categories without further optimization.
- Few-shot segmentation performs segmentation with only a few labeled data.
- Training and testing image sets contain disjoint object categories, requiring transfer from richly annotated base data to scarce-example test data.
- Each episode contains a small support set and a query set with a raw image and binary mask for a specific category.
- Models predict the query image under the support set during episodic training and are evaluated across test episodes without further optimization.
4. Proposed Method
BAM combines complementary base and meta learners with an ensemble module, using shared features and an adjustment factor to improve few-shot segmentation. It also adaptively weights support images in K-shot settings and extends the framework to generalized FSS.
- Overall architecture: BAM uses base and meta learners with a shared backbone, then integrates their predictions through an ensemble module.The base learner predicts base-class regions, while the meta learner recognizes the task-specific category.
- Base learner: The base learner is trained with standard supervised learning rather than episodic learning and shares the backbone with the meta learner.A two-stage training strategy addresses the tension between frozen-backbone FSS training and supervised base segmentation.
- Meta learner: The meta learner transfers support-mask information to the query image through feature guidance, prototype computation, and a decoder.Masked average pooling provides class-related prototype cues for query segmentation.
- Ensemble module: The adjustment factor ψ measures scene differences between support and query images using low-level feature Gram matrices and the Frobenius norm.The resulting indicator guides integration of the two learners’ coarse predictions.
- K-shot setting: In K-shot settings, BAM adaptively weights support images by their adjustment factors, giving greater contribution to samples with smaller values.This replaces equal prototype averaging when support images differ substantially from the query.
- Extension to generalized FSS: For generalized FSS, BAM fuses the base learner output with the ensembled prediction using a predefined threshold τ.The resulting prediction covers both base and novel classes without learnable fusion parameters.
5. Experiments
The experiments evaluate BAM on PASCAL-5i and COCO-20i using standard FSS metrics, comparisons, qualitative results, and ablations. BAM achieves state-of-the-art results, suppresses base-class false activations, and extends effectively to generalized FSS.
- Experimental Setup: BAM is evaluated on PASCAL-5i and COCO-20i using mIoU and FB-IoU under cross-validation.Both datasets divide categories into four folds, with 1,000 support-query pairs sampled per fold for validation.
- Quantitative Results: 4.71%p and 4.66%p mIoU improvements over previous best results are achieved on PASCAL-5i with VGG16 for 1-shot and 5-shot settings.
- Quantitative Results: 7.03%p and 4.26%p mIoU gains over HSNet are achieved on COCO-20i with ResNet50 for 1-shot and 5-shot settings.BAM also substantially improves FB-IoU on PASCAL-5i, especially in the 1-shot ResNet50 setting.
- Qualitative Results: Qualitative results show that BAM significantly suppresses falsely activated base-class targets compared with the baseline.Figure 4 compares support and query ground-truth masks with baseline and BAM predictions under 1-shot evaluation on both datasets.
- Ablation Study: Separately trained learners outperform jointly trained learners, while ensemble initialization with meta-learner weight 1 and base-learner weight 0 improves mIoU by 2.73%.The adjustment factor ψ is essential for ensemble forecasting, and B2 features provide the better accuracy-complexity trade-off.
- Generalized FSS: In generalized FSS, the ensemble improves segmentation across novel classes, base classes, and all classes, while support bounding-box annotations remain competitive with pixel-wise masks.The generalized output is formed by merging final and base-learner predictions using a predefined threshold τ.
6. Conclusion
The paper proposes BAM to reduce seen-class bias in FSS by using a base learner to identify confusable base regions and refine meta-learner predictions. It reports new state-of-the-art FSS results and strong baseline results for generalized FSS.
- BAM uses a base learner to identify confusable base regions and refine the meta learner’s query-image predictions.
- The scheme achieves new state-of-the-art results on FSS benchmarks despite using two plain learners.
- BAM is extended to generalized FSS, producing strong baseline results in the setting where base and novel classes must both be identified.
A. Calculation of FLOPs
The FLOPs calculation measures the computational complexity of estimating the scene-difference adjustment factor ψ from support and query low-level features.
- The method computes support and query Gram matrices, subtracts them, and takes the Frobenius norm to obtain ψ.The three operations correspond to Gram-matrix computation, matrix subtraction, and Frobenius-norm calculation.
- FLOPs quantify the computational complexity of these three operations, with N defined as H × W.
B. Implementation Details
Generalized FSS combines base-learner and ensembled predictions using a predefined threshold, while an alternative fusion scheme reverses their primary roles. The reported comparison favors the former slightly for BAM.
- The primary generalized-FSS scheme fuses base-learner and final ensembled predictions with threshold τ=0.9 to produce holistic segmentation results.
- The alternative scheme relies primarily on base-learner predictions and uses final predictions to determine novel pixels in background regions.
- BAM obtains similar results with both fusion schemes, with the primary scheme slightly better; the baseline is tolerable only with the alternative.
- For ResNet50, using B2 features in the adjustment-factor flow costs 3.78G FLOPs at 512×60×60 feature resolution.