Source-linked AI summary
Prototype Mixture Models for Few-shot Semantic Segmentation
Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, Qixiang Ye
TL;DR
Few-shot segmentation must transfer from limited masked support images to query images with potentially different appearances and poses, while single prototypes create semantic ambiguity. PMMs estimate multiple prototypes with EM, use them as representations and classifiers in a duplex strategy, and improve state-of-the-art few-shot segmentation performance. The method is illustrated for 1-shot segmentation and extended to 5-shot by feeding five support images to PMMs.
Problem
Few-shot segmentation must segment query images from a few masked support examples, even when target categories are absent from training and object appearances or poses differ.
Method
PMMs use EM to estimate mixed foreground and background prototypes from support features, then apply them as representations and classifiers for duplex query segmentation.
Results
PMMs significantly improve state-of-the-art few-shot segmentation performance, with a reported 3.34% improvement (55.27% vs. 51.93%) in semantic segmentation.
Takeaways & Limitations
PMMs provide a method for capturing diverse object-part semantics from few support examples while activating foreground and suppressing background regions.
Takeaways & Limitations
The architecture is illustrated for 1-shot segmentation and extended to 5-shot by feeding five support images to PMMs to estimate prototypes.
Abstract
from arXiv · showhide
Few-shot segmentation is challenging because objects within the support and query images could significantly differ in appearance and pose. Using a single prototype acquired directly from the support image to segment the query image causes semantic ambiguity. In this paper, we propose prototype mixture models (PMMs), which correlate diverse image regions with multiple prototypes to enforce the prototype-based semantic representation. Estimated by an Expectation-Maximization algorithm, PMMs incorporate rich channel-wised and spatial semantics from limited support images. Utilized as representations as well as classifiers, PMMs fully leverage the semantics to activate objects in the query image while depressing background regions in a duplex manner. Extensive experiments on Pascal VOC and MS-COCO datasets show that PMMs significantly improve upon state-of-the-arts. Particularly, PMMs improve 5-shot segmentation performance on MS-COCO by up to 5.82\% with only a moderate cost for model size and inference speed.
1 Introduction
Few-shot segmentation must generalize from limited support images to query images whose target objects may differ substantially in appearance and pose. PMMs address semantic ambiguity from single globally pooled prototypes by modeling diverse regions with multiple prototypes and using them in a duplex segmentation strategy.
- Few-shot segmentation uses a few masked support images to segment query images, including target categories absent from training data.
- Global average pooling discards object layout, causing a single prototype to mix object-part semantics and leave ambiguity unresolved.
- PMMs estimate mixed prototypes with EM by treating masked deep pixels as positive samples and modeling background regions.
- PMMs use a duplex strategy as spatially squeezed representations and C-dimensional linear classifiers to activate object-related channels and produce probability maps.
- PMMs, assembled into RPMMs with a residual structure, significantly improve upon state-of-the-art methods.
2 Related work
Related work frames few-shot segmentation primarily as metric learning with prototype-based comparison between support and query regions. Existing single-prototype approaches remain limited because global average pooling mixes spatially distinct object-part semantics.
- Few-shot Learning: Few-shot learning methods are broadly categorized as metric learning, meta-learning, or data augmentation.
- Few-shot Learning: Simple prototypes can achieve state-of-the-art results in metric learning, motivating their use for representative and discriminative features.
- Few-shot Segmentation: Few-shot segmentation methods commonly learn prototype vectors from support images and use metric modules to segment query images.
- Few-shot Segmentation: Global average pooling typically disregards object extent, mixes semantics from different parts, and reduces prototype diversity and feature representation capacity.
- Few-shot Segmentation: Prototypical networks use class prototypes, whereas this approach represents object parts and mixes multiple prototypes for a sample or class.
3 The Proposed Approach
The approach learns foreground and background prototype mixture models from support features, then uses them in two complementary operations to activate query features for segmentation. Residual stacking further enhances representation capacity through sequential refinement.
- Support and query branches: Support features are partitioned into foreground S+ and background S− samples using the support mask, and separate PMMs are learned for each.Foreground PMMs represent object parts, while background PMMs model non-object regions.
- Prototype mixture models: PMMs are probability mixture models whose prototype vectors represent clusters of feature samples associated with object regions.The mixture combines base distributions, with each prototype corresponding to a cluster and potentially an object-part region.
- Prototype mixture models: The models are estimated through iterative EM steps that update mixture expectations and prototype mean vectors from support samples.Foreground and background PMMs are estimated separately before query activation.
- Query feature activation: P-Match upsamples and concatenates prototype representations with query features, using convolution for channelwise comparison that activates foreground-related channels.This operation incorporates semantic information about the complete object extent into query features.
- Query feature activation: P-Conv multiplies each prototype vector element-wise with query features and applies channelwise Softmax to produce probability maps for semantic activation.The resulting maps are summarized and concatenated with query features before segmentation prediction.
- Residual prototype mixture models: Residual PMMs stack multiple mixture models and use residuals from earlier query branches to supervise later branches for fine-grained segmentation.This model-level ensemble minimizes residuals between branches rather than combining multiple models directly against the ground-truth mask.
4 Experiments
Experiments evaluate PMMs and RPMMs through visualizations, ablations, efficiency measurements, and Pascal-5i and MS COCO benchmarks. The results show improved segmentation, with moderate model and inference costs.
- Visual analysis: PMMs activate more complete object extents and multiple objects than CANet by producing and fusing multiple probability maps.CANet’s single prototype tends to miss object parts or whole objects.
- Visual analysis: PMMs improve recall by segmenting more target pixels, while background prototypes reduce false positives and RPMMs refine boundaries around hard pixels.These observations come from semantic segmentation comparisons against the baseline and proposed modules.
- Ablation study: 3.34%: PMMs improve Pascal-5i mean performance from 51.93% to 55.27% through P-Match and the duplex strategy.P-Match contributes 2.70%, and the duplex strategy adds 0.64%.
- Ablation study: 1.07%: RPMMs further improve performance from 55.27% to 56.34% by using residuals from previous query branches to supervise subsequent branches.The residual ensemble strategy is intended to reduce errors step by step.
- Ablation study: K = 2 significantly outperforms K = 1, while the best Pascal-5i mean performance occurs at K = 3.Performance slightly decreases for K = 4 and K = 5, partly because more prototypes divide the limited samples from one support image.
- Efficiency: PMMs have 19.5M parameters and run at 26 FPS, compared with 19M and 29 FPS for CANet; RPMMs run at 20 FPS with 19.6M parameters.The 1×1×C prototype vectors add little model size or computational cost.
- Benchmark performance: PMMs outperform state-of-the-art methods on Pascal-5i and MS COCO in both 1-shot and 5-shot settings.With a ResNet50 backbone, RPMMs improve Pascal-5i by 2.38% in 1-shot and 1.50% in 5-shot settings.
- Benchmark performance: 5.82%: In MS COCO 5-shot segmentation, RPMMs outperform PANet by 5.82% and FWB by 11.87%.RPMMs improve the baseline by 7.66% in the same setting.
5 Conclusion
Prototype mixture models (PMMs) address semantic ambiguity in few-shot segmentation by representing diverse object regions with multiple prototypes. They incorporate channel-wise and spatial semantics from limited support images and improve segmentation performance on MS COCO.
- PMMs correlate diverse image regions with multiple prototypes to address semantic ambiguity in few-shot segmentation.
- During training, PMMs incorporate channel-wise and spatial semantics from limited support images.
- During inference, PMMs match query features in a duplex manner to perform accurate semantic segmentation.
- PMMs improve few-shot segmentation performance on the large-scale MS COCO dataset.
- PMMs offer a general approach for capturing diverse object-part semantics from few support examples.