Source-linked AI summary
Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation
Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, Efstratios Gavves
TL;DR
Few-shot semantic segmentation must segment query objects from a few annotated support examples, but plain support-query interactions can miss intrinsic details. DPCN uses support-derived dynamic convolutions alongside modules for query-mask generation and background filtering; the paper reports state-of-the-art results.
Problem
Prototype-based FSS interactions using limited prototypes and plain operations can lose intrinsic query-object details, including holes and slots.
Source: §1 · p. 2Method
Results
DPCN achieves state-of-the-art results in extensive experiments.
Source: §5 · p. 8Takeaways & Limitations
DCM is designed as a plug-and-play module for sufficient interaction between support and query features.
Source: §1 · p. 2, §5 · p. 8The setting uses k annotated support image-mask pairs and disjoint training and test classes.
Source: §3.1 · pp. 3–4
Abstract
from arXiv · showhide
The key challenge for few-shot semantic segmentation (FSS) is how to tailor a desirable interaction among support and query features and/or their prototypes, under the episodic training scenario. Most existing FSS methods implement such support-query interactions by solely leveraging plain operations - e.g., cosine similarity and feature concatenation - for segmenting the query objects. However, these interaction approaches usually cannot well capture the intrinsic object details in the query images that are widely encountered in FSS, e.g., if the query object to be segmented has holes and slots, inaccurate segmentation almost always happens. To this end, we propose a dynamic prototype convolution network (DPCN) to fully capture the aforementioned intrinsic details for accurate FSS. Specifically, in DPCN, a dynamic convolution module (DCM) is firstly proposed to generate dynamic kernels from support foreground, then information interaction is achieved by convolution operations over query features using these kernels. Moreover, we equip DPCN with a support activation module (SAM) and a feature filtering module (FFM) to generate pseudo mask and filter out background information for the query images, respectively. SAM and FFM together can mine enriched context information from the query features. Our DPCN is also flexible and efficient under the k-shot FSS setting. Extensive experiments on PASCAL-5i and COCO-20i show that DPCN yields superior performances under both 1-shot and 5-shot settings.
1. Introduction
Few-shot semantic segmentation must segment novel query objects from a few annotated support examples, yet prototype-based interactions can miss intrinsic details. DPCN uses support-derived dynamic kernels and complementary query-context modules to address this challenge.
1. Introduction
- Prototype-based FSS interactions can lose query-object details such as holes and slots, especially amid appearance and scale variation.The cited methods use support prototypes with operations such as cosine similarity, summation, and feature concatenation.Source: §1 · p. 2
- DPCN's dynamic convolution module applies square and asymmetric kernels generated from support foreground features to query features, capturing object bodies and subtle details.The paper describes square kernels as capturing an object's main information and asymmetric kernels as capturing subtle details.Source: §1 · p. 2
- SAM generates an initial query pseudo-mask, and FFM refines it to retain more foreground context while filtering noise.The refined mask and query features aggregate object-related context from support and query images.Source: §1 · p. 2
- DCM is presented as a plug-and-play component for improving existing prototype-learning methods.Source: §1 · p. 2
2. Related work
The related work covers semantic segmentation context, prototype and graph-based approaches to few-shot segmentation, and dynamic convolution. DPCN's distinction is generating kernels from support foreground features, including asymmetric kernels for subtle details.
2.1. Semantic Segmentation
- Semantic segmentation methods use enlarged receptive fields and attention for context, but can lose performance when training data are insufficient.Source: §2.1 · p. 2
2.2. Few-Shot Semantic Segmentation
- Prototype-based FSS methods guide query segmentation with support prototypes, while graph-based methods use many-to-many reasoning to preserve structural information.Source: §2.2 · pp. 2–3
2.3. Dynamic Convolution Networks
- Unlike prior dynamic-convolution approaches, DPCN generates kernels from support foreground and adds asymmetric kernels to capture subtle object details.The passage contrasts this design with a prior approach using square kernels.Source: §2.3 · p. 3
3. Method
DPCN uses episodic training with disjoint training and test classes, combining support-derived activation, background filtering, and dynamic convolutions to predict query masks. Its design extends to multiple support images and trains with query- and support-mask losses.
3.1. Problem Setting
- Training and test episodes use disjoint classes, and each episode pairs support image-mask examples with a query whose mask supervises training but is unavailable at test time.The support set contains k image-mask pairs.Source: §3.1 · pp. 3–4
3.2. Overview
- DPCN extracts shared-backbone features, then uses SAM to form an initial query pseudo-mask, FFM to refine it and filter background, and DCM to apply support-derived kernels before decoding.The decoder predicts the final query segmentation mask from the computed features and pseudo-masks.Source: §3.2 · p. 4
3.3. Support Activation Module
- SAM matches support and query regions with sliding windows, using square and asymmetric shapes to capture context for regular and slender object parts.It aggregates regional matches into three activation maps and an initial pseudo-mask indicating the target's rough query location.Source: §3.3 · p. 4
3.4. Feature Filtering Module
- FFM combines a masked-average-pooled support prototype with the initial query pseudo-mask to refine the mask and filter irrelevant query background.The refined mask is used to filter the query feature.Source: §3.4 · pp. 4–5
3.5. Dynamic Convolution Module
- DCM extracts support foreground vectors, pools them into prototype groups, and generates vertical, horizontal, and square kernels for convolutions over the filtered query feature.The resulting enhanced query features are concatenated with support foreground features, activation maps, and the refined pseudo-mask for decoding.Source: §3.5 · p. 5
- The decoder predicts the query mask from the concatenated representative feature through convolution, ASPP, and classification modules.The concatenated feature includes enhanced query features and support and mask information.Source: §3.5 · p. 5
3.6. Extension to k-shot setting
- For k-shot episodes, DPCN collects foreground vectors from all support images so the kernel generator can produce more robust dynamic kernels.The paper states this leads to more adequate interaction and accurate query-mask estimation.Source: §3.6 · pp. 5–6
3.7. Training Loss
- Training combines query-mask and support-mask binary cross-entropy losses, with the support-branch loss weighted by λ = 1.0 in all experiments.The query and support mask prediction processes share the same structure and parameters.Source: §3.7 · p. 6
4. Experiments
Across PASCAL-5i and COCO-20i, DPCN outperforms prior methods under 1-shot and 5-shot evaluation. Ablations report gains from its components, dynamic-kernel choices, and 5-shot fusion strategy.
4.1. Experimental Settings
- Evaluation uses four-fold class splits on PASCAL-5i and COCO-20i, with mIoU and FB-IoU calculated over 1,000 test episodes.Source: §4.1 · p. 6
4.2. Comparisons with State-of-the-Arts
- 2.0% and 2.7%: DPCN surpasses HSNet in 1-shot PASCAL-5i with VGG16 and ResNet50, respectively.With ResNet50 in 1-shot, DPCN reaches mIoU 66.7% versus 61.4% for the baseline.Source: §4.2 · p. 6, Table 1 · p. 7
- 3.8% and 2.9% mIoU improvement over HSNet: DPCN achieves these gains on COCO-20i with ResNet50 under 1-shot and 5-shot settings, respectively.Source: Table 2 · p. 7, §4.2 · p. 7
4.3. Ablations
- DCM adds 2.2% mIoU and is the most important component; SAM and FFM are also indispensable, and all three together achieve state-of-the-art performance.Source: Table 3 · p. 7, §4.3 · pp. 7–8
- Kernel size 5 performs best (size 9 second-best), while combining square and asymmetric dynamic kernels performs better than either variant alone.Source: Table 4 · p. 7, §4.3 · p. 8, Table 5 · p. 8
4.2. Comparisons with State-of-the-Arts
- DPCN better captures object details than the baseline, preserving more tiny details in plant and bike segmentations.Source: §4.2 · p. 7, Figure 5 · p. 8
4.3. Ablations
- Adding DCM as a plug-and-play module significantly improves both CANet and PFENet.Source: Table 6 · p. 8, §4.3 · p. 8
- The proposed 5-shot fusion strategy improves mIoU by 3.2% and outperforms voting, average, and OR fusion strategies.Source: §4.3 · p. 8
5. Conclusion
DPCN combines SAM and FFM for mining query-image information with a plug-and-play DCM for support-query interaction. The paper reports state-of-the-art results for the method.
5. Conclusion
- DPCN combines SAM, FFM, and DCM to address few-shot semantic segmentation.SAM and FFM generate a pseudo query mask and filter background information, while DCM implements support-query interaction.Source: §5 · p. 8
- The method uses SAM and FFM to mine query-image information and DCM as a plug-and-play module for support-query interaction.The conclusion describes SAM and FFM as generating a pseudo query mask and filtering background information, respectively.Source: §5 · p. 8
- Extensive experiments demonstrate state-of-the-art results for DPCN.Source: §5 · p. 8
Appendix A. Implementation details
Experiments use pretrained ResNet-50 or VGG-16 backbones, with most backbone weights fixed and a shared training and evaluation setup across the two benchmarks.
Appendix A. Implementation details
- Experiments use ImageNet-pretrained ResNet-50 or VGG-16, with dilation preserving layer2–layer4 spatial resolution and only layer4 trainable.The backbone weights are fixed except for layer4.Source: Appendix A · p. 10
- Training runs for 200 PASCAL-5i epochs or 50 COCO-20i epochs, using SGD, benchmark-specific learning rates and batch sizes, and a poly schedule.Learning rates are 0.005 and 0.002, and batch sizes are 8 and 32, respectively.Source: Appendix A · p. 10
- Both benchmarks use 473 × 473 crops and the augmentation strategy in [25], with multi-scale testing and original query ground truth for metric calculation.Source: Appendix A · p. 10
- SAM uses 5 × 1, 3 × 3, and 1 × 5 windows; DCM uses 5 × 1, 5 × 5, and 1 × 5 kernels.The model was implemented in PyTorch 1.7.0 and run on Nvidia Tesla V100 GPUs with CUDA 11.3.Source: Appendix A · p. 10
Appendix B. Additional results and analyses
Appendix analyses examine bounding-box supervision, DCM transfer to existing methods, multi-scale testing, and other training and evaluation choices. The reported results include gains over PANet and CANet with bounding-box supervision and improved CANet and PFENet performance after adding DCM.
Kernel Generation Variants
- The two kernel-generation variants achieve similar performance, 66.0 vs 66.7, demonstrating DCM robustness to these generation schemes.The variants generate asymmetric and symmetric kernel weights in parallel or generate asymmetric weights first and use them to generate symmetric weights.Source: Appendix B · p. 10, Table A1 · p. 11
Experiments with Bounding Box Annotations
- With bounding-box support annotations, DPCN significantly outperforms PANet and CANet, while performing slightly worse than with pixel-wise annotations.The comparison is under 1-shot bounding-box supervision.Source: Appendix B · pp. 10–11, Table A2 · p. 11
Multi-Scale Testing
- Multi-scale testing adds 1% mIoU for ResNet-50 and a slight improvement for VGG-16.DPCN without multi-scale testing also achieves state-of-the-art results with both backbones.Source: Table A3 · p. 11, Appendix B · p. 11
Generalization Ability of DCM
- Adding DCM improves CANet and PFENet under both 1-shot and 5-shot settings.The appendix presents the comparisons in Table A4.Source: Appendix B · p. 11
Training and validation performance
- Training mIoU exceeds validation mIoU in every fold; validation is relatively stable in folds 0 and 1 but fluctuates in folds 2 and 3.The performance curves are shown in Figure A1.Source: Figure A1 · p. 11, Appendix B · p. 11
Support Loss Coefficient λ
- The support loss coefficient performs best at λ = 1 on folds 0 and 1, and λ is set to 1 in all experiments.The appendix studies λ values 0, 0.1, 0.5, 1, and 2.Source: Figure A2 · p. 11, Appendix B · p. 11
Evaluation using Original Groundtruth
- DPCN obtains similar performance when evaluated with original or resized query ground truth.The comparison uses the original ground truth or a version resized to 473 × 473.Source: Appendix B · p. 11
Appendix C. Additional qualitative results
The appendix presents qualitative comparisons on COCO-20i and PASCAL-5i, including examples of appearance and scale variation. It also visualizes activation maps and pseudo masks at different stages of the method.
Appendix C. Additional qualitative results
- On COCO-20i examples with large appearance variation, DPCN alleviates appearance variation relative to its baseline.The examples are shown in Figure A3.Source: Appendix C · p. 12, Figure A3 · p. 13
- On COCO-20i examples with large scale variation, DPCN alleviates scale variation relative to its baseline.The examples are shown in Figure A4.Source: Appendix C · p. 12, Figure A4 · p. 13
- PASCAL-5i examples compare qualitative segmentation results from DPCN and its baseline.The comparison is shown in Figure A5.Source: Appendix C · p. 12, Figure A5 · p. 14
- DPCN occasionally predicts masks more accurate than the human-annotated ground truth in examples from PASCAL-5i and COCO-20i.These examples are shown in Figure A6.Source: Appendix C · p. 12, Figure A6 · p. 14
- SAM activation maps capture complementary query-object details; the initial pseudo mask gives a rough location, and FFM's refined mask estimates it more accurately.The visualizations are shown in Figure A7.Source: Appendix C · pp. 12, 15, Figure A7 · p. 15