Source-linked AI summary
Hierarchical Dense Correlation Distillation for Few-Shot Segmentation
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chenyao Wang, Shu Liu, Jingyong Su, Jiaya Jia
TL;DR
Few-shot segmentation seeks class-agnostic prediction of unseen classes from a few annotations, but existing approaches suffer from coarse granularity and train-set overfitting. HDMNet hierarchically decouples feature parsing and matching, introduces correlation-based matching and distillation, and achieves state-of-the-art benchmark performance, including 4.8% and 7.5% mIoU gains on COCO-20i over prior state of the art.
Problem
Few-shot segmentation methods suffer from coarse segmentation granularity, train-set overfitting, and weak generalization when relying on class-specific features.
Method
HDMNet hierarchically decouples feature parsing and matching, uses pixel-level correlation for class-agnostic matching, and distills deep semantic correspondence into shallow layers.
Results
HDMNet achieves new state-of-the-art performance on COCO-20i and PASCAL-5i, with 4.8% one-shot and 7.5% five-shot mIoU improvements over prior state of the art on COCO-20i.
Takeaways & Limitations
The design alleviates training-class overfitting and improves generality while maintaining efficiency on standard few-shot segmentation benchmarks.
Takeaways & Limitations
HDMNet may fail to clearly distinguish categories with similar appearances and close semantic relations, such as trucks and cars or chairs and sofas.
Abstract
from arXiv · showhide
Few-shot semantic segmentation (FSS) aims to form class-agnostic models segmenting unseen classes with only a handful of annotations. Previous methods limited to the semantic feature and prototype representation suffer from coarse segmentation granularity and train-set overfitting. In this work, we design Hierarchically Decoupled Matching Network (HDMNet) mining pixel-level support correlation based on the transformer architecture. The self-attention modules are used to assist in establishing hierarchical dense features, as a means to accomplish the cascade matching between query and support features. Moreover, we propose a matching module to reduce train-set overfitting and introduce correlation distillation leveraging semantic correspondence from coarse resolution to boost fine-grained segmentation. Our method performs decently in experiments. We achieve $50.0\%$ mIoU on \coco~dataset one-shot setting and $56.0\%$ on five-shot segmentation, respectively.
1. Introduction
Few-shot semantic segmentation adapts class-agnostic models to unseen classes with limited annotations, but prior approaches face coarse granularity, train-set overfitting, and weak generalization. HDMNet addresses these issues by hierarchically decoupling feature parsing and matching, using pixel-level correlation and correlation map distillation.
- Few-shot segmentation conditions query-mask prediction on semantic clues from annotated support images under an episodic paradigm.
- Previous prototype-based and adaptive-classifier methods rely on global class-wise characteristics, making dense query-support correspondence difficult.
- Matching-based methods mine pixel-level correlation but may rely heavily on class-specific features, causing overfitting and weak generalization.
- HDMNet hierarchically decouples feature parsing and matching while extending transformers to construct feature pyramids and perform dense matching.
- Correlation map distillation transfers semantic correspondence from deeper to shallower layers, making shallow features more context-aware for higher-quality prediction.
- HDMNet reports new state-of-the-art results on COCO-20i and PASCAL-5i without compromising efficiency.
2. Related Work
Prior few-shot segmentation work includes prototype-based and matching-based approaches, while transformer research supplies architectures for image feature processing. HDMNet builds on dense correspondence and distillation to transfer semantic relations across feature stages.
- Few-shot semantic segmentation: Few-shot segmentation predicts dense masks for novel classes using only a few annotations.
- Few-shot semantic segmentation: Prototype-based methods represent specific classes with prototypes, while multiple-prototype variants address their limited coverage of object regions.
- Few-shot semantic segmentation: Matching-based methods mine dense query-support correspondence with pixel-level features and provide more detailed support context.
- Transformer: The proposed architecture extends transformer-based feature processing to hierarchical feature stages used for dense matching.
- Distillation: Correlation map distillation transfers relations between query and support sets from deep sections to shallow ones in a multi-level, multi-scale structure.
3. Task Definition and Our Method
HDMNet formulates few-shot segmentation as query-mask prediction conditioned on support masks, using hierarchically decoupled feature parsing and pixel-level matching. Its correlation-based matching and correlation-map distillation target overfitting while transferring coarse semantic correspondence to finer prediction.
- Task Definition: Few-shot segmentation evaluates on disjoint novel classes, predicting query masks from support images and masks without test-time parameter optimization.Training uses both query and support masks, whereas testing accesses only support masks.
- Hierarchically Decoupled Matching: The decoupled structure avoids directly interleaving cross-attention with self-attention, which can mix query and support information and weaken parsing purity and matching consistency.The method instead separates down-sampling and matching while retaining hierarchical feature construction.
- Hierarchically Decoupled Matching: HDMNet independently parses query and support features with self-attention and inserts down-sampling between transformer blocks to build hierarchical representations.Intermediate feature maps from multiple stages are collected for subsequent matching and decoding.
- Coarse-to-Fine Decoder: The decoder progressively fuses coarse and fine hierarchical features through resizing, MLP fusion, residual connections, convolution, and bilinear up-sampling to predict the query mask.Coarse features are scaled to the fine feature resolution before fusion.
- Correlation Matching: The matching module computes pixel-level cosine-similarity correlations between transformed query features and masked support features, then fuses correspondence scores with a prior mask.Cosine similarity is used instead of dot products to reduce reliance on category-specific feature norms; inverse softmax normalizes correlations along the query axis.
- Correlation Map Distillation: Correlation-map distillation transfers semantic correspondence from deeper, coarse-resolution stages to earlier, finer stages so shallow maps retain contextual information for fine-grained segmentation.The hierarchical maps provide complementary cues: deeper stages offer semantic information, while lower stages preserve local detail.
4. Experiments
Experiments evaluate HDMNet on PASCAL-5i and COCO-20i using mIoU and FB-IoU, with component ablations examining its hierarchical matching, correlation mechanism, and distillation. HDMNet achieves state-of-the-art results while reported ablations show gains from decoupling, correlation matching, and correlation-map distillation.
- Experimental Setup: HDMNet is evaluated on PASCAL-5i and COCO-20i, using mIoU as the main metric and FB-IoU as a supplementary metric.The datasets divide classes into four folds, with three folds for training and one for testing.
- Comparison with State-of-the-Art Methods: 4.8% and 7.5% mIoU improvements over the ResNet-50 SOTA are reported on COCO-20i for 1-shot and 5-shot segmentation, respectively.The paper also reports 69.4% mIoU for 1-shot and 71.8% for 5-shot PASCAL-5i with ResNet-50.
- Ablation Study: 2.1% mIoU is gained by converting the ensemble-only framework to the hierarchically decoupled matching structure.The authors attribute this improvement to reduced noise interference from decoupling feature down-sampling and matching.
- Ablation Study: The matching pyramid compares decoder designs across accuracy, efficiency, and model size, including variants with different numbers of matching stages.Qualitative correlation maps are shown for one to three matching stages.
- Ablation Study: 0.4% mIoU is added by replacing attention with the proposed correlation mechanism in the matching module.The correlation mechanism is designed for pixel-level correspondence without directly relying on semantic-specific features.
- Ablation Study: 1.7% mIoU is added by correlation-map distillation, which transfers correlation information across matching stages.Distillation has limited influence on training results but largely improves validation results across COCO-20i splits.
5. Concluding Remarks
The paper concludes that HDMNet uses hierarchically decoupled matching and correlation-based distillation to address few-shot segmentation and improve generality. It also identifies confusion between visually similar and semantically related categories as a limitation.
- Concluding Remarks: HDMNet decouples downsampling and matching to prevent information interference and uses correlation-based matching and distillation.The paper reports that this design alleviates training-class overfitting and improves generality.
- Concluding Remarks: The model may fail to clearly distinguish categories with similar appearances and close semantic relations, such as truck and car or chair and sofa.The paper presents this limitation as a direction for future research.
A.1. Statistical Analysis
The statistical analysis argues that image complexity contributes to the performance gap between COCO-20i and PASCAL-5i. PASCAL-5i images often contain a single foreground, making the task closer to foreground segmentation than semantic discrimination.
- Statistical Analysis: PASCAL-5i contains mostly single-foreground images, whereas image complexity contributes to the performance gap between the two datasets.The analysis also considers class amount and data quantity as factors.
A.2. Ablation Experiment of the Support Mask
The support-mask ablation tests whether performance depends on supervision from support annotations by comparing standard episodes with inputs lacking the support mask. The experiment is conducted on both PASCAL-5i and COCO-20i under the 1-shot setting.
- Ablation Experiment of the Support Mask: The ablation removes the support mask to test whether the model extracts supervision information from support annotations.Quantitative and qualitative results are reported in Table 8 and Figure 11.
- Ablation Experiment of the Support Mask: The support-mask effects are evaluated on PASCAL-5i and COCO-20i under the 1-shot setting.The dataset statistics are also illustrated for the two benchmarks.
B. Number of Test Episodes
The evaluation uses episode sampling to estimate mIoU, with 10k episodes adopted for COCO-20i because 1k episodes can produce unstable results. The reported fluctuation reaches 6.0% mIoU in one fold.
- COCO-20i evaluation uses 10k random episodes per fold and averages their mIoU.
- 1k episodes are standard for PASCAL-5i because each fold contains at most 584 images.
- Figure 9 compares 1k and 10k test-sample sets for each COCO-20i fold.
- 6.0% mIoU is the largest reported difference between 1k- and 10k-episode results in COCO-20i Fold-0.
C.2. Experimental Environment
The experimental environment uses PyTorch 1.12.1 on an NVIDIA GeForce RTX 3090 GPU and an Intel Xeon Gold 6326 CPU.
- The experiments use PyTorch version 1.12.1.
- The GPU is an NVIDIA GeForce RTX 3090.
- The CPU is an Intel(R) Xeon(R) Gold 6326 CPU running at 2.90GHz.
C.3. Additional Qualitative Results
Additional visualizations compare correlation maps and predictions with or without distillation, while multistage pyramids show coarse localization followed by progressively finer segmentation detail.
- Correlation maps and dense predictions are visualized with and without correlation distillation on PASCAL-5i and COCO-20i 1-shot settings.
- The correlation pyramid visualization covers the first through third matching stages before the model output.
- Coarse-resolution correlation maps provide rough class locations, while fine layers provide more detailed features for segmentation.
D. Ablation Experiments
The ablation experiments study distillation temperature and visualize the effects of support masks, distillation, and successive correlation-pyramid stages.
- D.1. Distillation Temperature: Temperature T controls the distillation distribution, and flattened correlation maps are spatially normalized with a softmax layer.
- D.1. Distillation Temperature: When T equals 1, the distillation ablation achieves the best performance and this value is retained by default.
- Figure 11 visualizes support-mask effects using single-foreground and complex-scene query images under PASCAL-5i 1-shot evaluation.
- Figure 12 compares support and query examples, correlation maps, and predictions with or without distillation.
- Figure 13 displays support and query examples, correlation pyramids from stages 1–3, and the final prediction.