Source-linked AI summary
CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, Chunhua Shen
TL;DR
Pixel-wise segmentation requires costly annotations and trained models are limited to predefined classes. CANet performs few-shot segmentation of new classes using dense support-query comparison, iterative refinement, attention-based support fusion, and bounding-box support annotations. On PASCAL VOC 2012, it achieves 55.4% mIoU for 1-shot and 57.1% for 5-shot segmentation, outperforming state-of-the-art results by 14.6% and 13.2%.
Problem
Pixel-wise segmentation data is expensive to label, and trained models have difficulty predicting classes outside their predefined class set.
Method
CANet uses two-branch multi-level dense comparison, recurrent iterative optimization, attention-based k-shot fusion, and bounding-box annotated support sets.
Results
55.4% mIoU for 1-shot and 57.1% for 5-shot segmentation outperform state-of-the-art results by 14.6% and 13.2%, respectively.
Takeaways & Limitations
CANet supports few-shot segmentation of new classes and can use weak bounding-box support annotations to reduce labeling effort.
Takeaways & Limitations
The task assumes test classes are different from training classes and provides only a few annotated examples for those classes.
Abstract
from arXiv · showhide
Recent progress in semantic segmentation is driven by deep Convolutional Neural Networks and large-scale labeled image datasets. However, data labeling for pixel-wise segmentation is tedious and costly. Moreover, a trained model can only make predictions within a set of pre-defined classes. In this paper, we present CANet, a class-agnostic segmentation network that performs few-shot segmentation on new classes with only a few annotated images available. Our network consists of a two-branch dense comparison module which performs multi-level feature comparison between the support image and the query image, and an iterative optimization module which iteratively refines the predicted results. Furthermore, we introduce an attention mechanism to effectively fuse information from multiple support examples under the setting of k-shot learning. Experiments on PASCAL VOC 2012 show that our method achieves a mean Intersection-over-Union score of 55.4% for 1-shot segmentation and 57.1% for 5-shot segmentation, outperforming state-of-the-art methods by a large margin of 14.6% and 13.2%, respectively.
1. Introduction
CANet addresses costly pixel-level labeling and limited generalization to unseen classes through few-shot, class-agnostic segmentation. It combines dense feature comparison, iterative refinement, attention-based k-shot fusion, and weak support annotations.
- Few-shot segmentation aims to generalize to new classes using only a few annotated training images.
- CANet compares support and query images through a two-branch dense comparison module using multi-level CNN representations.
- Iterative optimization refines dense-comparison predictions recurrently to produce fine-grained segmentation maps.
- Attention fuses information from multiple support examples in k-shot learning instead of using non-learnable fusion methods.
- Bounding-box support annotations are explored to reduce labeling effort for few-shot segmentation.
- 55.4% mIoU for 1-shot and 57.1% for 5-shot segmentation outperform state-of-the-art results by 14.6% and 13.2%, respectively.
2. Related Work
Related work covers conventional semantic segmentation, few-shot learning, and few-shot semantic segmentation. Prior few-shot segmentation methods commonly use two-branch architectures to transfer support information to query prediction.
- Semantic Segmentation: Semantic segmentation classifies every image pixel into a predefined category set.
- Semantic Segmentation: Fully supervised segmentation commonly uses CNN backbones and requires expensive pixel-level annotations.
- Few-shot Learning: Few-shot learning seeks transferable knowledge that generalizes to new classes from scarce labeled training data.
- Few-shot Semantic Segmentation: Previous few-shot semantic segmentation methods use two-branch structures in which support information guides query-branch segmentation.
3. Task Description
The task trains on one class set and tests on different, unseen classes using only a few annotated support examples. During testing, the trained model remains fixed and predicts a query-image mask from the support set and query image.
- Training uses a class set Ctrain, while testing uses a different class set Ctest containing new classes.
- The model is expected to segment a new class from only a few annotated examples.
- After training, model parameters are fixed and require no optimization on the new test dataset.
- The model receives the support set and query image, then outputs the predicted mask for the target class in the query image.
4. Method
CANet combines dense support-query comparison with iterative refinement to segment new classes from few examples. It extends the framework to k-shot inputs through attention-based support fusion and to cheaper bounding-box support annotations.
- Dense Comparison Module: CANet uses a two-branch dense comparison module to compare support and query representations at multiple feature levels.A shared feature extractor produces representations, while the comparison module performs dense feature matching.
- Dense Comparison Module: Foreground-pooled support features are expanded across the query feature map and concatenated for dense comparison.Global average pooling over the support foreground filters irrelevant background regions before comparison.
- Attention Mechanism for k-shot Segmentation: Attention-based fusion merges comparison results from multiple support examples for k-shot segmentation.Attention outputs become weights, are normalized with softmax, and produce a weighted sum of support-specific features.
- Iterative Optimization Module: CANet iteratively refines masks by combining dense-comparison features with the previous iteration’s prediction in residual form.The iterative module uses residual blocks and ASPP to capture multi-scale information before producing confidence maps for the next iteration.
- Support Annotations: Bounding-box annotated support sets provide a cheaper alternative to pixel-wise annotations by treating the box area as foreground.The paper evaluates this relaxed support annotation setting for segmentation in the query image.
5. Experiments
Experiments evaluate CANet across few-shot segmentation benchmarks, support annotation types, feature choices, iterative refinement, fusion strategies, and multi-scale testing. The results show strong performance and benefits from the proposed modules.
- PASCAL-5i divides 20 PASCAL VOC categories into four splits, using three for training and one for testing with 1000 sampled support-query pairs.
- 14.6% and 13.2% meanIoU gains over state-of-the-art results are reported for the 1-shot and 5-shot tasks, respectively.
- Bounding-box annotated support achieves performance comparable to pixel-wise annotations, despite background noise inside the box.
- The combination of ResNet block2 and block3 features achieves the best comparison result, matching class-agnostic object parts.
- 2.8% improvement over the initial prediction is obtained with iterative optimization, while DenseCRF does not significantly improve few-shot segmentation.
- In the 5-shot setting, attention outperforms non-learnable fusion methods, while multi-scale evaluation adds 1.9% meanIoU gain.
6. Conclusion
The paper concludes that CANet combines dense multi-level comparison, iterative refinement, and attention-based support fusion for class-agnostic few-shot segmentation. Comprehensive experiments report performance significantly above previous work.
- CANet is a class-agnostic segmentation network designed for few-shot learning.
- Its dense comparison module uses multiple CNN feature levels, while its iterative optimization module refines predicted results iteratively.
- The attention mechanism is more effective than non-learnable methods for the k-shot problem.
- Comprehensive experiments show that the framework significantly outperforms previous work.