PaperlayerView exact revision on arXiv

Source-linked AI summary

CRNet: Cross-Reference Networks for Few-Shot Segmentation

Weide Liu, Chi Zhang, Guosheng Lin, Fayao Liu

arXiv:2003.10658v1cs.CV

TL;DR

Pixel-level annotation is costly, and models trained on existing categories may not transfer well to novel classes. CRNet addresses few-shot segmentation with mutual support-query prediction, cross-reference features, recurrent mask refinement, and k-shot finetuning. On PASCAL VOC 2012, it achieves state-of-the-art performance.

  • Problem

    Pixel-level annotation requires costly large-scale labeled datasets, while knowledge learned from previous categories transfers poorly to unseen classes.

  • Method

    CRNet jointly predicts support and query masks, mines co-occurrent features, recurrently refines cached confidence maps, and finetunes network parts for multiple support images.

  • Results

    CRNet achieves state-of-the-art performance on the PASCAL VOC 2012 dataset.

  • Takeaways & Limitations

    Mutual support-query guidance, mask refinement, and support-set finetuning form an effective design for few-shot segmentation.

Abstract

from arXiv · show

Over the past few years, state-of-the-art image segmentation algorithms are based on deep convolutional neural networks. To render a deep network with the ability to understand a concept, humans need to collect a large amount of pixel-level annotated data to train the models, which is time-consuming and tedious. Recently, few-shot segmentation is proposed to solve this problem. Few-shot segmentation aims to learn a segmentation model that can be generalized to novel classes with only a few training images. In this paper, we propose a cross-reference network (CRNet) for few-shot segmentation. Unlike previous works which only predict the mask in the query image, our proposed model concurrently make predictions for both the support image and the query image. With a cross-reference mechanism, our network can better find the co-occurrent objects in the two images, thus helping the few-shot segmentation task. We also develop a mask refinement module to recurrently refine the prediction of the foreground regions. For the $k$-shot learning, we propose to finetune parts of networks to take advantage of multiple labeled support images. Experiments on the PASCAL VOC 2012 dataset show that our network achieves state-of-the-art performance.

1. Introduction

Few-shot segmentation addresses the cost and limited transferability of pixel-level annotation by learning from a few labeled examples. CRNet uses mutual support-query guidance, recurrent mask refinement, and support-set finetuning, achieving state-of-the-art PASCAL VOC 2012 performance.

  • Motivation: Large-scale pixel-level datasets are expensive to collect, and knowledge learned from specific categories transfers poorly to unseen classes.
  • Motivation: Few-shot segmentation seeks foreground regions of a novel category using only a few labeled examples.
  • Cross-reference network: CRNet concurrently predicts support and query masks, mining co-occurrent features through a symmetric cross-reference mechanism.The cross-reference module reinforces representations and supplies an auxiliary co-occurrence prediction loss during training.
  • Mask refinement: A mask refinement module recurrently updates cached confidence maps to improve foreground predictions after several refinement steps.The initial prediction identifies high-confidence seeds, whose probability map is cached for later predictions.
  • k-shot learning: For k-shot learning, CRNet finetunes parts of the network using labeled support-image pairs, benefiting from additional support images.The method uses at most k^2 image pairs and is evaluated in 1-shot, 5-shot, and 10-shot settings.
  • Results: Experiments on PASCAL VOC 2012 show that CRNet significantly outperforms baseline results and achieves state-of-the-art performance on 5-shot segmentation.

2. Related Work

Related work covers few-shot learning, semantic segmentation, few-shot segmentation, and image co-segmentation. Prior few-shot segmentation generally guides query prediction from support images, whereas co-segmentation jointly segments common objects in paired images.

  • Few-shot learning: Few-shot learning methods transfer models to new tasks with limited training data, using either testing-time finetuning or fixed metric-based embeddings.
  • Segmentation: Semantic segmentation classifies each image pixel, commonly using fully convolutional encoder-decoder networks with skip connections.
  • Few-shot segmentation: Few-shot segmentation extends few-shot classification to pixel-level prediction, with prior methods typically guiding query segmentation from labeled support images.Existing approaches include parameter regression, dense embedding comparison, prototype learning, and similarity guidance.
  • Image co-segmentation: Image co-segmentation jointly segments common objects in paired images using methods including graphical models, dense correspondences, clustering, Siamese networks, and channel attention.

3. Task Definition

Few-shot segmentation evaluates whether a model can identify a novel category’s foreground from few labeled support images, using episodically sampled support and query sets. The model predicts the query mask and learns with cross-entropy supervision.

  • Few-shot segmentation finds foreground pixels of a target category in test images using only a few pixel-level annotated images.
  • Training and testing use disjoint category sets, with labeled examples forming the support set and an unlabeled image forming the query set.
  • Episodes sample a target category and k + 1 labeled images containing it, assigning k images to support and one to query.
  • The model predicts the query mask from the support set and query image, then optimizes cross-entropy loss against the query ground-truth mask.

4. Method

CRNet uses a symmetric architecture in which support and query images guide each other through cross-reference features, category conditioning, and recurrent mask refinement. For k-shot learning, parts of the network are finetuned using support-image pairs.

  • Method overview: CRNet concurrently predicts masks for support and query images, allowing each image to guide segmentation of the other.
  • Method overview: The architecture comprises a Siamese encoder, cross-reference module, condition module, and mask refinement module.
  • Siamese encoder: The Siamese encoder uses shared convolutional networks to map support and query images into feature maps.
  • Cross-Reference Module: The cross-reference module compares support and query features to generate reinforced representations and supports prediction of co-occurrent objects.
  • Condition Module: The condition module fuses a category-relevant vector with reinforced feature maps to emphasize the assigned foreground category.
  • Mask Refinement Module: The mask refinement module caches confidence or probability maps and recurrently refines predictions from high-confidence seed regions toward fuller foreground masks.
  • Finetuning for K-Shot Learning: For k-shot learning, CRNet finetunes non-encoder modules using at most k2 support-image pairs, and the method consistently improves with more labeled support images.

5. Experiment

The experiments evaluate CRNet through four-fold cross-validation on PASCAL VOC 2012, using disjoint category folds and mean Intersection-over-Union on the held-out testing fold.

  • Dataset and Evaluation Metric: Experiments use PASCAL VOC 2012 cross-validation with 20 object categories divided evenly into four folds.
  • Dataset and Evaluation Metric: Each experiment trains on three category folds, tests on the remaining fold, and reports average performance over all four testing folds.
  • Dataset and Evaluation Metric: Performance is measured by mean Intersection-over-Union of classes in the testing fold.

6. Ablation study

The ablations show that cross-reference, multi-level features, multi-scale testing, and mask refinement each improve segmentation, while finetuning benefits from more support images. Comparisons also report state-of-the-art performance on PASCAL VOC 2012.

  • Component ablations: Removing either the condition or cross-reference module causes a significant performance drop, with cross-reference having the larger impact.The cross-reference module improves its counterpart without cross-reference by more than 10%.
  • Component ablations: 1.2 mIoU improvement comes from multi-scale input testing in the 1-shot setting.Support and query images are evaluated at multiple scales, and predictions are fused.
  • Component ablations: 1.8 mIoU improvement comes from using multi-level features rather than features solely from the backbone’s last layer.The result supports the usefulness of middle-level features for locating common objects across the two images.
  • Component ablations: 3.1 mIoU improvement comes from the mask refinement module over a baseline without its cached branch.The refinement module recurrently updates a confidence cache and is compared against a single-pass baseline.
  • k-shot experiments: 8.4 mIoU separates the finetuning method from the 1-shot baseline in the 5-shot setting, while fusion-based performance begins to drop with 10 supports.The finetuning method’s performance continues increasing as more support images become available.
  • Comparison with the State-of-the-Art Results: The method achieves state-of-the-art performance under both evaluation metrics in the 1-shot and 5-shot PASCAL VOC 2012 comparisons.The 5-shot comparison reports new state-of-the-art performance against previous methods.

7. Conclusion

The paper presents CRNet for few-shot segmentation, using mutual support-query prediction, cross-reference features, mask refinement, and k-shot finetuning. Experiments and ablations on PASCAL VOC 2012 validate the design and report state-of-the-art performance.

  • 7. Conclusion: CRNet jointly predicts masks for support and query images instead of unilaterally guiding query segmentation.Its cross-reference mechanism mines co-occurring features between the two images.
  • 7. Conclusion: The cached mask refinement module improves prediction performance through recurrent foreground refinement.The module uses confidence information from earlier predictions during later refinement steps.
  • 7. Conclusion: The finetuning-based k-shot method takes advantage of additional annotated support images and significantly improves performance.The conclusion attributes the k-shot gain to using more labeled support data.
  • 7. Conclusion: Ablation experiments validate the network design, and CRNet achieves state-of-the-art performance on PASCAL VOC 2012.The conclusion reports both component validation and the paper’s benchmark outcome.
Loading 2003.10658v1…