PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Meta-class Memory for Few-Shot Semantic Segmentation

Zhonghua Wu, Xiangxi Shi, Guosheng lin, Jianfei Cai

arXiv:2108.02958v3cs.CV

TL;DR

Few-shot segmentation methods treat classes independently despite shareable middle-level information, while fully supervised segmentation requires extensive pixel-level annotation. MM-Net learns transferable meta-class embeddings and weights support features by image quality, reporting state-of-the-art results on both benchmarks and a 5.1% mean mIoU gain on COCO.

  • Problem

    Few-shot segmentation methods treat classes independently despite shareable middle-level information, while fully supervised segmentation requires extensive pixel-level annotation.

    Source: §1 · pp. 1–2

  • Method

    MM-Net learns meta-class embeddings during base-class training for transfer to novel classes and uses support-image quality measures to weight feature fusion.

    Source: §1 · p. 2

  • Results

    MM-Net performs best in all settings on PASCAL-5i and COCO, with a 5.1% mean mIoU gain on COCO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors frame MM-Net’s memory embeddings as a way to transfer common knowledge between base and novel classes.

    Source: §5 · p. 9

    MM-Net uses 2466 MiB GPU memory at 17 FPS, versus PFENet’s 1920 MiB and 42 FPS, with the added cost attributed to its Meta-class Memory Module.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Currently, the state-of-the-art methods treat few-shot semantic segmentation task as a conditional foreground-background segmentation problem, assuming each class is independent. In this paper, we introduce the concept of meta-class, which is the meta information (e.g. certain middle-level features) shareable among all classes. To explicitly learn meta-class representations in few-shot segmentation task, we propose a novel Meta-class Memory based few-shot segmentation method (MM-Net), where we introduce a set of learnable memory embeddings to memorize the meta-class information during the base class training and transfer to novel classes during the inference stage. Moreover, for the $k$-shot scenario, we propose a novel image quality measurement module to select images from the set of support images. A high-quality class prototype could be obtained with the weighted sum of support image features based on the quality measure. Experiments on both PASCAL-$5^i$ and COCO dataset shows that our proposed method is able to achieve state-of-the-art results in both 1-shot and 5-shot settings. Particularly, our proposed MM-Net achieves 37.5\% mIoU on the COCO dataset in 1-shot setting, which is 5.1\% higher than the previous state-of-the-art.

1. Introduction

Few-shot segmentation seeks to reduce the annotation burden for novel classes, but existing methods treat support-query segmentation as class-agnostic. MM-Net instead learns shareable meta-class representations and quality-weights support features; the paper reports state-of-the-art results across two datasets and shot settings.

1. Introduction

  • Few-shot segmentation predicts a novel-class query mask from only a few labeled support images, reducing the need for extensive novel-class labeling.Conventional segmentation methods require abundant pixel-level annotations.Source: §1 · p. 1
  • Existing methods use class-agnostic foreground-background comparisons, while MM-Net targets middle-level attributes shareable across classes.The paper calls this shared information meta-class information.Source: §1 · p. 2
  • MM-Net learns meta-class memory embeddings during base-class training, then uses them to form support and query activation maps for novel-class transfer.The resulting maps interact to propagate support-mask information into the query representation used for mask prediction.Source: §1 · p. 2
  • For k-shot settings, QMM measures each support image's quality and weights its features to form a class prototype.This gives greater weight to higher-quality support samples than an unweighted average.Source: §1 · p. 2
  • 5.1% mean mIoU gain over SOTA on COCO; MM-Net also achieves SOTA on PASCAL-5i and COCO in 1-shot and 5-shot settings.The experiments evaluate four splits on each dataset.Source: §1 · p. 2

2. Related Works

The related work contrasts fully supervised segmentation, few-shot support-query methods, and broader meta-learning approaches. MM-Net's distinction is a learnable memory of shareable middle-level features rather than a meta-learner.

2.1. Semantic Segmentation

  • Fully supervised semantic segmentation classifies each pixel but typically needs abundant pixel-level annotations and does not generalize to novel classes from only a few labels.The overview also places MM-Net's MMM, APM, and FCM in a pipeline that produces query masks.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2. Few-Shot Semantic Segmentation

  • Few-shot segmentation predicts dense masks for new-class queries from a few labeled supports; prior work uses comparison, prototype alignment, graph attention, or multi-scale decoding.MM-Net differs by introducing shareable representations across base and novel classes.Source: §2.2 · p. 3

2.3. Meta Learning

  • Meta-learning seeks to transfer knowledge or support faster adaptation, including through meta-learners, parameter initialization, metric learning, and external memory.MM-Net instead builds a meta-class memory to capture representative middle-level features for transfer between base and novel classes.Source: §2.3 · p. 3

3. Method

MM-Net combines learnable meta-class memory with modules that propagate support-mask information, add high-level foreground confidence, and weight support images in k-shot settings. Its training losses supervise segmentation and encourage the memory embeddings to remain diverse.

3. Method

  • MM-Net combines meta-class memory, activation propagation, foreground confidence, and feature enrichment, with quality weighting added for k-shot inputs.APM transfers support-mask information to query activation maps, while FCM adds high-level query-support interactions.Source: §3 · p. 4

3.1. Meta-class Memory Module

  • Learnable meta-class embeddings encode middle-level features into activation maps for support and query images, with 50 embeddings of dimension 256.The embeddings are learned by back-propagation from concatenated second- and third-level ResNet features.Source: §3.1 · p. 4

3.2. Activation Propagation Module

  • APM compares query and support activation-map nodes, then uses support-mask-filtered similarities to propagate foreground evidence to query nodes.The method applies cosine similarity between node pairs and uses the resulting weights to fuse support-node features with query features.Source: §3.2 · p. 4

3.3. Foreground Confidence Module

  • FCM adds a high-level foreground confidence map based on interactions between query and support features from the fourth ResNet level.For each query feature node, it takes the maximum similarity to support nodes, reshapes the values into a map, and applies min-max normalization.Source: §3.3 · p. 5

3.4. Quality Measurement Module

  • QMM assigns support-image quality weights and fuses the support activation maps with a weighted average instead of a simple mean.The quality maps are normalized across support images with softmax before weighting and summing their activation maps.Source: §3.4 · pp. 5–6

3.5. Training Loss

  • Training uses segmentation losses for final and intermediate masks plus a reconstruction loss that encourages distinct meta-class embeddings.The reconstruction loss maximizes the log-likelihood of the correlation matrix's diagonal elements; the loss trade-off parameters are 1, 1, and 0.1.Source: §3.5 · p. 6

4. Experiments

MM-Net achieves the best reported results across the evaluated PASCAL-5i and COCO settings, including a 5.1% advantage over PFENet in 1-shot COCO. Its ablations evaluate the memory embeddings, feature levels, propagation, reconstruction loss, and support-image quality module.

4.1. Implementation details

  • Experiments use four splits and 1-shot and 5-shot settings on PASCAL-5i and COCO, with ImageNet-pretrained backbones fixed during training.The backbones are VGG-16, ResNet-50, and ResNet-50-v2; evaluation uses class mean intersection over union.Source: §4.1 · p. 6

4.2. Comparisons with state-of-the-art

  • MM-Net achieves the best results under all reported training conditions, and its 1-shot COCO results outperform PFENet by 5.1%.The comparisons include different image sizes and backbones; the COCO comparison uses ResNet-50-v2 and 473 × 473 for MM-Net due to GPU memory restrictions.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.3. Ablation studies

  • Fifty meta-class embeddings yield the best performance among the tested counts and are used in subsequent experiments.The authors report that 50 meta-classes learn more meaningful and effective features.Source: §4.3 · p. 7, Table 3 · p. 7
  • FCM+MMM improves mean mIoU by 2.9% over FCM+Feat, which directly uses middle-level image features for mask prediction.The comparison tests the Meta-class Memory Module against direct middle-level feature decoding.Source: Table 4 · p. 7, §4.3 · p. 8
  • Fusing the 2nd- and 3rd-level features yields the best memory-learning performance among the tested feature choices.The authors conjecture that combining edge information from the 2nd level with part and object information from the 3rd helps learn the embeddings.Source: Table 5 · p. 7, §4.3 · p. 8
  • The Activation Propagation Module improves mIoU by 0.6% over global foreground pooling and element-wise propagation.The alternative propagates support-mask information from support activation maps to query activation maps.Source: §4.3 · p. 8
  • The memory reconstruction loss improves results, with applying it to support features giving the best overall performance among the tested options.The ablation compares applying the loss to query features, support features, both, or neither.Source: Table 6 · p. 7, §4.3 · p. 8
  • The Quality Measurement Module improves mIoU by 0.8% over averaging activation maps from five support images.The reported comparison is under the 5-shot PASCAL-5i setting.Source: Table 7 · p. 7, §4.3 · p. 8

5. Conclusion

MM-Net combines memory-based meta-class embeddings with modules for activation propagation, foreground confidence, and support-image quality measurement. The authors report that it improves state-of-the-art results on PASCAL-5i and COCO.

5. Conclusion

  • MM-Net combines the Meta-class Memory Module, Activation Propagation Module, Foreground Confidence Module, and Quality Measurement Module.The authors identify the learnable meta-class embeddings and support-image quality measurement as the method's novelties.Source: §5 · pp. 8–9
  • Learnable meta-class embeddings are intended to transfer common knowledge from base classes to novel classes.Source: §5 · p. 9
  • The authors report that MM-Net significantly improves state-of-the-art results on both PASCAL-5i and COCO.Source: §5 · p. 9

A. Effect of Foreground Confidence Module.

The Foreground Confidence Module improves mIoU in the reported PASCAL-5i 1-shot ablation. The comparison is against passing fused meta-class activation maps directly to the mask-generation module.

A. Effect of Foreground Confidence Module.

B. Comparison between Quality Measurement Module and the attention mechanism.

On 5-shot PASCAL-5i, the reported QMM result exceeds the result with CANet's attention mechanism. The paper describes the attention alternative's feature-fusion procedure.

B. Comparison between Quality Measurement Module and the attention mechanism.

C. Visualization results for ablation studies “Features for memory learning”

Memory learned from second-level features activates on local information such as edges, while third-level features activate on whole objects or parts. Combining the levels produces stronger meta-class memory activation on meta-class regions.

C. Visualization results for ablation studies “Features for memory learning”

Loading 2108.02958v3…