PaperlayerView exact revision on arXiv

Source-linked AI summary

FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation

Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, Chi-Keung Tang

arXiv:1907.12347v2cs.CV

TL;DR

Existing vision datasets cover limited and uneven object categories, while few-shot segmentation lacks a large-scale benchmark with dense annotations. The paper introduces FSS-1000 and adapts a relation network for five-shot segmentation. Trained from scratch, the model achieves higher accuracy than previous methods, including on test sets unseen by FSS-1000.

  • Problem

    Few-shot segmentation lacks a large-scale dataset, while existing datasets have biased class coverage and require labor-intensive pixelwise annotation.

  • Method

    The paper builds FSS-1000 and adapts a relation network to predict per-pixel segmentation from a labeled support set and unlabeled query images.

  • Results

    The baseline trained exclusively on FSS-1000 without pretrained weights achieves higher accuracy than previous methods, including on test sets unseen by FSS-1000.

  • Takeaways & Limitations

    FSS-1000 emphasizes class diversity and can be extended to new object classes with few annotated examples.

  • Takeaways & Limitations

    The baseline is not robust to scale variance, occlusion, and background noise, and support-query differences in scale or pose can degrade segmentation.

Abstract

from arXiv · show

Over the past few years, we have witnessed the success of deep learning in image recognition thanks to the availability of large-scale human-annotated datasets such as PASCAL VOC, ImageNet, and COCO. Although these datasets have covered a wide range of object categories, there are still a significant number of objects that are not included. Can we perform the same task without a lot of human annotations? In this paper, we are interested in few-shot object segmentation where the number of annotated training examples are limited to 5 only. To evaluate and validate the performance of our approach, we have built a few-shot segmentation dataset, FSS-1000, which consists of 1000 object classes with pixelwise annotation of ground-truth segmentation. Unique in FSS-1000, our dataset contains significant number of objects that have never been seen or annotated in previous datasets, such as tiny daily objects, merchandise, cartoon characters, logos, etc. We build our baseline model using standard backbone networks such as VGG-16, ResNet-101, and Inception. To our surprise, we found that training our model from scratch using FSS-1000 achieves comparable and even better results than training with weights pre-trained by ImageNet which is more than 100 times larger than FSS-1000. Both our approach and dataset are simple, effective, and easily extensible to learn segmentation of new object classes given very few annotated training examples. Dataset is available at https://github.com/HKUSTCV/FSS-1000.

1. Introduction

Existing datasets cover relatively few object categories, often with uneven class distributions and substantial annotation costs. FSS-1000 addresses this gap with a scalable few-shot segmentation dataset and a baseline that performs strongly without pretraining.

  • Motivation: Existing datasets cover less than 1% of an estimated 500,000–700,000 visual object categories.ImageNet class sizes also range from 1 to 3,047 images, introducing uneven distribution.
  • Motivation: Extending object categories is costly because segmentation requires labor-intensive pixelwise labeling and existing datasets contain limited segmentation classes.PASCAL VOC and COCO contain only 20 and 80 classes with pixelwise segmentation information, respectively.
  • Contribution: FSS-1000 is a large-scale few-shot segmentation dataset emphasizing object-class count over image count and requiring only 10 annotated images to extend a class.The dataset provides a built-in object-category hierarchy and ground-truth segmentation.
  • Contribution: The baseline adapts a relation network with a decoder and, trained from scratch on FSS-1000, outperforms models fine-tuned from ImageNet/COCO weights.The paper states that FSS-1000 is less than 1% the size of contemporary large-scale datasets.

2. Related Work

Prior work provides large recognition and segmentation datasets or few-shot classification datasets, but not a large-scale dataset combining few-shot learning with dense pixelwise segmentation. FSS-1000 instead broadens segmentation class diversity and adapts metric-based few-shot learning to segmentation.

  • Dataset landscape: Existing large-scale datasets support recognition or segmentation, while recent few-shot datasets generally lack dense pixelwise segmentation labels.FSS-1000 contains 1,000 classes and 10,000 images with pixelwise segmentation labels.
  • FSS-1000: FSS-1000 prioritizes broadening segmentation class coverage rather than increasing dataset size.Each of its 1,000 classes has 10 images with binary segmentation annotation.
  • Few-shot segmentation: The relation network is extended because it provides a simple, general framework for learning a metric between few-shot samples and queries.The modification changes the output from class probabilities to pixelwise segmentation differences.
  • Semantic segmentation: Modern high-accuracy segmentation architectures are difficult to adapt to segment new classes from few training examples.The paper follows fully convolutional segmentation principles by decoding foreground features into dense predictions.

3. FSS-1000

FSS-1000 was designed as a diverse, hierarchical, and extensible dataset for general-object few-shot segmentation. Its annotations support binary and, for many classes, instance-level segmentation.

  • Dataset scope: FSS-1000 targets general-object few-shot segmentation rather than a restricted subdomain.Its comparison excludes datasets focused only on handwritten characters, faces, or road scenes.
  • Object classes: The dataset combines 584 ILSVRC-overlapping classes with 486 new classes, adding daily, artificial, and manmade objects to reduce animal-centered bias.The 1,000 classes are arranged in a hierarchy.
  • Data collection: Raw images were collected by querying object keywords across Google, Bing, and Yahoo, with aspect-ratio and resolution filtering before annotation.Images with extreme aspect ratios or dimensions below 224 pixels were discarded because all images were resized to 224×224.
  • Scalability: FSS-1000 can be extended with 10 images carrying pixelwise binary segmentation labels for each new class.Binary annotation avoids requiring old classes to be reannotated when adding new ones.
  • Hierarchy: The dataset uses a three-level hierarchy with 12 super-categories and 1,000 leaf classes, allowing a class to belong to multiple superclasses.For example, an apple can belong to both fruit and food.
  • Instance annotation: Instance-level segmentation labels are available for 758 of the 1,000 classes, with at most 10 instances annotated per image.The dataset annotates one object type per image even when other categories appear in the background.

4. Methodology

The method formulates few-shot segmentation as category-disjoint support-query learning and uses an encoder–relation–decoder pipeline to predict pixelwise query masks. The baseline supports standard backbones, averages multiple support features, and trains with binary cross entropy.

  • Problem Formulation: Few-shot segmentation splits inputs into annotated support samples and unlabeled query samples, with testing categories unseen during training.
  • Problem Formulation: Under the paper’s binary setting, the model predicts a pixelwise classification map for each query image using five annotated support examples.
  • Network Architecture: The pipeline encodes support and query images, combines their feature maps in a relation module, and decodes coarse results into full-resolution prediction maps.
  • Network Architecture: For K-shot forwarding, support feature maps are averaged element-wise over depth channels so their output depth remains independent of support-set size.
  • Training Objective: Binary cross entropy trains the network, while experiments report similar performance for BCE and mean square error under this network setting.
  • Network Module Details: The architecture can use VGG-16, ResNet, or Inception encoders, while the relation module concatenates support and query features before processing them with convolutional layers.

5. Experiments

Experiments evaluate FSS-1000 across benchmark comparisons, pre-training choices, support-set size and composition, generalization to unseen classes, and iterative refinement. Results show strong cross-dataset performance, benefits from varied support examples, and remaining sensitivity to scale, pose, occlusion, and background noise.

  • Experimental setup: FSS-1000-trained models are evaluated against benchmarks and across backbones, datasets, pre-training stages, and few-shot support settings.The experiments use IoU for binary segmentation and compare models trained on FSS-1000, fsPASCAL, fsCOCO, and other pre-training configurations.
  • Benchmark comparison: On PASCAL-5i, the baseline marginally outperforms OSLSM and Guided Network, while training only on FSS-1000 achieves much better results than PASCAL-5i-trained models.The FSS-1000-only model also exceeds the reported state-of-the-art performance of recent methods.
  • Effect of pre-training: 8% and 5.8%: Model II trained on FSS-1000 outperforms the fsCOCO-trained Model I on the two test sets.The authors attribute this to FSS-1000 containing COCO training classes with more variety.
  • Effect of pre-training: Model IV, pre-trained on FSS-1000 and fine-tuned on fsCOCO, achieves the best result on both test sets.It outperforms both the exclusively FSS-1000-trained Model III and the ImageNet-pre-trained, fsCOCO-fine-tuned Model I.
  • Effect of support set: More support images generally improve accuracy, but gains become negligible beyond five images; varied support examples are especially important for deformable objects.Support images provide variation in object color, pose, and scale, while rigid objects may be adequately represented by one image.
  • Unseen classes and limitations: FSS-1000-trained models generalize to novel unseen classes and can auto-label them from a few support examples, although scale variation, occlusion, and background noise remain weaknesses.The experiments include novel objects and applications such as satellite-image rivers, cells, and animals; large or small sunflower instances may be incomplete or omitted.
  • Iterative few-shot segmentation: Correcting and adding failure cases to the support set improves subsequent segmentation, as shown in iterative Eiffel Tower experiments.The initial support set lacked sufficient view and scale variation, while the revised set enabled correct segmentation of previous hard cases.

6. Conclusion

The paper addresses the lack of large-scale, scalable few-shot segmentation resources by introducing FSS-1000 and adapting relation networks for segmentation. Its model, trained without pre-trained weights, performs well on unseen classes and supports several related few-shot tasks.

  • FSS-1000 addresses the absence of a large-scale few-shot segmentation dataset by emphasizing class diversity rather than dataset size.
  • The authors adapt the relation network architecture to few-shot segmentation.
  • The baseline model trained exclusively on FSS-1000 without pre-trained weights achieves higher accuracy than previous methods, including on test sets unseen by FSS-1000.
  • FSS-1000 supports large-scale segmentation on totally unseen classes without retraining or fine-tuning.
  • The paper also demonstrates promise for few-shot instance segmentation and iterative few-shot recognition tasks.
Loading 1907.12347v2…