PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Shot Instance Segmentation

Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, Alexander S. Ecker

arXiv:1811.11507v2cs.CV

TL;DR

One-shot instance segmentation asks whether a system can find and segment instances of a novel category in a complex scene from one reference image. Siamese Mask R-CNN combines a Siamese backbone with Mask R-CNN for this task; results show generalization to held-out categories, while targeting detection to the reference category remains difficult.

  • Problem

    Few-shot vision has focused on image classification, while detecting and segmenting novel categories in complex scenes from limited examples remains a challenge.

    Source: §1 · pp. 1–2

  • Method

    Siamese Mask R-CNN uses a Siamese approach with Mask R-CNN to detect and segment novel-category instances from a single reference image.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On held-out categories, one-shot detection mAP50 is 16.3% and segmentation performance is 14.5%, while category targeting is often harder than producing good boxes and masks.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The results provide a strong baseline for one-shot instance segmentation and show generalization to categories outside the training set.

    Source: §8 · p. 8

    Evaluation ensures that each reference category is present in its test image, simplifying the procedure relative to real-world use.

    Source: §3 · p. 3

Abstract

from arXiv · show

We tackle the problem of one-shot instance segmentation: Given an example image of a novel, previously unknown object category, find and segment all objects of this category within a complex scene. To address this challenging new task, we propose Siamese Mask R-CNN. It extends Mask R-CNN by a Siamese backbone encoding both reference image and scene, allowing it to target detection and segmentation towards the reference category. We demonstrate empirical results on MS Coco highlighting challenges of the one-shot setting: while transferring knowledge about instance segmentation to novel object categories works very well, targeting the detection network towards the reference category appears to be more difficult. Our work provides a first strong baseline for one-shot instance segmentation and will hopefully inspire further research into more powerful and flexible scene analysis algorithms. Code is available at: https://github.com/bethgelab/siamese-mask-rcnn

1 Introduction

The paper introduces one-shot instance segmentation: finding and segmenting objects of a novel category in complex scenes from a single visual example. Siamese Mask R-CNN adapts Mask R-CNN with a Siamese, metric-learning approach, and the paper identifies category-targeted detection as the main challenge.

Task and motivation

  • Few-shot methods have focused on image classification and are limited to simple object-centered images, so they cannot trivially handle detection.The paper motivates detection and segmentation for settings without large annotated datasets, including robotics and scientific imaging.Source: §1 · pp. 1–2
  • The task is to detect and segment arbitrary objects in complex real-world scenes based on a single visual example.The task is formalized as producing a bounding box and segmentation mask for every instance of the reference category.Source: §1 · p. 2

Proposed approach

  • Siamese Mask R-CNN combines metric learning with Mask R-CNN to learn a similarity metric that generalizes to previously unknown object categories.The model incorporates Siamese networks into Mask R-CNN.Source: §1 · p. 2

Contributions

  • The paper introduces the task and model, establishes an MS-COCO evaluation protocol, and reports that targeting detection to the reference category is the main challenge.For the model, segmenting objects that are correctly identified works well.Source: §1 · p. 2

2 Background

The background distinguishes fixed-category object detection and instance segmentation from few-shot learning on novel categories. It describes metric-learning and meta-learning approaches and the category split used to test generalization.

Object detection and instance segmentation

  • Object detection localizes and classifies individual objects from a fixed set of categories, using either direct predictions or class-agnostic proposals followed by classification.The passage describes convolutional neural networks as the feature-extracting backbone.Source: §2 · p. 2
  • Instance segmentation labels pixels for individual object instances and inherits object detection's difficulties, making it harder than semantic segmentation.Mask R-CNN extends Faster R-CNN to perform instance segmentation.Source: §2 · p. 2

Few-shot learning

  • Few-shot learning evaluates generalization to novel categories through episodes with support examples and query images; one example per category is one-shot learning.An n-way, k-shot episode has k examples from each of n categories.Source: §2 · p. 2
  • Few-shot methods either learn a metric for classifying novel-category examples or learn a strategy applied within each episode.Training and test categories are separated to evaluate generalization to novel categories.Source: §2 · pp. 2–3

3 One-shot object detection and instance segmentation on MS-COCO

The section defines an example-based task on MS-COCO: given an unannotated close-up reference of a novel category, find and segment its instances in a separate complex scene. It describes the category split and evaluation procedure, including simplifications and distinctions from related tasks.

Task setup

  • Given a close-up reference of a novel category, the system must find and segment its instances in a separate scene containing many objects.The reference is cropped to the object's bounding box and supplied without mask annotations.Source: §3 · p. 3

Category split

  • MS-COCO's 80 categories are split into 60 training and 20 test categories, with four different training/test splits.Each split assigns every fourth category to the test set, with the starting category varied across the four splits.Source: §3 · p. 3
  • Test-category objects may appear in training images, but they receive no annotations and are never used as references.The authors describe this as a feasible and realistic setup for an agent that may encounter unlabeled objects before receiving label information.Source: §3 · p. 3

Evaluation procedure

  • Evaluation selects a test image, samples references for novel categories present in it, predicts boxes separately per reference, and computes mAP50.For instance segmentation, predictions must provide masks rather than boxes; mAP50 is chosen to reflect finding novel objects from one reference.Source: §3 · p. 3
  • The evaluation is simplified because each reference category is guaranteed to be present in its test image.The passage notes that a real-world application would require removing this restriction.Source: §3 · pp. 3–4

Related tasks

  • The task differs from few-shot object detection by searching from an example without retraining and requiring instance masks, and from zero-shot detection by using an image rather than text.The paper focuses on the challenging MS-COCO dataset.Source: §3 · p. 4
  • Unlike one-shot semantic segmentation, the task requires instance-level masks that distinguish individual objects; related video and homogeneous-cluster tasks instead track or retrieve particular objects.The paper focuses on instances of the same object category rather than the same particular object.Source: §3 · p. 4

4 Siamese Mask R-CNN

Siamese Mask R-CNN conditions Mask R-CNN on a reference image by learning reference-to-scene similarity, allowing its heads to target matching objects, including novel categories without retraining.

4 Siamese Mask R-CNN

  • A learned similarity metric between the reference and scene regions supports proposals and match classification for novel categories without retraining.The approach generates proposals and classifies them as matches or non-matches.Source: §4 · p. 4
  • The model uses a shared-weight Siamese backbone to extract features from both the reference image and the scene.It uses ResNet50 with Feature Pyramid Networks.Source: §4 · p. 5, Figure 2 · p. 5
  • Matching averages reference features into an embedding, compares it with each scene location, and concatenates the absolute differences to scene features.A 1 × 1 convolution then reduces the feature count; multiple resolutions are matched independently.Source: §4 · p. 5
  • The model reuses Mask R-CNN’s proposal and ROI operations, but replaces category classification with binary match discrimination and predicts class-agnostic boxes and a single mask.Similarity-conditioned features also inform the proposal, classification, and mask heads.Source: §4 · p. 5
  • Figure 4 shows successful novel-category examples in its top two rows and results with many false positives in its last row.Each example uses one reference image as the category information.Source: Figure 4 · p. 6

5 Experiments

Experiments train Siamese Mask R-CNN episodically on MS-COCO and evaluate across category splits, with repeated evaluations to account for random reference selection.

5 Experiments

  • Models are trained jointly for example-based object detection and instance segmentation, then evaluated on known and held-out categories using the MS-COCO validation set.A separate model is trained for each of four category splits.Source: §5 · p. 6
  • Backbone pretraining uses ImageNet categories without MS-COCO correspondences to avoid using test-category label information.Each training image is paired with a randomly selected reference category present in the image and a random instance from another image; other objects are treated as background.Source: §5 · p. 6
  • Evaluation scores are averaged across four category splits and five stochastic runs, with 95% confidence intervals reported.References are selected randomly, making the evaluation stochastic.Source: §5 · p. 6
  • Table 1 reports MS-COCO mAP50 for training categories, novel categories, and random boxes, using one or five references.The random-box baseline shifts ground-truth boxes while retaining ground-truth masks and certain annotation statistics.Source: §5 · p. 6, Table 1 · p. 7

6 Results

Siamese Mask R-CNN generalizes to held-out categories above chance, though performance remains below that on training categories. Category targeting and cluttered scenes remain challenges.

Example-based detection and segmentation

  • For training categories, one reference yields 37.6% detection and 34.9% segmentation mAP50; five references yield 41.3% and 38.4%, respectively.Across five evaluation runs, variation was below 0.2 percentage points in all cases.Source: §6 · p. 7

One-shot instance segmentation

  • On held-out categories, one-shot detection reaches 16.3% mAP50 and segmentation 14.5%; five-shot scores rise to 18.5% and 16.7%, respectively.The one-shot scores are far above chance (1.2% detection, 0.5% segmentation), but the authors report substantial overfitting to training categories.Source: §6 · p. 7

Qualitative analysis

  • Successful examples include markedly different or partial references, and the model uses its learned metric rather than reference masks to match categories.In the bus and horse examples, it finds and segments complete query objects despite incomplete references and no training masks or boxes for those categories.Source: §6 · p. 7

False positives when evaluating on novel categories

  • Failures often involve boxes and masks aimed at the wrong category; novel-category searches frequently trigger false detections of overrepresented training classes.Common false detections include people, cars, airplanes, and clocks.Source: §6 · pp. 7–8

Effect of image clutter

  • Both detection and segmentation scores are substantially higher in images with fewer instances, highlighting the challenge of clutter.Figure 5 reports results on split S2 by the number of instances per image, in mAP50.Source: Figure 5 · p. 7, §6 · p. 8

7 Related work

The work combines few-shot metric learning, object detection, and instance segmentation. Related approaches address few-shot detection with varied training and feature-matching strategies, but the authors report no prior category-based few-shot instance segmentation.

Related work

  • The authors locate their novelty in combining few-shot metric learning, object detection or visual search, and instance segmentation.They report being unaware of previous work on category-based few-shot instance segmentation.Source: §7 · p. 8
  • Prior few-shot detectors include semi-supervised label generation [13], transfer regularization [7], category-specific feature reweighting [24], and prototype-based metric learning.Concurrent work [66] evaluates few-shot object detection on Pascal VOC with separate feature fusions in the RPN and classifier head.Source: §7 · p. 8
  • Zero-shot detection methods [3] [42] [11] [69] target novel categories through a joint embedding of query images and textual descriptions.The authors contrast text-based descriptions with their use of visual descriptions.Source: §7 · p. 8

8 Discussion

The paper introduces one-shot instance segmentation and proposes Siamese Mask R-CNN, which detects and segments novel categories from a single reference image. Its results are above chance but weaker on novel than training categories, establishing a baseline for this difficult setting.

Discussion

  • Siamese Mask R-CNN combines Mask R-CNN with metric learning to detect and segment novel categories from one reference image, performing above chance but less successfully than on trained categories.The authors present its performance as a strong baseline given the task's difficulty.Source: §8 · p. 8

Changes to previous version

This version adds evaluation and training details, five-shot results, and discussion of concurrent work, while revising substantial portions of the text and figures.

Changes to previous version

  • The updated evaluation runs each split five times and reports means with 95% confidence intervals, and it adds five-shot results using a prototypical approach.The revision also adds background, concurrent-work discussion, and appendix details on training, evaluation, and leaderboard metrics.Source: Changes to previous version · p. 13
  • The authors reworked substantial parts of the text to clarify the task setup, related tasks, and corresponding solutions, and revised figures for qualitative examples, task comparisons, and model color coding.The qualitative figure combines good and bad examples.Source: Changes to previous version · p. 13

A1 Training and testing categories

This section introduces the appendix’s description of the training and testing categories.

A1 Training and testing categories

A1.1 Splits S1-S4

The appendix details the category splits, Siamese Mask R-CNN architecture, training, and evaluation, then reports split-specific results and additional analyses. Its evaluation finds variation across splits, with the largest train–test gap in S1, which the authors associate with the over-representation of people in MS-COCO.

A1.1 Splits S1-S4; A1.2 Rationale

  • Four splits hold out every fourth MS-COCO category, rotating the starting category; this tests all categories while distributing held-out supercategories as evenly as possible [56] [33].The rationale assumes knowledge from broad object categories can help infer missing categories, unlike tieredImageNet’s generalization to vastly different categories [48].Source: A1.1 Splits S1-S4 · p. 13, A1.2 Rationale · p. 13

A2.1 Backbone; A2.2 Feature matching

  • The model uses an unmodified ResNet-50 backbone and C2–C5 features to form multi-scale representations and scene–reference similarity features.The feature representation concatenates scene features with pixel-wise differences and applies a 1 × 1 convolution to produce 384 features.Source: A2.1 Backbone · p. 13, A2.2 Feature matching · pp. 13–14

A2.3 Region Proposal Network (RPN); A2.4 Classification and bounding box regression head; A2.5 Segmentation head

A3 Training details

  • Backbone pre-training uses ImageNet-687, a reduced set without categories corresponding to MS-COCO, so test-category labels are not used during training.Training then proceeds end-to-end after a 1,000-step heads-only warm-up; the multi-task losses use binary match/non-match classification and task-specific weights.Source: A3 Training details · p. 15

A4 Evaluation details; A4.1 Category selection; A4.2 Evaluation procedure; A4.3 Noise induced by random reference sampling; A4.4 Comment on the evaluation procedure; A4.6 Choice of evaluation metric

  • Evaluation uses 60 training and 20 held-out categories per split, samples a reference for each present category in each image, and averages five runs because random-reference results vary.Evaluating only categories present in an image makes the task easier and penalizes false positives less than evaluating all categories; the authors nevertheless consider it sufficiently difficult. They use mAP50, noting correct masking is less problematic than classification.Source: A4.1 Category selection · p. 16, A4.2 Evaluation procedure · p. 16, A4.3 Noise induced by random reference… · p. 16

A5 Confusion matrix

  • For S2, the confusion matrix measures AP50 for detections from category i evaluated as category j; its column sums and histogram identify commonly false-predicted categories.People, cars, airplanes, and clocks are among the categories commonly falsely predicted.Source: A5 Confusion matrix · p. 17, Figure A1 · p. 18

A6 Additional results; A6.1 Noisiness of evaluation; A6.3 Full MS-COCO style results

A6.2 Results for each split

  • Split S1 shows the largest train–test performance gap; the authors attribute it to the over-representation of people in MS-COCO.They say removing people makes the training set considerably easier, while detecting them at test time is hard.Source: A6.2 Results for each split · p. 17
Loading 1811.11507v2…