Source-linked AI summary
Semantic Instance Segmentation via Deep Metric Learning
Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, Kevin P. Murphy
TL;DR
Semantic instance segmentation requires masks that identify individual objects and their categories, but box-based and center-based methods face difficulties with multiple, elongated, or centerless objects. This paper learns pixel similarity with a deep embedding model, groups pixels around selected seed points, and obtains competitive Pascal VOC performance, including 62.21% mAP at 0.5 IoU.
Problem
Semantic instance segmentation must distinguish individual same-class objects while representing their shapes, and existing box- or center-based approaches have stated scope limitations.
Method
The method learns pixel embeddings and a seediness model, then groups pixels sufficiently similar to selected seed points into labeled instance masks.
Results
62.21% mAP at 0.5 IoU was obtained on the Pascal VOC 2012 instance segmentation benchmark, ranking fourth among the compared methods.
Takeaways & Limitations
The approach provides competitive preliminary Pascal VOC results while avoiding the need to identify object centers.
Abstract
from arXiv · showhide
We propose a new method for semantic instance segmentation, by first computing how likely two pixels are to belong to the same object, and then by grouping similar pixels together. Our similarity metric is based on a deep, fully convolutional embedding model. Our grouping method is based on selecting all points that are sufficiently similar to a set of "seed points", chosen from a deep, fully convolutional scoring model. We show competitive results on the Pascal VOC instance segmentation benchmark.
1. Introduction
Semantic instance segmentation must identify object instances and categories while producing masks that distinguish same-class objects. The paper replaces box- or center-based strategies with learned pixel similarity, seed selection, and grouping, achieving competitive Pascal VOC results.
- Problem: Semantic instance segmentation identifies object categories and individual instances using object-shaped masks rather than bounding boxes.Unlike semantic segmentation, it distinguishes separate objects of the same class, creating an unbounded instance label space.
- Existing limitations: Box-based methods can fail when multiple instances occupy one box, while box geometry poorly approximates elongated, articulated, or non-axis-aligned objects.These limitations motivate detecting masks directly and deriving boxes only when needed.
- Existing limitations: Center-based box-free methods predict centeredness, masks, and class labels but require each object to fit within the predicting unit’s receptive field.Elongated objects and categories without well-defined centers are particularly difficult for this approach.
- Proposed approach: The proposed method learns pixel-pair similarity with a deep embedding model and groups pixels likely to belong to the same object instance.The supervised object extent supplies a well-defined target, avoiding ambiguities such as splitting a person’s clothing into separate segments.
- Proposed approach: Seediness scores select representative seed pixels, whose embedding-space neighborhoods generate instance masks without computing all pixel-pair similarities.The seed search uses the top 100 seeds in practice to obtain good coverage of nearly all objects.
- Results: 62.21% mAP at 0.5 IoU places the method fourth on Pascal VOC 2012, behind three proposal-based methods but ahead of the previous fourth-place proposal-free result.The authors characterize the result as competitive rather than state of the art on this benchmark.
2. Related work
Prior instance-segmentation methods use box proposals, sliding-window or sequential mask prediction, watershed regions, or one-dimensional embeddings. This paper instead combines a different similarity loss, basin-based mask formation, and D-dimensional pixel embeddings.
- Proposal-based methods: Proposal-based methods predict object boxes before segmenting and classifying within each box.The approach can fail when one proposal contains multiple instances.
- Parallel mask prediction: Sliding-window methods predict masks and category likelihoods at image locations, often using image or feature pyramids for multiple object sizes.These methods operate in parallel across the image.
- Sequential mask prediction: Sequential methods use CNN features and an RNN to emit one binary instance mask per step while tracking already segmented locations.Their main drawbacks are speed and difficulty scaling to large numbers of objects.
- Region-based methods: Watershed-based methods partition predicted energy or boundary maps into candidate regions, sometimes followed by Multi-Cut grouping and semantic labeling.These techniques cannot group disconnected regions into one instance when occlusion separates an object.
- Embedding-based methods: Compared with a related one-dimensional embedding approach, this method changes the loss, forms masks from basins of attraction, and learns D-dimensional pixel embeddings.The cited comparison reports 62.21% mAP at 0.5 IoU for this method versus 35.1% for [19].
3. Method
The method learns pixel embeddings that separate object instances, then grows masks from selected seed pixels using embedding similarity and seed-quality predictions. A mask-classification head assigns labels and confidence while supporting multiple object scales.
- Architecture: The network adds embedding and mask-classification heads to a pretrained semantic-segmentation model for instance segmentation.The embedding head produces per-pixel vectors, while the classification head predicts the label and confidence of the mask generated by each seed.
- Embedding model: Pixels from the same instance are trained to be close in a 64-dimensional embedding space, while pixels from different objects and background are pushed apart.Training samples K points per instance and applies pairwise cross-entropy to same-instance versus different-instance labels.
- Creating masks: A seed pixel generates a mask by retaining every pixel whose embedding similarity exceeds threshold τ, with τ ∈ {0.25, 0.5, 0.75}.Varying τ allows the method to detect objects of different sizes, and tensor operations compute similarities efficiently for many seeds.
- Seed selection: The method learns a seediness heatmap that predicts whether growing a mask from each pixel will produce a good proposal.Seediness is derived from foreground mask-class probabilities and excludes the background class when selecting candidate seeds.
- Seed selection: Seeds are selected greedily using high seediness and embedding-space diversity, balancing proposal precision with coverage.The procedure encourages diversity in embedding space rather than suppressing nearby points in image coordinates, and the paper reports that the top 100 seeds provide good coverage of nearly all objects.
- Classification and seediness model: For each selected seed, the system chooses a threshold and class label, converts the similarity region into a mask, and attaches confidence and semantic class information.Separate classification models are trained for τ ∈ {0.25, 0.5, 0.75, 0.9} to handle different object sizes.
4. Results
The method is evaluated on PASCAL VOC 2012 using standard instance-segmentation metrics and compared across IoU thresholds, seed-sampling settings, and proposal counts. It is competitive overall, with particularly strong performance on large objects but poor bicycle performance.
- Experimental setup: The evaluation trains on PASCAL VOC 2012 with additional instance-mask annotations and evaluates on the validation set.
- Metrics: mAPr is computed by averaging per-class average precision across IoU thresholds of 0.5, 0.6, and 0.7.A predicted mask is a true positive when its IoU with an undetected true mask exceeds threshold β.
- Ablations: The experiments vary seed-sampling α, embedding size, and the number of sampled seed points, including class-agnostic average recall.The best embedding size was 64, and Table 3 reports performance for different numbers of mask proposals.
- Overall results: The method ranks 4th at 0.5 IoU, 2nd at 0.6 IoU, and tied 3rd at 0.7 IoU.The paper describes these results as competitive, if not state of the art.
- Per-class results: Performance is strong on large objects such as trains, dogs, and motorbikes but poor for bicycles.The paper attributes bicycle difficulty to coarse training masks versus test masks that separately segment wheel spokes.
5. Conclusion and future work
The paper presents a novel semantic instance-segmentation approach with promising preliminary PASCAL VOC results. Future work includes evaluation on COCO and Cityscapes and differentiable region growing for end-to-end training.
- Conclusion and future work: The authors report a novel approach with promising preliminary results on the PASCAL VOC dataset.
- Conclusion and future work: Future work will evaluate the method on COCO and Cityscapes and make region growing differentiable for end-to-end training.