Source-linked AI summary
One-Shot Learning for Semantic Segmentation
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, Byron Boots
TL;DR
One-shot semantic segmentation asks how to predict pixel-level masks for new classes from sparse annotated examples rather than many segmented training images. The paper uses a two-branched network that generates parameters for a segmentation model from a support set and applies them to a query image. On a PASCAL VOC 2012 benchmark with unseen test classes, the architecture improves over baselines and is considerably faster.
Problem
One-shot semantic segmentation seeks pixel-level masks for a new class from a single annotated image, with test classes absent from the training classes.
Method
A two-branched network converts a labeled support image into parameters for a learned segmentation model, which predicts a mask on a query image.
Results
The method significantly improves meanIoU over baselines on unseen classes in the PASCAL VOC 2012 benchmark and is considerably faster than other baselines.
Takeaways & Limitations
The architecture supports one-shot segmentation without weak labels for new classes while remaining fast and using less memory than other baselines.
Takeaways & Limitations
The benchmark contains only 20 PASCAL VOC classes, although the authors report that simulated one-shot training performs well despite this limited class count.
Abstract
from arXiv · showhide
Low-shot learning methods for image classification support learning from sparse data. We extend these techniques to support dense semantic image segmentation. Specifically, we train a network that, given a small set of annotated images, produces parameters for a Fully Convolutional Network (FCN). We use this FCN to perform dense pixel-level prediction on a test image for the new semantic class. Our architecture shows a 25% relative meanIoU improvement compared to the best baseline methods for one-shot segmentation on unseen classes in the PASCAL VOC 2012 dataset and is at least 3 times faster.
1 Introduction
The paper introduces one-shot semantic segmentation, where a model predicts a new class's pixel-level mask from one annotated image. Its two-branched architecture generates segmentation parameters in one forward pass and improves benchmark performance while remaining fast and scalable.
- Problem: One-shot semantic segmentation predicts a new class's pixel-level mask from a single image and its corresponding annotation.The annotated image-label pair is called the support set; k-shot learning generalizes this to k such pairs.
- Approach: The proposed two-branched model maps a labeled support image to parameters, then combines them with a query image to produce its segmentation mask.The first branch produces parameters, while the second branch uses them to generate the mask.
- Advantages: A single forward pass computes segmentation parameters, avoiding the many SGD iterations required by fine-tuning approaches.The authors also state that the approach is fully differentiable and jointly trainable with the segmentation branch.
- Advantages: The generated parameter vector is independent of image size, so the approach avoids the scaling problem caused by dense feature computation.This is contrasted with direct extensions of one-shot classification methods, which do not scale well to thousands of dense features.
- Evaluation: The PASCAL VOC 2012 benchmark tests on classes absent from training and reports significant improvements over baselines using meanIoU.The benchmark is designed for one-shot semantic segmentation.
- Extension: The paper extends the method to k-shot learning by independently segmenting each support image and aggregating masks with pixel-level logical OR.This requires no retraining for different support-set sizes and is reported to increase meanIoU as images are added.
2 Related Work
The related work situates the method between fully convolutional semantic segmentation and low-shot learning. It emphasizes that dense pixel classification creates scalability challenges absent from image-level classification.
- Semantic segmentation: Semantic image segmentation classifies every pixel into a predefined category, with CNN methods either classifying super-pixels or pixels directly.The paper bases its approach on Fully Convolutional Networks for efficient pixel-wise classification.
- Low-shot setting: Unlike standard FCN-based segmentation, the paper does not assume many annotated training examples for the test classes.This distinguishes the proposed setting from conventional semantic segmentation.
- Weak supervision: Weak and semi-supervised segmentation reduces pixel-level annotation requirements by using coarse labels such as bounding boxes or image labels.Co-segmentation is identified as a related task involving objects from the same semantic class.
- Metric learning: Metric learning addresses low-shot learning by learning embeddings that bring same-category objects together and separate different-category objects.The paper also draws ideas from related architectures for adapting dense classification.
3 Problem Setup
The task learns a model from image-mask pairs for training classes and evaluates it on query images containing unseen semantic classes. Given a support set and query image, the model predicts a binary mask for the target class.
- Task definition: The support set consists of a small collection of image-binary mask pairs for a semantic class.Each mask identifies the target class in its corresponding image.
- Task definition: The learned model takes a support set and query image as input and predicts a binary mask for the target semantic class.The problem illustration specializes this formulation to k = 1.
- Train/test split: Training uses a large set of image-mask pairs, while testing uses query annotations for classes disjoint from the training classes.This disjointness is the key difference from typical segmentation, where training and testing classes overlap.
- Annotation protocol: Images containing test-class objects may still enter training when they also contain a training-class object, but annotations for test-class objects are excluded.The procedure treats unannotated test-class objects as background during training.
4 Proposed Method
The method uses a two-branch network: one branch converts a labeled support image into classifier parameters, while the other extracts query-image features for pixel-level prediction. Training simulates one-shot episodes, and k-shot inference combines classifiers generated from multiple support images.
- Architecture: The conditioning branch maps a labeled support image to parameters {w,b} for a pixel-level logistic classifier.The segmentation branch extracts dense query-image features, which the classifier converts into a predicted mask.
- Architecture: The predicted mask is upsampled to the original image size and thresholded at 0.5 to produce a binary mask.
- Architecture: Masking isolates the target object in the support image, while weight hashing maps VGG features to classifier parameters without a massive fully connected layer.The masking choice addresses responses biased toward the largest object and increased variance from background information.
- Architecture: The embedding branch uses an FCN-32s architecture through conv-fc7, while a dilated-FCN provides a higher-resolution alternative.The conv-fc7 feature volume has 4096 channels; the dilated-FCN uses stride 8.
- Training: Training samples a support image, query image, class, and binary mask from the training set, then predicts the query mask in a forward pass.The objective maximizes the ground-truth mask’s log likelihood; training uses SGD for 60k iterations.
- k-shot inference: In k-shot segmentation, each support image produces an independent classifier, and a pixel is included if any classifier labels it as the object.This union combines classifiers described as having high precision but low recall.
5 Dataset and Metric
The PASCAL-5i benchmark partitions PASCAL VOC semantic classes into training and held-out test label sets for k-shot segmentation. Performance is measured with meanIoU, the average per-class Intersection over Union.
- Dataset: PASCAL-5i samples five of PASCAL VOC’s twenty classes as the test label set and uses the remaining fifteen for training.The benchmark is constructed from PASCAL VOC 2012 and extended SDS annotations.
- Dataset: The benchmark excludes training images that overlap with the PASCAL VOC 2012 validation set.
- Evaluation: Table 1 reports meanIoU across all folds for both 1-shot and 5-shot semantic segmentation tasks.The table’s top section lists test classes for each PASCAL-5i fold.
- Dataset: Training and test image-mask pairs are selected according to whether they contain pixels from their respective label sets, with other classes mapped to background.
- Evaluation: MeanIoU averages per-class IoU, where IoU_l = tp_l/(tp_l+fp_l+fn_l) over the predicted binary masks.This is the standard mean Intersection over Union metric adapted to binary classification.
6 Baselines
The evaluation compares the proposed approach with adapted dense-prediction baselines because one-shot semantic segmentation is a new problem. The baselines include feature-based classifiers, fine-tuning, co-segmentation, and Siamese dense matching.
- Baseline design: The study adapts prior dense pixel-prediction methods as baselines for the new one-shot segmentation task.
- Base Classifiers: Base Classifiers fine-tune an FCN for pixel-wise predictions, then train classifiers such as 1-NN or logistic regression on dense support-set features.
- Fine-tuning: The Fine-tuning baseline updates only fc6, fc7, and fc8 on support examples to reduce overfitting and inference time.Batch normalization of fc-7 features was found to make the fine-tuned network converge faster.
- Co-segmentation: Co-segmentation by Composition is evaluated using its publicly available implementation on PASCAL-5i.
- Siamese matching: The Siamese baseline uses two FCNs for dense feature extraction and learns pixel verification by comparing query pixels with support-set pixels.Pixels are labeled according to nearest neighbors in the learned similarity space.
7 Experiments
Experiments show that the proposed method generalizes better to unseen classes than baseline approaches, while substantially reducing inference time. Pretraining and meta-learning also support generalization without weak labels for the test classes.
- Sub-sampling support features degraded the 5-shot Siamese network enough that its results were excluded.The 5-shot results were worse than the 1-shot version after sub-sampling.
- 25% relative meanIoU improvement over 1-NN and fine-tuning demonstrates better one-shot generalization to new classes.Other methods overfit to the support image, whereas the proposed method performs better on unseen classes.
- 16% improvement over unsupervised co-segmentation shows that one strongly annotated image can outperform five weakly annotated images.The comparison concerns 5-shot segmentation.
- ∼3× faster one-shot inference and ∼10× faster 5-shot inference than logistic regression establish a substantial runtime advantage.Measurements used a 4GHz Intel Core-i7 CPU, 32GB RAM, and Titan X GPU.
- AlexNet-771 eventually performs on par with AlexNet-1000 and exceeds the best VGG baseline, despite excluding ImageNet classes overlapping PASCAL categories.AlexNet-771 achieved 33.6% meanIoU, compared with 28.1% for the Siamese VGG baseline on PASCAL-50.
8 Conclusion
The paper presents a new architecture for few-shot image segmentation that learns an ensemble classifier for pixel-level prediction. Experiments report clear superiority over baselines, with faster execution and a smaller memory footprint.
- Qualitative one-shot results visualize support ground truth in yellow and query predictions in red.
- The architecture addresses the challenge of learning new image-segmentation concepts from few examples.
- The model learns an ensemble classifier and uses it to classify pixels in a query image.
- The proposed method is considerably faster than other baselines and has a smaller memory footprint.
Supplementary Materials
The supplementary material describes weight hashing as a compact mapping from a lower-dimensional vector to a larger parameter vector. The implementation fixes the random hashing structure while training the surrounding model.
- Weight hashing maps a 1000-dimensional VGG output to the 4097 dimensions of {w,b}.
- The mapping reduces variance and overfitting relative to introducing a massive fully connected layer.
- The mapping replicates each input coefficient across multiple output locations and randomly flips signs.
- Random functions κ(i) and ζ(i) determine the source coefficient and sign used for each output coordinate.
- The hashing values are implemented in a fully connected layer, set before training, and kept fixed.
B Siamese Network for Dense Matching
The dense-matching baseline adapts a Siamese image-recognition network to pixel-level segmentation. Identical FCNs extract dense features, and learned pixel similarities determine query labels.
- The adapted Siamese network uses two FCNs to extract dense feature volumes for the query and support images.
- Training samples query-support image pairs and uses cross-entropy between the sigmoid similarity and the true binary pixel labels.
Qualitative Results
Qualitative results show that the method segments varied classes from a single support image and changes predictions when the support set changes. Increasing the support set from one to five examples improves the predicted mask on the same query image.
- The method segments a variety of classes and distinguishes an object from others using only a single support image.
- Changing the support set changes the predicted mask for a fixed query image.The comparison uses support pairs from unseen cow and car classes.
- Increasing the support set from 1-shot to 5-shot improves the predicted mask on the same query image.Ground-truth regions are green, predictions red, and their overlap yellow.
- In qualitative tiles, support annotations are shown in yellow and query predictions in red.