Source-linked AI summary
Fully Convolutional Multi-Class Multiple Instance Learning
Deepak Pathak, Evan Shelhamer, Jonathan Long, Trevor Darrell
TL;DR
The paper addresses semantic segmentation from weak image-level labels, reducing reliance on costly strong annotation. It combines multi-class MIL with an end-to-end FCN that selects latent pixel instances, and reports a 96% relative improvement over baseline results on PASCAL VOC. The authors describe these as preliminary but encouraging results, with coarse outputs currently only interpolated.
Problem
Weakly supervised semantic segmentation could avoid the costly collection of strongly annotated data, but prior MIL methods have limitations including fixed representations and sensitivity to initial latent hypotheses.
Method
The paper jointly learns an FCN representation and pixel classifier using a multi-class MIL loss that selects maximum-scoring pixels from class heat-maps.
Results
96% relative improvement over baseline results is achieved on the PASCAL VOC segmentation evaluation when the classifier is fine-tuned from common classes.
Takeaways & Limitations
The model enables end-to-end weakly supervised segmentation without proposal or instance-hypothesis mechanisms, with inference taking approximately 1/5 second.
Takeaways & Limitations
The coarse output is merely interpolated, while conditional random field regularization or super-pixel projection could further refine predictions.
Abstract
from arXiv · showhide
Multiple instance learning (MIL) can reduce the need for costly annotation in tasks such as semantic segmentation by weakening the required degree of supervision. We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network. In this setting, we seek to learn a semantic segmentation model from just weak image-level labels. The model is trained end-to-end to jointly optimize the representation while disambiguating the pixel-image label assignment. Fully convolutional training accepts inputs of any size, does not need object proposal pre-processing, and offers a pixelwise loss map for selecting latent instances. Our multi-class MIL loss exploits the further supervision given by images with multiple labels. We evaluate this approach through preliminary experiments on the PASCAL VOC segmentation challenge.
1 INTRODUCTION
The paper targets semantic segmentation from weak image-level labels to reduce costly strong annotation. It introduces end-to-end multi-class MIL in an FCN, using pixel-level instances and inter-class information to learn segmentation.
- Weak supervision could reduce the costly collection of strongly annotated data for scalable learning from available image-level labels.
- The proposed framework learns pixel-level semantic segmentation from labels indicating only object presence or absence.Each image is treated as a bag of pixel-level instances for a multi-class MIL loss.
- The method jointly learns the convolutional representation and pixel classifier through fully convolutional multiple instance learning.
- The approach addresses limitations of prior MIL methods involving fixed representations and sensitivity to initial latent instance-label hypotheses.
- The multi-class pixel-level loss uses inter-class competition to narrow latent instance hypotheses while maximizing pixel-instance classification scores.
2 FULLY CONVOLUTIONAL MIL
The fully convolutional MIL design produces spatial outputs and loss maps for arbitrary-sized images, enabling efficient latent-instance selection and end-to-end training.
- An FCN applies local operations on spatial coordinates, accepting inputs of any size and producing outputs with corresponding dimensions.
- The FCN predicts an output and loss map for every pixel, which can be masked or re-weighted to select instances for loss computation and back-propagation.
- The implementation converts VGG into fully convolutional form and fine-tunes from image-level ILSVRC classifier weights.
- Classifier-layer initialization is varied experimentally, with these initializations serving as baselines before MIL fine-tuning.
3 MULTI-CLASS MIL LOSS
The multi-class MIL loss computes logistic loss only at maximum-scoring pixels for labels present in each image and background. At inference, coarse predictions are interpolated into pixelwise segmentation.
- The loss selects the maximum-scoring pixel in each coarse heat-map for present classes and background, then computes multi-class logistic loss at those points.
- Ignoring non-maximal points avoids biasing FCN learning toward the background class.
- Inter-class confusion from multi-label images helps refine intra-class pixel accuracy during simultaneous training.
- At inference, the model selects the top class at each coarse-map location and bilinearly interpolates predictions to image resolution.
4 EXPERIMENTS
The method is evaluated on PASCAL VOC segmentation using intersection over union, with validation examples and test results reported. MIL fine-tuning substantially improves the baseline.
- The experiments train and validate on augmented PASCAL VOC 2011 data and evaluate on the held-out VOC 2012 test set.
- Intersection over union measures the percentage of pixels shared by ground-truth and predicted masks relative to their union.
- Table 1 reports results on PASCAL VOC 2011 validation and VOC 2012 test data.
- Figure 1 compares input images, ground-truth segmentations, and MIL-FCN outputs in each row.
- 96% relative improvement over the baseline is achieved when the classifier is fine-tuned from classes common to PASCAL and ILSVRC.
5 DISCUSSION
The proposed end-to-end FCN model performs weakly supervised semantic segmentation with a multi-class pixelwise MIL loss and avoids proposal or instance-hypothesis mechanisms. The preliminary results are encouraging but remain limited by coarse interpolated outputs that could be refined further.
- The model jointly performs multiple instance and representation learning with a multi-class pixelwise loss in an end-to-end FCN.
- Inference is fast, taking approximately 1/5 second.
- The coarse output is currently merely interpolated, while conditional random fields or super-pixel projection could refine predictions.