Source-linked AI summary
Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation
Alexander Kolesnikov, Christoph H. Lampert
TL;DR
Weakly supervised segmentation seeks to reduce reliance on costly pixel-level annotations while closing the performance gap caused by image-level labels. The paper introduces SEC, a composite loss that seeds from localization cues, expands objects using image-level class information, and constrains masks to boundaries. On PASCAL VOC 2012, SEC substantially outperforms prior state-of-the-art methods and provides ablations explaining the contributions of its loss terms.
Problem
Fully annotated segmentation data is costly, and models trained from only image-level labels have a substantial performance gap relative to models trained from full segmentation masks.
Method
SEC trains deep segmentation networks with a composite loss that seeds predictions from weak localization cues, expands objects using image-level class information, and constrains masks to object boundaries.
Results
13.5% higher mean intersection-over-union score than the state-of-the-art approaches on the PASCAL VOC 2012 test data, with new best scores on 20 out of 21 semantic classes.
Takeaways & Limitations
SEC substantially improves weakly supervised segmentation under the same experimental conditions and its ablation study examines how individual loss terms and their combinations affect segmentation quality.
Takeaways & Limitations
The method can confuse boats with water in their backgrounds, and stronger segmentation priors or automatically estimated object sizes are identified as desirable improvements.
Abstract
from arXiv · showhide
We introduce a new loss function for the weakly-supervised training of semantic image segmentation models based on three guiding principles: to seed with weak localization cues, to expand objects based on the information about which classes can occur in an image, and to constrain the segmentations to coincide with object boundaries. We show experimentally that training a deep convolutional neural network using the proposed loss function leads to substantially better segmentations than previous state-of-the-art methods on the challenging PASCAL VOC 2012 dataset. We furthermore give insight into the working mechanism of our method by a detailed experimental study that illustrates how the segmentation quality is affected by each term of the proposed loss function as well as their combinations.
1 Introduction
Weakly supervised segmentation addresses the costly bottleneck of fully annotated data, but image-level supervision leaves a substantial gap relative to full masks. SEC proposes a composite loss that seeds from localization cues, expands objects using image-level class information, and constrains predictions to object boundaries.
- Fully annotated segmentation data is costly to produce, motivating methods that learn from unlabeled or partially labeled data.Manual segmentation masks require several worker-minutes per image, making annotation a bottleneck.
- Image-level labels are efficiently collected, but models trained from them retain a large performance gap compared with models trained from full segmentation masks.
- SEC introduces a composite loss for weakly supervised segmentation built around three principles: Seed, Expand and Constrain.
- Localization cues provide reliable seeds but not exact object extents, so the seeding loss matches cues while ignoring the rest of the image.
- Global weighted rank pooling supports expansion by enlarging seed regions to a reasonable size while generalizing max- and average-pooling.
- The constrain-to-boundary loss addresses imprecise boundaries during training by encouraging predicted masks to respect low-level image information and object boundaries.
2 Related work
Related work seeks to reduce annotation requirements for semantic segmentation, ranging from partial masks and bounding boxes to image-level labels. Deep convolutional networks are the strongest approaches for difficult benchmarks such as PASCAL VOC 2012, while learning from only image-level labels remains challenging.
- Semantic image segmentation assigns a semantic class label to every pixel, but fully annotated training data is a major bottleneck.
- Earlier weakly supervised methods reduced annotation requirements by using partial segmentations or object bounding boxes.
- Learning segmentation models from only image-level labels remains a challenge, with graph-based and multiple-instance-learning approaches forming major categories.
- Deep convolutional-network methods are currently strong enough to address datasets as difficult as PASCAL VOC 2012.
3 Weakly supervised segmentation from image-level labels
SEC trains weakly supervised segmentation networks with a three-term loss that seeds predictions from localization cues, expands objects using image-level labels, and constrains masks to image boundaries.
- SEC loss: SEC minimizes seeding, expansion, and constrain-to-boundary losses to train a segmentation network from weak annotations.The combined objective is optimized end-to-end with back-propagation, including gradients through the fully connected CRF.
- Seeding loss: Weak localization cues guide predictions at labeled landmarks while the seeding loss ignores image regions without reliable localization evidence.The localization sets can be generated by existing weak localization techniques used as black boxes.
- Expansion loss: Global weighted rank pooling expands seeds toward reasonable object sizes by weighting more promising locations more heavily.GWRP generalizes global max-pooling and global average-pooling: its decay parameter interpolates between GMP at d_c = 0 and GAP at d_c = 1.
- Expansion loss: The expansion loss uses image-level class presence and absence information to penalize missing present classes, responses for absent classes, and inappropriate background predictions.The supplied objective specifies separate aggregation behavior for occurring classes, non-occurring classes, and background.
- Constrain-to-boundary loss: The constrain-to-boundary loss penalizes segmentations discontinuous with spatial and color information, encouraging masks to align with object boundaries.It matches network outputs to fully connected CRF outputs through a mean KL-divergence, using image-dependent pairwise potentials.
4 Experiments
SEC is evaluated on PASCAL VOC 2012 using weak image-level labels, with experiments measuring both benchmark performance and the effects of its loss components. The method substantially improves over prior approaches, while detailed analyses explain how seeding, expansion, boundary constraints, and pooling affect segmentation quality.
- Numeric Results: SEC improves test-set mIoU by 13.5% over prior state-of-the-art methods and sets best scores on 20 of 21 semantic classes.On the validation set, it improves over the state of the art by 14.1% and sets best scores on 19 of 21 classes.
- Qualitative Results: Successful examples show accurate segmentations that can recover fine boundary details, while failures involve objects tied to stereotyped backgrounds or incorrect class assignments.Wrong labels are especially observed when objects overlap or contain multiple components with substantially different appearances.
- Pooling Strategies: GWRP produces better segmentation quality than GMP and GAP, with object sizes closer to correct on average.GMP tends to underestimate object sizes, whereas GAP tends to overestimate them; the comparison reports foreground-pixel fraction and mean IoU.
- Loss-Term Interactions: The seeding loss is crucial because removing it loses object localization despite retaining global label statistics.The authors attribute this tension partly to the segmentation network’s large 378x378 field of view; a smaller 211x211 field of view performs clearly worse.
- Loss-Term Interactions: The constrain-to-boundary loss aligns predictions with image structure but may not expand seeds across visually dissimilar object parts.This limitation is illustrated for objects such as people whose clothing contains different colors.
- Loss-Term Interactions: Expansion increases predicted object sizes and suppresses absent classes, but combined with seeding alone it can reduce performance because added regions do not follow image boundaries.The combined SEC terms address this by integrating seed locations, reasonable object extent, and boundary alignment.
5 Conclusion
The paper concludes that SEC substantially outperforms prior state-of-the-art methods under matched conditions, while identifying object-size information and stronger priors as promising improvements.
- SEC outperforms previous state-of-the-art methods by a large margin under the same experimental conditions.The paper also includes a detailed ablation study.
- Object-size information can dramatically improve weakly-supervised segmentation performance.
- SEC can incorporate size information through decay parameters, but object sizes are not automatically estimated.
- Stronger segmentation priors about shape or materials could help avoid mistakes such as confusing boats with background water.
6 Supplementary material
The supplementary material details how SEC obtains weak localization cues and how the small-field-of-view architecture is derived for its experiments.
- Weak localization procedure: SEC uses classification-network cues to seed foreground and background regions.Foreground localization uses one technique, while background localization uses an alternative saliency-based technique.
- Weak localization procedure: The foreground localization network is modified from VGG and fine-tuned on PASCAL VOC 2012 to produce class-specific heat maps.It uses a larger input resolution, omits the final two max-pooling layers, and is trained with multilabel logistic loss.
- Weak localization procedure: Background cues are derived from smoothed class-independent saliency maps by selecting the 10% least salient locations in each image.
- Small field-of-view architecture: The small-field-of-view architecture replaces atrous convolutions with standard convolutional layers.This reduces the field-of-view from 378x378 to 211x211.