Source-linked AI summary
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation
Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Jiashi Feng, Yao Zhao, Shuicheng Yan
TL;DR
Semantic segmentation normally depends on costly pixel-level masks, while image-level-only methods have limited performance. STC progressively learns from saliency-supervised simple images to predicted masks on complex images, using only image-level annotations. On PASCAL VOC 2012, the framework achieves state-of-the-art performance relative to the cited methods, while remaining subject to failures from visually similar or background-confusable objects.
Problem
Pixel-level segmentation masks are expensive to annotate, and prior image-level-only methods perform far below fully supervised schemes.
Method
STC trains Initial-, Enhanced-, and Powerful-DCNNs progressively from saliency maps and predicted masks of simple and complex images, using image-level labels.
Results
STC achieves state-of-the-art performance on the PASCAL VOC 2012 segmentation benchmark using 40K simple Flickr images and 10K complex PASCAL VOC images.
Takeaways & Limitations
The framework demonstrates that image-level annotations can support progressively improved weakly supervised semantic segmentation through simple-to-complex training.
Takeaways & Limitations
Qualitative failures include confusing chairs with sofas and predicting large sofa regions as background; cleaner, more varied samples or classification-based post-processing may help.
Abstract
from arXiv · showhide
Recently, significant improvement has been made on semantic object segmentation due to the development of deep convolutional neural networks (DCNNs). Training such a DCNN usually relies on a large number of images with pixel-level segmentation masks, and annotating these images is very costly in terms of both finance and human effort. In this paper, we propose a simple to complex (STC) framework in which only image-level annotations are utilized to learn DCNNs for semantic segmentation. Specifically, we first train an initial segmentation network called Initial-DCNN with the saliency maps of simple images (i.e., those with a single category of major object(s) and clean background). These saliency maps can be automatically obtained by existing bottom-up salient object detection techniques, where no supervision information is needed. Then, a better network called Enhanced-DCNN is learned with supervision from the predicted segmentation masks of simple images based on the Initial-DCNN as well as the image-level annotations. Finally, more pixel-level segmentation masks of complex images (two or more categories of objects with cluttered background), which are inferred by using Enhanced-DCNN and image-level annotations, are utilized as the supervision information to learn the Powerful-DCNN for semantic segmentation. Our method utilizes $40$K simple images from Flickr.com and 10K complex images from PASCAL VOC for step-wisely boosting the segmentation network. Extensive experimental results on PASCAL VOC 2012 segmentation benchmark well demonstrate the superiority of the proposed STC framework compared with other state-of-the-arts.
1 INTRODUCTION
STC addresses the cost and limitations of pixel-level supervision by learning segmentation progressively from simple images to complex ones using only image-level labels.
- Pixel-level mask annotation for semantic segmentation requires substantial financial expense and human effort compared with image-level labeling.
- 40.6% vs. 66.4%: prior image-level-only methods remained far below fully supervised semantic segmentation performance.
- Simple images with clean backgrounds and one major object category allow salient-object methods to separate foreground and background pixels.
- STC first learns an initial segmenter from saliency-derived masks and image-level labels, then uses it to segment more objects in complex images.
- Retrieved simple images receive DRFI saliency maps, whose foreground pixels are assigned the image-level semantic label for segmentation training.
- The framework uses only image-level labels, introduces a saliency-map multi-label cross-entropy loss, and achieves state-of-the-art performance on PASCAL VOC 2012.
2 RELATED WORK
The related work spans weakly supervised segmentation, self-paced learning, and salient-object detection, while STC combines these ideas through a simple-to-complex training progression.
- STC uses DRFI saliency maps generated for simple images to provide foreground/background supervision for its initial segmentation network.
- Weakly supervised segmentation methods reduce annotation burden by replacing pixel masks or bounding boxes with image-level labels, including MIL and EM-based approaches.
- Self-paced learning motivates starting with easy samples, but STC defines sample simplicity from appearance, such as object multiplicity and background clutter.
- Other related methods include graph propagation and latent structured prediction for assigning image-level labels to semantic regions.
3 PROPOSED METHOD
The STC framework progressively trains segmentation networks from saliency-supervised simple images to predicted-mask supervision on increasingly complex images. It uses soft pixel supervision initially, refines predictions with image-level labels, and then trains stronger networks with single-label masks.
- Initial-DCNN: STC begins by using saliency maps from simple images to train an Initial-DCNN for semantic segmentation.Saliency values indicate foreground likelihood, while image-level labels assign foreground pixels their semantic class.
- Initial-DCNN: A SaliencyCut-based alternative lowers VOC 2012 validation performance by 3% because inaccurate masks introduce noise, whereas multi-label loss reduces this effect.Correct semantic labels continue contributing to optimization despite low-quality saliency maps.
- Enhanced-DCNN: Image-level labels refine Initial-DCNN predictions on simple images, enabling training of an Enhanced-DCNN with predicted segmentation masks.This refinement removes some false predictions before single-label cross-entropy training.
- Powerful-DCNN: The framework incorporates complex images and their image-level labels to produce more predicted masks and train a Powerful-DCNN.Compared with Initial-DCNN, Enhanced-DCNN uses many predicted masks and gains stronger segmentation capability before the final stage.
- Initial-DCNN: The Initial-DCNN uses multi-label cross-entropy because each pixel is softly associated with background and one foreground class.The saliency map provides class probabilities after resizing to the network output resolution.
- Training losses: The three networks are trained with two cross-entropy variants: single-label loss for Enhanced-DCNN and Powerful-DCNN, and multi-label loss for Initial-DCNN.The distinction reflects whether pixel class information is exact or represented probabilistically from saliency maps.
4 EXPERIMENTAL RESULTS
Experiments evaluate the STC framework on PASCAL VOC 2012 using simple Flickr-Clean images, complex VOC images, and progressively trained segmentation networks. Results show improvements across training stages and competitive performance against weakly and fully supervised methods, alongside identified failure cases and comparison caveats.
- Dataset: 41,625 Flickr-Clean images were collected after saliency-based filtering to train I-DCNN.The filtering retained images whose foreground regions satisfied 0.3*NI < Nf < 0.5*NI; using all 100K crawled images caused a 4% performance drop.
- Training Strategies: 17.6% mIoU resulted when I-DCNN was trained with complex images instead of simple images.The experiment was used to validate the necessity of using simple images.
- Comparison with State-of-the-art Methods: 49.8% versus 42.0% improved over the best result of MIL-ILP while using 50K rather than 700K training images.The comparison involves 40K Flickr-Clean simple images and 10K PASCAL VOC complex images, whereas MIL-ILP used 700K images.
- Comparison with State-of-the-art Methods: 51.2% versus 51.7% was reported on the PASCAL VOC 2012 test set compared with the state-of-the-art weakly-supervised performance.The framework also approached fully supervised results and achieved similar results to SDS in the reported comparison.
- Qualitative Results: Failure cases included chair pixels classified as sofa and large sofa regions classified as background.The paper suggests more clean-background samples with varied appearances or classification-based post-processing as possible remedies.