Source-linked AI summary
Box-driven Class-wise Region Masking and Filling Rate Guided Loss for Weakly Supervised Semantic Segmentation
Chunfeng Song, Yan Huang, Wanli Ouyang, Liang Wang
TL;DR
Pixel-level annotations make semantic segmentation costly, while bounding-box supervision yields imperfect proposals that must be used selectively. The paper combines BCM class-wise masking with FR-Loss, and reports state-of-the-art performance on PASCAL VOC 2012, comparable to fully supervised performance.
Problem
Pixel-level segmentation labels are expensive, and weakly supervised methods must guide FCNs toward correct regions while ignoring wrongly labeled proposal pixels.
Method
The method learns box-driven class-wise masks and uses class mean filling rates in FR-Loss to select confident score-map locations while ignoring weak proposal pixels.
Results
The combined BCM and FR-Loss method achieves state-of-the-art results on PASCAL VOC 2012 and performance comparable to a fully supervised model.
Takeaways & Limitations
Class filling rates provide global statistical information that adjusts learning and helps reduce negative effects from wrongly labeled proposals.
Takeaways & Limitations
Filling-rate guidance may require refinement because objects of the same class can differ substantially in shape and pose.
Abstract
from arXiv · showhide
Semantic segmentation has achieved huge progress via adopting deep Fully Convolutional Networks (FCN). However, the performance of FCN based models severely rely on the amounts of pixel-level annotations which are expensive and time-consuming. To address this problem, it is a good choice to learn to segment with weak supervision from bounding boxes. How to make full use of the class-level and region-level supervisions from bounding boxes is the critical challenge for the weakly supervised learning task. In this paper, we first introduce a box-driven class-wise masking model (BCM) to remove irrelevant regions of each class. Moreover, based on the pixel-level segment proposal generated from the bounding box supervision, we could calculate the mean filling rates of each class to serve as an important prior cue, then we propose a filling rate guided adaptive loss (FR-Loss) to help the model ignore the wrongly labeled pixels in proposals. Unlike previous methods directly training models with the fixed individual segment proposals, our method can adjust the model learning with global statistical information. Thus it can help reduce the negative impacts from wrongly labeled proposals. We evaluate the proposed method on the challenging PASCAL VOC 2012 benchmark and compare with other methods. Extensive experimental results show that the proposed method is effective and achieves the state-of-the-art results.
1. Introduction
The paper addresses weakly supervised semantic segmentation from bounding boxes by combining class-wise region masking with filling-rate-guided selection of reliable proposal pixels. It targets irrelevant regions and mislabeled pseudo-label pixels while reporting strong results on PASCAL VOC 2012.
- Motivation: Pixel-level segmentation annotations are expensive, motivating segmentation from bounding boxes despite the challenge of exploiting their class-level and region-level supervision.Pixel-level annotation costs about 15 times more than bounding-box labeling and 60 times more than image-class labeling.
- Motivation: Previous methods generate pixel-level proposals but often train with fixed proposals or simple iteration, leaving mismatches between generated proposals and ground truth.The paper identifies guiding the FCN toward correct object regions while ignoring wrongly labeled proposal regions as a core problem.
- Box-driven class-wise masking: BCM learns class-specific masks from box-like supervision to softly remove irrelevant regions and provide foreground shape and location hints.Unlike a global mask, class-wise masks can represent multiple accurate shape templates for different classes.
- Filling-rate guidance: Class filling rates provide a more stable cue for selecting confident score-map locations because foreground proportions differ across classes but are similar within a class.The paper notes that bus boxes contain about 80% foreground pixels while bicycle boxes contain about 20%, and proposes clustering within classes to handle shape and pose variation.
- Filling-rate guidance: FR-Loss uses mean class filling rates to select confident locations for back propagation and ignore wrongly labeled proposal pixels.Together, BCM and FR-Loss adjust learning using class-aware spatial information and global filling-rate statistics.
- Results: Experiments on PASCAL VOC 2012 report that the combined method is effective, achieves state-of-the-art results, and is comparable to a fully supervised model.The supplied passages do not provide a numerical benchmark value.
2. Related Work
Related work spans fully supervised FCN-based segmentation and weakly supervised learning from image labels, points, scribbles, and bounding boxes. Bounding-box methods commonly rely on generated segment proposals and recursive or expectation-maximization training.
- 2.1. Fully Supervised Semantic Segmentation: Fully supervised semantic segmentation has progressed through FCN-based architectures and subsequent network redesigns, including atrous convolution and encoder-decoder models.FCN was the first structure introduced into the segmentation field in the cited progression.
- 2.2. Weakly Supervised Semantic Segmentation: Weakly supervised methods use image labels, points, scribbles, or bounding boxes; bounding-box approaches are most closely related to this paper.The related methods include BoxSup, WSSL, and SDI.
- 2.2. Weakly Supervised Semantic Segmentation: BoxSup uses recursive training with segment proposals, while WSSL applies expectation-maximization to proposals generated by dense CRF.The supplied passage also begins describing SDI as combining MCG and GrabCut methods.
3. Our Method
The method learns weakly supervised semantic segmentation from bounding boxes by combining class-wise feature masking with filling-rate-guided adaptive supervision. It generates proposals and class statistics, then trains an FCN with BCM and FR-loss, including sub-class refinement for varied object shapes.
- Overall Pipeline: The framework generates pixel-level proposals and mean filling rates from bounding-box annotations before training an FCN with BCM and FR-loss.The backbone is DeepLab-LargeFOV, initialized from VGG-16 pretrained on ImageNet.
- Box-driven Class-wise Masking: BCM learns separate box-supervised attention maps for each class and spatially masks corresponding feature branches to suppress irrelevant regions.The class branches are recombined to produce the final segmentation score map.
- Box-driven Class-wise Masking: Class-wise masking supports different shape and size templates while providing foreground cues through bounding-box supervision.The method contrasts this with global spatial attention, which does not learn class-specific masking maps.
- Filling Rate Guided Adaptive Loss: FR-loss ranks class scores and backpropagates through the most confident locations according to each class’s mean filling rate, ignoring weaker proposal pixels.This adaptive selection is intended to reduce negative effects from wrongly labeled proposal regions.
- Filling Rate Guided Adaptive Loss: K-means clustering refines filling-rate guidance by dividing each class into sub-classes whose mean rates better represent shape and pose variation.Figure 3 illustrates three airplane sub-classes with different filling rates.
4. Experiments
Experiments on PASCAL VOC 2012 evaluate the method under weakly and semi-supervised settings, with ablations, comparisons, and qualitative analyses. BCM and FR-Loss jointly improve segmentation performance, while filling rates and class-wise masking provide useful guidance for learning from imperfect proposals.
- Experimental setup: The framework is evaluated on Pascal VOC2012 using mean pixel Intersection-over-Union under weakly and semi-supervised conditions.The benchmark contains 21 classes, with bounding-box supervision in the weakly supervised setting and additional labeled segments in the semi-supervised setting.
- Effectiveness of BCM and FR-Loss: 66.8% mean IoU is achieved when BCM and FR-loss are combined, exceeding the baseline and either module used separately.With CRF-box proposals, BCM and FR-loss individually achieve 65.6% and 65.8% mean IoU, respectively.
- Comparison with the State-of-the-art Methods: 67.5% mean IoU is achieved under semi-supervision, outperforming all compared methods with only 1/10 of the training segments labeled.The semi-supervised model improves by 0.7% over its weakly supervised version and reaches comparable performance with the fully supervised model.
- Discussions: Segmentation remains difficult for classes such as person and chair and for chaotic scenes, while airplane and sheep are easier to segment.The paper also notes frustrated cases, including a dog wearing cloth that confuses the model.
- Effectiveness of BCM and FR-Loss: The learned BCM masks remove irrelevant and cluttered background regions, helping class-wise attention guide segmentation learning.Qualitative visualizations show most backgrounds removed, while the method compares rectangle-box, CRF, and semi-supervised results against ground-truth segmentations.
- Effectiveness of BCM and FR-Loss: Mean filling rates provide class-specific guidance for selecting confident score-map locations and reducing the influence of wrongly labeled proposal pixels.Filling rates are calculated from generated pixel-level proposals and are used by FR-Loss as prior information for adaptive learning.
5. Conclusion
The paper concludes that BCM provides class-aware attention maps, while FR-Loss uses class-level filling-rate statistics to reduce errors from imperfect proposals. Together, they are effective on PASCAL VOC 2012, with joint detection and segmentation learning identified as future work.
- Conclusion: BCM learns class-aware attention maps that indicate whether regions contain a specific class and support segmentation learning.The maps help remove irrelevant regions and provide foreground hints.
- Conclusion: FR-loss uses global filling-rate statistics to help ignore wrongly labeled pixels in region-level proposals.The adaptive loss adjusts model learning rather than relying only on fixed individual proposals.
- Conclusion: BCM and FR-loss together are effective on the PASCAL VOC 2012 benchmark and achieve state-of-the-art results.The conclusion reports extensive experimental support for the proposed method.
- Conclusion: Future work will explore jointly learning object detection and segmentation to identify positive interactions between the tasks.