Source-linked AI summary
Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach
Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, Kaizhu Huang
TL;DR
Image-level supervision simplifies annotation but leaves a difficult gap between image labels and pixel-level predictions. RRM closes this gap with an end-to-end classification-and-segmentation network that mines reliable regions and uses dense energy loss, achieving 62.6/62.9 mIoU in one step and 66.3/66.5 in an extended two-step framework on Pascal VOC.
Problem
Image-level labels are easy to collect but do not directly specify pixel-level semantic assignments, making weakly supervised segmentation challenging.
Method
RRM combines a classification branch that mines reliable CAM-based regions with a parallel segmentation branch optimized using dense energy loss and pixel-wise cross-entropy.
Results
62.6 and 62.9 mIoU on Pascal VOC val and test sets are achieved by one-step RRM, while its two-step extension reaches 66.3 and 66.5.
Takeaways & Limitations
RRM provides an end-to-end one-step alternative to complex pseudo-mask pipelines while retaining competitive segmentation performance.
Abstract
from arXiv · showhide
Weakly supervised semantic segmentation is a challenging task as it only takes image-level information as supervision for training but produces pixel-level predictions for testing. To address such a challenging task, most recent state-of-the-art approaches propose to adopt two-step solutions, \emph{i.e. } 1) learn to generate pseudo pixel-level masks, and 2) engage FCNs to train the semantic segmentation networks with the pseudo masks. However, the two-step solutions usually employ many bells and whistles in producing high-quality pseudo masks, making this kind of methods complicated and inelegant. In this work, we harness the image-level labels to produce reliable pixel-level annotations and design a fully end-to-end network to learn to predict segmentation maps. Concretely, we firstly leverage an image classification branch to generate class activation maps for the annotated categories, which are further pruned into confident yet tiny object/background regions. Such reliable regions are then directly served as ground-truth labels for the parallel segmentation branch, where a newly designed dense energy loss function is adopted for optimization. Despite its apparent simplicity, our one-step solution achieves competitive mIoU scores (\emph{val}: 62.6, \emph{test}: 62.9) on Pascal VOC compared with those two-step state-of-the-arts. By extending our one-step method to two-step, we get a new state-of-the-art performance on the Pascal VOC (\emph{val}: 66.3, \emph{test}: 66.5).
Introduction
Image-level supervision makes semantic segmentation accessible but difficult because image labels do not directly map to pixels. RRM addresses this with an end-to-end network that mines reliable regions and achieves competitive segmentation performance.
- Image-level annotations are simpler to collect than pixel-level masks but provide no direct mapping between semantic labels and pixels.
- Existing one-step methods are elegant and easy to implement, but their segmentation accuracy trails fully supervised counterparts.
- RRM uses separate branches to generate pseudo pixel-level masks from image-level annotations and predict semantic segmentation results.
- RRM mines confident, tiny object/background regions from class activation maps and applies CRF pruning before using them as segmentation supervision.
- 62.6 and 62.9 mIoU on Pascal VOC val and test sets make the one-step method competitive with complex two-step approaches.
Related Work
Prior image-level segmentation methods commonly transform class activation maps into pseudo labels through additional networks and techniques. This work instead seeks to simplify the task with one network covering the full process.
- Image-level weakly supervised segmentation commonly uses class activation maps to generate initial object seeds or regions.
- Prior approaches refine initial regions using techniques such as iterative erasing, saliency maps, region networks, pixel networks, and affinity networks.
- These methods often require extra networks or processing before fully convolutional segmentation training.
- The paper proposes a single network for the whole image-level weakly supervised segmentation task to simplify the process.
Proposed Method
RRM jointly generates reliable, sparse pixel annotations and segmentation predictions in parallel, using shared features and end-to-end optimization. Its pipeline combines CAM-based region mining, CRF pruning, and a dense energy loss to train with limited labels.
- RRM uses parallel classification and semantic segmentation branches that share a backbone and update the whole network jointly during training.The classification branch generates annotations, while the segmentation branch predicts pixel-level labels.
- Classification Branch: Generating Labels for Reliable Regions: The classification branch converts CAMs into reliable yet tiny labeled regions, treating remaining pixels as unlabeled supervision targets.Original CAMs are processed into reliable pixel-level annotations before training the segmentation branch.
- Semantic Segmentation Branch: Making Predictions: The segmentation branch optimizes cross-entropy and dense energy losses, with cross-entropy using labeled pixels and dense energy modeling RGB color and spatial relationships across pixels.The joint segmentation loss is defined as Ljoint-seg = Lce + Lenergy; the overall RRM loss also includes classification loss.
- Classification Branch: Generating Labels for Reliable Regions: Multi-scale CAMs improve object localization over single-scale CAMs, while dense CRF post-processing removes mislabeled pixels and preserves reliable regions.The method combines CAM-derived and CRF-derived labels, especially improving reliability near object boundaries.
- Semantic Segmentation Branch: Making Predictions: Dense energy loss mitigates errors from imperfect pixel labels by using a soft filter and exploiting information from unlabeled regions.Its pixel-pair energy uses segmentation probabilities together with RGB colors and spatial positions.
Experiments
Experiments evaluate RRM on PASCAL VOC 2012 using ablations, end-to-end comparisons, two-step extensions, and qualitative results. The method benefits from reliable region mining and joint loss optimization, achieving competitive one-step and state-of-the-art two-step performance.
- Experimental Setup: RRM is trained and evaluated on PASCAL VOC 2012 and augmented data using mIoU, with 10,582 training, 1,449 validation, and 1,456 test images.The backbone is a ResNet-38 with dilated convolutions in the final three blocks.
- Ablation Studies: Using 20%-60% labeled pixels provides the best performance because too few labels are insufficient while too many introduce incorrect labels as noise.The labeled-pixel proportion is controlled by changing α.
- Ablation Studies: RRM-generated pseudo labels outperform original CAM labels, and the joint loss further improves segmentation performance.The ablation isolates reliable region mining and the joint loss as the two main components.
- Ablation Studies: 14.1 mIoU points are gained by adding the joint loss to RRM, increasing performance from 48.5 with cross-entropy loss to 62.6.The joint loss combines dense energy loss with pixel-wise cross-entropy loss.
- Two-Step Extension: The two-step extension trains Deeplab models on RRM-generated pseudo masks, using VGG and ResNet-101 variants.The ResNet variant uses Deeplab-v2 with a ResNet-101 backbone.
- Comparisons with Previous Approaches: Ours-RRM-VGG and Ours-RRM-ResNet outperform AffinityNet when using the same backbone, while RRM uses one end-to-end network instead of three DNNs.AffinityNet uses ResNet-38 as a baseline and multiple networks with additional processing.
- Two-Step Extension: 66.3 and 66.5 mIoU on PASCAL VOC val and test sets make Ours-RRM-ResNet the reported two-step state of the art, without extra data or information.These scores improve over FickleNet by 1.4 and 1.2, respectively.
- Comparisons with Previous Approaches: The one-step RRM produces more accurate qualitative segmentation than EM-Adapt for large and small objects, including more accurate boundaries.The comparison is shown on the PASCAL VOC 2012 validation set.
Conclusion
The paper presents RRM as an end-to-end one-step approach that mines reliable tiny regions for supervision, then extends it to a two-step method. Both approaches achieve state-of-the-art performance, offering an alternative to traditional two-step solutions.
- RRM mines reliable yet tiny regions and uses them directly as ground-truth labels for training the segmentation model.
- The dense energy loss combines shallow RGB and spatial features with pixel-wise cross-entropy under limited-pixel supervision.
- The authors extend the one-step RRM into a two-step method, with both variants achieving state-of-the-art performance.