Source-linked AI summary
Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation
Seungho Lee, Minhyun Lee, Jongwuk Lee, Hyunjung Shim
TL;DR
WSSS with image-level supervision struggles with object coverage, boundaries, and co-occurring pixels. EPS combines localization maps for object identity with saliency maps for boundaries and jointly trains with saliency and classification losses. It reports new state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014, while being less effective for many small objects.
Problem
Image-level WSSS produces pseudo-masks with sparse object coverage, inaccurate boundaries, and co-occurring pixels from non-target objects.
Method
EPS uses C + 1 target-and-background localization maps with saliency maps as pseudo-pixel feedback, jointly optimized through saliency and multi-label classification losses.
Results
EPS achieves new state-of-the-art WSSS performance on both PASCAL VOC 2012 and MS COCO 2014 datasets.
Takeaways & Limitations
Combining localization and saliency information improves pseudo-mask boundaries and removes co-occurring pixels from non-target objects.
Takeaways & Limitations
EPS is less effective when many small objects appear.
Abstract
from arXiv · showhide
Existing studies in weakly-supervised semantic segmentation (WSSS) using image-level weak supervision have several limitations: sparse object coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects. To overcome these challenges, we propose a novel framework, namely Explicit Pseudo-pixel Supervision (EPS), which learns from pixel-level feedback by combining two weak supervisions; the image-level label provides the object identity via the localization map and the saliency map from the off-the-shelf saliency detection model offers rich boundaries. We devise a joint training strategy to fully utilize the complementary relationship between both information. Our method can obtain accurate object boundaries and discard co-occurring pixels, thereby significantly improving the quality of pseudo-masks. Experimental results show that the proposed method remarkably outperforms existing methods by resolving key challenges of WSSS and achieves the new state-of-the-art performance on both PASCAL VOC 2012 and MS COCO 2014 datasets.
1. Introduction
WSSS commonly generates pseudo-masks from image-level labels, but CAM-based localization remains limited in object coverage, boundary accuracy, and separation of co-occurring pixels. EPS combines localization and saliency information through joint training to address these challenges.
- 1. Introduction: WSSS generates pseudo-masks with an image classifier before training a segmentation model on them.CAM provides localization maps corresponding to image-level labels.
- 1. Introduction: Existing image-level WSSS methods face sparse target coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects.The railroad accompanying a train is given as an example of a co-occurring non-target region.
- 1. Introduction: Localization maps distinguish object identities but poorly separate boundaries, whereas saliency maps provide boundary information without revealing object identity.EPS is motivated by this complementary relationship.
- 1. Introduction: EPS predicts C + 1 classes, estimates saliency from target and background localization maps, and jointly optimizes saliency and multi-label classification losses.The saliency loss supplies pixel-wise feedback across foreground and background classes.
- 1. Introduction: Saliency loss penalizes boundary mismatches, expands maps toward object boundaries, and assigns co-occurring railroad pixels to the background class.The paper reports new state-of-the-art WSSS accuracies on PASCAL VOC 2012 and MS COCO 2014.
2. Related Work
Related WSSS work improves pseudo-masks through affinity, graph, contrastive, self-supervised, and saliency-based signals. EPS differs by using saliency as pseudo-pixel supervision while addressing co-occurring pixels and noisy saliency maps.
- 2. Related Work: WSSS methods improve pseudo-masks using cross-image affinity, knowledge graphs, contrastive optimization, and self-supervised sub-category discovery.These approaches target the limitations of pseudo-masks generated from image-level supervision.
- 2. Related Work: Saliency-guided methods use foreground-background maps as cues, with some approaches applying saliency as full supervision for single-object images.Saliency maps are produced using external saliency datasets with pixel-level or image-level annotations.
- 2. Related Work: EPS incorporates saliency as pseudo-pixel supervision to provide boundary and co-occurring-pixel cues, unlike prior methods that do not fully exploit it.The cited similar work is described as addressing neither co-occurring pixels nor noisy saliency maps.
3. Proposed Method
EPS jointly uses image-level localization maps for object identity and saliency maps for boundary information, treating saliency as pseudo-pixel feedback. Its classifier and joint losses refine foreground and background maps to improve boundaries and suppress co-occurring pixels.
- 3.1. Motivation: EPS combines target-class localization maps with saliency-map boundaries as complementary pseudo-pixel supervision.The method adds a background class and uses localization maps for both target objects and background.
- 3.2. Explicit Pseudo-pixel Supervision: A C + 1-class classifier produces C target-class localization maps and one background localization map for estimating saliency.The target maps provide object identity, while the background map supports foreground-background separation.
- 3.3. Joint Training Procedure: Joint training restores missing saliency objects, removes saliency noise, improves boundaries, and separates co-occurring pixels from target objects.The reported examples include restoring boats or chairs and removing water bubbles or contrails; segmentation accuracy gains reach 3.8–10.6%.
- 3.3. Joint Training Procedure: The saliency loss measures pixel-wise differences between actual and estimated saliency maps, while classification loss uses image-level labels.The total objective is Ltotal = Lcls + Lsal, and Lsal updates target and background classes whereas Lcls evaluates target classes.
- 3.2. Explicit Pseudo-pixel Supervision: Localization maps are assigned to foreground or background according to their overlap with the saliency map, mitigating systematic saliency bias.The overlap threshold is τ = 0.4 across datasets and backbone models; unselected maps are combined with the background map.
4. Experimental Setup
Experiments use PASCAL VOC 2012 and MS COCO 2014, with qualitative pseudo-mask comparisons and a ResNet38-based training setup. Boundary quality is evaluated using class-agnostic boundary precision, recall, and F1-score.
- Datasets: Experiments are conducted on PASCAL VOC 2012 and MS COCO 2014 benchmark datasets.The augmented PASCAL VOC training set contains 10,582 images; COCO images without target classes are excluded.
- Qualitative Evaluation: Figure 4 compares pseudo-masks from CAM, SEAM, ICD, SGAN, and EPS against input images and ground truth.The comparison is qualitative and focuses on pseudo-mask outputs on PASCAL VOC 2012.
- Implementation Details: EPS uses ResNet38 with output stride 8, ImageNet-pretrained backbones, SGD, batch size 8, and 20k training iterations.Training uses learning rate 0.01, random scaling, flipping, and 448 × 448 random crops.
5. Experimental Results
EPS improves pseudo-mask quality and segmentation across PASCAL VOC 2012 and MS COCO 2014 by sharpening boundaries, reducing co-occurring pixels, and using adaptive saliency-map selection. It achieves state-of-the-art segmentation results while remaining effective across segmentation networks and saliency detectors.
- Boundary and co-occurrence: EPS consistently lowers confusion ratios for co-occurring objects while improving target-class IoU compared with representative methods.The evaluation considers boat–water, train–railroad, and train–platform pairs.
- Map selection: The adaptive map-selection strategy automatically handles systematic saliency bias more effectively than naïve or manually predefined selections.Manual predefined classes can mitigate bias but require human selection and cannot optimize decisions per image.
- Pseudo-mask accuracy: EPS achieves 7–21% gaps over CAM, SEAM, ICD, and SGAN in pseudo-mask accuracy on PASCAL VOC 2012.The qualitative comparisons show sharper boundaries, fuller object coverage, and fewer coincident pixels.
- PASCAL VOC 2012: 71.0 validation and 71.8 test mIoU establish new PASCAL VOC 2012 state-of-the-art performance with ResNet101-based DeepLab-V1 and saliency maps.The gains exceed 3% over the previous best record, whose improvements were approximately 1%.
- Robustness: EPS remains more accurate than existing methods across segmentation networks and when using PFAN, DSS, or USPS saliency detectors.With ResNet101-based DeepLab-V1, PFAN, DSS, and USPS produce validation/test scores of 71.0/71.8, 70.0/70.1, and 68.8/69.9, respectively.
- MS COCO 2014: 35.7 mIoU on MS COCO 2014 validation is 1.9% higher than SGAN, establishing new state-of-the-art accuracy.The comparison uses VGG16-based DeepLab-V2.
6. Conclusion
EPS combines localization and saliency maps through joint pseudo-pixel supervision to improve pseudo-mask quality. It captures precise object boundaries and discards co-occurring pixels from non-target objects.
- EPS combines saliency maps and localization maps through joint pseudo-pixel supervision.
- The framework complements noisy or missing information between the two map types.
- EPS captures precise object boundaries and discards co-occurring pixels from non-target objects.
- Evaluations report new state-of-the-art WSSS accuracies on PASCAL VOC 2012 and MS COCO 2014.