Source-linked AI summary
Joint Learning of Saliency Detection and Weakly Supervised Semantic Segmentation
Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang
TL;DR
Existing WSSS methods often use pre-trained saliency models without explicitly modeling their interaction with segmentation. The paper proposes SSNet, a single end-to-end network that jointly learns both tasks from image-level category labels and pixel-level saliency annotations, and reports state-of-the-art performance across WSSS and saliency benchmarks.
Problem
Existing WSSS methods use pre-trained saliency models as separate preprocessing and do not explicitly model interactions between saliency detection and segmentation.
Method
SSNet jointly learns saliency detection and WSSS in one end-to-end model comprising a segmentation network and saliency aggregation module.
Results
The method achieves state-of-the-art performance among fully supervised saliency detection methods and WSSS methods.
Takeaways & Limitations
Joint modeling lets WSSS use segmentation cues from saliency annotations while allowing one shared model to predict both segmentation and saliency.
Takeaways & Limitations
Training uses two datasets: a saliency dataset with pixel-level saliency annotations and a classification dataset with image-level semantic category labels.
Abstract
from arXiv · showhide
Existing weakly supervised semantic segmentation (WSSS) methods usually utilize the results of pre-trained saliency detection (SD) models without explicitly modeling the connections between the two tasks, which is not the most efficient configuration. Here we propose a unified multi-task learning framework to jointly solve WSSS and SD using a single network, \ie saliency, and segmentation network (SSNet). SSNet consists of a segmentation network (SN) and a saliency aggregation module (SAM). For an input image, SN generates the segmentation result and, SAM predicts the saliency of each category and aggregating the segmentation masks of all categories into a saliency map. The proposed network is trained end-to-end with image-level category labels and class-agnostic pixel-level saliency labels. Experiments on PASCAL VOC 2012 segmentation dataset and four saliency benchmark datasets show the performance of our method compares favorably against state-of-the-art weakly supervised segmentation methods and fully supervised saliency detection methods.
1. Introduction
The paper identifies inefficiencies in using pre-trained saliency models as separate preprocessing for WSSS and proposes SSNet, a unified end-to-end framework that explicitly links saliency detection and segmentation.
- Motivation: Existing WSSS methods commonly combine class activation maps for object regions with pre-trained saliency maps for background regions.These methods exploit the close connection between saliency detection and semantic segmentation, since both require accurate pixel-wise masks.
- Motivation: Pre-trained saliency models are used as preprocessing, while heuristic background-selection rules complicate training and prevent an end-to-end process.This configuration also ignores interactions between saliency detection and WSSS.
- Proposed framework: SSNet jointly solves saliency detection and WSSS with a segmentation network and a saliency aggregation module.The framework uses image-level category labels together with pixel-level saliency labels.
- Proposed framework: SAM predicts category saliency scores and aggregates category segmentation masks into a saliency map, explicitly bridging the two tasks.The resulting interaction allows segmentation and saliency inference to provide cues to one another.
- Advantages: The framework transfers segmentation knowledge from class-agnostic saliency annotations to unseen semantic categories with only image-level annotations.The authors also state that saliency pixel-level ground truth requires less labeling cost than semantic segmentation annotations.
- Advantages: A single model simultaneously predicts segmentation and saliency results, with most parameters shared between the two tasks.Experiments on PASCAL VOC 2012 and four saliency benchmarks demonstrate favorable performance against WSSS and fully supervised saliency methods.
2. Related work
Related work covers deep saliency detection, image-level weakly supervised segmentation, and unified multi-task architectures that share representations across computer vision tasks.
- Saliency detection: Earlier saliency methods used low-level features and heuristic priors, while later deep learning methods improved saliency detection performance.The passage contrasts these approaches with the paper's joint treatment of saliency and segmentation.
- Positioning: The paper differs from single-task saliency methods by jointly targeting saliency detection and semantic segmentation despite a modest increase in labeling cost.It reports state-of-the-art performance in both tasks.
- Weakly supervised segmentation: Weakly supervised segmentation reduces annotation costs by using supervision such as image-level labels, bounding boxes, and scribbles.Image-level supervision is described as the weakest and most studied form among these alternatives.
- Weakly supervised segmentation: Some image-level WSSS methods use pre-trained saliency detection results to support segmentation.This establishes the related-work setting that the proposed joint framework seeks to replace.
- Multi-task learning: Prior multi-task architectures jointly addressed several vision tasks through shared encoders or unified networks.Examples include joint classification, detection, segmentation, saliency estimation, depth prediction, and surface-normal estimation.
3. The proposed approach
The proposed SSNet jointly learns saliency detection and weakly supervised semantic segmentation through a segmentation network and saliency aggregation module. Two training stages combine image-level category labels, pixel-level saliency annotations, and pseudo segmentation labels to improve segmentation supervision.
- Overview: SSNet uses a segmentation network and saliency aggregation module to jointly model semantic segmentation and saliency detection.The module predicts category saliency scores and aggregates category masks into a saliency map.
- Segmentation networks: SSNet-2 uses four dilated 3 × 3 convolution layers with dilation rates 6, 12, 18, and 24 to generate finer segmentation results.Its stronger supervision comes from the segmentation results predicted by SSNet-1.
- Joint learning: The first training stage trains SSNet-1 with category-classification loss and saliency loss using image-level semantic labels and pixel-level saliency annotations.The saliency loss encourages segmentation masks to be spatially consistent with saliency ground truth.
- Segmentation networks: The segmentation network averages spatial segmentation outputs to estimate category probabilities and trains them against image-level labels.This provides semantic supervision despite the absence of pixel-level segmentation labels.
- Joint learning: In the second stage, SSNet-1 predictions are class-filtered, CRF-refined, and converted into pseudo labels for training SSNet-2.SSNet-2 combines the saliency loss with pseudo-label segmentation loss and uses a more complex segmentation network for finer results.
4. Experiments
Experiments evaluate SSNet on saliency detection, weakly supervised semantic segmentation, and joint-learning ablations. The method achieves favorable performance across both tasks while multi-task training improves segmentation without harming saliency detection.
- Datasets and settings: The experiments use PASCAL VOC 2012 for semantic segmentation and compare saliency detection against fully supervised methods across four datasets.VOC training uses only image-level labels; saliency comparisons use MAE and maximum F-measure.
- Saliency detection: The proposed method achieves the smallest MAE across all saliency datasets and ranks second on one dataset and third on three datasets by maximum F-measure.Together, these metrics are reported as state-of-the-art saliency detection performance.
- Semantic segmentation: SSNet compares favorably with WSSS methods using image-level labels, pre-trained saliency models, bounding boxes, or scribbles.It simultaneously solves semantic segmentation and saliency detection in an end-to-end manner.
- Ablation study: Multi-task training in the first stage provides better training data for the second segmentation stage than single-task training.This conclusion follows from comparing SSNet-SS with SSNet-MS.
- Ablation study: When trained with the same pixel-level segmentation labels, the multi-task setting still outperforms the single-task setting.The comparison is between SSNet-MS and SSNet-MM.
- Ablation study: Jointly learning semantic segmentation produces comparable saliency performance to single-task training, with higher maximum F-measure but worse MAE.The authors conclude that segmentation learning does not harm saliency detection.
5. Conclusion
The paper presents SSNet, a single-model framework that jointly learns saliency detection and weakly supervised semantic segmentation. It reports state-of-the-art performance across both task settings.
- SSNet jointly learns saliency detection and weakly supervised semantic segmentation with a single model.The framework is designed as a unified learning approach for both tasks.
- The method makes fuller use of segmentation cues from saliency annotations than WSSS methods relying on pre-trained saliency models.It also avoids the pre-processing configuration used by those methods.
- The framework is easier to train than WSSS methods exploiting pre-trained saliency detection models.
- The method provides more informative results than existing fully supervised saliency detection methods.
- Experiments show state-of-the-art performance among fully supervised saliency detection methods and weakly supervised segmentation methods.