Source-linked AI summary
Multi-source weak supervision for saliency detection
Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, Yizhou Yu
TL;DR
Pixel-level annotations are costly, while individual weak supervision sources provide incomplete information for saliency detection. The paper combines category labels, captions, and unlabelled data through CNet, PNet, attention-transfer and attention-coherence losses, then trains SNet from pseudo labels. Experiments compare favorably with unsupervised and weakly supervised methods and outperform some supervised methods.
Problem
Pixel-level annotations are expensive, and a single weak supervision source usually lacks enough information for well-performing saliency detection.
Method
A unified framework trains CNet and PNet from category labels and captions, uses attention transfer and attention coherence, and trains SNet with their pixel-level pseudo labels.
Results
The method compares favorably with unsupervised and weakly supervised methods and performs better than some supervised methods.
Takeaways & Limitations
Diverse weak supervision sources can be integrated into one saliency-detection training framework, while only SNet is needed during testing.
Takeaways & Limitations
Many existing deep saliency methods remain dependent on expensive pixel-level annotations, defining the annotation-cost boundary motivating this work.
Abstract
from arXiv · showhide
The high cost of pixel-level annotations makes it appealing to train saliency detection models with weak supervision. However, a single weak supervision source usually does not contain enough information to train a well-performing model. To this end, we propose a unified framework to train saliency detection models with diverse weak supervision sources. In this paper, we use category labels, captions, and unlabelled data for training, yet other supervision sources can also be plugged into this flexible framework. We design a classification network (CNet) and a caption generation network (PNet), which learn to predict object categories and generate captions, respectively, meanwhile highlight the most important regions for corresponding tasks. An attention transfer loss is designed to transmit supervision signal between networks, such that the network designed to be trained with one supervision source can benefit from another. An attention coherence loss is defined on unlabelled data to encourage the networks to detect generally salient regions instead of task-specific regions. We use CNet and PNet to generate pixel-level pseudo labels to train a saliency prediction network (SNet). During the testing phases, we only need SNet to predict saliency maps. Experiments demonstrate the performance of our method compares favourably against unsupervised and weakly supervised methods and even some supervised methods.
1. Introduction
The paper motivates combining category labels, captions, and unlabelled data because each weak cue is incomplete, then proposes a unified framework with three networks and cross-network attention losses.
- Motivation: Pixel-level saliency annotations are expensive, motivating weakly supervised alternatives based on higher-level supervision.Image-level supervision reduces annotation requirements but provides less spatial information.
- Weak-supervision challenges: Category labels may highlight only discriminative object regions, whereas captions can include background details and produce inaccurate saliency maps.The paper illustrates category supervision focusing on a dog’s face and caption supervision highlighting background elements.
- Weak-supervision challenges: Combining weak cues is difficult because large-scale datasets with matched annotation types are scarce and different cues often require different network structures.The paper identifies these obstacles as reasons for designing a unified framework.
- Proposed framework: The framework uses category labels, captions, and unlabelled images with CNet, PNet, and SNet to generate and refine saliency predictions.CNet classifies objects, PNet generates captions, and SNet is trained from pseudo labels produced by the first two networks.
- Proposed framework: Attention transfer lets each network learn from the other network’s coarse saliency maps when its own corresponding annotation is unavailable.Category-labelled images supervise PNet through CNet maps, while caption-labelled images supervise CNet through PNet maps.
- Proposed framework: Attention coherence uses unlabelled data and color-similarity refinement to encourage generally salient rather than task-specific regions.The refined maps provide a shared target for CNet and PNet.
2. Related work
Prior saliency methods include handcrafted, deep-learning, and weakly supervised approaches, but existing methods commonly require pixel-level labels or rely on a single image-level cue.
- Salient objects detection: Earlier saliency detection methods used handcrafted features and heuristic priors such as center and boundary-background priors.Later methods adopted deep CNNs for feature extraction and saliency prediction.
- Salient objects detection: Deep saliency methods achieved strong performance but generally required expensive pixel-level annotations for training.This annotation requirement limits the practicality of otherwise effective approaches.
- Weakly supervised learning: Weakly supervised methods reduce hand-labelling costs by learning from image-level supervision and related coarse prediction mechanisms.The related work describes segmentation, image-level aggregation, and semantic-affinity approaches.
- Weakly supervised learning: The paper distinguishes its approach from prior methods by integrating complementary supervision cues instead of relying on a single image-level source.This integration is presented as a route toward a more robust saliency model.
3. The proposed method
The method combines classification, caption generation, and unlabelled data through attention-based subnetworks and losses, then uses their saliency maps to supervise saliency prediction.
- Network architecture: CNet predicts image categories while producing a coarse saliency map, and PNet generates captions while locating caption-relevant regions.CNet uses a fully connected layer, whereas PNet replaces it with an LSTM that generates candidate-word distributions.
- Attention module: The attention module computes region saliency scores, attended features, normalized spatial weights, and a weighted global feature for downstream prediction.The saliency scores form a map, while softmax-normalized weights constrain the contribution of all image regions to sum to 1.
- Training objectives: Four losses train the networks: category localization, caption localization, attention transfer, and attention coherence.Localization losses focus each network on task-relevant regions; transfer shares supervision across annotation types, while coherence encourages common salient regions.
- Training objectives: The category localization objective combines category log-likelihood with regularization that penalizes uniformly high saliency responses.The regularization coefficient β is set to 0.005 and limits information flowing from image regions to the classifier.
- Training objectives: Attention transfer selects positive and negative regions from one network’s saliency map to supervise the other network when its corresponding annotation is unavailable.Category-labelled images transfer CNet’s regional supervision to PNet, while caption-labelled images transfer PNet’s supervision to CNet.
- Training objectives: On unlabelled images, superpixels and low-level color similarity refine saliency seeds, and the resulting positive and negative samples supervise both networks through attention coherence.The refined ranking uses a normalized graph Laplacian; superpixels whose optimized ranking exceeds the mean are selected as positive samples.
4. Experiments
Experiments assess the framework across five benchmark datasets, ablate its components, and compare it with unsupervised, weakly supervised, and fully supervised methods. Multiple supervision sources and joint training improve results, while the full method compares favorably with the evaluated alternatives.
- Experimental setup: The method is evaluated on five benchmark datasets using Precision-Recall curves, MAE, and maximum F-measure.The datasets are ECSSD, PASCAL-S, SOD, MSRA5K, and DUT-OMRON; maximum Fβ uses β2=0.3.
- Ablation studies: CNet and PNet alone provide unsatisfactory results, while averaging their outputs performs better because the two supervision sources are complementary.The comparison examines category-label and caption supervision separately before averaging their predictions.
- Ablation studies: Joint training with attention transfer substantially outperforms simply averaging the separately trained networks.Attention transfer lets CNet benefit from captions and PNet benefit from category labels.
- Ablation studies: Adding unlabelled data with attention coherence further evaluates whether the jointly trained networks detect generally salient rather than task-specific regions.The ablation isolates the contribution of unlabelled-data regularization.
- Performance comparison: The method outperforms all compared unsupervised methods and the weakly supervised method WSS, while achieving comparable or better performance than fully supervised methods.The reported comparisons use Precision-Recall curves and score tables; the method has larger recall at the same precision and outperforms DRFI and LEGS.
5. Conclusion and future work
The paper concludes that its unified framework combines category labels, captions, and unlabelled data for weakly supervised saliency detection. Its modular design supports pseudo-label training and can be extended to additional supervision sources and semi-supervised settings.
- Conclusion: The framework combines category labels, captions, and unlabelled data through CNet, PNet, and SNet.CNet and PNet generate supervision-related saliency information, while SNet is trained using their pixel-level pseudo labels.
- Conclusion: Attention transfer shares supervision between CNet and PNet, while attention coherence encourages generally salient rather than task-specific regions.The two losses connect complementary supervision sources and regularize learning on unlabelled data.
- Conclusion: SNet produces the final saliency predictions from pseudo labels generated by CNet and PNet.The testing phase requires only SNet to predict saliency maps.
- Future work: The framework can incorporate additional supervision sources such as bounding boxes, scribbles, noisy saliency maps, and pixel-level annotations.The paper identifies these extensions as possible future directions, including semi-supervised learning.