Weakly-Supervised Salient Object Detection via Scribble Annotations

Jing Zhang; Xin Yu; Aixuan Li; Peipei Song; Bowen Liu; Yuchao Dai

Source-linked AI summary

Weakly-Supervised Salient Object Detection via Scribble Annotations

Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, Yuchao Dai

arXiv:2003.07685v1Mar 17, 2020cs.CV

TL;DR

Pixel-wise annotation is costly, and scribble supervision leaves object structure and boundaries incomplete. The paper introduces a scribble-trained saliency framework with edge guidance, gated structure-aware loss, annotation boosting, and a structure-alignment metric; it outperforms weakly supervised or unsupervised methods and is on par with fully supervised methods.

Problem
Pixel-wise salient-object labels are time-consuming and laborious, while using scribbles for salient object detection had not been explored.
Method
The framework trains on the S-DUTS scribble dataset using auxiliary edge detection, gated structure-aware loss, scribble boosting, and saliency structure measure Bµ.
Results
The method significantly outperforms state-of-the-art weakly supervised or unsupervised methods and is on par with fully supervised methods.
Takeaways & Limitations
Scribble supervision can support high-quality salient object detection when structure recovery and annotation refinement are incorporated.
Takeaways & Limitations
DenseCRF fails to generate denser annotations from the very sparse scribbles.

Abstract

from arXiv · show

Compared with laborious pixel-wise dense labeling, it is much easier to label data by scribbles, which only costs 1$\sim$2 seconds to label one image. However, using scribble labels to learn salient object detection has not been explored. In this paper, we propose a weakly-supervised salient object detection model to learn saliency from such annotations. In doing so, we first relabel an existing large-scale salient object detection dataset with scribbles, namely S-DUTS dataset. Since object structure and detail information is not identified by scribbles, directly training with scribble labels will lead to saliency maps of poor boundary localization. To mitigate this problem, we propose an auxiliary edge detection task to localize object edges explicitly, and a gated structure-aware loss to place constraints on the scope of structure to be recovered. Moreover, we design a scribble boosting scheme to iteratively consolidate our scribble annotations, which are then employed as supervision to learn high-quality saliency maps. As existing saliency evaluation metrics neglect to measure structure alignment of the predictions, the saliency map ranking metric may not comply with human perception. We present a new metric, termed saliency structure measure, to measure the structure alignment of the predicted saliency maps, which is more consistent with human perception. Extensive experiments on six benchmark datasets demonstrate that our method not only outperforms existing weakly-supervised/unsupervised methods, but also is on par with several fully-supervised state-of-the-art models. Our code and data is publicly available at https://github.com/JingZhang617/Scribble_Saliency.

1. Introduction

The paper introduces weakly supervised salient object detection from sparse scribbles, addressing poor boundary localization through structure guidance and annotation boosting. It also proposes S-DUTS and a structure-aware evaluation metric, with results outperforming weakly supervised methods and matching fully supervised methods.

Pixel-wise labels improve saliency detection but require time-consuming, laborious annotation, motivating learning from sparse or noisy labels.
The proposed framework learns salient object detection from flexible scribble annotations, which mark only small portions of foreground and background.
Direct training on sparse scribbles produces poor boundary localization because scribbles do not identify complete object structure and details.
An auxiliary edge detector, gated structure-aware loss, and scribble boosting scheme provide structure guidance and expand high-confidence labels.The loss focuses structure recovery on salient regions while the boosting scheme propagates labels to larger receptive fields.
S-DUTS relabels 10,553 DUTS training images with scribbles, averaging around 3% labeled pixels and leaving the remainder unknown.
Saliency structure measure Bµ evaluates structure alignment and produces rankings more consistent with human perception than MAE.
Experiments on six benchmarks show the method outperforms state-of-the-art weakly supervised algorithms.

2. Related Work

Related work reduces annotation demands through weakly supervised and unsupervised saliency methods, while regularization helps recover structure from incomplete labels. The paper distinguishes scribble-based saliency detection from semantic segmentation through its class-agnostic setting, edge alignment, and scribble requirements.

Weakly supervised saliency methods learn from bounding boxes, image-level labels, sparse labels, or noisy annotations instead of accurate pixel-wise labels.
Weakly supervised dense prediction commonly uses graph-based regularizers such as CRF or boundary-based losses to recover incomplete object structure.
Scribble-based semantic segmentation targets class-specific objects, whereas salient object detection does not focus on object categories.
The proposed method aligns saliency-map derivatives with image structure and uses scribble boosting to support scribbles on any salient region.Semantic segmentation methods may require scribbles traversing all semantic categories, unlike this salient-object setting.

3. Learning Saliency from Scribbles

The framework learns saliency from sparse scribble annotations using separate saliency, edge, and refinement components. Structure-aware constraints and iterative scribble boosting address incomplete object coverage, boundary localization, and background-structure distraction.

Problem setup: Scribble annotations encode foreground, background, and unknown pixels, with only around 3% of training-dataset pixels labeled.The S-DUTS training data uses scribbles rather than pixel-wise labels, and unknown pixels are excluded from partial cross-entropy supervision.
Network architecture: The network combines a saliency prediction network, an edge detection network, and an edge-enhanced saliency prediction module.The SPN produces a coarse map, the EDN enhances structural information, and the ESPM refines the coarse map into an edge-preserving output.
Saliency prediction network: The SPN uses VGG16 features and DenseASPP with varying dilation rates to generate a coarse saliency map supervised by partial cross-entropy.DenseASPP enlarges receptive fields, while the loss operates on the labeled scribble pixels.
Edge detection network: The EDN combines multistage SPN features into an edge map trained with cross-entropy against edges from an existing edge detector.Each intermediate feature is projected with a 1 × 1 convolution before concatenation and edge prediction.
Gated structure-aware constraint: The gated structure-aware loss applies locally smooth, edge-aware gradient constraints within a dilated foreground-derived gate, suppressing distracting background structure.The predicted saliency map is dilated with a square kernel of size k = 11 and adaptively thresholded to form the gate; image intensity gradients preserve distinctions at edges.
Scribble boosting: Scribble boosting propagates labels to high-confidence regions, whereas DenseCRF fails to produce sufficiently dense annotations from the sparse scribbles.The confidence map identifies consistent foreground and background regions for new supervision, and training on DenseCRF-expanded scribbles remains similar to training on the original scribbles.

4. Experimental Results

Experiments evaluate the method on six benchmarks using four metrics and compare it with weakly-supervised, unsupervised, and fully-supervised methods. The method consistently outperforms weakly-supervised and unsupervised competitors, while ablations support the roles of structure-aware guidance and scribble boosting.

Datasets and setup: The method is trained on S-DUTS and evaluated on six widely used salient object detection benchmarks.S-DUTS contains scribble annotations, while evaluation uses DUTS testing, ECSSD, DUT, PASCAL-S, HKU-IS, and THUR.
Datasets and setup: Evaluation uses MAE, F-measure, E-measure, and the proposed saliency structure measure.The structure measure evaluates boundary or structure alignment in addition to conventional saliency metrics.
Comparison with the state of the art: The method consistently achieves the best performance among weakly-supervised and unsupervised methods across the four evaluation metrics.The comparison is reported on the six benchmark datasets in Table 1 and Figure 11.
Comparison with the state of the art: The method is comparable or superior to some fully-supervised models, including DGRL and PiCANet.The paper also reports stronger performance than weakly-supervised and unsupervised competitors across different thresholds on HKU-IS and THUR.
Ablation study: The gated structure-aware loss improves performance over direct scribble training and conventional smoothness loss, while the edge task reduces the Bµ measure.These ablations indicate that salient-region gating reduces structure-recovery ambiguity and edge guidance supplies structural information.
Ablation study: One iteration of scribble boosting improves performance over the initial estimate, whereas GrabCut enlargement performs worse because its pseudo-labels contain noise.The paper also reports stable performance across different scribble annotations, indicating robustness to sparse, low-overlap labeling.

5. Conclusions

The paper presents a weakly-supervised SOD network trained on the S-DUTS scribble dataset. Auxiliary edge detection, gated structure-aware loss, and scribble boosting produce structurally rich saliency maps, with performance exceeding weakly-supervised or unsupervised methods and matching fully-supervised methods.

Conclusions: The proposed network learns salient object detection from the newly labeled S-DUTS scribble dataset.This reduces the requirement for accurate pixel-wise training labels.
Conclusions: Auxiliary edge detection and gated structure-aware loss produce saliency maps with rich structure, while scribble boosting enriches the labels.The resulting structure is evaluated using the proposed saliency structure measure.
Conclusions: The method significantly outperforms state-of-the-art weakly-supervised or unsupervised methods and is on par with fully-supervised methods.The conclusion summarizes the reported results across extensive experiments.