Source-linked AI summary
Structure-Consistent Weakly Supervised Salient Object Detection with Local Saliency Coherence
Siyue Yu, Bingfeng Zhang, Jimin Xiao, Eng Gee Lim
TL;DR
Weakly supervised salient object detection needs label-efficient methods that avoid incomplete scribble supervision and complex training. The paper introduces one-round end-to-end scribble-supervised learning with local coherence, scale consistency, and feature aggregation, and reports state-of-the-art results across six benchmarks. Its consistency formulation assumes saliency predictions for differently scaled versions of an image should agree.
Problem
Scribble annotations are inexpensive and flexible but leave many pixels unlabeled and do not directly provide complete object structures, while prior methods may require extra supervision or complex training.
Method
The method uses one-stage end-to-end scribble supervision with local saliency coherence, saliency structure consistency, and AGGM feature aggregation.
Results
The approach achieves new state-of-the-art performance on six benchmarks, with average gains of 4.60% F-measure, 2.05% E-measure, and 1.88% MAE over the previous best method.
Takeaways & Limitations
The paper demonstrates that scribble-supervised SOD can use intrinsic image coherence and scale consistency to learn integral saliency maps without extra assisting data.
Takeaways & Limitations
The self-consistency formulation assumes that saliency predictions for different scales of the same image should be consistent.
Abstract
from arXiv · showhide
Sparse labels have been attracting much attention in recent years. However, the performance gap between weakly supervised and fully supervised salient object detection methods is huge, and most previous weakly supervised works adopt complex training methods with many bells and whistles. In this work, we propose a one-round end-to-end training approach for weakly supervised salient object detection via scribble annotations without pre/post-processing operations or extra supervision data. Since scribble labels fail to offer detailed salient regions, we propose a local coherence loss to propagate the labels to unlabeled regions based on image features and pixel distance, so as to predict integral salient regions with complete object structures. We design a saliency structure consistency loss as self-consistent mechanism to ensure consistent saliency maps are predicted with different scales of the same image as input, which could be viewed as a regularization technique to enhance the model generalization ability. Additionally, we design an aggregation module (AGGM) to better integrate high-level features, low-level features and global context information for the decoder to aggregate various information. Extensive experiments show that our method achieves a new state-of-the-art performance on six benchmarks (e.g. for the ECSSD dataset: F_β= 0.8995, E_ξ= 0.9079 and MAE = 0.0489$), with an average gain of 4.60\% for F-measure, 2.05\% for E-measure and 1.88\% for MAE over the previous best method on this task. Source code is available at http://github.com/siyueyu/SCWSSOD.
Introduction
The paper targets weakly supervised salient object detection with scribble annotations, addressing incomplete object structures and inconsistent predictions without extra supervision or complex training. It combines coherence-based supervision, scale consistency, and feature aggregation, achieving strong benchmark performance.
- Introduction: Scribble annotations reduce labeling cost but cannot cover whole objects or directly provide their structures.Existing approaches may introduce edge maps, extra supervision, or iterative training to recover integral structures.
- Introduction: A local saliency coherence loss supervises unlabeled points using image features and spatial proximity to learn integral salient object structures.The approach uses intrinsic image properties rather than extra edge or assisting information.
- Introduction: A saliency structure consistency loss regularizes predictions so different scales of the same image produce consistent saliency maps.The authors present this self-consistent mechanism as improving model generalization ability.
- Introduction: AGGM integrates global context with high-level and low-level features in the decoder to improve feature representations for weakly supervised SOD.Global context captures relationships among salient regions, while high- and low-level features provide semantic and spatial information.
- Introduction: 4.60% average gain in F-measure, 2.05% in E-measure, and 1.88% in MAE over the previous best method across six benchmarks.The experiments report new state-of-the-art performance among scribble-supervised SOD algorithms.
Related Work
Related work reduces saliency annotation requirements through image-level labels, bounding boxes, points, and other weak supervision, while the presented framework combines multi-level feature integration with weakly supervised learning.
- Salient Object Detection: Fully supervised salient object detection methods commonly aggregate multi-level features and may use edge supervision, but require costly pixel-level annotations.These methods target stronger feature representation and object-boundary refinement under full supervision.
- Weakly Supervised Salient Object Detection: Image-level labels have been used to reduce annotation effort, but some methods rely on saliency localization followed by multi-stage training.This line of work includes global smooth pooling and foreground inference for salient object detection.
- Weakly Supervised Semantic Segmentation: Bounding boxes and point-level labels provide alternative weak annotations for semantic or instance segmentation.Examples include box supervision and point-seed supervision for predicting segmentation masks.
- Weakly Supervised Semantic Segmentation: The proposed framework applies AGGM in decoder layers to integrate low-level, high-level, and global-context information during learning.The framework also combines local coherence and structure consistency losses with partial cross entropy, including auxiliary supervision for intermediate maps.
Methodology
The method uses an encoder-decoder network with AGGM, local saliency coherence, saliency structure consistency, and combined dominant and auxiliary losses for scribble-supervised SOD.
- Network Architecture: The network uses an encoder-decoder architecture whose decoder applies AGGM to propagate high-level, low-level, and global context information.AGGM learns feature importance through convolution and global average pooling before normalization and aggregation.
- Local Saliency Coherence Loss: Local saliency coherence enforces similar saliency scores for nearby pixels with similar features, propagating scribble labels to unlabeled regions.The loss uses adjacent points within a k × k neighborhood and a filter based on pixel position and RGB color.
- Saliency Structure Consistency: The consistency mechanism regularizes predictions from different input scales so the network learns more consistent object structures and generalizes across scales.It compares saliency maps from normal and down-scaled versions of the same image using a structure consistency loss.
- Training Objective: The final training objective combines partial cross-entropy with proposed auxiliary and dominant losses, using labeled scribble pixels for supervision.The dominant loss is defined as Ldom = Lce + Lssc + βLlsc, while stage-specific auxiliary losses are balanced by λq.
Implementation Details and Setup
The model is trained on scribble-annotated S-DUTS and evaluated across six standard salient object detection benchmarks using three metrics.
- The network is trained on the scribble-annotated S-DUTS dataset.
- Evaluation covers six benchmarks: ECSSD, DUT-OMRON, PASCAL-S, HKU-IS, THUR, and DUTS-TEST.
- Performance is compared against six weakly supervised or unsupervised methods and ten fully supervised methods.
- The reported metrics are mean F-measure (Fβ), mean E-measure (Eξ), and Mean Absolute Error (MAE).
Comparison with State-of-the-arts
The method achieves state-of-the-art performance among weakly supervised or unsupervised approaches, with gains over the previous best method across six datasets and strong qualitative predictions.
- The method achieves new state-of-the-art performance among weakly supervised or unsupervised approaches under all evaluation metrics.
- 4.60% Fβ, 2.05% Eξ, and 1.88% MAE are the average gains over WSSA across six datasets.
- Predicted saliency maps are reported as more complete, precise, general across object classes, and robust to foreground-background disturbance than prior weakly supervised methods.
Ablation Study
The ablations show that the proposed losses and AGGM jointly produce the strongest results, while each component contributes under its tested setting.
- The complete method achieves the best ablation performance when all loss functions and AGGM are enabled on DUTS-TEST.
- 2.56% Fβ, 1.52% Eξ and 0.63% MAE are the average gains from AGGM on DUT-OMRON and DUTS-TEST when all losses are enabled.Without the proposed losses, AGGM contributes little compared with the baseline.
- 5.29% Fβ, 2.81% Eξ and 0.47% MAE improve when saliency structure consistency loss is added to partial cross-entropy training.The study describes this loss as regularizing partial cross-entropy and enhancing model generalization.
- SSIM produces higher scores on all three evaluation metrics than saliency structure consistency loss without SSIM.The comparison evaluates the consistency loss with and without SSIM.
- Adding local saliency coherence loss improves Fβ from 0.7584 to 0.8226, Eξ from 0.8732 to 0.8904 and MAE from 0.0589 to 0.0487.The authors associate these improvements with learning integral salient object structures without extra edge supervision.
Conclusions
The paper presents one-round scribble-supervised SOD built from coherence, consistency, and feature-aggregation mechanisms, with experiments reporting superior performance across evaluation metrics.
- The method uses local saliency coherence to supervise unlabeled points, saliency structure consistency to improve generalization, and aggregation to integrate multiple feature levels.
- Experiments report that the approach outperforms previous state-of-the-art methods under different evaluation metrics.
6 datasets
The proposed loss functions use intrinsic image properties to supervise unlabeled points without introducing extra supervision, supporting the reported multi-dataset performance.
- The loss functions use intrinsic properties of input images to supervise unlabeled points without introducing extra supervision.