PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient Object Detection in the Deep Learning Era: An In-Depth Survey

Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, Haibin Ling, Ruigang Yang

arXiv:1904.09146v5cs.CV

TL;DR

Salient object detection requires identifying complete visually salient objects, but deep SOD needs systematic understanding across models, datasets, metrics, and difficult conditions. The paper surveys deep SOD through multiple taxonomies, benchmarks representative models, constructs attribute-rich evaluations, and studies robustness and generalization. It reports broad performance advantages for deep methods while identifying dataset, annotation, and visual-fixation limitations that remain open.

  • Problem

    Deep SOD has advanced rapidly, but its models, datasets, evaluation metrics, robustness, transferability, and behavior across conditions require comprehensive analysis.

  • Method

    The paper conducts a comprehensive survey, taxonomy, benchmark, attribute-based evaluation, perturbation and adversarial robustness analysis, and cross-dataset generalization study.

  • Results

    Deep SOD models significantly outperform heuristic methods overall, while performance is saturated on some datasets and weaknesses remain for small objects and indoor scenes.

  • Takeaways & Limitations

    The survey provides comparative evidence and identifies robustness, dataset bias, annotation strategy, and links between SOD and fixation prediction as research directions.

  • Takeaways & Limitations

    The survey restricts coverage to single-image SOD and excludes RGB-D SOD, co-saliency detection, and video SOD.

Abstract

from arXiv · show

As an essential problem in computer vision, salient object detection (SOD) has attracted an increasing amount of research attention over the years. Recent advances in SOD are predominantly led by deep learning-based solutions (named deep SOD). To enable in-depth understanding of deep SOD, in this paper, we provide a comprehensive survey covering various aspects, ranging from algorithm taxonomy to unsolved issues. In particular, we first review deep SOD algorithms from different perspectives, including network architecture, level of supervision, learning paradigm, and object-/instance-level detection. Following that, we summarize and analyze existing SOD datasets and evaluation metrics. Then, we benchmark a large group of representative SOD models, and provide detailed analyses of the comparison results. Moreover, we study the performance of SOD algorithms under different attribute settings, which has not been thoroughly explored previously, by constructing a novel SOD dataset with rich attribute annotations covering various salient object types, challenging factors, and scene categories. We further analyze, for the first time in the field, the robustness of SOD models to random input perturbations and adversarial attacks. We also look into the generalization and difficulty of existing SOD datasets. Finally, we discuss several open issues of SOD and outline future research directions.

1 INTRODUCTION

SOD detects whole visually salient object regions, differing from fixation prediction, and deep learning has driven rapid progress since its introduction in 2015. This survey systematically reviews deep SOD, benchmarks models, and examines datasets, robustness, transferability, attributes, and open issues.

  • History and Scope: Deep learning entered SOD in 2015 and quickly achieved superior performance over traditional solutions.
  • Survey Scope: It analyzes representative datasets and evaluation metrics, benchmarks deep and non-deep models, and studies attribute-specific strengths and weaknesses.
  • Open Issues: The survey investigates robustness to random perturbations and adversarial attacks, dataset generalization and difficulty, and future research directions.
  • SOD detects whole attentive object regions, whereas fixation prediction estimates sparse eye-fixation positions.
  • Scope and Limitations: Its scope is restricted to single-image SOD, leaving RGB-D SOD, co-saliency detection, and video SOD separate.
  • Survey Scope: The survey taxonomizes deep SOD models by network architecture, supervision, learning paradigm, and object- versus instance-level detection.

2 DEEP LEARNING BASED SOD MODELS

Deep SOD maps images to continuous saliency maps learned from ground-truth annotations and is organized through complementary taxonomies. Architectures evolved from subunit-based MLPs toward dominant FCNs, alongside hybrid and Capsule approaches.

  • Problem Formulation: An SOD model maps an input image I to a continuous saliency map S=f(I), trained by minimizing prediction error against ground-truth masks.
  • Taxonomies: The survey categorizes deep SOD by network architecture, supervision level, learning paradigm, and object- or instance-level operation.
  • Network Architecture: Deep SOD architectures comprise MLP-based, FCN-based, hybrid network-based, and Capsule-based methods.
  • MLP-Based Methods: MLP methods process superpixels, patches, proposals, or bounding boxes individually, limiting spatial information and increasing computation.
  • FCN-Based Methods: FCN methods enable end-to-end spatial representation learning and fast feed-forward prediction, becoming the dominant SOD architecture.
  • FCN-Based Methods: FCN variants include single-stream, multi-stream, side-fusion, bottom-up/top-down, and branched networks, while hybrids combine pixel- and region-level cues.
  • Level of Supervision: Most deep SOD models use pixel-level human annotations, but their cost and tendency to overfit motivate weaker supervision.

3 SOD DATASETS

SOD datasets have evolved from small, simple images with coarse annotations to larger, more diverse datasets with complex scenes, multiple salient objects, and precise pixel-wise labels. Specialized datasets add annotations for counting, instances, categories, eye fixation, and challenging factors.

  • Modern datasets: Recent standard datasets include challenging backgrounds, multiple salient objects, boundary-touching objects, and disconnected objects.Examples include SOD, ECSSD, DUT-OMRON, PASCAL-S, and HKU-IS.
  • Training datasets: MSRA10K and DUTS are large-scale datasets widely used to train deep SOD models.MSRA10K contains 10,000 images, while DUTS contains 10,553 training and 5,019 test images.
  • Specialized datasets: Specialized datasets support SOD subitizing, instance-level detection, geographic and object-tag analysis, eye-fixation studies, and category- or challenge-aware evaluation.SOS labels images by salient-object count, ILSO provides instance annotations, and SOC includes categories, instance masks, and challenging factors.
  • Dataset evolution: Modern SOD datasets are generally larger, more complex, and more general than early datasets, which often contained one or two salient objects and simple backgrounds.Early datasets also commonly used rough bounding-box annotations, whereas modern datasets typically provide precise pixel-wise ground truths.
  • Dataset evaluation: Performance on early datasets has become saturated, while modern datasets offer more demanding and reliable evaluation settings.Some special datasets additionally include background-only images and finer-grained annotations.

4 EVALUATION METRICS

SOD evaluation uses pixel-wise and structure-aware metrics, each capturing different aspects of prediction quality. The survey recommends combining F-measure, S-measure, and MAE for more complete evaluation.

  • Metric families: PR, F-measure, MAE, and Fbw evaluate pixel-wise errors, whereas S-measure and E-measure incorporate structural cues.F-measure and E-measure assess binarized maps; PR, MAE, Fbw, and S-measure assess non-binary maps.
  • Pixel-level metrics: F-measure combines precision and recall through a weighted harmonic mean, with β^2 empirically set to 0.3 to emphasize precision.Predictions can be thresholded across [0−255] to form a precision-recall curve.
  • Structure-aware metrics: Fbw extends F-measure by weighting errors according to their locations and neighborhood information, while S-measure compares object- and region-aware structural similarity.E-measure simultaneously considers global means and local pixel matching.
  • Metric limitations: PR, F-measure, and Fbw ignore true-negative assignments, while MAE accounts for them but performs poorly when salient objects are small.S-measure is more popular than E-measure because SOD focuses on continuous predictions.
  • Recommendations: The survey recommends maximal Fβ, S-measure, and MAE to balance popularity, metric characteristics, and evaluation completeness.These metrics cover complementary aspects of binarized, structural, and pixel-wise prediction quality.

5 BENCHMARKING AND EMPIRICAL ANALYSIS

The survey uses empirical analyses to examine key SOD challenges beyond aggregate benchmarking. It studies attribute-specific performance, robustness to random and adversarial perturbations, and dataset generalizability and difficulty.

  • Empirical analyses: The empirical analysis evaluates SOD models across object, scene, and challenging-factor attributes, random input perturbations, adversarial samples, and dataset generalizability and difficulty.These analyses are conducted after large-scale benchmarking to expose model strengths and limitations under specific conditions.

5.1 Quick Overview

Deep SOD models outperform heuristic methods overall, but performance saturates on some datasets and remains uneven across attributes and perturbations. The overview identifies vulnerabilities involving small objects, indoor scenes, noise and blur, adversarial attacks, and dataset limitations.

  • Overall benchmarks: Deep SOD models significantly outperform heuristic methods overall, although performance on some datasets has become saturated.The survey identifies models,,, and as current state-of-the-art methods.
  • Attribute-based analysis: Deep methods have significant advantages on semantic-rich objects such as animals, while both deep and non-deep methods struggle with small salient objects.Indoor scenes are especially challenging for application scenarios.
  • Random perturbations: Deep methods are more sensitive than heuristic methods to random input perturbations, with both method types more robust to rotation than Gaussian blur or Gaussian noise.The comparison is reported in the survey’s random-perturbation analysis.
  • Adversarial attacks: Adversarial attacks cause drastic performance degradation in deep SOD models and are worse than random perturbations.The survey reports this result from its adversarial-attack analysis.

5.2 Performance Benchmarking

The benchmark compares 44 deep SOD models with three classic methods across six datasets using maximal Fβ, S-measure, and MAE. Deep models substantially outperform heuristic methods, but gains have saturated on some older datasets.

  • Evaluation caveat: Benchmarking results can differ from scores reported in papers when evaluated using finally released saliency maps.This motivates an open and standardized evaluation system for consistent results and conclusions.
  • Overall results: Deep SOD models greatly outperform conventional heuristic methods, reflecting stronger learning of visually salient patterns.Performance has generally increased since 2015 with the progress of deep learning.
  • Overall results: Performance on older datasets such as ECSSD and HKU-IS tends to be saturated, while improvement rates declined after 2018.The authors identify this pattern as motivating more effective model designs and new machine learning technologies.
  • Top methods: PoolNet, BANet, EGNet, and SCRN are nominated as four state-of-the-art methods with promising performance across diverse datasets.The nomination follows the benchmark comparison of 44 deep SOD models.

5.3 Attribute-Based Study

The attribute-based study uses a hybrid benchmark with rich annotations to analyze how SOD models behave across object types, challenges, and scenes. Deep models excel on semantically rich objects but both model groups struggle with small salient objects and complex environments.

  • Evaluation design: Table 7 compares heuristic and deep models on subsets defined by salient-object, challenge, and scene attributes.D-avg aggregates DGRL, PAGR, and PiCANet; ND-avg aggregates HS, DRFI, and wCtr.
  • Benchmark construction: The hybrid benchmark contains 1,800 images sampled from six SOD datasets, annotated for object types, challenging factors, and scene categories.It supports the attribute-based, random-perturbation, and adversarial-perturbation experiments.
  • Object categories: Deep models significantly outperform non-deep models on almost all object categories except NatObj, while traditional methods struggle particularly with Human.NatObj is difficult for deep models, likely because of few training samples and complex topologies; Animal is easiest for them because of strong semantics.
  • Scenes and model strengths: Deep models show strong advantages on semantically rich objects and complex shapes, and narrow performance gaps between Indoor and Natural scenes.Both deep and heuristic methods find Natural easiest, while Indoor is harder than Urban because of uneven illumination and greater scene complexity.
  • Remaining challenges: Small salient objects remain difficult for both deep and non-deep methods, and SOD utility in real, especially complex, environments remains limited.The study uses top- and bottom-100 predictions to examine how attribute distributions relate to model difficulty.

5.4 Robustness Against General Input Perturbations

The study tests SOD stability under blur, noise, rotation, and grayscale conversion. Heuristic methods are less sensitive overall, while deep models are especially vulnerable to strong blur and noise but can benefit from suitable architectural designs.

  • Perturbation protocol: Robustness is evaluated by comparing predictions on randomly perturbed images with the same underlying content.The perturbations include Gaussian blur, Gaussian noise, rotation, and grayscale conversion.
  • Perturbation protocol: The experiments use blur sigmas of 2 and 4, noise variances of 0.01 and 0.08, rotations of ±15°, and grayscale conversion with rgb2gray.These settings cover multiple perturbation types and noise magnitudes.
  • Comparative robustness: Heuristic methods are less sensitive overall, rarely affected by rotation but performing worse under strong blur, strong noise, and grayscale inputs.The compactness and abstractness of superpixels are proposed as an explanation.
  • Deep-model behavior: Deep methods suffer most under Gaussian blur and strong Gaussian noise but remain relatively robust to rotation.The authors associate the blur and noise sensitivity with damage to shallow-layer features and rotation robustness with pooling-induced invariance.
  • Architecture-specific result: PiCANet shows excellent robustness across Gaussian blur, Gaussian noise, and rotation among the three tested deep models.The authors attribute this to its effective non-local operation, suggesting network design can improve robustness.

5.5 Robustness Against Manually Designed Input Perturbations

The paper evaluates deep SOD models against manually designed adversarial perturbations and tests whether attacks transfer across networks. The models are vulnerable to targeted attacks, while transfer between different architectures is comparatively weak.

  • Motivation: The study examines adversarial robustness because visually imperceptible perturbations can produce substantially different predictions and SOD attacks are rarely explored.The motivation is strengthened by SOD’s use in security systems and commercial projects.
  • Attack protocol: A modified Dense Adversary Generation attack is applied to SRM, DGRL, and PiCANet on the hybrid benchmark.Average perturbation perceptibility values are 3.54×10^-3, 3.57×10^-3, and 3.51×10^-3, respectively.
  • Adversarial vulnerability: The three tested deep SOD models are vulnerable to adversarial input perturbations that prevent reliable salient-object candidates.Compared with random perturbations, adversarial inputs cause larger degradation, consistent with their closer proximity to decision boundaries.
  • Transferability: Adversarial perturbations cause severe performance drops on targeted models but much less degradation on other architectures.The authors report weak transferability between models with different network structures.
  • Transferability: Weak cross-model transfer may reflect orthogonal gradient directions, although ensemble-generated attacks could achieve better non-targeted transferability.The latter possibility is identified as a potential threat to deep SOD models.

5.6 Cross-Dataset Generalization Evaluation

The survey evaluates how well SOD models transfer across datasets using a standard bottom-up/top-down model and a controlled cross-dataset protocol. Results show substantial dataset-dependent differences in difficulty and generalization, motivating dataset-specific recommendations.

  • Evaluation Protocol: Cross-dataset evaluation trains a typical SOD model on one dataset and tests it on the others, comparing results with the native test set.The model uses a bottom-up/top-down architecture; six datasets are evaluated under a controlled training/testing protocol.
  • Evaluation Protocol: Max F measures performance when each dataset serves separately as the training source and test target.Table 11 organizes training datasets by rows and testing datasets by columns; diagonal entries represent same-dataset evaluation.
  • Findings: Same-dataset training and testing achieves the best results, while cross-dataset performance reveals how difficult each dataset is and how well it generalizes.Column-wise comparisons indicate test difficulty, whereas row-wise comparisons indicate generalization ability.
  • Findings: SOC is the most difficult dataset, with Mean others 0.614, whereas MSRA10K is the easiest, with Mean others 0.811.MSRA10K nevertheless generalizes worst, with a Percent drop of 17%.
  • Implications: DUTS has the best generalization ability, with the lowest Percent drop of −16%.The survey recommends DUTS for training and SOC for assessing worst-case performance; DUT-OMRON and DUTS-test are also considered difficult.

6 MORE DISCUSSIONS

The discussion identifies limitations in current SOD model design, data collection, saliency ranking, visual-fixation links, supervision, pretraining, and efficiency. It frames these limitations as open directions for more human-consistent, robust, transferable, and practical SOD systems.

  • 6.1 Model Design: Hand-designed network topologies and standard binary cross-entropy losses leave room for AutoML-based architectures and losses better aligned with SOD metrics.The survey notes that network topology affects information flow, model capacity, training difficulty, and attainable performance.
  • 6.2 Data Collection: Annotation inconsistencies arise from different subjects and rules, while fine pixel-level labels are costly and vary in quality across datasets.The survey calls for consistent annotations, optimized coarse-versus-fine labeling strategies, and models robust to imperfect labels.
  • 6.3 Saliency Ranking and Relative Saliency: Current methods often regress saliency maps for benchmark scores while neglecting relative saliency among objects.Deep models struggle especially in complex, indoor, and multi-object scenes, motivating explicit reasoning about relative importance.
  • 6.4 Linking SOD to Visual Fixations: Despite strong correlations between eye movements and object saliency, most computational models treat fixation prediction and SOD as largely separate tasks.The survey links this separation to SOD-specific dataset design bias and advocates incorporating visual-attention theories for consistency and explainability.
  • 6.5 Learning SOD in a Weakly-/Unsupervised Manner: Fully supervised SOD depends on costly, time-consuming pixel-level annotations, and limited-supervision methods still lag behind fully supervised counterparts.Weakly or unsupervised learning is proposed as a way to reduce annotation demands and study the factors underlying saliency patterns.
  • 6.6 Self-Supervised Learning: Self-supervised learning from unlabeled images or videos is presented as an alternative to relying exclusively on ImageNet pretraining.The survey describes pretext tasks such as image inpainting for learning visual features.
  • 6.7 Efficient SOD for Real-World Application: Leading SOD models are complicated, creating a need for lightweight architectures that preserve accuracy and generalization in mobile and embedded applications.Model compression and knowledge distillation are proposed as routes to compact, fast models.

7 CONCLUSION

The paper presents a comprehensive deep-learning-focused SOD survey that combines taxonomy, dataset and metric analysis, benchmarking, and studies of robustness, transferability, and dataset properties. It concludes that deep learning has driven substantial progress, but efficient design, training, and inference remain under-explored.

  • Contributions: The survey systematically categorizes deep SOD models, reviews datasets and evaluation criteria, benchmarks major methods, and recommends datasets and metrics for consistent assessment.It also analyzes previously under-explored benchmarking and baseline issues.
  • Contributions: A newly compiled and annotated dataset supports analysis of model strengths and weaknesses, while perturbation and adversarial studies examine robustness.The survey additionally investigates adversarial-attack transferability and other under-explored evaluation issues.
  • Conclusion: Deep learning has produced notable SOD progress, but efficient model design, training, and inference remain open problems for research and real-world applications.The conclusion positions the survey as a guide to current state-of-the-art methods and future exploration.
Loading 1904.09146v5…