PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient Object Detection: A Survey

Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, Jia Li

arXiv:1411.5878v6cs.CVcs.AIq-bio.NC

TL;DR

Salient object detection has advanced substantially, but the field still lacks a deep understanding of its achievements, evaluation issues, and dataset biases. This paper reviews 228 publications, situates the task among related vision problems, and synthesizes modeling trends, datasets, metrics, and open questions. The survey concludes that salient object detection is useful for low-cost scene reasoning, while robust performance across nearly all images remains unresolved.

  • Problem

    A comprehensive understanding of salient object detection’s achievements and issues, including its relation to neighboring vision tasks, remains lacking.

  • Method

    The paper exhaustively reviews 228 publications covering the field’s roots, tasks, techniques, modeling trends, datasets, evaluation metrics, and related areas.

  • Results

    The survey finds that deep learning methods substantially outperform conventional heuristic methods and that fully convolutional approaches outperform methods using segment-level information.

  • Takeaways & Limitations

    Salient object detection can support later scene understanding because salient objects capture attention, while its general, fast, and low-annotation properties enable low-cost processing of many images.

  • Takeaways & Limitations

    A robust algorithm that produces high-quality results for nearly all images is still missing, and salient-object judgments can be ambiguous even for humans.

Abstract

from arXiv · show

Detecting and segmenting salient objects from natural scenes, often referred to as salient object detection, has attracted great interest in computer vision. While many models have been proposed and several applications have emerged, a deep understanding of achievements and issues remains lacking. We aim to provide a comprehensive review of recent progress in salient object detection and situate this field among other closely related areas such as generic scene segmentation, object proposal generation, and saliency for fixation prediction. Covering 228 publications, we survey i) roots, key concepts, and tasks, ii) core techniques and main modeling trends, and iii) datasets and evaluation metrics for salient object detection. We also discuss open problems such as evaluation metrics and dataset bias in model performance, and suggest future research directions.

1 Introduction

Salient object detection targets visually distinctive scene regions to support downstream understanding, while remaining closely related to fixation prediction, object proposals, and semantic segmentation. The field jointly detects salient objects and segments their regions, but evaluation and modeling criteria also emphasize accuracy, resolution, and efficiency.

  • Salient regions are detected rapidly by humans and then processed in greater detail for richer high-level information.
  • Salient object detection commonly combines detecting the most salient object with segmenting its accurate region rather than evaluating these stages separately.
  • Good saliency detection requires low missed-region and false-background rates, high-resolution maps, and computational efficiency.
  • Related areas: Fixation prediction estimates human fixation points, whereas salient object detection usually segments the most salient objects; both can produce continuous-valued saliency maps.
  • Related areas: Object proposal methods generate many overlapping candidate boxes or regions, and saliency estimation can serve as a cue for objectness.
  • Related areas: Semantic segmentation assigns one of several class labels to each pixel, unlike salient object detection, which produces a binary map.

2 Survey of the State of the Art

The survey organizes salient object detection around major modeling waves and related output types. Its chronicle moves from early saliency models to binary segmentation formulations and then to deep learning, while comparing representative results across neighboring tasks.

  • The review groups related work into salient object detection models, applications, and datasets, focusing on models representing major historical waves.
  • Related outputs: The survey’s sample comparison places salient object detection beside fixation prediction, image segmentation, and object proposals using distinct model outputs.
  • Second wave: The second wave defined saliency detection as a binary segmentation problem through work by Liu et al. and Achanta et al.
  • Modeling directions: Context-aware saliency and global contrast appear as representative modeling directions in the survey’s historical overview.
  • First wave: The first wave began with the Itti et al. model, which generated broad interest across cognitive psychology, neuroscience, and computer vision.
  • Third wave: The third wave started with deep learning models, including the Li et al. model, alongside new models and datasets.

2.1 Old Testament: Classic Models

Classic salient object detection models organize around visual subsets and cue sources, progressing from block-based intrinsic contrasts to region-based representations and extrinsic information. Region-based methods improve object alignment, while boundary assumptions remain an important limitation.

  • Most approaches first identify salient subsets by computing a saliency map, then integrate them to segment the entire salient object.
  • Classic models are categorized by block-based versus region-based analysis and intrinsic versus extrinsic cues, yielding three principal subgroups plus other models.The principal subgroups are block-based intrinsic, region-based intrinsic, and extrinsic-cue models.
  • Block-based Models with Intrinsic Cues: Block-based intrinsic models use pixels or patches and commonly measure uniqueness, distinctiveness, or rarity through center-surround, frequency-tuned, multi-scale, and patch-level contrasts.Multi-scale contrast improves robustness when salient-object size is unknown, while patch-level analysis extends pixel-level contrast for segmentation.
  • Region-based Models with Intrinsic Cues: Region-based analysis addresses block-based weaknesses because high-contrast edges can replace salient objects and large blocks can poorly preserve object boundaries.
  • Region-based Models with Intrinsic Cues: Region-based intrinsic models segment images into edge-aligned regions before computing regional saliency, often using global appearance contrast weighted by spatial distance and region size.Regions can be generated with graph-based segmentation, mean-shift, SLIC, or Turbopixels; higher global contrast receives higher saliency.
  • Region-based Models with Intrinsic Cues: Pseudo-background assumptions may fail when a salient object touches the image border, motivating boundary-connectivity priors based on border contact and region extent.These scores can be integrated into a quadratic objective to optimize the final saliency map.
  • Other Classic Models: Depth-based extensions reuse established saliency hypotheses, such as global contrast on depth maps, and can incorporate stereoscopic domain knowledge about attended-region disparities.The reviewed stereoscopic approach segments the image into regions and uses the tendency of attended regions to have small or zero disparities.

2.2 New Testament: Deep Learning Based Models

Deep-learning salient object detection models divide into CCN-based and FCN-based approaches. FCNs preserve spatial information, support end-to-end point-to-point learning, reduce time cost, and recent FCN approaches outperform segment-level methods, though failures remain in difficult scenes.

  • Model categories: Deep-learning models split into CCN-based methods using MLPs and FCN-based methods that preserve spatial information.CCN methods extract region features before saliency classification, whereas FCNs retain spatial information through fully convolutional architectures.
  • CCN-based Models: CCN-based methods lose spatial information because MLPs follow CNN feature extraction and classify resized image regions.Their segment-level framework also cannot fully leverage high-level semantic information and causes global information loss.
  • Model strategies: CNN-based models use varied strategies, including superpixels, local and global context, bounding-box regions, multiscale inputs, and high- and low-level features.These approaches include region-based 1D convolution, context branches, two-branch patch processing, Fast R-CNN, and multiscale subnetworks.
  • FCN-based Models: FCN architectures increasingly use multiscale skip layers, encoder-decoder structures, and side supervision to fuse features and capture detailed information.These designs combine high- and low-level features while incorporating multilevel supervision.
  • FCN-based Models: FCN-based approaches support point-to-point learning and end-to-end training while substantially decreasing time cost compared with CCN-based models.Recent FCN methods using CNN features substantially outperform methods based on segment-level information.

3 Applications of Salient Object Detection

Salient object detection supports applications across computer vision, graphics, and robotics. Reported uses include recognition, compression, summarization, media retargeting, quality assessment, and segmentation.

  • Applications: Salient object detection is applied in computer vision, graphics, and robotics.The survey lists applications including object detection and recognition, image and video compression, video summarization, media retargeting, image quality assessment, and image segmentation.

4 Datasets and Evaluation Measures

Salient object detection datasets evolved from bounding-box annotations toward pixel-wise masks, while evaluation uses complementary thresholded and continuous-map measures. The survey highlights binarization choices and the limitations of overlap-based metrics.

  • Datasets: Early datasets used bounding boxes, whereas later datasets introduced pixel-wise binary masks and increasingly included multiple objects.Datasets with accurate masks often contain limited objects and simple backgrounds.
  • Datasets: 22 salient object datasets include 20 image datasets and 2 video datasets, using binary masks or rectangles for annotation.Some image datasets also provide fixation data collected during free-viewing tasks.
  • Evaluation Measures: Precision-recall evaluation first binarizes the saliency map, using adaptive, variable, or GrabCut-like thresholds before comparing predictions with ground truth.The adaptive threshold is twice the image’s mean saliency, while variable thresholds produce a PR curve and GrabCut-like processing selects 95% recall.
  • Evaluation Measures: F-measure combines Precision and Recall, with β2 often set to 0.3 to weight Precision more because trivially achieving full recall is possible.Setting the entire map to foreground can achieve 100% recall.
  • Evaluation Measures: ROC plots true-positive rate against false-positive rate across thresholds, while AUC summarizes the curve as one scalar.A perfect model has AUC 1, whereas random guessing scores around 0.5.
  • Evaluation Measures: MAE complements overlap-based measures by evaluating continuous saliency maps and accounting for pixels correctly assigned as non-salient.Both the predicted map and binary ground truth are normalized to [0, 1].

5.1 Design Choices

The survey finds that salient object detection has explored many design choices across heuristic and learning-based methods, with deep learning achieving strong performance while retaining practical questions about simplicity. CNN research emphasizes deep supervision, encoder-decoder structures, and richer feature use.

  • Design Choices: Hundreds of classic and deep learning methods have explored many design choices, yet substantial room for improvement remains.The survey’s method summary identifies recurring design choices relevant to future algorithm development.
  • Heuristic vs. Learning From Data: Deep learning methods greatly outperform conventional heuristic methods by learning large amounts of extrinsic information from large datasets.The survey notes that learning-based methods trained on a small subset of MSRA5K still outperform others across differing datasets.
  • Heuristic vs. Learning From Data: Hand-crafted feature methods have limited generalization in cluttered scenes and can produce incomplete salient objects through over-segmentation.CNN-based models address these problems to some degree, including in complex scenes.
  • CNN Architectures: Deep supervision strengthens features at different layers, while encoder-decoder architectures progressively back-propagate high-level features for segmentation.Both are identified as promising CNN-based choices for further exploration.

5.2 Dataset Bias

Datasets enable training and comparison but represent only samples of an unlimited application domain and therefore contain bias. The survey identifies selection, capture, and negative-set biases as important dataset concerns.

  • Dataset Bias: Datasets provide large-scale training data and enable algorithm comparisons, but each samples an unlimited application domain and contains bias.The survey treats dataset utility and dataset bias as simultaneous properties.
  • Dataset Bias: Selection bias favors particular image types, producing qualitatively similar images and strong color contrast in frequently used salient-object benchmarks.The survey recommends independent image selection and annotation processes and detecting salient objects before segmenting them.
  • Dataset Bias: Negative-set bias arises from insufficiently rich and unbiased negative examples and can incorporate annotators’ preferences for particular object types.Including a variety of images is encouraged to reduce this concern.
  • Dataset Bias: Center-positioned predictions can score higher than many fixation-prediction models, illustrating how dataset structure can affect evaluation.The passage reports this comparison without establishing a causal mechanism.

5.3 Future Directions

Future work should extend salient object detection beyond predominantly single-image, object-agnostic settings and address evaluation questions. The survey points toward richer benchmarks, instance-level modeling, multimodal cues, and collaboration with related fields.

  • Beyond Working with Single Images: Video, co-salient, depth, and light-field salient object detection remain less explored because benchmark datasets are limited.Only two public video-saliency benchmarks are cited, and they mostly contain cartoons and news with bounding boxes on key frames.
  • Beyond Working with Single Images: Integrating spatio-temporal consistency and depth is suggested as a way to use increasingly accessible multimodal data for salient object detection.The recommendation is framed around improving efficiency in these multi-input settings.
  • Instance-Level Salient Object Detection: Existing saliency models are object-agnostic and do not split salient regions into individual objects, limiting instance-level saliency capabilities.Instance-level saliency is identified as useful for image editing and video compression.
  • Instance-Level Salient Object Detection: Instance-level detection could combine object proposals with segmentation or use edge information to distinguish different salient objects.The survey presents these as two possible approaches.
  • Model Development: Advanced baseline models, deeper networks, and both high-level and low-level features are highlighted as architectural directions for improving performance.Deeper networks capture high-level semantic information, while low-level features should also be considered.
  • Open Problems: Open questions include the number of salient objects needed to represent a scene, smoothing effects on scores and rankings, center-bias evaluation, and the gap between models and humans.The survey recommends engagement with fixation prediction, scene labeling, segmentation, detection, and recognition research.

6 Summary and Conclusion

Salient object detection offers efficient, general-purpose preprocessing for large-scale image reasoning, but a robust algorithm that performs well on nearly all images remains unavailable. The field would benefit from stronger links to fixation prediction and human judgments of salience.

  • Applications: Saliency models provide general, fast, and annotation-light alternatives to special-purpose detectors, enabling low-cost processing of many images.Their outputs can support subsequent scene understanding and detailed reasoning.
  • Future directions: Connections between salient object detection and fixation prediction could improve both model types.Datasets combining salient-object judgments with eye movements are particularly desirable for studying these connections.
  • Open problems: A highly robust detector producing high-quality results for nearly all images is still missing, partly because the most salient object can be ambiguous even for humans.This ambiguity limits expectations of universally reliable salient-object detection.
  • Applications: Efficiency and simplicity allow saliency methods to process noisy Internet images at scale despite unreliable individual-image performance.Large-scale processing can support image filtering, robust applications, and unsupervised learning.
Loading 1411.5878v6…