PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient Object Detection: A Benchmark

Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li

arXiv:1501.02741v2cs.CV

TL;DR

Salient object detection lacks clear evidence of progress and a precise distinction from related saliency tasks. The paper benchmarks models across categories and datasets, finding rapid gains, stronger performance from task-specific models, and dataset-dependent challenges involving bias, complexity, and object existence.

  • Problem

    The field needs clearer evidence of progress and a precise distinction between salient object detection and related tasks such as fixation prediction.

  • Method

    The paper benchmarks accessible salient object detection, fixation prediction, object proposal, and baseline models across datasets, using standard measures and analyses of dataset bias and salient-object existence.

  • Results

    The benchmark finds continuous improvement over publication years, with DRFI, QCUT, RBD, ST, DSR, and MC among the top salient object detection models.

  • Takeaways & Limitations

    Task-specific salient object detection models are evaluated as distinct from related models, while center bias, scene complexity, and difficult cases remain important directions for future benchmarks and models.

Abstract

from arXiv · show

We extensively compare, qualitatively and quantitatively, 40 state-of-the-art models (28 salient object detection, 10 fixation prediction, 1 objectness, and 1 baseline) over 6 challenging datasets for the purpose of benchmarking salient object detection and segmentation methods. From the results obtained so far, our evaluation shows a consistent rapid progress over the last few years in terms of both accuracy and running time. The top contenders in this benchmark significantly outperform the models identified as the best in the previous benchmark conducted just two years ago. We find that the models designed specifically for salient object detection generally work better than models in closely related areas, which in turn provides a precise definition and suggests an appropriate treatment of this problem that distinguishes it from other problems. In particular, we analyze the influences of center bias and scene complexity in model performance, which, along with the hard cases for state-of-the-art models, provide useful hints towards constructing more challenging large scale datasets and better saliency models. Finally, we propose probable solutions for tackling several open problems such as evaluation scores and dataset bias, which also suggest future research directions in the rapidly-growing field of salient object detection.

I. INTRODUCTION

The paper distinguishes salient object detection from fixation prediction by defining it as detecting and segmenting salient foreground objects. It benchmarks the rapidly expanding field to assess progress, cross-category differences, and dataset fitting.

  • Visual saliency models pursue different goals: some predict human fixations, while others identify salient regions or objects using explicit saliency judgments.These purposes produce noticeably different saliency maps, despite the models sometimes being treated as interchangeable.
  • Salient object detection is defined as detecting attention-grabbing objects and segmenting their complete regions from the background.Its output is a pixel-level saliency map whose intensities represent the probability that pixels belong to salient objects.
  • Salient object detection has applications in computer vision, graphics, and robotics, including recognition, compression, summarization, retargeting, retrieval, and image editing.
  • The field has expanded rapidly, creating uncertainty about whether newer algorithms genuinely improve over earlier models and generalize beyond particular datasets.The benchmark also tests older high-performing models on newer datasets.
  • The study compares salient object detection, fixation prediction, and object proposal models to test whether task-specific models have an advantage across categories.A baseline is included to examine the effect of center bias in model comparisons.

II. SALIENT OBJECT DETECTION BENCHMARK

The benchmark evaluates single-image models across datasets selected for varied scale, clutter, object multiplicity, annotation characteristics, and positional bias. Dataset statistics expose differences in center bias, scene complexity, and object size that affect comparison difficulty.

  • The benchmark focuses on models that take a single image as input because public comprehensive benchmarks for multi-image saliency settings are lacking.
  • 41 accessible models are evaluated: 29 salient object detection, 10 fixation prediction, 1 objectness proposal, and 1 baseline.The baseline, Average Annotation Map, averages ground-truth annotations and often has stronger image-center activation.
  • Seven datasets are chosen for broad use, image scale, varied biases, and potential value as future benchmarks.They include MSRA10K, THUR15K, ECSSD, JuddDB, DUT-OMRON, SED2, and PASCAL-S.
  • The datasets cover large-scale collections, structurally complex natural images, cluttered multi-object scenes, and images containing two salient objects.MSRA10K contains 10,000 annotated images, while SED2 tests performance when two salient objects are present.
  • JuddDB has the smallest positional bias, while ECSSD has the shortest and SED2 the longest distances between salient-object centroids and image centers.These statistics make positional variation explicit across benchmarks.
  • Scene complexity is estimated from average super-pixel counts, while object-size statistics show larger objects in MSRA10K and ECSSD and smaller objects in SED2.

C. Evaluation Measures

The benchmark evaluates salient object detection with complementary measures for overlap, thresholded classification, continuous-map error, and spatially weighted non-binary accuracy. Because metrics can disagree, the authors primarily use PR-derived F-Measure while reporting additional scores for comprehensive comparison.

  • Four evaluation measures are used: PR, ROC, MAE, and the weighted F-measure, covering overlap, classification, continuous-map error, and spatially weighted accuracy.The weighted measure extends Fβ to non-binary values and weights errors by location and neighborhood.
  • Binarization converts a saliency map into a binary mask for precision-recall evaluation, using adaptive, fixed, or SaliencyCut-based thresholds.Fixed thresholds produce PR curves, while SaliencyCut iteratively refines an initial mask with GrabCut.
  • Fβ combines precision and recall as a weighted harmonic mean, with β^2 set to 0.3 to emphasize precision.The weighting reflects the authors’ concern that high recall can be obtained trivially by labeling the whole image as foreground.
  • ROC evaluation reports TPR against FPR across fixed thresholds, while AUC summarizes the ROC curve as a single scalar.A perfect model has AUC 1, whereas random guessing has AUC around 0.5.
  • MAE compares continuous saliency maps with binary ground truth and includes correctly identified non-salient pixels, unlike overlap-only measures.This supports evaluation when continuous saliency-map quality matters more than binary-mask quality.
  • PR and ROC scores can disagree, so the benchmark mainly bases conclusions on PR-curve scores while also reporting other measures for broader comparison.The authors note that the abundance of negative pixels makes PR curves more informative for this task.

D. Quantitative Comparison of Models

Across the benchmark datasets, DRFI is the strongest overall model by PR, ROC, Fβ, and AUC, while rankings vary more under MAE. Salient object detection models generally outperform fixation prediction and object proposal models, and center bias affects dataset-relative performance.

  • DRFI outperforms all other models on PR and ROC curves across the seven benchmark datasets, with large margins.
  • DRFI consistently ranks first across the five datasets evaluated with Fβ, followed by MC, RBD, DSR, and GMR among the five best models.
  • DRFI ranks best on AUC across all seven datasets, while all models perform above chance level (AUC = 0.5).
  • MAE rankings are more diverse: DSR, RBD, and DRFI rank highly overall, PCA performs best on JuddDB, and MC never reaches the top three.
  • On average, fixation prediction and object proposal models perform worse than salient object detection models, although COV and BMS outperform several such models across all metrics.
  • AAM outperforms no model on SED2, whereas it exceeds only a subset of models on the other datasets, indicating weaker center bias in SED2.

E. Qualitative Comparison of Models

Qualitatively, leading salient object detection models suppress background and recover most of the salient object, whereas fixation and objectness models produce outputs less suited to whole-object segmentation.

  • DRFI and DSR suppress most background while nearly detecting the whole salient object, producing higher precision and lower false positive rates.
  • Fixation models typically generate sparse, blob-like fixation maps, while objectness produces rough salient-object indications rather than complete segmentations.

III. PERFORMANCE ANALYSIS

The performance analysis presents ROC curves for models across six benchmarks as part of a broader evaluation of benchmarking models and datasets.

  • The analysis evaluates benchmarking models and datasets through several experiments, including ROC curves across six benchmarks.

A. Analysis of Segmentation Methods

Segmentation performance generally improves when sophisticated saliency maps are combined with SaliencyCut, which enforces label consistency, but this advantage weakens on datasets with multiple or ambiguous salient objects and on difficult object structures.

  • SaliencyCut combined with sophisticated salient object detectors achieves the best segmentation results on all evaluated datasets except JuddDB, PASCAL-S, and SED2.
  • SaliencyCut's graph-based segmentation and global appearance statistics benefit segmentation by enforcing label consistency.
  • The default SaliencyCut program is less optimal on SED2, PASCAL-S, and JuddDB because it outputs only the dominant salient object, conflicting with their multiple-object or ambiguous-scene characteristics.
  • Label consistency often improves pixel labeling when most salient-object pixels are detected, but complex topology, spindle components, and object-background similarity remain challenging.
  • A SaliencyCut failure case segments only part of an object because complex topology, shading, and reflection make whole-object segmentation difficult.

B. Analysis of Center Bias

Center bias substantially affects evaluation, so the benchmark tests models on 1,000 less center-biased MSRA10K images and on SED2. Most models decline off-center, while DRFI and DSR remain strongest and DRFI is comparatively robust to object location.

  • Evaluation setup: 1,000 MSRA10K images with no or less center bias were selected by thresholding salient-object centroid distance from the image center.The selected images show off-center annotation peaks and are used to test location variation.
  • Evaluation results: Most models’ performance decreases on less center-biased images; MC’s AUC falls from 0.951 to 0.888.SVO is an exception, with AUC rising from 0.930 to 0.942 and reaching second rank.
  • Evaluation results: DRFI’s differences between all MSRA10K images and the less center-biased subset are 0.05 for Fβ, 0.05 for AUC, and 0.009 for MAE.The benchmark interprets these small differences as evidence that DRFI does not rely heavily on center bias.
  • Evaluation results: DRFI and DSR outperform other models on SED2 in Fβ, AUC, and MAE, indicating greater robustness to salient-object location variation.HS again ranks second by Fβ on SED2.
  • Evaluation results: AAM and JuddDB perform significantly worse on the less center-biased MSRA10K subset and SED2, despite all models remaining above chance.SED2 has less center activation in its average annotation map, making it useful for studying center bias.

C. Analysis of Salient Object Existence

The benchmark identifies salient-object existence as an underexplored condition because most models assume at least one salient object is present. Background images expose both qualitative failures and a lack of informative quantitative evaluation measures.

  • Problem: Most salient object detection models assume that every input image contains at least one salient object.This assumption can produce suboptimal behavior on background images containing no dominant salient object.
  • Model behavior: Top models such as DRFI, DSR, and MC do not perform well on background images, where a good model should produce a dark blank saliency map.Background images consist of textures or cluttered backgrounds without a dominant salient object.
  • Evaluation: PR, ROC, Fβ, and AUC cannot be calculated on background images because the ground-truth positive labeling is empty.MAE is also uninformative because methods commonly normalize saliency maps to [0,255] during post-processing.

D. Analysis of Worst and Best Cases for Top Models

Top models share easy and difficult stimuli: they succeed when a dominant object appears against a clean background but struggle with clutter, semantic saliency, low contrast, border contact, and small objects.

  • Best cases: DRFI and MC perform substantially well when a dominant salient object appears against a relatively clean background.The top models share the same easy and difficult stimuli.
  • Worst cases: Complex scenes with cluttered backgrounds or semantically salient objects can cause failures because existing models generally lack high-level prior knowledge.DRFI is specifically reported to fail on images with faces in MSRA10K.
  • Worst cases: Small salient objects are difficult for both DRFI and MC, especially in hard cases from DUT-OMRON and JuddDB.Object size is identified as one reason for poor saliency detection.
  • Worst cases: DRFI’s contrast-based saliency cues fail when salient objects resemble their backgrounds, including hard cases from MSRA10K and ECSSD.Close object-background appearance limits the effectiveness of contrast cues.
  • Worst cases: MC fails when salient objects touch the image border because it relies on a pseudo-background prior that treats border regions as background.The benchmark gives a gorilla image in MSRA10K as an example.

E. Runtime Analysis

The benchmark compares model efficacy with runtime and finds substantial speed differences. HC is fastest, whereas DRFI is the benchmark’s best-performing but slower model.

  • Measurement: Runtime was measured over all 10K MSRA10K images at approximately 400 × 300 resolution on an Intel Xeon E5645 2.40GHz CPU with 8 GB RAM.A scatter plot of F w β scores versus running time illustrates the efficacy–efficiency trade-off.
  • Runtime results: HC is fastest at about 0.017 seconds per image, followed by GC and SR.These measurements compare processing speed across all benchmarked methods.
  • Runtime results: DRFI, the benchmark’s best model, requires about 0.697 seconds to process one image.RC, GMR, MC, and RBD show similar efficacy–runtime trade-offs.

IV. DISCUSSIONS AND CONCLUSIONS

The benchmark identifies leading salient-object models and design choices while showing substantial progress in accuracy and runtime. It also highlights remaining challenges involving dataset bias, scene difficulty, and practical deployment.

  • DRFI, QCUT, RBD, ST, DSR, and MC are the top six salient-object detection models.
  • Performance shows a continuous ascending trend over recent publication years, indicating sustained progress in salient-object detection.
  • Top models commonly use superpixels because regions support effective features, preserve object boundaries, and may reduce computation relative to pixels or patches.
  • All six top models explicitly use a background prior, which is reported as more robust than a salient-object location prior.
  • Despite strong performance, salient-object detection remains challenging because leading methods rely on location priors, degrade on difficult datasets, and can be slow.Examples include runtimes of 40.9s for CA, 56.5s for SVO, and 140s for LMLC on 400 × 300 images.
  • Future work includes evaluation scores that address dataset bias, benchmarks for multiple-input settings, active segmentation, model aggregation, and behavioral studies of saliency judgments.
Loading 1501.02741v2…