PaperlayerView exact revision on arXiv

Source-linked AI summary

What is a salient object? A dataset and a baseline model for salient object detection

Ali Borji

arXiv:1412.5027v1cs.CV

TL;DR

Salient-object benchmarks often use single-object scenes, obscuring the challenge of choosing among multiple objects. The paper links explicit saliency judgments to fixations, introduces less biased multi-object datasets, and proposes a simple baseline whose results expose failures in existing models.

  • Problem

    The paper addresses biased evaluation datasets dominated by single-object, low-clutter, center-biased scenes.

  • Method

    It studies explicit saliency judgments and fixations, constructs Bruce-A and Judd-A, and proposes a superpixel-based model that separates detection from segmentation.

  • Results

    The new datasets challenge existing models, while the simple computationally efficient baseline wins over state-of-the-art models on the authors’ data.

  • Takeaways & Limitations

    The paper suggests defining the most salient object as the one attracting the highest fraction of fixations and using less biased multi-object benchmarks.

  • Takeaways & Limitations

    The fixation-based definition can disagree with explicit judgments in some cases, and model failures include segmenting the wrong object.

Abstract

from arXiv · show

Salient object detection or salient region detection models, diverging from fixation prediction models, have traditionally been dealing with locating and segmenting the most salient object or region in a scene. While the notion of most salient object is sensible when multiple objects exist in a scene, current datasets for evaluation of saliency detection approaches often have scenes with only one single object. We introduce three main contributions in this paper: First, we take an indepth look at the problem of salient object detection by studying the relationship between where people look in scenes and what they choose as the most salient object when they are explicitly asked. Based on the agreement between fixations and saliency judgments, we then suggest that the most salient object is the one that attracts the highest fraction of fixations. Second, we provide two new less biased benchmark datasets containing scenes with multiple objects that challenge existing saliency models. Indeed, we observed a severe drop in performance of 8 state-of-the-art models on our datasets (40% to 70%). Third, we propose a very simple yet powerful model based on superpixels to be used as a baseline for model evaluation and comparison. While on par with the best models on MSRA-5K dataset, our model wins over other models on our data highlighting a serious drawback of existing models, which is convoluting the processes of locating the most salient object and its segmentation. We also provide a review and statistical analysis of some labeled scene datasets that can be used for evaluating salient object detection models. We believe that our work can greatly help remedy the over-fitting of models to existing biased datasets and opens new venues for future research in this fast-evolving field.

I. INTRODUCTION

Existing salient-object datasets often use simple, low-clutter scenes with one centered object, leaving models’ ability to select among multiple objects unclear. This paper motivates less biased datasets and separates selecting the salient object from segmenting it.

  • Applications: Salient-object detection supports applications including recognition, compression, summarization, image editing, tracking, object discovery, and human-robot interaction.The paper lists these applications as motivations for modeling human-like prioritization of objects.
  • Motivation: Current models often bypass object selection by using single-object scenes, then achieve high segmentation scores on biased datasets.These datasets commonly contain low-clutter images with one centered object.
  • Related tasks: The paper distinguishes salient-object detection from related tasks including fixation prediction, object proposals, segmentation, and general object detection.It notes that saliency detection has been treated as binary segmentation, increasing overlap with established vision problems.
  • Contributions: The proposed datasets complement prior work by using more varied, less structured scenes rather than relying on simple biased benchmarks.The authors position these datasets as a possible shift toward more versatile evaluation.
  • Dataset limitations: Many datasets derived from MSRA contain single objects, low clutter, and strong center bias, making evaluation difficult for off-center salient objects.The paper argues that adding images without addressing clutter and object count does not resolve these biases.

III. WHAT IS A SALIENT OBJECT?

The paper examines salient objects as objects people explicitly choose, and tests whether those judgments correspond to where people look during free viewing.

  • Research question: The study investigates which objects people select as salient and whether those selections agree with their eye movements.It addresses a relationship that few prior studies had directly and quantitatively confirmed.
  • Research question: The paper treats explicit saliency judgments and free-viewing fixations as two indicators of visual attention.The comparison is used to characterize salient objects from a human perspective.
  • Research question: It also examines cases where saliency judgments and fixations disagree, rather than assuming that eye movements always identify the chosen salient object.The paper presents agreement as an empirical question.

A. Salient object detection vs. fixation prediction

Salient-object detection seeks object-level silhouettes, whereas fixation prediction seeks locations people look at; the two tasks therefore require different map properties and evaluations.

  • Task distinction: Fixation prediction models estimate gaze points during free viewing, while salient-object models detect and segment objects with pixel-accurate boundaries.The paper contrasts point prediction over several seconds with object-level silhouette extraction.
  • Task distinction: Eye-tracking noise makes blurred fixation maps useful, whereas salient-object detection requires accurate object boundaries.The paper notes that saccade landing and tracking noise can make smoothing improve fixation scores.
  • Figure 2: Figure 2 compares fixation agreement and annotation agreement across Bruce-A examples using eye-low/eye-high and seg-low/seg-high labels.Eye agreement is measured with shuffled AUC, while segmentation agreement uses the equation described in Section III.B.
  • Practical overlap: Despite their conceptual differences, models for the two tasks can be used interchangeably in practice by thresholding saliency maps for proto-object segmentation.The paper gives fixation-oriented models as examples that can be adapted this way.

B. Human explicit saliency judgments

Human explicit saliency judgments show moderate agreement and significantly above-chance correspondence with free-viewing fixations, supporting fixation-based identification of salient objects.

  • Agreement: Participants’ segmentation agreement averaged r = 0.37 with standard deviation 0.17, significantly above chance.Images with several foreground objects tended to have the lowest agreement.
  • Fixation relationship: Annotation maps predicted free-viewing fixations at sAUC 0.62±0.07 versus 0.50 chance across 120 images.The result was statistically significant with t-test p < 0.05 and matched the prediction power of the ITTI98 model.
  • Bruce-A construction: The Bruce-A benchmark merges 70 participants’ annotations, normalizes each map to [0,1], and thresholds it at 0.7.The dataset focuses on prevalent man-made home supplies in indoor scenes.

IV. OUR NEW LARGE-SCALE DATASET: JUDD-A

Judd-A was constructed by annotating objects in 900 scenes and identifying the most salient object from human fixation maps. The annotations followed explicit object-segmentation rules and were consolidated through observer voting.

  • Dataset construction: Judd-A uses scenes from the Judd fixation dataset, which contains 1003 scenes viewed freely by 15 observers.The authors used fixation data to determine which object among several annotated objects was most salient.
  • Dataset construction: Two observers manually outlined as many separable objects as possible using the LabelMe annotation tool.They were instructed to produce boundaries sufficient for recognizing the object from its polygon.
  • Annotation rules: Annotations excluded mirror reflections, inseparable object groups, and occlusion boundaries that would split one object into multiple parts.These cases occurred rarely according to the annotation procedure.
  • Saliency labeling: The object with the highest fraction of human fixations was selected as the most salient object.This operational definition links explicit object saliency to the peak of the human fixation map.

A. Dataset statistics

The datasets contain multiple, varied objects and are less centered, smaller, and more cluttered than common salient-object benchmarks. Fixation statistics support defining the most salient object by its share of scene fixations.

  • Object annotations: Annotated scenes contained 1–31 objects for the first observer and 1–24 for the second, with median counts of 3 and 4, respectively.Median object size was 10% and 9% of total image area for the two observers.
  • Fixation statistics: In about 55% of images, the most salient object attracted more than 50% of fixations, with a mean fixation ratio of 0.54.The image background received a mean fixation ratio of 0.45.
  • Object statistics: The most salient object occupied 0.1%–90.2% of image area, with a median size of 10.17%.Its bounding-box aspect ratio ranged from 0.04 to 13.7, with a median of 0.94.
  • Center-bias analysis: The datasets contained 667 on-centered and 223 off-centered scenes under the authors’ central-Gaussian overlap criterion.This partition was used to scrutinize model performance and address center-bias.
  • Dataset comparison: Compared with MSRA-5K and CSSD, objects in the authors’ datasets were farther from the image center and generally smaller than 10% of the image.The authors characterize the smaller objects as making salient-object detection more challenging.
  • Scene complexity: Scene complexity was analyzed through the number of graph-based superpixels in salient objects, backgrounds, and entire scenes.The method used Felzenszwalb and Huttenlocher segmentation with σ = 1 and K = 300.
  • Object categories: People appeared in 45% of images, more than two people in 27.2%, and a person was selected as most salient in about 27%.Other frequent categories included text, animals, cars, faces, flowers, and signs.

V. OUR BASELINE SALIENCY MODEL: SALBASE

SALBASE separates saliency localization from object segmentation using a fixation map and superpixels. Its deliberately simple design provides a baseline for diagnosing dataset bias and distinguishing detection errors from segmentation errors.

  • Model purpose: SALBASE is designed to assess how well the data are explained by a simple model and to gauge state-of-the-art progress.The model is deliberately kept simple for measuring dataset bias and complexity.
  • Model pipeline: The model combines a fixation-prediction saliency map with an over-segmented region map computed using graph-based superpixels.AWS and HouNIPS provide saliency maps, while Felzenszwalb and Huttenlocher provide the superpixels.
  • Model pipeline: SALBASE normalizes the saliency map, thresholds it at β = 0.7, selects overlapping superpixels, discards boundary-touching regions, and fills holes.The selected superpixels form the final salient-object segmentation.
  • Design rationale: The model explicitly dissociates locating the salient object from segmenting its boundaries, allowing errors in the two stages to be distinguished.The paper identifies this separation as important because existing models commonly conflate the stages.
  • Limitations: SALBASE has no training stage and uses manually fixed parameters.The authors identify learned superpixel-selection rules and multiscale extensions as possible future modeling directions.

VI. MODEL EVALUATION AND COMPARISON

The evaluation compares eight state-of-the-art models across salient-object, objectness, and fixation-prediction categories using precision-recall and ROC analyses. Performance is summarized with F-measure and AUC, with α = 0.3 weighting precision more than recall.

  • Compared models: Eight state-of-the-art models are compared across three categories: salient-object detection, generic objectness, and fixation prediction.The comparison includes CBsal, SVO, PCA, Goferman, FTS, Alexe, AWS, and HouNIPS.
  • Evaluation metrics: Precision-recall curves are generated by thresholding saliency maps from 0 to 255 and computing precision and recall against ground-truth masks.F-measure is also reported from these precision-recall values.
  • Evaluation metrics: α = 0.3 weights precision more than recall in the reported F-measure.This setting follows the cited prior evaluations.
  • Evaluation metrics: ROC curves report true-positive rate against false-positive rate while varying the saliency-map threshold.The binary mask and ground truth are complemented to compute false positives.

A. Quantitative evaluation

On MSRA-5K, the proposed models perform comparably to the strongest methods, but performance drops sharply on the more complex Bruce-A dataset.

  • MSRA-5K: On MSRA-5K, the proposed models achieve F-measures above 0.70, comparable to the best models, whose maximum is about 0.73 for Alexe.CBsal, PCA, SVO, and Alexe rank highest with F-measures above 0.55 and AUCs above 0.90.
  • Bruce-A: 0.24 is the best F-measure on Bruce-A, achieved by PCA, after a dramatic performance drop across all models.The proposed SalBase-Human model reaches about 0.31 on this dataset.
  • Bruce-A: 72% is the average performance drop from MSRA-5K to Bruce-A across CBsal, FTS, SVO, PCA, and Alexe.MAP is especially degraded on Bruce-A, where its F-measure is 0.1.
  • Bruce-A: High AUCs persist on Bruce-A because the objects are small, keeping the true-positive rate high.Thus AUC and F-measure show different behavior on this dataset.

B. Analysis of saliency map thresholding

The models’ F-measures depend on the saliency threshold and segmentation granularity, with dataset-specific thresholds affecting accuracy.

  • Saliency-map thresholding: Varying saliency threshold β changes F-measure for both SalBase-Human and SalBase-AWS.The threshold controls how many superpixels are included.
  • Saliency-map thresholding: Higher scores are achievable with different parameters: Judd-A favors a lower threshold, whereas Bruce-A favors the opposite setting.The difference reflects the datasets’ differing object sizes.
  • Segmentation parameters: Both over-segmenting and under-segmenting produce lower performance than the settings used for Fig. 11.The tested extremes were σ = 1, K = 100, min = 20 and σ = 1, K = 1000, min = 800.

D. Analysis of model failure cases

Failures arise from disagreement between fixations and selected salient objects, boundary and thresholding errors, and incorrect first-stage object selection.

  • Failure causes: Four failure causes are identified: fixation–judgment disagreement, border-touching segments, thresholding errors, and incorrect fixation-based object selection.The fourth occurs when the fixation-prediction model chooses the wrong object as most salient.
  • Fixation–judgment mismatch: The assumption that the object people look at is the object they choose as most salient holds in most cases but not all.This creates errors on Bruce-A when human fixation and annotation select different objects.
  • Segmentation errors: Border-touching segments and incorrect superpixel inclusion or exclusion make determining the salient object’s extent difficult.The paper identifies superpixel selection as an area for future modeling effort.
  • Model limitations: People and text can be the most salient objects even when the employed saliency models lack person and text detectors.This contributes to failures attributed to fixation-prediction shortcomings.
  • Qualitative comparison: Qualitative maps differ in appearance: some emphasize boundaries or centers, while others are sparse, smooth, or pointy.The comparison covers 12 Judd-A scenes and includes CBsal, SVO, Goferman, PCA, FTS, AWS, and HouNIPS.

VII. DISCUSSION AND CONCLUSION

The discussion argues that salient-object benchmarks should address object selection separately from segmentation, using less biased datasets and a model that keeps those stages distinct. It also identifies wrong-object selection as a central failure mode and notes remaining dataset limitations.

  • Model design: The proposed model runs in approximately 0.2 s for 400×300 saliency and segmentation maps and is intended as a computationally efficient baseline.This timing was measured on a 3.2 GHz Intel i7 CPU with 6GB RAM using Matlab.
  • Benchmarking: The paper recommends evaluating models on less biased scenes because prior modeling focused mainly on center-located objects.The authors argue that reliance on the center-biased ASD dataset may limit how well newer models handle complex scenes.
  • Dataset limitations: No reviewed dataset contains object annotations, eye movements, and explicit saliency judgments together, motivating larger datasets that collect all three.The paper uses the fixation-map peak as the most salient object when the full set of information is unavailable.
  • Model design: The model separates salient-object detection from segmentation, enabling its detection component to be replaced for other object-selection tasks.The authors also suggest augmenting it with top-down fixation selection for active observation.
  • Failure analysis: Models often perform poorly on complex scenes because they segment the wrong object rather than merely missegmenting the selected object.The analysis identifies wrong-object selection as the main cause of the performance drop on Judd-A relative to MSRA-5K.
  • Future analysis: The datasets support joint analysis of salient-object detection, fixation prediction, and object-proposal generation.Object annotations can help test whether object centers or semantically similar objects attract more fixations.
Loading 1412.5027v1…