Source-linked AI summary
Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground
Deng-Ping Fan, Ming-Ming Cheng, Jiang-Jiang Liu, Shang-Hua Gao, Qibin Hou, Ali Borji
TL;DR
Existing SOD benchmarks assume images contain obvious salient objects in low clutter, yet models perform poorly on realistic daily scenes. The paper analyzes this bias, proposes the SOC dataset with realistic annotations and attributes, and evaluates CNN-based models across these challenges. The resulting benchmark is intended to narrow the gap between existing evaluations and real-world conditions.
Problem
Existing SOD datasets favor images containing at least one clearly salient object in low clutter, while lacking attributes for analyzing performance across real-world challenges.
Method
The paper identifies seven dataset requirements, constructs the SOC dataset with salient and non-salient images, category labels, instance annotations, and challenge attributes, and evaluates CNN-based SOD models.
Results
State-of-the-art models exceed 0.9 F-measure on most existing datasets but fall below 0.45 F-measure on realistic scenes; SOC also supports attribute-based assessment.
Takeaways & Limitations
SOC provides a more realistic SOD benchmark intended to support deeper analysis of model weaknesses and application-oriented development.
Takeaways & Limitations
F-measure is not suitable for evaluating non-salient object detection because all-zero ground truth makes recall and precision indistinguishable across foreground maps.
Abstract
from arXiv · showhide
We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of existing SOD datasets which assumes that each image contains at least one clearly outstanding salient object in low clutter. The design bias has led to a saturated high performance for state-of-the-art SOD models when evaluated on existing datasets. The models, however, still perform far from being satisfactory when applied to real-world daily scenes. Based on our analyses, we first identify 7 crucial aspects that a comprehensive and balanced dataset should fulfill. Then, we propose a new high quality dataset and update the previous saliency benchmark. Specifically, our SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object category annotations, each salient image is accompanied by attributes that reflect common challenges in real-world scenes. Finally, we report attribute-based performance assessment on our dataset.
1 Introduction
Existing SOD datasets favor images with obvious salient objects in low clutter, producing saturated benchmark scores that do not reflect real-world performance. The paper introduces SOC and attribute-based evaluation to provide a more realistic benchmark.
- Motivation: Most existing SOD datasets discard images without salient objects and underrepresent cluttered multi-object scenes, creating a data selection bias.This differs from realistic daily scenes, where salient objects may be absent and clutter is common.
- Motivation: Top-performing SOD models exceed 0.9 F-measure on most existing datasets but achieve below 0.45 F-measure on realistic scenes.
- Motivation: Existing datasets provide only overall performance, lacking attributes that expose model strengths and weaknesses across real-world challenges and application perspectives.
- Contributions: SOC contains 6,000 images from more than 80 common categories, including category annotations, non-salient images, and attributes such as motion blur, occlusion, and cluttered backgrounds.
- Contributions: The paper evaluates state-of-the-art CNN-based SOD models using region, pixel-wise, structure, and attribute-based performance measures.
2 Related Works
Prior SOD datasets and models provide limited coverage of realistic scenes and annotation settings. The paper benchmarks CNN-based models on a dataset designed to expose these limitations and guide future research.
- Datasets: Previous SOD datasets annotate pixel-accurate salient-object silhouettes, whereas MS COCO segments object instances without requiring saliency.
- Datasets: Early SOD datasets were limited by image count, coarse annotations, or simple scenes containing only one or two salient objects.
- Datasets: Existing datasets mostly emphasize clear salient objects in simple backgrounds, motivating a benchmark with non-salient objects, in-the-wild textures, and salient-object attributes.
- Models: The reviewed SOD models include single-task architectures using local, global, multi-context, multi-scale, contrast, recurrent, or fine-grained cues.
- Models: Multi-task models jointly use saliency with segmentation or classification, including DS, weakly supervised WSS, and MSR instance-level refinement.
- Evaluation: The paper benchmarks a large set of state-of-the-art CNN-based models on SOC to highlight current issues and future research directions.
3 The Proposed Dataset
The SOC dataset is designed to represent realistic salient-object detection scenes through non-salient images, diverse categories, challenging contrasts and locations, high-quality boundaries, and attribute annotations.
- Dataset composition: Non-salient images include texture and realistic scenes without salient objects, making SOC closer to real-world scenes and more challenging.The collection includes 783 texture images from DTD and 2,217 additional realistic images.
- Dataset composition: SOC contains 6,000 images from more than 80 categories, split evenly between 3,000 salient-object and 3,000 non-salient images.The dataset is divided into training, validation, and test sets in a 6:2:2 ratio.
- Real-world challenges: SOC evaluates salient objects across global and local color contrast, with a larger proportion of low-global-contrast objects than ILSO.Foreground and background RGB histograms are compared using χ2 distance.
- Real-world challenges: SOC analyzes object locations using normalized center and margin distances and reports that its salient objects do not suffer from center bias.The distances are normalized by half the image diagonal, yielding ro, rm ∈[0, 1].
- Annotation quality: SOC uses pixel-wise salient-object annotations produced through viewer agreement, bounding-box filtering, and manual silhouette labeling.Five viewers identify salient objects; images are retained when at least three agree with bbox IOU > 0.8, followed by accurate silhouette annotation.
- Attribute annotations: Each salient image receives non-exclusive attributes describing real-world challenges, enabling performance analysis across difficult scene conditions.The attributes include situations such as motion blur, occlusion, and cluttered backgrounds; attribute dependencies are also analyzed.
4 Benchmarking Models
The SOC benchmark evaluates sixteen SOD models using region, pixel-wise, and structural metrics, including overall and attribute-based analyses. Results show different model strengths across cluttered-scene attributes and expose limitations in evaluating non-salient images.
- Evaluation Metrics: SOC evaluation uses region similarity F, normalized mean absolute error ε, and structure similarity S to assess SOD predictions against ground truth.The overall scores are averaged across images using Eqn. (3), with S-measure used for the reported overall analysis.
- Evaluation Metrics: F-measure is unsuitable for non-salient object detection because black ground truths make different foreground maps receive the same zero score.This occurs because recall and precision are not well defined for an all-zero ground truth in the F-measure calculation.
- Overall Model Performance: NLDF achieves the best overall structure similarity among single-task models with MS = 0.818, followed by RFCN with MS = 0.814.RFCN also combines edge and superpixel cues with FCN and reports MF = 0.435 and MS = 0.814 over the overall dataset.
- Overall Model Performance: MSR achieves the highest overall performance among multi-task models, while weakly supervised WSS remains comparable to other fully supervised single-task models.The authors identify weakly supervised and multi-task approaches as promising future directions.
- Attribute-based Evaluation: Attribute-based evaluation reveals distinct model sensitivities: large objects penalize local-detail methods, small objects suffer from CNN down-sampling, and occlusions favor models combining global and local features.For Big Object scenes, DSS loses 28.9%, MC loses 20.8%, and RFCN loses 23.8%, whereas IMC improves by 3.2%; for Small Object scenes, degradation ranges from 0.3% to 5.6%.
- Attribute-based Evaluation: Heterogeneous-object scenes improve every model's performance by 3.9% to 9.7%, possibly because this attribute occupies a substantial portion of the datasets.The authors interpret this pattern as consistent with their dataset statistics.
5 Discussion and Conclusion
The paper identifies dataset bias as a central weakness in SOD evaluation and introduces SOC as a more realistic, extensible benchmark with attribute-based analysis.
- Existing SOD datasets exhibit selection bias and low-clutter assumptions, making their evaluation settings less realistic than daily scenes.The analysis identifies seven aspects that comprehensive and balanced datasets should fulfill.
- SOC is a high-quality dataset containing salient objects from daily life in natural environments.The dataset is intended to reach closer to realistic settings and evolve over time.
- SOC supports research directions including salient object subitizing, instance-level SOD, and weakly supervised SOD.These directions are enabled by the dataset’s scope and annotations.
- The paper proposes attributes to analyze SOD more deeply, investigate model strengths and weaknesses, and assess performance objectively.Attribute-based assessment examines model behavior from perspectives relevant to real-world challenges.