Source-linked AI summary
The Secrets of Salient Object Segmentation
Yin Li, Xiaodi Hou, Christof Koch, James M. Rehg, Alan L. Yuille
TL;DR
Existing saliency benchmarks can emphasize stereotypical visual concepts, separating fixation prediction from salient-object segmentation and misleading algorithm design. The paper evaluates these tasks, builds a jointly annotated dataset, and uses their relationship to develop a segmentation model that improves results across three datasets.
Problem
Existing methodology overlooks the connection between fixation prediction and salient-object segmentation, while single-dataset benchmarks can overfit dataset bias.
Method
The paper augments 850 PASCAL 2010 images with eye fixations and salient-object labels, then combines fixation-based saliency models with segmentation techniques.
Results
The model outperforms the best salient-object segmentation algorithm by 11.82% on PASCAL-S, 7.06% on IS, and 2.47% on FT.
Takeaways & Limitations
The analysis supports studying salient objects through their strong correlation with fixations while moving beyond textbook examples of visual saliency.
Takeaways & Limitations
Bruce dataset masks are unsuitable for the analysis because over 30% are empty after a labeling process that forces one-object selections.
Abstract
from arXiv · showhide
In this paper we provide an extensive evaluation of fixation prediction and salient object segmentation algorithms as well as statistics of major datasets. Our analysis identifies serious design flaws of existing salient object benchmarks, called the dataset design bias, by over emphasizing the stereotypical concepts of saliency. The dataset design bias does not only create the discomforting disconnection between fixations and salient object segmentation, but also misleads the algorithm designing. Based on our analysis, we propose a new high quality dataset that offers both fixation and salient object segmentation ground-truth. With fixations and salient object being presented simultaneously, we are able to bridge the gap between fixations and salient objects, and propose a novel method for salient object segmentation. Finally, we report significant benchmark progress on three existing datasets of segmenting salient objects
1 Introduction
The paper examines fixation prediction and salient object segmentation as related but disconnected saliency tasks. It identifies dataset design bias and introduces a jointly annotated dataset and a model that connects the two tasks.
- Saliency research commonly focuses on either predicting human fixations or segmenting salient objects.Fixation prediction produces probabilistic gaze maps, whereas segmentation produces pixel-accurate salient-object masks.
- Existing methodology overlooks connections between the two saliency tasks and can overfit the bias of a single dataset.
- 850 PASCAL 2010 images were augmented with both eye fixations and salient-object segmentation labels.The authors argue that independent image acquisition and annotation can avoid dataset design bias caused by unnatural image selection.
- Joint annotations show consistent salient-object judgments, strong fixation–object correlation, and heavy bias in the widely used salient-object dataset.Top segmentation algorithms generalize poorly to more realistic images when tested beyond that biased dataset.
- A model combines fixation-based saliency models with segmentation techniques and significantly outperforms state-of-the-art segmentation algorithms on three datasets.
2 Related Works
Prior work developed separate fixation-prediction and salient-object-segmentation models despite deep connections between their tasks. Related analyses also show that ground-truth formats, object-proposal methods, and dataset biases shape their evaluation.
- Fixation prediction: Fixation models predict probabilistic maps of human eye movements using visual features and feature-interaction steps.
- Fixation prediction: Fixation algorithms are commonly evaluated with ROC AUC against recorded human eye fixations, using datasets of varying size and subject count.
- Salient object segmentation: Salient-object benchmarks evolved from bounding boxes to pixel-accurate silhouettes, motivating methods that predict full-resolution object masks.
- Task connections: Fixation and segmentation models remain isolated because their ground truths differ from sparse fixation dots to large positive pixel regions.These differing sparsity priors limit cross-task performance.
- Object proposals: Object-proposal methods generate category-independent candidate segments that can be ranked using saliency-related scores, but object detection is exhaustive while saliency is selective.
- Dataset bias: Dataset bias can arise from image selection or annotation, with center bias being a major saliency-specific influence on benchmark scores.Shuffled-AUC was proposed to normalize center-bias effects in fixation evaluation.
3 Dataset Analysis
The analysis compares fixation and salient-object datasets, evaluates inter-subject consistency and algorithm generalization, and identifies dataset design bias as a major source of misleading benchmarks. It introduces PASCAL-S with both fixation and object-label ground truth to analyze the relationship between the two tasks.
- PASCAL-S dataset: PASCAL-S derives salient-object labels from full image segmentation and aggregates multiple labelers’ selections into segment saliency values.The dataset’s labeling process supports both object-level selection and pixel-accurate masks.
- PASCAL-S dataset: PASCAL-S combines fixation data and salient-object masks for 850 images from the PASCAL VOC 2010 validation set.Eight subjects performed free viewing, while 12 subjects selected salient objects without restrictions on the number of selected objects.
- Dataset consistency: PASCAL-S and IS evaluate inter-subject consistency, while Bruce is unsuitable for analysis because over 30% of its segmentation maps are completely empty.Bruce’s sparsity is attributed to forcing labelers to choose only one object, which can eliminate images containing multiple equally salient objects after thresholding.
- Dataset consistency: Human labelers show highly consistent behavior in both eye-fixation and salient-object segmentation tasks despite saliency being considered ill-defined in complex natural scenes.The paper measures fixation consistency with AUC scores and segmentation consistency with F-measures.
- Benchmarking: The evaluation benchmarks seven fixation algorithms on five datasets and four salient-object algorithms on four datasets to assess generalization and interdataset differences.The study uses original author implementations and compares fixation prediction with salient-object segmentation.
- Dataset design bias: FT exhibits unusually strong local and global color contrast and boundary strength, creating dataset design bias by overexpressing desired saliency properties during image selection.The analysis also adjusts fixation algorithms with a fixed Gaussian center bias before comparison; after removing center and dataset design bias, fixation performance becomes competitive.
4 From Fixations to Salient Object Detection
The method combines generic object proposals with fixation-based saliency to rank candidate segments and produce salient-object masks. It achieves strong results with few segments, while its CPMC-based segmenter misses many small salient objects.
- Model: The model generates object-candidate segments with CPMC, then ranks them using fixation-based saliency and segment features.CPMC produces overlapping figure-ground candidates; the selector uses shape and fixation-distribution information.
- Model: The selector learns to predict each candidate’s intersection-over-union with ground truth from its mask and fixation map.A random regression forest scores candidate masks independently during testing.
- Upper bound: With ground-truth segments and human fixation maps, the selector reaches an F-Measure of 0.9201, with P = 0.9328 and R = 0.7989.This experiment measures the selector’s performance with a perfect segmenter.
- Results: The model consistently outperforms the original CPMC ranking function across fixation algorithms and converges faster as K increases.Its performance is already decent with K = 20 segments, whereas CPMC ranking does not converge even with K = 2004.
- Results: Using K = 20, the method improves over the best salient-object segmentation algorithm by 11.82% on PASCAL-S, 7.06% on IS, and 2.47% on FT.CPMC+GBVS gives the best results among the reported fixation combinations on all three datasets.
- Limitation: The method often captures the full salient-object region but misses many small salient objects when CPMC fails to generate corresponding segments.This limitation is attributed to CPMC’s sparse uniform seeds.
5 Conclusion
The paper introduces a dataset pairing fixations with salient-object annotations, analyzes benchmark bias, and uses the observed connection to build a new segmentation method. The method outperforms state-of-the-art salient-object segmentation algorithms across major datasets.
- Contributions: The paper provides a dataset containing both eye fixations and salient-object annotations for the same images.It uses this paired data to study the connection between fixation prediction and salient-object segmentation.
- Findings: The analysis finds that salient-object definitions are highly consistent among human subjects.This supports treating salient-object segmentation as a valid problem.
- Findings: Major salient-object benchmarks contain significant design bias from deliberately emphasizing stereotypical saliency concepts.The paper argues that evaluation should move beyond textbook examples of visual saliency.
- Method and results: The proposed method separates segment generation from fixation-based saliency scoring and outperforms state-of-the-art algorithms on all major datasets.The method combines candidate-region generation with fixation prediction to score salient regions.