Source-linked AI summary
Visual Saliency Based on Multiscale Deep Features
Guanbin Li, Yizhou Yu
TL;DR
Visual saliency modeling needs challenging datasets and models that capture multiscale, semantically rich visual information. This paper develops a CNN-based saliency framework with spatial coherence refinement and multilevel segmentation fusion, achieving leading benchmark performance and introducing the HKU-IS dataset.
Problem
Existing large-scale saliency training data, particularly MSRA-B, are less challenging because images commonly contain one salient object and strong center or boundary priors.
Method
The paper combines multiscale CNN features with fully connected saliency layers, spatial coherence modeling, and learned linear fusion of saliency maps from multiple segmentation levels.
Results
The method achieves the highest precision across almost the entire recall range on all evaluated datasets and the best overall F-measure, including 86.4% precision and 87.0% recall on MSRA-B.
Takeaways & Limitations
The work provides a complete CNN-based saliency framework and a publicly available challenging HKU-IS dataset for saliency research and evaluation.
Abstract
from arXiv · showhide
Visual saliency is a fundamental problem in both cognitive and computational sciences, including computer vision. In this CVPR 2015 paper, we discover that a high-quality visual saliency model can be trained with multiscale features extracted using a popular deep learning architecture, convolutional neural networks (CNNs), which have had many successes in visual recognition tasks. For learning such saliency models, we introduce a neural network architecture, which has fully connected layers on top of CNNs responsible for extracting features at three different scales. We then propose a refinement method to enhance the spatial coherence of our saliency results. Finally, aggregating multiple saliency maps computed for different levels of image segmentation can further boost the performance, yielding saliency maps better than those generated from a single segmentation. To promote further research and evaluation of visual saliency models, we also construct a new large database of 4447 challenging images and their pixelwise saliency annotation. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks, improving the F-Measure by 5.0% and 13.2% respectively on the MSRA-B dataset and our new dataset (HKU-IS), and lowering the mean absolute error by 5.7% and 35.1% respectively on these two datasets.
1. Introduction
The paper develops a CNN-based visual saliency model that learns contrast using multiscale deep features and proposes a complete framework with refinement, segmentation fusion, and a challenging dataset.
- Visual saliency models estimate attention directed toward image regions and support tasks including cropping, retargeting, and summarization.
- CNN features provide hierarchical semantic information suited to saliency because attention varies across object categories.
- The model extracts CNN features from three nested windows to compare each region with its neighborhood and the whole image.
- A fully connected neural network regresses each region’s saliency score from concatenated multiscale features trained on labeled saliency maps.
- The complete framework integrates the CNN model with spatial coherence refinement and multi-level image-segmentation maps.
- HKU-IS is introduced as a publicly available challenging dataset containing images with multiple salient objects and more general spatial distributions.
2. Saliency Inference with Deep Features
The saliency architecture combines three CNN feature scales with fully connected layers that predict region saliency from object, neighborhood, and whole-image context.
- The architecture places two fully connected hidden layers and one output layer on top of three deep convolutional neural networks.
- Multiscale Feature Extraction: Each CNN extracts features from nested windows covering the region, its immediate neighbors, and the entire image.
- Multiscale Feature Extraction: Irregular regions are represented by masked bounding boxes warped to 227x227 pixels, producing a 4096-dimensional feature vector.
- Multiscale Feature Extraction: Neighborhood and whole-image features supply contrast, uniqueness, and position cues absent from the region feature alone.
- Neural Network Training: The fully connected network is trained as a regressor on regions with at least 70% of pixels sharing the same binary saliency label.
- Neural Network Training: The penultimate layer yields a fine-tuned saliency feature, while the final logistic regression layer achieves state-of-the-art performance on the evaluated datasets.
3. The Complete Algorithm
The complete algorithm refines region-level predictions for spatial coherence, applies the process across segmentation scales, and linearly fuses the resulting saliency maps.
- 3.2. Spatial Coherence: A superpixel refinement minimizes a cost balancing fidelity to initial scores with spatial coherence across superpixels.
- 3.2. Spatial Coherence: Adjacent superpixels are connected in a weighted graph, while non-adjacent distances use shortest paths through that graph.
- 3.2. Spatial Coherence: Spatial coherence weights are larger within homogeneous regions and smaller across strong edges.
- 3.3. Saliency Map Fusion: The neural model and refinement are applied at M segmentation levels, producing maps that represent salient parts at different granularities.
- 3.3. Saliency Map Fusion: The final saliency map is a linear combination of the level-specific maps, with weights learned by least squares on a validation dataset.
- 3.3. Saliency Map Fusion: The authors report that linear fusion produces quality comparable to more complicated saliency-fusion techniques in this setting.
4. A New Dataset
The paper introduces HKU-IS to address the limited challenge of existing saliency data, selecting difficult images and retaining those with consistent annotations. The resulting dataset contains 4447 images with high-quality pixelwise annotations and more challenging visual structure than MSRA.
- The dataset targets a limitation of MSRA-B, whose images often contain one salient region away from the boundary and become less challenging with center and boundary priors.
- HKU-IS was constructed from images containing multiple disconnected salient objects, boundary-touching salient objects, or color contrast below 0.7.These criteria were used to select challenging images for saliency research and evaluation.
- Three people independently annotated all 7320 initially collected images using an interactive segmentation tool to reduce label inconsistency.Each annotation took 1–2 minutes on average, and annotation spanned more than three months.
- Label consistency was defined as the ratio of pixels marked salient by all three annotators to pixels marked salient by at least one annotator.
- 4447 images remained after excluding images with label consistency C < 0.9, and their ground-truth saliency maps used majority labels from the three annotations.
- HKU-IS contains 50.34% images with multiple disconnected salient objects and 21% with boundary-touching salient objects, compared with 6.24% and 13% in MSRA.Its mean color contrast is 0.69, compared with 0.78 for MSRA.
5. Experimental Results
The experiments evaluate MDF against established saliency methods across public benchmarks and the new HKU-IS dataset using precision-recall, adaptive-threshold metrics, and MAE. MDF achieves the strongest overall performance, with especially large gains on challenging datasets, while multiscale components and multilevel fusion improve results.
- Datasets and protocol: MDF is evaluated on MSRA-B, SED, SOD, iCoSeg, and the new HKU-IS dataset against multiple state-of-the-art saliency methods.MSRA-B and HKU-IS are partitioned for training, validation, and testing; smaller datasets are evaluated using a trained model.
- Qualitative comparison: MDF performs well on challenging cases involving disconnected objects, boundary-touching objects, cluttered backgrounds, and low object-background contrast.The authors attribute stronger advantages on more challenging datasets to multiscale CNN features characterizing contrast relationships across image regions.
- Quantitative comparison: On HKU-IS, MDF increases F-measure from 0.71 to 0.80 over DRFI, while improving precision by 9% and recall by 5.7%.The paper reports similar conclusions on the other evaluated datasets.
- Quantitative comparison: MDF lowers MAE by 5.7% on MSRA-B, 26.3% on iCoSeg, 17.1% on SOD, and 35.1% on HKU-IS relative to the corresponding second-best methods.MAE measures pixelwise distance between the predicted saliency map and ground truth and is used to provide a more balanced evaluation.