PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Saliency Detection Based on Multiscale Deep CNN Features

Guanbin Li, Yizhou Yu

arXiv:1609.02077v1cs.CV

TL;DR

Visual saliency models must identify visually important image regions, but existing datasets can be insufficiently challenging for training and evaluation. This paper develops a multiscale CNN-based saliency model with complementary handcrafted features and introduces a challenging dataset; the method consistently ranks among top performers on challenging datasets.

  • Problem

    Visual saliency is a fundamental problem in psychology, neural science, and computer vision, while commonly used datasets can be insufficiently challenging for saliency research.

  • Method

    The method learns multiscale CNN features with fully connected layers, combines its discriminative deep contrast feature with handcrafted low-level features, and predicts regional saliency scores.

  • Results

    The proposed models consistently rank among the top performers across challenging datasets, with advantages becoming more apparent as dataset difficulty increases.

  • Takeaways & Limitations

    The work provides a saliency framework and challenging dataset designed to support research and evaluation of advanced saliency models.

Abstract

from arXiv · show

Visual saliency is a fundamental problem in both cognitive and computational sciences, including computer vision. In this paper, we discover that a high-quality visual saliency model can be learned from multiscale features extracted using deep convolutional neural networks (CNNs), which have had many successes in visual recognition tasks. For learning such saliency models, we introduce a neural network architecture, which has fully connected layers on top of CNNs responsible for feature extraction at three different scales. The penultimate layer of our neural network has been confirmed to be a discriminative high-level feature vector for saliency detection, which we call deep contrast feature. To generate a more robust feature, we integrate handcrafted low-level features with our deep contrast feature. To promote further research and evaluation of visual saliency models, we also construct a new large database of 4447 challenging images and their pixelwise saliency annotations. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks, improving the F- measure by 6.12% and 10.0% respectively on the DUT-OMRON dataset and our new dataset (HKU-IS), and lowering the mean absolute error by 9% and 35.3% respectively on these two datasets.

I. INTRODUCTION

The paper addresses fragile low-level saliency cues and limited challenging datasets by learning saliency from multiscale CNN features, while introducing a larger annotated dataset and an integrated framework.

  • Visual saliency predicts attention or salient regions and supports image cropping, retargeting, summarization, thumbnail generation, recognition, and tracking.
  • Handcrafted low-level contrast features can fail in complex backgrounds because they do not effectively capture semantic context.A lion hidden in bushes illustrates a salient object that low-level cues alone may miss.
  • CNNs provide automatically learned feature hierarchies and rich semantic information, motivating their use for visual saliency.The features are extracted from a CNN originally trained on ImageNet.
  • The model extracts CNN features from three nested, increasingly larger windows to estimate contrast between each region, its neighborhood, and the rest of the image.
  • Fully connected layers regress region saliency from concatenated multiscale features, with the penultimate layer serving as a discriminative high-level saliency feature.This feature can be concatenated with handcrafted low-level features to improve performance.
  • The paper constructs a large challenging dataset because existing training and testing datasets lack sufficient difficulty, and reports strong performance across existing and new datasets.The new dataset includes multiple salient objects with general spatial distributions and consistent annotations from multiple users.

II. RELATED WORK

Prior saliency methods rely on low-level, top-down, or hybrid cues, while this paper learns contrast from multiscale CNN features and adds hybrid refinement components.

  • A. Salient Object Detection: Bottom-up saliency models combine low-level attributes such as color, intensity, texture, and orientation using local or global contrast.
  • A. Salient Object Detection: Local bottom-up methods may detect edges or noise and suppress homogeneous interiors of salient objects.
  • A. Salient Object Detection: Top-down methods incorporate high-level knowledge such as objectness, object detectors, and category-specific priors.
  • A. Salient Object Detection: Center and boundary priors heuristically adjust saliency by favoring central regions or downweighting image boundaries.
  • B. Deep Convolutional Neural Networks: The proposed CNN architecture learns contrast from multiscale deep features rather than only the content of an image region.
  • B. Deep Convolutional Neural Networks: The framework combines local context, deep contrast features, multilevel segmentation, handcrafted features, random-forest regression, and fully connected CRF refinement.These additions are presented as distinguishing the method from earlier CNN-based saliency models.

III. SALIENCY INFERENCE WITH DEEP FEATURES

The saliency model extracts three-scale CNN features for each image region, combining region, neighborhood, and whole-image context before regression.

  • The architecture uses three deep CNNs and fully connected layers to infer saliency from multiscale image-region features.The CNNs process nested windows at the region, immediate-neighborhood, and entire-image scales.
  • The CNN used for extraction was pretrained on ImageNet and fine-tuned for object detection before producing region descriptors.Its second-last fully connected layer provides the feature representation used by the saliency model.
  • Feature A represents the considered region, while Feature B adds its immediate neighboring regions for contextual contrast.Feature A masks pixels outside the region within its bounding box; Feature B preserves the surrounding neighborhood.
  • Feature C is extracted from the entire image with the considered region masked, capturing region position and image-wide uniqueness.The paper identifies color and content uniqueness relative to the whole image and spatial position as important saliency cues.

B. Neural Network Training

The network learns a saliency regressor from concatenated multiscale features and exposes its penultimate representation as a discriminative Deep Contrast Feature. Handcrafted regional features are then combined with DCF to address complementary low-level contrast information.

  • Neural Network Training: The saliency network regresses region scores from a 12288-dimensional concatenation of multiscale CNN features.It contains two fully connected hidden layers and one output layer, and is trained on labeled image regions.
  • Neural Network Training: Regions with at least 70% of pixels sharing a saliency label are selected as training samples with binary targets.The training images are decomposed into regions before region labels are estimated from pixelwise annotations.
  • Neural Network Training: The penultimate neural-network layer is named Deep Contrast Feature (DCF), a fine-tuned representation whose final logistic-regression layer predicts region saliency.DCF captures semantic contrast between a region and its surrounding neighborhood and can support other regressors.
  • HDHF: Hybrid Deep and Handcrafted Feature: DCF reflects semantic information but may lack sufficient low-level contrast information, motivating integration with handcrafted low-level features.The paper reports that hybrid features combining DCF and low-level regional features can further boost performance.
  • HDHF: Hybrid Deep and Handcrafted Feature: The handcrafted descriptor uses color and texture features, including contrasts between each region, pseudo-background, and entire image.For each segmented region, the descriptor contains 39 dimensions of contrast features and segment properties.

C. Training Saliency Regressor over HDHF

The hybrid deep-and-handcrafted feature is mapped to saliency scores with a random-forest regressor, while saliency maps from multiple segmentation levels are fused by learned linear weights.

  • A random forest regressor maps each 339-dimensional hybrid deep-and-handcrafted feature to a saliency score.Training samples are regions with at least 70% of pixels sharing the same binary saliency label.
  • The method produces one saliency map for each of M segmentation levels, representing salient content at different granularities.Initial maps can come from either the neural-network model or the HDHF-based random-forest regressor.
  • The aggregated saliency map is a linear combination of level-specific maps, with weights learned by least squares on a validation dataset.The authors report that this simple fusion performs comparably to more complicated techniques in their context.

E. Spatial Coherence Based on CRF

The method refines noisy, fragmented region-level saliency maps with a fully connected CRF that encourages spatial coherence and preserves salient-object boundaries.

  • The fully connected CRF refines the aggregated map by solving a binary pixel-labeling problem for salient versus nonsalient pixels.The aggregated pixel scores initialize the unary label probabilities.
  • The CRF pairwise potential combines a position-and-color bilateral kernel with a position-only kernel.The first encourages nearby, similarly colored pixels to receive similar labels; the second removes small isolated regions.
  • CRF refinement produces smoother maps, improves pixelwise accuracy, and better preserves salient-object boundaries than fragmented initial maps.The paper attributes the noisy initial scores to imperfect segmentation and reports the comparison in Fig. 4.
  • The refinement uses color in the bilateral term for efficiency and takes less than 0.5 second on a 300 × 400 image.Mean-field inference and high-dimensional filtering are used to accelerate energy minimization.
  • V. A NEW DATASET: The new dataset begins with images selected for multiple salient objects, boundary-touching objects, complex backgrounds, or low color contrast.Three people independently annotated 7320 initially collected images using an interactive segmentation tool.
  • V. A NEW DATASET: After excluding images with label consistency C < 0.9, HKU-IS contains 4447 images with high-quality pixelwise annotations.Ground-truth labels are generated using the majority label among three annotators.

VI. EXPERIMENTAL RESULTS

The method is evaluated on several public salient-object benchmarks and the new HKU-IS dataset, with component effectiveness examined through ablation figures.

  • Datasets: The evaluation covers MSRA-B, DUT-OMRON, SOD, PASCAL-S, ECSSD, and the new 4447-image HKU-IS dataset.MSRA-B is widely used, while DUT-OMRON, SOD, PASCAL-S, and ECSSD provide additional evaluation settings.
  • Datasets: HKU-IS contains 4447 images with pixelwise salient-object annotations.
  • Evaluation protocol: The study omits SED and ICOSEG because they are considered insufficiently challenging and not widely used.
  • Evaluation protocol: The ablation analysis investigates each component of MDF, while MDF and HDHF are compared with other state-of-the-art methods.The two result sets are produced using multilevel fusion and CRF enhancement.
  • Ablation analyses: Figure 5 compares precision-recall curves and adaptive-threshold precision, recall, and F-measure for models using different S-3CNN components.
  • Ablation analyses: Figure 6 compares single segmentation levels with multilevel fusion using precision-recall curves and adaptive-threshold precision, recall, and F-measure.

B. Implementation Details

The models are trained from segmented regions of MSRA-B, producing a large collection of S-3CNN training samples for saliency estimation.

  • Training setup: The saliency models are trained on MSRA-B and tested on all other datasets.
  • Training setup: The training set contains 2500 images, with 15 segmentation levels and around 800 extracted segments per image.
  • Training setup: Around 1.9 million training samples are formed from S-3CNN feature vectors extracted from the segments.The passage states that this sample size is sufficient to train the high-dimensional models without overfitting.

C. Evaluation Criteria

The evaluation uses threshold-based precision-recall and ROC analysis, F-measure, adaptive-threshold scores, and mean absolute error to assess saliency maps.

  • Threshold-based metrics: Precision-recall and ROC curves evaluate continuous saliency maps across thresholds from 0 to 1.Precision-recall curves are averaged over each dataset.
  • Threshold-based metrics: F-measure combines precision and recall, with β^2 = 0.3 weighting precision more heavily than recall.The maximum F-measure across precision-recall pairs is reported.
  • Threshold-based metrics: The adaptive threshold is twice the mean saliency of the image, and average precision, recall, and F-measure are reported after binarization.W and H denote map width and height, while S(x, y) denotes the pixel saliency score.
  • Error metric: Mean absolute error measures the average pixelwise absolute difference between the binary ground truth and saliency map.It is included because precision-recall curves do not account for true-negative pixels.

D. Ablation Study

Ablation studies test the complementarity of multiscale features, segmentation-level fusion, CRF refinement, and CNN architecture choices.

  • S-3CNN components: The full S-3CNN feature is compared with models using individual components and selected two-component concatenations on MSRA-B.
  • S-3CNN components: The three S-3CNN components are complementary, and the saliency model learns region-contrast information from their multiscale features.
  • Multilevel fusion: 2.15% higher average precision and 3.47% higher recall are obtained by fusing 15 segmentation levels versus the best single level.
  • CRF refinement: CRF improves MDF and HDHF results across all six evaluated datasets.
  • CRF refinement: On DUT-OMRON, CRF increases F-measure by 4.7% for HDHF and 4.2% for MDF.
  • CNN architectures: Among AlexNet, VGG16, VGG19, and R-CNN features, R-CNN achieves slightly better saliency-detection performance.Joint fine-tuning produces slightly better results, but much better per-scale descriptions remain difficult to obtain.

F. The Performance of HDHF

HDHF consistently outperforms MDF, LF, and existing saliency methods across challenging benchmark datasets, with gains in both F-measure and MAE. Its advantages are especially pronounced on difficult images and datasets.

  • Ablation comparison: 2.6%: HDHF improves F-measure over MDF on DUT-OMRON, while reducing MDF's MAE by 8.5%.Against LF, HDHF improves F-measure by 12.4% and reduces MAE by 46.3% on the same dataset.
  • Qualitative comparison: HDHF and MDF produce saliency maps closest to ground truth in cases involving clutter, multiple objects, low contrast, and boundary-touching objects.These visual comparisons cover several challenging image configurations.
  • Quantitative comparison: HDHF consistently achieves top performance across benchmark datasets in precision-recall and ROC evaluations.HDHF and MDF occupy the top two positions and outperform competing methods on all benchmark datasets.
  • Quantitative comparison: On HKU-IS, HDHF raises F-measure from 0.76 to 0.86 versus MC while improving precision by 10.9% and recall by 15.3%.On DUT-OMRON, HDHF achieves 70.9% precision and 75.7% recall, compared with MC's 62.2% precision and 78.5% recall.
  • Quantitative comparison: 10.0%: HDHF improves F-measure over the best existing algorithm on HKU-IS.The corresponding improvements are 6.4%, 2.3%, 6.1%, 5.5%, and 8.1% on MSRA-B, ECSSD, DUT-OMRON, PASCAL-S, and SOD.
  • Quantitative comparison: 35.3%: HDHF lowers MAE on HKU-IS relative to the best existing algorithm.MAE reductions are also reported for MSRA-B, ECSSD, DUT-OMRON, PASCAL-S, and SOD.
  • Interpretation: The advantage becomes more obvious on more challenging datasets, where multiscale CNN features identify subtle contrast among image regions.Models trained on MSRA-B remain among the top performers on other challenging datasets.

H. Efficiency

The system combines deep and handcrafted features with multiscale segmentation aggregation and a fully connected CRF, but regional deep-feature extraction remains inefficient. Future work targets shared computation to accelerate this stage.

  • Runtime: 4 seconds: the model detects salient objects in a 400 × 300 testing image using two NVIDIA GTX Titan Black GPUs and a 3.4GHz Intel processor.Training the deep model takes around 20 hours on the MSRA-B training set.
  • Framework: The complete framework combines multilevel segmentation saliency maps with a fully connected CRF to improve spatial coherence.The framework also uses learned deep contrast features integrated with handcrafted low-level features.
  • Efficiency limitation: Regional feature extraction processes each image region independently without shared computation, creating an efficiency bottleneck.The authors identify improving deep feature extraction efficiency as future work.
  • Future work: SPPnets are proposed for acceleration by computing one convolutional feature map per image and extracting all regional features from it.This is presented as a future direction rather than an evaluated result.
Loading 1609.02077v1…