PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Saliency with Encoded Low level Distance Map and High Level Features

Gayoung Lee, Yu-Wing Tai, Junmo Kim

arXiv:1604.05495v1cs.CV

TL;DR

Saliency detectors using deep features achieve strong performance but can localize boundaries imprecisely. This paper encodes low-level distances between superpixels and combines them with VGG16 high-level features; it reports improved benchmark performance and roughly 0.5 seconds per ECSSD image.

  • Problem

    Deep high-level features capture objectness but are relatively weak for precise localization, leaving adjacent salient and non-salient regions difficult to distinguish.

  • Method

    The method encodes low-level distances among superpixels with 1 × 1 convolutional and ReLU layers, then concatenates the ELD-map with VGG16 features for saliency classification.

  • Results

    The method reports better performance than prior CNN methods on most datasets, with the lowest MAE and highest maximum F-measure on most datasets.

  • Takeaways & Limitations

    Low-level features can play a complementary role alongside high-level features for precise salient-region detection.

  • Takeaways & Limitations

    The method can highlight non-salient regions and struggles with small, low-contrast, boundary-touching objects underrepresented in MSRA10K training data.

Abstract

from arXiv · show

Recent advances in saliency detection have utilized deep learning to obtain high level features to detect salient regions in a scene. These advances have demonstrated superior results over previous works that utilize hand-crafted low level features for saliency detection. In this paper, we demonstrate that hand-crafted features can provide complementary information to enhance performance of saliency detection that utilizes only high level features. Our method utilizes both high level and low level features for saliency detection under a unified deep learning framework. The high level features are extracted using the VGG-net, and the low level features are compared with other parts of an image to form a low level distance map. The low level distance map is then encoded using a convolutional neural network(CNN) with multiple 1X1 convolutional and ReLU layers. We concatenate the encoded low level distance map and the high level features, and connect them to a fully connected neural network classifier to evaluate the saliency of a query region. Our experiments show that our method can further improve the performance of state-of-the-art deep learning-based saliency detection methods.

1. Introduction

Saliency detection benefits from combining deep high-level features with complementary low-level feature distances. The proposed ELD-map integrates these signals to improve boundary precision and benchmark performance while remaining fast.

  • Motivation: High-level CNN features capture objectness but provide coarse localization, making adjacent salient and non-salient regions difficult to distinguish.Deep convolutional and pooling layers blur object boundaries, while prior methods did not directly encode feature distances.
  • Method: The ELD-map directly encodes pairwise distances between superpixels using colors, color distributions, Gabor responses, and locations.It uses multiple 1 × 1 convolutional and ReLU layers as an auto-encoder for low-level feature distances.
  • Method: Concatenating the ELD-map with VGG16 convolutional features forms a composite representation for precise superpixel saliency estimation.The method generates an accurate saliency map with precise boundaries without post-processing.
  • Contribution: Low-level features complement high-level features by supporting precise detection of salient regions.The paper presents this complementarity as a central contribution of the ELD-map.
  • Results: The method reports consistent improvements across benchmark datasets over approaches using either high-level or low-level features alone.This is stated as a cross-dataset comparison against prior feature regimes.
  • Results: Around 0.5 seconds per image is reported for testing on ECSSD, enabled by reusing high-level features across query regions.The reported runtime is specific to the ECSSD dataset.

2. Related Works

Earlier saliency methods relied on handcrafted priors or deep high-level features, while this work combines both feature types through a unified classifier. Its representation uses superpixel-to-grid distance maps to support region-level saliency prediction.

  • Benchmarks: Benchmark datasets cover varied scenes and subjects, with ground truth ranging from manually segmented masks to human fixation data.Examples include MSRA10K, ECSSD, Judd, THUR15K, DUTOMRON, PASCAL-S, and FT.
  • Low-level methods: Traditional top-performing methods use handcrafted features and heuristic priors such as global contrast and boundary information.These methods often construct high-dimensional features to distinguish salient from non-salient regions.
  • Deep methods: Deep methods including MDF, MCDL, and LEGS use CNN high-level features and achieved superior results over methods using only low-level features.MDF and MCDL query individual superpixel regions to assign saliency.
  • Distance-map construction: The initial low-level distance map represents uniformly gridded image areas using the dominant superpixel’s descriptor.The resulting representation is an N × N × K feature distance map.
  • Combined approach: The proposed method combines high-level objectness with low-level superpixel similarities and evaluates them using a multilevel fully connected classifier.The authors report significant improvements over methods using either feature type alone.

3. Algorithms

The method combines an encoded low-level distance map with VGG16 high-level features to classify superpixel saliency. Controlled experiments examine the effects of feature encoding, feature combinations, and statistical low-level features.

  • Algorithm pipeline: The pipeline constructs an ELD-map from superpixel feature distances and integrates it with VGG16 high-level features for saliency classification.The ELD-map is encoded using multiple 1 × 1 convolutional and ReLU layers before concatenation with the VGG16 feature map.
  • Superpixel representation: SLIC segments images into roughly regular superpixels, whose color, texture, and location descriptors are converted into a fixed 23 × 23 grid representation.Grid cells take the descriptor of the superpixel occupying the largest area, enabling fixed-size distance maps without resizing or cropping.
  • Low-level distance map: For each query superpixel, the method computes distances to grid-cell descriptors using feature differences and Chi-square distance for color histograms.The resulting initial distance map has dimensions 23 × 23 × 54 and includes average colors as reference information.
  • ELD-map encoding: Multiple 1 × 1 convolutional and ReLU layers reduce the initial distance map to a 23 × 23 × 3 ELD-map while learning nonlinear channel combinations.The layers act as fully connected transformations across channels, and controlled experiments report improved performance from encoding.
  • Feature integration: The model reuses one VGG16 conv5_3 feature map per image, concatenates it with the flattened ELD-map, and applies two 1024-node fully connected layers.This design evaluates each queried region using fixed-length low- and high-level representations; the high-level map has 512 channels at 14 × 14 resolution.
  • Self-evaluation: The ELD-HF configuration performs best in controlled quantitative comparisons, while visual results show that combining low- and high-level features captures salient objects and precise boundaries.Histogram features also improve performance, and 1 × 1 kernels are useful when comparing ELD-HF with non-encoded LD-HF.

4. Experiment and Discussion

The experiments evaluate saliency detection across five datasets using PR, F-measure, MAE, visual comparisons, and runtime analysis. The method generally outperforms prior approaches, especially on difficult low-contrast and complicated-background scenes, but remains vulnerable when training data lacks similarly difficult examples.

  • Quantitative results: The method achieves the lowest MAE and highest maximum F-measure on most datasets, while its PR graph outperforms prior CNN-based methods including MDF and MCDL.The reported comparisons include both overlap-based and absolute-error evaluation.
  • Visual results: Visual comparisons show especially strong performance on low-contrast salient objects and complicated backgrounds, with good results on other difficult scenes.The difficult cases also include small objects, multiple salient objects, and objects touching image boundaries.
  • Failure analysis: Failure cases include highlighting non-salient regions and imprecise detection of small, low-contrast, boundary-touching salient objects.MCDL and MDF also fail to locate the salient objects precisely in the most difficult examples.
  • Runtime: The method takes around 0.5 seconds per ECSSD image because high-level features are shared and computed once for the whole image.The measurement used 400 × 300 images on a server with an Intel i7 CPU, 8GB RAM, and GTX Titan-Black.

5. Conclusion

The paper integrates low-level and high-level features through the ELD-map and VGG16 features for saliency detection. The combined method achieves state-of-the-art visual and quantitative performance, with broader CNN architectures proposed as future work.

  • The ELD-map provides stronger discriminative power than original low-level feature distances for measuring similarities or dissimilarities among superpixels.
  • Concatenating the ELD-map with VGG16 high-level features produces state-of-the-art visual quality and quantitative performance.
  • Future work will explore additional CNN architectures to further improve the method.
Loading 1604.05495v1…