PaperlayerView exact revision on arXiv

Source-linked AI summary

DISC: Deep Image Saliency Computing via Progressive Representation Learning

Tianshui Chen, Liang Lin, Lingbo Liu, Xiaonan Luo, Xuelong Li

arXiv:1511.04192v2cs.CV

TL;DR

DISC addresses the difficulty of accurate saliency assignment without relying on hand-crafted priors or feature engineering. It learns saliency progressively with two CNNs, combining global coarse context, superpixel refinement, and local fine prediction. The paper reports superior benchmark performance and cross-dataset generalization, while coarse-map quality remains its main limitation.

  • Problem

    Accurate saliency assignment is ill-posed across diversified natural images, and existing methods often depend on priors or hand-crafted features.

  • Method

    DISC uses two stacked CNNs for progressive coarse-to-fine learning, refining the coarse map with superpixel local context before fine-level prediction.

  • Results

    DISC significantly outperforms state-of-the-art approaches and generalizes across datasets without additional training.

  • Takeaways & Limitations

    The framework produces detail-preserved saliency maps without feature engineering or heuristic assumptions and is presented as generalizable to similar vision tasks.

  • Takeaways & Limitations

    Fine-map accuracy depends heavily on the coarse map, which can fail when foreground and background appearances are similar or backgrounds are cluttered.

Abstract

from arXiv · show

Salient object detection increasingly receives attention as an important component or step in several pattern recognition and image processing tasks. Although a variety of powerful saliency models have been intensively proposed, they usually involve heavy feature (or model) engineering based on priors (or assumptions) about the properties of objects and backgrounds. Inspired by the effectiveness of recently developed feature learning, we provide a novel Deep Image Saliency Computing (DISC) framework for fine-grained image saliency computing. In particular, we model the image saliency from both the coarse- and fine-level observations, and utilize the deep convolutional neural network (CNN) to learn the saliency representation in a progressive manner. Specifically, our saliency model is built upon two stacked CNNs. The first CNN generates a coarse-level saliency map by taking the overall image as the input, roughly identifying saliency regions in the global context. Furthermore, we integrate superpixel-based local context information in the first CNN to refine the coarse-level saliency map. Guided by the coarse saliency map, the second CNN focuses on the local context to produce fine-grained and accurate saliency map while preserving object details. For a testing image, the two CNNs collaboratively conduct the saliency computing in one shot. Our DISC framework is capable of uniformly highlighting the objects-of-interest from complex background while preserving well object details. Extensive experiments on several standard benchmarks suggest that DISC outperforms other state-of-the-art methods and it also generalizes well across datasets without additional training. The executable version of DISC is available online: http://vision.sysu.edu.cn/projects/DISC.

I. INTRODUCTION

DISC addresses the ill-posedness of fine-grained saliency assignment by learning coarse-to-fine representations directly from image pixels rather than relying on hand-crafted priors. Its stacked CNNs combine global and local context, with superpixel information helping preserve salient-object structure.

  • Accurate saliency assignment is ill-posed because image saliency lacks a rigorous definition across diversified natural images.
  • Existing methods commonly rely on contrast, compactness, background, or higher-level semantic priors, often with hand-crafted features.
  • DISC formulates fine-grained saliency computing as progressive representation learning with coarse global context and fine local context.
  • The framework uses two stacked CNNs to learn saliency representations from raw pixels, with the first CNN producing coarse regions and the second rendering boundaries and subtle structures.
  • The separately supervised CNNs collaboratively infer a fine-grained saliency map in one shot, and experiments report strong performance and cross-dataset generalization without additional training.
  • Superpixel-based local context is embedded in the first CNN through intra-superpixel smoothing and inter-superpixel voting to preserve salient-object structure.

II. RELATED WORK

Prior saliency research spans bottom-up models based on mathematical priors and top-down models based on supervised visual knowledge. Deep CNN representation learning provides a newer route for dense prediction and saliency detection, including multi-scale and hierarchical approaches.

  • Existing saliency detection methods are broadly divided into bottom-up and top-down approaches.
  • Bottom-up approaches: Bottom-up models explain visual attention using mathematical principles or priors such as contrast, compactness, entropy, self-information, and low-rank structure.
  • Top-down approaches: Top-down approaches use supervised visual knowledge and are effective for task-specified saliency detection.
  • Deep representation learning: Deep CNN representation learning has been applied to dense pixel labeling and other vision tasks using raw-pixel, multi-scale, or hierarchical features.
  • Deep learning for saliency detection: DISC is illustrated as a two-stage model in which a whole-image CNN produces a coarse map and a local-patch CNN generates the fine-grained map.
  • Deep learning for saliency detection: Recent deep saliency methods use hierarchical contrast features or multi-scale CNN features fused through fully connected layers.

III. DEEP IMAGE SALIENCY COMPUTING FRAMEWORK

The DISC framework progressively computes saliency with two CNNs: a whole-image network first predicts a coarse map, then a local network uses the image and refined coarse patches to produce fine-level scores. Superpixel-based refinement preserves spatial structure before fine-level prediction.

  • DISC models fine-level saliency as progressive representation learning with two stacked CNNs.
  • The first CNN takes the original image and generates a coarse-level saliency map, while the second receives the image and coarse map to produce the final fine map.
  • The notation distinguishes the original image, coarse map, and fine map, while corresponding patches are centered at pixel i.
  • Coarse-level computing: For each coarse-map pixel, whole-image features are passed through a pixel-specific linear transformation to assign a saliency score.
  • Coarse-level computing: The coarse feature extractor is a CNN, and its pixelwise linear transformations are formulated as linear SVMs.
  • Coarse-level refinement: Superpixel-based local context refines the coarse map into a structure-preserved map before fine-level prediction.
  • Fine-level computing: For each fine-map pixel, the model combines a local image patch with the corresponding refined coarse-map patch, then applies a linear transformation to predict saliency.
  • The feature extractor and classifier of each CNN are jointly trained for saliency computing.

B. Superpixel-based Local Context Information (SLCI)

SLCI augments global coarse saliency prediction with superpixel-based local context. ISS smooths scores within small regions, while ISV uses larger neighboring regions to preserve spatial structure.

  • The coarse CNN can confuse similarly appearing small regions and lose salient-object structure in cluttered backgrounds.
  • Intra-superpixel smoothing (ISS): ISS replaces pixels’ saliency scores within each superpixel by their average, smoothing small-scale saliency structure.
  • Inter-superpixel voting (ISV): ISV replaces each region’s score with a weighted average over adjacent superpixels, preserving large-scale structure and uniform object labeling.
  • Inter-superpixel voting (ISV): Voting weights favor adjacent superpixels with similar color and structure histograms, using a scaling factor to balance the terms.
  • ISS and ISV are implemented as pooling layers integrated into the first CNN, while the fine CNN uses local neighborhoods directly.

C. Spatial Regularization

DISC uses spatial regularization in both CNN stages while retaining distinct global and local processing. The second CNN produces dense fine-level predictions from whole-image inputs to avoid redundant patch computation.

  • First CNN: The first CNN receives RGB data plus a spatial regularization channel and produces a coarse map through convolutional and pooling layers.
  • Second CNN: The second CNN uses original-image, spatial-regularization, and coarse-map channels to compute pixel saliency from local observations.
  • Second CNN: The second CNN is fully convolutional and produces dense outputs from the whole image rather than separately processing each local patch.
  • Second CNN: Whole-image dense computation avoids overlapping-patch redundancy and reduces training and inference time by hundreds of times.
  • The two CNNs are trained sequentially with stochastic gradient descent and hinge loss.

A. Optimization Formulation

DISC optimizes separate coarse- and fine-level saliency objectives for the two CNNs. The coarse stage uses binary classification with hinge loss, and gradients are propagated through the CNN layers by backpropagation.

  • The total objective is L = Lc + Lf, combining the coarse- and fine-level CNN objective functions.
  • The two objective terms are optimized separately for the first and second CNNs.
  • Coarse-level objective: The coarse saliency computation is formulated as pixelwise binary classification and optimized with squared hinge loss.
  • Fine-level objective: The fine-level saliency computation is likewise formulated as a binary classification problem with its own objective function.
  • Lower-layer parameters are learned by backpropagating gradients from the top layer through the CNN.
  • Training initializes and optimizes the first CNN, integrates ISS and ISV, fine-tunes the coarse model, and trains the second CNN similarly.

V. TASK-ORIENTED ADAPTATION

DISC can be adapted to task-oriented salient object detection through fine-tuning on images labeled for the target object classes. The experiments use multiple benchmark datasets and PR, F0.3, and MAE evaluation measures.

  • Task-oriented adaptation: Task-oriented detection highlights only selected object classes rather than every salient object in an image.Fine-tuning uses target-class pixels labeled 1 and other pixels labeled -1.
  • Datasets: The evaluation covers MSRA10K, SED1, ECSSD, PASCAL1500, and THUR15K, spanning single-object, complex-background, multi-object, and category-specific settings.THUR15K contains 15,000 images and 6,233 pixel-accurate annotations across five categories.
  • Evaluation protocol: Performance is assessed with precision-recall curves, F0.3, and mean absolute error.The F0.3 protocol uses an adaptive threshold based on twice the image’s mean saliency, with β2 = 0.3 weighting precision more than recall.
  • Experimental comparisons: Figures compare DISC with prior methods on MSRA10K and across SED1, ECSSD, and PASCAL1500 to assess performance and generalization.The cross-dataset comparison tests a model learned on MSRA10K without fine-tuning on the target datasets.
  • Visual analysis: Visual comparisons examine whether DISC highlights complete objects while preserving boundary and structural details.Examples include MSRA10K, SED1, ECSSD, and PASCAL1500.

B. Comparison with State-of-the-art Methods

On MSRA10K, DISC is compared with thirteen state-of-the-art saliency methods using precision-recall, F0.3, and MAE. DISC achieves stronger recall and overall performance while preserving details, and the reported comparisons attribute its advantage to coarse-to-fine deep architecture.

  • MSRA10K results: DISC reaches 97.3% accuracy on the MSRA10K precision-recall comparison, improving over prior methods that already exceed 93%.Its minimum recall is 34%, higher than the other compared methods.
  • MSRA10K results: DISC’s recall is 15.3% higher than the best previous work while precision remains comparable.This result is reported for the F0.3 comparison.
  • Detail preservation: DISC has significantly lower MAE than the compared methods, indicating better preservation of saliency-map details.The paper also reports visual evidence that DISC highlights overall salient objects while preserving detail.
  • Overall comparison: The comparisons report that DISC outperforms other state-of-the-art algorithms by large margins.The paper attributes the result to a coarse-to-fine architecture that captures different levels of image saliency information.

C. Comparision with CNN-based Methods

DISC is compared with three CNN-based saliency approaches on SED1 using precision-recall, F-measure, MAE, and running time. The comparison is designed to assess the effectiveness and efficiency of its coarse-to-fine architecture.

  • Compared methods: The SED1 comparison includes S-3CNN, LEGS, and an FCN-based method alongside DISC.S-3CNN and LEGS use multi-scale CNN structures, while FCN produces dense label maps for arbitrary input sizes.
  • Efficiency: DISC is reported as slightly faster than the FCN-based method, supporting the efficiency of its coarse-to-fine architecture.The passage presents this result as evidence for both architectural effectiveness and efficiency.
  • Evaluation measures: The SED1 table reports precision-recall with F-measure, MAE, and running time for DISC and the three CNN-based methods.These metrics jointly compare detection quality, saliency-map error, and computational speed.

D. Performance of Generalization

DISC generalizes across substantially different datasets without retraining, while fine tuning yields only slight additional improvement on ECSSD and PASCAL1500. Task-oriented adaptation improves precision while maintaining nearly the same recall.

  • Cross-dataset generalization: DISC outperforms previous methods on three evaluation metrics when directly tested on SED1, ECSSD, and PASCAL1500 after training on MSRA-10K.The datasets differ substantially in salient objects and backgrounds.
  • Fine-tuning: Fine tuning on ECSSD and PASCAL1500 produces results that are slightly better than without fine tuning.The authors attribute the limited gain partly to DISC's strong generalization and partly to insufficient fine-tuning samples.
  • Task-oriented adaptation: After fine tuning, precision improves significantly while recall remains nearly unchanged on the task-oriented adaptation dataset.The change indicates fewer non-target pixels are mislabeled as salient, so the model highlights the specified object classes while ignoring others.
  • Task-oriented adaptation: The task-oriented evaluation uses 5,233 images for fine tuning and 1,000 images for testing.

1) Contribution of progressive representation learning:

Progressive representation learning separates coarse global structure from fine object details, while local context and spatial regularization improve structural preservation and uniform highlighting.

  • Contribution of progressive representation learning: The progressive framework uses coarse representations for object location and global structure, then refines object details with fine representations.The evaluation compares the full framework with coarse-only maps and unguided fine maps.
  • Contribution of superpixel-based local context information (SLCI): Removing SLCI causes poor structure preservation by allowing visually similar small regions and background areas to be confused with salient objects.SLCI preserves local structural information and improves the resulting fine maps because fine-map quality depends heavily on the coarse map.
  • Contribution of spatial regularization (SR): Spatial regularization makes first-layer feature maps highlight objects more uniformly instead of focusing only on contours.Performance also consistently exceeds the version without spatial regularization across three evaluation metrics.

4) Hinge loss vs. cross entropy loss:

The paper evaluates hinge loss against cross entropy loss within the DISC CNNs, while also documenting challenging cases caused by coarse-map errors and coarse-map dependency.

  • Hinge loss vs. cross entropy loss: The hinge-loss evaluation replaces hinge loss with cross entropy while leaving the other CNN layers unchanged, then retrains both networks for comparison.Pixel labels are set to 1 for salient-object pixels and 0 otherwise.
  • Limitation: DISC produces poor coarse maps when foreground and background appearances are similar or when the background is cluttered.SLCI preserves target-object structure but does not work well for complex backgrounds.
  • Limitation: The fine saliency map depends heavily on coarse-map quality, making coarse-map errors the main limitation of the method.The paper suggests incorporating high-level knowledge such as object semantic shapes to refine coarse maps.
Loading 1511.04192v2…