PaperlayerView exact revision on arXiv

Source-linked AI summary

DISC: Deep Image Saliency Computing via Progressive Representation Learning

Tianshui Chen, Liang Lin, Lingbo Liu, Xiaonan Luo, Xuelong Li

arXiv:1511.04192v2cs.CV

TL;DR

DISC addresses the difficulty of accurate saliency assignment without relying on hand-crafted priors or feature engineering. It learns saliency progressively with two CNNs, combining global coarse context, superpixel refinement, and local fine prediction. The paper reports superior benchmark performance and cross-dataset generalization, while coarse-map quality remains its main limitation.

  • Problem

    Accurate saliency assignment is ill-posed across diversified natural images, and existing methods often depend on priors or hand-crafted features.

    Source: §I · p. 1

  • Method

    DISC uses two stacked CNNs for progressive coarse-to-fine learning, refining the coarse map with superpixel local context before fine-level prediction.

    Source: §I · pp. 2–3, B. Superpixel-based Local Context… · p. 5

  • Results

    DISC significantly outperforms state-of-the-art approaches and generalizes across datasets without additional training.

    Source: §I · p. 3, §VII · p. 13

  • Takeaways & Limitations

    The framework produces detail-preserved saliency maps without feature engineering or heuristic assumptions and is presented as generalizable to similar vision tasks.

    Source: §I · p. 3, §VII · p. 13

    Fine-map accuracy depends heavily on the coarse map, which can fail when foreground and background appearances are similar or backgrounds are cluttered.

    Source: H. Limitation · p. 13

Abstract

from arXiv · show

Salient object detection increasingly receives attention as an important component or step in several pattern recognition and image processing tasks. Although a variety of powerful saliency models have been intensively proposed, they usually involve heavy feature (or model) engineering based on priors (or assumptions) about the properties of objects and backgrounds. Inspired by the effectiveness of recently developed feature learning, we provide a novel Deep Image Saliency Computing (DISC) framework for fine-grained image saliency computing. In particular, we model the image saliency from both the coarse- and fine-level observations, and utilize the deep convolutional neural network (CNN) to learn the saliency representation in a progressive manner. Specifically, our saliency model is built upon two stacked CNNs. The first CNN generates a coarse-level saliency map by taking the overall image as the input, roughly identifying saliency regions in the global context. Furthermore, we integrate superpixel-based local context information in the first CNN to refine the coarse-level saliency map. Guided by the coarse saliency map, the second CNN focuses on the local context to produce fine-grained and accurate saliency map while preserving object details. For a testing image, the two CNNs collaboratively conduct the saliency computing in one shot. Our DISC framework is capable of uniformly highlighting the objects-of-interest from complex background while preserving well object details. Extensive experiments on several standard benchmarks suggest that DISC outperforms other state-of-the-art methods and it also generalizes well across datasets without additional training. The executable version of DISC is available online: http://vision.sysu.edu.cn/projects/DISC.

I. INTRODUCTION

DISC addresses the ill-posedness of fine-grained saliency assignment by learning coarse-to-fine representations directly from image pixels rather than relying on hand-crafted priors. Its stacked CNNs combine global and local context, with superpixel information helping preserve salient-object structure.

  • Accurate saliency assignment is ill-posed because image saliency lacks a rigorous definition across diversified natural images.Source: §I · p. 1
  • Existing methods commonly rely on contrast, compactness, background, or higher-level semantic priors, often with hand-crafted features.Source: §I · pp. 1–2
  • DISC formulates fine-grained saliency computing as progressive representation learning with coarse global context and fine local context.Source: §I · p. 2
  • The framework uses two stacked CNNs to learn saliency representations from raw pixels, with the first CNN producing coarse regions and the second rendering boundaries and subtle structures.Source: §I · p. 2
  • The separately supervised CNNs collaboratively infer a fine-grained saliency map in one shot, and experiments report strong performance and cross-dataset generalization without additional training.Source: §I · pp. 2–3
  • Superpixel-based local context is embedded in the first CNN through intra-superpixel smoothing and inter-superpixel voting to preserve salient-object structure.Source: §I · p. 3

II. RELATED WORK

Prior saliency research spans bottom-up models based on mathematical priors and top-down models based on supervised visual knowledge. Deep CNN representation learning provides a newer route for dense prediction and saliency detection, including multi-scale and hierarchical approaches.

Bottom-up approaches

  • Bottom-up models explain visual attention using mathematical principles or priors such as contrast, compactness, entropy, self-information, and low-rank structure.Source: A. Image Saliency Detection · p. 3

Top-down approaches

Deep representation learning

  • Deep CNN representation learning has been applied to dense pixel labeling and other vision tasks using raw-pixel, multi-scale, or hierarchical features.Source: B. Deep Representation Learning · p. 3

Deep learning for saliency detection

  • DISC is illustrated as a two-stage model in which a whole-image CNN produces a coarse map and a local-patch CNN generates the fine-grained map.Source: Figure 2 · p. 4
  • Recent deep saliency methods use hierarchical contrast features or multi-scale CNN features fused through fully connected layers.Source: C. Deep Learning for Saliency Detection · p. 4

III. DEEP IMAGE SALIENCY COMPUTING FRAMEWORK

The DISC framework progressively computes saliency with two CNNs: a whole-image network first predicts a coarse map, then a local network uses the image and refined coarse patches to produce fine-level scores. Superpixel-based refinement preserves spatial structure before fine-level prediction.

  • DISC models fine-level saliency as progressive representation learning with two stacked CNNs.Source: §III · p. 4
  • The first CNN takes the original image and generates a coarse-level saliency map, while the second receives the image and coarse map to produce the final fine map.Source: §III · p. 4
  • The notation distinguishes the original image, coarse map, and fine map, while corresponding patches are centered at pixel i.Source: A. Progressive Representation Learning · p. 4

Coarse-level computing

Coarse-level refinement

Fine-level computing

B. Superpixel-based Local Context Information (SLCI)

SLCI augments global coarse saliency prediction with superpixel-based local context. ISS smooths scores within small regions, while ISV uses larger neighboring regions to preserve spatial structure.

Intra-superpixel smoothing (ISS)

Inter-superpixel voting (ISV)

C. Spatial Regularization

DISC uses spatial regularization in both CNN stages while retaining distinct global and local processing. The second CNN produces dense fine-level predictions from whole-image inputs to avoid redundant patch computation.

First CNN

  • The first CNN receives RGB data plus a spatial regularization channel and produces a coarse map through convolutional and pooling layers.Source: C. Spatial Regularization · p. 6

Second CNN

  • The second CNN uses original-image, spatial-regularization, and coarse-map channels to compute pixel saliency from local observations.Source: C. Spatial Regularization · p. 6
  • The second CNN is fully convolutional and produces dense outputs from the whole image rather than separately processing each local patch.Source: C. Spatial Regularization · p. 6
  • Whole-image dense computation avoids overlapping-patch redundancy and reduces training and inference time by hundreds of times.Source: C. Spatial Regularization · p. 6
  • The two CNNs are trained sequentially with stochastic gradient descent and hinge loss.Source: §IV · p. 6

A. Optimization Formulation

DISC optimizes separate coarse- and fine-level saliency objectives for the two CNNs. The coarse stage uses binary classification with hinge loss, and gradients are propagated through the CNN layers by backpropagation.

Coarse-level objective

Fine-level objective

V. TASK-ORIENTED ADAPTATION

DISC can be adapted to task-oriented salient object detection through fine-tuning on images labeled for the target object classes. The experiments use multiple benchmark datasets and PR, F0.3, and MAE evaluation measures.

Task-oriented adaptation

  • Task-oriented detection highlights only selected object classes rather than every salient object in an image.Fine-tuning uses target-class pixels labeled 1 and other pixels labeled -1.Source: §V · p. 7

Datasets

  • The evaluation covers MSRA10K, SED1, ECSSD, PASCAL1500, and THUR15K, spanning single-object, complex-background, multi-object, and category-specific settings.THUR15K contains 15,000 images and 6,233 pixel-accurate annotations across five categories.Source: A. Experiment Setting · p. 7, Figure 5 · p. 7

Evaluation protocol

  • Performance is assessed with precision-recall curves, F0.3, and mean absolute error.The F0.3 protocol uses an adaptive threshold based on twice the image’s mean saliency, with β2 = 0.3 weighting precision more than recall.Source: Figure 5 · p. 7, A. Experiment Setting · p. 7

Experimental comparisons

  • Figures compare DISC with prior methods on MSRA10K and across SED1, ECSSD, and PASCAL1500 to assess performance and generalization.The cross-dataset comparison tests a model learned on MSRA10K without fine-tuning on the target datasets.Source: Figure 6 · p. 8, Figure 7 · p. 8

Visual analysis

  • Visual comparisons examine whether DISC highlights complete objects while preserving boundary and structural details.Examples include MSRA10K, SED1, ECSSD, and PASCAL1500.Source: Figure 8 · p. 9

B. Comparison with State-of-the-art Methods

On MSRA10K, DISC is compared with thirteen state-of-the-art saliency methods using precision-recall, F0.3, and MAE. DISC achieves stronger recall and overall performance while preserving details, and the reported comparisons attribute its advantage to coarse-to-fine deep architecture.

MSRA10K results

Detail preservation

  • DISC has significantly lower MAE than the compared methods, indicating better preservation of saliency-map details.The paper also reports visual evidence that DISC highlights overall salient objects while preserving detail.Source: B. Comparison with State-of-the-art… · p. 10

Overall comparison

  • The comparisons report that DISC outperforms other state-of-the-art algorithms by large margins.The paper attributes the result to a coarse-to-fine architecture that captures different levels of image saliency information.Source: B. Comparison with State-of-the-art… · p. 10

C. Comparision with CNN-based Methods

DISC is compared with three CNN-based saliency approaches on SED1 using precision-recall, F-measure, MAE, and running time. The comparison is designed to assess the effectiveness and efficiency of its coarse-to-fine architecture.

Compared methods

  • The SED1 comparison includes S-3CNN, LEGS, and an FCN-based method alongside DISC.S-3CNN and LEGS use multi-scale CNN structures, while FCN produces dense label maps for arbitrary input sizes.Source: C. Comparision with CNN-based Methods · p. 10

Efficiency

  • DISC is reported as slightly faster than the FCN-based method, supporting the efficiency of its coarse-to-fine architecture.The passage presents this result as evidence for both architectural effectiveness and efficiency.Source: C. Comparision with CNN-based Methods · p. 11

Evaluation measures

  • The SED1 table reports precision-recall with F-measure, MAE, and running time for DISC and the three CNN-based methods.These metrics jointly compare detection quality, saliency-map error, and computational speed.Source: Table I · p. 11

D. Performance of Generalization

DISC generalizes across substantially different datasets without retraining, while fine tuning yields only slight additional improvement on ECSSD and PASCAL1500. Task-oriented adaptation improves precision while maintaining nearly the same recall.

Cross-dataset generalization

  • DISC outperforms previous methods on three evaluation metrics when directly tested on SED1, ECSSD, and PASCAL1500 after training on MSRA-10K.The datasets differ substantially in salient objects and backgrounds.Source: D. Performance of Generalization · p. 11

Fine-tuning

  • Fine tuning on ECSSD and PASCAL1500 produces results that are slightly better than without fine tuning.The authors attribute the limited gain partly to DISC's strong generalization and partly to insufficient fine-tuning samples.Source: E. Fine Tuning on Other Datasets · p. 11, Figure 10 · p. 11

Task-oriented adaptation

  • After fine tuning, precision improves significantly while recall remains nearly unchanged on the task-oriented adaptation dataset.The change indicates fewer non-target pixels are mislabeled as salient, so the model highlights the specified object classes while ignoring others.Source: F. Task-oriented Adaptation · p. 11
  • The task-oriented evaluation uses 5,233 images for fine tuning and 1,000 images for testing.Source: F. Task-oriented Adaptation · p. 11

1) Contribution of progressive representation learning:

Progressive representation learning separates coarse global structure from fine object details, while local context and spatial regularization improve structural preservation and uniform highlighting.

Contribution of progressive representation learning

Contribution of superpixel-based local context information (SLCI)

Contribution of spatial regularization (SR)

4) Hinge loss vs. cross entropy loss:

The paper evaluates hinge loss against cross entropy loss within the DISC CNNs, while also documenting challenging cases caused by coarse-map errors and coarse-map dependency.

Hinge loss vs. cross entropy loss

Limitation

  • DISC produces poor coarse maps when foreground and background appearances are similar or when the background is cluttered.SLCI preserves target-object structure but does not work well for complex backgrounds.Source: H. Limitation · p. 13, Figure 15 · p. 13
  • The fine saliency map depends heavily on coarse-map quality, making coarse-map errors the main limitation of the method.The paper suggests incorporating high-level knowledge such as object semantic shapes to refine coarse maps.Source: H. Limitation · p. 13, §VII · p. 13

Related papers

Loading 1511.04192v2…