PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

Xi Li, Liming Zhao, Lina Wei, Ming-Hsuan Yang, Fei Wu, Yueting Zhuang, Haibin Ling, Jingdong Wang

arXiv:1510.05484v2cs.CV

TL;DR

Salient object detection needs data-driven modeling of semantic properties across multiple feature levels, beyond independently modeled patches and manually specified priors. The paper proposes a multi-task FCNN that shares features with semantic segmentation and adds graph-Laplacian nonlinear-regression refinement. Across eight benchmark datasets, the approach performs favorably against state-of-the-art methods on multiple evaluation metrics.

  • Problem

    Salient object detection needs a unified data-driven model for low-, mid-, and high-level semantic properties, while existing deep models can suffer feature redundancy from independent patch modeling.

  • Method

    A multi-task FCNN learns saliency detection and semantic segmentation through shared fully convolutional layers, followed by graph Laplacian regularized nonlinear regression for boundary-preserving refinement.

  • Results

    Across eight benchmark datasets, the approach performs favorably against state-of-the-art methods on aveF, maxF, AUC, and MAE in most cases.

  • Takeaways & Limitations

    Collaborative semantic feature learning and super-pixel graph refinement provide the paper’s data-driven route to object-aware and fine-grained saliency detection.

Abstract

from arXiv · show

A key problem in salient object detection is how to effectively model the semantic properties of salient objects in a data-driven manner. In this paper, we propose a multi-task deep saliency model based on a fully convolutional neural network (FCNN) with global input (whole raw images) and global output (whole saliency maps). In principle, the proposed saliency model takes a data-driven strategy for encoding the underlying saliency prior information, and then sets up a multi-task learning scheme for exploring the intrinsic correlations between saliency detection and semantic image segmentation. Through collaborative feature learning from such two correlated tasks, the shared fully convolutional layers produce effective features for object perception. Moreover, it is capable of capturing the semantic information on salient objects across different levels using the fully convolutional layers, which investigate the feature-sharing properties of salient object detection with great feature redundancy reduction. Finally, we present a graph Laplacian regularized nonlinear regression model for saliency refinement. Experimental results demonstrate the effectiveness of our approach in comparison with the state-of-the-art approaches.

I. INTRODUCTION

The paper frames salient object detection as a need to model low-, mid-, and high-level semantic properties in one data-driven framework. It proposes a multi-task FCNN with refinement to learn shared object-perception features and produce fine-grained saliency maps.

  • Motivation: Salient object detection must jointly model low-level cues, mid-level shape and context, and high-level semantic object information.Existing methods often rely on manually tailored background, center, or contrast priors.
  • Proposed approach: The proposed multi-task FCNN uses whole images as input and whole saliency maps as output while learning saliency detection with semantic image segmentation.The two tasks share fully convolutional layers to explore their intrinsic correlations.
  • Proposed approach: Collaborative feature learning extracts discriminative object-perception features and reduces feature redundancy through shared fully convolutional layers.The model captures salient-object semantics across multiple levels.
  • Saliency refinement: Graph Laplacian regularized nonlinear regression refines the FCNN output into a fine-grained saliency map while preserving object boundaries.The refinement model is driven by super-pixels and has a closed-form solution.

II. RELATED WORK

Prior saliency methods use contrast analysis or manually specified priors, while early deep models learn from annotations but may process patches independently. This work instead combines whole-image refinement with shared multi-task features, introducing semantic segmentation into saliency learning.

  • Saliency detection approaches: Early salient-object detectors rely on center-surround differences, region uniqueness, or predefined center, region, semantic, and background priors.The proposed direction replaces manually assumed priors with automatically learned prior knowledge.
  • Saliency detection approaches: The approach combines FCNN saliency prediction with Laplacian-regularized nonlinear regression on a super-pixel graph for boundary-preserving refinement.The FCNN first predicts a coarse map, followed by graph-based refinement.
  • Deep saliency networks: Existing deep saliency models supervise pixel-wise prediction but may independently model local patches, increasing feature redundancy and computational cost.They also learn relative object information without modeling what the objects are.
  • Multi-task neural networks: The proposed multi-task network shares convolutional features between object-class segmentation and saliency detection.The shared representation uses semantic segmentation to learn object information for saliency modeling.
  • Multi-task neural networks: Introducing semantic segmentation into deep networks is presented as a novel way to learn a better saliency model.The architecture uses two task-specific networks connected through shared features.

A. Overview

The approach combines a multi-task FCNN for saliency detection and semantic segmentation with graph-based nonlinear refinement. Shared convolutional layers learn multi-level object features, while task-specific layers produce saliency and segmentation outputs.

  • A. Overview: The proposed model jointly learns salient-object detection and pixel-wise semantic segmentation in a fully convolutional network.The two tasks share convolutional layers but use task-specific deconvolutional layers.
  • A. Overview: Shared convolutional layers extract features across multiple semantic levels for object perception.The shared component processes the input RGB image through stacked convolutional operations and pooling.
  • A. Overview: The segmentation branch predicts C probabilistic class maps using 1×1 convolution and learned deconvolutional upsampling.The upsampled outputs are cropped to the input-image size.
  • A. Overview: The saliency branch generates a sigmoid-normalized saliency map and optimizes squared Euclidean regression loss.Its output is normalized to [0, 1].
  • A. Overview: Training alternates between optimizing segmentation parameters and saliency parameters while updating the shared parameters with SGD.The segmentation and saliency objectives use cross-entropy and squared Euclidean loss terms, respectively.
  • A. Overview: The learned network captures saliency properties associated with color, image center, contrast, and semantic objects.Examples include assigning higher saliency to a red flower than a yellow one and detecting high-contrast objects.

C. Regularized Regression for Refinement

The refinement model represents images with super-pixels and propagates saliency through a spatial-feature adjacency graph. Graph Laplacian regularization combines regression fitting with spatial smoothness to estimate fine-grained saliency values.

  • C. Regularized Regression for Refinement: Fuzzy FCNN object boundaries motivate refinement with homogeneous, boundary-preserving SLIC super-pixels.Super-pixels provide the basic units for a graph that models spatial and feature relationships.
  • C. Regularized Regression for Refinement: The super-pixel adjacency graph connects spatially adjacent regions using an RBF kernel that measures feature similarity.The affinity is zero for non-adjacent super-pixels.
  • C. Regularized Regression for Refinement: Boundary super-pixels act as seed samples with predefined saliency scores, while unseeded regions begin with null scores for propagation.The seed vector ranges from [-1,1], with boundary samples commonly initialized as background.
  • C. Regularized Regression for Refinement: The nonlinear regression objective combines squared loss, RKHS regularization, and a graph-Laplacian term enforcing spatial smoothness.The trade-off factors are γA = 10^-6 and γI = 1 in the experiments.
  • C. Regularized Regression for Refinement: The learned function g(x) uses kernel coefficients to predict saliency scores for arbitrary super-pixels.The optimal coefficient vector is expressed using the kernel Gram matrix and regularization terms.

D. Generating Saliency Map

Saliency-map generation proceeds from FCNN object perception through boundary propagation, coarse fusion, and graph-based nonlinear refinement. The final stage produces more accurate maps with fine-grained object boundaries.

  • D. Generating Saliency Map: The pipeline has four stages: FCNN object perception, boundary propagation, coarse saliency fusion, and nonlinear-regression refinement.These stages combine foreground objectness with boundary-derived background information.
  • D. Generating Saliency Map: FCNN produces a pixel-wise objectness probability map called DeepMap, representing foreground information.DeepMap values range from 0 to 1.
  • D. Generating Saliency Map: Boundary propagation initializes image-boundary super-pixels at -1 and estimates their influence through the Laplacian graph.The resulting BoundaryMap is normalized to [0, 1].
  • D. Generating Saliency Map: DeepMap and BoundaryMap are combined by elementwise multiplication to form the coarse-grained saliency map CgMap.β controls the contribution of BoundaryMap in the fusion.
  • D. Generating Saliency Map: The normalized CgMap is refined over the super-pixel graph by Laplacian-regularized nonlinear regression.The resulting fine-grained map is reported to have more accurate saliency detection results and object boundaries.

A. Experimental Setup

The evaluation uses eight pixel-wise annotated benchmark datasets spanning simple object-centric scenes and challenging images with multiple salient objects or clutter. The FCNN is implemented with pretrained VGG layers, alternating task optimization, and fixed experimental settings.

  • Datasets: Experiments use eight benchmark datasets with pixel-wise ground-truth annotations, covering both relatively simple and challenging salient-object scenes.The challenging datasets contain multiple salient objects and background clutter.
  • Implementation: Super-pixel regression uses SLIC with N=200 and an RBF-kernel scaling factor ρ=0.1 on a desktop computer with an Intel E5-2609 CPU and 8 GB RAM.These settings are reported as the fixed implementation configuration.
  • Implementation: The method is implemented in Caffe using the first 13 convolutional layers of pretrained VGG-16, with upsampled deconvolution layers.Deconvolution parameters start from bilinear interpolation and are updated during training.
  • Training: Segmentation and saliency tasks are optimized alternately because existing datasets lack both annotation types, while joint training remains possible when such data exist.The alternating procedure transfers segmentation knowledge into saliency feature learning without recollecting training data.
  • Training: Training alternates SGD updates for shared, segmentation-related, and saliency-related parameters, repeating the procedure three times for relatively stable performance.The experimental configurations remain fixed throughout all experiments.

B. Evaluation Metrics

The paper evaluates saliency detection using precision-recall, F-measure, MAE, ROC, and AUC, with thresholding procedures defining the reported measures.

  • Precision-recall: Precision-recall curves assess object retrieval by measuring precision and recall after binarizing saliency maps across thresholds from 0 to 255.Precision and recall are evaluated at particular binarization thresholds.
  • F-measure: F-measure summarizes the balance between precision and recall, using η^2=0.3 in accordance with most existing literature.The reported variants are maximum F-measure and average F-measure.
  • F-measure: Maximum F-measure is the largest value from the precision-recall curve, whereas average F-measure uses an adaptive threshold equal to twice the mean saliency-map value.The adaptive threshold is used for binarization.
  • Error and ROC: MAE measures average pixel-wise error against ground truth, while ROC curves plot false-positive rate against true-positive rate over varying thresholds.AUC is the area under the resulting ROC curve and evaluates object detection performance.

C. State-of-the-art performance comparison

Across eight benchmark datasets and several state-of-the-art baselines, the proposed approach performs favorably in most reported comparisons and achieves the best Table II metrics for OurLO.

  • Baselines: The comparison includes deep-learning, random-forest, background-prior, objectness, contrast, and boundary-prior saliency methods.Baselines include DeepMC, LEGS, DRFI, MC, DSR, SVO, HS, Wco, GMR, FT, GC, BL15, and BSCA.
  • Precision-recall comparison: The proposed approach achieves higher precision at fixed recall on precision-recall curves across eight benchmark datasets.Figure 6 reports the corresponding curves for all competing approaches.
  • Quantitative comparison: OurLO achieves the best performance on aveF, maxF, AUC, and MAE in Table II, where smaller MAE is better.The final three comparison columns use results quoted from the original papers, while OurBR results are null because THUS supplies its training data.
  • ROC comparison: The proposed approach performs better than competing methods in most ROC-curve cases across the eight benchmark datasets.Figure 7 presents the ROC curves used for this comparison.
  • Qualitative comparison: Qualitative comparisons on challenging samples report more visually feasible saliency maps than competing approaches.The figure caption describes the displayed comparisons, while the accompanying discussion covers clutter and low foreground-background contrast.

2) Qualitative performance comparison:

Qualitative and ablation-oriented evaluations examine challenging images, alternative training-data strategies, multi-task learning, and segmentation comparisons. The reported results support robustness with smaller training sets and benefits from multi-task learning in most cases.

  • Qualitative performance comparison: On challenging samples, the approach performs well under background clutter and low foreground-background contrast.Figure 8 compares its saliency results with other state-of-the-art approaches.
  • Training strategies of FCNN: The training-data strategies OurLO, OurBR, and OurST use leave-one-out, baseline-reference, and small-training-set protocols, respectively.OurST selects 2500 MSRA5000 images, while OurBR trains on THUS and tests the other seven datasets.
  • Training strategies of FCNN: OurST achieves comparable results to OurLO and OurBR with more training data and outperforms DRFI in most cases.The comparison includes datasets not overlapped with training data and a 2000-image MSRA5000 test list.
  • Multi-task learning: Multi-task learning achieves better performance than the single-task saliency model in most cases across seven datasets excluding THUS.The comparison uses the same baseline-reference strategy for the multi-task model.
  • Segmentation comparison: A conventional semantic segmentation model cannot directly serve as salient-object detection, whereas the proposed segmentation component uses saliency information to help segment salient objects.This distinction is evaluated quantitatively on the ASD dataset.

3) Evaluation on regression-based propagation:

Regression-based propagation generally improves saliency detection across eight benchmark datasets, with benefits attributed to capturing super-pixel topology that supports salient-object boundary preservation. Performance remains relatively stable across β choices, with β = 0.2 selected for experiments.

  • In most cases, propagation achieves better saliency detection performance than using the approach without propagation.This comparison covers eight benchmark datasets.
  • Nonlinear regression-based propagation captures topological information among super-pixels that helps preserve salient-object boundaries.
  • CRF-based refinement is evaluated against the proposed regression-based refinement on eight benchmark datasets using the same metrics.The supplied passage defines the comparison but does not report its outcome.
  • β = 0.2 yields better performance in most cases, while performance remains relatively stable across the tested β values.Performance improves until β = 0.2 and then remains relatively stable; β controls the trade-off between nonlinear saliency regression and image boundary propagation.
Loading 1510.05484v2…