PaperlayerView exact revision on arXiv

Source-linked AI summary

PDNet: Prior-model Guided Depth-enhanced Network for Salient Object Detection

Chunbiao Zhu, Xing Cai, Kan Huang, Thomas H Li, Ge Li

arXiv:1803.08636v2cs.CVcs.AIcs.MM

TL;DR

Deep saliency detection is constrained by limited annotated RGB-D data and robustness challenges in complex scenes. PDNet addresses these issues with a pre-trained RGB master network and an independent depth sub-network, and it performs favorably across five benchmark datasets.

  • Problem

    Deep RGB-D saliency detection faces limited annotated training data and insufficient robustness for extracting salient objects in complex scenes.

  • Method

    PDNet pre-trains an RGB master network as a prior model and independently encodes depth cues through a subsidiary network integrated with the master network.

  • Results

    PDNet performs favorably against state-of-the-art methods across five RGB-D benchmark datasets.

  • Takeaways & Limitations

    The prior model provides a solid foundation for salient object detection, while the independent depth-enhanced network contributes substantially to final accuracy.

Abstract

from arXiv · show

Fully convolutional neural networks (FCNs) have shown outstanding performance in many computer vision tasks including salient object detection. However, there still remains two issues needed to be addressed in deep learning based saliency detection. One is the lack of tremendous amount of annotated data to train a network. The other is the lack of robustness for extracting salient objects in images containing complex scenes. In this paper, we present a new architecture$ - $PDNet, a robust prior-model guided depth-enhanced network for RGB-D salient object detection. In contrast to existing works, in which RGB-D values of image pixels are fed directly to a network, the proposed architecture is composed of a master network for processing RGB values, and a sub-network making full use of depth cues and incorporate depth-based features into the master network. To overcome the limited size of the labeled RGB-D dataset for training, we employ a large conventional RGB dataset to pre-train the master network, which proves to contribute largely to the final accuracy. Extensive evaluations over five benchmark datasets demonstrate that our proposed method performs favorably against the state-of-the-art approaches.

I. INTRODUCTION

RGB-D saliency detection addresses the limits of 2D-only methods in complex scenes, but annotated RGB-D data is scarce and depth integration remains challenging. PDNet combines RGB prior guidance with independent depth processing and reports improvements across five benchmark datasets.

  • I. INTRODUCTION: Visual attention filters irrelevant information and highlights noticeable foreground regions, motivating computational saliency models.Saliency methods are broadly categorized as task-driven top-down or low-level-cue-based bottom-up approaches.
  • I. INTRODUCTION: RGB-D saliency is motivated by complex visual scenes where 2D information alone is insufficient and depth helps distinguish similarly appearing objects.Depth sensors such as Time-of-Flight devices and Microsoft Kinect support structural cues for applications including surveillance, retrieval, and recognition.
  • I. INTRODUCTION: Limited annotated RGB-D data makes network training ineffective, while integrating depth with RGB remains a key unresolved issue.
  • I. INTRODUCTION: PDNet combines a prior-model guided RGB master network with an independent depth-processing sub-network for RGB-D saliency detection.The master network extracts hierarchical features and restores object shape, while the sub-network encodes depth cues to enhance robustness.
  • I. INTRODUCTION: PDNet demonstrates dramatic performance improvements over previous methods on five benchmark datasets.

II. RELATED WORK

Saliency detection has evolved from hand-crafted low-level features toward deep architectures, while RGB-D methods remain comparatively less explored. The section frames PDNet within this transition and the broader saliency literature.

  • II. RELATED WORK: The section reviews saliency methods for both RGB and RGB-D images, positioning PDNet within these two lines of research.
  • II. RELATED WORK: Earlier saliency detection methods primarily relied on low-level hand-crafted features, with comprehensive surveys covering this broader body of work.
  • II. RELATED WORK: Deep learning enabled convolutional architectures to learn useful salient-object representations as annotated RGB datasets expanded.

B. RGB-D Saliency Detection

RGB-D saliency methods exploit color and depth information, but prior approaches often directly concatenate depth as a fourth input or depend on hand-crafted features. PDNet instead separates RGB and depth processing within complementary network components.

  • B. RGB-D Saliency Detection: Prior RGB-D research includes depth mining, color-depth salient stimuli, fusion frameworks, and depth-image saliency methods.
  • B. RGB-D Saliency Detection: Unlike many prior methods, PDNet processes depth independently rather than directly feeding it as a fourth-dimensional input with RGB.The independent design is intended to make fuller use of depth cues and assist the main-stream network.
  • B. RGB-D Saliency Detection: PDNet uses a prior-model guided convolution-deconvolution master network alongside a subsidiary encoder that extracts depth cues.
  • B. RGB-D Saliency Detection: The master network uses encoder-decoder features, copy-crop connections, and multi-feature concatenation to combine hierarchical information for detailed saliency maps.Copy-crop adds early low-level features during up-sampling, while concatenation combines low- and high-level decoder features.
  • B. RGB-D Saliency Detection: The saliency model treats detection as regression and produces a pixel-wise saliency value from an input image and its receptive field.

2) Prior-model Guidance:

PDNet pre-trains its master network on conventional RGB saliency data, then uses the resulting fixed prior-model weights to guide RGB-D training. This transfers RGB supervision into the RGB-D setting.

  • 2) Prior-model Guidance:: Once trained, the network fixes its weights for detecting salient objects in new input images.
  • 2) Prior-model Guidance:: RGB saliency datasets are used for pre-training because annotated RGB-D datasets are limited.The training data include MSRA10K and DUTS-TR, totaling 20,553 RGB training images according to the implementation description.
  • 2) Prior-model Guidance:: Pre-training the master network produces fixed prior-model weights γ that guide the post-training weights θ.

B. Depth-enhanced Subsidiary Network

The depth-enhanced subsidiary network encodes depth cues and incorporates the resulting depth-based features into the prior-model guided master network.

  • B. Depth-enhanced Subsidiary Network: The subsidiary network encodes an input depth map and incorporates its depth-based features into the master network as a convolution layer.The depth map is denoted d.
  • B. Depth-enhanced Subsidiary Network: The depth-enhanced convolution output is computed as d_o = w · d + b, with b as bias and w obtained from the subsidiary network.The passage identifies w as the depth-enhanced weight matrix.
  • B. Depth-enhanced Subsidiary Network: The subsidiary network’s output features are directly used as the master network’s weight matrix, making it a depth-enhanced weight prediction network.The output is denoted d_o, while the resulting weight matrix is denoted w.
  • B. Depth-enhanced Subsidiary Network: The combination weight factor α controls the contribution of depth-based feature maps according to the number of feature maps.The RGB-based feature maps from the master encoder are denoted I_o.

A. Parameters and Running time

The model is pretrained on a large RGB dataset and evaluated across five RGB-D datasets, with visual comparisons reported alongside the experiments.

  • A. Parameters and Running time: The master network is pretrained on 20,553 RGB images for 15 epochs, requiring approximately 26 hours on the reported hardware.Implementation uses TensorFlow 1.4, an i7-7700 CPU, and an Nvidia GTX1060 GPU with 6G memory.
  • A. Parameters and Running time: Evaluation covers five RGB-D datasets: NJU2000, NLPR, LFSD, RGBD135, and SSD100.NJU2000 and NLPR are split into training and testing subsets, while the other datasets are used entirely for testing.
  • A. Parameters and Running time: The experiments include visual comparisons of different methods, PR curves on five RGB-D datasets, and visual results from the ablation study.The supplied captions identify the scopes of the three visuals.
  • A. Parameters and Running time: Table I reports ablation-study results on two RGB-D datasets.The supplied table caption identifies the table’s scope but does not provide its cell values.

C. Evaluation Metrics

The study evaluates saliency algorithms using precision-recall curves, F-measure, and mean absolute error.

  • C. Evaluation Metrics: The evaluation uses precision-recall curves, F-measure, and mean absolute error as the three listed performance metrics.These metrics are described as widely used for evaluating saliency algorithms.

D. Ablation Study

The ablation study compares a baseline with five PDNet variants to assess prior guidance, depth enhancement, and combination-weight settings.

  • D. Ablation Study: The baseline MNet is the master network without prior-model guidance, trained directly with four-dimensional RGB-D input.The ablation design uses MNet as the comparison baseline.
  • D. Ablation Study: PNet tests the prior-model-guided master network trained with three-dimensional RGB data, isolating the prior-model setting.PNet is one of the five evaluated variants.
  • D. Ablation Study: DNet connects the subsidiary depth-enhanced network to the master network without prior training, using α = 1.This variant tests depth enhancement without prior training.
  • D. Ablation Study: The study also evaluates PDNet with α = 1, α < 1, and α > 1, averaging four samples for each unequal-weight condition.The unequal-weight samples are 0.3, 0.5, 0.7, 0.9 and 1.3, 1.5, 1.7, 1.9, respectively.
  • D. Ablation Study: PDNetα=1 achieves approximately the best performance among the evaluated variants, so α = 1 is used in subsequent experiments.The passage reports accumulated gains after each component and identifies PDNetα=1 as approximately best.

E. Comparison with the State of the Art

PDNet is evaluated against RGB and RGB-D saliency methods using default author settings, with visual and quantitative comparisons across five datasets. It produces fine-detail salient regions and outperforms other methods across nearly all datasets and commonly used metrics.

  • Evaluation setup: The comparison includes three RGB saliency methods and seven RGB-D methods, reproduced using author-provided code and default settings.The evaluated RGB methods are BSCA15, LIP15, and HS16; the RGB-D methods are DES14, NLPR14, ACSD15, SE, TPF17, DF17, and CTMF17.
  • Visual comparison: PDNet produces fine-detail saliency maps that highlight attention-grabbing salient regions in visual comparisons.Fig.2 compares PDNet with three RGB and seven RGB-D methods.
  • Quantitative comparison: Across five RGB-D datasets, PDNet outperforms other methods almost everywhere on F-measure and MAE, while PR curves favor PDNet on all five datasets.Table II reports F-measure and MAE comparisons; Fig.3 provides the PR-curve comparison.
  • Interpretation: Pre-training the RGB master network provides a foundation for detection, while independently integrating depth features contributes substantially to final accuracy.The design avoids treating depth as a fourth-dimensional direct input and instead uses an independent depth-enhanced sub-network.
Loading 1803.08636v2…