PaperlayerView exact revision on arXiv

Source-linked AI summary

RGBD Salient Object Detection via Deep Fusion

Liangqiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, Qingxiong Yang

arXiv:1607.03333v1cs.CV

TL;DR

RGBD saliency detection lacks a well-established way to model interactions among low-level RGB and depth cues and fuse them into a master map. The paper learns cue interactions by feeding hand-designed saliency features into a CNN, then applies Laplacian propagation; experiments on three datasets show consistent outperformance of state-of-the-art methods.

  • Problem

    How to model interactions among RGB and depth saliency cues and integrate them effectively into a master saliency map remains challenging.

  • Method

    The method fuses hand-designed RGBD saliency feature vectors into CNN-learned hyper-features and refines their output with Laplacian propagation.

  • Results

    Quantitative and qualitative evaluations on three datasets show that the fused RGBD hyper-feature consistently outperforms state-of-the-art methods.

  • Takeaways & Limitations

    The framework provides an RGBD saliency detector that combines learned cue interactions with spatially consistent map refinement.

Abstract

from arXiv · show

Numerous efforts have been made to design different low level saliency cues for the RGBD saliency detection, such as color or depth contrast features, background and color compactness priors. However, how these saliency cues interact with each other and how to incorporate these low level saliency cues effectively to generate a master saliency map remain a challenging problem. In this paper, we design a new convolutional neural network (CNN) to fuse different low level saliency cues into hierarchical features for automatically detecting salient objects in RGBD images. In contrast to the existing works that directly feed raw image pixels to the CNN, the proposed method takes advantage of the knowledge in traditional saliency detection by adopting various meaningful and well-designed saliency feature vectors as input. This can guide the training of CNN towards detecting salient object more effectively due to the reduced learning ambiguity. We then integrate a Laplacian propagation framework with the learned CNN to extract a spatially consistent saliency map by exploiting the intrinsic structure of the input image. Extensive quantitative and qualitative experimental evaluations on three datasets demonstrate that the proposed method consistently outperforms state-of-the-art methods.

I. INTRODUCTION

RGBD saliency detection must integrate complementary RGB and depth cues, but their interaction and effective fusion remain unresolved. The paper proposes deep fusion of hand-designed saliency features with Laplacian propagation to produce spatially consistent saliency maps.

  • RGBD saliency detection is important for applications including 3D surveillance, retrieval, and image recognition.
  • Bottom-up methods derive salient regions from low-level cues such as contrast, background priors, and color compactness.
  • Different saliency features may not complement one another, and linear map averaging cannot guarantee improved results.
  • The proposed model uses CNN representation learning to fuse hand-designed RGBD saliency features into representative hyper-features.
  • Laplacian propagation refines the learned saliency map by enforcing spatial consistency and exploiting the input image's intrinsic structure.
  • Simple feature fusion achieves superior performance, while experiments show the fused RGBD hyper-feature outperforms state-of-the-art methods.

II. RELATED WORK

Prior work models saliency with low-level RGB or depth cues and increasingly uses learned feature representations. The proposed approach differs by learning interactions among multiple saliency cues rather than directly combining their resulting maps.

  • RGB saliency research commonly uses bottom-up contrast cues, especially local or global color contrast, to model salient regions.
  • Integration methods include CRF-based feature fusion and hierarchical multi-scale saliency-map combination.
  • The proposed pipeline generates RGB and depth saliency features, extracts hyper-features with CNNs, and applies Laplacian propagation for spatial consistency.
  • Existing deep networks often take raw image pixels as input, whereas this method learns interactions among different saliency cues in a unified CNN.
  • RGBD saliency has received less research attention than RGB saliency and includes approaches based on depth attention, depth-motion contrast, and appearance-depth fusion.

III. PROPOSED METHOD

The proposed RGBD saliency framework has three modules: saliency-feature extraction, hyper-feature representation learning, and Laplacian propagation for spatially consistent detection.

  • The first module generates various saliency feature vectors for each superpixel region.
  • The second module extracts a hyper-feature representation from the obtained saliency feature vectors.
  • The third module applies Laplacian propagation to help produce a spatially consistent saliency map.

A. Saliency feature vectors extraction

The method represents each RGBD image region with multiple hand-designed saliency feature vectors rather than directly assigning a single saliency value. These vectors capture local and global contrast, background proximity, and color compactness using superpixel regions.

  • Region representation: Each RGB image is segmented into N superpixels, and every region Pi receives a saliency feature vector ΓPi.The paper uses SLIC segmentation and illustrates feature computation for one example region.
  • Feature composition: The feature representation records seven vector types combining color and depth information with local and global scopes.The notation distinguishes color, depth, local, global, background, and color-compactness components.
  • Contrast features: Local color contrast weights neighboring regions by spatial distance, region size, and differences between their mean colors.The spatial scale σLr = 0.15 makes neighboring regions more influential while making other regions’ influence negligible.
  • Contrast features: Global contrast differs from local contrast through its spatial weighting, with σGr = 0.45 set to cover the entire image.Depth contrast is calculated analogously using mean depth values for the compared regions.
  • Prior features: Color compactness models the tendency of object colors to cluster, while the method omits depth compactness because depth levels and spatial distributions can be random.The authors report that including depth compactness has little effect on final results.
  • Prior features: The background prior estimates contrast with Nb boundary regions treated as pseudo-background, using N = 1024 superpixels and Nb = 160 boundary regions.The prior reflects the assumption that salient objects are less likely to lie near image boundaries.

B. Hyper-feature extraction with CNN

The CNN converts regional saliency vectors into unified hyper-features for saliency prediction. It processes six-channel 32 × 32 inputs through convolutional, pooling, fully connected, and logistic-regression layers.

  • CNN input and output: The CNN formulates saliency detection as binary logistic regression that predicts salient and non-salient class probabilities from a patch.Its input size is 32 × 32 × 6.
  • CNN input and output: For each superpixel, seven saliency vectors are integrated into a multiple-channel image representation.The construction uses the regional feature vectors as CNN channels rather than raw image pixels.
  • Network architecture: The network contains three convolutional layers, average-pooling layers after the first two convolutions, a fully connected layer, and a sigmoid logistic output.Sigmoid functions are used in the convolutional layers, while ReLUs are used in the final fully connected stages.
  • Network architecture: Its specified architecture is conv1(6, 5)−sig1−pool1(MEAN, 2)−conv2(12, 5)−sig2−pool2(MEAN, 2)−conv3(24, 3)−sig3−fc4(200)−relu4−dropout4−fc5(2).The model is trained with back-propagation.

C. Laplacian propagation

Laplacian propagation refines noisy CNN saliency predictions by propagating reliable salient and non-salient labels to ambiguous regions while enforcing spatial and manifold consistency.

  • Motivation: The CNN can omit salient regions and misclassify background, although its high-probability salient and non-salient regions are generally reliable.These errors motivate a refinement stage for producing a spatially consistent map.
  • Propagation setup: Propagation favors similar saliency for neighboring regions and regions lying within the same image manifold.The affinity matrix uses color and relative depth distances with a two-hierarchy neighboring connection model.
  • Propagation setup: Laplacian propagation labels superpixels using salient or non-salient CNN probabilities, then predicts labels for the remaining regions.Superpixels exceed thresholds τ1 or τ2 to receive initial labels.
  • Optimization: The optimization balances smoothness and fitting constraints, and its linear-system solution uses Conjugate Gradient with a preconditioner for fast convergence.The balance parameter is μ, with α = 1/(1 + μ).
  • Output refinement: The final normalized map corrects wrongly estimated regions by propagating from high-probability salient and non-salient regions.Parameters τ1 and τ2 are adaptively determined using Otsu’s method.

IV. EXPERIMENTAL EVALUATIONS

The method is evaluated on three RGBD saliency datasets using precision-recall analysis, mean average precision and recall, and F-measure.

  • Datasets: The evaluation uses the NLPR, NJUDS2000, and LFSD RGBD saliency datasets.NLPR contains 1,000 Kinect images, NJUDS2000 contains 2,000 stereo images with depth maps, and LFSD contains 100 depth-annotated images.
  • Qualitative evaluation: Figure 5 compares the proposed framework with four RGB saliency methods and three RGBD saliency methods.The compared RGB methods use RGB images, while LMH, ACSD, and GP use RGBD images.
  • Metrics: Performance is evaluated with precision-recall curves, mean average precision and recall, and F-measure.The precision-recall curve measures mean precision and recall across saliency-map thresholds.

A. Implementation details

The framework is trained on randomly sampled NLPR and NJUDS2000 images, with data augmentation used to address CNN data demands and reduce overfitting.

  • Training data: Training uses 750 NLPR images and 1,000 NJUDS2000 images, while the remaining portions and all LFSD images evaluate generalization.The selected training data cover more than 1,000 kinds of common objects under different circumstances.
  • Training procedure: Data augmentation enriches the training set because the available data are insufficient for training the data-hungry CNN.Dropout is also used, and the augmentation is intended to enable training without overfitting.
  • Training procedure: Training takes approximately 5–7 days to converge.The reported implementation uses Matlab, momentum 0.9, weight decay 0.0005, and a learning rate reduced from 1 to 0.001.

B. Performance Comparison

The proposed RGBD method is evaluated against RGB, RGBD, heuristic, CRF, cellular-automata, and CNN-based fusion methods on three datasets. It consistently performs favorably, while feature-level fusion and Laplacian propagation improve saliency estimation and spatial consistency.

  • Qualitative comparison: The proposed RGBD method consistently outperforms LMH, ACSD, and GP in qualitative comparisons across scenes.Depth helps detect salient objects sharing similar color with the background.
  • Quantitative comparison: Higher precision, recall, and F-measure scores are reported on NLPR, NJUDS2000, and LFSD than for existing algorithms.RGBD methods are especially advantageous when RGB appearance resembles the background, whereas rough depth can limit recall.
  • Fusion comparison: Feature-level fusion is compared with linear, CRF, MCA, CNN-F, heuristic, and sophisticated saliency-map merging methods using fundamental and sophisticated inputs.The experiments test whether jointly learning feature interactions is more flexible than directly merging saliency maps.
  • Fusion comparison: 0.8157 versus 0.8071: initial saliency feature fusion exceeds sophisticated saliency-map merging on LFSD without Laplacian propagation.The 0.8157 result uses simple features and pixel-level processing without spatial-consistency modeling.
  • Laplacian propagation: Almost 0.79, 0.79, and 0.84 F-measures are achieved on the three test datasets with Laplacian propagation.The framework also refines saliency maps from existing methods and can be used as a post-processing step.
  • Failure cases: The method may fail when salient objects have color and depth information very similar to the background.Depth is more helpful when objects have high depth contrast with the background or lie closer to the camera.

V. CONCLUSION

The conclusion presents a three-module RGBD saliency framework that learns interactions among hand-designed cues and refines predictions with Laplacian propagation. Experiments show that the fused hyper-feature outperforms state-of-the-art methods, while the framework can incorporate additional cues.

  • V. CONCLUSION: The framework generates low-level saliency vectors, learns RGB-depth feature interactions with CNNs, and produces a spatially consistent map with Laplacian propagation.Hand-designed inputs guide CNN learning toward saliency-optimized representations.
  • V. CONCLUSION: The fused RGBD hyper-feature outperforms all state-of-the-art methods in quantitative and qualitative experiments.
  • V. CONCLUSION: Optimized fusion leads to superior performance, and the hyper-feature framework can be extended with additional saliency cues such as flash cues.
Loading 1607.03333v1…