PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Fusion for RGB-D Salient Object Detection

Ningning Wang, Xiaojin Gong

arXiv:1901.01369v2cs.CV

TL;DR

RGB-D saliency detection must handle cases where color or depth alone is insufficient, motivating adaptive multimodal fusion. This paper uses two modality-specific CNN streams and a learned switch map with edge-preserving supervision, and reports consistent improvements over state-of-the-art methods across three datasets.

  • Problem

    Color-only saliency detection can fail when salient objects resemble their surroundings, while existing RGB-D methods mainly use fixed multimodal fusion operations.

  • Method

    An end-to-end two-stream CNN predicts RGB and depth saliency maps, then a supervised switch map adaptively fuses them with saliency and edge-preserving losses.

  • Results

    The method consistently outperforms state-of-the-art RGB-D saliency methods on three publicly available datasets.

  • Takeaways & Limitations

    Adaptive fusion can combine effective information from RGB and depth predictions while reducing blurry boundaries and improving spatial coherence.

Abstract

from arXiv · show

RGB-D salient object detection aims to identify the most visually distinctive objects in a pair of color and depth images. Based upon an observation that most of the salient objects may stand out at least in one modality, this paper proposes an adaptive fusion scheme to fuse saliency predictions generated from two modalities. Specifically, we design a two-streamed convolutional neural network (CNN), each of which extracts features and predicts a saliency map from either RGB or depth modality. Then, a saliency fusion module learns a switch map that is used to adaptively fuse the predicted saliency maps. A loss function composed of saliency supervision, switch map supervision, and edge-preserving constraints is designed to make full supervision, and the entire network is trained in an end-to-end manner. Benefited from the adaptive fusion strategy and the edge-preserving constraint, our approach outperforms state-of-the-art methods on three publicly available datasets.

I. INTRODUCTION

RGB-D saliency detection supplements color images with depth because salient objects may be distinguishable in at least one modality. The paper proposes an end-to-end two-stream network with supervised adaptive fusion and evaluates it across three datasets.

  • Color-only saliency methods often fail when objects resemble their surroundings or appear against complex backgrounds.
  • Depth data can address these challenges, but existing RGB-D methods commonly fuse modalities through feature concatenation or element-wise operations.
  • Four paired-image scenarios show that salient objects are detectable in at least one modality except when both color and depth are cluttered.
  • The proposed two-streamed CNN predicts separate RGB and depth saliency maps using multi-scale feature aggregation.
  • A saliency fusion module learns a supervised switch map to adaptively combine the two predicted saliency maps.
  • The approach is evaluated on NJUD, NLPR, and STEREO, consistently outperforming state-of-the-art methods on all three datasets.

II. RELATED WORK

Prior RGB-D saliency methods address depth saliency and multimodal fusion, but their fusion is generally based on fixed concatenation or element-wise operations. This paper instead adaptively fuses modality-specific predictions.

  • RGB Saliency Detection: RGB saliency research progressed from hand-crafted priors to CNN architectures using multi-scale, branched, hierarchical, and progressively connected features.
  • RGB-D Saliency Detection: RGB-D saliency research centers on modeling depth-induced saliency and fusing RGB and depth modalities.
  • RGB-D Saliency Detection: Hand-crafted depth features and priors can detect depth saliency, but their performance is limited by manual designs and multi-stage models.
  • RGB-D Saliency Detection: Existing multimodal approaches fuse inputs, features, or predictions using concatenation, addition, multiplication, or progressively structured fusion networks.
  • RGB-D Saliency Detection: The proposed method differs by adaptively fusing RGB and depth prediction results rather than relying mainly on fixed feature or prediction operations.

III. THE PROPOSED METHOD

The proposed framework uses parallel RGB and depth streams to generate modality-specific saliency maps, progressively aggregate multi-scale features, and combine the streams through a learned fusion pathway.

  • A two-streamed network predicts RGB and depth saliency maps separately from unimodal information, while concatenated final-layer features generate a switch map for fusion.
  • Each unimodal stream progressively aggregates features across scales before predicting its saliency map.
  • Each stream is built on VGG-16 with five convolutional blocks, omitting the final pooling and fully connected layers.
  • The feature-aggregation operation uses convolution, nonlinear activation, bilinear upsampling, channel-wise concatenation, and a final saliency prediction layer.
  • The RGB stream accepts a 3-channel color image and the depth stream accepts a 1-channel depth map; their structures are shared but parameters differ.

B. Saliency Fusion Module

The saliency fusion module learns a spatial switch map from RGB and depth features, using it to weight modality-specific predictions rather than combining them with fixed operations.

  • The module concatenates the streams’ last-layer features, predicts a switch map, and produces a fused saliency map from RGB and depth predictions.
  • A pseudo ground-truth switch map supervises the learned switch map so fusion can select the more reliable modality across scenarios.
  • The switch target assigns 1 when the RGB prediction agrees with the ground truth on saliency and nonsaliency, and 0 otherwise.
  • The switch map contains values in [0, 1], so the final prediction is a weighted sum rather than a hard modality choice.

C. Loss Function

The loss function jointly supervises unimodal and fused saliency predictions, the switch map, and edge preservation. This fully supervised design trains the network to fuse modality predictions while maintaining ground-truth boundaries.

  • Loss design: The total loss combines saliency, switch-map, and edge-preserving terms.These are denoted Lsal, Lsw, and Ledge, respectively.
  • Saliency loss: Ground-truth supervision is applied to the RGB, depth, and fused saliency maps using cross-entropy loss.The three predicted maps are Srgb, Sd, and Sfused.
  • Switch loss: The switch map is trained with cross-entropy against a pseudo ground-truth switch map.SWi,j represents the probability that a pixel chooses the RGB prediction.
  • Edge-preserving loss: The edge-preserving loss minimizes horizontal and vertical gradient differences between the fused saliency map and ground truth.The gradients are represented by ∂x(·) and ∂y(·).

D. Implementation Details

The network uses a VGG-16 backbone and is trained end-to-end with the proposed loss function. Training uses Adam with batch size 8, learning rate 10^-4, and 224 × 224 inputs.

  • Implementation: The implementation uses TensorFlow with VGG-16 as the backbone for comparison with previous works.Parameters outside VGG-16 are initialized via Xavier.
  • Optimization: The entire network is trained end-to-end using Adam with batch size 8 and learning rate 10^-4.The optimization uses the loss function described in the paper.
  • Preprocessing: Input images are resized to 224 × 224 for both training and testing.

A. Datasets

The evaluation uses three publicly available RGB-D saliency datasets with different sources and scene conditions. Performance is measured using PR curves, F-measures, and MAE.

  • Datasets: The experiments use NJUD, NLPR, and STEREO as publicly available evaluation datasets.NJUD contains 2003 binocular pairs, NLPR contains 1000 Kinect images, and STEREO provides 797 pairs.
  • Datasets: NJUD combines binocular image pairs collected from the Internet, 3D movies, and photographs.The dataset contains 2003 pairs.
  • Datasets: NLPR contains Kinect images spanning indoor and outdoor scenes under different illumination conditions.The dataset consists of 1000 images.
  • Data split: The test set uses the remaining NJUD and NLPR data together with the full STEREO dataset, after horizontal-flip augmentation of training samples.The split includes 1400 NJUD and 650 NLPR training samples, with 100 NJUD and 50 NLPR validation pairs.
  • Evaluation metrics: Performance is evaluated with PR curves, maximum and mean F-measures, and mean absolute error.PR curves compare thresholded predictions with ground truth across 255 thresholds in [0, 1]; β^2 is set to 0.3 for F-measure.

C. Ablation Study

The ablation study evaluates the full model against variants that remove edge-preserving loss, switch-map fusion, or modality streams. Component analysis reports mean F-measure scores across three datasets.

  • C. Ablation Study: The component analysis compares the full AF model with variants excluding edge-preserving loss, switch-map fusion, or individual modality streams.AF-Edge removes edge-preserving loss, while AF-Edge-SW removes both switch-map and edge-preserving losses.
  • The effectiveness of the saliency fusion module:: Without switch-map fusion, the model directly concatenates two-stream features and predicts the fused saliency map through a 1 × 1 convolutional layer.
  • The effectiveness of the saliency fusion module:: The saliency fusion module improves mean F-measure over the AF-Edge-SW variant without switch-map fusion.The comparison uses AF-Edge-SW and AF-Edge in Table I.

The effectiveness of the edge-preserving loss:

The edge-preserving loss improves both quantitative performance and saliency-map quality. It reduces boundary blur and supports more coherent, complete objects, while the full method outperforms state-of-the-art methods across evaluation metrics.

  • The effectiveness of the edge-preserving loss:: The full AF model achieves superior performance to AF-Edge when the edge-preserving loss is included.This comparison is reported in Table I.
  • The effectiveness of the edge-preserving loss:: The edge-preserving loss reduces blur around object boundaries and produces more coherent, complete salient objects.Figure 3 compares predictions with and without this loss.
  • Quantitative Comparison:: The proposed method outperforms other state-of-the-art methods on all evaluation metrics in Table II, Fig. 4, and Fig. 5.The comparison includes traditional methods and CNN-based RGB-D saliency networks.
  • Quantitative Comparison:: The proposed method consistently improves PCA's F-measure and MAE on all three datasets.The improvement is especially noted on NLPR, where accurate depth data are collected by Kinect.
  • Qualitative Comparison:: Visual comparisons show sharper boundaries and richer details from the proposed method than from competing approaches.The edge-preserving loss contributes to preserving details and boundaries in the last two rows of Fig. 6.

Failed Cases:

The approach works when salient objects stand out in at least one modality but fails when objects are indistinguishable in both color and depth.

  • Failed Cases:: The proposed approach can detect salient objects when they stand out in at least one modality.
  • Failed Cases:: The method fails when salient objects are not distinguishable in either modality, a scenario also challenging for existing methods.
  • V. CONCLUSION: The conclusion attributes the framework's performance to adaptive fusion of RGB and depth predictions plus an edge-preserving loss.
Loading 1901.01369v2…