PaperlayerView exact revision on arXiv

Source-linked AI summary

Bifurcated backbone strategy for RGB-D salient object detection

Yingjie Zhai, Deng-Ping Fan, Jufeng Yang, Ali Borji, Ling Shao, Junwei Han, Liang Wang

arXiv:2007.02713v3cs.CV

TL;DR

RGB-D salient object detection must aggregate multi-level features and fuse RGB with depth despite noisy low-level features and differing modality cues. BBS-Net uses bifurcated teacher-student features, cascaded refinement, and depth-enhanced attention, outperforming 18 SOTA models across eight datasets while supporting dataset generalization analysis.

  • Problem

    Existing RGB-D SOD methods directly fuse multi-level features and modalities, allowing noisy low-level features and inefficient RGB-depth combinations; dataset generalization is also underexplored.

  • Method

    BBS-Net splits cross-modal multi-level features into teacher and student groups, uses an initial map to refine student features, and enhances depth cues through channel and spatial attention.

  • Results

    BBS-Net outperforms 18 SOTA models on eight challenging datasets under multiple evaluation metrics.

  • Takeaways & Limitations

    The paper provides a cascaded RGB-D SOD architecture and a training set with strong generalization ability for future research.

Abstract

from arXiv · show

Multi-level feature fusion is a fundamental topic in computer vision. It has been exploited to detect, segment and classify objects at various scales. When multi-level features meet multi-modal cues, the optimal feature aggregation and multi-modal learning strategy become a hot potato. In this paper, we leverage the inherent multi-modal and multi-level nature of RGB-D salient object detection to devise a novel cascaded refinement network. In particular, first, we propose to regroup the multi-level features into teacher and student features using a bifurcated backbone strategy (BBS). Second, we introduce a depth-enhanced module (DEM) to excavate informative depth cues from the channel and spatial views. Then, RGB and depth modalities are fused in a complementary way. Our architecture, named Bifurcated Backbone Strategy Network (BBS-Net), is simple, efficient, and backbone-independent. Extensive experiments show that BBS-Net significantly outperforms eighteen SOTA models on eight challenging datasets under five evaluation measures, demonstrating the superiority of our approach ($\sim 4 \%$ improvement in S-measure $vs.$ the top-ranked model: DMRA-iccv2019). In addition, we provide a comprehensive analysis on the generalization ability of different RGB-D datasets and provide a powerful training set for future research.

I. INTRODUCTION

RGB-D salient object detection must combine complementary modalities and multi-level features while handling noisy low-level cues and dataset generalization. BBS-Net addresses these issues through cascaded refinement, depth enhancement, and cross-dataset analysis.

  • Challenges: Existing RGB-D methods struggle because direct multi-level fusion admits noisy low-level features that can confuse models with background regions.RGB and depth also encode different information, making simple fusion inefficient.
  • Method: BBS-Net uses teacher features to generate an initial saliency map that refines student features, suppressing distractors in lower-level representations.The method splits multi-level features into two groups and applies cascaded decoding.
  • Method: Depth-enhanced attention extracts informative channel and spatial cues before complementary RGB-depth fusion.The DEM applies two sequential attention operations to depth features from multiple side-output layers.
  • Results: BBS-Net outperforms 18 SOTA methods on eight public datasets across multiple evaluation metrics, while producing sharper edges and fewer background distractors.The comparison includes max F-measure, MAE, S-measure, max E-measure, and precision-recall curves.
  • Dataset analysis: Cross-dataset experiments evaluate dataset quality and introduce a training set intended to support fair comparisons and future RGB-D SOD research.The paper identifies dataset generalization ability as an overlooked issue.

II. RELATED WORKS

RGB-D salient object detection has evolved from handcrafted cues to deep multi-level and multi-modal feature fusion. Existing approaches differ in how they aggregate scales and combine RGB with depth, while noisy depth remains a challenge.

  • Traditional Models: Handcrafted RGB-D methods rely on contrast, edge, color, texture, region, spatial, and background priors to estimate saliency.These methods combine local and global cues but lack high-level semantic guidance.
  • RGB-D Fusion: Treating depth as a fourth RGB channel or combining modalities by simple operations can be unreliable because RGB and depth encode different information.RGB captures color and texture, whereas depth captures spatial relations.
  • Deep Models: Deep RGB-D models extract semantic, contextual, multi-level features with CNNs and combine RGB and depth through different fusion architectures.Reported architectures include single-stream, double-stream, and three-stream designs.
  • BBS-Net: BBS-Net differs from single-decoder aggregation by using bifurcated multi-level features, a depth-enhanced module, and a depth adapter for efficiency.The network combines cascaded refinement with enhanced depth processing.
  • RGB-D Fusion: Depth maps may contain noise or misleading information, motivating contrast enhancement, depth filtering, uncertainty modeling, depth prediction, and joint learning.These strategies aim to improve the usefulness of depth for saliency detection.

III. PROPOSED METHOD

BBS-Net organizes multi-level cross-modal features into teacher and student groups for cascaded saliency refinement. It uses lightweight decoders and progressive upsampling to suppress distractors while recovering details.

  • Bifurcated Backbone Strategy: BBS divides cross-modal features into G1 = {Conv1, Conv2, Conv3} and G2 = {Conv3, Conv4, Conv5}, preserving multi-scale information in both groups.Conv3 is the split point between the two groups.
  • Cascaded Refinement: The cascaded refinement mechanism first predicts an initial saliency map from teacher features, then uses it to refine student features for detail enhancement.The two stages use separate cascaded decoders, with the initial map guiding student-feature refinement.
  • Cascaded Decoder: A lightweight cascaded decoder combines each feature group through global context modules and pyramid multiplication followed by progressive concatenation.Higher-level features modulate lower-level features before aggregation.
  • Cascaded Decoder: The second decoder uses a progressively transposed module to generate the final saliency map from its lower-resolution output.The module contains residual-based transposed blocks and sequential 1 × 1 convolutions.
  • Cascaded Refinement: Unlike several prior refinement strategies, BBS-Net requires only one saliency-refinement round, reducing training time and computational resources.The comparison is made with CRN, SRM, R3Net, and RFCN.

C. Depth-Enhanced Module (DEM)

The depth-enhanced module improves RGB-depth compatibility and extracts informative depth cues through sequential channel and spatial attention. It uses global max pooling to focus on critical depth information while preserving multi-level features.

  • DEM Design: DEM addresses modality incompatibility and redundancy in low-quality depth maps before RGB-depth fusion.It is inserted before each side-out depth feature map and preserves multi-level, multiscale information.
  • Attention Operations: DEM applies sequential channel attention and spatial attention to excavate informative cues from depth features.The module operates on depth features from multiple side-out layers.
  • Attention Operations: The module uses global max pooling to select critical depth cues, reducing complexity for salient-object detection.Spatial attention pools along the channel axis, while channel attention pools each feature map.
  • DEM Design: DEM improves the compatibility of multi-modal features compared with direct concatenation, simple convolution, or contrast-prior enhancement.The paper reports its effectiveness through experiments referenced in Tab. VI and Fig. 8.

D. Improve the efficiency of BBS-Net.

The efficient BBS-Net version shares backbone parameters but first adapts depth inputs to account for RGB-depth modality differences. It also removes the final progressive transposed module with negligible performance degradation.

  • Motivation: The original BBS-Net uses separate, non-shared backbones, which improves modality-specific feature extraction but increases parameters.Sharing weights directly causes a substantial performance degradation according to the paper.
  • Depth Adapter Module: The depth adapter module computes modality differences between RGB and depth images before extracting adapted depth features with the shared backbone.The depth image is broadcast to RGB dimensionality when calculating the difference.
  • Efficient Version: In the efficient version, the backbones share parameters and the depth image is passed through DAM before backbone processing.The adapter is intended to preserve performance despite weight sharing.
  • Efficient Version: Removing the last progressively transposed module further reduces parameters with negligible performance degradation.This change is specific to the efficient version of BBS-Net.

E. Implementation Details

Training jointly supervises initial and final saliency maps using a combined loss. The implementation uses a ResNet-50 backbone, standard optimization settings, resized inputs, and five evaluation measures.

  • Training Loss: The model predicts initial and final saliency maps and jointly optimizes both cascaded stages against binary ground truth.The total loss combines the two stage losses with a trade-off parameter α.
  • Training Loss: Binary cross entropy is used to supervise predicted saliency maps against the binary ground-truth map.The supplied formulation defines the cross-entropy term using predicted map S and ground truth G.
  • Training and Test Protocol: The implementation uses ImageNet-pretrained ResNet-50 side outputs, Adam optimization, and 352 × 352 RGB-depth inputs.The two branches use separate weights in the main model, while the efficient version shares weights after depth adaptation.

A. Experimental Settings

The experiments evaluate BBS-Net on eight RGB-D salient object detection datasets using established training splits, five metrics, and comparisons with handcrafted and deep-learning methods.

  • Datasets: Eight challenging RGB-D SOD datasets are used, including NJU2K, NLPR, STERE, DES, LFSD, SSD, SIP, and DUT.NJU2K contains 1,985 image pairs, while several smaller datasets target indoor, outdoor, stereoscopic, and challenging scenes.
  • Training/Testing: The training set contains 1,485 NJU2K samples and 700 NLPR samples, while testing uses held-out NJU2K and NLPR images plus complete additional datasets.The evaluation follows settings from prior work for fair comparison.
  • Evaluation Metrics: Performance is evaluated with S-measure, E-measure, F-measure, MAE, and precision-recall curves.The reported metrics include Sα, Eξ, Fβ, and mean absolute error.

B. Comparison with SOTAs

BBS-Net outperforms the compared RGB-D salient object detection methods across broad quantitative and qualitative evaluations, while ablations support its refinement, depth, and decoder components.

  • Comparison with SOTAs: Performance gains over the best compared algorithms are 2.5% ∼3.5% in Sα, 0.7% ∼3.9% in maxFβ, 0.8% ∼2.3% in maxEξ, and 0.009 ∼0.016 in M on seven datasets.BBS-Net outperforms handcrafted-feature and SOTA CNN-based methods across all four reported metrics.
  • Comparison with SOTAs: BBS-Net achieves 24.32 fps on a single GTX 1080Ti GPU, while BBS-Net⋇ uses 25.96M versus 49.77M parameters with similar performance.The efficient version is described as suitable for processing more images with larger batch sizes.
  • Comparison with SOTAs: BBS-Net exceeds TANet, CPFP, and DMRA with VGG-16, VGG-19, or ResNet-50 backbones, with ResNet-50 performing best among the three.VGG-19 and VGG-16 have similar performances.
  • Comparison with SOTAs: Qualitative comparisons show sharper segmentation, fewer background distractors, and better handling of small objects, multiple objects, complex backgrounds, and low-contrast scenes.The method also reduces adverse effects from misleading or low-quality depth maps in the shown examples.
  • Ablation Study: High-level feature aggregation improves localization over low-level aggregation, while the proposed cascaded refinement suppresses low-level noise and yields sharper edges than direct five-level fusion.Reverse refinement performs worse because it cannot effectively suppress noise in low-level features.
  • Ablation Study: The progressively transposed module improves performance on all datasets and produces sharper edges with finer details, while the cascaded decoder outperforms element-wise summation on six of seven datasets.The cascaded decoder achieves comparable results on SIP.

A. Utility of Depth Information

The analysis shows that depth information improves salient-object detection by supplying spatial and contour cues, while post-processing and difficult scenes expose remaining strengths and failure modes.

  • Depth Information: Depth information provides spatial distance and contour guidance, improving salient-object detection and producing clearer borders in cross-modal features.Depth features activate strongly on object borders, making cross-modal features clearer than RGB features alone.
  • Post-processing Methods: CRF achieves the best overall post-processing performance, while adaptive thresholding performs worst and can expand background blur.Ostu also reduces background noise but may not fully eliminate it.
  • Failure Cases: The model fails on transparent objects, camouflage-like backgrounds, complex scenes, and images containing several separate salient objects.These failures arise when RGB contrast or layout is ambiguous, depth lacks useful information, or multiple objects are not all detected.
  • Failure Cases: Background interference from color, contrast, and content accounts for many failure cases, while high-quality depth maps may resolve some of them.The authors also suggest adding boundary priors and more challenging training examples.

D. Cross-Dataset Generalization Analysis

The cross-dataset analysis evaluates how RGB-D training datasets transfer across domains and how dataset combinations affect generalization. It identifies difficult datasets, a strong generalizing dataset, and the importance of selected training mixtures.

  • Single Dataset Generalization Analysis: NJU2K and DUT are the hardest datasets for cross-dataset generalization because they contain challenging scenes such as transparent objects, multiple objects, and complex backgrounds.The analysis compares self-dataset performance with mean performance on other datasets.
  • Single Dataset Generalization Analysis: STERE has the best generalization ability among the evaluated datasets, based on the reported average drops in Sα and Fβ.The supplied passage states this conclusion but truncates the corresponding values.
  • Dataset Combination for Generalization Improvement: More training examples do not necessarily improve performance on every test set, as several larger combinations perform similarly to NJU2K-plus-NLPR training on NLPR.This indicates that dataset size alone is insufficient to guarantee better cross-dataset results.
  • Dataset Combination for Generalization Improvement: Including NJU2K is important for generalizing to small datasets such as LFSD and SSD.The supplied passage contrasts combinations that include NJU2K with those that omit it.
  • Conclusion: The paper presents BBS-Net as a cascaded RGB-D refinement network with a depth-enhanced module and reports superiority over 18 SOTA models across eight datasets.It also analyzes RGB-D dataset generalization and introduces a training set for future research.
Loading 2007.02713v3…