PaperlayerView exact revision on arXiv

Source-linked AI summary

Siamese Network for RGB-D Salient Object Detection and Beyond

Keren Fu, Deng-Ping Fan, Ge-Peng Ji, Qijun Zhao, Jianbing Shen, Ce Zhu

arXiv:2008.12134v2cs.CV

TL;DR

RGB-D SOD models often use independent modality-specific networks despite limited high-quality depth data and challenging fusion requirements. JL-DCF instead shares a Siamese backbone for joint RGB-depth learning and uses densely cooperative fusion to discover complementary features. It improves max F-measure by about 2% on average across seven datasets and generalizes to RGBT SOD and video SOD, while its suitability for RGB-D semantic segmentation is constrained by the modalities’ weaker commonality there.

  • Problem

    RGB-D SOD is constrained by limited deep-model development, ineffective feature extraction and fusion, and sparse high-quality depth maps.

  • Method

    JL-DCF uses a Siamese network for shared RGB-depth feature extraction and densely cooperative fusion for complementary feature discovery.

  • Results

    ~2% average improvement in max F-measure over state-of-the-art methods is reported across seven challenging datasets, with comparable or better performance on RGBT SOD and video SOD.

  • Takeaways & Limitations

    JL-DCF provides a general framework reported as robust across RGB-D SOD, RGBT SOD, video SOD, and comparisons with semantic segmentation models.

  • Takeaways & Limitations

    For RGB-D semantic segmentation, Siamese learning may face a large RGB-depth commonality gap because the task identifies category-specific regions.

Abstract

from arXiv · show

Existing RGB-D salient object detection (SOD) models usually treat RGB and depth as independent information and design separate networks for feature extraction from each. Such schemes can easily be constrained by a limited amount of training data or over-reliance on an elaborately designed training process. Inspired by the observation that RGB and depth modalities actually present certain commonality in distinguishing salient objects, a novel joint learning and densely cooperative fusion (JL-DCF) architecture is designed to learn from both RGB and depth inputs through a shared network backbone, known as the Siamese architecture. In this paper, we propose two effective components: joint learning (JL), and densely cooperative fusion (DCF). The JL module provides robust saliency feature learning by exploiting cross-modal commonality via a Siamese network, while the DCF module is introduced for complementary feature discovery. Comprehensive experiments using five popular metrics show that the designed framework yields a robust RGB-D saliency detector with good generalization. As a result, JL-DCF significantly advances the state-of-the-art models by an average of ~2.0% (max F-measure) across seven challenging datasets. In addition, we show that JL-DCF is readily applicable to other related multi-modal detection tasks, including RGB-T (thermal infrared) SOD and video SOD, achieving comparable or even better performance against state-of-the-art methods. We also link JL-DCF to the RGB-D semantic segmentation field, showing its capability of outperforming several semantic segmentation models on the task of RGB-D SOD. These facts further confirm that the proposed framework could offer a potential solution for various applications and provide more insight into the cross-modal complementarity task.

1 INTRODUCTION

RGB-D SOD remains relatively underexplored, while existing fusion methods face feature-learning, training-data, and fusion challenges. JL-DCF addresses these issues by sharing a Siamese backbone across RGB and depth and adding densely cooperative fusion, improving performance and extending to related tasks.

  • Research gaps: Only a few deep learning-based works focus on RGB-D SOD, leaving substantial room for performance improvement.The first CNN-based RGB-D SOD model appeared in 2017, followed by only about a dozen deep models.
  • Research gaps: Early- and late-fusion methods struggle with representative multimodal feature extraction or effective feature fusion.Middle-fusion methods can instead require sophisticated architectures, many parameters, extensive training data, and elaborate training processes.
  • Proposed approach: JL-DCF uses joint learning and densely cooperative fusion to exploit commonality and complementarity between RGB and depth views.The architecture adopts a Siamese CNN for shared RGB-depth feature extraction and a cooperative fusion strategy for combining modalities.
  • Results: ~2% average improvement in max F-measure over state-of-the-art methods is reported across seven challenging datasets.The paper also reports further gains when bridging RGB and RGB-D SOD.
  • Results: JL-DCF is reported as applicable to RGBT SOD and video SOD, with comparable or better performance against state-of-the-art methods.The authors describe this as validating the framework’s robustness and generality across diverse tasks.

2 RELATED WORK

Prior RGB-D SOD research spans handcrafted and deep models organized around early-, late-, and middle-fusion strategies. JL-DCF differs by using a shared Siamese architecture to exchange cross-modal knowledge and reduce parameters, while applying the network beyond saliency detection.

  • RGB-D salient object detection: Traditional RGB-D SOD methods extended handcrafted RGB saliency cues such as contrast, depth, center-surround, and boundary priors.These approaches incorporated disparity contrast, domain knowledge, and other manually designed hypotheses.
  • RGB-D salient object detection: Deep RGB-D SOD models evolved from shallow CNN cue fusion toward fully convolutional architectures with increasingly explicit cross-modal processing.Examples include superpixel-based CNNs, depth-feature branches, recurrent CNNs, encoder-decoder models, and contrast-enhanced depth priors.
  • Fusion strategies: Early-fusion concatenates inputs, late-fusion uses parallel networks, and middle-fusion combines modality features at intermediate stages.These three categories are illustrated in Fig. 2.
  • Fusion strategies: Early-fusion may miss complementary interactions and become trapped learning primarily one modality, whereas late-fusion explicitly preserves both modality streams.The cited discussion contrasts early-fusion’s blended input with late-fusion’s parallel feature extraction and separate supervision.
  • Siamese networks: JL-DCF is a middle-fusion method whose RGB and depth streams share both network architecture and weights through a Siamese design.The shared architecture enables cross-modal knowledge-sharing and substantially reduces parameters by using one network.
  • Siamese networks: Unlike conventional Siamese applications based on matching or distance, this paper fuses RGB and depth cues for saliency prediction.The paper positions this as an introduction of Siamese networks to multimodal saliency detection.
  • Related tasks: RGB-D semantic segmentation labels pixels within predefined categories, whereas RGB-D SOD segments category-agnostic salient object regions.The paper identifies a potential correlation between the fields and evaluates JL-DCF against semantic segmentation models on RGB-D SOD.

3 METHODOLOGY

JL-DCF uses a shared Siamese backbone to learn RGB–depth commonality, then densely fuses complementary multi-scale features for saliency prediction. Its design also supports single-modality inference and joint RGB/RGB-D training.

  • 3 METHODOLOGY: JL-DCF comprises joint learning and densely cooperative fusion components within a bottom-up/top-down architecture.The JL component learns from both modalities with a Siamese network, while DCF performs multi-scale cross-modal fusion.
  • 3 METHODOLOGY: The shared Siamese backbone uses the same architecture and weights for RGB and depth feature extraction.Depth maps are normalized, converted to three channels, and batch-concatenated with RGB images for parallel processing.
  • 3.2 Densely Cooperative Fusion (DCF): The CM module splits batch features and combines RGB and depth tensors through element-wise addition and multiplication.Addition exploits feature complementarity, whereas multiplication emphasizes feature commonality; the blended output retains k channels.
  • 3.2 Densely Cooperative Fusion (DCF): Dense decoder connections blend RGB and depth features across scales through feature aggregation modules using multi-level convolutions and max-pooling.The FA modules use 1×1, 3×3, and 5×5 filters plus max-pooling while maintaining spatial feature sizes.
  • 3.3 Loss Function: The final saliency map is supervised alongside coarse predictions using a total loss that combines final and global guidance losses.The final prediction comes from FA1, while coarse predictions are produced after CP6; λ balances global guidance.
  • 3.4 Bridging between RGB and RGB-D SOD: JL-DCF can operate on a single RGB or depth modality by replacing batch-related operations with identity mappings.The dense decoder and deep supervision remain unchanged, producing a full-resolution saliency estimate.
  • 3.4 Bridging between RGB and RGB-D SOD: The JL component is jointly optimized for RGB and RGB-D SOD by combining RGB-only and RGB-D data during online training.RGB SOD datasets are larger than RGB-D datasets, and the RGB task supplies a coarse saliency map from JL.

4 EXPERIMENTS

JL-DCF consistently improves RGB-D SOD performance and generalizes across datasets and related multimodal tasks. Ablations attribute gains to joint learning, cross-modal fusion, dense aggregation, and stronger backbones, while identifying data-quality and efficiency boundaries.

  • Comparisons to SOTAs: JL-DCF achieves the best results compared with existing techniques on the evaluated RGB-D SOD datasets.The comparison reports notable gains over CPFP, D3Net, and DMRA across four metrics.
  • Comparisons to SOTAs: JL-DCF improves depth utilization through cross-modal compensation, while deeply supervised coarse predictions support subsequent adaptive refinement.The densely cooperative fusion architecture learns an image-dependent fusion process, and fusion does not degrade either RGB or depth views.
  • Ablation Studies: Replacing CM modules with concatenation biases learning toward RGB, whereas explicit RGB-depth fusion consistently improves performance except when depth maps are poor.On STERE, unreliable depth produces substantially worse depth-only performance, about 16%/20% lower on Sα/F max β than RGB.
  • Ablation Studies: A shared-backbone joint-learning configuration converges more reliably than separate learning, whose alternatives incur 1.1%/1.76% overall drops on Sα/F max.Separate backbones can become trapped at high loss or converge slowly depending on the learning rate.
  • Ablation Studies: Removing all FA modules causes an average ∼1.38% F max β drop, while dense connections improve robustness and DenseNet-161 further boosts performance.Residual connections are also competitive on several datasets, and stronger backbones improve the JL-DCF framework.
  • Computational Efficiency: The JL component dominates runtime, while DCF takes only 0.024s; lightweight backbones could accelerate inference but may affect accuracy.The computational cost of CM, FA, and dense connections is described as small relative to the backbone.
  • Application to Other Multi-modal Fusion Tasks: JL-DCF generalizes beyond RGB-D SOD: RGB-D-trained models perform well on RGB-T, and the framework achieves competitive VSOD results without long-term temporal modeling.On VT821, JL-DCF∗(T) surpasses MIED by 2.6% on Sα; for VSOD, JL-DCF achieves the best result on six of ten scores.

5 CONCLUSION

JL-DCF is a joint-learning and densely cooperative fusion framework for RGB-D salient object detection. It achieves strong benchmark performance and shows robustness across related multimodal tasks.

  • JL-DCF achieves superior performance against state-of-the-art methods on seven benchmark datasets.
  • JL-DCF generalizes to RGB-T SOD, video SOD, and comparisons with semantic segmentation models.
  • The framework is presented as a potential unified approach for multimodal feature learning and fusion tasks.
Loading 2008.12134v2…