PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection

Chen Zhang, Runmin Cong, Qinwei Lin, Lin Ma, Feng Li, Yao Zhao, Sam Kwong

arXiv:2108.01971v1cs.CV

TL;DR

RGB-D salient object detection must integrate RGB appearance and depth cues despite prior methods treating the modalities equally or using depth only as auxiliary information. CDINet assigns them different roles across feature levels and upgrades decoder skip guidance; it reports competitive performance on five benchmark datasets at 42 FPS.

  • Problem

    RGB-D SOD methods often treat RGB and depth equally or use depth only as auxiliary information, despite their distinct strengths in appearance and geometric cues.

    Source: §1 · pp. 1–2

  • Method

    CDINet uses RGB to enhance low-level depth detail, depth to guide high-level RGB semantics, and a dense semantic block to improve decoder skip guidance.

    Source: §1 · p. 2

  • Results

    CDINet achieves competitive performance against state-of-the-art methods on five benchmark datasets and runs at 42 FPS.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The design assigns RGB and depth complementary roles according to feature level rather than treating them as interchangeable inputs.

    Source: §1 · p. 2

Abstract

from arXiv · show

The popularity and promotion of depth maps have brought new vigor and vitality into salient object detection (SOD), and a mass of RGB-D SOD algorithms have been proposed, mainly concentrating on how to better integrate cross-modality features from RGB image and depth map. For the cross-modality interaction in feature encoder, existing methods either indiscriminately treat RGB and depth modalities, or only habitually utilize depth cues as auxiliary information of the RGB branch. Different from them, we reconsider the status of two modalities and propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD, which differentially models the dependence of two modalities according to the feature representations of different layers. To this end, two components are designed to implement the effective cross-modality interaction: 1) the RGB-induced Detail Enhancement (RDE) module leverages RGB modality to enhance the details of the depth features in low-level encoder stage. 2) the Depth-induced Semantic Enhancement (DSE) module transfers the object positioning and internal consistency of depth features to the RGB branch in high-level encoder stage. Furthermore, we also design a Dense Decoding Reconstruction (DDR) structure, which constructs a semantic block by combining multi-level encoder features to upgrade the skip connection in the feature decoding. Extensive experiments on five benchmark datasets demonstrate that our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively. Our code is publicly available at: https://rmcong.github.io/proj_CDINet.html.

1 INTRODUCTION

The introduction motivates layer-dependent interaction between RGB and depth, then presents CDINet’s distinct encoder guidance and dense decoding reconstruction. It reports competitive benchmark performance and illustrates improved detection in a difficult scene.

  • In a difficult scene, CDINet detects the salient object with complete structure and a clear background, unlike the illustrated ATSA and ASIF outputs.Source: Figure 1 · p. 1, §1 · p. 2
  • RGB supplies color and texture, while depth provides geometric structure and internal consistency that can help with low-contrast scenes and complex backgrounds.Source: §1 · pp. 1–2
  • RDE transfers RGB detail to low-level depth features, while DSE uses high-level depth positioning and internal consistency to enhance RGB semantics.Source: §1 · p. 2
  • DDR densely connects higher-level encoder features into a semantic block that provides more comprehensive guidance for decoder skip connections.The introduction notes that convolution loses spatial detail and direct addition or concatenation does not fully use features from different encoder layers.Source: §1 · p. 2
  • CDINet achieves competitive performance against 15 state-of-the-art methods on five RGB-D SOD datasets and runs at 42 FPS.Source: §1 · p. 2

2 RELATED WORK

Related work groups RGB-D SOD methods by how they integrate modalities and reviews attention mechanisms and skip-connection designs. The paper positions its approach as differentiated cross-modal interaction with dense higher-level encoder guidance for decoding.

  • Prior RGB-D SOD methods commonly either treat RGB and depth equally through bidirectional integration or use depth as auxiliary information for RGB.Examples include joint or dense fusion methods and methods that transfer depth knowledge or decoding features to the RGB stream.Source: §2 · p. 2
  • Attention methods include spatial, channel, self-attention, and mixed forms; this paper uses spatial- and channel-wise attention for cross-modal enhancement.Source: §2 · p. 3
  • Prior skip connections use direct addition or concatenation, or decoder-specific fusion blocks; this work densely connects higher-level encoder features for skip guidance.Source: §2 · p. 3

3 PROPOSED METHOD

CDINet uses layer-specific cross-modal guidance: RGB details enhance low-level depth features, depth semantics enrich high-level RGB features, and dense reconstruction refines decoder skip connections. Its encoder-decoder architecture uses separate RGB and depth encoders and a decoder.

3.1 Overview

  • CDINet differentially guides RGB and depth features by layer, using RGB detail cues for depth at low levels and depth positioning and consistency cues for RGB semantics at high levels.The overview describes separate RGB and depth encoders with VGG16 backbones and a decoder.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 RGB-induced Detail Enhancement

  • RDE fuses low-level RGB and depth features, then uses depth-derived spatial attention to supply relevant detail cues to the depth branch.A 1×1 convolution reduces channels and a 3×3 convolution forms the fused feature pool; the depth features guide masking to suppress irrelevant RGB information.Source: §3.2 · pp. 3–4

3.3 Depth-induced Semantic Enhancement

  • DSE uses high-level depth features to guide RGB semantics through spatial and channel attention, then fuses enhanced features across modalities for stronger salient-object responses.Depth features offer positioning accuracy and internal consistency, while feature-level addition is used to strengthen salient-object responses.Source: §3.3 · p. 5

3.4 Dense Decoding Reconstruction

  • DDR densely combines higher-level encoder features into a semantic block that filters each skip connection before it is joined with decoder features.Element-wise multiplication suppresses redundant skip information, while a residual connection preserves the original information as decoding progressively restores image details.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments compare CDINet with 15 CNN-based RGB-D SOD methods across five benchmark datasets and examine its components and interaction modes. Reported results favor CDINet overall, with ablations also exposing computational trade-offs.

4.1 Datasets and Evaluation Metrics

  • Training uses 2,985 images from NJUD, NLPR, and DUT; evaluation spans NLPR, NJUD, DUT, STEREO, and LFSD.The evaluation includes S-measure, max F-measure, and MAE.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Implementation Details

  • At 256 × 256, inference takes 0.023 seconds (42 FPS) on an RTX 2080Ti, and the model trains end-to-end for 100 epochs.The method requires no depth preprocessing or post-processing.Source: §4.2 · p. 6

4.3 Comparisons with SOTA Methods

  • Across five datasets, CDINet outperforms all compared methods except for S-measure on NLPR.The comparison reports S-measure, max F-measure, and MAE.Source: Table 1 · p. 7, §4.3 · p. 7
  • Qualitative comparisons show better handling of small and multiple objects, disturbing backgrounds, and confusing depth maps, with sharper boundaries and more consistent object interiors.The authors attribute improved internal consistency to DSE and sharper boundaries to DDR.Source: §4.3 · p. 7

4.4 Ablation Studies

  • Adding RDE yields MAE percentage gains of 4.4% on NLPR and 10.7% on DUT.RDE transfers detail-supplement information from RGB to depth features in the low-level encoder.Source: §4.4 · p. 8
  • Without DSE, max F-measure falls 1.1% on NLPR and 1.6% on DUT; visualization shows stronger object response and suppressed background noise.The visualization is of DSE in the backbone’s last layer.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Replacing DDR with corresponding-layer skips reduces performance; DDR gains 0.8% in max F-measure and 9.6% in MAE score on DUT.The ablation compares the proposed decoder with a U-net-like skip connection.Source: Table 2 · p. 7, §4.4 · p. 8
  • Compared with full model No.1, No.2 performs worse on DUT; bidirectional No.3 is optimal on almost all indicators except DUT S-measure, but raises parameters by 10M and changes inference speed from 42 to 32 FPS.Source: §4.4 · p. 8

5 CONCLUSION

CDINet differentiates cross-modality guidance across convolutional layers and uses multi-level encoder features to upgrade skip connections. It achieves competitive performance on five benchmark datasets at 42 FPS.

5 CONCLUSION

  • CDINet achieves competitive performance against state-of-the-art methods on five benchmark datasets and runs at 42 FPS.Source: §5 · p. 8
  • CDINet explicitly models the dependence of RGB and depth modalities in different convolutional layers through differentiated guidance.Source: §5 · p. 8
  • DDR generates a semantic block from multiple high-level features to upgrade the decoder skip connection.Source: §5 · p. 8

ABSTRACT

CDINet addresses RGB-D encoder interaction by assigning modality guidance according to feature level and augmenting decoder skip connections. The paper reports better quantitative and qualitative results than 15 state-of-the-art methods on five benchmark datasets.

ABSTRACT

  • CDINet outperforms 15 state-of-the-art methods quantitatively and qualitatively on five benchmark datasets.Source: ABSTRACT · p. 10
  • The paper addresses encoder methods that either treat RGB and depth indiscriminately or use depth only as auxiliary information for RGB.Source: ABSTRACT · p. 10
  • RDE uses RGB features to enhance details in depth features at the low-level encoder stage.Source: ABSTRACT · p. 10
  • DSE transfers depth-derived object positioning and internal consistency to the RGB branch at the high-level encoder stage.Source: ABSTRACT · p. 10
  • DDR combines multi-level encoder features into a semantic block to upgrade skip connections during decoding.Source: ABSTRACT · p. 10

CCS CONCEPTS

The CCS concept classifies the work under interest point and salient region detections within computing methodologies.

CCS CONCEPTS

  • The listed CCS concept is interest point and salient region detections within computing methodologies.Source: CCS CONCEPTS · p. 10

KEYWORDS

The keywords identify the paper's task, input modality, interaction approach, and decoding structure.

KEYWORDS

A MORE EXPLANATIONS

CDINet assigns different cross-modal roles to low- and high-level features: RGB details enhance depth features, while depth semantics guide RGB features. Ablations and stage placement experiments support these design choices.

The design of RDE and DSE

  • RDE addresses depth’s inability to distinguish same-depth object instances, whereas DSE uses depth cues to help position salient objects against complex backgrounds.Source: A MORE EXPLANATIONS · p. 10
  • RDE transfers RGB detail information to depth features, while DSE uses depth-derived attention and feature addition to guide RGB object positioning and internal consistency.RDE uses a mask generated from depth features to gate fusion features; DSE highlights salient-object locations and suppresses background in RGB features.Source: A MORE EXPLANATIONS · p. 10
  • Replacing RDE with DSE yields F-measures of 0.9142/0.9296 on NLPR/DUT; replacing DSE with RDE yields 0.9107/0.9261, and exchanging both yields 0.9066/0.9254.Source: A MORE EXPLANATIONS · p. 10

The location of RDE and DSE

  • Using RDE in the first three layers and DSE in the last two lowers F-measures from 0.9162/0.9372 to 0.9138/0.9218 on NLPR/DUT.The proposed placement treats the first two layers as low-level and the last three as high-level.Source: A MORE EXPLANATIONS · p. 10

B EXPERIMENTS

CDINet is evaluated against 15 CNN-based RGB-D SOD methods on five benchmark datasets. The reported precision-recall curves favor CDINet across all testing datasets, and visual comparisons report better effects in challenging scenes.

B EXPERIMENTS

  • The evaluation compares CDINet with 15 CNN-based RGB-D SOD methods on NLPR, NJUD, DUT, STEREO, and LFSD.Source: B EXPERIMENTS · pp. 10–11
  • Visual comparisons report better CDINet results for fine object structure, disturbing backgrounds, and easily mistaken objects such as pendants and frisbees.Source: B EXPERIMENTS · p. 11, Figure 5 · p. 12
  • CDINet achieves higher precision and recall than other methods on all five benchmark datasets, according to the reported P-R curves.The curves relate precision to recall; curves nearer the upper right indicate better performance.Source: B EXPERIMENTS · p. 11, Figure 6 · p. 13
Loading 2108.01971v1…