Source-linked AI summary
RGB-D Salient Object Detection via 3D Convolutional Neural Networks
Qian Chen, Ze Liu, Yi Zhang, Keren Fu, Qijun Zhao, Hongwei Du
TL;DR
RGB-D SOD models often fuse modalities at only one encoder-decoder stage, leaving cross-modal integration insufficiently explored. RD3D addresses this gap with progressive 3D-CNN fusion: an inflated 3D encoder pre-fuses RGB and depth, while a 3D decoder with rich back-projection paths performs further fusion. Across six benchmark datasets, it performs favorably against 14 state-of-the-art methods, with average gains of 0.68% in Sα and 0.50% in Fmax over the best SOTA scores.
Problem
Existing RGB-D SOD models generally fuse RGB and depth in only the encoder or decoder, leaving feature aggregation across both stages insufficiently investigated.
Method
RD3D treats depth as another temporal state of RGB and progressively fuses the modalities using an inflated 3D encoder and a 3D decoder with rich back-projection paths.
Results
RD3D surpasses 14 state-of-the-art methods on six benchmark datasets, advancing the best Sα and Fmax scores by average margins of 0.68% and 0.50%.
Takeaways & Limitations
Progressive fusion makes RD3D a fully 3D-CNN-based RGB-D SOD model that performs favorably against existing state-of-the-art approaches.
Abstract
from arXiv · showhide
RGB-D salient object detection (SOD) recently has attracted increasing research interest and many deep learning methods based on encoder-decoder architectures have emerged. However, most existing RGB-D SOD models conduct feature fusion either in the single encoder or the decoder stage, which hardly guarantees sufficient cross-modal fusion ability. In this paper, we make the first attempt in addressing RGB-D SOD through 3D convolutional neural networks. The proposed model, named RD3D, aims at pre-fusion in the encoder stage and in-depth fusion in the decoder stage to effectively promote the full integration of RGB and depth streams. Specifically, RD3D first conducts pre-fusion across RGB and depth modalities through an inflated 3D encoder, and later provides in-depth feature fusion by designing a 3D decoder equipped with rich back-projection paths (RBPP) for leveraging the extensive aggregation ability of 3D convolutions. With such a progressive fusion strategy involving both the encoder and decoder, effective and thorough interaction between the two modalities can be exploited and boost the detection accuracy. Extensive experiments on six widely used benchmark datasets demonstrate that RD3D performs favorably against 14 state-of-the-art RGB-D SOD approaches in terms of four key evaluation metrics. Our code will be made publicly available: https://github.com/PPOLYpubki/RD3D.
Introduction
RGB-D SOD addresses difficult salient-object detection conditions by combining RGB and depth, but existing models generally fuse modalities in only one stage. RD3D instead uses progressive fusion across both encoder and decoder stages, with 3D CNNs and RBPP, and reports gains over 14 SOTA methods on six datasets.
- RGB-D SOD supplements RGB with depth to address weak foreground-background differences and complex scenes affecting single-modal SOD.
- Existing RGB-D models typically perform feature fusion in either the encoder or decoder, limiting cross-modal fusion across the architecture.
- RD3D uses 3D CNNs to pre-fuse RGB and depth in the encoder while enabling explicit decoder fusion without dedicated cross-modal modules.
- The model adds a 3D decoder with rich back-projection paths to exploit 3D convolutional aggregation and complete progressive fusion.
Related Work
RGB-D SOD has progressed from hand-crafted features to deep models grouped by how they fuse RGB and depth. RD3D is framed against these strategies as a 3D-CNN approach using progressive fusion.
- Traditional RGB-D SOD methods rely on hand-crafted contrast, center or boundary priors, and center-surround differences, performing poorly in complex scenes.
- Deep RGB-D models are commonly classified as early-fusion, middle-fusion, or late-fusion methods according to their fusion stage.
- The paper further categorizes methods by cross-modal feature extraction and fusion, including two-stream architectures and input-fusion approaches.
- The authors position 3D convolutions as a new approach for RGB-D saliency detection, treating depth as another time state of RGB.
Methodology
RD3D follows an encoder-decoder architecture in which a 3D encoder pre-fuses modalities and a 3D decoder performs further feature decoding with rich back-projection paths and attention.
- RD3D comprises a 3D encoder and 3D decoder within a typical encoder-decoder architecture.
- The 3D encoder extends a ResNet/VGG-like backbone with 3D convolutions to produce modality-aware multi-level features through cross-modal pre-fusion.
- The 3D decoder uses a UNet-like top-down design augmented with rich back-projection paths and channel-modality attention.
3D Encoder
The 3D encoder stacks RGB and depth as two temporal states and applies inflated 3D convolutions to fuse them progressively. Shared and modality-specific kernel slices preserve modality-aware features while enabling mutual enhancement.
- The encoder normalizes and replicates the single-channel depth map, then stacks RGB and depth into a tensor with temporal dimension T = 2.
- Inflated 3D ResNet replaces conventional 2D convolutions with 3D convolutions whose temporal kernel size is 3.
- A 3D convolution combines RGB and depth slices through temporal weight slices, mutually enhancing the modalities during feature propagation.
- Successive 3D convolutions provide progressive fusion while retaining modality-aware individuality through shared and modality-specific weights.
- The encoder is initialized from 2D ResNet weights by placing them in the central temporal kernel slice and setting the other slices to zero.
3D Decoder with Rich Back-Projection Paths
The decoder aggregates features across resolutions using rich back-projection paths and enhances them with channel-modality attention over channel and temporal dimensions.
- Rich back-projection paths: RBPP adds downsampling flows from high-resolution features to low-resolution ones alongside the decoder’s usual upsampling path.This extends UNet-like top-down refinement with hierarchical cross-resolution aggregation.
- Rich back-projection paths: At level i, downsampling blocks back-project higher-resolution features while an upsampling block incorporates nearby aggregated outputs.Both blocks preserve the temporal-dimension size.
- Decoder feature computation: The decoder computation concatenates reduced encoder features, back-projected lower-level outputs, and nearby higher-level outputs before further processing.The formulation uses temporal concatenation, temporal reduction, and channel-modality attention to produce final level-wise features.
- Channel-modality attention: The channel-modality attention module learns attention weights across both channel and temporal dimensions and adds the attended tensor residually to the input.Its inputs are reshaped to combine modality information into the channel dimension before channel attention is applied.
Experiments and Results
RD3D is evaluated on six paired RGB-depth benchmark datasets using four established metrics that jointly assess salient-object detection quality.
- Datasets: Six public datasets with paired RGB and depth images are used: NJU2K, NLPR, STERE, DES, SIP, and DUTLF-D.The datasets contain between 135 and 1,985 image pairs.
- Metrics: Evaluation uses S-measure (Sα), E-measure (Eφ), F-measure (Fβ), and Mean Absolute Error (M).The four metrics are described as providing comprehensive and reliable comparisons.
3D ResNet
RD3D uses a fully 3D CNN-based encoder-decoder design and progressive RGB-depth fusion. Experiments show benefits from its backbone strategy, RBPP, and CMA components, alongside favorable quantitative and qualitative results.
- 3D ResNet: RD3D replaces 2D ResNet-50 kernels with 3D versions initialized from ImageNet-pretrained 2D weights.Channel-reduction modules map side outputs to 32 channels.
- Comparisons with SOTAs: Across six datasets, RD3D advances the best SOTA scores by 0.68% on Sα and 0.50% on Fmax.The comparison covers 14 deep RGB-D SOD models.
- Comparisons with SOTAs: Visual comparisons report more accurate RD3D detections, including correctly identifying two people as foreground where other methods fail.The qualitative results are described as consistent with the quantitative analysis.
- Backbone Strategies: RD3D compares 3D, input-fusion, two-stream, and Siamese backbone strategies across four large datasets.The decoder receives repeated encoder outputs when needed to maintain temporal dimension T = 2.
- Backbone Strategies: RD3D outperforms the other backbone strategies by a notable margin, while retaining model speed and size close to the two-stream network.Input fusion performs worst, whereas two-stream and Siamese strategies are comparable but below RD3D.
- Other Modules: Ablations show that RBPP and CMA together substantially enhance performance, while moving CMA to the encoder slightly improves benchmarks but increases computation and model size.The final model keeps CMA in the decoder for fair comparison with prior work.
Conclusion
RD3D is a fully 3D CNN-based RGB-D SOD framework that progressively fuses RGB and depth features. Experiments on six benchmark datasets show favorable performance against existing state-of-the-art approaches.
- Conclusion: RD3D progressively fuses RGB and depth through 3D encoder pre-fusion and a 3D decoder with RBPP and CMA modules.The framework performs initial cross-modal fusion, followed by explicit modality-aware feature fusion.
- Conclusion: RD3D is reported as the first fully 3D CNN-based model for RGB-D salient object detection.The paper presents the framework as a novel RGB-D SOD design based on 3D CNNs.
- Conclusion: Experiments on six benchmark datasets show that RD3D performs favorably against existing state-of-the-art RGB-D SOD approaches.The paper also reports detailed ablation studies validating its key components.