Source-linked AI summary
A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection
Xiaoqi Zhao, Lihe Zhang, Youwei Pang, Huchuan Lu, Lei Zhang
TL;DR
Existing RGB-D salient object detection methods leave challenging appearance and low-contrast scenes unresolved and underexplore depth-map information. This paper proposes a single-stream network with depth-guided fusion and attention modules; it reports stronger performance than competitors while remaining lightweight and real-time at 32 FPS.
Problem
RGB-only saliency is insufficient in complex scenes, while prior RGB-D methods do not explore internal relationships between parallel multi-scale features or the depth map's role deeply.
Method
A single-stream 4-channel encoder performs early RGB-depth fusion using ImageNet pre-training, while depth-enhanced dual attention and pyramidally attended features support decoder fusion and multi-scale localization.
Results
The method performs much better than ten state-of-the-art competitors across six challenging datasets, is much lighter than other models, and runs at 32 FPS.
Takeaways & Limitations
The proposed depth-fusion design improves saliency detection while almost no parameters are increased, supporting robust real-time RGB-D salient object detection.
Abstract
from arXiv · showhide
Existing RGB-D salient object detection (SOD) approaches concentrate on the cross-modal fusion between the RGB stream and the depth stream. They do not deeply explore the effect of the depth map itself. In this work, we design a single stream network to directly use the depth map to guide early fusion and middle fusion between RGB and depth, which saves the feature encoder of the depth stream and achieves a lightweight and real-time model. We tactfully utilize depth information from two perspectives: (1) Overcoming the incompatibility problem caused by the great difference between modalities, we build a single stream encoder to achieve the early fusion, which can take full advantage of ImageNet pre-trained backbone model to extract rich and discriminative features. (2) We design a novel depth-enhanced dual attention module (DEDA) to efficiently provide the fore-/back-ground branches with the spatially filtered features, which enables the decoder to optimally perform the middle fusion. Besides, we put forward a pyramidally attended feature extraction module (PAFE) to accurately localize the objects of different scales. Extensive experiments demonstrate that the proposed model performs favorably against most state-of-the-art methods under different evaluation metrics. Furthermore, this model is 55.5\% lighter than the current lightest model and runs at a real-time speed of 32 FPS when processing a $384 \times 384$ image.
1 Introduction
The paper targets robust, lightweight, and real-time RGB-D salient object detection by replacing costly two-stream fusion with a single-stream design that uses depth in early and middle fusion. It adds depth-enhanced attention and pyramidally attended features, reporting stronger results across challenging datasets and 32 FPS operation.
- Motivation: RGB-only saliency is insufficient in complex scenes where salient objects resemble the background or inter-object contrast is low.Depth maps provide geometric structures that are robust against illumination and texture changes.
- Motivation: Two-stream RGB-D methods increase network parameters, while limited datasets and modality differences make training a separate depth encoder difficult.These constraints motivate a smaller architecture that can exploit pre-trained RGB backbones.
- Proposed approach: The proposed single-stream network performs early and middle fusion, using a 4-channel encoder and explicitly enhancing cross-modal interaction in the decoder.The design preserves ImageNet pre-training for feature extraction while avoiding a separate depth-stream encoder.
- Proposed approach: DEDA uses depth to strengthen mask-guided attention and computes foreground- and background-attended features for the decoder.The module filters mutual interference between depth and appearance priors.
- Proposed approach: PAFE uses self-attention to model spatial relationships across scales and provide contextual guidance for localizing objects with different receptive fields.The module addresses limitations in prior multi-scale feature extraction methods.
- Results: 32 FPS is achieved while the method performs much better than competing methods on six challenging RGB-D SOD datasets.The introduction compares the model with ten state-of-the-art RGB-D SOD methods.
2 Related Work
Prior RGB-D saliency methods commonly fuse separately encoded RGB and depth streams, but this increases parameters and limits depth learning under small datasets. The paper instead integrates depth with RGB from the input stage in a single-stream design.
- Fusion strategies: Depth information can be fused at early, middle, or late stages, while existing deep saliency methods mainly use two streams with fusion at selected stages.The paper identifies early fusion as unexplored in existing deep-learning saliency methods.
- Limitations of prior work: Two-stream designs significantly increase parameters, and their separate depth streams are difficult to train effectively on small RGB-D datasets.The paper also notes that modality differences hinder training from scratch.
- Proposed direction: The proposed approach integrates RGB and depth from the start to build a real single-stream network rather than adding a separate depth-processing stream.This design is presented as an alternative to prior trade-off methods using shallow depth subnets.
- Architecture: The overall architecture uses VGG-16, transition layers, saliency and background decoder layers, PAFE, and DEDA.Residual connections fuse the outputs of the first saliency and background layers for final prediction.
3 Proposed Method
The proposed method uses a single-stream RGB-D encoder and depth-enhanced decoder attention to reduce modality and parameter costs while improving saliency feature extraction. A pyramidally attended module further captures spatial dependencies across multi-scale features.
- Overall Architecture: The FPN-based architecture uses encoder, transition, saliency, and background layers, with the saliency and background layers forming the decoder.Five levels are indexed by i ∈ {1, 2, 3, 4, 5}, and transition layers use 3×3 convolutions for channel matching.
- Single Stream Encoder: The single-stream encoder concatenates RGB and depth into a 4-channel input, retaining ImageNet-pretrained parameters beyond the first convolutional layer.The first block outputs 64 channels and uses He initialization for its first convolution.
- Depth-enhanced Dual Attention: DEDA combines transition features, decoder features, and depth to compute mask-guided attention, then constructs separate foreground and background attended features.Depth-enhanced attention refines misclassified foreground and background regions while preserving contrast between them.
- Depth-enhanced Dual Attention: DEDA remains operative when depth values are small or zero because its attention formulas include terms independent of the depth map.The module also preserves foreground-background contrast despite depth noise and produces dynamic gradients during back-propagation.
- Pyramidally Attended Feature Extraction: PAFE applies parallel dilated convolutions at different scales and spatial attention to emphasize visually important regions and model pairwise feature relationships.The attention map uses reshaped feature representations, while the 1×1 convolution and global-pooling branches preserve input properties without attention.
4 Experiments
Experiments evaluate the model on six RGB-D SOD datasets using standard accuracy metrics, efficiency measures, precision-recall curves, qualitative comparisons, and component ablations. The results report strong accuracy, lightweight real-time operation, and benefits from depth fusion, DEDA, and PAFE.
- Experimental Setup: The model is evaluated on six public RGB-D SOD datasets using PR curves, F-measure, MAE, S-measure, and E-measure.MAE is lower-is-better, while the other listed metrics are higher-is-better.
- Comparison with State-of-the-art Results: The comparison includes ten state-of-the-art methods, with saliency maps supplied by authors or generated from released code for fairness.The compared methods include DES, DCMC, CDCP, DF, CTMF, PCA, MMCI, TANet, CPFP, and DMRA.
- Comparison with State-of-the-art Results: The proposed DANet achieves the best results across six datasets and all six reported metrics, while also attaining the smallest model size and fastest speed.It saves 55.5% of the parameters compared with the second-lightest method, DMRA.
- Comparison with State-of-the-art Results: Qualitative results show more accurate localization and detail capture for multiple objects, slender parts, complex environments, and degraded depth information.Depth guidance helps identify whole objects in complex environments, while the network still outperforms competitors when depth separates foreground and background poorly.
- Ablation Studies: Ablations compare early-fusion strategies, DEDA variants, and PAFE against baselines across five datasets, with CatHe and PAFE showing favorable results.CatHe outperforms other early-fusion methods, while PAFE is more competitive than ASPP without much additional computation.
5 Conclusions
The proposed single-stream RGB-D network combines depth-enhanced attention with pyramidally weighted multi-scale features for robust, real-time salient object detection. It outperforms ten state-of-the-art methods, while using 106.7 MB and running at 32 FPS with VGG-16.
- The model uses a single-stream network with depth-enhanced dual attention to jointly optimize foreground and background predictions.
- Self-attention pyramidally weights multi-scale features to provide contextual information for salient object segmentation.
- The model notably outperforms ten state-of-the-art methods under different evaluation metrics.
- 106.7 MB is the model size with the VGG-16 backbone, and it runs at 32 FPS.