Source-linked AI summary
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion
Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, Xi Li
TL;DR
RGB-D SOD requires effective depth-aware feature modeling and fusion of heterogeneous RGB and depth information. The paper uses depth-sensitive attention to enhance RGB features and reduce background distraction, while NAS searches a task-specific multi-modal multi-scale fusion architecture. Across seven benchmarks, the resulting framework is reported to outperform previous state-of-the-art methods.
Problem
RGB-D SOD must exploit depth geometry and effectively fuse heterogeneous RGB and depth features, but existing fusion designs rely heavily on manual expertise.
Method
The framework uses depth-sensitive attention for RGB feature enhancement and background-distraction reduction, plus NAS over a task-specific multi-modal multi-scale search space.
Results
The framework outperforms previous state-of-the-art approaches across seven standard benchmarks.
Takeaways & Limitations
Depth geometry can guide RGB feature modeling, while automatically searched fusion can provide a powerful architecture for RGB-D saliency detection.
Abstract
from arXiv · showhide
RGB-D salient object detection (SOD) is usually formulated as a problem of classification or regression over two modalities, i.e., RGB and depth. Hence, effective RGBD feature modeling and multi-modal feature fusion both play a vital role in RGB-D SOD. In this paper, we propose a depth-sensitive RGB feature modeling scheme using the depth-wise geometric prior of salient objects. In principle, the feature modeling scheme is carried out in a depth-sensitive attention module, which leads to the RGB feature enhancement as well as the background distraction reduction by capturing the depth geometry prior. Moreover, to perform effective multi-modal feature fusion, we further present an automatic architecture search approach for RGB-D SOD, which does well in finding out a feasible architecture from our specially designed multi-modal multi-scale search space. Extensive experiments on seven standard benchmarks demonstrate the effectiveness of the proposed approach against the state-of-the-art.
1. Introduction
RGB-D SOD must exploit depth geometry while effectively fusing heterogeneous RGB and depth features. The paper addresses these issues with depth-sensitive RGB enhancement and automatically searched multi-modal fusion.
- RGB-D SOD aims to localize and segment salient regions from RGB and depth inputs in an end-to-end image-to-mask pipeline.
- The paper targets two challenges: exploiting depth geometry for saliency analysis and effectively fusing RGB and depth features.
- Existing methods often treat depth as an auxiliary CNN input, limiting use of geometric layouts that distinguish salient objects.Salient objects may occupy particular depth intervals, motivating depth-aware RGB feature extraction.
- A depth-sensitive attention module enhances RGB features and reduces background distraction using depth prior knowledge.
- The method introduces a task-specific multi-modal multi-scale search space and NAS to automatically discover the feature-fusion architecture.The search space is designed for heterogeneous RGB-depth fusion across multiple scales.
- Seven-benchmark experiments show that the method outperforms other state-of-the-art approaches.
2. Related Work
RGB-D SOD research has progressed from handcrafted features to CNN-based single- and multi-stream models, but depth geometry and fusion design remain open concerns. This paper explicitly exploits depth geometry and introduces NAS for RGB-D multi-modal multi-scale fusion.
- Earlier RGB-D SOD methods used handcrafted cues such as contrast, shape, and local background enclosure.
- CNN-based approaches are broadly organized into single-stream models that directly fuse modalities and multi-stream models that process them separately.
- The paper argues that depth cues contain geometric prior knowledge that can explicitly reduce background distraction through depth-sensitive attention.
- NAS automates network architecture design, with gradient-based methods offering efficient search compared with earlier reinforcement-learning and evolutionary approaches.
- RGB-D SOD fusion architectures were still hand-designed, motivating the paper's first NAS attempt for multi-modal multi-scale fusion in this task.
3. Method
DSA2F combines depth-sensitive RGB feature modeling with an automatically searched multi-modal, multi-scale fusion module. Depth decomposition guides RGB enhancement and background suppression, while NAS searches fusion cells across RGB and depth features.
- Framework overview: DSA2F uses RGB and depth branches with DSAMs in the RGB branch and a specially designed fusion module.The RGB branch is based on VGG-19, while the depth branch extracts depth features at different scales.
- Depth-sensitive attention: DSAM decomposes the raw depth map into T + 1 regions and uses the resulting masks to extract depth-sensitive RGB features.The T largest depth-distribution modes define depth interval windows, while the remaining histogram values form the final region.
- Depth-sensitive attention: Each DSAM processes masked RGB features through T + 1 parallel sub-branches, refines them with 1 × 1 convolutions, aggregates them, and adds a residual connection.The masks are resized to the RGB feature-map resolution before channel-wise multiplication and aggregation.
- Depth-sensitive attention: DSAM provides depth-wise geometric prior knowledge for RGB features and eliminates background distraction such as cluttered objects or similar texture.The paper reports that ablation experiments verify the effectiveness of DSAM.
- Automatic multi-modal multi-scale fusion: The searchable fusion module uses MM, MS, GA, and SR cells to combine RGB and depth features across modalities and scales.MM cells fuse adjacent RGB-depth features, MS cells perform dense multi-scale fusion, GA aggregates global context, and SR combines low- and high-level features.
- Architecture search: The fusion search space represents cells as directed acyclic graphs whose edges use softmax-weighted candidate operations with learnable architecture parameters.After searching, each mixed operation is replaced by the most likely operation, and the resulting module is used with a decoder containing bilinear upsampling and convolutional layers.
4. Experiments
Experiments evaluate DSA2F across seven RGB-D benchmarks, compare it with 18 state-of-the-art methods, and test its components through ablations. DSA2F achieves broad benchmark gains, while the studies identify effective depth-region and attention choices.
- Experimental Setup: Experiments use seven RGB-D datasets, four evaluation metrics, and comparisons against 18 state-of-the-art methods.The datasets include DUT-RGBD, NJUD, NLPR, SSD, STEREO, LFSD, and RGBD135; metrics are Fβ, M, Sλ, and Eξ.
- Quantitative Comparison: DSA2F consistently surpasses competing methods in overall Fβ performance across all seven datasets.It also achieves especially large margins on the challenging LFSD and DUT-RGBD datasets.
- Qualitative Comparison: Qualitative results show more accurate salient-object segmentation under low contrast, clutter, blurry depth, and fine-structure conditions.The reported outputs also have clearer, sharper boundaries that preserve more details.
- Ablation Analysis: The depth-sensitive attention module improves the baseline, with element-wise multiplication selected as the best overall fusion operation.The operation directly serves as a spatial attention mechanism.
- Ablation Analysis: DSAM achieves its best accuracy when the depth decomposition uses T + 1 = 3 regions.The number of depth regions is evaluated as a method hyper-parameter.
- Ablation Analysis: Adding spatial and channel attention operations substantially improves performance, and each DSA2F component contributes a significant gain.The search-space ablations also examine MM, MS, GA, SR, and attention operations.
5. Conclusion
The proposed DSA2F framework combines depth-sensitive RGB enhancement with automatically searched multi-modal fusion for RGB-D saliency detection. Experiments demonstrate effectiveness against previous state-of-the-art methods, with visualizations showing precise salient-region capture in challenging scenes.
- DSA2F uses a depth-sensitive attention module to enhance RGB features and reduce background distraction through depth geometry information.
- Its task-specific search space automatically produces a fusion architecture for multi-modal, multiscale feature fusion.
- Experiments demonstrate DSA2F’s effectiveness against previous state-of-the-art RGB-D saliency methods.
- Visualizations show that the network precisely captures salient regions in challenging scenes.