Source-linked AI summary
RGB-D Salient Object Detection with Cross-Modality Modulation and Selection
Chongyi Li, Runmin Cong, Yongri Piao, Qianqian Xu, Chen Change Loy
TL;DR
RGB-D SOD must integrate complementary RGB and depth cues despite noisy or misaligned depth information. This paper progressively modulates, selects, and refines cross-modal features, and reports state-of-the-art performance on six RGB-D benchmarks.
Problem
Depth maps provide useful SOD cues but can be noisy or misaligned with RGB, making effective integration of the modalities challenging.
Source: §1 · p. 1Method
The network uses depth-conditioned RGB feature modulation, adaptive selection of spatial and channel features, and saliency-guided position-edge refinement in a coarse-to-fine process.
Source: §1 · pp. 2–3Results
The method achieves state-of-the-art performance on six RGB-D SOD benchmarks.
Source: §5 · p. 14Takeaways & Limitations
The authors report that elaborate cross-modality integration and adaptive selection of spatial and channel features can boost SOD performance.
Source: §5 · p. 14
Abstract
from arXiv · showhide
We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD). The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features. First, we propose a cross-modality feature modulation (cmFM) module to enhance feature representations by taking the depth features as prior, which models the complementary relations of RGB-D data. Second, we propose an adaptive feature selection (AFS) module to select saliency-related features and suppress the inferior ones. The AFS module exploits multi-modality spatial feature fusion with the self-modality and cross-modality interdependencies of channel features are considered. Third, we employ a saliency-guided position-edge attention (sg-PEA) module to encourage our network to focus more on saliency-related regions. The above modules as a whole, called cmMS block, facilitates the refinement of saliency features in a coarse-to-fine fashion. Coupled with a bottom-up inference, the refined saliency features enable accurate and edge-preserving SOD. Extensive experiments demonstrate that our network outperforms state-of-the-art saliency detectors on six popular RGB-D SOD benchmarks.
1 Introduction
RGB-D SOD can use depth cues such as shape and boundaries, but noisy or misaligned depth maps complicate their integration. The paper proposes a coarse-to-fine network that modulates, selects, and refines cross-modal features, reporting state-of-the-art results on six benchmarks.
1 Introduction
- Depth maps offer useful cues, including depth of field, shape, and boundaries, but can be noisy, inconsistent, or misaligned with RGB.The cited detectors CPFP and A2dele can miss salient objects or produce incomplete or serrated boundaries.Source: §1 · p. 1
- cmFM uses corresponding depth features as a prior to modulate RGB features, modeling cross-modal relations while reducing interference from modality inconsistency.This feature transformation is contrasted with fusion strategies that crudely concatenate or add modalities.Source: §1 · p. 2
- AFS gates the fusion of spatial features and emphasizes self- and cross-modal channel features to select saliency-related information and suppress inferior features.The module is also described as mitigating the negative influence of poorly captured depth maps.Source: §1 · p. 2
- The network consistently outperforms state-of-the-art SOD methods on six popular RGB-D benchmarks.Its cross-modality feature modulation and adaptive feature selection are central components.Source: §1 · p. 3
- The cmFM, AFS, and sg-PEA modules coordinate in a coarse-to-fine sequence that modulates, selects, and refines features for bottom-up SOD.The paper reports complete, sharp, and edge-preserving detections in its motivating examples.Source: §1 · pp. 2–3, Figure 1 · p. 2
2 Related Work
Prior SOD work includes bottom-up and top-down approaches, while RGB-D methods learn complementary features from paired images. This paper distinguishes its approach through depth-conditioned feature modulation and joint treatment of self-modal, cross-modal, and spatial features.
2 Related Work
- RGB-D SOD methods use depth cues alongside color, and supervised approaches learn discriminative, complementary features from RGB-D images.The related-work discussion also distinguishes bottom-up from top-down SOD methods.Source: §2 · p. 3
- The paper distinguishes its method by using depth features as priors for multi-level RGB feature modulation and by modeling self-modal, cross-modal, and spatial feature relations.Its AFS module uses channel interdependencies and gated fusion of spatial features.Source: §2 · p. 3
- The architecture sends RGB and depth inputs through cmMS blocks containing cmFM, AFS, and sg-PEA, and produces five saliency maps and five edge maps.The maps range from 14×14 to 224×224, and Smap1 is the final result.Source: Figure 2 · p. 4
3 Our Method
The method progressively integrates RGB-D features, selects saliency-related information, and refines it with position and edge attention. Its cmMS blocks operate across feature levels, followed by bottom-up inference and jointly supervised saliency and edge prediction.
3.1 Overview of Network Architecture
- In each cmMS block, cmFM integrates RGB-D features, AFS selects informative channel and spatial features, and sg-PEA refines them before bottom-up inference produces the fine-scaled saliency map.Higher-level saliency maps and predicted edge maps guide refinement at each level.Source: §3.1 · pp. 4–5
3.2 Cross-modality Feature Modulation (cmFM)
- Depth-conditioned cmFM estimates pixel-wise affine parameters and uses them to scale and shift RGB features, integrating cross-modality cues through feature transformation.The method presents this as an alternative to strategies that crudely concatenate or add modalities.Source: §3.2 · pp. 5–6
3.3 Adaptive Feature Selection (AFS)
- AFS models channel interdependencies within and across modalities, then combines channel attention with gated fusion of multi-modality spatial features to retain informative features and filter misleading ones.The module uses confidence maps for spatial fusion and progressively re-scales channel-wise features.Source: §3.3 · pp. 6–7
3.4 Saliency-Guided Position-Edge Attention (sg-PEA)
- sg-PEA uses up-sampled higher-level saliency maps for position attention and supervised edge maps for edge attention, refining features to emphasize salient positions and edges while reducing cluttered backgrounds.The refined features support bottom-up prediction of the final saliency map.Source: §3.4 · pp. 8–9
3.5 Loss Function
- The network uses standard cross-entropy losses to jointly optimize saliency-map and saliency-edge predictions at multiple levels, with corresponding loss weights.The two weighted losses are specified for each level.Source: §3.5 · pp. 9–10
4 Experiments
Experiments on six RGB-D SOD benchmarks compare the proposed method with state-of-the-art detectors and test its component modules. The method achieves the best reported quantitative results across metrics, while ablations support the contributions of cmFM, AFS, and sg-PEA.
4.1 Benchmark Datasets and Evaluation Metrics
- The evaluation uses six RGB-D datasets and P-R curves, F-measure, MAE score, and S-measure to compare saliency predictions with ground truth.The datasets range from LFSD (100 images) to NJUD (1,985 images).Source: §4.1 · p. 10
4.2 Implementation Details
- Training follows the splits described in, uses ADAM with a fixed learning rate of 1e−4, and averages 0.037s per 224×224 RGB-D image on an Nvidia Tesla V100 GPU.The batch size is 4, and the final saliency-map loss weight λ1 is 1.2 while other weights are 1.Source: §4.2 · p. 10
4.3 Comparisons with State-of-the-art Methods
- Visual comparisons report more complete objects and sharper boundaries, including cases with unsatisfactory depth maps, complex backgrounds, and small salient objects.Examples also report better conservation of plant-leaf details and suppression of distracting background objects.Source: §4.3 · pp. 10–12
- The method achieves the best quantitative results across all metrics on the six testing datasets, with gains over the second-best method reported for NJUD-Test and DUT-test.On NJUD-Test, gains are 1.6% for F-measure, 6.2% for MAE score, and 1.7% for S-measure; on DUT-test, they are 2.0% for F-measure, 14.1% for MAE score, and 2.5% for S-measure.Source: Table 1 · p. 12, §4.3 · pp. 12–13
4.4 Ablation Studies
- Ablations support the three modules: cmFM improves on simple addition or concatenation, AFS benefits from channel attention and spatial fusion, and the full sg-PEA model beats its removal baselines.Removing AFS performs worse than removing GFF or CACA individually; removing position or edge attention yields different completeness and boundary trade-offs.Source: §4.4 · pp. 13–14, Table 2 · p. 14
5 Conclusion
The conclusion presents a network combining cross-modality feature modulation and adaptive feature selection. Experiments report state-of-the-art performance on six benchmarks.
5 Conclusion
- The network combines cross-modality feature modulation to integrate multimodal information with adaptive selection of saliency-related features, and achieves state-of-the-art performance on six benchmarks.The conclusion also states that adaptive selection covers multimodality spatial and channel features.Source: §5 · p. 14