Source-linked AI summary
Cross-Modal Weighting Network for RGB-D Salient Object Detection
Gongyang Li, Zhi Liu, Linwei Ye, Yang Wang, Haibin Ling
TL;DR
RGB-D SOD needs to exploit complementary geometric information from depth while improving interactions between RGB and depth features. CMWNet uses three level-specific cross-modal weighting modules and a three-level decoder to enhance details and localization. It achieves the best performance against 15 state-of-the-art methods on seven public RGB-D SOD benchmarks.
Problem
RGB-D SOD requires effective exploitation of complementary information between RGB images and depth maps across feature levels and scales.
Method
CMWNet uses CMW-L, CMW-M, and CMW-H with Depth-to-RGB and RGB-to-RGB Weighting to model cross-modal and cross-scale interactions in a three-level encoder-decoder.
Results
CMWNet achieves the best performance on seven public RGB-D SOD benchmarks compared with 15 state-of-the-art methods.
Takeaways & Limitations
The proposed network jointly enhances salient-object details and localization by treating low-, middle-, and high-level features differently.
Abstract
from arXiv · showhide
Depth maps contain geometric clues for assisting Salient Object Detection (SOD). In this paper, we propose a novel Cross-Modal Weighting (CMW) strategy to encourage comprehensive interactions between RGB and depth channels for RGB-D SOD. Specifically, three RGB-depth interaction modules, named CMW-L, CMW-M and CMW-H, are developed to deal with respectively low-, middle- and high-level cross-modal information fusion. These modules use Depth-to-RGB Weighing (DW) and RGB-to-RGB Weighting (RW) to allow rich cross-modal and cross-scale interactions among feature layers generated by different network blocks. To effectively train the proposed Cross-Modal Weighting Network (CMWNet), we design a composite loss function that summarizes the errors between intermediate predictions and ground truth over different scales. With all these novel components working together, CMWNet effectively fuses information from RGB and depth channels, and meanwhile explores object localization and details across scales. Thorough evaluations demonstrate CMWNet consistently outperforms 15 state-of-the-art RGB-D SOD methods on seven popular benchmarks.
1 Introduction
CMWNet addresses limited RGB-D feature interaction by modeling cross-modal and cross-scale relationships across three representation levels. Its specialized modules enhance details and localization, and experiments report best performance against 15 state-of-the-art methods on seven benchmarks.
- RGB-D SOD uses depth information alongside RGB images to address challenges that can affect RGB-only saliency detection.Depth sensors have motivated extensive RGB-D SOD research, while earlier handcrafted methods could produce blocky or confused salient objects in complex scenes.
- CMWNet jointly explores RGB and depth features through cross-modal and cross-scale interactions in a three-level encoder-decoder structure.The design targets microscopic details from shallow blocks and macroscopic object localization from deep blocks.
- CMW-L and CMW-M enhance salient-object details, whereas CMW-H enhances salient-object localization across low-, middle-, and high-level features.The modules are specialized to different feature properties rather than applying one identical operation at every network block.
- Depth-to-RGB Weighting and RGB-to-RGB Weighting strengthen RGB features using corresponding response maps, while a three-level decoder predicts final salient objects.These operations provide the core weighting mechanism for the proposed cross-modal interactions.
- CMWNet achieves the best performance against 15 state-of-the-art RGB-D SOD methods across seven public datasets and six evaluation metrics.The reported evaluation covers extensive comparisons across the benchmark suite.
2 Related Work
Earlier RGB-D SOD methods used contrast, fusion, or measurement strategies, while CNN approaches introduced end-to-end and two-stream processing. CMWNet instead treats detail and localization features differently and models interactions across modalities and scales.
- Traditional RGB-D SOD methods commonly used disparity, depth, or multi-contextual contrast, along with multi-scale or multi-cue fusion.These approaches combined handcrafted cues or saliency maps to exploit RGB-D information.
- Some CNN-based methods fuse only saliency maps and high-level features, limiting cross-modal interaction at other network levels.This leaves lower-level and intermediate feature relationships less directly modeled.
- Other methods process cross-modal CNN features with the same module across blocks, overlooking the differing properties of features at different depths.The related-work discussion identifies this as a limitation of uniform processing across network blocks.
- CMWNet uses separate treatments for detail and localization features and performs cross-modal, cross-scale processing to enhance both before three-level decoding.The approach is designed to capture interactions across modalities and scales while distinguishing feature roles.
3 Proposed Method
CMWNet uses a three-level Siamese encoder-decoder to strengthen RGB-depth interactions across modalities and scales. CMW-L and CMW-M enhance details, while CMW-H enhances high-level salient-object localization before decoder aggregation.
- Network Overview: CMWNet uses shared-weight RGB and depth encoders divided into low-, middle-, and high-level feature parts.The decoder contains corresponding levels and uses skip-connections to fuse enhanced encoder features.
- Low- and Middle-level Weighting: CMW-L and CMW-M enhance object details by applying cross-scale Depth-to-RGB Weighting between adjacent encoder blocks.Higher-depth responses modulate lower RGB features, while lower-depth responses modulate higher RGB features.
- Low- and Middle-level Weighting: RGB-to-RGB Weighting adaptively enhances RGB features using response maps generated from RGB features, complementing depth-based weighting.The two weighting operations are applied to low- and middle-level features and combined with the original RGB features.
- High-level Weighting: CMW-H uses the highest RGB and depth encoder blocks to enhance macroscopic salient-object localization.This module is distinct from the detail-oriented low- and middle-level modules.
- Decoder and Training: The decoder progressively fuses enhanced features across three levels, with deep-scale supervision applied behind each level during training.The architecture connects high-, middle-, and low-level enhanced features for final salient-object inference.
- Weighting Mechanism: The weighting operations modulate feature maps through element-wise multiplication and summation, strengthening responses for salient regions.The formulation interprets the two response-map products as Depth-to-RGB and RGB-to-RGB weighting.
4 Experiments
CMWNet is evaluated on seven public RGB-D SOD benchmarks using six metrics and comparisons with 15 state-of-the-art methods. Results show consistent superiority across metrics, datasets, challenging visual cases, and ablations of its weighting and supervision components.
- Evaluation Setup: Seven public datasets and six widely used metrics are used to evaluate CMWNet against 15 state-of-the-art RGB-D SOD methods.The datasets are STEREO, NJU2K, LFSD, DES, NLPR, SSD, and SIP; metrics include Fβ, Fwβ, MAE, PR, Sλ, and Eξ.
- Quantitative Comparison: 1.6% and 2.0% improvements on STEREO are reported for Sλ and Fβ, respectively.On DES, the increases are 3.0% for Sλ and 4.2% for Fβ; SIP improves by 1.2% in Fβ.
- Quantitative Comparison: CMWNet consistently outperforms all compared methods across different evaluation metrics, with superiority especially visible on STEREO, LFSD, and DES.The PR curves, F-measure curves, and Sλ-Fwβ coordinates agree with the quantitative table results.
- Visual Comparison: CMWNet accurately highlights salient objects with fine details across low contrast, disturbing backgrounds, salient persons, fine structures, multiple objects, and small objects.In a fine-structure example, DW and RW preserve a mask with three holes while other methods fail.
- Ablation Studies: Removing depth input or CMW modules degrades performance, supporting the contributions of depth-based enhancement, detail enhancement, and localization enhancement.Examples include Sλ decreasing from 0.903 to 0.891 on NJU2K without CMW-L&M and Fβ decreasing from 0.902 to 0.894 without CMW-H.
- Ablation Studies: Cross-scale weighting between adjacent CNN blocks and deep scale supervision outperform nonadjacent weighting and single final-prediction supervision.The reported analyses attribute the adjacent-block advantage to feature continuity and show that coarse-to-fine prediction refinement benefits from deep scale supervision.
5 Conclusion
CMWNet combines three cross-modal, cross-scale weighting modules with progressive decoding for RGB-D SOD. It achieves the best performance on seven public benchmarks against 15 state-of-the-art methods.
- CMWNet uses CMW-L, CMW-M and CMW-H modules to encourage cross-modal and cross-scale feature interactions.
- A three-level decoder progressively refines salient objects after cross-modal weighting.
- CMWNet achieves the best performance on seven public RGB-D SOD benchmarks compared with 15 state-of-the-art methods.