Source-linked AI summary
JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection
Keren Fu, Deng-Ping Fan, Ge-Peng Ji, Qijun Zhao
TL;DR
RGB-D salient object detection needs effective use of complementary RGB and depth information, while prior separate-stream schemes can be constrained by limited data and elaborate training. JL-DCF uses shared Siamese joint learning and densely-cooperative fusion, and reports superior performance against state-of-the-art methods across six benchmark datasets.
Problem
Prior RGB-D SOD methods often treat RGB and depth independently, while limited high-quality depth data and elaborate training requirements constrain deep multimodal learning.
Method
JL-DCF jointly learns RGB and depth features with a shared Siamese network and discovers cross-modal complementarity through densely-cooperative fusion.
Results
JL-DCF shows superior performance against state-of-the-art methods on six benchmark datasets.
Takeaways & Limitations
The framework provides a flexible RGB-D saliency detector whose inner modules can be replaced for further improvement.
Abstract
from arXiv · showhide
This paper proposes a novel joint learning and densely-cooperative fusion (JL-DCF) architecture for RGB-D salient object detection. Existing models usually treat RGB and depth as independent information and design separate networks for feature extraction from each. Such schemes can easily be constrained by a limited amount of training data or over-reliance on an elaborately-designed training process. In contrast, our JL-DCF learns from both RGB and depth inputs through a Siamese network. To this end, we propose two effective components: joint learning (JL), and densely-cooperative fusion (DCF). The JL module provides robust saliency feature learning, while the latter is introduced for complementary feature discovery. Comprehensive experiments on four popular metrics show that the designed framework yields a robust RGB-D saliency detector with good generalization. As a result, JL-DCF significantly advances the top-1 D3Net model by an average of ~1.9% (S-measure) across six challenging datasets, showing that the proposed framework offers a potential solution for real-world applications and could provide more insight into the cross-modality complementarity task. The code will be available at https://github.com/kerenfu/JLDCF/.
1. Introduction
RGB-D salient object detection addresses a growing multimodal problem, but existing deep methods remain limited and often rely on separate feature extraction with demanding training requirements. JL-DCF instead jointly learns RGB and depth features through a shared Siamese network and combines them with densely-cooperative fusion.
- Motivation: RGB-D salient object detection is emerging as an attractive topic as depth sensors become more popular.Depth is treated as the additional modality in RGB-D inputs.
- Research gap: Only a few deep learning-based RGB-D SOD works had appeared, leaving substantial room for performance improvement.The paper contrasts this small body of work with more than one hundred RGB SOD papers published since 2015.
- Research gap: Early- and late-fusion strategies face feature-extraction or fusion challenges, while independent middle-fusion networks require elaborate training and abundant data.The paper notes that high-quality depth maps remain sparse, which can lead to sub-optimal deep-learning solutions.
- Proposed approach: JL-DCF jointly extracts hierarchical RGB and depth features through a Siamese network with a shared backbone.This differs from prior approaches that independently extract features from RGB and depth views.
- Proposed approach: JL-DCF combines joint learning with densely-cooperative fusion to provide robust saliency feature learning and complementary feature discovery.The framework is presented as a general RGB-D SOD architecture with two principal sub-modules.
- Reported outcome: ~2% average F-measure improvement across six challenging datasets advances the state of the art reported by the paper.The result is stated in the contribution summary across six datasets.
2. Related Work
RGB-D SOD research progressed from heuristic handcrafted features toward deep multimodal fusion, commonly organized as early-, late-, or middle-fusion. JL-DCF uses middle-fusion while replacing independent modality streams with a shared Siamese architecture and cross-modal cooperation.
- Traditional methods: Traditional RGB-D SOD methods extend handcrafted RGB saliency cues, but their reliance on heuristics limits generalizability in complex scenarios.Examples include contrast, center or boundary priors, compactness, and combinations of saliency measures.
- Deep methods: Recent deep RGB-D SOD approaches use fully convolutional networks and diverse cross-modal fusion designs.Reported examples include bottom-up/top-down fusion, fully connected fusion, and attention-aware architectures.
- Fusion strategies: RGB-D fusion methods are categorized as early-fusion, late-fusion, or middle-fusion according to where modality integration occurs.Middle-fusion handles feature extraction and subsequent fusion with relatively deep CNNs.
- Fusion strategies: Middle-fusion can learn high-level concepts from both modalities and mine complex integration rules.The strategy also supports individual deep supervision for RGB and depth.
- JL-DCF positioning: JL-DCF belongs to the middle-fusion category while using a Siamese architecture with shared network architecture and weights.The shared design enables cross-modal knowledge sharing and substantially reduces model parameters.
3. Methodology
JL-DCF combines Siamese joint learning with densely-cooperative, multi-scale fusion to extract and integrate RGB-D saliency features. Its decoder uses dense connections, feature aggregation, and jointly supervised predictions to produce the final saliency map.
- Joint Learning: The JL component jointly learns RGB and depth representations through a Siamese network with shared architecture and weights.This enables cross-modal knowledge sharing while reducing parameters relative to independent extraction streams.
- Joint Learning: RGB and depth inputs are processed as three-channel maps, with depth normalized to [0, 255] and converted through color mapping.Naive gray mapping replicates the single depth channel across three channels.
- Densely-cooperative Fusion: The DCF component performs multi-scale cross-modal fusion by combining RGB and depth batch features with element-wise addition and multiplication.Addition exploits feature complementarity, while multiplication emphasizes commonality; the fused output retains k channels.
- Densely-cooperative Fusion: Replacing cooperative fusion with channel concatenation can bias learning toward RGB information and cause performance degeneration.Concatenation produces 2k-channel features but performs feature selection rather than explicit fusion, according to the authors.
- Densely-cooperative Fusion: Fused features from CM1∼CM6 enter a densely connected decoder whose aggregation modules combine deeper multi-scale features through non-linear Inception-based processing.The FA module uses 1 × 1, 3 × 3, and 5 × 5 convolutions together with max-pooling, while preserving spatial sizes and using k input/output channels.
- Loss Function: The final FA1 output is converted into a saliency map and trained with final supervision alongside global guidance losses from coarse predictions.The total loss combines Lf with λ-weighted Lg, using cross-entropy for both losses.
4. Experiments
Experiments evaluate JL-DCF on six RGB-D benchmarks using four metrics, backbone and fusion ablations, modality comparisons, and learning-curve analyses. JL-DCF consistently outperforms competing methods, while joint learning, cooperative fusion, and combined modalities improve results under suitable depth quality.
- Datasets and evaluation: JL-DCF is evaluated on six public RGB-D datasets using four saliency metrics, with training performed on selected NLPR and NJU2K samples.The model trained on this split is also applied to the other datasets for fair comparison.
- Comparison with existing methods: JL-DCF shows notable gains over CPFP, D3Net, and DMRA across all four reported metrics, supporting consistent effectiveness and generalizability.The comparisons include 14 state-of-the-art methods spanning deep learning and traditional approaches.
- Ablation studies: The full ResNet+CM+RGB-D implementation is the reference configuration for ablations, compared against VGG, concatenation, RGB-only, depth-only, and separate-learning variants.CM denotes cooperative fusion modules, while RGB-D denotes using both modalities.
- Ablation studies: Replacing CM modules with concatenation degrades performance because learning can become biased toward RGB while ignoring depth; CM instead performs explicit cross-modal fusion.The ablation attributes the difference to concatenation enabling feature selection without sufficiently explicit fusion.
- Ablation studies: Combining RGB and depth improves performance on most datasets, but unreliable STERE depth maps can degrade results, with depth-only performance about 16%/20% lower than RGB on Sα/F max β.STERE depth maps are described as coarse and misaligned with object boundaries.
- Ablation studies: Separate learning is prone to training difficulty: lr = 10^-9 becomes trapped at high loss, whereas joint learning converges nicely; after 40 epochs, separate learning drops 1.1%/1.76% on Sα/F max β.The reported drop is relative to JL-DCF and follows slower convergence for the separate-learning model.
5. Conclusion
JL-DCF is a flexible RGB-D salient-object-detection framework combining joint learning with densely-cooperative fusion. It achieves superior performance on six benchmark datasets, supported by ablation studies.
- JL-DCF combines joint learning and densely-cooperative fusion for RGB-D salient object detection.
- The framework jointly learns salient-object localization from RGB and depth views using a shared network.
- Densely-cooperative fusion exploits complementary information across RGB and depth modalities.
- JL-DCF outperforms state-of-the-art methods on six benchmark datasets and is supported by comprehensive ablation studies.
- The framework is flexible and general, allowing its internal modules to be replaced by counterparts for further improvement.