Source-linked AI summary
Accurate RGB-D Salient Object Detection via Collaborative Learning
Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, Huchuan Lu
TL;DR
RGB-D salient object detectors face blurred boundaries, costly depth subnetworks, and dependence on depth inputs during testing. CoNet collaboratively learns edge, saliency, and depth information, integrating depth into high-level feature learning rather than using depth at inference. Across seven benchmark datasets, it reports superior performance while operating at 34 FPS and improving FPS by 55% over DMRA.
Problem
Existing RGB-D saliency methods can blur object boundaries, incur computation and storage costs through extra depth networks, and require depth inputs during testing.
Method
CoNet jointly uses edge detection, coarse salient object detection, and depth estimation, integrating depth and saliency learning into high-level feature learning with collaborative knowledge transfer.
Results
34 FPS, improving FPS by 55% over DMRA, while experiments on seven benchmark datasets show superior performance over 22 state-of-the-art RGB and RGB-D methods.
Takeaways & Limitations
The framework produces accurate saliency results with sharp boundaries while avoiding extra depth subnetworks and depth inputs during testing.
Takeaways & Limitations
The quality of Attdepth and Attf can cause irrecoverable inhibition of salient areas, motivating residual connections in the knowledge collector.
Abstract
from arXiv · showhide
Benefiting from the spatial cues embedded in depth images, recent progress on RGB-D saliency detection shows impressive ability on some challenge scenarios. However, there are still two limitations. One hand is that the pooling and upsampling operations in FCNs might cause blur object boundaries. On the other hand, using an additional depth-network to extract depth features might lead to high computation and storage cost. The reliance on depth inputs during testing also limits the practical applications of current RGB-D models. In this paper, we propose a novel collaborative learning framework where edge, depth and saliency are leveraged in a more efficient way, which solves those problems tactfully. The explicitly extracted edge information goes together with saliency to give more emphasis to the salient regions and object boundaries. Depth and saliency learning is innovatively integrated into the high-level feature learning process in a mutual-benefit manner. This strategy enables the network to be free of using extra depth networks and depth inputs to make inference. To this end, it makes our model more lightweight, faster and more versatile. Experiment results on seven benchmark datasets show its superior performance.
1 Introduction
RGB-D salient object detection improves localization in complex scenes through depth cues, but existing approaches can blur boundaries and require costly depth processing. CoNet collaboratively learns edge, saliency, and depth information while avoiding depth input at inference, achieving strong performance and real-time speed.
- Motivation: FCN pooling and upsampling can produce coarse, blurred salient-object boundaries despite strong high-level semantic localization.Low-level features preserve local details but may introduce background noise.
- Motivation: Previous RGB-D models either separately process RGB and depth with cross-modal fusion or use dedicated depth subnetworks to compensate for RGB features.These structures are illustrated as two common network designs.
- Method: CoNet uses edge, coarse saliency, and depth collaborators, combining edge knowledge with saliency and integrating depth learning into high-level feature learning.A knowledge collector transfers information from the individual collaborators to the group.
- Method: The framework avoids an extra depth network and depth input during testing, making the model more lightweight, faster, and versatile.Inference uses RGB input without relying on depth images.
- Results: 34 FPS, enhancing FPS by 55% compared with DMRA, supports the model’s suitability for real-time prediction.The reported comparison is against the current best-performing method DMRA.
2 Related Work
RGB-D saliency methods evolved from handcrafted and two-stream designs toward richer fusion and depth-specific subnetworks, but these approaches retain practical and boundary-quality limitations. CoNet instead integrates depth learning into feature learning and combines edge and saliency knowledge without depth input during testing.
- Prior RGB-D methods: RGB-based saliency methods became mainstream but degrade in complex scenarios where depth provides useful spatial information.Earlier approaches primarily relied on handcrafted features, while later deep models used RGB images.
- Prior RGB-D methods: Two-stream RGB-D models process RGB and depth separately before combining cross-modal features, with later variants adding progressive fusion or attention.These designs aim to select complementary information from the two modalities.
- Limitations: Dedicated depth networks increase computation and storage costs, while requiring depth images during testing limits practical applications.These constraints are identified as limitations of existing RGB-D models.
- Limitations: Existing methods can produce coarse, blurred saliency boundaries because FCN operations lose local details and boundary emphasis is insufficient.The limitation is linked specifically to pooling and upsampling operations.
- Proposed direction: CoNet integrates depth into high-level feature learning and combines learned edge knowledge with saliency knowledge, avoiding depth subnetworks and depth input at testing.The design is presented as a more efficient use of edge, depth, and saliency information.
3 Collaborative Learning Framework
CoNet combines edge detection, coarse saliency detection, and depth estimation as mutually beneficial collaborators. Low-level edge guidance sharpens boundaries, while high-level saliency and depth learning enhance semantic features without requiring depth input during inference.
- Collaborative Learning: CoNet uses three collaborators—edge detection, coarse salient object detection, and depth estimation—to support accurate RGB-D salient object detection.Their knowledge is transferred to a unified knowledge collector for final saliency prediction.
- Feature Preprocessing: The backbone and transition layers produce integrated low-level and high-level features for subsequent collaborative processing.Five side-output features are transformed by transition layers, while the global guidance module enriches high-level semantics.
- Feature Preprocessing: The global guidance module applies parallel dilated convolutions and a 1×1 convolution to high-level features, capturing multi-scale global semantics.It combines current-layer features with outputs from high-level global perception modules to reduce information dilution.
- Edge Collaborator: The edge collaborator extracts edge information from low-level features and uses it to emphasize salient-object boundaries.Edge supervision is generated from saliency ground truth with a Canny operator and trained using binary cross entropy.
- Saliency and Depth Collaborators: Saliency and depth collaborators jointly refine high-level features by learning coarse saliency attention and depth-aware channel attention.Depth is learned from depth ground truth through a depth head rather than supplied as an inference input, eliminating the need for an extra depth network during inference.
- Knowledge Collector: The knowledge collector fuses edge, saliency, depth, and multi-level features with triple attention before producing the final saliency map.Residual connections retain original features because attention quality could otherwise inhibit salient areas irrecoverably.
4 Experiments
Experiments evaluate CoNet on seven benchmark datasets, analyze its components and collaborator interactions, and compare accuracy, qualitative behavior, and complexity against prior methods.
- Evaluation Setup: CoNet is evaluated on seven benchmark datasets using PR curves, F-measure, MAE, weighted F-measure, S-measure, and E-measure.The datasets include DUT-D, NJUD, NLPR, SIP, LFSD, STEREO, and RGBD135.
- Ablation Analysis: Adding the Global Guidance Module extracts richer semantic features and prevents information dilution during decoding.This comparison uses the backbone network with and without GGM.
- Ablation Analysis: Mutual-benefit saliency-depth learning improves salient-object localization, and the knowledge collector further improves average MAE on two datasets by nearly 9.6% points.The knowledge collector combines edge, depth, and saliency knowledge to emphasize salient regions and object boundaries.
- Ablation Analysis: Edge supervision sharpens saliency boundaries, while saliency supervision on low-level features can substantially reduce performance because the tasks emphasize different structures.The paper attributes this decrease partly to coarse, information-heavy low-level features and partly to task incompatibility.
- Comparison with Prior Methods: Across seven benchmark datasets, CoNet achieves superior performance over 13 RGB-D methods and comparable results to 2D methods.PR curves and qualitative comparisons also support the reported superiority, including complex low-contrast and multiple-object scenes.
- Complexity Comparison: CoNet enhances FPS by 55% compared with the current best-performing RGB-D model DMRA, supporting its suitability in model size and running speed.The comparison is reported in the complexity evaluation.
5 Conclusion
CoNet uses collaborative edge, saliency, and depth learning to produce accurate RGB-D saliency maps with sharp boundaries. It avoids extra depth subnetworks and depth inputs during testing, and outperforms 22 state-of-the-art RGB and RGB-D methods on seven benchmark datasets.
- Conclusion: CoNet jointly uses edge detection, coarse salient-object detection, and depth estimation as mutually beneficial collaborators.The collaborators address the SOD task from different perspectives.
- Conclusion: The framework produces accurate saliency results while preserving sharp boundaries.This outcome is attributed to the collaborative learning strategy.
- Conclusion: CoNet avoids extra depth subnetworks and depth inputs during testing, making the network more lightweight and versatile.
- Conclusion: Experiments on seven benchmark datasets show superior performance over 22 state-of-the-art RGB and RGB-D methods.