PaperlayerView exact revision on arXiv

Source-linked AI summary

DPANet: Depth Potentiality-Aware Gated Attention Network for RGB-D Salient Object Detection

Zuyao Chen, Runmin Cong, Qianqian Xu, Qingming Huang

arXiv:2003.08608v4cs.CV

TL;DR

RGB-D salient object detection must exploit complementary RGB-depth information without being contaminated by unreliable depth. DPANet learns depth potentiality and uses gated multi-modality attention to control fusion, outperforming 16 state-of-the-art methods across 8 datasets in quantitative and qualitative evaluations.

  • Problem

    RGB-D salient object detection lacks a holistic solution that both integrates cross-modal complementarity and accounts for unreliable depth-map quality.

  • Method

    DPANet learns saliency-oriented depth potentiality and combines it with gated multi-modality attention to regulate RGB-depth fusion.

  • Results

    DPANet outperforms 16 state-of-the-art methods on 8 RGB-D salient-object-detection datasets in quantitative and qualitative evaluations.

  • Takeaways & Limitations

    Depth-potentiality modeling and gated fusion jointly provide DPANet’s framework for reducing unreliable-depth contamination while aggregating RGB-depth complementarity.

Abstract

from arXiv · show

There are two main issues in RGB-D salient object detection: (1) how to effectively integrate the complementarity from the cross-modal RGB-D data; (2) how to prevent the contamination effect from the unreliable depth map. In fact, these two problems are linked and intertwined, but the previous methods tend to focus only on the first problem and ignore the consideration of depth map quality, which may yield the model fall into the sub-optimal state. In this paper, we address these two issues in a holistic model synergistically, and propose a novel network named DPANet to explicitly model the potentiality of the depth map and effectively integrate the cross-modal complementarity. By introducing the depth potentiality perception, the network can perceive the potentiality of depth information in a learning-based manner, and guide the fusion process of two modal data to prevent the contamination occurred. The gated multi-modality attention module in the fusion process exploits the attention mechanism with a gate controller to capture long-range dependencies from a cross-modal perspective. Experimental results compared with 15 state-of-the-art methods on 8 datasets demonstrate the validity of the proposed approach both quantitatively and qualitatively.

I. INTRODUCTION

DPANet jointly addresses cross-modal RGB-D fusion and unreliable-depth contamination by modeling depth potentiality and regulating feature fusion. Its gated attention design and end-to-end formulation are evaluated across eight RGB-D datasets against state-of-the-art methods.

  • Motivation and proposed solution: DPANet models depth-map potentiality while optimizing RGB-depth fusion through a gated attention mechanism.The approach adaptively fuses modalities rather than integrating them indiscriminately.
  • Depth potentiality perception: Depth potentiality perception guides cross-modal aggregation to reduce contamination from unreliable depth without requiring depth-quality labels.It uses saliency-oriented supervision based on the relationship between thresholded depth and the saliency mask.
  • Gated multi-modality attention: The gated multi-modality attention module uses spatial attention to reduce redundancy and a gate controller to regulate cross-modal fusion.This design targets complementary information while controlling how much depth information enters the fusion process.
  • Experimental validation: The network outperforms 16 state-of-the-art methods on 8 RGB-D SOD datasets without HHA or CRF pre- or post-processing.The comparison includes both quantitative and qualitative evaluations.

II. RELATED WORK

Prior RGB-D salient-object detectors progressively incorporated multimodal cues, but generally fused RGB and depth indiscriminately despite unreliable depth. DPANet instead estimates depth potentiality from saliency-oriented cues and uses that estimate to guide fusion.

  • RGB-D salient object detection: Deep learning has become mainstream in RGB-D salient object detection after earlier unsupervised methods used handcrafted depth, color, and spatial cues.Prior approaches progressively introduced hierarchical saliency fusion, master-subnetworks, and cross-modal integration.
  • Network overview: The proposed network is a symmetrical two-stream encoder-decoder whose learned depth-potentiality score controls gated fusion across enhanced RGB and depth features.Its decoders progressively integrate multiscale features before producing the saliency map.
  • Limitations of prior fusion: Previous RGB-D methods generally integrate RGB and depth features indiscriminately, leaving unreliable-depth contamination insufficiently addressed.A prior depth depurator switched RGB and RGB-D paths mechanically and unsupervisedly, whereas DPANet explicitly models depth confidence.
  • Depth potentiality perception: DPANet uses a saliency-oriented task to learn depth potentiality because depth-quality labels are unavailable.Otsu thresholding produces a binary depth map whose relationship with the ground-truth saliency mask provides the learning signal.
  • Depth potentiality perception: The depth potentiality metric combines IoU with coverage so salient-object coverage remains represented when thresholded depth includes background.The coverage term reflects the intersection area relative to the ground truth, while γ is set to 0.3 to emphasize coverage.

C. Gated Multi-modality Attention Module

DPANet’s GMA module addresses cross-modal complementarity and inconsistency by combining spatial attention, symmetrical cross-modal attention, and a learned depth-reliability gate. The gate controls how strongly RGB and depth features interact during fusion.

  • Gated Multi-modality Attention Module: The gate controller prevents contamination from unreliable depth maps while preserving useful cross-modal information.The design responds to the complementarity and inconsistency of RGB-D data rather than directly integrating both modalities.
  • Gated Multi-modality Attention Module: GMA applies spatial attention to RGB and depth features to reduce single-modal redundancy and emphasize salient-region responses.The modified features are unified into 256 dimensions, with separate weights for the two branches.
  • Gated Multi-modality Attention Module: Symmetrical attention sub-modules capture long-range dependencies by using one modality to generate spatial weights for refining the other.Adr uses depth information to generate spatial weights for RGB features, while Ard performs the symmetric operation for depth features.
  • Gated Multi-modality Attention Module: The interaction weights satisfy g1 + g2 = 1, with g1 set to the predicted depth potentiality score ĝ.When ĝ approaches 1, more depth information is introduced into the RGB branch; when ĝ approaches 0, the RGB branch dominates.

D. Multi-level Feature Fusion

DPANet fuses features across scales and modalities to combine complementary information while suppressing background noise and unreliable depth effects. Its multi-modality fusion uses learned channel and gate weights.

  • Multi-scale Feature Fusion: Multi-scale progressive fusion integrates single-modal features from coarse to fine because different scales provide complementary information.The strategy is motivated by the distinct information carried by features at different scales.
  • Multi-scale Feature Fusion: Multiplication strengthens salient-object responses and suppresses background noise when combining higher- and lower-level features.This differs from concatenation or summation commonly used for low-level and high-level feature fusion.
  • Multi-modality Feature Fusion: Weighted channel attention selects useful channels when fusing cross-modal features such as rd2 and dd2.The learned weight α balances complementary information from the RGB and depth modalities.
  • Multi-modality Feature Fusion: The fusion equations combine channel selection α and the depth-potentiality gate ĝ to account for modality complementarity and inconsistency.The resulting response represents common information associated with salient objects.

E. Loss Function

DPANet is trained with classification and regression losses in an end-to-end objective. Classification supervises saliency prediction, while regression models the depth-potentiality response.

  • Loss Function: The classification loss constrains saliency prediction, and the regression loss models the depth potentiality response.The final objective combines both losses.
  • Classification Loss: Binary cross-entropy measures the relation between the predicted saliency map and ground truth.The formulation uses image height H, width W, ground truth G, and predicted saliency map S.
  • Classification Loss: Auxiliary classification losses are added at four decoder stages in addition to the dominant output loss.Each stage is projected to one channel, up-sampled to ground-truth size, and normalized with a sigmoid function.
  • Regression Loss: Smooth L1 loss supervises the depth-potentiality estimation using pseudo label g and network estimation ĝ.The regression target is the pseudo label generated for depth potentiality perception.
  • Final Loss: The final loss is a linear combination of classification and regression losses with λ set to 1 for the regression term.The entire training process is conducted end to end.

A. Datasets

DPANet is evaluated on eight public RGB-D salient-object-detection datasets with pixel-wise ground truth. The datasets vary in size, acquisition method, scene complexity, and image content.

  • Datasets: The evaluation uses eight public RGB-D SOD datasets with corresponding pixel-wise ground truth.This provides the benchmark setting for assessing the proposed method.
  • Datasets: NJUD contains 2,003 RGB-depth image pairs with diverse objects and complex scenarios, using depth estimated from stereo images.NLPR contains 1,000 Kinect-captured RGB-D images and includes multiple salient objects.
  • Datasets: STEREO797 contains 797 Internet-collected stereoscopic images with stereo-estimated depth maps, while LFSD contains 100 RGB-D images.These datasets represent different sources and scales of RGB-D data.
  • Datasets: The dataset collection also includes scenes with multiple or transparent objects and 929 high-resolution person RGB-D images captured by Huawei Meta10.The multiple-or-transparent-object dataset uses 800 training and 400 testing images.

B. Evaluation Metrics

The evaluation uses PR curves, F-measure, MAE, and S-measure to assess saliency detection quality from complementary precision-recall, pixel-error, and structural perspectives.

  • Precision-recall evaluation: PR curves evaluate precision and recall across thresholds by comparing thresholded saliency maps with ground truth.Each threshold produces a precision-recall pair.
  • F-measure: F-measure combines precision and recall into a single score, with β^2 set to 0.3 to emphasize precision.
  • Mean Absolute Error: MAE measures the average pixel-wise absolute difference between the saliency map and ground truth.S denotes the saliency map, G the ground truth, and H and W its height and width.
  • S-measure: S-measure evaluates structural similarity between the saliency map and ground truth.It balances object-aware and region-aware structural similarity with α = 0.5.

C. Implementation Details

The evaluation compares DPANet with 16 state-of-the-art methods across eight RGB-D datasets using quantitative, qualitative, runtime, and ablation analyses. Results report stronger overall measurements, visual quality, robustness to difficult depth inputs, and efficiency relative to most compared RGB-D methods.

  • Qualitative evaluation: Qualitatively, DPANet produces precise locations, clean backgrounds, complete structures, and sharp boundaries in challenging scenes.The examples include low contrast, complex backgrounds, background disturbance, and multiple salient objects.
  • Qualitative evaluation: DPANet yields more complete structures and sharper boundaries than comparison methods in examples where salient-object parts are missed or boundaries are blurred.Reported cases include incomplete horns and bodies, missed dog legs, and retained background regions.
  • Qualitative evaluation: DPANet handles low-contrast and complex scenes by detecting salient people, suppressing distracting poles, and preserving clear object structures.
  • Robustness: DPANet remains robust to similar object-background appearances and inaccurate or blurred depth information through cross-modal complementarity and contamination prevention.The paper attributes this robustness in challenging depth scenarios to the GMA module.
  • Ablation study: The ablation study verifies the effectiveness of the GMA module, regression loss, and multi-level feature fusion on NJUD-test, SIP, and STEREO797.
  • Quantitative evaluation: Quantitatively, DPANet achieves higher precision and recall on all eight datasets and outperforms the compared methods on every reported measurement except DUT MAE.
  • Runtime comparison: Runtime comparisons show that DPANet is faster than most compared deep-learning RGB-D salient object detection methods.

E. Ablation Study

The ablation studies evaluate DPANet’s fusion components, depth-potentiality learning, and the contribution of depth information. Visualizations and dataset-level analyses indicate that gated cross-modal fusion can exploit reliable depth while limiting contamination from unreliable depth.

  • Module Verification: The ablation study evaluates key components on NJUD-test, SIP, and STEREO797, including GMA, regression loss, multi-scale fusion, and multi-modality fusion.The baseline removes or replaces these components for comparison.
  • GMA Module: GMA combines spatial attention with adaptive gating to extract discriminative cross-modal features while reducing redundancy.Its gate controller regulates how RGB and depth features contribute to each branch.
  • GMA Module: When depth is reliable, depth features complement RGB features affected by similar-background interference and suppress background noise.The enhanced features are combined with RGB features through the gate weight g1.
  • GMA Module: When depth is unreliable, the gate controller largely prevents imperfect depth information from affecting the RGB branch while RGB information refines the depth branch.The authors summarize this as learning cross-modal complementarity while preventing contamination.
  • Depth-related Information Verification: The soft depth-potentiality learning manner outperforms the hard classification manner overall and is more robust because it is not affected by threshold selection.The soft setting uses regression supervision, whereas the hard setting binarizes the pseudo-label at 0.5.
  • Depth-related Information Verification: On NJUD-test, adding depth improves F-measure from 0.908 to 0.930 and MAE from 0.043 to 0.035, corresponding to gains of 2.4% and 18.6%.These comparisons are against the RGB-only ‘w/o depth’ setting.
  • Depth-related Information Verification: On NLPR-test, the dataset with the lowest DP score of 0.481, DPANet achieves a minimum F-measure percentage gain of 1.7%.Across eight datasets, the authors report that DPANet handles both poor and good depth perception potential.

F. Failure Cases

The reported failure cases involve small distant objects, conflicts between depth emphasis and saliency, and complex backgrounds with inaccurate depth. These conditions can produce missed salient objects or false alarms.

  • Failure Cases: Long-distance scenes with multiple small salient objects are difficult because the depth map may not provide effective information for distant targets.Large size differences among salient targets further hinder complete detection.
  • Failure Cases: When depth highlights a nearby but nonsalient object, algorithms may struggle to suppress that interference, producing false alarms.The example involves a close-range person emphasized in the depth map rather than the final salient object.
  • Failure Cases: Complex, cluttered backgrounds remain difficult when the depth map fails to provide accurate information for separating salient objects from background regions.The authors identify this as a failure condition for both RGB and RGB-D approaches.

V. CONCLUSION

The conclusion presents DPANet as an RGB-D salient object detection framework that estimates depth potentiality, regulates cross-modal fusion, and generates saliency maps through multi-stage feature aggregation.

  • Conclusion: DPANet evaluates depth-map potentiality with a saliency-oriented perception module and weakens contamination from unreliable depth information.The module is integrated into the RGB-D processing framework.
  • Conclusion: The GMA module aggregates RGB-depth complementarity by highlighting saliency responses and regulating the fusion rate of cross-modal information.Multi-stage and multi-modality feature fusion then produces discriminative RGB-D features and the saliency map.
  • Conclusion: Experiments on eight RGB-D datasets report that DPANet outperforms 16 state-of-the-art methods under different evaluation metrics.The conclusion states this as the overall experimental finding.
Loading 2003.08608v4…