PaperlayerView exact revision on arXiv

Source-linked AI summary

Specificity-preserving RGB-D Saliency Detection

Tao Zhou, Deng-Ping Fan, Geng Chen, Yi Zhou, Huazhu Fu

arXiv:2108.08162v2cs.CV

TL;DR

RGB-D SOD requires fusing RGB and depth while retaining modality-specific information, which existing shared-representation approaches often do not explicitly preserve. SP-Net combines modality-specific and shared networks with CIM and MFA, and experiments report effectiveness across six RGB-D SOD and three COD benchmarks, despite failures in complex backgrounds and with low-quality depth.

  • Problem

    Existing RGB-D SOD models mainly learn shared representations, while few explicitly preserve modality-specific characteristics during RGB-depth fusion.

  • Method

    SP-Net uses modality-specific and shared networks, with CIM for cross-modal and cross-level integration and MFA for incorporating modality-specific decoder features.

  • Results

    SP-Net outperforms cutting-edge approaches on six RGB-D SOD and three COD benchmarks, with additional attribute-based evaluation of challenging factors.

  • Takeaways & Limitations

    The framework explores shared cross-modal information while compensating modality-specific characteristics, and its components are validated through ablation and attribute-based evaluations.

  • Takeaways & Limitations

    SP-Net can fail in complex backgrounds and with low-quality depth, producing rough detections or missing fine object details.

Abstract

from arXiv · show

Salient object detection (SOD) on RGB and depth images has attracted more and more research interests, due to its effectiveness and the fact that depth cues can now be conveniently captured. Existing RGB-D SOD models usually adopt different fusion strategies to learn a shared representation from the two modalities (\ie, RGB and depth), while few methods explicitly consider how to preserve modality-specific characteristics. In this study, we propose a novel framework, termed SPNet} (Specificity-preserving network), which benefits SOD performance by exploring both the shared information and modality-specific properties (\eg, specificity). Specifically, we propose to adopt two modality-specific networks and a shared learning network to generate individual and shared saliency prediction maps, respectively. To effectively fuse cross-modal features in the shared learning network, we propose a cross-enhanced integration module (CIM) and then propagate the fused feature to the next layer for integrating cross-level information. Moreover, to capture rich complementary multi-modal information for boosting the SOD performance, we propose a multi-modal feature aggregation (MFA) module to integrate the modality-specific features from each individual decoder into the shared decoder. By using a skip connection, the hierarchical features between the encoder and decoder layers can be fully combined. Extensive experiments demonstrate that our~\ours~outperforms cutting-edge approaches on six popular RGB-D SOD and three camouflaged object detection benchmarks. The project is publicly available at: https://github.com/taozh2017/SPNet.

1. Introduction

RGB-D SOD must adaptively fuse RGB and depth while preserving their modality-specific characteristics. SP-Net addresses this with modality-specific and shared networks, CIM and MFA modules, and reports superiority across RGB-D SOD and COD benchmarks.

  • RGB-D SOD uses depth for geometric and spatial information, but adaptively fusing RGB and depth remains challenging.
  • Existing approaches commonly fuse modalities through independent streams, auxiliary depth subnetworks, or shared representations.
  • SP-Net uses two modality-specific networks and a shared learning network to preserve individual properties while exploring shared information.
  • CIM integrates cross-modal features into shared representations and propagates its output across layers to capture cross-level information.
  • MFA integrates modality-specific decoder features into the shared decoder to use complementary multi-modal information.
  • Extensive experiments cover six RGB-D SOD and three COD datasets, alongside attribute-based evaluation under challenging factors.

2. Related Work

Related work spans RGB SOD, RGB-D SOD, and multi-modal learning. Prior methods include handcrafted priors, feature-scale integration, and fusion strategies that may not fully model cross-modal correlations.

  • Prior SOD methods used handcrafted features and priors such as contrast, center or boundary, compactness, and background enclosure.
  • Deep SOD methods integrate multi-level or multi-scale features to address scale variation in salient objects.
  • Multi-modal learning methods seek shared representations, but direct feature concatenation may fail to exploit complex correlations across modalities.
  • RGB-D SOD frameworks include modality-specific streams followed by late fusion, while other approaches combine RGB and depth through auxiliary subnetworks.

3. Methodology

SP-Net uses separate modality-specific and shared learning pathways to preserve RGB/depth characteristics while learning shared representations. CIM fuses cross-modal features across levels, and MFA injects modality-specific decoder features into the shared decoder.

  • Overall Framework: SP-Net uses two modality-specific encoder subnetworks and a shared learning network to model individual and shared RGB-depth representations.The framework generates modality-specific and shared saliency predictions, with skip connections combining hierarchical encoder-decoder features.
  • Cross-enhanced Integration Module: CIM first uses sigmoid-generated attention maps from one modality to enhance the other, while residual connections preserve each modality’s original information.The normalized maps act as feature-level attention, and element-wise multiplication combines cross-modal enhancement with residual features.
  • Cross-enhanced Integration Module: CIM applies element-wise multiplication and maximization to cross-enhanced features, concatenates both results, and uses convolution to adaptively weight them.The fused output is combined with the preceding CIM output before producing the current shared representation.
  • Cross-enhanced Integration Module: CIM integrates cross-modal features at different layers and propagates each fused representation to the next layer for cross-level information.This design is intended to learn shared representations while exploiting correlations between RGB and depth features.
  • Multi-modal Feature Aggregation: MFA multiplies modality-specific decoder features with the current shared decoder representation and concatenates the resulting features.Its purpose is to make fuller use of modality-specific decoder features in the shared decoder.
  • Training Objective: The framework is trained end to end with separate modality-specific and shared decoder losses based on RGB, depth, shared predictions, and ground truth.The overall loss contains Lsp for modality-specific decoders and Lsh for the shared decoder.

4. Experimental Results and Analysis

The experimental section evaluates SP-Net through quantitative and qualitative comparisons, ablation studies, and attribute-based analysis under different challenges.

  • Experimental Analysis: Experiments include performance comparisons, qualitative evaluation, ablation studies, and attribute-based evaluation across different challenging factors.The analysis covers datasets, metrics, implementation details, component effectiveness, and robustness to challenge attributes.

4.1. Experimental Setup

The study evaluates RGB-D saliency detection on six public datasets using four established metrics and a fixed training-test protocol. The implementation uses a pretrained Res2Net-50 backbone and Adam optimization.

  • Datasets: Evaluation uses six RGB-D SOD datasets: NJU2K, NLPR, DES, SSD, STERE, and SIP.The protocol uses 2,195 training samples from NJU2K and NLPR, with remaining samples and the other datasets reserved for testing.
  • Evaluation Metrics: Precision-recall curves are computed by thresholding saliency maps from 0 to 255 and pairing the resulting precision and recall values.Precision is based on predicted-ground-truth overlap relative to the binary prediction, while recall uses overlap relative to ground truth.
  • Evaluation Metrics: Sα combines regional and object perception with α set to 0.5, while Eϕ captures image-level statistics and local pixel matching.The enhanced-alignment measure averages the enhanced-alignment matrix over all map pixels.
  • Evaluation Metrics: The model is assessed with Sα, max Eϕ, max Fβ, and mean absolute error M.The table caption specifies that higher is better for Sα, Eϕ, and Fβ, while lower is better for M.
  • Evaluation Metrics: Fβ combines precision and recall as a weighted harmonic mean, with β^2 set to 0.3 to emphasize precision.The reported experiments use the maximum Fβ obtained across thresholds from 0 to 255.
  • Evaluation Metrics: Mean absolute error M measures average pixel-level error between the normalized saliency prediction and ground truth, normalized to [0, 1].W and H denote the map width and height in the metric definition.
  • Implementation: Implementation uses PyTorch, a pretrained Res2Net-50 backbone, a one-channel depth encoder input, Adam optimization, and a learning rate reduced tenfold every 60 epochs.Training uses one NVIDIA Tesla V100 GPU with 32 GB memory and an initial learning rate of 1e-4.

4.2. Performance Comparison

SP-Net is evaluated against traditional and deep RGB-D saliency detectors across multiple datasets, metrics, visual examples, backbones, and efficiency measures. It generally achieves strong or superior detection performance, but its three-network design increases model size and inference time.

  • SP-Net outperforms eight traditional methods by a large margin on all six datasets.
  • SP-Net achieves the best performance on four evaluation metrics for NJU2K, DES, and SIP.
  • SP-Net performs better than most compared methods on ReDWeb-S, while remaining comparable with UCNet and JL-DCF.
  • Res2Net-50 gives SP-Net better performance than the tested backbone alternatives, while ResNet-50 still exceeds other compared methods.
  • Qualitative comparisons show reliable detection in small-object, complex-background, multiple-object, and low-light examples, with sharper boundaries in multiple-object scenes.
  • Because SP-Net uses two modality-specific networks and a shared learning network, it has relatively large model size and slower inference than compared methods.

4.3. Ablation Studies

Ablation studies evaluate the cross-enhanced integration module, multi-modal feature aggregation, modality-specific decoders, shared decoding, and the number of CIMs. The results support the contributions of the proposed fusion and specificity-preserving components.

  • The ablation section evaluates model components by removing or replacing them in the full model.
  • Effectiveness of CIM: CIM outperforms direct feature concatenation, and its cross-modal enhancement, adaptive fusion, and propagation components contribute to saliency performance.
  • Effectiveness of MFA: MFA outperforms both cross-modal feature enhancement fusion and simple concatenation by integrating modality-specific decoder features into the shared decoder.
  • Modality-specific decoders: Removing the two modality-specific decoders degrades performance, indicating that they provide supervision signals for learning modality-specific properties.
  • Shared decoder: The shared decoder outperforms the combination of the two modality-specific decoder outputs.
  • Number of CIMs: The full model uses five CIMs, while degraded variants use one or three CIMs for comparison.

4.4. Attribute-based Evaluation

Attribute-based evaluation examines RGB-D SOD under object-count, scene-setting, lighting, and object-scale conditions. Performance generally varies with these factors, while SP-Net remains competitive across the evaluated attributes.

  • Single vs. Multiple Objects: SP-Net outperforms other state-of-the-art methods for both single and multiple salient objects in a 1,229-image hybrid dataset.
  • Indoor vs. Outdoor: Many models struggle more indoors than outdoors, while SP-Net is among the methods performing well across both scene types.
  • Light Conditions: All models perform worse under low-light than sunny conditions, confirming that low light negatively impacts SOD performance.
  • Object Scale: Object scales are defined as small below 0.1, medium from 0.1 to 0.4, and large above 0.4 by salient-region-to-image-area ratio.
  • Object Scale: Compared methods perform better on small salient objects than large ones, while SP-Net and several recent models obtain promising scale-related performance.

4.5. Failure Cases and Discussion

The failure-case analysis identifies challenging conditions involving poor depth quality, similar object appearances, and fine object details. These cases define practical boundaries for SP-Net's current performance.

  • Poor-quality depth maps can leave SP-Net locating an object only roughly, without fine details.
  • Objects resembling other scene objects remain difficult to detect accurately.
  • Fine object structures may be omitted, with predictions capturing main regions but not detailed boundaries.
  • The authors identify depth enhancement or filtering and improved handling of fine structures as directions for future work.

4.6. Application for RGB-D Camouflaged Object Detection

SP-Net extends RGB-D salient object detection to camouflaged object detection, where target-background similarity makes localization difficult. Across three public benchmarks, it outperforms comparison methods and produces more accurate camouflaged-object boundaries, with depth cues contributing spatial information.

  • RGB-D camouflaged object detection is challenging because target objects can be highly similar to their surrounding backgrounds.
  • The extension is evaluated on CHAMELEON, CAMO, and COD10K public benchmark datasets.CHAMELEON contains 76 images, CAMO contains 1,250 images, and COD10K contains 5,066 images.
  • Table 7 compares COD methods using Sα and M, with larger or smaller values preferred according to the metric direction.
  • SP-Net performs better than the comparison COD methods on the three public datasets.
  • Depth-based methods, including SP-Net, outperform methods without depth cues, indicating that depth provides spatial information useful for COD.
  • Qualitative comparisons show that SP-Net detects more accurate boundaries of camouflaged objects than other COD models.

5. Conclusion

SP-Net addresses RGB-D salient object detection by combining shared cross-modal information with modality-specific characteristics. Its CIM and MFA modules support cross-level and complementary multimodal integration, and experiments validate the framework on RGB-D SOD and camouflaged object detection.

  • SP-Net combines shared cross-modal information with modality-specific characteristics rather than focusing only on shared representations.
  • CIM fuses cross-modal features and propagates its output across layers to explore cross-level information.
  • MFA integrates modality-specific decoder features to enhance complementary multimodal information.
  • Benchmark experiments report effectiveness against state-of-the-art RGB-D SOD methods and validate key framework components.
  • An attribute-based evaluation examines RGB-D SOD performance under different challenging factors.
  • Extending SP-Net to RGB-D camouflaged object detection also yields validated effectiveness.
Loading 2108.08162v2…