PaperlayerView exact revision on arXiv

Source-linked AI summary

CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection

Runmin Cong, Qinwei Lin, Chen Zhang, Chongyi Li, Xiaochun Cao, Qingming Huang, Yao Zhao

arXiv:2210.02843v1cs.CV

TL;DR

RGB-D SOD requires effective use of complementary RGB and depth information despite their differing modality attributes. CIR-Net combines cross-modality interaction in both encoder and decoder stages with self- and cross-modality refinement, achieving competitive performance against state-of-the-art methods on six datasets without HHA or CRF processing.

  • Problem

    Effectively utilizing and integrating RGB information and depth cues remains an open issue because the modalities have different attributes.

  • Method

    CIR-Net uses a three-stream-like architecture with PAI encoder integration, IGF-based decoder aggregation, and middleware applying smAR followed by cmWR refinement.

  • Results

    CIR-Net achieves competitive performance against state-of-the-art methods on six RGB-D SOD datasets without HHA preprocessing or CRF post-processing.

  • Takeaways & Limitations

    Joint interaction and refinement across RGB, depth, and RGB-D features supports saliency prediction with qualitative and quantitative competitiveness across six datasets.

Abstract

from arXiv · show

Focusing on the issue of how to effectively capture and utilize cross-modality information in RGB-D salient object detection (SOD) task, we present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. For the cross-modality interaction, 1) a progressive attention guided integration unit is proposed to sufficiently integrate RGB-D feature representations in the encoder stage, and 2) a convergence aggregation structure is proposed, which flows the RGB and depth decoding features into the corresponding RGB-D decoding streams via an importance gated fusion unit in the decoder stage. For the cross-modality refinement, we insert a refinement middleware structure between the encoder and the decoder, in which the RGB, depth, and RGB-D encoder features are further refined by successively using a self-modality attention refinement unit and a cross-modality weighting refinement unit. At last, with the gradually refined features, we predict the saliency map in the decoder stage. Extensive experiments on six popular RGB-D SOD benchmarks demonstrate that our network outperforms the state-of-the-art saliency detectors both qualitatively and quantitatively.

I. INTRODUCTION

RGB-D SOD combines RGB appearance details with depth cues, but effectively integrating their differing modalities remains open. CIR-Net addresses this through cross-modality interaction in encoder and decoder stages, refinement middleware, and a three-stream-like architecture.

  • Motivation: RGB images provide appearance details, while depth maps add complementary shape, surface-normal, and internal-consistency cues for RGB-D SOD.Depth information can help suppress background interference in complex scenes.
  • Cross-modality interaction: The progressive attention guided integration unit fuses cross-modality and cross-level encoder features, while importance gated fusion supports decoder-stage interaction.The PAI unit progressively combines multi-level information and selects or highlights effective features.
  • Cross-modality refinement: A refinement middleware successively applies self-modality attention and cross-modality weighting to reduce redundancy and encode complementary contextual dependencies.The refinement units operate on RGB, depth, and RGB-D encoder features before decoding.
  • Results: Experiments on six RGB-D SOD datasets show competitive performance against state-of-the-art methods without HHA preprocessing or CRF post-processing.Visual comparisons report better object positioning, completeness preservation, and background suppression for the proposed method.
  • Related work: Existing single-stream models concatenate RGB and depth as four input channels, but ignore modality differences and lack comprehensive cross-modality interaction.Two-stream models separately process RGB and depth, interacting in either the encoder or decoder stage.
  • Architecture: CIR-Net uses a three-stream-like architecture that preserves independent RGB and depth information while forming RGB-D features through interaction with high-level branch features.The authors classify the architecture as three-stream while describing it as conceptually between two-stream and three-stream designs.

III. PROPOSED METHOD

CIR-Net uses an encoder-decoder three-stream architecture with RGB, depth, and RGB-D streams, connected by refinement middleware. Cross-modality features are generated in the encoder, refined between stages, and interact again during decoding.

  • CIR-Net is an encoder-decoder three-stream architecture with RGB, depth, and RGB-D branches plus refinement middleware between encoder and decoder.The RGB-D stream is formed from interactions between RGB and depth features.
  • Top-layer RGB, depth, and RGB-D encoder features are progressively refined by self-modality and cross-modality units before decoding.The refinement middleware contains smAR and cmWR units.
  • During decoding, RGB and depth features flow into the corresponding RGB-D stream through an IGF unit for cross-modality interaction.The decoder uses convergence aggregation to integrate corresponding features dynamically.
  • The backbone extracts multi-level RGB and depth features, while the PAI unit generates RGB-D encoder features from high-level features.RGB-D features are generated at encoder levels 3, 4, and 5.

B. Progressive Attention Guided Integration Unit

The PAI unit integrates RGB and depth features progressively across encoder levels while using spatial attention to emphasize complementary information and suppress redundancy.

  • The PAI unit addresses encoder integration through multi-level information fusion and feature selection and highlighting.It is designed around differences in scale, receptive field, and content across feature levels.
  • Spatial attention guides fusion by highlighting complementary information and suppressing irrelevant redundancy.
  • High-level RGB and depth features are combined from the third encoder layer because shallow depth features contain background noise and high-level features lack details.This produces initial cross-modality features while retaining complementary information.
  • Progressive cross-level fusion uses the preceding RGB-D spatial attention map to guide current-level feature integration.The final RGB-D features at the fourth and fifth levels are updated progressively.
  • A residual connection combines learned and original features to remain effective when attention is inaccurate in challenging cases.The residual design is included in the PAI update equations.

C. Refinement Middleware

The refinement middleware filters encoder features before decoding through successive self-modality and cross-modality refinement. Its smAR unit combines spatial and channel attention to reduce redundancy and emphasize useful information.

  • The middleware connects encoder and decoder stages to refine modality features before information is transmitted into decoding.It targets redundant self-modality information and indiscriminate transmission.
  • Self-modality Attention Refinement Unit: The smAR design differs from separate, serial, and parallel feature-fusion uses of spatial and channel attention described in prior RGB-D SOD work.The cited alternatives have limitations involving level suitability, operation order, or feature fusion.
  • Self-modality Attention Refinement Unit: smAR computes spatial and channel attention in parallel and fuses their attention maps by matrix multiplication into a 3D attention tensor.The tensor refines RGB, depth, and RGB-D top-layer features.
  • Self-modality Attention Refinement Unit: A residual connection is used to refine each modality feature after applying the 3D attention tensor.The paper reports ablation studies for different attention combinations and this design.

2) Cross-modality Weighting Refinement Unit:

The cmWR unit complements smAR by modeling relationships across RGB, depth, and RGB-D features. It computes cross-modality and RGB-D self-correlation, then uses global dependency weights to refine modality features.

  • cmWR addresses the RGB-depth complementarity that smAR does not fully use, including RGB color and texture and depth spatial relations.It is inspired by non-local modeling.
  • smAR outputs from RGB, depth, and RGB-D modalities are embedded into a unified feature space after bottleneck convolutions halve channel numbers.
  • cmWR calculates pixel-wise RGB-depth correlation and RGB-D self-correlation using scaled dot-product attention.M1 captures common RGB-depth responses, while M2 models RGB-D dependencies.
  • The separate M1 and M2 correlations are mapped into RGB-D feature space before generating the final similarity interaction.
  • Cross-modality global dependency weights generated from M1 and M2 refine the original modality features from a global perspective.The paper states that this can improve detection completeness and accuracy.

D. Importance Gated Fusion Unit

The IGF unit dynamically selects and combines RGB-D decoder information, extending cross-modality interaction into the decoder while producing features for saliency prediction.

  • Importance Gated Fusion Unit: IGF addresses the need to select valuable information from RGB and depth decoder streams rather than combining modalities equally.It learns an importance map to control each modality’s contribution dynamically.
  • Importance Gated Fusion Unit: RGB and depth decoder features are fused with skip-connection encoder features, then concatenated to form RGB-D decoder features.These features are combined with the previous IGF output for the current decoder level.
  • Importance Gated Fusion Unit: The importance map P_i is generated by concatenating current RGB-D decoder features with upsampled previous IGF features, reducing channels with 1×1 convolution, and applying channel attention with sigmoid activation.P_i measures the importance of RGB-D decoder features during fusion.
  • Importance Gated Fusion Unit: The last decoder features from the RGB, depth, and RGB-D streams separately predict saliency maps, while binary cross-entropy jointly optimizes all three streams.At testing time, only the RGB-D stream prediction is used as the final saliency map.

IV. EXPERIMENTS

The experiments evaluate CIR-Net on six RGB-D SOD benchmarks using precision-recall curves, F-measure, MAE, and S-measure, alongside visual comparisons of competing methods.

  • Experimental Setup: Experiments compare the proposed model across six RGB-D SOD benchmark datasets and against 15 state-of-the-art CNN-based methods.The datasets include STEREO797, NLPR, NJUD, DUT, LFSD, and SIP.
  • Evaluation Metrics: The evaluation uses precision-recall curves, F-measure, Mean Absolute Error, and S-measure to quantify saliency performance.Precision and recall are computed across thresholds from 0 to 255.
  • Qualitative Evaluation: Visual comparisons assess predictions for common scenes, unreliable depth maps, multiple objects, low contrast, and small objects.The figure lists RGB, depth, ground truth, CIR-Net, and competing-method outputs for these scene types.
  • Evaluation Metrics: F-measure combines precision and recall with β^2 set to 0.3, emphasizing precision.MAE measures average pixel-wise absolute prediction error, while S-measure evaluates structural similarity.

3) Implementation Details:

CIR-Net is trained and evaluated under specified dataset splits and input-processing settings, with comparisons showing strong results across datasets and challenging scene subsets.

  • Implementation Details: Training uses 1485 NJUD, 700 NLPR, and 800 DUT samples, while remaining samples and three additional datasets are reserved for testing.Training applies random flipping, rotation, and multi-scale resizing; inference uses 352 × 352 inputs.
  • Implementation Details: The reported results use ResNet as the backbone unless otherwise stated.This is the stated default backbone assumption for the paper’s experiments.
  • Overall Comparisons: The model achieves higher precision and recall than compared methods across all six benchmark datasets in the P-R curves.The visual comparison also reports accurate localization and complete salient-object structure.
  • Overall Comparisons: CIR-Net achieves the best performance except for MAE on SIP, with minimum gains of 3.0%, 15.3%, and 10.7% in MAE on NLPR-test, DUT-test, and STEREO797.These gains are reported relative to the second-best method.
  • Challenging Scenes: On unreliable-depth subsets, CIR-Net improves robustness when depth values resemble the background and interfere with salient-object detection.The reported comparison is against DANet, with percentage gains of 3.8% for MAE, F-measure, and S-measure.
  • Challenging Scenes: Compared with the second-best method, gains on multi-object scenes reach 1.6% for both F-measure and S-measure.For low-contrast scenes, reported gains are 25.9% in MAE, 3.0% in F-measure, and 3.7% in S-measure; for small objects, they are 18.7%, 5.5%, and 3.3%, respectively.

C. Ablation Study

Ablation studies progressively add CIR-Net’s modules and show improvements in feature completeness, boundary clarity, background suppression, and quantitative performance.

  • Ablation Design: The baseline replaces PAI with fifth-layer feature concatenation, removes the refinement middleware, and replaces IGF with simple deconvolutional layers.The ablations are evaluated on NJUD-test, STEREO797, and LFSD.
  • Ablation Design: Modules are added progressively as +PAI, +IGF, +cmWR, and +smAR under the same training configurations as CIR-Net.This design isolates the cumulative contribution of the proposed components.
  • Module Effects: Adding PAI increases F-measure from 0.8880 to 0.8952 on NJUD-test and from 0.8769 to 0.8853 on STEREO797.The qualitative results show more complete and consistent structural information, although background regions remain incorrectly detected.
  • Module Effects: Adding IGF raises NJUD-test F-measure to 0.9135 and improves it by 2.0% over the +PAI model, while producing clearer object boundaries.Adding cmWR further improves background suppression and object structure to a certain extent.
  • Module Effects: The full model gains 4.5% and 4.2% in F-measure over the baseline on NJUD-test and STEREO797, respectively.The final addition of smAR highlights important single-modality cues, and the ablations support the effectiveness of the proposed modules.

2) Analysis of the converged three-stream architecture:

The converged three-stream architecture improves RGB-D saliency by combining modality-specific strengths in an RGB-D branch and refining features through middleware attention units.

  • Three-stream architecture: CIR-Net outperforms the two-stream architecture because its three-stream structure enables more comprehensive feature interaction.The comparison removes the decoder RGB-D branch and fuses RGB and depth features by concatenation.
  • Three-stream architecture: RGB and depth branches have complementary regional strengths, while the final RGB-D branch suppresses their disadvantages and produces sharper, more complete saliency structures.On LFSD, F-measure rises from 0.8324 to 0.8828 and S-measure from 0.8339 to 0.8753 over the RGB branch.
  • Refinement middleware: The middleware ablation evaluates self-modality attention refinement and cross-modality weighting refinement on NJUD-TEST, STEREO797, and LFSD.The study compares alternative attention configurations and weighting maps.
  • Refinement middleware: The smAR unit's 3D attention tensor is more effective than single-channel, single-spatial, and serial SA-CA attention variants.The reported comparison is conducted on the STEREO797 dataset.
  • Refinement middleware: Using the combined M1 × M2 weight map in cmWR is more effective than using either M1 or M2 alone.On LFSD, F-measure increases from 0.8582 with only M2 to 0.8828 with M1 × M2.

4) Analysis of different feature interaction strategy in PAI and IGF units:

Ablations show that PAI and IGF design choices improve feature interaction, while PAI helps exploit RGB-depth correlations and IGF selectively controls depth contributions in difficult scenes.

  • PAI unit: Starting PAI combination from the third layer achieves the best performance among the tested propagation strategies.On LFSD, F-measure improves from 0.8555 to 0.8828 over first-layer forward propagation.
  • PAI unit: PAI outperforms the commonly used feature fusion strategy on LFSD.S-measure increases from 0.8480 with feature fusion to 0.8753 with PAI.
  • IGF unit: IGF improves performance over addition and concatenation fusion strategies.On LFSD, F-measure increases from 0.8605 with concatenation to 0.8828 with IGF, while S-measure also improves.
  • Residual features: Residual connections improve quantitative performance over direct multiplication alone.On LFSD, F-measure rises from 0.8500 without the addition in Eq. (2) to 0.8828 with the residual connection.
  • Challenging scenes: For unreliable depth maps, PAI exploits RGB-depth correlations and IGF selectively reduces the contribution of depth features.These mechanisms are described as suppressing interference from unreliable depth information.
  • Challenging scenes: In multiple-object scenes, cmWR uses long-range cross-modality dependencies to refine features globally and improve saliency completeness.With cmWR, F-measure increases from 0.8626 to 0.8715, a 1.0% gain.

D. Discussion

CIR-Net combines a three-stream-like architecture with cross-modality interaction and refinement, achieving competitive performance while retaining clear failure modes and future research boundaries.

  • Discussion: CIR-Net has failure cases involving multiple small salient objects, high-contrast nonsalient objects, and complex background noise.Unreliable or misleading depth information contributes to these failures in the reported scenes.
  • Future work: The paper does not specifically solve unreliable depth quality, but uses cmWR and IGF to reduce its negative impact.Stable and explicit detection under poor depth-map quality remains a stated future research problem.
  • Future work: Removing NLPR training data degrades performance by varying amounts across datasets.The F-measure drop is 0.5% on STEREO797 and 4.4% on LFSD.
  • Discussion: The model architecture forms RGB-D features by fusing high-level RGB and depth features through PAI rather than learning the RGB-D stream from scratch.This design is described as reducing computation while producing more discriminative RGB-D features.
  • Discussion: The decoder convergence structure centers interaction on RGB-D features to capture complementarity among RGB, depth, and RGB-D modalities.The stated goal is to obtain more discriminative and saliency-related features.
  • Discussion: PAI integrates cross-modality information in the encoder, IGF routes selected RGB and depth decoder information into the RGB-D stream, and middleware refines features before decoding.The refinement middleware operates from both self-modality and cross-modality perspectives.
  • Discussion: The method achieves competitive qualitative and quantitative performance on six datasets through cooperation between its architecture and technical modules.The conclusion attributes the result to their mutual cooperation and facilitation.
Loading 2210.02843v1…