PaperlayerView exact revision on arXiv

Source-linked AI summary

CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection

Runmin Cong, Qinwei Lin, Chen Zhang, Chongyi Li, Xiaochun Cao, Qingming Huang, Yao Zhao

arXiv:2210.02843v1cs.CV

TL;DR

RGB-D SOD requires effective use of complementary RGB and depth information despite their differing modality attributes. CIR-Net combines cross-modality interaction in both encoder and decoder stages with self- and cross-modality refinement, achieving competitive performance against state-of-the-art methods on six datasets without HHA or CRF processing.

  • Problem

    Effectively utilizing and integrating RGB information and depth cues remains an open issue because the modalities have different attributes.

    Source: §I · p. 2

  • Method

    CIR-Net uses a three-stream-like architecture with PAI encoder integration, IGF-based decoder aggregation, and middleware applying smAR followed by cmWR refinement.

    Source: §I · p. 2, B. Progressive Attention Guided… · p. 5, 2) Cross-modality Weighting Refinement… · p. 6

  • Results

    CIR-Net achieves competitive performance against state-of-the-art methods on six RGB-D SOD datasets without HHA preprocessing or CRF post-processing.

    Source: §I · p. 2, §V · p. 14

  • Takeaways & Limitations

    Joint interaction and refinement across RGB, depth, and RGB-D features supports saliency prediction with qualitative and quantitative competitiveness across six datasets.

    Source: §I · p. 2, §V · p. 14

Abstract

from arXiv · show

Focusing on the issue of how to effectively capture and utilize cross-modality information in RGB-D salient object detection (SOD) task, we present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. For the cross-modality interaction, 1) a progressive attention guided integration unit is proposed to sufficiently integrate RGB-D feature representations in the encoder stage, and 2) a convergence aggregation structure is proposed, which flows the RGB and depth decoding features into the corresponding RGB-D decoding streams via an importance gated fusion unit in the decoder stage. For the cross-modality refinement, we insert a refinement middleware structure between the encoder and the decoder, in which the RGB, depth, and RGB-D encoder features are further refined by successively using a self-modality attention refinement unit and a cross-modality weighting refinement unit. At last, with the gradually refined features, we predict the saliency map in the decoder stage. Extensive experiments on six popular RGB-D SOD benchmarks demonstrate that our network outperforms the state-of-the-art saliency detectors both qualitatively and quantitatively.

I. INTRODUCTION

RGB-D SOD combines RGB appearance details with depth cues, but effectively integrating their differing modalities remains open. CIR-Net addresses this through cross-modality interaction in encoder and decoder stages, refinement middleware, and a three-stream-like architecture.

Motivation

  • RGB images provide appearance details, while depth maps add complementary shape, surface-normal, and internal-consistency cues for RGB-D SOD.Depth information can help suppress background interference in complex scenes.Source: §I · p. 1

Cross-modality interaction

  • The progressive attention guided integration unit fuses cross-modality and cross-level encoder features, while importance gated fusion supports decoder-stage interaction.The PAI unit progressively combines multi-level information and selects or highlights effective features.Source: §I · p. 2

Cross-modality refinement

  • A refinement middleware successively applies self-modality attention and cross-modality weighting to reduce redundancy and encode complementary contextual dependencies.The refinement units operate on RGB, depth, and RGB-D encoder features before decoding.Source: §I · p. 2

Results

  • Experiments on six RGB-D SOD datasets show competitive performance against state-of-the-art methods without HHA preprocessing or CRF post-processing.Visual comparisons report better object positioning, completeness preservation, and background suppression for the proposed method.Source: §I · p. 2

Related work

  • Existing single-stream models concatenate RGB and depth as four input channels, but ignore modality differences and lack comprehensive cross-modality interaction.Two-stream models separately process RGB and depth, interacting in either the encoder or decoder stage.Source: §II · p. 3

Architecture

  • CIR-Net uses a three-stream-like architecture that preserves independent RGB and depth information while forming RGB-D features through interaction with high-level branch features.The authors classify the architecture as three-stream while describing it as conceptually between two-stream and three-stream designs.Source: §I · p. 2, §II · p. 3

III. PROPOSED METHOD

CIR-Net uses an encoder-decoder three-stream architecture with RGB, depth, and RGB-D streams, connected by refinement middleware. Cross-modality features are generated in the encoder, refined between stages, and interact again during decoding.

  • CIR-Net is an encoder-decoder three-stream architecture with RGB, depth, and RGB-D branches plus refinement middleware between encoder and decoder.The RGB-D stream is formed from interactions between RGB and depth features.Source: A. Overview · p. 3
  • Top-layer RGB, depth, and RGB-D encoder features are progressively refined by self-modality and cross-modality units before decoding.The refinement middleware contains smAR and cmWR units.Source: A. Overview · p. 4
  • During decoding, RGB and depth features flow into the corresponding RGB-D stream through an IGF unit for cross-modality interaction.The decoder uses convergence aggregation to integrate corresponding features dynamically.Source: A. Overview · p. 4
  • The backbone extracts multi-level RGB and depth features, while the PAI unit generates RGB-D encoder features from high-level features.RGB-D features are generated at encoder levels 3, 4, and 5.Source: A. Overview · p. 4

B. Progressive Attention Guided Integration Unit

The PAI unit integrates RGB and depth features progressively across encoder levels while using spatial attention to emphasize complementary information and suppress redundancy.

  • The PAI unit addresses encoder integration through multi-level information fusion and feature selection and highlighting.It is designed around differences in scale, receptive field, and content across feature levels.Source: B. Progressive Attention Guided… · p. 4
  • Spatial attention guides fusion by highlighting complementary information and suppressing irrelevant redundancy.Source: B. Progressive Attention Guided… · p. 5
  • High-level RGB and depth features are combined from the third encoder layer because shallow depth features contain background noise and high-level features lack details.This produces initial cross-modality features while retaining complementary information.Source: B. Progressive Attention Guided… · p. 5
  • Progressive cross-level fusion uses the preceding RGB-D spatial attention map to guide current-level feature integration.The final RGB-D features at the fourth and fifth levels are updated progressively.Source: B. Progressive Attention Guided… · p. 5
  • A residual connection combines learned and original features to remain effective when attention is inaccurate in challenging cases.The residual design is included in the PAI update equations.Source: B. Progressive Attention Guided… · p. 5

C. Refinement Middleware

The refinement middleware filters encoder features before decoding through successive self-modality and cross-modality refinement. Its smAR unit combines spatial and channel attention to reduce redundancy and emphasize useful information.

Self-modality Attention Refinement Unit

  • The smAR design differs from separate, serial, and parallel feature-fusion uses of spatial and channel attention described in prior RGB-D SOD work.The cited alternatives have limitations involving level suitability, operation order, or feature fusion.Source: 1) Self-modality Attention Refinement… · pp. 5–6
  • smAR computes spatial and channel attention in parallel and fuses their attention maps by matrix multiplication into a 3D attention tensor.The tensor refines RGB, depth, and RGB-D top-layer features.Source: 1) Self-modality Attention Refinement… · p. 6
  • A residual connection is used to refine each modality feature after applying the 3D attention tensor.The paper reports ablation studies for different attention combinations and this design.Source: 1) Self-modality Attention Refinement… · p. 6

2) Cross-modality Weighting Refinement Unit:

The cmWR unit complements smAR by modeling relationships across RGB, depth, and RGB-D features. It computes cross-modality and RGB-D self-correlation, then uses global dependency weights to refine modality features.

D. Importance Gated Fusion Unit

The IGF unit dynamically selects and combines RGB-D decoder information, extending cross-modality interaction into the decoder while producing features for saliency prediction.

Importance Gated Fusion Unit

  • IGF addresses the need to select valuable information from RGB and depth decoder streams rather than combining modalities equally.It learns an importance map to control each modality’s contribution dynamically.Source: D. Importance Gated Fusion Unit · p. 7
  • RGB and depth decoder features are fused with skip-connection encoder features, then concatenated to form RGB-D decoder features.These features are combined with the previous IGF output for the current decoder level.Source: D. Importance Gated Fusion Unit · p. 7
  • The importance map P_i is generated by concatenating current RGB-D decoder features with upsampled previous IGF features, reducing channels with 1×1 convolution, and applying channel attention with sigmoid activation.P_i measures the importance of RGB-D decoder features during fusion.Source: D. Importance Gated Fusion Unit · p. 7
  • The last decoder features from the RGB, depth, and RGB-D streams separately predict saliency maps, while binary cross-entropy jointly optimizes all three streams.At testing time, only the RGB-D stream prediction is used as the final saliency map.Source: E. Loss Function · p. 7

IV. EXPERIMENTS

The experiments evaluate CIR-Net on six RGB-D SOD benchmarks using precision-recall curves, F-measure, MAE, and S-measure, alongside visual comparisons of competing methods.

Experimental Setup

  • Experiments compare the proposed model across six RGB-D SOD benchmark datasets and against 15 state-of-the-art CNN-based methods.The datasets include STEREO797, NLPR, NJUD, DUT, LFSD, and SIP.Source: §IV · p. 7, 1) Benchmark Datasets · p. 7

Evaluation Metrics

  • The evaluation uses precision-recall curves, F-measure, Mean Absolute Error, and S-measure to quantify saliency performance.Precision and recall are computed across thresholds from 0 to 255.Source: 2) Evaluation Metrics · pp. 7–8

Qualitative Evaluation

  • Visual comparisons assess predictions for common scenes, unreliable depth maps, multiple objects, low contrast, and small objects.The figure lists RGB, depth, ground truth, CIR-Net, and competing-method outputs for these scene types.Source: Figure 7 · p. 8

Evaluation Metrics

  • F-measure combines precision and recall with β^2 set to 0.3, emphasizing precision.MAE measures average pixel-wise absolute prediction error, while S-measure evaluates structural similarity.Source: 2) Evaluation Metrics · p. 8

3) Implementation Details:

CIR-Net is trained and evaluated under specified dataset splits and input-processing settings, with comparisons showing strong results across datasets and challenging scene subsets.

Implementation Details

  • Training uses 1485 NJUD, 700 NLPR, and 800 DUT samples, while remaining samples and three additional datasets are reserved for testing.Training applies random flipping, rotation, and multi-scale resizing; inference uses 352 × 352 inputs.Source: 3) Implementation Details · p. 8
  • The reported results use ResNet as the backbone unless otherwise stated.This is the stated default backbone assumption for the paper’s experiments.Source: 3) Implementation Details · p. 8

Overall Comparisons

  • The model achieves higher precision and recall than compared methods across all six benchmark datasets in the P-R curves.The visual comparison also reports accurate localization and complete salient-object structure.Source: Figure 8 · p. 9, B. Comparison with the… · p. 9
  • CIR-Net achieves the best performance except for MAE on SIP, with minimum gains of 3.0%, 15.3%, and 10.7% in MAE on NLPR-test, DUT-test, and STEREO797.These gains are reported relative to the second-best method.Source: B. Comparison with the… · p. 10

Challenging Scenes

  • On unreliable-depth subsets, CIR-Net improves robustness when depth values resemble the background and interfere with salient-object detection.The reported comparison is against DANet, with percentage gains of 3.8% for MAE, F-measure, and S-measure.Source: B. Comparison with the… · p. 10
  • Compared with the second-best method, gains on multi-object scenes reach 1.6% for both F-measure and S-measure.For low-contrast scenes, reported gains are 25.9% in MAE, 3.0% in F-measure, and 3.7% in S-measure; for small objects, they are 18.7%, 5.5%, and 3.3%, respectively.Source: B. Comparison with the… · p. 10

C. Ablation Study

Ablation studies progressively add CIR-Net’s modules and show improvements in feature completeness, boundary clarity, background suppression, and quantitative performance.

Ablation Design

  • The baseline replaces PAI with fifth-layer feature concatenation, removes the refinement middleware, and replaces IGF with simple deconvolutional layers.The ablations are evaluated on NJUD-test, STEREO797, and LFSD.Source: 1) Analysis of Different Modules · pp. 10–11
  • Modules are added progressively as +PAI, +IGF, +cmWR, and +smAR under the same training configurations as CIR-Net.This design isolates the cumulative contribution of the proposed components.Source: 1) Analysis of Different Modules · p. 11

Module Effects

  • Adding PAI increases F-measure from 0.8880 to 0.8952 on NJUD-test and from 0.8769 to 0.8853 on STEREO797.The qualitative results show more complete and consistent structural information, although background regions remain incorrectly detected.Source: 1) Analysis of Different Modules · p. 11
  • Adding IGF raises NJUD-test F-measure to 0.9135 and improves it by 2.0% over the +PAI model, while producing clearer object boundaries.Adding cmWR further improves background suppression and object structure to a certain extent.Source: 1) Analysis of Different Modules · p. 11
  • The full model gains 4.5% and 4.2% in F-measure over the baseline on NJUD-test and STEREO797, respectively.The final addition of smAR highlights important single-modality cues, and the ablations support the effectiveness of the proposed modules.Source: 1) Analysis of Different Modules · p. 11

2) Analysis of the converged three-stream architecture:

The converged three-stream architecture improves RGB-D saliency by combining modality-specific strengths in an RGB-D branch and refining features through middleware attention units.

Three-stream architecture

  • CIR-Net outperforms the two-stream architecture because its three-stream structure enables more comprehensive feature interaction.The comparison removes the decoder RGB-D branch and fuses RGB and depth features by concatenation.Source: 2) Analysis of the converged… · p. 11
  • RGB and depth branches have complementary regional strengths, while the final RGB-D branch suppresses their disadvantages and produces sharper, more complete saliency structures.On LFSD, F-measure rises from 0.8324 to 0.8828 and S-measure from 0.8339 to 0.8753 over the RGB branch.Source: 2) Analysis of the converged… · p. 12

Refinement middleware

  • The middleware ablation evaluates self-modality attention refinement and cross-modality weighting refinement on NJUD-TEST, STEREO797, and LFSD.The study compares alternative attention configurations and weighting maps.Source: 3) Analysis of Refinement Middleware · p. 12, Table V · p. 12
  • The smAR unit's 3D attention tensor is more effective than single-channel, single-spatial, and serial SA-CA attention variants.The reported comparison is conducted on the STEREO797 dataset.Source: 3) Analysis of Refinement Middleware · p. 12
  • Using the combined M1 × M2 weight map in cmWR is more effective than using either M1 or M2 alone.On LFSD, F-measure increases from 0.8582 with only M2 to 0.8828 with M1 × M2.Source: 3) Analysis of Refinement Middleware · p. 12

4) Analysis of different feature interaction strategy in PAI and IGF units:

Ablations show that PAI and IGF design choices improve feature interaction, while PAI helps exploit RGB-depth correlations and IGF selectively controls depth contributions in difficult scenes.

PAI unit

  • Starting PAI combination from the third layer achieves the best performance among the tested propagation strategies.On LFSD, F-measure improves from 0.8555 to 0.8828 over first-layer forward propagation.Source: 4) Analysis of different feature… · p. 12
  • PAI outperforms the commonly used feature fusion strategy on LFSD.S-measure increases from 0.8480 with feature fusion to 0.8753 with PAI.Source: 4) Analysis of different feature… · p. 12

IGF unit

Residual features

  • Residual connections improve quantitative performance over direct multiplication alone.On LFSD, F-measure rises from 0.8500 without the addition in Eq. (2) to 0.8828 with the residual connection.Source: 5) Analysis of residual connection · p. 13

Challenging scenes

  • For unreliable depth maps, PAI exploits RGB-depth correlations and IGF selectively reduces the contribution of depth features.These mechanisms are described as suppressing interference from unreliable depth information.Source: 6) Analysis of effectiveness to… · p. 13
  • In multiple-object scenes, cmWR uses long-range cross-modality dependencies to refine features globally and improve saliency completeness.With cmWR, F-measure increases from 0.8626 to 0.8715, a 1.0% gain.Source: 6) Analysis of effectiveness to… · p. 13

D. Discussion

CIR-Net combines a three-stream-like architecture with cross-modality interaction and refinement, achieving competitive performance while retaining clear failure modes and future research boundaries.

Discussion

  • CIR-Net has failure cases involving multiple small salient objects, high-contrast nonsalient objects, and complex background noise.Unreliable or misleading depth information contributes to these failures in the reported scenes.Source: 1) Failure Cases · p. 13

Future work

  • The paper does not specifically solve unreliable depth quality, but uses cmWR and IGF to reduce its negative impact.Stable and explicit detection under poor depth-map quality remains a stated future research problem.Source: 2) Future Work · p. 13
  • Removing NLPR training data degrades performance by varying amounts across datasets.The F-measure drop is 0.5% on STEREO797 and 4.4% on LFSD.Source: 2) Future Work · p. 14

Discussion

  • The model architecture forms RGB-D features by fusing high-level RGB and depth features through PAI rather than learning the RGB-D stream from scratch.This design is described as reducing computation while producing more discriminative RGB-D features.Source: §V · p. 14
  • The decoder convergence structure centers interaction on RGB-D features to capture complementarity among RGB, depth, and RGB-D modalities.The stated goal is to obtain more discriminative and saliency-related features.Source: §V · p. 14
  • PAI integrates cross-modality information in the encoder, IGF routes selected RGB and depth decoder information into the RGB-D stream, and middleware refines features before decoding.The refinement middleware operates from both self-modality and cross-modality perspectives.Source: §V · p. 14
  • The method achieves competitive qualitative and quantitative performance on six datasets through cooperation between its architecture and technical modules.The conclusion attributes the result to their mutual cooperation and facilitation.Source: §V · p. 14

Related papers

Loading 2210.02843v1…