PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

Zhengyi Liu, Yacheng Tan, Qian He, Yun Xiao

arXiv:2204.05585v1cs.CV

TL;DR

RGB-D and RGB-T salient object detection needs effective fusion of complementary modalities, especially under difficult lighting and weather conditions. SwinNet uses two-stream Swin Transformers, attention-based cross-modality refinement, and edge-guided decoding, and it outperforms state-of-the-art methods on both task types. The model's higher accuracy comes with high computational complexity and reduced speed.

  • Problem

    Effective cross-modality fusion remains challenging in RGB-D and RGB-T salient object detection, although depth and thermal data complement RGB under adverse conditions.

  • Method

    SwinNet uses two-stream Swin Transformer encoders, spatial alignment and channel re-calibration, and edge-guided decoding for multimodal salient object detection.

  • Results

    SwinNet outperforms state-of-the-art RGB-D and RGB-T SOD models; on RGB-D, its average improvements over VST are about 0.007 in S-measure, 0.017 in F-measure, 0.010 in E-measure, and 0.005 in MAE.

  • Takeaways & Limitations

    SwinNet provides a cross-modality fusion model that combines hierarchical transformer features, attention-based modality refinement, and edge-guided contour sharpening.

  • Takeaways & Limitations

    SwinNet has 198.7M parameters, about 124.3G FLOPs, and approximately 10 FPS inference speed, with most computation in its two Swin Transformer backbones.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are good at extracting contexture features within certain receptive fields, while transformers can model the global long-range dependency features. By absorbing the advantage of transformer and the merit of CNN, Swin Transformer shows strong feature representation ability. Based on it, we propose a cross-modality fusion model SwinNet for RGB-D and RGB-T salient object detection. It is driven by Swin Transformer to extract the hierarchical features, boosted by attention mechanism to bridge the gap between two modalities, and guided by edge information to sharp the contour of salient object. To be specific, two-stream Swin Transformer encoder first extracts multi-modality features, and then spatial alignment and channel re-calibration module is presented to optimize intra-level cross-modality features. To clarify the fuzzy boundary, edge-guided decoder achieves inter-level cross-modality fusion under the guidance of edge features. The proposed model outperforms the state-of-the-art models on RGB-D and RGB-T datasets, showing that it provides more insight into the cross-modality complementarity task.

I. INTRODUCTION

SwinNet addresses difficult RGB-D and RGB-T salient object detection by combining Swin Transformer representations with cross-modality attention and edge guidance. The model targets robust performance in challenging conditions and reports superior results against existing methods.

  • Motivation: RGB-D and RGB-T salient object detection remains challenging under insufficient light, cluttered backgrounds, and adverse imaging conditions.Depth supplies geometry, while thermal imagery captures radiated heat under difficult weather and lighting; effective cross-modality fusion remains challenging.
  • Motivation: Swin Transformer combines CNN-like locality and hierarchy with transformer-based long-range dependency modeling for modality-specific feature extraction.Its shifted-window design establishes dependencies across local windows while retaining hierarchical representations.
  • Contributions: SwinNet is a Swin Transformer-based salient object detector for both RGB-D and RGB-T tasks that outperforms state-of-the-art models.The architecture is designed to extract discriminative features for both cross-modality settings.
  • Contributions: Attention-based spatial alignment and channel re-calibration optimize intra-layer cross-modality features from spatial and channel perspectives.The module addresses shared salient positions and modality-specific channel importance.
  • Contributions: The edge-guided decoder performs inter-layer cross-modal fusion under edge-aware guidance to generate sharper salient-object contours.Edge information is used to clarify fuzzy boundaries during decoding.

B. RGB-T salient object detection

RGB-T salient object detection uses thermal imagery to complement RGB information in difficult scenes. SwinNet applies a two-stream Swin-based multimodal pipeline whose enhanced features and edge features are combined to produce saliency maps.

  • RGB-T motivation: Thermal images capture object heat and are insensitive to lighting and weather, supporting RGB-T detection in darkness, fog, and cluttered backgrounds.Thermal imagery is presented as a promising supplement to RGB for adverse conditions.
  • Related work: Prior RGB-T SOD methods include machine-learning approaches, CNN-based multimodal fusion, multi-interactive decoders, staged fusion, and context-guided refinement.The surveyed methods progressively introduce attention, multi-scale fusion, multi-level interaction, and cross-modality refinement.
  • Framework: The framework contains a two-stream backbone, spatial alignment and channel re-calibration module, edge-aware module, and edge-guided decoder.Enhanced multimodal features are generated before decoding, while edge features guide the final saliency-map generation.
  • Two-stream backbone: SwinNet uses two Swin Transformers to extract hierarchical features from paired modalities, with Swin-B selected for complexity and efficiency.Patch embedding and patch merging produce modality-specific hierarchical representations.

C. Spatial alignment and channel re-calibration module

The spatial alignment and channel re-calibration module improves multimodal feature fusion by first emphasizing common salient locations and then weighting modality-specific channels. This addresses complementary spatial and semantic information in RGB and depth streams.

  • Spatial alignment: RGB and depth features require spatial alignment because corresponding salient objects occupy the same positions across modalities.A common spatial attention map is computed from the two modality features and used to align them.
  • Channel re-calibration: The module re-calibrates channels separately because RGB emphasizes appearance and texture whereas depth provides spatial cues.Channel attention assigns greater weights to more salient content within each modality.
  • Channel re-calibration: Channel attention maps are multiplied with the original features to obtain channel-recalibrated representations.The operation emphasizes modality-specific salient channels after spatial alignment.
  • Output: The enhanced features after alignment and re-calibration show stronger representation ability.

D. Edge-aware module

The edge-aware module extracts boundary information from shallow depth features, where salient objects commonly exhibit useful popout structure. Attention and residual processing clarify these edge features for decoder guidance.

  • Edge extraction: Shallow depth-backbone features are used to produce edge features because depth contrast can depict salient-object contours.Shallow layers retain details, while salient objects are more likely to show popout structure in depth images.
  • Edge extraction: Three features are aligned in size through 1×1 convolutions and upsampling, then concatenated to generate the edge feature.The construction combines shallow-layer features from three levels.
  • Edge refinement: Channel attention and a residual connection are applied to the edge feature to generate clearer edge information.The resulting edge-aware output is used to guide decoding and enhance details.
  • Decoder guidance: The edge-aware output guides the model's decoding process and enhances salient-object details.

E. Edge-guided decoder

The edge-guided decoder fuses enhanced cross-modality features across hierarchy levels and uses edge features to produce salient features with sharper contours.

  • The decoder combines enhanced hierarchical features from different modalities with edge features to produce an edge-guided salient feature.
  • Aligned and re-calibrated color and depth features at each hierarchy are fused through addition, multiplication, and concatenation.
  • High-level fused features are progressively aggregated into shallow-layer fused features following the U-Net decoding strategy.
  • The edge-aware feature is combined with the fused feature to generate the final edge-guided salient feature.

F. Loss function

The loss-function design supervises both edge prediction and saliency prediction, using edge information derived from saliency annotations.

  • The loss function includes separate edge and saliency losses, denoted Le and Ls, respectively.
  • The edge map Se is generated by applying convolution and upsampling to the edge feature from the edge-aware module.
  • Edge supervision derives ground truth from saliency maps using a Canny edge detector and applies cross-entropy edge loss.

2) Saliency loss:

The edge-guided salient feature is transformed into a saliency map and trained with saliency loss, while performance is evaluated on RGB-D and RGB-T benchmarks using established metrics.

  • The edge-guided salient feature Fs is processed by convolution and upsampling to generate the final saliency map S.
  • Saliency loss Ls uses cross-entropy over salient-region and nonsalient pixels, with prediction values representing salient-region confidence.
  • RGB-D evaluation uses datasets including NLPR, NJU2K, STERE, and DUT, with training sets containing 2,185 or 2,985 paired images depending on the DUT protocol.
  • RGB-T evaluation uses VT821, VT1000, and VT5000, with 2,500 VT5000 image pairs used for training under the stated comparison protocol.
  • Evaluation uses PR curves, S-measure, F-measure, E-measure, and MAE to compare the model with state-of-the-art RGB-D and RGB-T SOD models.

B. Implementation details

SwinNet is evaluated on challenging RGB-D and RGB-T datasets using standard metrics and comparisons with state-of-the-art models. It achieves stronger quantitative and qualitative results across both modalities, including difficult scenes and imaging conditions.

  • Implementation details: Inputs are resized to 384×384, depth images are replicated to three channels, and training uses flipping, rotation, and border clipping.The Swin-B backbone uses pretrained parameters, while remaining parameters use PyTorch defaults and Adam optimization.
  • Evaluation protocol: SwinNet is compared against multiple state-of-the-art RGB-D and RGB-T salient object detection algorithms using author-provided or reproduced saliency maps.The evaluation uses precision-recall curves, S-measure, F-measure, E-measure, and MAE.
  • RGB-D results: On RGB-D datasets, SwinNet improves over transformer-based VST by about 0.007 S-measure, 0.017 F-measure, 0.010 E-measure, and 0.005 MAE on average.The largest improvements occur on NLPR, NJU2K, STERE, SIP, and DUT, while gains are smaller on DES.
  • RGB-T results: On RGB-T datasets, SwinNet achieves the best reported evaluation metrics and outperforms competing models by a large margin.Qualitative comparisons also cover poor illumination, low thermal contrast, small objects, multiple objects, and noise disturbance.

D. Ablation studies

Ablation studies examine the backbone, modality-fusion module, and feature representations. The results favor Swin Transformer and spatial alignment with channel re-calibration for cleaner, more complete cross-modal features.

  • The effectiveness of Swin Transformer backbone: Replacing Swin Transformer with CNN and alternative transformer backbones shows that Swin Transformer significantly improves detection performance.The authors attribute this to combining CNN locality with transformer global awareness.
  • The effectiveness of Swin Transformer backbone: Swin Transformer produces more complete salient features than ResNet, which can ignore small objects in depth features.The comparison includes fourth-layer color and depth features and final prediction maps.
  • Cross-modality fusion: Spatial alignment and channel re-calibration purify color features and reduce noise, while color assistance makes depth features closer to ground truth.These effects are especially visible in the reported first, second, and third columns.
  • Cross-modality fusion: Compared with the depth-enhanced module, spatial alignment and channel re-calibration improves S-measure, F-measure, E-measure, and MAE by about 0.006, 0.006, 0.005, and 0.002.The comparison tests the added alignment operation against a module with similar spatial and channel attention.

3) The effectiveness of edge guidance:

Removing edge guidance weakens object-detail representation and reduces quantitative performance. The ablation therefore supports edge-guided decoding for sharper detected-object details.

  • The effectiveness of edge guidance: Removing edge guidance from the decoder reduces detected-object detail in the visual ablation comparison.The comparison contrasts decoder features and predictions with and without edge guidance.
  • The effectiveness of edge guidance: Edge guidance improves S-measure, F-measure, E-measure, and MAE by about 0.004, 0.009, 0.006, and 0.003, respectively.These results are reported in the edge-guidance ablation.

4) The effectiveness of each modality:

The modality ablation indicates that depth generally improves RGB-D salient object detection, but low-quality depth can reduce performance. Complexity analysis attributes most computation to the two Swin Transformer backbones, while the auxiliary fusion and edge modules add little cost.

  • The effectiveness of each modality: Depth information generally improves RGB-D salient object detection, but its benefit is weaker than color and can become negative on low-quality STERE depth images.The ablation compares independent modalities and reports depth's dataset-dependent contribution.
  • Model complexity analysis: 198.7M parameters and 124.3G FLOPs make SwinNet computationally complex, with inference reaching about 10 FPS including input/output and preprocessing.The reported complexity combines model size, computation, and end-to-end inference speed.
  • Model complexity analysis: Most computation resides in the two Swin Transformer backbones, whereas spatial alignment, channel re-calibration, and edge-aware modules add nearly no cost.The edge-guided decoder contributes only a small amount through convolution operations during upsampling.
  • The effectiveness of each modality: The study separately evaluates edge-guided decoder effectiveness and spatial alignment, channel re-calibration, and edge guidance through visual comparison and ablation tables.The supplied captions identify the evaluation targets but do not provide their numerical outcomes.
Loading 2204.05585v1…