PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection

Zhengzheng Tu, Zhun Li, Chenglong Li, Yang Lang, Jin Tang

arXiv:2005.02315v3cs.CV

TL;DR

RGBT SOD must identify common salient regions while handling incomplete modality complementation, noise, and saliency bias. The paper uses a multi-interactive dual-decoder with hierarchical, modality, and global-context interactions, reporting state-of-the-art performance on public RGBT and RGBD SOD datasets.

  • Problem

    Existing RGBT SOD methods do not fully exploit modality complementarity and multi-type cues, while effective complementation, noise suppression, and saliency-bias reduction remain key problems.

  • Method

    The method encodes RGB and thermal inputs separately, then uses cascaded dual-decoder interactions among modalities, multi-level features, and global contexts.

  • Results

    Extensive experiments report state-of-the-art performance on several public RGBT and RGBD SOD datasets.

  • Takeaways & Limitations

    The network is designed to capture common saliency, preserve complementary information, and remain effective when one modality is invalid or noisy.

Abstract

from arXiv · show

RGB-thermal salient object detection (SOD) aims to segment the common prominent regions of visible image and corresponding thermal infrared image that we call it RGBT SOD. Existing methods don't fully explore and exploit the potentials of complementarity of different modalities and multi-type cues of image contents, which play a vital role in achieving accurate results. In this paper, we propose a multi-interactive dual-decoder to mine and model the multi-type interactions for accurate RGBT SOD. In specific, we first encode two modalities into multi-level multi-modal feature representations. Then, we design a novel dual-decoder to conduct the interactions of multi-level features, two modalities and global contexts. With these interactions, our method works well in diversely challenging scenarios even in the presence of invalid modality. Finally, we carry out extensive experiments on public RGBT and RGBD SOD datasets, and the results show that the proposed method achieves the outstanding performance against state-of-the-art algorithms. The source code has been released at:https://github.com/lz118/Multi-interactive-Dual-decoder.

I. INTRODUCTION

The paper targets RGBT SOD challenges involving modality complementation, noise, and saliency bias. It proposes a dual-decoder with multi-type interactions and augmentation for robust common-saliency prediction.

  • Motivation: RGBT SOD estimates common conspicuous regions from aligned visible and thermal infrared images.Thermal images provide temperature-related cues that can assist difficult vision tasks.
  • Challenges: Prior RGBT SOD methods do not fully solve effective modality complementation, noise interference, or saliency bias toward one modality.The desired output should focus on common conspicuous regions in both modalities.
  • Approach: The proposed dual-decoder separately decodes the modalities while enabling decoded-feature interaction to adjust saliency toward common regions.Its multi-interaction blocks model modality, multi-level-feature, and global-context interactions.
  • Robustness: Randomly replacing a modality with zeros or Gaussian noise trains the network for invalid or unreliable inputs.The augmentation increases challenging training examples and is reported to improve defective-input handling.
  • Results: Extensive experiments on public RGBT and RGBD SOD datasets report state-of-the-art performance against existing methods.The same network structure treats the two modalities equally, unlike the usual supplementary-depth framing in RGBD SOD.
  • Approach: The method integrates spatial details, complementary modality information, and global context progressively during decoding.These cues support clearer salient-object structure and complete-region highlighting.

II. RELATED WORK

Related work covers deep SOD, multimodal RGBD SOD, and early RGBT SOD methods. This paper distinguishes its approach by performing modality fusion during decoding with a dual-decoder.

  • Salient Object Detection: Deep SOD methods progressively combine spatial, semantic, contextual, residual, and boundary cues to predict salient objects.Examples include hierarchical refinement, context-aware attention, cascaded decoding, and edge detection.
  • RGBT SOD: RGBT SOD research includes manifold ranking, modality weighting, collaboration, heterogeneity modeling, and collaborative graph learning.These methods aim to fuse visible and thermal information for saliency estimation.
  • RGBD SOD: RGBD SOD methods use handcrafted multimodal features, feature contrasts, modality concatenation, recurrent decoding, or location priors.Many approaches combine RGB and depth before or during encoding.
  • Difference from Prior Work: Unlike cited RGBD SOD methods, this work fuses modalities during decoding and separately decodes them before interaction.The design seeks common saliency across the two modalities.

III. MULTI-INTERACTIVE DUAL-DECODER NETWORK

The network uses independent RGB and thermal backbones, a global information module, and cascaded multi-interaction blocks in a dual-decoder. Final decoder features are fused into a saliency map.

  • Overall Architecture: Independent backbones extract hierarchical features from RGB and thermal images before joint decoding.The architecture assigns modality-specific processing to the two input streams.
  • Global Information Module: The global information module combines top-level RGB and thermal features to produce global contexts for salient-region localization.Multiple receptive fields are used to capture region-based context.
  • Prediction: The final features from the two decoder streams are fused to predict the final saliency map.Progressive interactive decoding suppresses modality bias and noise according to the architecture description.
  • Feature Processing: Channel attention computes modality-aware feature weighting and is used to reconstruct features with reduced channel dimensionality.The method marks this operation as CA and decreases intermediate channels to 256 in the global module.
  • Global Information Module: Four adaptive max-pooling branches with output sizes 1 × 1, 5 × 5, 9 × 9, and 13 × 13 provide different receptive fields.Their up-sampled outputs are concatenated with the base feature to generate global contexts.

D. Dual-decoder Network

The dual-decoder progressively restores modality-specific details while exchanging information between branches and integrating global context. Cascaded MIBs support complementary, consistent, and spatially detailed decoding.

  • Dual-decoder Network: Cascaded multi-interaction blocks decode each modality progressively while interacting across modalities, hierarchical features, and global contexts.The blocks are embedded in each decoder stream.
  • Interaction with hierarchical features: Encoded hierarchical features refine up-sampled decoder outputs, preserving modality-specific characteristics and restoring spatial details.Shallow encoded features contain abundant detail, while decoder outputs maintain modality characteristics.
  • Interaction between modalities: Concatenated outputs from both decoders provide fused and modality-specific information for adaptive feature reconstruction.Channel attention limits excessive disturbance between modalities during fusion.

3) Interaction with global contexts:

The global information module supplies global context to each multi-interaction block, helping preserve salient-region localization as spatial details are fused. Interacting global context with other features emphasizes salient regions and suppresses background noise.

  • 3) Interaction with global contexts:: Global context is integrated into each MIB to preserve salient-region locations during progressive top-down decoding.High-level semantic information can be diluted when spatial details are fused, while limited receptive fields may omit parts of large or multiple objects.
  • 3) Interaction with global contexts:: Global-context interaction emphasizes the main salient region and suppresses background noise.The paper reports this effect in its visualization of the global information module.
  • 3) Interaction with global contexts:: The MIB reconstructs fused features by directly summing modality-interaction, global-context, and multi-level feature inputs before convolution.The operation is expressed as Zi = Conv( ˜ Mi + ˜ Gi + ˜Ai), for i = 2, 3, 4.
  • 3) Interaction with global contexts:: Final dual-decoder features retain modality-specific characteristics while also containing valid fused information from prior decoder interactions.The final features are concatenated and passed through channel-wise attention to predict the final saliency map.
  • 3) Interaction with global contexts:: Visualization shows progressively more accurate details in MIB outputs and increasing consistency between the two decoder branches.The authors interpret this consistency as evidence of modality complementarity and reduced modality bias.

E. Loss Function

The loss design supervises decoder branches, global context, and the final prediction, while adding smoothness and equal-modality considerations to support training under defective inputs.

  • E. Loss Function: Binary cross-entropy supervises the two decoder saliency maps S1 and S2 against the ground truth Y.The paper defines the standard BCE loss for predicted saliency maps and corresponding ground-truth pixels.
  • E. Loss Function: A separate BCE loss trains the global information module using a saliency map Sg and a ground truth Y down-sampled by a factor of 16.The down-sampled target matches the spatial size of Sg.
  • E. Loss Function: The final predicted map Sf receives its own loss function, while smoothness loss constrains region consistency and clearer edges.The smoothness term uses first-order derivatives of the saliency map.
  • E. Loss Function: The network is trained through the cooperation of four constraints, with β = 0.5 empirically balancing the smoothness loss.The supplied passages state the balancing value but do not enumerate all four constraints in one passage.
  • E. Loss Function: The method treats the two modalities with equal importance in the dual-decoder rather than reducing a modality’s loss weight for defective inputs.Instead, modality effectiveness is addressed during data augmentation by creating more defective samples.

F. Noisy Data Augmentation

The experiments use public RGBT and RGBD datasets, standard SOD metrics, and augmentation designed to expose the model to invalid or noisy modalities.

  • F. Noisy Data Augmentation: The augmentation strategy randomly replaces one modality with a zero map or a noisy map to train robustness to defective inputs.The paper motivates this setting because RGB can be indiscernible in low illumination, thermal can fail when temperatures are similar, and both can contain noise.
  • F. Noisy Data Augmentation: Experiments are conducted on public RGBT and RGBD SOD datasets with comparison and ablation analyses.The experimental section introduces datasets, setup, evaluation metrics, comparison experiments, and ablation experiments.
  • F. Noisy Data Augmentation: The RGBT datasets include VT821 with 821 registered pairs, VT1000 with 1000 relatively simple pairs, and VT5000 with 5000 complex aligned pairs.VT821 includes added noise and manually registered thermal images with vacant regions; VT5000 contains varied objects and complex scenes.
  • F. Noisy Data Augmentation: Evaluation uses F-measure, S-measure, E-measure, and mean absolute error as standard salient-object-detection metrics.The paper describes these metrics as widely used in SOD.
  • F. Noisy Data Augmentation: F-measure is computed from precision and recall with β2 = 0.3, emphasizing precision, and weighted F-measure is also reported.The evaluation code uses 20 equally divided saliency thresholds for PR-curve computation.
  • F. Noisy Data Augmentation: MAE measures the difference between predicted saliency maps and ground truth, with values averaged across test samples.S-measure evaluates spatial-structure similarity, while E-measure captures image-level statistics and pixel-level matching.

C. Comparison with State-of-the-Art Methods

The comparison evaluates the proposed method against 12 existing methods across three testing datasets, using highlighted rankings to distinguish the strongest scores.

  • C. Comparison with State-of-the-Art Methods: The study compares the proposed method with 12 existing methods on three testing datasets.The comparison includes traditional and deep-learning RGBT SOD methods, RGBD SOD methods, and single-modality SOD methods.
  • C. Comparison with State-of-the-Art Methods: Table I marks the best, second-best, and third-best scores in red, green, and blue, respectively.The caption specifies the ranking colors across the three testing datasets and 12 methods.
  • C. Comparison with State-of-the-Art Methods: Single-modality baselines receive early fusion of the modalities to preserve their architectures for a fair comparison.The paper also states that its own method uses no post-processing, though the supplied passage truncates the surrounding sentence.

1) Quantitative evaluation:

The proposed method outperforms existing methods across RGBT SOD evaluations, including challenging modality conditions and diverse image pairs. Challenge-based analysis further reports the best performance across all evaluated challenges.

  • The method outperforms four existing RGBT SOD methods across the reported evaluation metrics, with especially strong superiority on VT821.
  • RGBD SOD methods can perform on RGBT data, but their depth-oriented designs are not fully suited to equally important RGB and thermal modalities.DMRA performs poorly on RGBT data, while S2MA performs well; the paper attributes this difference to the distinct goals and modality roles of RGBD and RGBT SOD.
  • Qualitatively, the method produces better predictions across eight challenging RGBT image pairs by exploiting modality complementarity and suppressing modality bias.The method remains less affected when one modality is useless and combines useful information when both modalities are unreliable.
  • The method achieves the best performance on all VT5000 challenges and quality attributes reported in the challenge-based comparison.Low illumination, small salient objects, and bad weather are identified as especially difficult because they can create deficient modalities or ambiguous context.

1) Supervision effectiveness in Dual-decoder:

Ablation studies support the dual-decoder design, branch supervision, modality interaction, channel attention, and independent decoder parameters. These components improve feature cooperation, modality-bias suppression, and robustness to noisy inputs.

  • 1) Supervision effectiveness in Dual-decoder:: Removing supervision from the two decoder branches decreases performance across all four reported metrics and weakens modality complementarity.Without branch supervision, the decoder streams are less likely to produce consistent feature maps and cooperate effectively.
  • 2) Effectiveness of MIB:: Disabling modality interaction prevents the network from withstanding noise in a deficient modality and limits its ability to eliminate modality bias.With modality interaction, saliency maps show larger confidence in both foreground and background regions.
  • 2) Effectiveness of MIB:: Channel attention is used to select informative channels when fusing the two top decoded features, rather than immediately concatenating or summing them.The supplied ablation passage introduces this design choice but does not report its numerical outcome.
  • 4) Effectiveness of dual-decoder network:: The single-decoder variant performs obviously worse than the dual-decoder baseline, supporting separate decoding before feature interaction.The variant directly concatenates hierarchical features from both modalities in one decoder stream while retaining global-context interaction.
  • 4) Effectiveness of dual-decoder network:: Independent decoder parameters achieve maximum gains of 2.0%, 0.4%, 1.5%, 10.0%, and 2.0% across the four evaluation metrics compared with shared parameters.The paper attributes the gains to learning modality-specific saliency before the two streams interact to obtain common saliency.

5) Effectiveness of noisy data augmentation:

Noisy data augmentation improves performance across the reported test datasets and yields stable training gains. The backbone comparison indicates that stronger feature representation alone does not guarantee better results when spatial resolution is lower.

  • 5) Effectiveness of noisy data augmentation:: Training with noisy data improves the four evaluation metrics by average 1.5%, 1.0%, 2.1%, and 9.0% on three test datasets.The training curves show stable gains, and the network converges after 90 epochs.
  • 6) Effectiveness of different backbones:: Replacing VGG16 with ResNet50 produces little performance difference because ResNet50 extracts lower-resolution features with fewer spatial details.
  • 6) Effectiveness of different backbones:: The evaluated network setups all outperform the state-of-the-art methods, while different setups show advantages in different scenes.The supplied passage notes that the MIDD(R+) setup improves performance in some aspects but is not optimal overall.

2) Experiment setup:

Experiments evaluate the proposed method against RGBD SOD baselines across multiple datasets and standard saliency metrics, while also identifying remaining multimodal SOD challenges.

  • Datasets: 1485 NJU2K image pairs and 700 NLPR image pairs are sampled for training, with testing on DES, LFSD, SIP, SSD, and STERE.
  • Evaluation protocol: The evaluation compares the method with ten RGBD SOD methods using S-measure, E-measure, F-measure, and MAE.
  • Results: The method generally outperforms competing methods on RGBD datasets, although SSD shows lower stability because of fewer and less diverse samples.
  • Future work: Future work includes explicit cross-modal correlation, alignment-free multimodal SOD, visual-saliency uncertainty, and related multimodal detection problems.
  • Method: The proposed network uses cascaded multi-interactive modules to fuse modalities, hierarchical features, and global contexts while implicitly suppressing modality bias.
Loading 2005.02315v3…