PaperlayerView exact revision on arXiv

Source-linked AI summary

RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

Hongbo Gao, Zhengyu Li, Xueru Nie, Dihao Zhu, Lijun Zhao, Yunke Wang, Chang Xu

arXiv:2609.12622v1cs.CV

TL;DR

RGB-T salient object detection relies on complementary visible and thermal cues, but modality degradation can make those cues unreliable and hinder fusion. RA-SOD propagates reliability through representation learning, refinement, and fusion; it reports state-of-the-art performance and strong degradation robustness across four benchmarks.

  • Problem

    RGB and thermal degradation can create unreliable or misleading evidence, while existing methods often assume encoded features remain sufficiently reliable for later fusion.

    Source: §1 · p. 2

  • Method

    RA-SOD models reliability across feature representation, uncertainty-guided dual-stream refinement, and pixel-wise modality competition for fusion.

    Source: §1 · pp. 2–3

  • Results

    RA-SOD reports state-of-the-art performance and strong robustness under challenging modality degradation scenarios on VT821, VT1000, VT5000, and VT-IMAG.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The reported experiments support reliability-aware modeling across representation, decoding, and fusion for RGB-T SOD under modality degradation.

    Source: §1 · p. 3, §5 · p. 15

Abstract

from arXiv · show

RGB-Thermal (RGB-T) salient object detection leverages complementary cues from visible and thermal modalities to improve robustness in challenging environments. However, in real-world scenarios, the reliability of each modality is inherently unstable: RGB images degrade under low illumination, motion blur, and noise, while thermal imagery often suffers from contrast compression and sensor artifacts. Such degradation introduces unreliable perceptual evidence that can mislead cross-modal fusion and significantly deteriorate detection performance. To address this challenge, we propose RA-SOD, a reliability-aware RGB-T salient object detection framework that explicitly models modality reliability and integrates it into feature learning and cross-modal fusion. First, we introduce a reliability-conditioned representation that adaptively compensates degraded modality features while preserving structural cues. Second, an uncertainty-guided dual-stream refinement strategy progressively corrects cross-modal representations while suppressing unreliable evidence. Finally, we propose a pixel-wise modality competition mechanism that dynamically selects modality cues according to spatial reliability for fine-grained fusion. Extensive experiments on four benchmarks (VT821, VT1000, VT5000, and VT-IMAG) demonstrate that RA-SOD achieves state-of-the-art performance and exhibits strong robustness under severe modality degradation. Code and models are available at https://github.com/zaoxienian/RA-SOD.

1 Introduction

RGB-T SOD uses complementary visible and thermal cues, but degradation can make either modality unreliable and undermine fusion. RA-SOD models reliability across feature representation, refinement, and fusion, and reports strong performance on four benchmarks.

1 Introduction

  • RGB and thermal degradation can distort local responses and obscure object boundaries, making modality evidence unreliable or misleading for cross-modal fusion.RGB failures include low light, motion blur, and noise; thermal failures include contrast compression, suppressed temperature gradients, and block-like artifacts.Source: §1 · p. 2, Figure 1 · p. 2
  • Existing RGB-T methods emphasize elaborate fusion but often assume encoded modality features remain reliable, although degradation can arise early and spread through the network.Fixed feature transformations may propagate or amplify distorted signals, leaving later fusion to disentangle structure from modality-specific noise.Source: §1 · p. 2
  • RA-SOD conditions backbone representations to compensate for degraded features while preserving structural priors, then uses uncertainty-guided refinement and pixel-wise modality competition for fusion.Predicted uncertainty regulates bidirectional correction, while the competition mechanism selects RGB or thermal cues according to spatial reliability.Source: §1 · pp. 2–3
  • RA-SOD reports state-of-the-art performance and strong robustness under challenging modality degradation scenarios on VT821, VT1000, VT5000, and VT-IMAG.The supplied passage gives no metric values for this overall comparison.Source: §1 · p. 3

2 Related Work

Prior SOD work develops from handcrafted and unimodal approaches toward multi-modal fusion and interaction, while degradation-aware representation learning remains less explored. The section also situates RA-SOD's reliability-aware architecture among these approaches.

2.1 RGB Salient Object Detection

  • Unimodal RGB SOD evolved from handcrafted priors to deep pixel-wise models and richer representations, but remains vulnerable to low illumination, motion blur, and sensor noise.Later approaches include multi-scale aggregation, attention, boundary guidance, and generalist or state-space architectures.Source: §2.1 · p. 3

2.2 RGB-X Salient Object Detection

  • RGB-X SOD uses depth geometry or thermal radiation cues, while multi-modal methods progress from input concatenation to feature interaction, graph learning, and boundary enhancement.Depth can be compromised by low illumination or complex reflections; thermal cues are described as robust to lighting changes.Source: §2.2 · p. 3
  • Fig. 2 depicts RA-SOD's three components: reliability-conditioned encoding, uncertainty-guided dual-stream refinement, and pixel-wise modality competition.The final saliency map aggregates predictions from the RGB, Thermal, and Fusion streams.Source: Figure 2 · p. 4

2.3 Multi-modal Learning in RGB-T SOD

  • RGB-T methods increasingly model feature fusion and interaction, but modality-degradation-aware representation learning remains less explored.RA-SOD applies input-adaptive routing through a lightweight shared expert bank to adapt representations under modality degradation.Source: §2.3 · p. 4

3 Methodology

RA-SOD propagates modality reliability through encoding, uncertainty-guided refinement, and spatially adaptive fusion. Its hierarchical design aims to mitigate unreliable cues caused by modality degradation.

3 Methodology

  • RA-SOD models reliability across representation, refinement, and fusion to mitigate the propagation of unreliable cues caused by modality degradation.Source: §3 · p. 5

3.1 Overall Architecture

  • The architecture uses a reliability-conditioned encoder and parallel RGB, Thermal, and Fusion decoding streams whose independent predictions are linearly aggregated.Source: §3.1 · p. 5

3.2 Reliability-Conditioned Backbone Representation

3.3 Uncertainty-Guided Dual-Stream Decoding

  • Dual top-down RGB and Thermal streams predict uncertainty maps and use them for bidirectional cross-modal refinement before recursively propagating refined states.Source: §3.3 · p. 6, Top-down recursion · p. 7

3.4 Pixel-wise Fusion Stream Decoding

  • Fusion decoding uses pixel-wise modality competition to select more reliable RGB or Thermal cues at each location and suppress locally degraded responses.Source: Pixel-wise modality competition · p. 8

3.5 Multi-Stream Saliency Prediction

  • Independent lightweight heads predict saliency from RGB, Thermal, and fused streams; linear aggregation combines outputs to exploit complementary cues.Source: §3.5 · p. 8

3.6 Training Objective

  • Deep supervision jointly optimizes RGB, Thermal, and Fusion predictions with BCE and IoU losses against ground-truth maps.Source: §3.6 · p. 8

4 Experiments

Across standard and degraded RGB-T benchmarks, RA-SOD ranks at or near the top and is particularly robust under modality corruption. Its analysis and ablations examine efficiency, backbone generalization, uncertainty-guided refinement, reliability-conditioned representations, and pixel-wise fusion.

4.2 Comparison on Standard Benchmarks

  • On VT5000, RA-SOD ranks first on all five metrics, while remaining in the top two on VT821 and VT1000; it also improves on ConTriNet across reported VT5000 metrics.On VT1000, it achieves the best Fβ, F w β, and Em, and remains competitive in Sm and M compared with CAVER.Source: Quantitative Results · p. 10

4.3 Robustness Analysis under Degradations

  • Under five synthetic RGB and thermal degradations on VT1000, RA-SOD improves Sm from 0.921 to 0.927, Fβ from 0.895 to 0.906, and F w β from 0.895 to 0.907, while reducing M to 0.020.The test corruptions include RGB low-light, noise, and blur, plus thermal hot-block artifacts and contrast compression.Source: Synthetic Degradation Settings · p. 11, Table 2 · p. 11
  • On VT-IMAG, designed for severe sensor failures and environmental degradation, RA-SOD outperforms ConTriNet on Fβ and F w β and achieves the highest Sm and Em.It is superior on four of five metrics among state-of-the-art CNN-based methods.Source: Performance on the VT-IMAG Challenge Set · p. 12, Table 3 · p. 12

4.4 Analysis of RA-SOD

4.5 Ablation Studies

  • Removing uncertainty-guided dual-stream decoding lowers F w β on VT1000 and VT-IMAG, while uncertainty gates rise for the modality affected by RGB low-light or thermal contrast degradation.Without uncertainty guidance, the degraded-setting score also drops to 0.903.Source: Table 7 · p. 14, Table 8 · p. 14
  • The full reliability-conditioned backbone model outperforms the frozen-backbone ablation, improving VT5000 F w β from 0.851 to 0.855 and reducing M to 0.029.Modality-specific adapters and a shared expert bank also improve performance over the frozen backbone.Source: Effectiveness of… · p. 14, Table 6 · p. 14
  • On VT5000, pixel-wise modality competition raises Fβ from 0.865 to 0.870 and F w β from 0.850 to 0.855; three experts provide the best reported accuracy-robustness trade-off.At K = 3, Sm is 0.897, F w β is 0.855, and M is 0.029.Source: Impact of the Number of Experts · p. 14, Effectiveness of Pixel-wise Modality… · p. 15, Table 10 · p. 15

5 Conclusion

RA-SOD addresses modality degradation by modeling reliability across representation learning, decoding refinement, and fusion. On multiple benchmarks, it achieves state-of-the-art performance and strong robustness under severe degradation conditions.

  • RA-SOD achieves state-of-the-art performance on multiple benchmarks and shows strong robustness under severe degradation conditions.Source: §5 · p. 15
  • RA-SOD models reliability across representation learning, decoding refinement, and fusion rather than focusing mainly on fusion strategies.Source: §5 · p. 15
  • Its reliability-conditioned backbone compensates for degraded features while preserving structural priors, and its decoder and pixel-wise competition strategy regulate cross-modal integration.The decoder prevents unreliable cues from being propagated; pixel-wise competition enables spatially adaptive integration.Source: §5 · p. 15

Related papers

Loading 2609.12622v1…