PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

Runmin Cong, Kepu Zhang, Chen Zhang, Feng Zheng, Yao Zhao, Qingming Huang, Sam Kwong

arXiv:2210.04266v1cs.CV

TL;DR

RGB-T SOD uses thermal imagery to address difficult low-light scenes, but thermal information is not always aligned with saliency. TNet regulates modality interaction and assigns different roles to thermal features in encoding and decoding. It outperforms 20 state-of-the-art methods on three public benchmark datasets.

  • Problem

    Thermal information is not inherently aligned with saliency, while existing RGB-T models focus on feature fusion and RGB-only SOD can struggle in low light.

    Source: §I · p. 1, C. Semantic Constraint Provider (SCP)… · p. 4

  • Method

    TNet uses global illumination estimation to regulate modality interaction, enriches thermal features with RGB semantics during encoding, and uses thermal cues for RGB decoding.

    Source: §I · p. 2

  • Results

    TNet outperforms 20 state-of-the-art methods on three public benchmark datasets.

    Source: §I · p. 2

  • Takeaways & Limitations

    TNet treats thermal imagery as auxiliary information whose role can be adjusted according to scene brightness.

    Source: §V · p. 10

Abstract

from arXiv · show

In recent years, RGB-T salient object detection (SOD) has attracted continuous attention, which makes it possible to identify salient objects in environments such as low light by introducing thermal image. However, most of the existing RGB-T SOD models focus on how to perform cross-modality feature fusion, ignoring whether thermal image is really always matter in SOD task. Starting from the definition and nature of this task, this paper rethinks the connotation of thermal modality, and proposes a network named TNet to solve the RGB-T SOD task. In this paper, we introduce a global illumination estimation module to predict the global illuminance score of the image, so as to regulate the role played by the two modalities. In addition, considering the role of thermal modality, we set up different cross-modality interaction mechanisms in the encoding phase and the decoding phase. On the one hand, we introduce a semantic constraint provider to enrich the semantics of thermal images in the encoding phase, which makes thermal modality more suitable for the SOD task. On the other hand, we introduce a two-stage localization and complementation module in the decoding phase to transfer object localization cue and internal integrity cue in thermal features to the RGB modality. Extensive experiments on three datasets show that the proposed TNet achieves competitive performance compared with 20 state-of-the-art methods.

I. INTRODUCTION

RGB-T SOD addresses RGB-based detection failures in dark scenes, but thermal information is not inherently aligned with saliency. TNet regulates the modalities’ roles and uses distinct interactions during encoding and decoding.

I. INTRODUCTION

  • Thermal images can complement RGB in low light, where RGB-only SOD may fail to detect salient objects.Thermal images reflect surface temperature, which is not necessarily aligned with saliency.Source: §I · p. 1
  • RGB-T annotations and saliency judgments remain mainly based on RGB, so directly porting RGB-D models can produce poor predictions.Thermal intensity does not imply saliency, and salient RGB objects can differ from hotter thermal objects.Source: §I · pp. 1–2
  • GIE predicts global illuminance to regulate thermal interaction, assigning thermal information a greater role in low-light and dark scenes.It derives an illumination map using a pretrained Retinex network, then pools and applies sigmoid activation to obtain an illuminance score.Source: §I · p. 2
  • SCP enriches thermal features with semantic content, while LC uses thermal features to provide localization and integrity information for RGB decoding.The thermal modality is treated as auxiliary during decoding, with interaction regulated by global illumination.Source: §I · p. 2
  • TNet outperforms 20 state-of-the-art methods on three public benchmark datasets.Source: §I · p. 2

II. RELATED WORK

Prior SOD work includes RGB-D methods that exploit depth cues and RGB-T methods that emphasize cross-modal fusion. TNet addresses the distinct role of thermal information by regulating interaction through GIE, SCP, and LC.

A. RGB and RGB-D Salient Object Detection

B. RGB-T Salient Object Detection

  • RGB-T SOD methods have progressed from graph-based approaches to CNN-based methods, with cross-modality cue extraction and integration central to CNN approaches.The cited CNN methods use multi-interaction decoding, multi-stage fusion, cross-guidance, bilateral fusion, perceptual importance, and multi-source feature fusion.Source: B. RGB-T Salient Object Detection · p. 3
  • TNet differentiates itself by regulating RGB-thermal interaction with GIE and using SCP during encoding and LC during decoding.Source: B. RGB-T Salient Object Detection · pp. 3–4

III. PROPOSED METHOD

TNet uses a two-stream encoder-decoder architecture in which a global illuminance score regulates interaction between RGB and thermal features. Its GIE module estimates scene brightness, while thermal features provide semantic guidance and decoding cues under brightness-dependent control.

A. Architecture Overview

  • TNet’s two-stream encoder-decoder uses the illuminance score to regulate modality interaction, with RGB-dominant decoding guided by semantically refined thermal features.The final localization and complementation module produces the saliency map.Source: A. Architecture Overview · p. 4

B. Global Illumination Estimation (GIE) module

  • The GIE module estimates an illuminance map from the input RGB image using a pretrained Retinex decomposition network.The paper describes the map as reflecting image brightness distribution.Source: B. Global Illumination Estimation… · p. 4
  • The GIE module converts the illuminance map into a global score using global average pooling and sigmoid activation, rather than retaining a pixel-level map that may introduce noise.The score is intended to quantify the light intensity of the whole scene.Source: B. Global Illumination Estimation… · p. 4
  • A smaller illuminance score reduces RGB semantic guidance to thermal features during encoding while increasing thermal influence on RGB localization and complementation during decoding.The stated intent in darker scenes is to preserve original thermal features and give them a greater decoding role.Source: B. Global Illumination Estimation… · p. 4

C. Semantic Constraint Provider (SCP) module

Thermal information can be weakly related to saliency and may mislead detection, so TNet uses the Semantic Constraint Provider to add RGB-derived semantic guidance to thermal features. The guidance is weighted by scene illumination, and feature visualizations show examples of misleading thermal responses being suppressed.

C. Semantic Constraint Provider (SCP) module

  • Thermal cues may mislead saliency detection because hot regions need not be salient and salient objects can be cooler than their surroundings.The paper also notes that people cannot directly perceive thermal information and RGB-T annotations mainly rely on RGB except in dark scenes.Source: C. Semantic Constraint Provider (SCP)… · pp. 4–5
  • The SCP derives a semantic mask from top-layer RGB features using channel compression and sigmoid activation, with saliency ground truth supervising mask learning.The RGB features supply high-level semantic information to guide the thermal modality.Source: C. Semantic Constraint Provider (SCP)… · p. 5
  • Illumination-weighted semantic guidance suppresses nonsalient high-thermal locations in normal light, while reducing RGB guidance in dim scenes to preserve thermal features.The mask is upsampled as needed, weighted by the global illuminance score, and used with a residual connection to update thermal features.Source: C. Semantic Constraint Provider (SCP)… · p. 5
  • Fig. 4’s fourth-layer feature visualizations show SCP suppressing misleading thermal responses, including hot ground and temperature-similar objects near salient regions.The comparison is between features extracted with and without SCP.Source: Figure 4 · p. 5, C. Semantic Constraint Provider (SCP)… · p. 5

D. Localization and Complementation (LC) module

The LC module keeps RGB features dominant in decoding and uses thermal features as auxiliary cues for localization and feature complementation. Its two stages use illumination-aware thermal guidance to refine decoder features and provide skip-connection information.

D. Localization and Complementation (LC) module

  • The LC module uses thermal features as auxiliary information in a single-stream decoder dominated by RGB features, reflecting the weak link between thermal imagery and saliency.It is decoupled into localization guidance and skip-connection complementation stages.Source: D. Localization and Complementation… · pp. 5–6, Figure 5 · p. 6
  • The localization guidance stage learns a weight map from semantically enhanced thermal features to refine RGB-based decoder features, with global illuminance controlling thermal influence.Thermal localization cues are intended to contribute more in low-light scenes.Source: D. Localization and Complementation… · p. 6
  • The skip-connection complementation stage fuses RGB and thermal encoder features according to global illuminance to complement decoder features.The resulting skip features are concatenated with location-refined decoder features, followed by channel attention and a 1×1 convolution.Source: D. Localization and Complementation… · p. 6
  • The final decoder feature is reduced to one channel, upsampled to the input resolution, and passed through sigmoid activation to produce the saliency map.Source: D. Localization and Complementation… · p. 6

E. Loss Function

Training supervises both saliency maps and semantic masks using BCE and IoU losses.

E. Loss Function

  • BCE and IoU losses supervise the saliency map and semantic mask during training.The final loss function is defined after specifying these two losses.Source: E. Loss Function · p. 6

A. Datasets and Evaluation Metrics

The evaluation uses three RGB-T SOD benchmarks, training on half of VT5000 and testing on the remaining VT5000 images plus VT1000 and VT821. Performance is assessed with five commonly used measures.

A. Datasets and Evaluation Metrics

B. Implementation details

TNet is compared with 20 SOD methods and outperforms the other deep-learning RGB-T methods on nearly all reported metrics. The experiments also specify the implementation and include qualitative comparisons across method families.

B. Implementation details

  • Implementation uses PyTorch with an ImageNet-pretrained backbone, Adam optimization, and 100 training epochs; BCE-only supervision is used for the first 30 epochs.The initial learning rate is 1e-4, divided by 10 every 45 epochs.Source: B. Implementation details · p. 7

C. Comparison with State-of-the-Art Methods

  • The comparison covers 20 state-of-the-art SOD methods, and RGB and RGB-D models are retrained on the same RGB-T training data for fairness.The compared methods include RGB, RGB-D, traditional RGB-T, and deep-learning RGB-T approaches.Source: C. Comparison with State-of-the-Art… · p. 7

1) Qualitative Comparison:

  • The qualitative comparison groups methods into nine RGB-T, two RGB-D, and two RGB SOD methods, with the groups marked by different colored boxes.Source: Figure 7 · p. 8

C. Comparison with State-of-the-Art Methods

  • On VT821, TNet gains 16.7% MAE, 1.6% S-measure, 2.4% F-measure, and 1.1% E-measure over the second-best method.TNet performs best among deep-learning RGB-T methods on all metrics except E-measure on VT1000; Table I reports the quantitative comparisons.Source: 1) Qualitative Comparison · pp. 7–8, Table I · p. 8

1) Qualitative Comparison:

2) Quantitative Comparison:

The supplied comparisons report visual performance across challenging scenes and test the contribution of thermal input against an RGB-only model. The ablation setup also compares the full model with variants that remove or replace components.

2) Quantitative Comparison:

  • The visual comparisons report strong localization, background suppression, detail representation, and robustness across challenging scenarios.The examples include small or multiple objects, complex backgrounds, low light, and hollow objects.Source: 2) Quantitative Comparison · p. 8

D. Ablation Study

1) Effectiveness of thermal modality:

  • On VT821, the full model raises F-measure from 0.851 to 0.904 versus the RGB-only model, a 6.2% gain.The MAE score also gains 41.2%; RGB-only predictions leave some background noise unsuppressed.Source: 1) Effectiveness of thermal modality · p. 9

2) Effectiveness of GIE module:

The ablations report that removing GIE, SCP, or LC degrades performance, and show that SCP and LC design choices matter. Their effects include thermal-noise regulation, semantic guidance, localization, and detection completeness.

2) Effectiveness of GIE module:

  • On VT5000, removing GIE reduces maximum F-measure from 0.895 to 0.880; the ablation also reports degradation on all three datasets.The authors associate GIE with regulating RGB and thermal feature roles when thermal images contain non-salient hot objects.Source: 2) Effectiveness of GIE module · p. 9

3) Effectiveness of SCP module:

  • Replacing SCP with concatenation does not outperform removing SCP, while removing SCP lowers performance relative to the full model.SCP uses high-level RGB semantics to encode semantic attributes in thermal features, making them more suitable for SOD.Source: 3) Effectiveness of SCP module · pp. 9–10

4) Effectiveness of LC module:

  • On VT5000, removing LC lowers maximum F-measure from 0.895 to 0.882; removing localization or complementation lowers it to 0.885 or 0.873.Direct addition yields VT5000 F-measure 0.882 versus 0.895 with adaptive skip connections; MAE changes from 0.037 to 0.033.Source: 4) Effectiveness of LC module · p. 10

E. Failure Cases and Future Work

TNet is reported to struggle with small, indistinct objects and slender, low-contrast objects. The authors also note that public RGB-T SOD datasets contain few nighttime scenes, while concluding that TNet performs competitively across three datasets.

E. Failure Cases and Future Work

  • TNet misses small objects indistinguishable from their surroundings and may incompletely detect slender, low-contrast objects.The authors suggest stronger feature extractors or reasoning mechanisms as future directions.Source: E. Failure Cases and Future Work · p. 10
  • VT821, VT1000, and VT5000 contain mostly daytime scenes and very few nighttime scenes, limiting their coverage of thermal-image use cases.The conclusion reports competitive TNet results on three datasets.Source: E. Failure Cases and Future Work · p. 10, §V · p. 11

V. Conclusion

  • TNet adjusts RGB and thermal roles according to scene brightness, with GIE, SCP, and LC mechanisms across encoding and decoding.The conclusion reports experiments on three datasets and improved detection accuracy and completeness.Source: §V · pp. 10–11
Loading 2210.04266v1…