PaperlayerView exact revision on arXiv

Source-linked AI summary

RGBT Salient Object Detection: A Large-scale Dataset and Benchmark

Zhengzheng Tu, Yan Ma, Zhun Li, Chenglong Li, Jieming Xu, Yongtao Liu

arXiv:2007.03262v6cs.CV

TL;DR

RGBT salient object detection lacks sufficiently large and comprehensive benchmarks for complex environments. The paper introduces VT5000 and an attention-based baseline, reporting superior performance to state-of-the-art methods on VT5000 and two public datasets.

  • Problem

    RGBT salient object detection is constrained by limited large-scale datasets and benchmarks despite the complementary information available from RGB and thermal images.

  • Method

    The paper constructs VT5000 and develops a CNN baseline that extracts modality-specific multi-level features and fuses them with attention, including edge supervision.

  • Results

    The proposed baseline outperforms state-of-the-art methods on VT5000, VT821, and VT1000.

  • Takeaways & Limitations

    VT5000 supports robustness analysis across 11 challenges and provides annotations that can support future weakly supervised RGBT salient object detection research.

Abstract

from arXiv · show

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared images becomes a new research direction for detecting salient object in complex scenes recently, as thermal infrared spectrum imaging provides the complementary information and has been applied to many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multi-level features within each modality and aggregates these features of all modalities with the attention mechanism, for accurate RGBT salient object detection. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection.

I. INTRODUCTION

The paper addresses limitations in RGBT salient object detection by introducing the large-scale VT5000 benchmark and an attention-based CNN baseline. VT5000 broadens scene diversity and challenge coverage, while experiments report stronger performance than existing methods.

  • RGB salient object detection remains challenging when salient objects resemble their surroundings or scenes contain adverse conditions.
  • Existing RGBT datasets and methods are limited by insufficient scale, alignment errors, simple scenarios, ineffective feature fusion, background noise, and unclear object boundaries.
  • VT5000 contains 5000 RGB–thermal image pairs collected across environments, with automatic alignment, diverse object and scene factors, 11 challenge annotations, and imaging-quality labels.
  • The baseline uses an end-to-end two-stream CNN to extract multi-scale modality-specific features and selectively fuse them using attention mechanisms.
  • The proposed approach outperforms state-of-the-art methods on VT5000, VT821, and VT1000, alongside a comprehensive benchmark analysis.

II. RELATED WORK

Related work progresses from RGBD saliency and attention-based feature selection toward RGBT methods that exploit complementary RGB and thermal information. Prior RGBT datasets and algorithms motivate larger, more capable benchmarks and improved modality fusion.

  • RGBD salient object detection established multimodal saliency datasets using stereo cameras, Kinect, Internet images, movies, and real-world or virtual scenes.
  • Earlier RGBT benchmarks include VT821 with 821 image pairs and VT1000 as a more challenging dataset.
  • Attention mechanisms support feature selection in diverse vision tasks and can emphasize channels responsive to salient objects while suppressing background regions.
  • RGB and thermal infrared data provide complementary cues because RGB is affected by illumination and weather, while depth imaging has limitations for equal-distance surfaces.
  • Prior RGBT saliency methods include manifold ranking, superpixel graph optimization, and cooperative ranking with modality reliability weights.

III. VT5000 BENCHMARK

VT5000 is designed as a diverse RGBT salient object detection benchmark with aligned image pairs, manual annotations, varied imaging conditions, and challenge labels. Its statistics emphasize difficult object sizes and backgrounds for evaluating algorithm robustness.

  • Data Collection: VT5000 contains 5000 RGB–thermal image pairs captured with matching camera parameters to reduce manual alignment errors.
  • Data Annotation: Six viewers identify salient objects for annotation, and the dataset includes manually labeled ground truth and imaging-quality information for RGB and thermal modalities.
  • Dataset Diversity: The benchmark varies illumination, object categories, sizes, positions, quantities, and backgrounds across recorded places and environments.
  • Challenge Annotations: Backgrounds challenge detection when their temperature or appearance resembles salient objects or when clutter makes separation difficult.
  • Challenge Annotations: VT5000 annotates 11 challenges, including object size, multiple objects, low illumination, center bias, boundary crossing, similar appearance, thermal crossover, clutter, blur, and bad weather.
  • Dataset Statistics: The training set contains more large salient objects, whereas the test set contains more small objects and therefore more hard samples.

D. Advantages of Our Dataset

VT5000 is designed as a larger, more diverse, and more challenging RGBT benchmark than earlier datasets, with improved imaging alignment and broad challenge coverage.

  • VT5000 uses 2500 training samples and 2500 testing samples, making it a large-scale dataset.
  • VT5000 samples vary in salient-object size, category, surroundings, imaging conditions, and spatial location.The dataset includes statistical analysis of object diversity and 11 annotated challenges for testing algorithm sensitivity.
  • The training set contains more large salient objects, whereas the testing set contains more small salient objects and therefore harder samples.
  • Unlike VT821 and VT1000, VT5000 contains more complex scenes and RGBT image pairs that do not require manual alignment.The dataset also uses thermal cameras with automatic focusing and captures thermal texture information effectively.

IV. ATTENTION-BASED DEEP FUSION NETWORK

ADFNet uses two modality-specific VGG16 streams, attention-guided feature selection, and multi-level fusion to detect salient objects in RGBT images.

  • ADFNet uses a two-stream CNN that separately extracts RGB and thermal infrared features before RGBT salient object detection.Pretrained VGG16 initializes the two branches for feature representation.
  • The network applies attention modules to weight informative features from the RGB and thermal infrared branches before fusion.
  • CBAM combines channel-wise and spatial-wise attention to select less noisy features from complex-scene inputs.Channel attention emphasizes informative channels, while spatial attention highlights informative spatial positions.
  • Channel attention aggregates average- and max-pooled spatial information, applies 1×1 convolutions, sums the outputs, and uses a sigmoid weight vector.
  • Spatial attention pools features along the channel axis, concatenates the descriptors, applies convolution, and produces weights with a sigmoid function.

C. Multi-modal Multi-layer Feature Fusion

The method fuses RGB and thermal infrared representations across multiple VGG16 layers to retain complementary information at different feature scales.

  • The network uses two VGG16 streams and fuses RGB and thermal infrared features from multiple layers during downsampling.This strategy avoids relying only on shallow or deep fusion to exploit multimodal information.

D. Pyramid Pooling Module

The architecture augments the top-down pathway with pyramid pooling and global guidance to preserve context while integrating features across scales.

  • The top-down pathway can dilute higher-level features, while deeper CNN layers provide insufficient global receptive field for image context.
  • The Pyramid Pooling Module uses global average pooling, identity mapping, and adaptive pooling branches with 3×3 and 5×5 outputs.
  • Global guidance information from the pyramid pooling module is integrated with feature maps at different levels of the top-down pathway.

E. Feature Aggregation Module

The framework aggregates multi-scale features through feature aggregation modules and trains saliency and boundary predictions with complementary losses.

  • Feature Aggregation Module: The network extracts pyramid features F = {F2, F3, F4, F5} through five sequential VGG16 convolution blocks.A global guidance flow passes global guidance information to features at different pyramid levels.
  • Feature Aggregation Module: Feature aggregation modules convert inputs to multiple scales through average pooling, then merge branches using upsampling and 3×3 convolution.The convolution reduces aliasing effects caused by upsampling.
  • Cross Entropy Loss: Cross entropy loss measures the error between the final saliency map and the ground-truth salient-object map.Y denotes ground truth, P the network’s saliency-map output, and N the number of image pixels.
  • Edge Loss: The method adds edge supervision by applying a Laplace operator to ground-truth and predicted saliency maps and comparing their boundaries with cross entropy.This strategy is designed to sharpen salient-object boundaries.

V. EXPERIMENTS

Experiments compare the proposed method with ten state-of-the-art baselines across VT5000 and additional RGBT datasets using saliency-quality metrics. The setup includes specified training procedures and challenge-level F-measure comparisons.

  • Experiment Design: Experiments conduct ablation studies and compare the proposed method with state-of-the-art methods.The evaluation uses a comparison platform involving ten baseline methods.
  • Baseline Methods: The ten comparison methods include PoolNet, RAS, BASNet, CPD, R3Net, PFA, PiCANet, EGNet, MTMR, and SGDL.These methods receive RGB and thermal images as input for RGBT salient object detection.
  • Datasets and Implementation: The training uses 2500 VT5000 image pairs, while the remaining VT5000 pairs, VT821, and VT1000 form the test sets.The network is trained for 25 epochs with Adam, weight decay 5e-4, and a learning-rate reduction from 1e-4 to 1e-5 after epoch 20.
  • Evaluation Metrics: Performance is evaluated with PR curves, F-measure, and MAE.PR curves compare thresholded saliency maps with ground truth; F-measure combines precision and recall, while MAE measures average pixel-level difference.
  • Challenge Analysis: Table III reports F-measure values for the proposed method and ten comparison methods across each challenge.The table is specifically organized around challenge-level performance.

B. Comparison with State-of-the-Art Methods

The proposed method is evaluated against eight deep learning-based and two traditional methods on three RGBT datasets and across VT5000’s diverse challenges. It achieves the strongest reported performance, with gains over PoolNet and robust qualitative results in difficult scenes.

  • Quantitative Comparisons: The benchmark tests eight deep learning-based and two traditional state-of-the-art methods on RGBT data using the same evaluation code without post-processing.The evaluated methods include PoolNet, RAS, BASNet, CPD, R3Net, PFA, PiCANet, EGNet, MTMR, and SCGL.
  • Challenge-sensitive performance: 10.8% and 14.8%: F-measure gains over PoolNet on the SA and SSO challenges, respectively.The passage links thermal infrared information to distinguishing objects from backgrounds with similar RGB appearance, while global guidance helps locate small objects.
  • Quantitative Comparisons: 5.5%-9.9%: the proposed model’s F-measure gain over PoolNet across VT821, VT1000, and VT5000.The comparison attributes the improvement to the synergy of the thermal infrared branch.
  • Quantitative Comparisons: 7.5% higher F-measure and 1.8% lower MAE than PiCANet on VT5000.On VT1000, the corresponding differences are 5.9% higher F-measure and 2.6% lower MAE; these are separate dataset-specific comparisons.
  • PR Curves: The proposed method’s precision-recall curve is particularly prominent, retaining higher accuracy than compared methods when recall approaches 1.The passage interprets this as a higher truth-positive rate in the saliency maps.
  • Visual Comparison: Across varied scenes including low illumination, cluttered backgrounds, small objects, and similar appearance, the method highlights salient objects, suppresses backgrounds, and produces well-defined contours.The qualitative comparison covers eleven challenge types and reports the best results in the shown challenging scenes.

VI. ABLATION ANALYSIS

The ablation analysis examines component effectiveness, module time costs, and transfer of RGBD methods to RGBT salient object detection. Attention-based multimodal aggregation performs strongly without thermal-specific architectural design, while added modules incur modest reported time costs.

  • Component Ablation: The ablation study on VT5000 evaluates the effects of CBAM and edge loss using max F-measure and MAE on the test set.The basic network is compared with variants that add these components.
  • Time Cost: 0.095, 0.079, and 0.051 second per sample: the reported training-time increases for CBAM, PPM, and FAM, respectively.These measurements report the training time of one epoch containing 2500 samples.
  • Time Cost: 0.061, 0.042, and 0.025 second per sample: the reported testing-time increases for CBAM, PPM, and FAM, respectively.Testing time is measured over VT5000’s 2500-sample test set.
  • RGBD Methods on RGBT: Five advanced RGBD salient object detection methods are applied to VT5000 because RGBD and RGBT tasks both use two modalities.The comparison evaluates max F-measure and MAE on the RGBT dataset.
  • RGBD Methods on RGBT: The proposed ADFNet achieves optimal or suboptimal results among the RGBD methods on RGBT datasets without thermal-image-specific design.The architecture uses attention for multimodal information aggregation, while some methods designed for depth maps perform below average.

B. The performance of our method on RGBD SOD datasets

The method is evaluated on RGBD salient object detection datasets against advanced RGBD methods, achieving similar performance and competitive results. The paper also identifies broader opportunities and constraints for multimodal salient object detection, including annotation cost and alignment requirements.

  • RGBD evaluation: The evaluation compares ADFNet with seven advanced RGBD salient object detection methods on DES, LFSD, SSD, and STERE test datasets.The compared methods are PCF, CTMF, MMCI, AFNet, TANet, D3Net, and S2MA.
  • RGBD evaluation: On three test datasets, ADFNet achieves similar performance to state-of-the-art RGBD methods, indicating competitive applicability to other multimodal SOD tasks.The comparison uses maximum F-measure and MAE scores reported in Table VII.
  • Multimodal implications: Thermal imagery provides complementary information for salient object detection in cluttered backgrounds and low illumination, while thermal crossover makes thermal data unreliable.Visible-spectrum imaging is not influenced by temperature in the thermal-crossover case.
  • Limitations and directions: Pixel-level annotation requirements create unacceptable manual costs, motivating unsupervised, weakly supervised, and alignment-free RGBT SOD research directions.The paper also notes that existing datasets can contain misaligned image pairs and that most RGBT platforms are non-aligned.
Loading 2007.03262v6…