RGB-T Image Saliency Detection via Collaborative Graph Learning

Zhengzheng Tu; Tian Xia; Chenglong Li; Xiaoxiao Wang; Yan Ma; Jin Tang

Source-linked AI summary

RGB-T Image Saliency Detection via Collaborative Graph Learning

Zhengzheng Tu, Tian Xia, Chenglong Li, Xiaoxiao Wang, Yan Ma, Jin Tang

arXiv:1905.06741v1May 16, 2019cs.CV

TL;DR

RGB-T saliency detection needs methods that address limitations in handcraft features, fixed graph structures, and independently computed saliency. The paper uses collaborative graph learning over superpixels with hierarchical deep features and introduces a 1000-pair aligned dataset. Experiments on public and new datasets demonstrate effectiveness, while failure cases remain when salient objects cross image boundaries or resemble the background in both modalities.

Problem
Existing RGB-T saliency methods are limited by handcraft features, fixed local graphs, and separate graph construction and saliency computation.
Method
The method jointly learns graph structure, edge and node weights, modality weights, and feature weights using hierarchical deep features in a unified optimization framework.
Results
The approach demonstrates effectiveness on both public and newly created RGB-T saliency datasets.
Takeaways & Limitations
The paper provides a collaborative graph learning approach and a comprehensive benchmark dataset for RGB-T saliency detection.
Takeaways & Limitations
The method struggles to preserve good contours when objects cross image boundaries or resemble the background in both modalities.

Abstract

from arXiv · show

Image saliency detection is an active research topic in the community of computer vision and multimedia. Fusing complementary RGB and thermal infrared data has been proven to be effective for image saliency detection. In this paper, we propose an effective approach for RGB-T image saliency detection. Our approach relies on a novel collaborative graph learning algorithm. In particular, we take superpixels as graph nodes, and collaboratively use hierarchical deep features to jointly learn graph affinity and node saliency in a unified optimization framework. Moreover, we contribute a more challenging dataset for the purpose of RGB-T image saliency detection, which contains 1000 spatially aligned RGB-T image pairs and their ground truth annotations. Extensive experiments on the public dataset and the newly created dataset suggest that the proposed approach performs favorably against the state-of-the-art RGB-T saliency detection methods.

I. INTRODUCTION

RGB-T saliency detection addresses challenges in adverse environments by combining complementary visible and thermal information. The paper formulates the task as collaborative graph learning and introduces a highly aligned, diverse 1000-pair benchmark dataset.

RGB-T saliency detection remains relatively new, with limited existing methods and benchmark resources.
Existing methods are limited by handcraft features, fixed local graph structures, and separate graph construction and saliency computation.
The approach jointly learns graph structure, edge weights, and node saliency in one unified optimization framework using multi-level deep features from both modalities.
The dataset uses highly aligned RGB and thermal cameras, avoiding blank boundaries and restricting transformations to translation and scale.
Experiments on public and newly created datasets demonstrate the effectiveness of the proposed graph learning approach.
The authors create a public benchmark containing 1000 aligned RGB-thermal image pairs with ground-truth annotations.

A. RGB Image Saliency Detection

RGB saliency research includes bottom-up models based on hand-designed visual cues and top-down models increasingly based on deep networks. Related RGB-T work extends graph, ranking, decomposition, and tracking methods to combine complementary modalities and adaptive reliability.

Bottom-up saliency models use image cues such as color, orientation, texture, boundary, and contrast.
Top-down saliency models learn salient-object detectors, with recent approaches largely based on deep learning networks.
RGB-T research applies collaborative decomposition, sparse representation, patch-based graphs, and cross-modal ranking to related vision tasks.
RGB data can provide more robust saliency estimates than grayscale data when color distinguishes objects with similar intensities.
These related methods adaptively incorporate modality information or reliability in multimodal processing.

III. COLLABORATIVE GRAPH LEARNING ALGORITHM

The algorithm represents RGB-T superpixels as graph nodes and learns graph structure, affinities, modality and feature weights, and saliency jointly. It adaptively fuses hierarchical features and modalities in a unified graph-learning framework.

SLIC generates n non-overlapping superpixels from paired RGB and thermal images, using thermal data as an image channel for segmentation consistency.
The fixed graph connects spatially adjacent superpixels, but its local structure can miss intrinsic relationships among nodes.
The method learns an affinity matrix W whose entries measure the possibility that one node is the true neighbor of another.
CIE-LAB color and multi-layer FCN-32S deep features are extracted from RGB and thermal superpixels to form multiple modality- and layer-specific graphs.
Modality weights α and feature weights β adaptively fuse graph affinities according to imaging quality and feature reliability.
Graph structure, edge affinities, and superpixel saliency values are jointly optimized rather than learned in separate stages.

B. Model Optimization

The model is optimized by alternating updates of the affinity matrix, feature weights, modality weights, and saliency vector. Each subproblem is convex, while the overall algorithm has cubic complexity per iteration in the number of nodes.

The objective uses graph Laplacians from different modalities and features, with α and β constrained as normalized nonnegative weights.
Algorithm 1 alternates updates of W, β, α, and s until variable changes fall below ε or the maximum iteration count is reached.The stated initialization is α(m) = 1/M, β(m,k) = 1/K, with ε = 10^-4 and maxIter = 50.
The W subproblem is reformulated in matrix form using S, whose ij-th element is (s_i − s_j)^2.
β and α are updated to estimate the importance of multi-layer features and modalities, respectively.
Because each subproblem is convex, the alternating solution satisfies the Nash equilibrium conditions.
Each subproblem has complexity O(n^3), giving overall optimization complexity O(Tn^3) for T iterations.

IV. TWO-STAGE RGB-T SALIENCY DETECTION

The method performs two-stage RGB-T saliency detection by using boundary and foreground queries within unified graph learning and saliency inference.

Boundary superpixels serve as background queries, while high-confidence foreground superpixels are selected from low ranking scores across modalities.
The approach constructs four saliency maps from top, bottom, left, and right boundary queries before combining them into a background-query saliency map.
Graph learning and saliency inference are performed in a unified framework for each boundary-query ranking vector.
An adaptive threshold applied to the background-query map generates foreground queries for a second graph-learning and saliency-inference stage.
The final saliency map is normalized to the range from 0 to 1 after inference with foreground queries.

V. VT1000 DATASET

The paper introduces VT1000, a dataset of 1000 visible–thermal image pairs captured in diverse scenes for RGB-T saliency detection research.

VT1000 contains 1000 image pairs consisting of visible images and corresponding thermal maps from diverse scenes.
The dataset is presented with statistical analysis, including a table listing its annotated challenges.

A. Platform

The imaging platform uses a combined thermal and CCD camera system designed to produce highly aligned RGB-T image pairs.

The FLIR SC620 contains a thermal infrared camera and a CCD camera in one imaging platform.
The cameras share imaging parameters except focus and have parallel optical axes, supporting manual alignment by enlarging and cropping the visible image.
The resulting visible and thermal images are cropped to totally overlap, producing highly aligned RGB-T pairs.

B. Annotation

VT1000 is constructed and annotated to support challenging, attribute-sensitive RGB-T saliency evaluation, with aligned image pairs spanning varied scenes and conditions.

Annotation: Approximately 2000 natural RGB-T pairs were captured, 1500 were manually selected, and the 1000 most consistently labeled pairs were retained.
Annotation: Six participants identified the first-glance salient object, while four participants cropped overlapping RGB regions and segmented salient objects.
Statistics: The dataset covers varied illumination, object categories, sizes, and numbers, and annotates ten challenges for challenge-sensitive evaluation.
Advantages over existing Datasets: Compared with earlier datasets, VT1000 addresses limited image numbers and coarse annotations while emphasizing more complex and cluttered scenes.
Advantages over existing Datasets: VT1000 includes more than 400 common object types collected across ten indoor and outdoor scene categories, with single- and multiple-object images.
Advantages over existing Datasets: Its imaging platform enables highly aligned modalities without blank boundary noise because the transformation is limited to translation and scale.
Statistics: Thermal crossover is common because images were collected in summer, motivating ten attribute categories for occlusion-sensitive evaluation.

VI. EXPERIMENTS

The experiments evaluate the proposed method on VT1000 and VT821 using precision-recall curves and F-measure, comparing it with traditional, RGB-T, and deep-learning baselines. Parameter settings and component analyses support controlled evaluation.

The method is evaluated on the public VT821 dataset and newly created VT1000 dataset against state-of-the-art methods.
Evaluation criteria: Performance is assessed using precision-recall curves and F-measure, with precision and recall computed from thresholded saliency maps.Thresholds range from 0 to 255, and F-measure combines precision and recall with β^2 = 0.3.
Baseline methods: The comparison includes eight baselines spanning traditional, RGB-T, and deep-learning saliency detection methods.The baselines are MR, RBD, CA, RRWR, MILPS, FCNN, DSS, and MTMR; extended RGB-T variants concatenate RGB and thermal features.
Parameter settings: Experiments fix the graph to 300 superpixels and use σ_RGB=20 and σ_T=40 for graph construction.
Parameter settings: The model uses fixed parameters {γ1, γ2, θ, µ, λ1} = {0.5, 8, 0.0001, 0.001, 0.004}, while γ1 and γ2 control modality and affinity-graph weighting.The reported parameter variations do not substantially affect performance, as examined in Fig. 11.

B. Evaluation on the VT1000 Dataset

On VT1000, the proposed collaborative graph method outperforms the compared RGB, RGB-T, and deep-learning approaches overall, while performance varies across challenge attributes. Its graph-based fusion also avoids large training and pretraining requirements.

The proposed method achieves a 5.4% F-measure gain over MTMR and outperforms other extended RGB-T methods on VT1000.Its precision-recall curve exceeds competing methods with a clear margin, while visual results show salient regions and well-defined contours.
Comparison with traditional RGB methods: RGB-T input improves F-measure over RGB-only baselines by 9.2% against MR and 6.4% against MILPS.The comparison supports the complementary use of RGB and thermal information in the evaluated challenging scenarios.
Comparison with deep learning methods: The method outperforms FCNN by 5.6% and DSS by 9.3% in F-measure on VT1000.Although FCNN has higher recall, the proposed method has higher precision and F-measure; the paper attributes this partly to limited RGB-T training data.
Comparison with deep learning methods: The graph-based approach does not require laborious pretraining or a large training set and has closed-form solutions for its subproblems.It is also reported to perform favorably against FCNN and DSS in efficiency on cheaper hardware.
Challenge-sensitive performance: On VT1000 attributes, the method outperforms other RGB-T methods on most challenges except BSO and LI.It ranks fourth on BSO, 4.2% below RBD, and third on LI, 3.2% below FCNN.

C. Evaluation on the VT821 Dataset

On VT821, the proposed method again outperforms the compared RGB-T methods and performs strongly across most challenge attributes. Ablation analyses examine feature, modality, and weighting contributions.

The proposed method achieves a 6.4% F-measure gain over MTMR and exceeds FCNN and DSS by 9.1% and 10.5%, respectively, on VT821.
On VT821 challenge attributes, the method outperforms other RGB-T methods on most subsets except BSO and CIB.It ranks second on both exceptions, while RBD and FCNN obtain the best results on BSO and CIB, respectively.
Feature Analysis: Removing high-level, low-level, or handcrafted color features reduces performance, with high-level deep features identified as most important.The paper associates high-level features with object semantics and distinguishing objects from background.
Feature Analysis: Removing feature weights lowers performance, supporting their role in adaptively incorporating information from different features.
Modality Analysis: Modality analysis evaluates removing RGB, removing thermal information, and removing modality weights to test their contributions.The results report that modality weights support adaptive incorporation of information from the two modalities.

E. Runtime Comparison

On VT821, the proposed algorithm requires 2.23 seconds per 480 × 640 RGB-T image pair, excluding deep-feature extraction, and is not very fast because two subproblems require matrix inversion.

2.23s is the proposed algorithm’s average runtime for a 480 × 640 RGB-T image pair on VT821, excluding deep-feature extraction.
Matrix inversion in the W-subproblem and S-subproblem contributes to the method’s relatively slow runtime.
The authors plan to adopt a linearized operation to avoid matrix inversion and improve efficiency.