Source-linked AI summary
Learning Deep Multi-Level Similarity for Thermal Infrared Object Tracking
Qiao Liu, Xin Li, Zhenyu He, Nana Fan, Di Yuan, Hongpeng Wang
TL;DR
Deep TIR trackers often rely on semantic features and RGB-trained representations that are insufficient for distinguishing similar distractors. The paper introduces MLSSNet, which combines structural and semantic similarities with adaptive relative-entropy fusion and trains it using a large-scale annotated TIR dataset. Experiments on VOT-TIR2015 and VOT-TIR2017 report favorable performance against state-of-the-art methods.
Problem
Existing deep TIR trackers mainly use semantic features, limiting discriminative capacity for similar distractors, especially when feature extractors are trained only on RGB images.
Method
MLSSNet uses Siamese structural and semantic correlation networks plus a relative entropy-based adaptive ensemble network to combine their similarities.
Results
MLSSNet performs favorably against state-of-the-art methods on the VOT-TIR2015 and VOT-TIR2017 benchmarks.
Takeaways & Limitations
A large-scale manually annotated TIR dataset supports training for TIR tracking and several other TIR vision tasks.
Abstract
from arXiv · showhide
Existing deep Thermal InfraRed (TIR) trackers only use semantic features to describe the TIR object, which lack the sufficient discriminative capacity for handling distractors. This becomes worse when the feature extraction network is only trained on RGB images.To address this issue, we propose a multi-level similarity model under a Siamese framework for robust TIR object tracking. Specifically, we compute different pattern similarities on two convolutional layers using the proposed multi-level similarity network. One of them focuses on the global semantic similarity and the other computes the local structural similarity of the TIR object. These two similarities complement each other and hence enhance the discriminative capacity of the network for handling distractors. In addition, we design a simple while effective relative entropy based ensemble subnetwork to integrate the semantic and structural similarities. This subnetwork can adaptive learn the weights of the semantic and structural similarities at the training stage. To further enhance the discriminative capacity of the tracker, we construct the first large scale TIR video sequence dataset for training the proposed model. The proposed TIR dataset not only benefits the training for TIR tracking but also can be applied to numerous TIR vision tasks. Extensive experimental results on the VOT-TIR2015 and VOT-TIR2017 benchmarks demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.
1. Introduction
TIR tracking remains challenged by occlusion, appearance changes, and similar distractors, while many deep trackers rely mainly on semantic features. The paper proposes MLSSNet, a large-scale TIR-trained multi-level similarity tracker, and reports favorable benchmark performance.
- TIR tracking faces occlusion, appearance changes, and similar distractors.
- Existing CNN-based TIR trackers mostly use deep semantic features, although TIR objects lack color and rich texture information.
- MLSSNet computes structural and semantic similarities within a Siamese framework to improve discriminative capacity for handling distractors.
- The authors construct a large-scale manually annotated TIR image dataset for training deep networks across several TIR vision tasks.
- Extensive benchmark experiments report favorable performance against state-of-the-art TIR tracking methods.
2. Related Work
Prior TIR tracking work includes handcrafted, CNN-based, Siamese, and ensemble approaches, but the paper positions MLSSNet as using multiple feature levels and training-stage adaptive fusion.
- Siamese based trackers: Siamese trackers formulate object tracking as similarity verification between target and candidate samples.
- Siamese based trackers: Prior Siamese methods differ in branch training, optimization, and fusion, whereas MLSSNet uses end-to-end training and relative entropy-based adaptive fusion.
- CNN based ensemble trackers: CNN ensemble trackers combine multiple weak trackers or classifiers using adaptive weighting, sequential sampling, stochastic regularization, or tree structures.
- TIR training dataset: The lack of a large-scale TIR image training dataset has hindered CNN development for TIR object tracking.
3. Multi-Level Similarity Network
The proposed multi-level similarity network combines local structural and global semantic similarities in a Siamese framework, then adaptively integrates them for TIR tracking. It is trained with a newly constructed diverse TIR video dataset and used as a matching function during tracking.
- Network architecture: MLSSNet computes local structural and global semantic similarities through dedicated subnetworks on different convolutional layers.The structural CSN operates on shallow Conv3 features, while the semantic CSN operates on deeper Conv5 features.
- Structural CSN: The structural CSN enhances local object parts through learned weighting, supporting accurate localization when TIR objects lack color and texture.It captures local structure with convolutional and deconvolutional layers, generates a spatial weight map, and scales the original feature.
- Semantic CSN: The semantic CSN generates channel weights that emphasize discriminative regions in deep convolutional features.Its channel-reweighting design strengthens semantic feature representation before similarity computation.
- REN: REN integrates structural and semantic similarity maps through a KL-divergence-based adaptive ensemble subnetwork.The KL layer yields a weighted-sum integration whose convolution-filter parameters are learned adaptively.
- TIR training dataset: The training dataset contains 430 TIR videos, over 180,000 images, 20 classes, and over 200,000 annotated bounding boxes.Its varied sources, resolutions, scenes, and capture times provide diverse data distributions for training.
- Network training and tracking: Training mixes the TIR dataset with ILSVRC2015 images converted to grayscale and uses cropped positive and negative pairs with logistic mean loss.At inference, the trained model compares a target image with candidates in each search region without online updating.
4. Experiments
Experiments evaluate MLSSNet through ablations, benchmark comparisons, challenge-specific analyses, and qualitative tracking examples. Results show that multi-level similarity and TIR-data training improve robustness, localization, and overall EAO.
- Experimental setup: The experiments evaluate MLSSNet and its variants on VOT-TIR2015 and VOT-TIR2017, using Accuracy, Robustness, and EAO.The study also compares MLSSNet with fifteen state-of-the-art trackers.
- Ablation studies: Adding the semantic CSN improves baseline robustness on both benchmarks, while adding the structural CSN further improves accuracy and EAO on VOT-TIR2015.The structural CSN raises accuracy by about two percent and EAO by about one percent relative to Baseline Sem on VOT-TIR2015.
- Ablation studies: Training with the constructed TIR dataset remarkably boosts the accuracy and robustness of the multi-level model on both benchmarks.The comparison is between Baseline Sem+Str trained on ILSVRC2015 and MLSSNet trained on ILSVRC2015 plus the TIR dataset.
- Comparison with state-of-the-art methods: 0.278 and 0.316 are MLSSNet’s best EAO values on VOT-TIR2017 and VOT-TIR2015, respectively, among the compared trackers.The version without the proposed TIR dataset also achieves the best and second-best EAO values, 0.272 and 0.309, on the two benchmarks.
- Challenge analysis: On challenge-specific evaluations, MLSSNet performs best on dynamics change and empty for VOT-TIR2017 and achieves strong motion-change and size-change results across both benchmarks.TIR-data training adds about 3% EAO on dynamics change, while comparisons report about 5% and 3% gains over CFNet for motion change and about 18% and 5% gains over Siamese-FC for size change.
- Qualitative analysis: Qualitative sequences show more accurate localization during drastic scale change, severe appearance variation, distractors, and simultaneous appearance variation with distractors.The tracker remains more accurate than competing methods in Boat2, Car1, Street, Soccer, and Crouching.
5. Conclusion
The paper presents MLSSNet, a Siamese multi-level similarity model combining structural and semantic representations for robust TIR tracking. A large-scale TIR dataset supports training, and experiments on two benchmarks report favorable state-of-the-art performance.
- MLSSNet combines structural and semantic correlation similarity networks to capture local structure and global semantic representations.The structural network supports precise localization, while the semantic network supports robust identification.
- A relative entropy based adaptive ensemble network integrates the structural and semantic similarities.The ensemble network adaptively combines the two similarity signals within the Siamese framework.
- The authors construct a large-scale TIR image dataset for training and for broader TIR vision tasks such as classification and detection.
- Experiments on VOT-TIR2015 and VOT-TIR2017 show favorable performance against state-of-the-art methods.