Source-linked AI summary
Hierarchical Spatial-aware Siamese Network for Thermal Infrared Object Tracking
Xin Li, Qiao Liu, Nana Fan, Zhenyu He, Hongzhi Wang
TL;DR
TIR trackers based on classification are not well coupled to the tracking objective, which centers on locating the same object. HSSNet uses a spatial-aware Siamese network with hierarchical features, pretrained on visible video for similarity verification. Experiments report favorable performance against state-of-the-art methods on VOT-TIR 2015 and 2016.
Problem
Classification-based TIR tracking does not couple label prediction to location estimation, despite tracking's focus on within-class differences of the same object.
Source: §1 · p. 2Method
HSSNet fuses hierarchical Siamese features, adds a spatial-aware network, and learns paired-object similarity on visible video before transfer to TIR.
Source: §1 · p. 2Results
HSSNet achieves favorable performance against state-of-the-art methods on VOT-TIR 2015 and VOT-TIR 2016.
Source: §5 · p. 14Takeaways & Limitations
The proposed spatial-aware network is described as good at handling dynamic changes of the target object.
Source: §5 · p. 14When two TIR objects have very similar appearances, the network is hard to distinguish them.
Source: §4.4 · p. 14
Abstract
from arXiv · showhide
Most thermal infrared (TIR) tracking methods are discriminative, treating the tracking problem as a classification task. However, the objective of the classifier (label prediction) is not coupled to the objective of the tracker (location estimation). The classification task focuses on the between-class difference of the arbitrary objects, while the tracking task mainly deals with the within-class difference of the same objects. In this paper, we cast the TIR tracking problem as a similarity verification task, which is coupled well to the objective of the tracking task. We propose a TIR tracker via a Hierarchical Spatial-aware Siamese Convolutional Neural Network (CNN), named HSSNet. To obtain both spatial and semantic features of the TIR object, we design a Siamese CNN that coalesces the multiple hierarchical convolutional layers. Then, we propose a spatial-aware network to enhance the discriminative ability of the coalesced hierarchical feature. Subsequently, we train this network end to end on a large visible video detection dataset to learn the similarity between paired objects before we transfer the network into the TIR domain. Next, this pre-trained Siamese network is used to evaluate the similarity between the target template and target candidates. Finally, we locate the candidate that is most similar to the tracked target. Extensive experimental results on the benchmarks VOT-TIR 2015 and VOT-TIR 2016 show that our proposed method achieves favourable performance compared to the state-of-the-art methods.
1. Introduction
TIR tracking remains challenged by weak image features, complex backgrounds, and appearance changes, while classification-based CNN features are not tailored to location estimation. HSSNet uses hierarchical Siamese features and spatial awareness to verify target similarity, with favorable benchmark comparisons reported.
1. Introduction
- TIR tracking is challenged by low-resolution, blurry, colorless objects, complicated backgrounds, and deformation, occlusion, and scale variation.These conditions make discriminative feature extraction difficult for existing hand-crafted methods.Source: §1 · p. 2
- Classification-based CNN features are not explicitly coupled to tracking objectives, often use one layer, and have limited TIR training data.The paper contrasts label prediction with tracking's location estimation and within-class matching needs.Source: §1 · p. 2
- HSSNet fuses deep and shallow Siamese features for semantic and spatial information, then applies a spatial-aware network to strengthen discrimination.It is trained on a large visible video dataset for paired-object similarity before transfer to TIR, where it compares a target template with candidates.Source: §1 · pp. 2–3
- HSSNet is reported to perform favorably against state-of-the-art TIR trackers on benchmark datasets.Source: §1 · p. 3
2. Related Work
The related work contrasts classification-based TIR trackers with verification-based Siamese trackers. It argues that similarity verification better matches tracking and motivates hierarchical features plus spatial awareness for TIR.
Classification-based trackers
- Classification-based TIR trackers span sparse representation, multiple-instance learning, correlation filters, and deep learning, among other approaches.Source: §2 · p. 3
- The paper argues classification is a poor fit because label prediction emphasizes between-class differences, whereas tracking estimates position and focuses on within-class differences.Source: §2 · p. 4
Verification-based trackers
- Verification-based Siamese trackers match templates to candidates, but many use a single convolutional layer that the paper says is not robust for TIR.Examples discussed include YCNN, SINT, Siamese-fc, and CFNet.Source: §2 · p. 4
- HSSNet addresses this TIR-specific concern by combining hierarchical convolutional features with a spatial-aware network.Source: §2 · p. 4
3. Hierarchical Spatial-aware Siamese Network
HSSNet couples a Siamese similarity-verification pipeline with hierarchical feature fusion and spatial-aware processing. It learns from visible video detections, then ranks target candidates by similarity.
3.1. Algorithmic Overview
- The Siamese network compares a target template with candidates and selects the candidate at the response-map maximum as the tracked target.The response map represents candidate-to-template similarity, and the selected coordinates are mapped to the original frame.Source: §3.1 · p. 5, Figure 1 · p. 5
3.2. The Network Architecture
- Hierarchical fusion pools and batch-normalizes Conv3, Conv4, and Conv5 features before concatenating them, combining shallow spatial cues with deep semantic information.A 1 × 1 convolution reduces the fused map’s channel dimension from 1024 to 256.Source: §3.2 · pp. 6–7
- The spatial-aware network cascades a spatial transformer and channel-attention network to address spatial variation and unequal channel importance.The transformer learns a 2D affine transformation covering rotation, scaling, and translation; channel attention reweights feature channels.Source: §3.2 · pp. 7–8, Figure 2 · p. 7
- The correlation-filter layer is differentiable for end-to-end training and can update the target template to adapt to appearance variation.The filter computes a template from the feature map by solving a ridge-regression problem in the Fourier domain.Source: §3.1 · p. 5, §3.2 · p. 8
3.3. Training the Network
- Training uses ILSVRC2015, with more than 4,000 videos and 2 million labeled boxes, to learn similarity before transfer to the TIR domain.Pairs comprise a target crop and a centered search region sampled from frames within an interval of T frames in the same video.Source: §3.3 · p. 8
- A logistic loss trains the network using ground-truth labels of +1 or −1 for target–candidate pairs, with the response-map loss averaging individual losses.The network parameters are obtained by minimizing this loss using stochastic gradient descent.Source: §3.3 · p. 9
3.4. Tracking Interface
- At tracking time, the learned similarity function evaluates candidates without adaptation, and the highest-scoring candidate is selected.Three search-region scales are compared, and the maximum-response map and its corresponding scale are returned.Source: §3.4 · p. 9
4. Experiments
The experiments combine component ablations on VOT-TIR 2015 with state-of-the-art comparisons on VOT-TIR 2016. HSSNet leads the reported 2016 EAO comparison, while its component gains and metric-specific rankings show a more mixed profile.
4.1. Implementation Details
- The implementation trains on ILSVRC2015 for 40 epochs, searches three fixed scales during tracking, and runs at an average 10 fps.The tracking algorithm crops three scale-specific search regions and returns the best scale factor.Source: §4.1 · p. 10
4.2. Evaluation Criteria
- Evaluation reports accuracy and robustness, with robustness counting failures, and combines both measures into EAO for overall performance.The paper also visualizes tracker results with A-R raw and ranking plots and EAO curves and scores.Source: §4.2 · p. 10
4.4. Comparisons with the State-of-the-Art on VOT-TIR 2016
- VOT-TIR 2015 has 20 sequences with local attributes, while 2016 adds harder sequences; the 2016 comparison tests 14 trackers across four categories.The categories are CNN-based, correlation-filter-based, part-based, and fusion-based trackers.Source: §4.3 · p. 11, §4.4 · pp. 11–12
4.3. Ablation Experiments on VOT-TIR 2015
- On VOT-TIR 2015, hierarchical fusion raises accuracy about 2% over HSNet-lastlayer; the spatial transformer adds about 4%, channel attention another 2%, and scale estimation helps accuracy without changing robustness.The ablation results are shown in the A-R ranking and raw plots.Source: §4.3 · p. 11, Figure 3 · p. 12
4.4. Comparisons with the State-of-the-Art on VOT-TIR 2016
- HSSNet ranks second in robustness and fourth in accuracy; across dynamics, motion, and camera changes it has best robustness, and it has best accuracy under occlusion.CNN-based trackers generally outperform conventional hand-crafted-feature trackers in these TIR comparisons, although HSSNet does not lead every metric.Source: §4.4 · pp. 12–13, Figure 5 · p. 14
- HSSNet scores 0.2619 EAO on VOT-TIR 2016, about 1% above CFNet and 3% above DSST.The EAO curve and score identify the overall comparison; the paper reports HSSNet as the best performer.Source: §4.4 · p. 12, Figure 4 · p. 13
- In visual examples, HSSNet tracks through drastic appearance change and occlusion, but can drift amid heavy background noise or very similar objects.The authors attribute this boundary partly to visible-image training not covering all TIR situations and propose a large-scale TIR dataset for fine-tuning.Source: §4.4 · pp. 13–14
5. Conclusion
HSSNet casts TIR tracking as similarity verification and combines hierarchical Siamese features with a spatial-aware network. The authors report favorable performance against state-of-the-art methods.
5. Conclusion
- HSSNet combines hierarchical convolutional layers in a Siamese CNN with a spatial-aware network to enrich spatial and semantic features and enhance discrimination.The method casts tracking as similarity verification.Source: §5 · p. 14
- The authors report that HSSNet achieves favorable performance against state-of-the-art methods.Source: §5 · p. 14
- Figure 6 compares baseline experiments across VOT-TIR 2016 attribute subsets using A-R ranking and raw plots; better trackers appear nearer the top-right.Source: Figure 6 · p. 15
- Figure 7 presents visual tracking results for several state-of-the-art trackers on representative challenging TIR sequences.Source: Figure 7 · p. 16