Source-linked AI summary
Unsupervised Deep Tracking
Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, Houqiang Li
TL;DR
The paper addresses whether visual tracking can be learned from unlabeled videos rather than extensive ground-truth annotations. It trains a Siamese correlation filter network with forward-backward consistency, multiple-frame validation, and cost-sensitive loss. The resulting unsupervised tracker performs comparably to classic fully supervised trackers and can benefit from additional unlabeled or weakly labeled data.
Problem
Visual tracking methods commonly require large quantities of expensive, time-consuming ground-truth annotations, motivating the use of readily available unlabeled videos.
Method
The method integrates forward-backward tracking into a Siamese correlation filter network and improves training with multiple-frame validation and a cost-sensitive loss.
Results
Without bells and whistles, the unsupervised tracker achieves comparable performance with classic fully supervised trackers on standard benchmarks.
Takeaways & Limitations
The framework shows potential for exploiting more unlabeled or weakly labeled data and for further improving tracking accuracy.
Takeaways & Limitations
The approach may lack objectness information in complex scenarios, and its forward-backward tracking introduces additional computational load.
Abstract
from arXiv · showhide
We propose an unsupervised visual tracking method in this paper. Different from existing approaches using extensive annotated data for supervised learning, our CNN model is trained on large-scale unlabeled videos in an unsupervised manner. Our motivation is that a robust tracker should be effective in both the forward and backward predictions (i.e., the tracker can forward localize the target object in successive frames and backtrace to its initial position in the first frame). We build our framework on a Siamese correlation filter network, which is trained using unlabeled raw videos. Meanwhile, we propose a multiple-frame validation method and a cost-sensitive loss to facilitate unsupervised learning. Without bells and whistles, the proposed unsupervised tracker achieves the baseline accuracy of fully supervised trackers, which require complete and accurate labels during training. Furthermore, unsupervised framework exhibits a potential in leveraging unlabeled or weakly labeled data to further improve the tracking accuracy.
1. Introduction
The paper investigates learning visual trackers from unlabeled videos instead of expensive frame-level annotations. It uses forward-backward consistency within a Siamese correlation filter framework, adding validation and loss design to improve unsupervised training.
- Manual frame-level annotations are expensive and time-consuming, while extensive unlabeled videos are readily available for visual tracking.
- Forward tracking predicts later locations, while backward tracking returns from the final prediction toward the initial frame to measure trajectory consistency.
- The method initializes training with randomly cropped bounding boxes, allowing it to track subregions rather than requiring complete objects.
- Multiple-frame validation reduces verification failures, while cost-sensitive loss mitigates interference from noisy samples and occlusions.
- Without bells and whistles, the unsupervised tracker achieves comparable performance with baseline fully supervised trackers on benchmark datasets.
- The contributions combine a Siamese correlation filter backbone, forward-backward learning, multiple-frame validation, and cost-sensitive loss.
2. Related Work
The paper situates unsupervised deep tracking within deep tracking, forward-backward trajectory analysis, and unsupervised representation learning. It distinguishes its framework by integrating tracking into unsupervised training and coupling learned representations with a tracking objective.
- The related work reviews deep tracking methods, forward-backward trajectory analysis, and unsupervised representation learning.
- Siamese trackers formulate visual tracking as a similarity matching process.
- Forward and backward tracking produces a consistency loss for network training, while online tracking uses only forward prediction.
- The proposed framework integrates the tracking algorithm into unsupervised training rather than using an off-the-shelf tracker only for data preprocessing.
- The unsupervised framework couples learned feature representation with a tracking objective so it can represent generic target objects effectively.
3. Proposed Method
The proposed method trains a Siamese correlation filter tracker without ground-truth annotations by enforcing forward–backward tracking consistency on unlabeled videos. Multiple-frame validation and cost-sensitive weighting address verification failures and noisy training samples.
- Forward-backward tracking: Forward tracking predicts a target location in a later frame, while backward tracking uses that prediction as a pseudo-label to return toward the initial frame.The method switches the roles of search and template patches during backward tracking.
- Correlation filter tracking: The Siamese correlation filter framework extracts shared CNN features, learns a target template from one patch, and produces a response map for a search patch.The template and search branches form an end-to-end tracking framework tied to the tracking objective.
- Unsupervised learning: The consistency loss compares the backward response map with the original template label, enabling network training without ground-truth labels.The loss is back-propagated to update the representation network.
- Multiple Frames Validation: Multiple-frame validation amplifies localization errors across successive frames, making inconsistent forward–backward predictions more likely to be penalized.The implementation uses three frames and an additional backward response map.
- Cost-sensitive Loss: Cost-sensitive loss reduces the influence of noisy training pairs and emphasizes pairs containing larger target motion.The method excludes the 10% of training pairs with the highest loss and assigns motion weights to training pairs.
- Unsupervised training data: Training pairs use one template patch and two subsequent search patches cropped from unlabeled video frames, with randomly initialized boxes that may include background context.Center regions are cropped and resized, and the tracked content need not correspond to a specified object category.
4. Experiments
Experiments evaluate UDT through configuration studies and benchmark comparisons on OTB-2015, Temple-Color, and VOT2016. The results show favorable performance against supervised baselines, while also identifying weaknesses under difficult appearance variations.
- Experimental Protocol: Experiments use OPE with DP at 20 pixels and AUC on OTB-2015 and Temple-Color, and EAO on VOT2016.These metrics define the evaluation protocol across the three benchmarks.
- Ablation Study: Multiple-frame validation and cost-sensitive loss improve UDT accuracy over configurations without these components.The ablation compares hard-sample reweighting and single-trajectory training against the full configuration.
- Ablation Study: 0.9% DP and 0.7% AUC gains result when UDT is trained with additional unlabeled videos from OxUvA.The added data comprise 337 raw videos from the OxUvA benchmark, a subset of Youtube-BB.
- State-of-the-art Comparison: UDT is comparable with fully supervised SiamFC and CFNet on OTB-2015 and exceeds the DCF-based DSST tracker by a large margin.UDT+ is reported as comparable with recent state-of-the-art supervised trackers in the OTB-2015 AUC comparison.
- State-of-the-art Comparison: UDT performs favorably against SiamFC and CFNet on Temple-Color, while UDT+ performs favorably against supervised trackers on VOT2016.VOT2016 ranking uses Expected Average Overlap, alongside Accuracy and Failures in the comparison table.
- Attribute Analysis: UDT outperforms SiamFC and CFNet on most OTB-2015 challenge attributes but is weaker under illumination variation, occlusion, and fast motion.The paper attributes these weaknesses to significant target appearance changes and insufficient objectness information from unsupervised representation learning.
5. Conclusion
The paper demonstrates that visual trackers can be trained from unlabeled video using an unsupervised Siamese correlation filter network. Multiple-frame validation and cost-sensitive loss support this pipeline, which performs comparably to classic fully supervised trackers and may benefit from additional unlabeled or weakly labeled data.
- The proposed framework trains a visual tracker from unlabeled video sequences using an unsupervised Siamese correlation filter network.
- Multiple-frame validation and a cost-sensitive loss are introduced to facilitate the unsupervised training process.
- Without additional enhancements, the unsupervised tracker performs comparably to classic fully supervised trackers.
- The framework shows potential for using more unlabeled or weakly labeled data to further improve tracking accuracy.