Source-linked AI summary
Unsupervised Deep Representation Learning for Real-Time Tracking
Ning Wang, Wengang Zhou, Yibing Song, Chao Ma, Wei Liu, Houqiang Li
TL;DR
Visual tracking models typically rely on expensive labeled data, while many videos are unlabeled. The paper trains a Siamese correlation filter tracker from scratch using forward–backward trajectory consistency and training enhancements. It reports performance comparable to baseline fully supervised trackers, with real-time speed.
Problem
Deep tracking models typically require expensive ground-truth annotations, despite the availability of unlabeled videos.
Source: §1 · p. 1Method
A Siamese correlation filter tracker is trained from scratch on unlabeled videos using forward–backward trajectory consistency, multi-frame validation, and a cost-sensitive loss.
Source: §1 · p. 2Results
The unsupervised tracker is comparable to baseline fully supervised trackers across the reported benchmarks, and achieves real-time speed.
Source: §1 · p. 2Takeaways & Limitations
The framework shows potential to improve tracking accuracy by using more unlabeled or weakly labeled data.
Source: §1 · p. 2Under occlusion or drastic appearance change, the unsupervised tracker tends to drift; its bidirectional training also adds offline computational load.
Source: §4.5 · p. 16
Abstract
from arXiv · showhide
The advancement of visual tracking has continuously been brought by deep learning models. Typically, supervised learning is employed to train these models with expensive labeled data. In order to reduce the workload of manual annotations and learn to track arbitrary objects, we propose an unsupervised learning method for visual tracking. The motivation of our unsupervised learning is that a robust tracker should be effective in bidirectional tracking. Specifically, the tracker is able to forward localize a target object in successive frames and backtrace to its initial position in the first frame. Based on such a motivation, in the training process, we measure the consistency between forward and backward trajectories to learn a robust tracker from scratch merely using unlabeled videos. We build our framework on a Siamese correlation filter network, and propose a multi-frame validation scheme and a cost-sensitive loss to facilitate unsupervised learning. Without bells and whistles, the proposed unsupervised tracker achieves the baseline accuracy as classic fully supervised trackers while achieving a real-time speed. Furthermore, our unsupervised framework exhibits a potential in leveraging more unlabeled or weakly labeled data to further improve the tracking accuracy.
1 Introduction
The paper seeks to reduce reliance on expensive frame annotations by training visual trackers from unlabeled videos. It uses forward–backward trajectory consistency and additional training strategies, and reports performance comparable to supervised baselines.
1 Introduction
- The motivation is that deep trackers typically rely on expensive ground-truth labels, while unlabeled videos are readily available and tracking labels depend on the initial target selection.The paper frames manual annotation as expensive and time-consuming.Source: §1 · pp. 1–2
- Forward tracking followed by backward tracking measures trajectory consistency to train a tracker from scratch without supervision.The backward pass starts from the predicted location in the last frame; ideally it returns to the initial bounding box.Source: §1 · p. 2, Figure 1 · p. 2
- The training targets high-entropy local regions rather than requiring a complete object annotation, aiming to learn generic features for arbitrary image content.Entropy-based initialization avoids manual annotation in the first frame.Source: §1 · p. 2
- Across seven benchmarks, the unsupervised tracker is comparable to baseline fully supervised trackers, and with adaptive online updates shows state-of-the-art performance.The paper also reports comparable performance to supervised training when labels are limited or noisy, and potential gains from more unlabeled data.Source: §1 · p. 2
- The paper identifies its threefold contribution and directs readers to the related work, method, experiments, and conclusion sections.The supplied contribution list emphasizes the unsupervised method, training enhancements, and benchmark evaluation.Source: §1 · pp. 2–3
- Multi-frame validation is proposed to enlarge trajectory inconsistency when the tracker loses the target.The paper also proposes a cost-sensitive loss and entropy selection metric to reduce the contribution of easy samples.Source: §1 · p. 2
2 Related Work
Prior deep trackers commonly use supervised training or pretrained models, while forward–backward analysis has mostly served other purposes. This work uses the trajectory loop to train a tracking network from unlabeled videos.
2.1 Deep Visual Tracking
- Unlike deep trackers that use supervised CNN training or off-the-shelf models, the proposed method learns a tracking network from scratch using unlabeled videos.It builds on a Siamese correlation filter network.Source: §2.1 · p. 3
2.2 Forward-Backward Analysis
- The paper repurposes forward–backward consistency from motion analysis and tracking reliability checks as a training signal for a deep visual tracker.Earlier approaches cited in the passage use empirical metrics to identify reliable trajectories, while repeated bidirectional tracking can be costly for online use.Source: §2.2 · p. 3
2.3 Unsupervised Representation Learning
- Unlike, which uses an off-the-shelf tracker for video preprocessing, this method integrates tracking into unsupervised training with a tracking objective.The authors state that this objective is intended to characterize generic target objects.Source: §2.3 · p. 4
- The authors frame forward–backward motion in videos as a source of self-supervision suited to tracking, a task they say has rarely used unsupervised learning.They distinguish their tracking-specific approach from the general encoder–decoder framework discussed in the passage.Source: §2.3 · p. 4
3 Proposed Method
The method uses a forward–backward tracking loop to compute a consistency loss, then integrates unsupervised learning into a Siamese correlation filter network.
3 Proposed Method
- A content-rich local region is initialized as the target, tracked forward through subsequent frames, then used as a pseudo label for backward verification.The backward prediction in the first frame is ideally identical to the original bounding box.Source: §3 · p. 4
- The difference between forward and backward trajectories supplies a consistency loss for network training.The method overview places this training scheme within an unsupervised Siamese correlation filter network.Source: §3 · p. 4
- The method section proceeds from the correlation filter and Siamese network to the unsupervised prototype, training enhancements, training details, and online tracking.The supplied passage is a roadmap of Sections 3.1–3.5.Source: §3 · p. 4
3.1 Revisiting Correlation Tracking
The section reviews discriminative correlation filters (DCFs), which localize targets by regressing shifted input features to a soft target response. It then describes how Siamese correlation filter networks integrate this filter with learned features.
3.1 Revisiting Correlation Tracking
- DCFs regress circularly shifted search-patch features to a soft target response map for target localization.The Gaussian-shaped label peaks at the target position.Source: §3.1 · p. 4
- The DCF filter is learned with ridge regression and can be computed efficiently in the Fourier domain.The filter is then convolved with a search patch to generate a response map.Source: §3.1 · p. 5
- Siamese DCF networks use shared-weight branches to extract template and search features, integrating feature extraction and the correlation filter end to end.The template branch generates a filter; the search branch uses it to produce the response map.Source: §3.1 · p. 5
3.2 Unsupervised Learning Prototype
The prototype trains a Siamese correlation filter tracker without additional supervision by comparing forward tracking with backward verification. Its consistency objective encourages the response map to return to the initial label.
3.2 Unsupervised Learning Prototype
- Forward-backward verification lets the tracker train from video patches without ground-truth annotations for subsequent frames.The difference between the initial and back-predicted bounding boxes in the first frame forms a consistency loss.Source: §3.2 · p. 5
3.2.1 Forward Tracking
- A template patch and its Gaussian center label produce a filter that generates a response map for a search patch in the next frame.The feature extractor has trainable parameters, and the response can be supervised with an available search-patch label.Source: §3.2.1 · p. 5
3.2.2 Backward Tracking
- Backward tracking swaps the search and template roles, using the forward prediction as a pseudo-label to generate a response map for the original template.The same network is used in both directions, with its parameters fixed during tracking.Source: §3.2.2 · p. 5
3.2.3 Consistency Loss Computation
- Training minimizes reconstruction error between the backward response map and the original template label.The response map should ideally be a Gaussian label peaking at the initialized target position; pseudo-labeled training pairs update the network through back-propagation.Source: §3.2.3 · pp. 5–6, Figure 3 · p. 6
3.3 Enhancement for Unsupervised Learning
The enhancements address false consistency when a tracker loses the target and noisy or uninformative training samples. They extend validation across frames and weight or exclude training examples to improve the unsupervised objective.
3.3 Enhancement for Unsupervised Learning
- A tracker can deviate from the target yet return to its starting position, while textureless or occluded samples can impair training.The consistency loss alone does not penalize the false-return case, motivating multi-frame validation and a cost-sensitive loss.Source: §3.3 · p. 6
3.3.1 Multi-frame Validation
- Using more frames makes localization errors accumulate, making false-consistent trajectories less likely to escape the consistency objective.The approach extends the cycle beyond two frames and combines consistency losses across trajectory cycles.Source: §3.3.1 · pp. 6–7
3.3.2 Cost-sensitive Loss
- The cost-sensitive scheme removes the 10% of training triples with the highest losses to reduce outlier effects that hinder convergence.Triples contain one initialized template and search patches from subsequent frames.Source: §3.3.2 · p. 7
- Motion weights give greater emphasis to training pairs with larger motion, while binary weights mark excluded triples and the remaining pairs.The motion weight reflects target motion across consecutive frames and identifies harder pairs for the network to focus on.Source: §3.3.2 · p. 7
3.4 Unsupervised Training Details
The tracker uses a lightweight Siamese correlation filter network and unlabeled video data. Training patches are initialized from high-entropy regions in the first frame, although those regions may not accurately identify a target.
3.4 Unsupervised Training Details
- The tracking network is a shallow Siamese model with two convolutional layers and a local response normalization layer.The lightweight structure supports efficient online forward inference.Source: §3.4 · p. 8
- ILSVRC 2015 is used for training, and unlike supervised approaches, the data preparation does not require manual per-frame labels.Supervised preparation also removes frames with occlusion, partial out-of-view targets, or irregular shapes.Source: §3.4 · p. 8
- The method selects the highest-entropy patch among 25 center-region candidates to initialize tracking, then samples patches from nearby frames for a training trajectory.The entropy heuristic may select an inaccurate target region, but is intended to alleviate meaningless background patches.Source: §3.4 · p. 8, Figure 7 · p. 8
3.5 Online Object Tracking
After offline unsupervised learning, tracking proceeds forward while the DCF adapts to target appearance changes through an online moving-average update. The method also uses adaptive updating and a three-level scale pyramid.
3.5 Online Object Tracking
- Online tracking updates the DCF to adapt to changes in target appearance.The update uses a moving average; α_t is the linear interpolation coefficient.Source: §3.5 · p. 8
- LUDT+ adds adaptive model updating to LUDT, following the DCF formulation used in ECO.The target scale is estimated using a patch pyramid with scale factors 1.015^s for s ∈ {-1, 0, 1}.Source: §3.5 · p. 8
- LUDT combines different trajectory cycles and selects informative image patches by entropy rather than using a center crop.LUDT+ and UDT+ add online tracking techniques such as adaptive updating to LUDT and UDT, respectively.Source: §3.5 · p. 9
4 Experiments
The experiments evaluate the unsupervised tracker across seven benchmarks and examine its training choices, representation, and performance. Results indicate that trajectory enlargement, entropy-based region selection, and additional unlabeled data can improve tracking outcomes.
4.1 Experimental Details
- Experiments cover seven benchmarks, using OPE distance and overlap precision on OTB, TempleColor, LaSOT, and TrackingNet, and EAO on VOT2016 and VOT2017/2018.LUDT and LUDT+ run at about 70 FPS and 55 FPS, respectively, on the reported PC.Source: §4 · p. 9, §4.1 · p. 9
4.2.1 Improvements upon UDT
- On OTB-2015, LUDT reaches 60.2% AUC and on Temple-Color it reaches 51.5% AUC after trajectory enlargement and RoI selection improve UDT.The study evaluates these changes against UDT using DP and AUC on both datasets.Source: §4.2.1 · p. 9, Table 1 · p. 9
4.2.2 Baseline Performance
- The unsupervised forward-backward tracker outperforms the encoder-decoder feature baseline in this tracking scenario.Randomly initialized features retain weak tracking capability, while HOG performs significantly worse than the proposed approach.Source: §4.2.2 · p. 9, Table 2 · p. 9
4.2.3 Training Data
- Unsupervised training reaches 60.0% AUC versus 62.6% with full supervision, while weakly labeled training reaches 61.4% AUC versus 62.6%.Entropy-based RoI selection achieves 60.0% AUC, compared with 59.4% for center cropping; it can suppress meaningless background samples such as sky and grass.Source: §4.2.3 · p. 10, Table 3 · p. 10
4.2.4 Trajectory Length
- On OTB-2015, adding a third validation frame improves results by 2.8% DP and 2.0% AUC; four-frame validation further improves accuracy, while five frames are less effective.The authors suggest four frames already provide adequate self-supervision for measuring consistency loss.Source: §4.2.4 · p. 10, Table 4 · p. 10
4.2.5 Cost-sensitive Loss
- Without hard-sample reweighing, LUDT drops about 1.5% DP and 1% AUC on OTB-2015; sample dropout was not ablated because training did not converge well without it..Source: §4.2.5 · p. 10
4.2.6 Unlabled Data Augmentation
- Adding OxUvA videos improves the tracker by 0.7% DP and 1.2% AUC, and adding LaSOT videos further improves performance.Few-shot domain adaptation uses several initial frames, with only the first-frame bounding box available, and takes about six minutes for 100 fine-tuning iterations.Source: Table 5 · p. 10, §4.2.6 · p. 11
4.2.7 Empirical Features Embedding
- With more unlabeled training videos, LUDT+ outperforms ECOhc, and adding HOG yields 65.7% AUC on OTB-2015.The feature-embedding visualization contrasts LUDT's unsupervised and fully supervised representations with VGG-19 feature maps.Source: §4.2.7 · p. 11, Figure 8 · p. 11, Table 6 · p. 11
4.3 Visualization of Unsupervised Representation
The learned unsupervised representation resembles supervised features while emphasizing central object responses, and the tracker compares favorably with several supervised and hand-crafted-feature trackers. Performance remains competitive across multiple benchmarks, with reported boundaries around specific challenge attributes and operating speeds.
4.3 Visualization of Unsupervised Representation
- Unsupervised and supervised LUDT responses are similar, though supervised responses emphasize object boundaries more and unsupervised responses still focus on object centers.The visualization compares these responses with VGG-19 feature maps across five layers.Source: §4.3 · p. 11
4.4 Comparison with State-of-the-art Methods
- On OTB-2015, LUDT achieves 76.9% DP and 60.2% AUC, compared with SiamFC's 77.1% DP and 58.2% AUC.LUDT exceeds CFNet by 2.1% DP and 3.4% AUC; Table 7 compares supervised baselines and recent trackers.Source: §4.4 · p. 12, Figure 10 · p. 12, Table 7 · p. 12
- On Temple-Color, LUDT achieves 51.3% AUC versus SiamFC's 50.3%, and exceeds CFNet by 6.4% DP and 4.7% AUC.The success plots compare trackers on Temple-Color.Source: §4.4 · p. 13, Figure 11 · p. 13
- On VOT2016, LUDT+ performs slightly worse than C-COT but runs much faster, and is reported to compare favorably with MDNet.The AR plot places better-performing trackers toward the upper right; the accompanying EAO graph ranks trackers from right to left.Source: Figure 12 · p. 13, Table 8 · p. 13, §4.4 · p. 14
- On VOT2017/2018, LUDT+ achieves 0.230 EAO, comparable with advanced Siamese trackers SA-Siam and SiamRPN.LUDT remains superior to the standard hand-crafted-feature DCF trackers DSST and KCF.Source: §4.4 · p. 14, Table 9 · p. 14
- On LaSOT, LUDT+ reaches 30.5% AUC, compared with ECO at 32.4%, STRCF at 30.8%, and TRACA at 25.7%.LUDT also outperforms the listed hand-crafted-feature DCF trackers; the figure plots success-curve AUC.Source: §4.4 · pp. 14–15, Figure 13 · p. 14
- On TrackingNet, LUDT+ outperforms ECO by 0.9% in AUC without labeled training data.LUDT records 54.3% AUC, exceeding ECOhc, CSR-DCF, and BACF by 0.2%, 0.9%, and 2.0%, respectively.Source: Table 10 · p. 14, §4.4 · p. 15
- On OTB-2015, LUDT outperforms SiamFC and CFNet on most challenge scenes, while its DP advantage is less apparent than its AUC advantage.The main gaps from fully supervised LUDT occur on illumination variation, occlusion, and fast motion.Source: Figure 14 · p. 15, §4.4 · p. 16
4.5 Limitations
The unsupervised tracker can drift under occlusion or drastic appearance change, and its forward-backward training adds computational load offline.
4.5 Limitations
- The unsupervised tracker tends to drift when targets are occluded or undergo drastic appearance change, unlike its fully supervised version.The illustrated failure sequences are Skiing and Soccer.Source: §4.5 · p. 16, Figure 16 · p. 16
- Forward and backward tracking increase computational load during training, although learning is offline.Source: §4.5 · p. 16
5 Concluding Remarks
The paper presents an unlabeled-video training framework for visual tracking and finds that its baseline tracker performs comparably to classic fully supervised trackers. An online-improved variant performs favorably against state-of-the-art tracking algorithms.
5 Concluding Remarks
- The baseline unsupervised tracker achieves results comparable to classic fully supervised trackers, while LUDT+ performs favorably against state-of-the-art tracking algorithms.LUDT+ adds online improvements such as a sophisticated update scheme.Source: §5 · p. 16
- The framework trains a Siamese correlation filter network from unlabeled videos using forward-backward tracking, multi-frame validation, and a cost-sensitive loss.Source: §5 · p. 16
- The authors identify potential to improve tracking accuracy by using more unlabeled or weakly labeled data.Source: §5 · p. 16