Source-linked AI summary
Fully-Convolutional Siamese Networks for Object Tracking
Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, Philip H. S. Torr
TL;DR
Arbitrary object tracking lacks pre-trained object-specific detectors, while online-only learning limits model richness and deep adaptation compromises speed. This paper instead uses an offline-trained fully-convolutional Siamese network, achieving strong benchmark performance at frame-rate speed.
Problem
Arbitrary object tracking must handle objects identified only at test time, while video-only online learning limits the complexity of learnable appearance models.
Method
The paper trains a fully-convolutional Siamese similarity network offline and evaluates it online to locate an exemplar within a larger search image.
Results
The tracker improves over recent state-of-the-art real-time trackers and outperforms most top VOT-15 methods while remaining the only one at frame-rate speed.
Takeaways & Limitations
Offline-learned deep embeddings provide rich features that enable simple online tracking strategies to perform well across tracking benchmarks.
Takeaways & Limitations
The method assumes uniform exemplar sizes for simpler mini-batch implementation, although this assumption could be relaxed.
Abstract
from arXiv · showhide
The problem of arbitrary object tracking has traditionally been tackled by learning a model of the object's appearance exclusively online, using as sole training data the video itself. Despite the success of these methods, their online-only approach inherently limits the richness of the model they can learn. Recently, several attempts have been made to exploit the expressive power of deep convolutional networks. However, when the object to track is not known beforehand, it is necessary to perform Stochastic Gradient Descent online to adapt the weights of the network, severely compromising the speed of the system. In this paper we equip a basic tracking algorithm with a novel fully-convolutional Siamese network trained end-to-end on the ILSVRC15 dataset for object detection in video. Our tracker operates at frame-rates beyond real-time and, despite its extreme simplicity, achieves state-of-the-art performance in multiple benchmarks.
1 Introduction
The introduction frames arbitrary object tracking as a setting where object-specific detectors cannot be pre-trained, making online appearance learning necessary but limiting model complexity. It proposes offline-trained Siamese similarity learning, enabled by ImageNet Video, followed by simple online evaluation for fast tracking.
- Problem setting: Arbitrary object tracking identifies the target only by a rectangle in the first frame, so a specific detector cannot have been trained beforehand.
- Limitations of online learning: Online learning from examples extracted from the current video has been successful, but it restricts trackers to comparatively simple appearance models.The introduction cites TLD, Struck, and KCF as examples of this paradigm.
- Related approaches: Prior deep-learning approaches either use shallow methods on network features or apply SGD to fine-tune multiple network layers.
- Proposed approach: The paper instead trains a deep convolutional network offline for general similarity learning and evaluates the learned function online during tracking.The proposed Siamese network is trained to locate an exemplar image within a larger image.
- Data opportunity: The emergence of the ILSVRC dataset for object detection in video is presented as enabling training of similarity models for tracking.The authors argue that earlier tracking datasets contained only a few hundred annotated videos.
2 Deep similarity learning for tracking
The tracker learns a similarity function that compares an exemplar with candidate images, then locates the target by selecting the highest-scoring candidate. A fully-convolutional Siamese architecture computes similarities for all translated sub-windows simultaneously using a larger search image and produces a score map.
- Similarity learning: Similarity learning trains f(z, x) to score image pairs highly when they depict the same object and low otherwise.Tracking selects the candidate location with maximum similarity to the target’s previous appearance.
- Siamese architecture: A Siamese network applies the same transformation ϕ to exemplar and candidate images, then combines their representations through g.The similarity function is f(z, x) = g(ϕ(z), ϕ(x)); with a simple metric g, ϕ acts as an embedding.
- Fully-convolutional design: The fully-convolutional network evaluates a larger search image once, computing similarities for all translated sub-windows on a dense grid.A convolutional embedding and cross-correlation layer produce a spatial score map rather than a single score.
- Training objective: Training uses exemplar–search pairs with a logistic loss, generating many labeled score-map examples from each pair and optimizing the network parameters with SGD.Each score-map position receives a positive or negative label, and positive and negative losses are weighted to address class imbalance.
- Architecture assumptions: The network is symmetric, so f(z, x) = f(x, z), although uniform exemplar sizes are assumed to simplify mini-batch implementation.Different exemplar sizes would be possible in theory because the network is also fully-convolutional in the exemplar.
3 Related work
Related work explored recurrent networks, fixation-based distance metrics, test-time fine-tuning, and image-pair convolutional trackers. The paper distinguishes its approach by training without videos from the benchmark domain, avoiding the overfitting concern raised for competing methods.
- Recurrent tracking: RNN-based trackers predict target positions or use differentiable attention, but had not yet demonstrated competitive modern-benchmark results.These methods were identified as a promising avenue for future research.
- Fixation-based tracking: Denil et al. combine a particle filter with an RBM-learned distance metric between foveated fixations, plus an object-specific stochastic fixation policy.The method was demonstrated on synthetic MNIST sequences and qualitatively on face and person tracking.
- Test-time fine-tuning: SO-DLT and MDNet fine-tune offline-trained convolutional networks with video-derived examples at test time, but cannot operate at frame-rate.They follow the conventional tracking-as-detector-learning paradigm and use SGD online.
- Image-pair convolutional tracking: GOTURN learns a convolutional function of image pairs that directly regresses the second image’s object rectangle, handling scale and aspect-ratio changes without exhaustive evaluation.The rectangle prediction differs from predicting only a position.
- Training-domain separation: MDNet, SINT, and GOTURN train on video data from the ALOV/OTB/VOT benchmark domain, a practice forbidden in VOT over overfitting concerns.The paper’s contribution is demonstrating effective tracking without videos from the testing set’s distribution.
4 Experiments · 4.1 Implementation details · Training.
Training uses straightforward SGD for 50 epochs over sampled pairs, with Xavier-scaled Gaussian initialization, mini-batches, and geometric learning-rate annealing. Online tracking remains minimal: the initial exemplar is fixed and compared convolutionally with later-frame sub-windows, while score-map upsampling improves spatial resolution.
- Training.: The embedding-function parameters are optimized with straightforward SGD using MatConvNet.Initialization follows the improved Xavier method, with parameters drawn from a scaled Gaussian distribution.
- Training.: 50 epochs each contain 50,000 sampled training pairs.The pairs are sampled according to the procedure described in Section 2.2.
- Training.: Mini-batches contain 8 examples for estimating gradients at each iteration.
- Training.: The learning rate is geometrically annealed each epoch from 10−2 to 10−5.
- Training.: The initial object embedding ϕ(z) is computed once and compared convolutionally with sub-windows in subsequent frames.Online exemplar updates through simple strategies such as linear interpolation provide little performance gain, so the exemplar remains fixed.
- Training.: Bicubic interpolation upsamples the score map from 17 × 17 to 272 × 272.
Tracking. · 4.2 Evaluation
The paper evaluates two variants of its fully-convolutional Siamese tracker and reports real-time performance for the online tracking pipeline across different scale-search settings. Training and tracking code, along with dataset-generation scripts, are shared to support reproducibility.
- Tracking.: The full online tracking pipeline runs at 86 frames-per-second when searching over 3 scales.This measurement uses a single NVIDIA GeForce GTX Titan X and an Intel Core i7-4790K at 4.0GHz.
- Tracking.: The full online tracking pipeline runs at 58 frames-per-second when searching over 5 scales.The 5-scale configuration is slower than the 3-scale configuration on the reported hardware.
- Tracking.: The reported performance measurements use a single NVIDIA GeForce GTX Titan X.The hardware setup also includes an Intel Core i7-4790K at 4.0GHz.
- Tracking.: The reported performance measurements use an Intel Core i7-4790K processor running at 4.0GHz.The system also includes a single NVIDIA GeForce GTX Titan X.
- Tracking.: The authors share training and tracking code to make their experimental results reproducible.They also share scripts for generating the curated dataset.
- Tracking.: The authors share scripts to generate the curated dataset.These scripts are provided alongside the training and tracking code.
- 4.2 Evaluation: The evaluation compares SiamFC with SiamFC-3s, a variant that searches over 3 scales instead of 5.SiamFC denotes Siamese Fully-Convolutional.
- 4.2 Evaluation: SiamFC-3s searches over 3 scales, whereas SiamFC searches over 5 scales.The two variants differ in their number of searched scales.
4.3 The OTB-13 benchmark
The OTB-13 benchmark evaluates trackers by the area under success-rate curves across IoU thresholds. The paper compares the tracker against established and recent real-time state-of-the-art methods under OPE, TRE, and SRE protocols.
- Evaluation metric: OTB-13 defines per-frame success by whether the estimate’s intersection-over-union with ground truth exceeds a threshold.Trackers are compared across different threshold values using their success rates.
- Evaluation protocols: Figure 3 reports one pass evaluation, temporal robustness evaluation, and spatial robustness evaluation for the OTB-13 benchmark.These protocols are abbreviated OPE, TRE, and SRE, respectively.
- Evaluation metric: Trackers are ranked by the area under the curve of success rates over different IoU thresholds.
- Compared trackers: The comparison includes Staple, LCT, CCT, SCT4, DLSSVM NU, DSST, and KCFDP as recent state-of-the-art trackers capable of frame-rate speed.CCT, SCT4, and KCFDP were available only for OPE when the paper was written.
4.4 The VOT benchmarks
The VOT2015 benchmark evaluates trackers on sequences selected to represent seven challenging situations, with automatic re-initialization after failures. SiamFC and SiamFC-3s are compared with leading VOT 2014 trackers and recent real-time methods using accuracy and robustness.
- Benchmark setup: VOT2015 uses 356 sequences selected to represent seven different challenging situations, including sequences originating from datasets such as ALOV and OTB.The benchmark uses the stable vot2015-final version of the VOT toolkit.
- Benchmark setup: Trackers are automatically re-initialized five frames after failure in the VOT benchmark.Failure is defined using the intersection-over-union criterion described by the benchmark protocol.
- Compared trackers: SiamFC and SiamFC-3s are compared against the best 10 trackers from VOT 2014, plus Staple and GOTURN, two recent real-time trackers.Staple was presented at CVPR 2016 and GOTURN at ECCV 2016.
- Evaluation measures: Accuracy is measured by average IoU, whereas robustness is measured by the total number of failures.The benchmark reports these as two distinct performance measures.
VOT-14 results. · VOT-15 results.
The paper compares its tracker with top participants in VOT-14 and reports expected-overlap results for VOT-15 and an early VOT-16 comparison. Expected average overlap is computed from accuracy and failures, with SiamFC-3s exceeding SiamFC in the reported VOT-16 scores.
- VOT-14 results.: The method is compared against the 40 best participants in the 2015 edition.
- VOT-14 results.: VOT-14 evaluation uses raw accuracy scores and the number of failures.
- VOT-15 results.: The VOT-14 accuracy-robustness plot ranks trackers by proximity to the top-right corner.
- VOT-15 results.: Expected average overlap represents average IoU without re-initialization after failure.
- VOT-15 results.: Figure 5 shows the final expected-average-overlap ranking, while Table 2 reports scores and speed for the 15 highest-ranked trackers.
- VOT-15 results.: 0.3876 and 0.4051 are the overall expected overlaps reported for SiamFC and SiamFC-3s, respectively.These values average the baseline and unsupervised experiments.
- VOT-15 results.: The reported 2016 scores were provided as an early comparison because the official results were unavailable at writing.The paper notes that these scores differ from the VOT-16 report because its challenge entry was a preliminary version.
VOT-16 results.
The method improves over recent real-time trackers and outperforms most leading methods on the challenging VOT-15 benchmark. It is also the only method reported to achieve frame-rate speed there.
- VOT-16 results.: The method improves over recent state-of-the-art real-time trackers despite its simplicity.The passage attributes these comparisons to Figures 3 and 4.
- VOT-16 results.: On the challenging VOT-15 benchmark, the method outperforms most of the best methods.The passage cites Figure 5 and Table 2 for this result.
- VOT-16 results.: The method is the only one reported to achieve frame-rate speed on VOT-15.This result is presented alongside the benchmark comparison in Figure 5 and Table 2.
4.5 Dataset size
Increasing the training dataset from 5% to 100% steadily improves VOT-15 expected average overlap from 0.168 to 0.274, suggesting that larger video datasets could improve performance further.
- 4.5 Dataset size: 0.168 to 0.274: VOT-15 expected average overlap steadily improves as the training dataset increases from 5% to 100%.The results indicate that using a larger video dataset could increase performance further.
- 4.5 Dataset size: The 2 million supervised bounding boxes used for training still come from a relatively moderate number of videos compared with datasets normally used to train conv-nets.This leaves room for performance gains from substantially larger video datasets.
5 Conclusion
The work replaces traditional online tracking updates with offline learning of strong embeddings using a fully-convolutional Siamese network. This approach improves data efficiency while enabling efficient spatial searches at test time, with robustness to several challenging visual conditions without model updates.
- 5 Conclusion: The proposed approach learns strong object embeddings offline instead of relying on traditional online learning during tracking.The network is trained before tracking, avoiding online stochastic-gradient updates to adapt its weights.
- 5 Conclusion: Fully-convolutional Siamese networks use available tracking data more efficiently than their use in classification settings.The paper attributes this efficiency to the tracking architecture and training approach.
- 5 Conclusion: The tracker performs efficient spatial searches at test time without requiring model updates after the first frame.The tracker computes ϕ(z) from only the first frame and does not update its model during tracking.
- 5 Conclusion: The tracker remains robust under motion blur, drastic appearance changes, poor illumination, and scale change.These conditions are illustrated in snapshots of the tracker equipped with the proposed Siamese network trained from scratch on ImageNet Video.