Source-linked AI summary
Siamese Instance Search for Tracking
Ran Tao, Efstratios Gavves, Arnold W. M. Smeulders
TL;DR
Tracking requires matching a target across frames despite varied appearance changes, while conventional approaches often explicitly model distortions or update target-specific representations. The paper learns a generic two-stream Siamese matching function from external videos and applies it without adaptation to unseen targets. The resulting SINT tracker reaches state-of-the-art performance and supports re-identification after a complete video shot.
Problem
Tracking needs a matching function that remains effective under occlusion, scale, rotation, illumination, camera motion, and other appearance changes.
Method
SINT learns a generic two-stream Siamese matching function from external videos, then matches each new frame’s candidates to the target’s initial patch without updating the function.
Results
SINT reaches state-of-the-art tracking performance and allows target re-identification after absence for a complete video shot.
Takeaways & Limitations
A learned generic matching function can support simple tracking of previously unseen targets without model updating, occlusion detection, tracker combination, or geometric matching.
Takeaways & Limitations
The training data must be sufficiently varied and include bounding-box annotations while remaining separate from evaluation videos; similar objects may cause the tracker to jump.
Abstract
from arXiv · showhide
In this paper we present a tracker, which is radically different from state-of-the-art trackers: we apply no model updating, no occlusion detection, no combination of trackers, no geometric matching, and still deliver state-of-the-art tracking performance, as demonstrated on the popular online tracking benchmark (OTB) and six very challenging YouTube videos. The presented tracker simply matches the initial patch of the target in the first frame with candidates in a new frame and returns the most similar patch by a learned matching function. The strength of the matching function comes from being extensively trained generically, i.e., without any data of the target, using a Siamese deep neural network, which we design for tracking. Once learned, the matching function is used as is, without any adapting, to track previously unseen targets. It turns out that the learned matching function is so powerful that a simple tracker built upon it, coined Siamese INstance search Tracker, SINT, which only uses the original observation of the target from the first frame, suffices to reach state-of-the-art performance. Further, we show the proposed tracker even allows for target re-identification after the target was absent for a complete video shot.
1. Introduction
The paper learns a generic matching function from external videos and uses it without adaptation to match an initial target patch against candidates in later frames. Built on this function, SINT avoids several conventional tracking components while achieving state-of-the-art performance and supporting re-identification after long absence.
- Learning the matching function: The paper learns tracking invariances from external videos rather than explicitly modeling particular distortions such as occlusion, scale change, rotation, illumination, or camera motion.The training videos are kept separate from evaluation videos and do not contain the tracking targets.
- Generic application: The learned matching function is applied as is, without adaptation, to previously unseen target objects and new tracking videos.The target is excluded from the training data, avoiding offline learning of target-specific information.
- Tracker design: SINT matches the target’s initial first-frame patch with candidates in each new frame and returns the candidate with the highest learned similarity.The tracker focuses on the matching function instead of building a more elaborate tracker.
- Performance and simplicity: SINT uses no model updating, occlusion detection, tracker combination, or geometric matching while reaching state-of-the-art tracking performance.Its design is deliberately limited to matching the initial target patch against candidate patches.
- Re-identification: The absence of on-the-fly model updating allows successful target re-identification after the target has been absent for a long period.The paper reports this capability after a complete video shot.
2. Related Work
Prior tracking work develops matching functions around explicit distortions, online updates, composite trackers, or target-specific learning. This paper instead connects tracking with instance search and learns a generic deep matching function for arbitrary objects and realistic appearance variations.
- Matching functions in tracking: Early trackers directly matched target and incoming-image patches, while later methods incorporated affine, probabilistic, earth mover’s distance, eigen-image, fragment, or super-pixel matching.Normalized cross-correlation exemplifies the direct-matching approach.
- Recent tracking methods: Composite trackers combine matching with recovery, continuous displacement prediction, cautious updating, or historical expert snapshots.Examples include TLD, Struck, and MEEM.
- Deep learning in tracking: Earlier deep-tracking methods learn features, target classifiers, or target-specific saliency and objectness, often using online adaptation or target-related data.The paper positions its approach as learning a generic matching function from rich external data instead.
- Instance search: Instance search retrieves a particular object from one example, and learned matching supports generic objects despite substantial appearance changes.The paper adapts this paradigm to tracking rather than category-specific retrieval.
- Siamese architectures: The two-stream Siamese architecture has been used for pairwise matching tasks, and this work designs one to learn robust representations for object tracking.The network processes paired inputs to support matching under appearance variation.
3. Siamese Instance Search Tracker
SINT learns a generic matching function with a two-stream Siamese network and applies it directly to tracking, using first-frame target appearance to select candidates. Its architecture is designed for precise, efficient comparison of many regions while training on external videos without target-specific data.
- 3. Siamese Instance Search Tracker: SINT is introduced as a tracker built around a learned matching function and simple online tracking inference.The method is named Siamese INstance search Tracker, or SINT.
- 3.1. Matching Function: The matching model uses two Siamese branches that process paired inputs separately through convolutional networks.The authors compare AlexNet-like and VGG-like branch architectures.
- 3.1. Matching Function: The network limits max pooling to preserve spatial resolution needed for precise tracking localization.The AlexNet-like network has no max pooling, while the VGG-like network uses two early max-pooling layers.
- 3.1. Matching Function: Region pooling efficiently converts many overlapping candidate regions into fixed-length representations after shared image processing.This reduces computation compared with independently processing hundreds of candidate regions.
- 3.1. Matching Function: Training uses paired video data and margin contrastive loss to bring same-object representations close and separate different-object representations by a minimum margin.The distance is computed between ℓ2-normalized latent representations; y_jk indicates whether the pair depicts the same object, and ϵ is the negative-pair margin.
- 3.1. Matching Function: The network is trained on varied external videos excluded from evaluation sets, then applied without target-specific adaptation.The external data require bounding-box annotations but no labels for individual variation types.
- 3.2. Tracking Inference: During tracking, SINT compares each frame’s candidate boxes with the first-frame target and selects the candidate with the highest learned match.Candidate locations are sampled around the previous prediction at multiple radii and scales, then the selected box is refined with Ridge regressors trained on the first frame and kept fixed.
4. Experimenents
The experiments evaluate SINT’s training setup, architecture, robustness, and benchmark performance on OTB and additional sequences. SINT is competitive with state-of-the-art trackers, while SINT+ improves further through adaptive sampling and optical-flow filtering.
- Dataset and evaluation metrics: OTB contains 50 videos and 51 annotated tracking sequences covering challenges including fast motion, deformation, background clutter, and occlusion.Performance is ranked using AUC for the success plot and Prec@20 for the precision plot.
- Design evaluation: Generic Siamese fine-tuning on external ALOV video data significantly improves over ImageNet pretraining, whereas first-frame target-specific fine-tuning provides only marginal improvement.The benchmark excludes 12 ALOV videos overlapping with OTB, leaving no common objects between training and evaluation.
- Design evaluation: Removing max-pooling layers improves localization accuracy because pooling reduces feature-map resolution; multi-layer features and deeper VGG-style networks also improve performance.The experiments report advantages for multi-layer features and significant improvement from the deeper network.
- State-of-the-art comparison: SINT reaches state-of-the-art performance on OTB despite simple NCC-like inference, while SINT+ outperforms the other state-of-the-art trackers.SINT+ uses adaptive candidate sampling and optical flow to remove motion-inconsistent candidates containing less than 25% of the tracked pixels.
- State-of-the-art comparison: SINT is temporally and spatially at least as robust as MEEM and MUSTer, and performs better than MUSTer in 6 of 11 AUC attribute groups and 7 of 11 Prec@20 groups.The 11 annotated distortion types include illumination variation and occlusion.
- Additional sequences and re-identification: With whole-image window sampling, SINT can re-identify a target after it is absent for a complete video shot.The paper illustrates this capability by tracking Yoda when the target re-enters the scene.
5. Conclusion
SINT tracks by matching the initial target patch against candidates using a learned function, without adapting the tracker to each target. It reaches state-of-the-art performance on OTB and supports re-identification after a complete video shot.
- SINT matches the initial target patch with candidates in each new frame using a learned matching function.The matching function is learned on external video data and used without adapting to the tracked target.
- The tracker reaches state-of-the-art performance on OTB without target updating, tracker combination, or occlusion detection.
- SINT enables target re-identification after the target is absent for a complete video shot.The demonstration uses a 1500-frame, 12-shot Star Wars video.