Source-linked AI summary
Long-term Tracking in the Wild: A Benchmark
Jack Valmadre, Luca Bertinetto, João F. Henriques, Ran Tao, Andrea Vedaldi, Arnold Smeulders, Philip Torr, Efstratios Gavves
TL;DR
Short-term benchmarks rarely test long videos or target disappearance, although practical tracking requires both localization and presence decisions. OxUvA constructs and curates a large long-term dataset, introduces an evaluation for these two abilities, and finds that its benchmark contains 366 tracks with frequent disappearances while emphasizing reliable sparse annotation and representative tracker comparisons. The benchmark is intended to support tracking evaluation in long-term scenarios.
Problem
Existing benchmarks focus on short sequences with continuously visible targets, providing limited evaluation of long-term tracking with disappearance and re-appearance.
Method
The authors curate longer YTBB-based sequences, split them into development and secret test sets, and evaluate presence decisions jointly with target localization.
Results
OxUvA contains 366 tracks with target disappearance labeled in 52% of tracks, and representative trackers are compared using presence-and-localization measures.
Takeaways & Limitations
Reliable long-term evaluation can use sparse 1Hz labels when it includes many videos, while tracking methods must account for target absence as well as location.
Takeaways & Limitations
The dataset uses sparse 1Hz labels, relying on the assumption that failures and recoveries within one second are unlikely or relatively harmless.
Abstract
from arXiv · showhide
We introduce the OxUvA dataset and benchmark for evaluating single-object tracking algorithms. Benchmarks have enabled great strides in the field of object tracking by defining standardized evaluations on large sets of diverse videos. However, these works have focused exclusively on sequences that are just tens of seconds in length and in which the target is always visible. Consequently, most researchers have designed methods tailored to this "short-term" scenario, which is poorly representative of practitioners' needs. Aiming to address this disparity, we compile a long-term, large-scale tracking dataset of sequences with average length greater than two minutes and with frequent target object disappearance. The OxUvA dataset is much larger than the object tracking datasets of recent years: it comprises 366 sequences spanning 14 hours of video. We assess the performance of several algorithms, considering both the ability to locate the target and to determine whether it is present or absent. Our goal is to offer the community a large and diverse benchmark to enable the design and evaluation of tracking methods ready to be used "in the wild". The project website is http://oxuva.net
1 Introduction
Existing tracking benchmarks emphasize short videos in which targets remain visible, whereas practical tracking requires handling long sequences and target disappearance. OxUvA addresses this gap with a long-term benchmark that evaluates both localization and presence decisions.
- Visual object tracking locates a user-specified target throughout a video despite appearance changes such as deformation, occlusion, blur, and clutter.
- Standard benchmarks focus on short-term tracking, where the target is assumed to remain present and re-detection is unnecessary.
- Practical applications require trackers to handle target disappearance and re-appearance while determining whether the object is present.
- Existing videos average only 20–30 seconds, limiting knowledge about tracker behavior in practical videos lasting several minutes or longer.
- OxUvA contains sequences averaging 2.4 minutes and 14 hours of video, with target disappearance occurring in roughly half the videos.
- The benchmark introduces secret test ground truth, a presence-and-localization evaluation, continuous attributes, and comparisons among representative trackers.
2 Related Work
Prior tracking benchmarks expanded evaluation across diverse sequences but generally remained short-term and did not require re-detection. Earlier long-term efforts had limited scale or lacked secret test ground truth.
- ALOV, VOT, and OTB broadened tracking evaluation to larger collections spanning object classes and variation factors.
- Standard benchmarks typically use sequences averaging 7–30 seconds and do not include target disappearance requiring re-detection.
- The LTDT workshop evaluated long sequences with disappearances, but its five sequences were insufficient for general-purpose tracker assessment.
- Two concurrent long-term datasets included labeled target absences but, to the authors’ knowledge, neither provided a test set with secret ground truth.
3 Long-Term Tracking Dataset
OxUvA is constructed by curating and joining YouTube Bounding Boxes tracklets into longer, disappearance-containing sequences, then evaluating annotation and dataset design choices. The resulting benchmark emphasizes many sparsely labeled videos, secret testing, and long-term challenges.
- 3.1 Dataset Compilation and Curation: The dataset starts from YTBB tracklets and combines multiple tracklets referring to the same object instance to create longer sequences.
- 3.1 Dataset Compilation and Curation: Manual curation selects clear initial target frames, excludes uninformative sequences, and uses five expert annotators for annotation quality.
- 3.1 Dataset Compilation and Curation: 366 tracks in 337 videos contain 26 times OTB-100’s frames, an average of 2.2 absent labels per track, disappearances in 52% of tracks, and 2.3-minute average duration.
- 3.2 Data Subsets and Challenges: The dataset is split into 200 development and 166 test tracks with disjoint classes, while test labels remain secret and accessible through evaluation.
- 3.3 Annotation Density: At fixed video count, increasing annotation frequency above 1Hz only marginally improves reliability, while increasing video count steadily reduces variance.
- 3.3 Annotation Density: 4–5× smaller standard deviation results from annotating more videos sparsely at 1Hz rather than fewer videos densely at 25Hz.
- 3.3 Annotation Density: The authors conclude that 1Hz labeling preserves evaluation robustness and that a large number of videos is paramount.
4 Tracker Evaluation
OxUvA evaluates trackers jointly on whether the target is present and whether its location is correct. Because absent labels are rare, it uses class-imbalance-resistant rates and causal operating-point analysis rather than conventional accuracy-based evaluation.
- 4.1 Evaluating Object Presence and Localization: Trackers must output present or absent for each frame and provide an axis-aligned bounding box when the target is present.
- 4.1 Evaluating Object Presence and Localization: A present prediction is correct only when the target is correctly localized according to an IOU threshold, linking classification and localization.
- 4.1 Evaluating Object Presence and Localization: Only 4% of annotations are absent despite disappearances in roughly half of sequences, making accuracy, precision, and recall potentially misleading.
- 4.1 Evaluating Object Presence and Localization: TPR measures correctly localized present targets, while TNR measures correctly identified absent targets; both are invariant to class imbalance.
- 4.2 Operating Points: Conventional precision-recall evaluation is unsuitable because causal trackers’ internal state can change when prediction thresholds are applied.
- 4.2 Operating Points: Randomly flipping present predictions to absent with probability p traces a TPR–TNR lower bound toward the all-absent operating point.
- 4.2 Operating Points: The maximum geometric mean along this lower bound enables comparison when trackers that never predict absence would otherwise have TNR = 0.
5 Evaluated Trackers
The evaluation covers ten baseline trackers spanning long-term, correlation-filter, and deep-network approaches, using authors’ code and default settings without OxUvA-specific tuning.
- Baseline selection: Ten baselines are selected because they show strong performance, suit long-term tracking, or represent major tracker families.The methods comprise long-term-affine, correlation-filter, and deep convolutional approaches.
- Long-term-affine methods: LCT, EBT, and TLD can search across the frame, unlike methods restricted to local neighborhoods, but EBT does not report object presence.Global localization is important when the target disappears.
- Correlation-filter trackers: ECO-HC, BACF, and Staple are recent real-time correlation-filter trackers selected for high performance.
- Deep trackers: MDNet, SINT, and SiamFC are deep trackers; MDNet fine-tunes online, whereas SINT and SiamFC evaluate offline-learned similarity functions during tracking.SiamFC is fully convolutional and trained from scratch as a similarity function; SINT uses a pretrained VGG-16 architecture and bounding-box regression.
- Presence detection: Because few available methods determine object presence, SiamFC is augmented with re-detection logic that enters absent mode when its response score falls below a threshold.The augmented tracker searches random locations while in absent mode.
- Evaluation settings: All trackers use authors’ code and default hyper-parameters, and none are trained or tuned specifically for YTBB or the long-term dataset.Some models use external pretraining datasets with overlapping object classes.
6 Analysis
The analysis evaluates presence-aware localization, temporal degradation, disappearance robustness, and score thresholding. MDNet, SiamFC+R, and TLD perform strongest overall, while performance generally decays over time and thresholding offers substantial gains.
- Main evaluation: MDNet, SiamFC+R, and TLD dominate other methods because their collective lower bounds exceed those of all other trackers.The main evaluation uses TPR and TNR with IOU ≥0.5; most methods have TNR = 0 because they do not report absent predictions.
- Performance over time: The TPR plots show that, except possibly SINT, most methods’ performance decays rapidly after the first minute.Figure 4 compares cumulative early-frame TPR with TPR over later frames.
- Object disappearance: Performance is better on videos without disappearance, while TLD and SINT appear less affected because they lie closer to the equal-performance diagonal.The comparison uses videos with at least one absent annotation versus videos whose annotations are always present.
- Performance over time: SINT appears more robust to temporal degradation than most other methods, although variance is large beyond four minutes because fewer annotations remain.
- Post-hoc thresholding: Large gaps between lower-bound and post-hoc threshold curves indicate substantial gains from simply thresholding prediction scores.The re-detection module of SiamFC+R hardly improves over SiamFC’s post-hoc threshold curve.
- Post-hoc thresholding: Offline-trained Siamese approaches appear more promising than online-trained MDNet and Staple in the high-TNR region.
7 Continuous Attributes
The benchmark replaces subjective binary attributes with continuous quantities derived from annotations and metadata, then analyzes tracker performance across bins. Results expose sensitivity to speed, scale variation, distractors, object size, and absence.
- Definition: Continuous attributes replace subjective binary labels by measuring quantities directly from bounding boxes and metadata.The approach supports finer-grained analysis than manually choosing thresholds such as “fast motion” or “scale change.”
- Definition: Relative speed is normalized by object size because perspective makes closer, larger objects appear to move more.The instantaneous speed is divided by a size-related factor to reduce this effect.
- Analysis: Six bins are used for each attribute except distractors, which has three; shaded histograms indicate bin prevalence and reliability.Results from bins containing fewer videos have higher variance and are harder to interpret.
- Results: All local-search trackers lose performance as relative speed increases, while TLD is least affected among full-frame methods.The comparison includes TLD, LCT, EBT, and SiamFC+R.
- Results: Performance dips near 6× scale variation, whereas one distractor usually preserves performance and two distractors particularly hurt EBT and LCT.All trackers peak near object size 0.2, while MDNet and LCT remain strong for the largest objects.
- Results: Target disappearance reduces performance for all methods, and long videos compound small errors that can cause tracker drift.Video length is measured as elapsed time between the first and last target annotations.
8 Conclusion
The paper introduces OxUvA as a large long-term tracking benchmark with sparse labels, target disappearances, and evaluation of both presence decisions and localization. It evaluates trackers across several factors to characterize performance in realistic sequences.
- Dataset: OxUvA contains minute-long sequences with frequent target disappearances and exceeds OTB-100 by more than 25× in frames.The dataset uses relatively sparse target labeling at 1 Hz.
- Evaluation: The evaluation adapts True Positive and True Negative Rates to measure whether the target is present and where it is located.The benchmark evaluates 166 test sequences and examines factors including speed, size, sequence length, and distractors.
A Dev and Test Subsets
The development and test subsets have different absolute TPR and TNR values because they contain disjoint classes, but tracker rankings show the same relative trends.
- Subset comparison: TPR and TNR values differ significantly between dev and test because the subsets contain disjoint classes with different distributions.The comparison preserves relative tracker trends despite these absolute differences.
B Constrained and Open Challenges
The constrained challenge excludes trackers using deep convolutional networks or certain pretrained data, and TLD achieves the best score among eligible methods. Other methods are disadvantaged because they never predict absence.
- Constrained challenge: TLD obtains the best score among constrained-challenge trackers after excluding methods based on deep conv-nets or pretrained ImageNet VID weights.The constrained setting differs from the main-paper evaluation.
- Constrained challenge: Trackers that always assume the object is present are seriously disadvantaged in the challenge.Their inability to predict absence conflicts with the benchmark’s long-term setting.
C Object Classes
OxUvA reports target-object classes while tracking algorithms do not receive class labels during development or testing. Compared with other benchmarks, its videos are more evenly distributed across object classes.
- OxUvA reports class labels and class-instance counts for itself and several popular tracking benchmarks.The statistics are reported in Table 2.
- The dataset’s object classes are more equally distributed than those in other tracking benchmarks.
- 14%: the most frequent OxUvA class, bear, appears in only 14% of videos.