Source-linked AI summary
TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild
Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, Bernard Ghanem
TL;DR
Object tracking is constrained by small, saturated datasets and the scarcity of dedicated large-scale data for deep trackers. TrackingNet introduces a large-scale dataset and benchmark with dense annotations, diverse real-world videos, and a segregated test set. Its use improves tracker performance by up to 1.7% on other datasets, while the benchmark indicates that tracking in the wild remains unsolved.
Problem
Small tracking datasets and sparse detection annotations limit dedicated training data for data-hungry deep trackers.
Method
TrackingNet combines large-scale YouTube-derived tracking videos, dense annotations generated from coarse labels, and a separately annotated test set evaluated through an online server.
Results
Pretraining deep models on TrackingNet improves performance on other datasets by up to 1.7%, while more than 20 trackers are benchmarked on TrackingNet.
Takeaways & Limitations
TrackingNet provides a large-scale basis for training and evaluating trackers in the wild, and the results suggest the task remains far from solved.
Takeaways & Limitations
Bounding boxes are adjusted to visible object parts, while estimating occluded parts is avoided because it is not deterministic.
Abstract
from arXiv · showhide
Despite the numerous developments in object tracking, further development of current tracking algorithms is limited by small and mostly saturated datasets. As a matter of fact, data-hungry trackers based on deep-learning currently rely on object detection datasets due to the scarcity of dedicated large-scale tracking datasets. In this work, we present TrackingNet, the first large-scale dataset and benchmark for object tracking in the wild. We provide more than 30K videos with more than 14 million dense bounding box annotations. Our dataset covers a wide selection of object classes in broad and diverse context. By releasing such a large-scale dataset, we expect deep trackers to further improve and generalize. In addition, we introduce a new benchmark composed of 500 novel videos, modeled with a distribution similar to our training dataset. By sequestering the annotation of the test set and providing an online evaluation server, we provide a fair benchmark for future development of object trackers. Deep trackers fine-tuned on a fraction of our dataset improve their performance by up to 1.6% on OTB100 and up to 1.7% on TrackingNet Test. We provide an extensive benchmark on TrackingNet by evaluating more than 20 trackers. Our results suggest that object tracking in the wild is far from being solved.
1 Introduction
Object tracking remains challenging because established datasets are small, while deep trackers lack a dedicated large-scale training resource. TrackingNet addresses this gap with diverse tracking data, a segregated benchmark, and dense annotations for training and evaluation.
- Established OTB and VOT datasets are small and do not fully represent challenges in tracking objects in the wild.
- Nine of the ten top-performing VOT17 trackers rely on deep features, yet the community lacks a dedicated large-scale dataset for training deep trackers.
- Existing dataset overlap and the absence of a clear train-test split make it difficult to train deep tracking networks fairly.
- TrackingNet provides a large-scale tracking dataset with dense temporal annotations, diverse object classes, and real-world YouTube video contexts.
- TrackingNet includes a segregated test set with similar object-class and motion distributions, withheld annotations, and an evaluation server.
- Pretraining deep models on TrackingNet improves performance on other datasets by up to 1.7%.
2 Related Work
Prior tracking research spans single-object and multi-object settings, with single-object tracking commonly combining target representation and object search. Existing datasets and detection resources support progress, but remain limited for training deep trackers because annotations are sparse in time.
- Single-object tracking uses tracking-by-detection, combining model representation with object search.
- Model representation may be generative or discriminative, while object search balances computational cost against dense sampling of the region of interest.
- Correlation-filter trackers learn an object filter from the first frame, localize it in later frames, and update the filter over time.
- Deep trackers use generic detection features, online domain-specific fine-tuning, fully convolutional networks, or Siamese networks trained offline.
- Common tracking datasets include OTB, VOT, ALOV300, TC128, and application-specific collections such as UAV123 and NUS PRO.
- Object detection datasets provide sparse or low-frame-rate video boxes, so they lack motion information about object dynamics in consecutive frames.
3 TrackingNet
TrackingNet is a large-scale object-tracking dataset and benchmark built from diverse YouTube videos, with dense annotations, separate training and testing sets, and controlled evaluation. Its design targets deep trackers by providing temporal annotations and distributions intended to reflect tracking in the wild.
- Dataset scale: 30,643 video segments contain 14,431,266 densely annotated frames from 140 hours of visual content.The videos have an average duration of 16.6 seconds, and each frame has one upright bounding box.
- Dataset splits: TrackingNet uses 30,132 training videos and 511 novel testing videos selected to give the two splits similar distributions.The test videos are sourced from YouTube with Creative Commons licences so they can be shared.
- Object classes: Object classes are adapted from YT-BB by removing motionless categories and splitting person into seven context-based classes.The resulting training distribution preserves the original distribution of 21 retained object classes.
- Annotation construction: The training set converts YT-BB’s 1-fps coarse annotations into dense annotations using weighted forward and backward tracker passes.The authors rely on the claim that trackers are reliable over one-second intervals and use a mixture of state-of-the-art trackers, including DCF.
- Testing annotations: Testing videos are annotated by Mechanical Turk workers using the smallest upright bounding box fitting any visible part of the tracked object.VATIC and optical flow assist annotation, while every frame is visually inspected after annotation rounds.
- Dataset characteristics: TrackingNet’s test set has an attribute distribution similar to its training set and broader diversity in bounding-box resolution, motion, and object context than established datasets.The evaluation considers 15 attributes, with five estimated automatically and ten checked manually.
4 Dataset Experiments
TrackingNet addresses sparse temporal annotations by using tracker-aided interpolation to generate dense bounding boxes for training. Experiments compare trackers and annotation-combination strategies, selecting DCF with bidirectional weighted averaging for the training set.
- Dense Annotation Generation: Sparse bounding boxes from YT-BT do not capture the temporal evolution required by deep network-based trackers.TrackingNet Training Set contains approximately 30K videos originally annotated every second.
- Dense Annotation Generation: Four experiments evaluate 12 trackers for densely annotating OTB100 sequences split into 30-frame segments at 30fps.The candidate trackers span correlation-filter, Siamese, and other tracking approaches.
- Dense Annotation Generation: Forward and backward tracking results are combined by direct averaging or a convex weighted average that favors the forward pass near the first frame.The weighting uses α = 0.05 and sequence length t ∈ [1, 30].
- Dense Annotation Generation: The weighted-average strategy generally performs best, and DCF is selected for annotating TrackingNet because it combines reasonable accuracy with a 300fps frame rate.DCF runs in both forward and backward directions before weighted combination.
5 Tracking Benchmark
The TrackingNet benchmark evaluates diverse tracking algorithms under standard, speed-aware, fine-tuning, and attribute-specific settings. Results show substantial difficulty on TrackingNet, sensitivity to runtime, measurable gains from limited fine-tuning, and persistent challenges from rotation, low resolution, and occlusion.
- 5 Tracking Benchmark: The benchmark covers a broad range of tracking principles, especially discriminative correlation filters using handcrafted or deep features.Evaluated trackers vary in representations, kernels, and search strategies.
- 5.1 State-of-the-art Benchmark on TrackingNet: About 60% is the highest success rate on TrackingNet, versus around 90% on OTB, with MD-Net ranking highest but running very slowly.CFNet and SiamFC follow and benefit from large-scale training on ImageNet Videos.
- 5.2 Real-Time Tracking: Up to 50% degradation occurs for trackers running below real-time when frames are skipped according to tracker speed.CFNet does not degrade despite seeing every third frame, while newer deep trackers are generally less affected.
- 5.3 Retraining on TrainingNet: 0.533 to 0.543 precision on TrackingNet Test and 0.765 to 0.781 on OTB100 follow fine-tuning SiameseFC on one 2,511-video TrackingNet chunk.Normalized precision and success also increase on both datasets.
- 5.4 Attribute Specific Results: In-plane rotation, low-resolution targets, and full occlusion are consistently the most difficult attributes, while illumination variation, partial occlusion, and deformation affect trackers least.TrackingNet Test videos are annotated with 15 attributes.
6 Conclusion
TrackingNet presents a large-scale tracking dataset, dense-annotation approach, and benchmark for evaluating modern trackers. The authors conclude that it improves deep trackers and exposes difficult attributes, while planning to expand the test set for generalization evaluation.
- 6 Conclusion: TrackingNet is presented as the largest object-tracking dataset, using a novel interpolation method to leverage large-scale detection datasets.The paper also benchmarks more than 20 tracking algorithms.
- 6 Conclusion: The benchmark identifies especially difficult tracking attributes and verifies that the dataset improves some deep-learning-based trackers.The stated conclusion connects dataset scale with both evaluation and tracker improvement.
- 6 Conclusion: The planned test-set expansion from 500 to 1,000 videos is intended to evaluate generalization across additional classes within the same category.The extra videos are planned to come from different classes, such as tortoise and animal.