PaperlayerView exact revision on arXiv

Source-linked AI summary

GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild

Lianghua Huang, Xin Zhao, Kaiqi Huang

arXiv:1810.11981v3cs.CV

TL;DR

Generic object tracking lacks broad, class-agnostic evaluation data, especially for unseen object classes. GOT-10k addresses this with a WordNet-guided, high-diversity dataset and a one-shot protocol, providing large-scale annotations and benchmark infrastructure. Its test design and extensive tracker evaluations support more reliable assessment of generalization, within the stated dataset scope.

  • Problem

    Existing large tracking benchmarks have limited class diversity and overlapping training-test classes, making evaluation biased toward familiar objects and hindering assessment of class-agnostic generalization.

  • Method

    GOT-10k uses WordNet to populate object and motion classes, provides large manually annotated videos and tracking attributes, and evaluates trackers with zero-overlap training and test classes.

  • Results

    GOT-10k contains over 10,000 videos covering 563 object classes, with 1.5 million manually annotated bounding boxes, and benchmarks 39 tracking approaches.

  • Takeaways & Limitations

    The benchmark offers broad real-world object coverage and a one-shot evaluation setting intended to promote tracker generalization and support stable comparison.

  • Takeaways & Limitations

    Persons are included in both training and test sets, although their motion classes are non-overlapping to introduce a domain gap.

Abstract

from arXiv · show

We introduce here a large tracking database that offers an unprecedentedly wide coverage of common moving objects in the wild, called GOT-10k. Specifically, GOT-10k is built upon the backbone of WordNet structure and it populates the majority of over 560 classes of moving objects and 87 motion patterns, magnitudes wider than the most recent similar-scale counterparts. The contributions of this paper are summarized in the following: (1) GOT-10k offers over 10,000 video segments with more than 1.5 million manually labeled bounding boxes, enabling unified training and stable evaluation of deep trackers. (2) GOT-10k is by far the first video trajectory dataset that uses the semantic hierarchy of WordNet to guide class population. (3) For the first time, GOT-10k introduces the one-shot protocol for tracker evaluation, where the training and test classes are zero-overlapped. The protocol avoids biased evaluation results towards familiar objects and it promotes generalization in tracker development. (4) We conduct extensive tracking experiments with 39 typical tracking algorithms on GOT-10k and analyze their results in this paper. (5) Finally, we develop a comprehensive platform for the tracking community that offers full-featured evaluation toolkits, an online evaluation server, and a responsive leaderboard. The annotations of GOT-10k's test data are kept private to avoid tuning parameters on it. The database, toolkits, evaluation server and baseline results are available at http://got-10k.aitestunion.com.

1 INTRODUCTION

GOT-10k addresses limited object diversity and class-overlapping evaluation in generic tracking by constructing a broad benchmark guided by WordNet and introducing one-shot evaluation. It combines large-scale annotations, additional attributes, class-balanced metrics, experiments, and evaluation infrastructure.

  • Motivation: Generic object tracking must locate moving objects sequentially without prior object-class or environment knowledge, despite occlusion, deformation, and background distraction.The task supports applications including surveillance, augmented reality, biology, and robotics.
  • Motivation: Existing large benchmarks use limited manually defined classes and overlapping training-test classes, biasing evaluation toward familiar objects and limiting generalization assessment.LaSOT and TrackingNet provide large unified data, but their training and test classes are fully overlapped with close distributions.
  • Benchmark construction: GOT-10k contains over 10,000 video segments and more than 1.5 million manually annotated bounding boxes for unified training and fair comparison of deep trackers.The benchmark is designed for short-term generic object tracking.
  • Benchmark construction: GOT-10k covers 563 object classes and 87 motion forms, using WordNet to guide broader and less creator-dependent class population.Its object classes are expanded from WordNet-based semantic subtrees, while motion labels increase dataset variety.
  • Evaluation and resources: The one-shot protocol uses zero-overlap between training and test classes to avoid familiar-object bias and promote generalization in tracker development.The benchmark also supplies motion, visible-ratio, and absence labels for motion-aware and occlusion-aware tracking.
  • Evaluation and resources: The benchmark includes a 420-video, 84-object-class, 31-motion-class test set, class-balanced mAO and mSR metrics, retrained evaluations of 39 tracking approaches, and an online platform.Test annotations remain private to prevent parameter tuning on the test set.

2 RELATED WORK

Tracking datasets evolved from small unified evaluation sets toward specialized and large-scale benchmarks. TrackingNet and LaSOT provide substantial scale, but GOT-10k emphasizes broader object coverage and non-overlapping evaluation classes, while some video sources contain tracking-unfavorable noise.

  • Tracking datasets: Early unified tracking datasets included OTB, ALOV++, and VOT, while later datasets targeted specific settings such as people, aerial, long-term, thermal, RGBD, and high-frame-rate tracking.Many specialized datasets remain relatively small and provide only test data.
  • Large-scale benchmarks: TrackingNet and LaSOT introduced scale comparable to GOT-10k, with roughly 30 thousand and 1.4 thousand videos respectively in their described collections.TrackingNet selected videos from YouTube-BB for training and added 511 evaluation videos.
  • Large-scale benchmarks: TrackingNet and LaSOT use relatively narrow manually defined class sets and fully overlapping training-test classes, limiting evaluation of tracker generalization to unseen classes.GOT-10k is described as offering wider object coverage and the one-shot protocol.
  • Training data: YouTube-BB and ImageNet-VID contain noisy segments, including incomplete objects and shot changes, making some videos less optimal for tracking.

3 CONSTRUCTION OF GOT-10K

GOT-10k is constructed as a broad, high-quality generic tracking benchmark using WordNet to organize object and motion classes, with controlled collection, annotation, and dataset splitting. Its design emphasizes diverse coverage, compatible labels, occlusion information, and stable one-shot evaluation.

  • 3.1 Collection of Videos: WordNet guides selection of object and motion classes to provide broad, comprehensive, and unbiased coverage of real-world moving objects, motion patterns, and scenes.The construction expands WordNet categories and manually filters or prunes class subtrees before collection.
  • 3.1 Collection of Videos: One collection stage and five verification stages are used to filter stationary, noisy, incomplete, repeated, or otherwise defective videos.Verification involves the data company, trained verifiers, and final screening by the authors.
  • 3.1 Collection of Videos: 563 object classes, 87 motion classes, and around 10 thousand videos form GOT-10k’s final collection, although video counts are imbalanced across 121 object-class groups.The imbalance is mainly attributed to filtering during collection and verification.
  • 3.2 Annotation of Trajectories: GOT-10k uses tight bounding boxes compatible with object-detection standards, supporting applications that reuse tracking annotations for related vision tasks.This labeling differs from datasets using rotated rectangles around the target.
  • 3.2 Annotation of Trajectories: Visible-ratio labels provide seven ranges for more continuous occlusion annotation, while annotation quality is checked through one annotation stage and four verification stages.Visible ratios represent the approximate visible proportion of an object, with occluded or boundary-truncated pixels treated as invisible.
  • 3.3 Dataset Splitting: The one-shot protocol separates training and test object classes, and test-set design varies video number and class variety to improve ranking stability.With 420 videos and 84 non-overlapping test classes, the benchmark targets a stability-efficiency trade-off; increasing video count lowers rank standard deviation, with marginal gains beyond 1,000 videos.

4 EXPERIMENTS

The experiments evaluate tracker performance using overlap- and success-based metrics, compare methods across datasets and challenges, and examine how training-data diversity affects generalization. Results show substantial difficulty on unconstrained videos, strong sensitivity to object and motion conditions, and benefits from broader training diversity.

  • 4.2 Evaluation Methodology: AO averages predicted and ground-truth overlaps, while SR measures the percentage of frames whose overlap exceeds a threshold such as 0.5.AO is equivalent to the area under the curve used by several tracking benchmarks.
  • 4.2 Evaluation Methodology: Class-balanced mAO and mSR first average scores within each object class, then average across classes to prevent dominant classes from receiving higher weight.The same class-balancing principle is applied to both AO and SR.
  • 4.3 Overall Performance: 46.0% is the highest mAO score on GOT-10k, indicating that tracking in real-world unconstrained videos remains difficult for current trackers.MemTracker ranks first and exceeds the second-place tracker by 1.1% in mAO, 3.3% in mSR50, and 0.3% in mSR75.
  • 4.3 Overall Performance: Method rankings change significantly between GOT-10k and OTB2015, with ECO strongest on OTB2015 but much worse on GOT-10k, while GOTURN shows the opposite pattern.The authors suggest that overfitting to small datasets or dependence on hyperparameter tuning may contribute to these differences.
  • 4.4 Evaluation by Challenges: Performance degrades significantly as difficulty increases across all six challenge attributes, especially when visible ratios fall below 0.5 or object resolution falls below 0.3.Fast state and appearance changes, truncated targets, distractors, and very small objects remain particularly challenging.
  • 4.4 Evaluation by Challenges: GOTURN improves its ranking by 3 places on scale-variation subsets and 2 places on aspect-ratio-variation subsets compared with the full test set.ECO and CCOT perform well on fast motion and low resolution, while siamese trackers perform well on occlusion/truncation subsets.
  • 4.6 Impact of Training Data: MemTracker and SiamFCv2 increase mAO by nearly 15% when training object classes grow from 5 to 405, with trends that do not appear to converge at 405 classes.This result highlights the effect of object diversity in training data on these trackers’ generalization performance.
  • 4.6 Impact of Training Data: Training on GOT-10k improves all evaluated trackers on GOT-10k by 1.1%–2.2% in absolute AO compared with training on ImageNet-VID.On OTB2015, the effect is mixed: MemTracker and SiamFCv2 gain around 1%, whereas GOTURN and MDNet perform worse.

5 CONCLUSION

GOT-10k is a large, high-diversity, one-shot tracking database covering real-world moving objects, with extensive experiments analyzing generic tracking challenges and training-data effects.

  • GOT-10k collects over 10,000 videos spanning 563 object classes and manually annotates 1.5 million tight bounding boxes.
  • The dataset follows a one-shot protocol intended to promote generalization in tracker development.
  • The paper describes GOT-10k’s construction, diversity and quality controls, and evaluation platform for generic-purpose trackers.
  • Experiments with recent tracking approaches analyze challenges in unconstrained real-world tracking and the impact of training data on performance.
Loading 1810.11981v3…