PaperlayerView exact revision on arXiv

Source-linked AI summary

TAO: A Large-Scale Benchmark for Tracking Any Object

Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, Deva Ramanan

arXiv:2005.10356v1cs.CV

TL;DR

Existing multi-object tracking benchmarks cover limited categories and domains, motivating TAO, a large and diverse benchmark built through bottom-up vocabulary discovery and federated annotation. Across its evaluation, current trackers struggle in the wild, while detection-based multi-object trackers are competitive with user-initialized trackers. TAO consequently provides a broader basis for measuring tracking progress and raises questions about generic versus specialized tracking.

  • Problem

    Existing tracking benchmarks are small, domain-biased, and focused on few categories, providing limited coverage for evaluating tracking in diverse settings.

  • Method

    TAO combines 2,907 diverse high-resolution videos with bottom-up discovery of 833 object categories and federated annotation of relevant tracks.

  • Results

    Detection-based multi-object trackers are competitive with user-initialized trackers on a large vocabulary, while state-of-the-art trackers struggle to generalize in the wild.

  • Takeaways & Limitations

    TAO provides a broad benchmark for measuring in-the-wild tracking progress and supports evaluation of generic large-vocabulary tracking approaches.

  • Takeaways & Limitations

    TAO intentionally dedicates most videos to in-the-wild benchmark evaluation, constraining its coverage of other tracking settings.

Abstract

from arXiv · show

For many years, multi-object tracking benchmarks have focused on a handful of categories. Motivated primarily by surveillance and self-driving applications, these datasets provide tracks for people, vehicles, and animals, ignoring the vast majority of objects in the world. By contrast, in the related field of object detection, the introduction of large-scale, diverse datasets (e.g., COCO) have fostered significant progress in developing highly robust solutions. To bridge this gap, we introduce a similarly diverse dataset for Tracking Any Object (TAO). It consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. Importantly, we adopt a bottom-up approach for discovering a large vocabulary of 833 categories, an order of magnitude more than prior tracking benchmarks. To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum. Our vocabulary is both significantly larger and qualitatively different from existing tracking datasets. To ensure scalability of annotation, we employ a federated approach that focuses manual effort on labeling tracks for those relevant objects in a video (e.g., those that move). We perform an extensive evaluation of state-of-the-art trackers and make a number of important discoveries regarding large-vocabulary tracking in an open-world. In particular, we show that existing single- and multi-object trackers struggle when applied to this scenario in the wild, and that detection-based, multi-object trackers are in fact competitive with user-initialized ones. We hope that our dataset and analysis will boost further progress in the tracking community.

1 Introduction

TAO addresses the narrow, domain-biased scope of existing multi-object tracking benchmarks with a large, diverse dataset spanning many object categories and environments. Its bottom-up vocabulary discovery and federated annotation support scalable evaluation, revealing that current trackers struggle in the open-world setting.

  • Motivation: Existing tracking datasets are small, short, and concentrated on people, vehicles, and animals, limiting coverage of diverse tracking scenarios.The paper contrasts this with the broader diversity that helped drive progress in object detection benchmarks.
  • TAO benchmark: TAO samples videos from indoor, movie, outdoor, road, and in-the-wild environments to reduce domain bias.Its source datasets include Charades, AVA, LaSOT, ArgoVerse, HACS, and YFCC100M.
  • Dataset construction: Annotators discover categories bottom-up by labeling objects that move, then assign names after observing the videos.This approach avoids predefining the full category vocabulary before collection.
  • Dataset construction: Federated annotation focuses effort on up to 10 objects per video and separately verifies exhaustive labels for selected category-video pairs.Evaluation uses exhaustive-label videos for precision and all videos for other measurements.
  • Findings: Current trackers struggle with infrequent categories, diverse people-tracking conditions, and open-world generalization, while multi-object trackers can match user-initialized trackers at scale.The reported findings motivate broader evaluation beyond the narrow conditions of previous benchmarks.

2 Related work

Related tracking benchmarks differ in vocabulary, domain diversity, initialization, and annotation format. TAO extends multi-object tracking toward a larger, more diverse, labeled open-world benchmark while retaining box-based evaluation.

  • Multi-object tracking: Multi-object tracking follows an unknown number of objects from a known category set, but common benchmarks emphasize people or vehicles in limited domains.This narrow design can encourage methods that overfit benchmark scenarios.
  • Multi-object tracking: ImageNet-Vid provides 30 categories across 1,314 videos, while YTVIS covers 40 classes but remains dominated by people, vehicles, and animals.These benchmarks broaden scale or vocabulary relative to standard MOT but remain limited in category coverage or video duration.
  • Annotation formats: TAO prioritizes scale and diversity over pixel-accurate masks because box-level tracking already challenges current methods.Mask annotations are described as substantially more expensive than bounding boxes.
  • Tracking formulations: User-initialized tracking removes the fixed category vocabulary by supplying test-time bounding boxes, but its benchmarks trade off video count against average video length.This formulation differs from multi-object tracking in how tracks are initialized.
  • Open-world tracking: Saliency-based tracking defines objects through bottom-up motion or appearance cues rather than fixed categories or manual annotations.TAO similarly uses motion-based discovery but adds category labels for evaluating large-vocabulary recognition.
  • Comparative evaluation: User-initialized trackers achieve only moderate improvement over a multi-object tracking baseline on TAO despite receiving ground-truth track initialization.The comparison evaluates publicly implemented state-of-the-art methods in this paradigm.

3 Dataset design

TAO’s dataset design combines bottom-up category discovery, federated annotation, and 3D IoU-based evaluation to make diverse large-vocabulary tracking measurable at scale.

  • Design goals: TAO’s collection strategy addresses video selection, vocabulary discovery, scalable annotation, and evaluation as separate design requirements.The paper details vocabulary discovery, annotation, and evaluation in this section and defers video collection to Section 4.1.
  • Category discovery: Annotators mark every object that moves without a predefined vocabulary, then provide free-form names while using LVIS categories when possible.The resulting vocabulary is discovered from unlabeled videos spanning diverse operating domains.
  • Federation: Federated annotation replaces exhaustive labeling across every category and video with targeted track labeling plus exhaustive verification on selected subsets.This reallocates annotation effort toward a large-scale, diverse dataset.
  • Evaluation: 3D IoU compares predicted and ground-truth tracks across all video frames, assigning empty boxes when an object or detection is absent.TAO uses 3D IoU with a threshold of 0.5 as its default metric.
  • Evaluation: Track confidence combined with 3D IoU supports mean average precision across categories, with precision computed only where category labels are exhaustively verified.Frame-level scores are averaged when methods provide a score for each frame.

4 Dataset collection

TAO is constructed from diverse video sources and uses a bottom-up, federated annotation process to discover and label a broad object vocabulary at scale.

  • Video diversity: TAO samples videos across indoor, scripted film, outdoor, road, action-recognition, and in-the-wild domains to reduce single-domain bias.The collection combines Charades, AVA, LaSOT, ArgoVerse, HACS, and YFCC100M sources.
  • Federated evaluation: Federated verification records which categories are exhaustively labeled or absent in each video, enabling category-specific evaluation with reliable false-positive and missed-detection penalties.Exhaustive labels are used for precision, while absence verification identifies categories with no instances.
  • Object discovery: Annotators first mine objects that move, track them throughout each video, and label up to 10 objects per video without a predefined category search.Motion serves as a category-agnostic saliency cue, limiting annotation expenditure on crowded videos.
  • Vocabulary construction: The vocabulary is built post factum by labeling discovered tracks with LVIS categories or free-form text when LVIS lacks an appropriate category.An autocomplete interface suggests LVIS synsets, names, synonyms, and definitions.
  • Vocabulary construction: 17,038 objects are labeled: 16,144 objects use 488 LVIS categories, while 894 objects use 345 free-form categories.The 488 LVIS categories support multi-object tracking experiments; all categories are used for user-initialized tracking experiments.
  • Benchmark design: TAO is split into 500 training, 988 validation, and 1,419 test videos, prioritizing large-scale in-the-wild evaluation and held-out validation and test sets.The small training split supports hyperparameter tuning while reserving most annotation budget for evaluation.

5 Analysis of state-of-the-art trackers

The TAO analysis finds that existing trackers struggle with its large vocabulary and diverse settings, while oracle analyses expose both tracking and classification as major bottlenecks.

  • Detection: 27.1 detection mAP on TAO val at IoU 0.5 shows that single-frame detection remains challenging.The reported detector is trained on LVIS and COCO because LVIS-only training struggles to detect people.
  • Multi-object tracking: SORT outperforms Viterbi linking, but both methods achieve low performance, indicating that TAO remains a major tracking challenge.SORT is favored for its scalability to many categories and long videos, whereas Viterbi is difficult to tune at TAO scale.
  • Tracking and classification: 31.5 mAP from the track oracle versus 13.2 for SORT shows that imperfect track linking substantially limits performance.The track oracle perfectly links per-frame detections while retaining their class predictions.
  • Tracking and classification: 30.2 mAP with SORT tracks and an oracle class label shows that classification errors are another substantial limitation.The analysis treats tracking and classification improvements as orthogonal and tests their combination.
  • Tracking and classification: 83.6% mAP from combining oracle tracks with oracle class labels indicates that large-vocabulary tracking requires improving tracking and classification jointly.The authors connect this result to semantic flicker between similar classes.
  • People tracking: 18.6 versus 16.5 AP shows SORT modestly outperforming Viterbi for people, while overall SORT performance is 13.2 mAP across categories.People are easier partly because of focused research, whereas hand-held objects introduce repeated occlusions.
  • People tracking: Removing Tracktor++’s appearance re-identification and pixel-level frame alignment reduces AP from 36.7 to 25.9, a drop of over 8 points.These components account for a majority of its improvement over the evaluated baselines.
  • User-initialized tracking: User-initialized trackers provide at most modest improvements over SORT despite oracle box initialization, with benefits diminishing in longer-term tracking.The authors relate this limited generalization to training on small-vocabulary videos with limited occlusions.

6 Discussion

The discussion positions TAO as both an evaluation benchmark for tracking in the wild and a source of broader questions about generic versus specialized tracking. It also identifies video object detection as an additional use case.

  • 6 Discussion: TAO provides a robust benchmark for measuring tracking performance in-the-wild across nearly 3,000 videos.The authors argue that its analysis raises new conclusions and future research questions.
  • 6 Discussion: Large-vocabulary datasets such as TAO and LVIS allow multi-object trackers to match or outperform user-initialized tracking for some categories.
  • 6 Discussion: TAO supports evaluating both generic trackers and trackers specialized for particular scenarios or categories.The authors argue that diverse scenarios can benefit generic approaches while TAO also supports application-specific evaluation.
  • 6 Discussion: TAO can measure progress in video object detection, where systems detect objects in each frame without maintaining object identity.Its large vocabulary also offers opportunities to use temporal information to address classification errors.
  • 6 Discussion: Appendices extend the paper with analyses of annotation quality, track statistics, metrics, non-LVIS categories, and hyperparameter tuning.

A.1 Annotation diversity and quality

TAO’s annotation audit indicates both diverse object selection and high-quality labels. Re-annotation of 50 videos found many newly selected objects while most corresponding tracks showed substantial overlap.

  • A.1 Annotation diversity and quality: The re-annotation procedure treats IoU > 0.75 as evidence that a re-annotated track corresponds to an originally labeled object.
  • A.1 Annotation diversity and quality: 310 tracks from 50 re-annotated videos were used to assess annotation diversity and quality.
  • A.1 Annotation diversity and quality: 57% of re-annotated tracks overlapped original tracks with IoU > 0.75, while the remainder were newly labeled objects.The authors interpret the new objects as evidence that annotators selected a diverse set of objects.
  • A.1 Annotation diversity and quality: Only 6 tracks, or 3%, received more precise, more general, similar, or erroneous labels, supporting high class-label quality.
  • A.1 Annotation diversity and quality: Annotation quality was assessed by checking IoU and class-label agreement for the 177 corresponding tracks.The expected criterion was very high overlap, above 0.9, together with matching class labels.

A.2 Annotation statistics

TAO is analyzed through comparisons of motion, size, aspect-ratio change, track length, and dataset scale. These statistics show broad variation in object behavior and evaluation coverage, alongside split-construction safeguards.

  • A.2 Annotation statistics: Figure 5 compares TAO with MOT-17 and ImageNet-Vid using statistics of annotated tracks.
  • A.2 Annotation statistics: TAO has a more diverse distribution of aspect-ratio changes than MOT-17 and ImageNet-Vid.MOT-17 largely contains people with stable aspect ratios, while ImageNet-Vid is somewhat more varied.
  • A.2 Annotation statistics: TAO and ImageNet-Vid contain varied object sizes, whereas MOT-17 tends to contain smaller bounding boxes.
  • A.2 Annotation statistics: TAO includes varied object motions, including extremely fast motion for small objects with motion changes larger than 5.0.The distributions are plotted at 1FPS so annotation rate does not affect the comparison.
  • A.2 Annotation statistics: TAO provides short, medium, and long tracks, requiring tracking over long periods and recognition when objects are missing.The three bins correspond to less than 1/3, between 1/3 and 2/3, and greater than 2/3 of video length.
  • A.2 Annotation statistics: TAO contains nearly 5x as many tracks per video as standard user-initialized tracking datasets.The authors also emphasize its larger category and video coverage for evaluating diverse scenarios.
  • A.3 Split construction: TAO’s split construction keeps related videos together by source user, channel, driving sequence, worker, or movie, depending on the dataset.

B Metrics

The paper analyzes 3D IoU-based mAP as an interpretable tracking metric and relates it to identity swaps. The metric rewards sustained, precise tracking and explicitly evaluates confidence.

  • B Metrics: The metrics appendix analyzes 3D IoU, MOT Challenge metrics, and per-category AP for SORT.
  • B Metrics: 3D IoU is correlated with identity swaps, a key tracking metric measured by MOT Challenge metrics.
  • B Metrics: At an IoU threshold of 0.5, trackers receive no partial credit for tracking an object for only a short portion of its track.Trackers that cover 30% and 5% of a track can receive the same mAP under this threshold.
  • B Metrics: The 3D IoU mAP metric evaluates confidence through the precision-recall curve, unlike MOTA and ID-F1.
  • B Metrics: Partial credit can be introduced by averaging mAP across multiple IoU thresholds, following the COCO evaluation approach.

B.2 MOTA results

The appendix adapts MOT metrics to TAO’s federated, 833-category setting and finds conclusions broadly consistent with mAP, with important differences for user-initialized tracking and person performance.

  • Metric adaptations: MOTA and ID-F1 count false positives only in videos with exhaustive category annotations, providing a simple federated-dataset adaptation that may overestimate performance.The modified metrics average MOTA and ID-F1 across categories, while counters are summed across categories.
  • Overall comparison: The mAP-based conclusions remain consistent with MOTA experiments overall, with exceptions involving user initialization and the person category.MOTA is tuned on TAO train and evaluated on validation for multi-object, user-initialized, and person-tracking settings.
  • User initialization: User-initialized trackers significantly improve over SORT on MOTA and ID-F1, unlike mAP, because these metrics give partial credit for short tracking periods.At IoU 0.1, DIMP reaches 71.0 mAP versus 36.9 for SORT, reinforcing that initialization helps mainly over shorter periods.
  • Person tracking: MOTA-person is 54.8 versus 6.7 MOTA-overall for SORT, while mAP is 18.5 versus 13.2, because MOT metrics reward detection accuracy more heavily.Person detectors substantially outperform other category detectors on average, producing the large MOTA-person advantage.
  • Tracker robustness: Tracktor++ gains significantly on TAO from re-identification and camera-motion compensation, suggesting TAO rewards robustness to occlusion and camera motion.The comparison is made against Tracktor++ performance on MOT-17.

B.3 AP per category

Per-category SORT accuracy is uneven across TAO, with many categories receiving zero AP and individual-category AP remaining noisy under federated annotation.

  • Per-category accuracy: 180 TAO categories receive 0 AP from SORT, so Figure 7 plots only categories with non-zero AP.The omitted categories are excluded for conciseness rather than removed from the benchmark.
  • Per-category accuracy: Individual-category AP can be noisy because TAO uses a federated annotation setup.This noise is noted as a property of per-category evaluation.

C Additional tracking results

Additional experiments examine user-initialized trackers on TAO categories outside LVIS and show the broad per-category SORT distribution, including substantial zero-AP coverage.

  • Non-LVIS categories: User-initialized trackers are evaluated on TAO’s remaining non-LVIS categories because they do not require object detectors.The appendix reports these results in Table 12 and generally finds consistency with results on LVIS categories.
  • Per-category AP: SORT achieves 0 AP for 180 categories, while Figure 7 omits those categories and displays the remaining per-category AP values.Individual-category AP may be noisy under federated annotation.

C.2 Hyperparameter tuning

The appendix tunes tracker, detector, and confidence-threshold parameters on TAO, finding especially strong sensitivity for SORT and user-initialized trackers while documenting detector-training choices.

  • Score thresholds: TAO tuning defines detection, tracker per-frame, and track scores, with track scores ranking tracks for mAP and thresholding reported tracks for MOTA.Detection scores precede tracking; tracker per-frame scores are produced after tracking.
  • SORT: SORT improves when max age increases, min iou and min hits decrease, and more detections are output per image, reflecting TAO’s long, high-motion sequences.Lowering the score threshold from 0.1 to 0.0005 improves accuracy from 8.2 to 11.3, a 2.1-point gain.
  • Viterbi: Viterbi gains only a small improvement from post-hoc per-frame score tuning, increasing from 8.5 to 9.0 without internal-parameter tuning.Internal tuning was avoided because processing TAO train required over a week on four machines.
  • Tracktor++: Tracktor++ is relatively robust to detector-threshold tuning, unlike SORT, potentially because detections are reused as proposals for subsequent frames.The appendix reports this comparison from TAO train threshold experiments.
  • User-initialized trackers: User-initialized trackers require a confidence threshold to report object absence, and tuning it can change accuracy by 5.2% for DIMP.The optimal threshold varies across trackers; DIMP’s example compares 0.5 with the default of 0.
  • Detector: The tracking experiments use a Mask R-CNN ResNet-101 detector retrained jointly on LVIS and COCO, with duplicate COCO annotations removed above IoU 0.7.Joint LVIS+COCO training noticeably improves detection quality, particularly for people, relative to LVIS alone.
Loading 2005.10356v1…