Source-linked AI summary
LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking
Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, Haibin Ling
TL;DR
Tracking research lacks a benchmark combining large scale, long-term sequences, and precise dense annotations. LaSOT addresses this gap with manually annotated videos, language descriptions, and evaluations of existing trackers; results indicate substantial room for improvement.
Problem
Existing benchmarks do not jointly provide large-scale data, long-term sequences, and high-quality dense annotations for training and evaluating trackers.
Method
LaSOT constructs a large dense tracking benchmark with consistent manual annotations, long-term sequences, and natural-language specifications.
Results
35 tracking algorithms are evaluated on LaSOT, and the results demonstrate substantial room for improvement.
Takeaways & Limitations
LaSOT provides a dedicated platform for training deep trackers and assessing long-term tracking performance with visual and linguistic information.
Abstract
from arXiv · showhide
In this paper, we present LaSOT, a high-quality benchmark for Large-scale Single Object Tracking. LaSOT consists of 1,400 sequences with more than 3.5M frames in total. Each frame in these sequences is carefully and manually annotated with a bounding box, making LaSOT the largest, to the best of our knowledge, densely annotated tracking benchmark. The average video length of LaSOT is more than 2,500 frames, and each sequence comprises various challenges deriving from the wild where target objects may disappear and re-appear again in the view. By releasing LaSOT, we expect to provide the community with a large-scale dedicated benchmark with high quality for both the training of deep trackers and the veritable evaluation of tracking algorithms. Moreover, considering the close connections of visual appearance and natural language, we enrich LaSOT by providing additional language specification, aiming at encouraging the exploration of natural linguistic feature for tracking. A thorough experimental evaluation of 35 tracking algorithms on LaSOT is presented with detailed analysis, and the results demonstrate that there is still a big room for improvements.
1. Introduction
Existing tracking benchmarks limit progress through small scale and incomplete annotation coverage, motivating LaSOT as a large, densely annotated benchmark for training and evaluation.
- Motivation: Existing datasets seldom exceed 400 sequences, limiting tracking-specific data available for training deep trackers.Researchers consequently rely on image-classification models or video object-detection sequences for deep feature learning.
- Motivation: Dense per-frame annotations improve evaluation reliability, provide training samples, and preserve temporal context between consecutive frames.These benefits address both tracker assessment and learning temporal dynamics.
- Motivation: No existing benchmark addresses large scale, long-term tracking, and precise dense annotations simultaneously, motivating LaSOT.Prior datasets address subsets of these requirements but not all of them together.
- LaSOT: LaSOT contains 1,400 videos with an average of 2,512 frames per sequence and manually inspected frame-level labels.The benchmark generates approximately 3.52 million high-quality bounding box annotations.
- LaSOT: LaSOT additionally provides natural-language specifications and evaluates 35 representative trackers under different protocols and metrics.These additions support exploration of visual-linguistic tracking and provide baselines for future comparisons.
2. Related Work
Prior tracking benchmarks divide into densely manually annotated datasets and sparse or semi-automatically annotated datasets. LaSOT extends the dense category with substantially larger scale, long sequences, and language descriptions.
- Dense Benchmarks: Dense benchmarks provide manually inspected bounding boxes for every video frame to support precise tracker training and assessment.Examples include OTB, TC-128, VOT, NUS-PRO, UAV, NfS, and GOT-10k.
- Dense Benchmarks: LaSOT is the largest dense benchmark, with 3.52 million frames, an average sequence length of 2,512 frames, and language descriptions for each video.Other dense benchmarks do not provide the additional lingual descriptions.
- Other Benchmarks: Other benchmarks use sparse or semi-automatic annotations, including ALOV, TrackingNet, and OxUvA.ALOV labels every 5 frames, while TrackingNet annotations are produced by a tracker.
- Other Benchmarks: OxUvA targets long-term tracking with 366 videos averaging around 4,200 frames, but annotates each video only every 30 frames.This omits temporal context between consecutive frames.
- Other Benchmarks: Sparse or potentially unreliable annotations may reduce evaluation fidelity and hinder trackers that learn temporal models from annotations.LaSOT instead provides large-scale, high-quality dense bounding box annotations.
3. The Proposed LaSOT Benchmark
LaSOT is designed as a large-scale, densely annotated, long-term tracking benchmark with balanced categories, natural-language specifications, and comprehensive challenge labels. Its construction combines deterministic frame-level annotation with repeated validation and supports both tracker training and evaluation.
- Annotation: Each frame receives a manually inspected bounding box or an absent label for out-of-view and full-occlusion cases.The deterministic strategy draws the tightest upright box around visible target parts and is intended to provide consistent learning signals.
- Annotation: Special annotation rules handle highly deformable object parts, such as excluding mouse tails when they add noise and little localization information.
- Language specification: LaSOT provides one natural-language sentence for every sequence describing target color, behavior, and surroundings as auxiliary semantic guidance.The descriptions are intended to help disambiguate tracker proposals.
- Annotation: Annotation quality is improved through labeling, expert validation, and repeated revision; around 40% of initial annotations fail the first validation round.Many frames are revised more than three times before the benchmark is finalized.
- Attributes and evaluation: Fourteen attributes characterize tracking challenges, with scale variation, occlusion, deformation, and rotation among the most common; LaSOT includes more than 1,300 scale-variation videos and 477 out-of-view sequences.The benchmark also proposes two evaluation protocols: one using all 1,400 sequences and another splitting them into training and testing subsets.
4. Evaluation
LaSOT evaluation uses OPE with three metrics across two protocols and 35 trackers, exposing performance patterns, hard attributes, and remaining training challenges.
- Evaluation metrics: Precision uses pixel distance, normalized precision uses AUC from 0 to 0.5, and success uses IoU with AUC from 0 to 1.
- Evaluation setup: 35 tracking algorithms are evaluated using precision, normalized precision, and success under two protocols.
- Protocol I results: Under protocol I, MDNet leads precision at 0.374 and success at 0.413, while VITAL leads normalized precision at 0.484.
- Protocol I results: SiamFC reaches 0.341 precision, 0.449 normalized precision, and 0.358 success without time-consuming online adaptation, while running in real time.
- Attribute-based performance: Fast motion, out-of-view, and full occlusion are among the most challenging attributes, where trackers often lose targets during local-region localization.
- Qualitative evaluation: Low resolution, aspect-ratio change, and background clutter cause drift linked to weak representations, scale handling, or insufficient target-background discrimination.
- Protocol II results: Under protocol II, MDNet and VITAL remain top performers, while SiamFC ranks third and balances slightly lower accuracy with real-time speed.
- Retraining study: Retraining MDNet and SiamFC produced similar performances to evaluation without retraining, suggesting dedicated configurations may be needed for LaSOT’s long-term sequences.
5. Conclusion
The paper presents LaSOT as a high-quality benchmark for training and evaluating visual object trackers. Its two protocols and extensive evaluations indicate substantial room for improvement.
- LaSOT provides high-quality dense bounding-box annotations and natural-language specifications for visual object tracking.
- The benchmark is intended to support deep-tracker training and long-term tracking evaluation while encouraging integration of visual and linguistic features.
- Extensive evaluations under two protocols demonstrate substantial room for improvement in visual tracking.
—– Supplementary Material —–
LaSOT’s supplementary material includes details of its 70 object categories and comparisons with existing dense benchmarks.
- Table 5 summarizes LaSOT’s 70 object categories and compares them with existing dense benchmarks.
1. Details of 70 Object Categories in LaSOT and Comparison with Existing Dense Benchmarks
LaSOT contains 70 object categories, each represented by 20 videos, with categories selected for tracking usability.
- LaSOT contains 70 object categories with 20 videos per category.
- Most categories are selected from ImageNet, with exceptions such as drone and gametarget chosen by experts for tracking.
- Experts must unanimously agree on each category’s selection to ensure its usability for visual tracking.
2. Traing/Testing Split in Protocol II
Protocol II divides LaSOT into separate training and testing subsets, with similar sequence-length statistics and attribute distributions supporting split consistency.
- Dataset split: 1,120 training videos contain 2.83M frames, while 280 testing videos contain 690K frames.The split assigns 16 sequences per category to training and 4 sequences per category to testing.
- Dataset split: The training/testing split compares sequence counts, frame counts, and sequence-length statistics between the two subsets.Table 6 is identified as reporting the detailed comparison.
- Split consistency: Sequence ratios across all 14 attributes are similar between training and testing sets.Figure 9 provides the attribute-distribution comparison.
- Split consistency: Minimum, mean, median, and maximum frame counts are similar between the training and testing subsets.These comparable length statistics support consistency between the subsets.
3. Detailed Attribute-based Performance under Protocol I
Protocol I evaluates tracker performance separately across attributes using precision, normalized precision, and success measures.
- Precision: Tracker performance is shown for each attribute using precision under protocol I.Figure 10 provides the attribute-wise precision results.
- Normalized precision: Tracker performance is shown for each attribute using normalized precision under protocol I.Figure 11 provides the corresponding normalized-precision results.
- Success: Tracker performance is shown for each attribute using success under protocol I.Figure 12 provides the attribute-wise success results.
4. Detailed Attribute-based Performance under Protocol II
Protocol II evaluates tracker performance across attributes using precision, normalized precision, and success measures.
- Precision: Tracker performance is shown for each attribute using precision under protocol II.Figure 13 provides the attribute-wise precision results.
- Normalized precision: Tracker performance is shown for each attribute using normalized precision under protocol II.Figure 14 provides the corresponding normalized-precision results.
- Success: Tracker performance is shown for each attribute using success under protocol II.Figure 15 provides the attribute-wise success results.