Source-linked AI summary
LaSOT: A High-quality Large-scale Single Object Tracking Benchmark
Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, Yong Xu, Chunyuan Liao, Lin Yuan, Haibin Ling
TL;DR
Existing tracking benchmarks limit deep representation learning and authentic real-world evaluation. LaSOT addresses these gaps with a large, densely annotated benchmark and flexible protocols, while experiments on 48 trackers show significant room for improvement.
Problem
Existing tracking benchmarks are often too small, short-term, limited in object categories, or insufficiently densely annotated for deep representation learning and authentic real-world evaluation.
Method
LaSOT provides a large-scale single-object tracking benchmark with manually verified annotations, language specifications, and full-overlap and one-shot evaluation protocols.
Results
Experiments assessing 48 trackers on LaSOT indicate significant room for future improvement, while one-shot evaluation gives LTMU a SUC score of 0.414.
Takeaways & Limitations
LaSOT offers a dedicated platform for developing deep trackers and evaluating long-term tracking performance.
Takeaways & Limitations
Attribute evaluation remains uncertain because video sequences may contain multiple challenge attributes, making it difficult to identify which attribute causes failure.
Abstract
from arXiv · showhide
Despite great recent advances in visual tracking, its further development, including both algorithm design and evaluation, is limited due to lack of dedicated large-scale benchmarks. To address this problem, we present LaSOT, a high-quality Large-scale Single Object Tracking benchmark. LaSOT contains a diverse selection of 85 object classes, and offers 1,550 totaling more than 3.87 million frames. Each video frame is carefully and manually annotated with a bounding box. This makes LaSOT, to our knowledge, the largest densely annotated tracking benchmark. Our goal in releasing LaSOT is to provide a dedicated high quality platform for both training and evaluation of trackers. The average video length of LaSOT is around 2,500 frames, where each video contains various challenge factors that exist in real world video footage,such as the targets disappearing and re-appearing. These longer video lengths allow for the assessment of long-term trackers. To take advantage of the close connection between visual appearance and natural language, we provide language specification for each video in LaSOT. We believe such additions will allow for future research to use linguistic features to improve tracking. Two protocols, full-overlap and one-shot, are designated for flexible assessment of trackers. We extensively evaluate 48 baseline trackers on LaSOT with in-depth analysis, and results reveal that there still exists significant room for improvement. The complete benchmark, tracking results as well as analysis are available at http://vision.cs.stonybrook.edu/~lasot/.
1 Introduction
Existing tracking benchmarks are limited in scale, video duration, annotation quality, and category diversity, motivating LaSOT as a dedicated large-scale benchmark with flexible evaluation protocols.
- Model-free generic object tracking locates a target sequentially from its bounding box in the initial frame.
- Most existing datasets contain fewer than 400 videos, limiting tracking-specific deep representation learning and extensive evaluation of tracker generality.Researchers consequently use ImageNet-pretrained models or video-object-detection sequences, which may differ intrinsically from tracking.
- Existing short-term benchmarks average fewer than 600 frames, and targets almost always remain visible, limiting real-world performance assessment.
- LaSOT provides 85 object categories, 1,550 videos, and more than 3.87M densely annotated frames, making it the largest benchmark of its kind.Each frame is manually labeled and checked, while equal numbers of videos per class reduce category bias.
- LaSOT averages around 2,500 frames per video, enabling long-term tracking assessment beyond shorter existing benchmarks.Its shortest sequence has 1,000 frames and its longest has 11,397 frames.
- LaSOT adds natural-language specifications and full-overlap and one-shot protocols for evaluating trackers on seen and unseen categories.The one-shot protocol uses training and testing categories with zero overlap.
2 Related Work
Prior tracking benchmarks span dense, sparse, and semi-automatic annotation strategies, but differ in scale, sequence length, category coverage, and suitability for training or long-term evaluation. LaSOT is presented as a large dense benchmark with long sequences, two evaluation protocols, and language specifications.
- Dense benchmarks: Dense benchmarks provide manually inspected, per-frame bounding-box annotations for training and evaluating tracking algorithms.Other benchmarks use sparse or semi-automatic annotation, which can limit detailed per-frame evaluation and temporal information.
- Dense benchmarks: Existing benchmarks vary substantially in scale and focus, including specialized datasets for depth, human and rigid objects, UAVs, high-frame-rate video, and motion trajectories.GOT-10k contains 9,695 videos and introduced one-shot evaluation, while other benchmarks target narrower settings.
- Dense benchmarks: LaSOT is a dense benchmark with more than 3.87 million frames, approximately 2,500-frame videos, language specifications, and two protocols: full-overlap and one-shot.It targets long-term tracking, unlike short-term GOT-10k, and is described as the only listed benchmark offering language specifications for each sequence.
- Other tracking benchmarks: Sparse or semi-automatic benchmarks reduce annotation cost but may omit detailed per-frame performance information and provide imprecise temporal or motion cues.TrackingNet averages fewer than 500 frames, while OxUvA averages more than 4,200 frames but is labeled every 30 frames.
- Other tracking benchmarks: Video object detection datasets are large but are not ideally suited to tracking because targets may be static, initially out of view, or sparsely annotated.VID contains more than two million frames and YT-BB more than five million frames, yet both differ from tracking requirements.
- Other vision benchmarks: Large-scale benchmarks in classification, detection, segmentation, and multi-object tracking demonstrate the broader role of benchmark scale in advancing vision research.Examples include ImageNet, COCO, ADE20K, Cityscapes, LVIS, MOT Challenge, and TAO.
3 The LaSOT Benchmark
LaSOT is a large-scale, densely annotated tracking benchmark designed for training and evaluation, with long videos, balanced categories, rich challenge labels, and flexible language-assisted analysis.
- Dataset composition: 2,502 frames is LaSOT’s average video length, supporting long-term tracking evaluation with targets that may disappear and reappear.Sequences range from 1,000 to 11,397 frames, and videos were selected from YouTube to provide diverse, in-the-wild footage.
- Annotation protocol: Each frame receives a manually edited axis-aligned bounding box when the target is visible, or a full-occlusion or out-of-view absence label otherwise.The protocol is intended to improve evaluation accuracy and support occlusion- and out-of-view-aware trackers.
- Additional annotations: LaSOT adds one language description per sequence, describing target color, behavior, and surroundings for potential linguistic guidance in tracking.The benchmark contains 1,550 such sentences, which may help suppress background distractors through global semantic guidance.
- Quality assessment: Around 40% of initial annotations fail the first validation round, and many frames are revised at least three times before finalization.Two-team labeling and validation, including expert review and correction, are used to produce high-quality dense annotations.
4.1 Evaluation Metric
LaSOT evaluates trackers with One-Pass Evaluation under two protocols using precision, normalized precision, and success metrics. The metrics capture center-location accuracy, scale-normalized accuracy, and overlap-based tracking success.
- Evaluation metrics: One-Pass Evaluation measures tracker performance with precision, normalized precision, and success under two protocols.The protocols are full-overlap and one-shot.
- Evaluation metrics: Precision compares ground-truth and predicted box-center distances in pixels at a chosen threshold, such as 20 pixels.Because it ignores object scale, precision is sensitive to target size and image resolution.
- Evaluation metrics: Normalized precision normalizes center-distance precision with object scales to improve consistency across different target sizes.It is adopted to address precision’s sensitivity to target scale and image resolution.
- Evaluation metrics: Success is the ratio of frames whose predicted and ground-truth boxes have IoU above a predefined threshold, typically 0.5.The ratio is computed over all frames in a sequence.
4.2 Evaluated Tracking Algorithms
The benchmark evaluates 48 tracking algorithms using their original published configurations, providing a broad baseline set while avoiding potentially unfair or suboptimal retraining choices.
- Evaluated algorithms: 48 tracking algorithms are evaluated to establish baselines for future comparisons on LaSOT.The evaluated approaches include deep trackers and correlation trackers with handcrafted features.
- Evaluated algorithms: The evaluated trackers are summarized by representation schemes and search strategies in chronological order.This organization supports comparison across different tracker designs.
- Evaluation protocol: Each tracker is tested as reported in its original paper rather than being uniformly retrained for LaSOT.The authors cite differing training requirements and the risk that inappropriate settings could reduce some trackers’ performance.
4.3 Evaluation with Full Overlap Protocol
Under the full overlap protocol, DiMP achieves the strongest overall and attribute-based performance, while tracker strengths vary across long-term challenges such as out-of-view and fast motion.
- Overall performance: 0.563 PRE, 0.642 N-PRE, and 0.560 SUC make DiMP the top full-overlap tracker.LTMU ranks second with 0.535 PRE, 0.621 N-PRE, and 0.539 SUC; DaSiamRPN ranks third.
- Tracker design observations: All top 18 trackers use deep features, and trackers with online model updates outperform several trackers without updates.The comparison includes DiMP and LTMU with updates versus GlobalTrack, SiamRPN++, and SiamMask without updates.
- Attribute-based evaluation: DiMP achieves the best SUC performance on 13 of 14 attributes, while LTMU ranks second on 11 of 14.GlobalTrack and LTMU outperform DiMP on out-of-view, indicating an advantage for full-image search.
- Qualitative evaluation: Fast motion, occlusion, low resolution, out-of-view, aspect-ratio change, and background clutter remain difficult tracking conditions.Fast motion can induce blur that harms re-localization, while clutter causes drift and rotation complicates scale estimation.
4.4 Evaluation with One-Shot Protocol
The one-shot protocol evaluates generalization to 150 sequences from 15 unseen classes. LTMU leads overall, while attribute results and qualitative examples expose persistent challenges in unfamiliar targets and difficult conditions.
- Protocol and overall performance: 150 sequences from 15 classes form the one-shot test set, with none of those classes appearing in training or ImageNet.This protocol therefore assesses performance on unseen object categories.
- Overall performance: 0.473 PRE, 0.499 N-PRE, and 0.414 SUC make LTMU the top one-shot tracker.DiMP ranks second with 0.451 PRE, 0.476 N-PRE, and 0.392 SUC; ATOM ranks third with 0.43 PRE, 0.459 N-PRE, and 0.376 SUC.
- Attribute-based evaluation: DiMP achieves the best results on 10 of 14 attributes, but GlobalTrack outperforms DiMP on out-of-view.GlobalTrack benefits from its global search strategy in that challenge.
- Qualitative evaluation: Fast motion, full occlusion, low resolution, rotation, background clutter, and scale variation cause trackers to drift or lose targets.The reported remedies include mining motion features and exploiting more spatial target detail.
4.5 Retraining on LaSOT
Retraining SiamFC and CFNet on LaSOT sequences improves their success scores across evaluated benchmarks and protocols. The gains are larger on smaller OTB datasets than on LaSOT test sets.
- Cross-benchmark comparison: On OTB-15, retraining raises the reported score from 0.568 to 0.593 and 0.598.The experiments retain the original training settings while substituting LaSOT sequences for VID.
- Cross-benchmark comparison: Performance gains are larger on OTB-13 and OTB-15 than on LaSOT test sets.The paper attributes this difference partly to LaSOT’s more challenging long-term sequences and the original trackers’ short-term design.
5 Discussion
Discussion results show that one-shot evaluation is harder because it requires generalization to unfamiliar categories, while deeper features, larger search regions, and model updates are central directions for long-term tracking.
- Protocol comparison: 0.037-0.18 performance drops occur for all trackers when moving from full overlap to one-shot evaluation.The paper links this degradation to an unresolved domain gap between object categories.
- Short-term and long-term tracking: DiMP leads full-overlap evaluation, whereas LTMU leads one-shot evaluation with 0.414 SUC versus DiMP’s 0.392 SUC.The paper associates LTMU’s one-shot advantage with global re-localization for small targets and fast motion.
- Future directions: The proposed improvement directions are deeper feature representations, larger search regions, and continued use of model updating.These choices target background discrimination, occlusion or out-of-view handling, and robust long-term tracking.
- Deeper feature representation: SiamRPN++ success rises from 0.433 with AlexNet to 0.495 with ResNet-50 under full overlap.DiMP also improves from 0.534 with ResNet-18 to 0.560 with ResNet-50.
- Deeper feature representation: Using AlexNet instead of ResNet-18 causes a 0.039 SiamRPN++ drop under full overlap and a 0.071 drop under one-shot evaluation.The larger one-shot degradation indicates greater importance of deeper representations for unseen targets.
- Model update: Trackers with model updates outperform listed trackers without updates on both protocols.With updates, the cited SUC scores are 0.560, 0.539, 0.515, and 0.499 under full overlap, versus 0.517, 0.495, 0.467, and 0.455 without updates.
6 Conclusion
LaSOT provides a large-scale benchmark with language specifications and flexible evaluation protocols for developing and assessing single-object trackers. Experiments on 48 trackers indicate substantial room for improvement.
- 1,550 videos and more than 3.87 million frames make LaSOT a large-scale, precisely annotated platform for tracker development and evaluation.The benchmark is presented as the largest tracking benchmark in terms of precisely annotated frames.
- Language specifications for each sequence support exploration of linguistic features for improving tracking performance.
- Full-overlap and one-shot protocols provide flexible settings for evaluating trackers.
- 48-tracker experiments indicate significant room for future improvement in visual tracking.