Source-linked AI summary
LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking
Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, Dengdi Sun
TL;DR
RGBT tracking lacks a large-scale, high-diversity benchmark needed for deep-tracker training and comprehensive evaluation. LasHeR provides densely annotated aligned and unaligned multimodal data and evaluates 12 trackers, revealing substantial difficulty and consistently low performance on several newly introduced challenges.
Problem
RGBT tracking lacks a large-scale, high-diversity benchmark suitable for training deep trackers and comprehensively evaluating methods.
Method
The paper constructs LasHeR with aligned, densely annotated visible–thermal pairs, releases an unaligned version, and evaluates 12 RGBT trackers.
Results
Trackers show significant performance degradation on LasHeR relative to existing datasets and consistently low performance on its newly introduced challenges.
Takeaways & Limitations
LasHeR provides a challenging benchmark for comprehensive RGBT tracking evaluation and for studying alignment-free tracking.
Abstract
from arXiv · showhide
RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a Large-scale High-diversity benchmark for RGBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and densely annotated. LasHeR is highly diverse capturing from a broad range of object categories, camera viewpoints, scene complexities and environmental factors across seasons, weathers, day and night. We conduct a comprehensive performance evaluation of 12 RGBT tracking algorithms on the LasHeR dataset and present detailed analysis to clarify the research room in RGBT tracking. In addition, we release the unaligned version of LasHeR to attract the research interest for alignment-free RGBT tracking, which is a more practical task in real-world applications. The datasets and evaluation protocols are available at: https://github.com/BUGPLEASEOUT/LasHeR.
I. INTRODUCTION
Existing RGBT benchmarks are limited in scale, diversity, imaging setups, and real-world challenges. LasHeR addresses these gaps with densely annotated aligned data, broad diversity, an unaligned release, and comprehensive tracker evaluation.
- Benchmark gaps: Existing RGBT datasets contain fewer than 300 sequences, limiting the potential of deep RGBT tracking models.The paper notes task differences and the small size of GTOT can lead to suboptimal evaluation performance.
- Benchmark gaps: Single imaging platforms restrict dataset diversity through repeated imaging parameters, fixed viewpoints, and limited scenes or categories.Examples include tripod-mounted or turnable platforms with constrained capture conditions.
- Benchmark gaps: Existing datasets provide limited scenes and categories and omit challenges such as occlusion, frame loss, illumination variation, out-of-view targets, similar appearance, and aspect-ratio change.These omissions weaken the basis for reliable evaluation under real-world conditions.
- LasHeR contributions: LasHeR contains 1224 aligned visible–thermal video pairs and more than 730K densely annotated frame pairs.Every frame pair is manually annotated with a bounding box.
- LasHeR contributions: LasHeR captures diverse object categories, viewpoints, scene complexities, seasons, weather conditions, and day–night settings.The dataset is intended to support practical tracking research.
- LasHeR contributions: The authors release an unaligned LasHeR version and evaluate 12 RGBT trackers to support alignment-free tracking and clarify the field’s research space.The unaligned task is described as more practical for real-world applications.
II. RELATED WORK
Related work covers established RGBT tracking benchmarks and notes that commonly used real datasets remain limited in sequence count, categories, and challenge diversity.
- Scope: The review frames benchmark datasets and current RGBT tracking methods as the two main areas of related work.Method details are referred to a separate survey.
- RGBT tracking datasets: Early benchmarks include OSU-CT with six sequences, LITIV with nine, GTOT with 50, and larger datasets such as RGBT210 and RGBT234.These datasets introduced increasing scale and challenge annotations across successive benchmarks.
- RGBT tracking datasets: The independent sequence count across RGBT210, RGBT234, and VOT-RGBTIR2019 does not exceed 234 because the datasets are inclusive.The paper also questions whether pseudo-RGBT data generated by pix2pix provides complementary thermal information.
B. RGBT Tracking Methods
RGBT tracking methods span sparse representation, graph-based, correlation-filter, and deep learning approaches that exploit complementary visible and thermal information.
- Method taxonomy: RGBT tracking methods are broadly divided into sparse representation-based, graph-based, correlation-filter-based, and deep learning-based methods.The taxonomy organizes the subsequent review of existing approaches.
- Sparse representation: Sparse representation methods use modality weights or discriminative learning to adaptively fuse modalities and suppress noise or errors.These methods are described within Bayesian filtering and discriminative learning frameworks.
- Graph-based methods: Graph-based methods represent both modalities collaboratively and use node weighting, global relations, or cross-modal ranking to address interference and heterogeneity.The reviewed methods also consider noisy labels and seed noise.
- Correlation filters: Correlation-filter methods target efficient fusion while modeling modality consistency, heterogeneity, or low-rank structure.The review includes soft-consistency, jointly learned, and discriminative fusion correlation filters.
- Deep learning methods: Deep methods include fast Siamese trackers, robust but less efficient MDNet-based fusion, and newer DiMP- and SSD-based frameworks.Siamese methods may be weak on low-resolution objects, while MDNet capacity is limited by its VGG backbone.
III. LASHER BENCHMARK
LasHeR is constructed as a large-scale, diverse RGBT benchmark with multiple imaging platforms, spatial alignment, dense annotations, and expanded challenge coverage.
- Dataset scale: LasHeR contains 1224 RGBT sequence pairs and approximately 730K frames to address deep trackers’ data demands.The dataset is positioned against existing smaller RGBT benchmarks.
- Data collection: Multiple imaging platforms are used to overcome the diversity limits imposed by single-platform capture.The design targets variation in imaging parameters, viewpoints, and scenes.
- Multi-modal alignment: LasHeR spatially aligns visible and thermal images because its platforms collect multimodal videos at the same frame rate.The processing pipeline begins by aligning the two modalities.
- Challenge attributes: LasHeR introduces seven new challenge attributes, including hyaline occlusion, illumination changes, similar appearance, aspect-ratio change, out-of-view, and frame loss.The full benchmark lists 19 attributes and marks the seven newly introduced ones.
- Annotation: Each frame is manually checked and fine-tuned with a minimum target bounding box, while out-of-view targets receive zero width and height.Strict alignment allows one modality’s annotation to serve both modalities.
D. Data Statistics
LasHeR is designed to be diverse across object categories, scenes, imaging conditions, and real-world challenges. Its statistics include 32 target-object types, more than 20 scene types, and newly labeled challenges such as hyaline occlusion and frame lost.
- Diversity: LasHeR spans broad object categories, camera viewpoints, scene complexities, and environmental factors across seasons, weather, day, and night.
- Scene type: More than 20 indoor and outdoor scene types contribute varied scene complexity to LasHeR.
- Object category: 32 target-object types include rigid and non-rigid objects, with a long-tail class distribution.
- Challenge: LasHeR labels seven additional challenges, including hyaline occlusion, high illumination, abrupt illumination variation, similar appearance, aspect ratio change, out-of-view, and frame lost.
- Challenge: Figure 4 compares attribute distributions across all LasHeR sequences and attributes common to RGBT234 and LasHeR.
- Challenge: Hyaline occlusion makes targets invisible to thermal imaging but visible in visible imagery, while frame lost occurs when thermal-camera correction temporarily stops imaging.
E. Unaligned LasHeR
The unaligned LasHeR release targets alignment-free RGBT tracking, where RGB and thermal frames do not share ground truths. The dataset derives unaligned boxes from accurately annotated aligned pairs using SIFT-based homography estimation and modality transformations.
- Alignment-free RGBT tracking is motivated by the cost, expense, and unavoidable errors of aligning RGB and thermal imaging systems.
- Unaligned LasHeR provides separate ground truths for RGB and thermal frames without requiring a second manual annotation process.
- SIFT matches between aligned and unaligned RGB images are used to estimate homography transformations for generating unaligned ground-truth boxes.
- The data-creation pipeline captures unaligned pairs, locally aligns them around targets, annotates the aligned dataset, and transforms those annotations back to unaligned pairs.
F. Evaluated Trackers
LasHeR evaluates a broad set of RGBT trackers using standard precision, normalized precision, and success metrics under whole-dataset and train/test protocols. The protocols support both comprehensive assessment and training of deep trackers.
- The benchmark evaluates 12 RGBT tracking algorithms, including deep-model and handcrafted-feature trackers.
- Metrics: Precision rate measures the percentage of frames whose predicted position is within 20 pixels of the ground truth.
- Metrics: Normalized precision reduces sensitivity to image resolution and bounding-box size.
- Metrics: Success rate measures the proportion of frames exceeding an overlap threshold, summarized by area under the curve.
- Protocols: Two protocols evaluate trackers on the entire dataset or on a testing subset after a training/testing split, with retraining on the training subset.
A. Overall Evaluation Results
On the full 1,224-sequence LasHeR test set, DMCNet leads precision and normalized precision while mfDiMP leads success, but overall performance is substantially lower than on earlier datasets. Challenge-specific results show severe difficulty with occlusion, disappearance, deformation, and newly introduced attributes.
- Overall results: DMCNet achieves the best precision score of 0.557 and normalized precision score of 0.494, while mfDiMP achieves the top success score of 0.399.
- Overall results: mfDiMP uses an offline DiMP-based classification model for faster testing, whereas DMCNet's online MDNet-based training has higher computational complexity.
- Challenge-based results: Total occlusion, hyaline occlusion, and out-of-view commonly cause trackers to lose targets, producing low precision and success scores.
- Challenge-based results: Deformation and aspect-ratio changes degrade performance because trackers struggle to model drastically changing target appearance.
- Challenge-based results: Trackers consistently perform poorly on seven newly introduced challenges, especially hyaline occlusion, frame lost, and aspect ratio change.
C. Qualitative Evaluation
Qualitative and testing evaluations show that LasHeR exposes substantial tracking difficulty, while retraining on its large-scale data improves deep RGBT trackers. The testing protocol establishes challenging baselines and the retraining results support the dataset’s training value.
- Qualitative Evaluation: Most evaluated trackers struggle with aspect-ratio changes and scale variation, while mfDiMP handles scale variation best.The authors attribute mfDiMP’s scale-handling ability to DiMP’s model-update mechanism.
- Testing Protocol: The second protocol splits LasHeR into training and testing subsets according to target-class distribution.Training sequences are used to train trackers, which are then assessed on the testing subset.
- Testing Protocol: 0.490 precision, 0.431 normalized precision, and 0.355 success are DMCNet’s scores on the LasHeR testing subset.All trackers perform significantly worse than on existing datasets.
- Retraining Experiment: Retraining MANet and mfDiMP on LasHeR improves their performance, with retrained mfDiMP exceeding the unretrained DMCNet on LasHeR.The comparison also reports gains over mfDiMP trained on GTOT and a larger synthetic RGBT dataset.
- Retraining Experiment: A synthetic training set nearly nine times larger than LasHeR does not improve mfDiMP performance, whereas LasHeR training improves it.The authors therefore emphasize the value of large-scale real RGBT data for tracker training.
E. Evaluation of RGB Trackers on LasHeR
RGB trackers perform substantially worse on LasHeR than on LaSOT and TrackingNet, indicating that LasHeR is more challenging. However, mfDiMP retrained on LasHeR’s RGBT data outperforms the best RGB tracker, highlighting the complementary value of RGBT data.
- Evaluation of RGB Trackers on LasHeR: RGB tracking performance on LasHeR is greatly lower than on LaSOT and TrackingNet, suggesting that LasHeR is more challenging.The comparison covers TransT, SiamFC++, PrDiMP, DiMP, and ATOM variants.
- Evaluation of RGB Trackers on LasHeR: TransT performs best among the evaluated RGB trackers on the LasHeR testing set.The authors relate the strong RGB-tracker performance to training on large-scale datasets compared with the small-scale training of RGBT trackers.
- Evaluation of RGB Trackers on LasHeR: mfDiMP retrained on the LasHeR training set greatly outperforms TransT, demonstrating the complementary benefits of RGBT data for visual tracking.The comparison uses results from Tables V and VII.
V. CONCLUSION
The paper concludes that LasHeR is a large-scale, densely annotated, multichallenge benchmark designed to address the training and evaluation limitations of existing RGBT datasets. Its evaluation framework targets scenarios closer to real-world tracking applications.
- V. CONCLUSION: LasHeR is presented as the largest RGBT tracking dataset with high-quality dense bounding-box annotations at the time of publication.The conclusion positions this scale as addressing deep trackers’ data requirements.
- V. CONCLUSION: The LasHeR testing-set evaluation includes RGB-tracker results alongside RGBT-tracker evaluation.Table VII reports evaluation results for RGB trackers on the LasHeR testing set.
- V. CONCLUSION: LasHeR is intended to address existing datasets’ limited coverage of real-world challenges and scenarios.The conclusion describes the dataset as large-scale and multichallenge.