OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning

• OneTracker unifies RGB and RGB+N/M/D/T/E tracking, whose shared objective is locating a target in the search frame from its initial appearance.Source: §1 · pp. 1–2
• Task-specific trackers require customized architectures and redundant parameters, while limited data constrains some tasks and separate designs miss unified temporal attention.Source: §1 · pp. 1–2
• Earlier unified approaches were not tracking-specific [1] [102], used RGB only [68] [87] [101], or were limited to RGB+D/T/E [108] [123].The paper says these models also fail to capture unified temporal attention mechanisms.Source: §1 · p. 2
• Foundation Tracker is pretrained on several RGB tracking datasets and gains strong localization ability for finding targets from template appearance.The paper identifies Foundation Tracker [35] and datasets [28] [45] [69].Source: §1 · p. 2
• Prompt Tracker adds CMT Prompters and TTP Transformer layers with few trainable parameters, enabling quick RGB+X fine-tuning while retaining similar speed to Foundation Tracker.CMT Prompters integrate multimodal features with RGB; TTP layers introduce parameters into Transformer linear layers to bridge RGB and RGB+X tracking.Source: §1 · p. 2
• OneTracker achieves state-of-the-art performance on 11 benchmarks spanning 6 tracking tasks.Source: §1 · p. 3

Large-scale Pretraining Vision Models

• Inspired by transferable vision foundation models, OneTracker trains Foundation Tracker on diverse tracking datasets to learn strong temporal matching.Prior vision work includes scaling ViT to 2 billion parameters, masked-image representation learning [3] [39] [99], and vision-language alignment [30] [78] [82] [89] [98].Source: §2 · p. 3

Parameter-Efficient Transfer Learning

• Existing PETL methods and RGB+D/T/E tracking prompts leave transfer to RGB+M and RGB+N unanswered; OneTracker introduces CMT Prompter and TTP Transformer for RGB+X finetuning.PETL freezes pretrained models and adds trainable parameters; examples include VPT [47], adapters [11, 14, 33, 118], and tracking methods ProTrack [108] and ViPT [123].Source: §2 · p. 3

Visual Tracking

• RGB+X tracking augments RGB with thermal, event, depth, language, or first-frame mask information, while OneTracker proposes a general approach to unify RGB and RGB+X tasks.Prior task-specific [13] [111] and multi-task trackers [1] [68] [87] [101] [102] are described as lacking multimodal tracking capability and human-like temporal matching.Source: §2 · p. 3

3.1. Tracking Unification

• OneTracker expresses RGB tracking and RGB+X tracking in a shared temporal-matching framework, with X representing language, masks, depth, thermal, or event data.RGB+M tracking uses the first-frame mask instead of an initial box; the framework combines Tracking Task Perception Transformer layers with Cross Modality Tracking Prompters.Source: §3.1 · p. 4

3.2. Foundation Tracker

• The Foundation Tracker processes template and search-frame tokens with transformer encoders, matches them temporally, and predicts bounding boxes and masks.It is pretrained on LaSOT [28], TrackingNet [69], and GOT-10K [45]; its segmentation head uses box supervision because tracking datasets lack mask labels.Source: §3.2 · p. 4

3.3. Prompt Tracker

• The Prompt Tracker embeds downstream modalities as tokens and uses CMT Prompters to fuse them with matching features, adding the fused prompts residually.BERT [23] encodes language and patch embeddings encode masks and multimodal maps; CMT uses cross-attention for RGB+N and linear fusion for other RGB+X tasks.Source: §3.3 · pp. 4–6
• TTP Transformer adapters specialize the pretrained tracker for downstream tasks while retaining its temporal-matching knowledge.Adapters are inserted into query/key/value projections and FFN output layers; during finetuning, pretrained weights are frozen and only low-rank adapter matrices are updated.Source: §3.3 · p. 6

3.4. Training and Inference

• Training has two stages: RGB pretraining followed by RGB+X finetuning with the Foundation Tracker frozen; inference procedures vary by modality without changing model structure.The second stage trains CMT Prompters and TTP adapters; RGB+M uses mask annotations instead of BoxInst supervision, and mask-based matching avoids cropping.Source: §3.4 · p. 6

4.1. Implementation Details

• The implementation uses a 12-layer ViT-B encoder, freezes Foundation Tracker during downstream tuning, and trains CMT Prompters and TTP Transformer layers for 60 epochs.Foundation Tracker is pretrained for 300 epochs; RGB+N tasks use BERT as the text encoder.Source: §4.1 · pp. 6–7

4.3. Ablation Study.

• With only 2.8M parameters adjusted, Prompt Tracker outperforms full finetuning, and ablations show gains from CMT Prompters and TTP Transformer layers.Attention visualizations show Foundation Tracker capturing temporal dependencies and Prompt Tracker further refining predictions using multimodal information.Source: Table 2 · p. 7, §4.3 · pp. 8–9, Figure 3 · p. 8
• Adding more CMT Prompter layers improves Prompt Tracker on most tasks but significantly degrades RGB+M performance, indicating mask embeddings mainly capture superficial features.Prompters were tested at 1, 2, 3, 6, and 12 transformer blocks; zero layers denotes direct addition of multimodal embeddings to RGB embeddings.Source: Table 3 · p. 7, §4.3 · pp. 8–9

4.2. Benchmark Results on 6 Tracking Tasks

• 70.5 AUC on LaSOT and 69.7 AUC on TrackingNet make Foundation Tracker the top-performing tracker on both benchmarks; it beats GRM by 0.6 AUC on LaSOT.It also exceeds several unified or perception-focused models by at least 1.3 AUC on LaSOT and can generate target masks using its segmentation head.Source: §4.2 · p. 7
• On OTB99, Prompt Tracker surpasses existing RGB+N trackers by at least 1.7 AUC and 2.5 precision, despite not being designed specifically for RGB+N.The RGB+N evaluation uses OTB99 and TNL2K.Source: §4.2 · p. 7
• Prompt Tracker greatly exceeds other trackers on DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent for RGB+D, RGB+T, and RGB+E tracking.The authors attribute its advantage over ViPT to CMT Prompters and TTP Transformer layers.Source: §4.2 · p. 7
• On DAVIS16 and DAVIS17, Prompt Tracker beats multi-task and unified trackers by a large margin despite less training data and cost, while remaining slightly behind memory-based specific models.It relies only on the first and previous frame, so it can handle videos of any length; cited memory mechanisms include [15] [16] [110].Source: §4.2 · pp. 7–8

4.3. Ablation Study.

• Separating CMT Prompters and TTP Transformer layers by modality progressively improves joint training, while task-specific training performs better because each downstream dataset is limited.The tested joint-training strategies progressively separate embedding layers, CMT Prompters, and TTP Transformer layers, then separate both latter modules.Source: Table 4 · p. 8, §4.3 · p. 9

5. Conclusion

• OneTracker achieves state-of-the-art results across 11 benchmarks spanning 6 tracking tasks.The authors report this performance as demonstrating the framework’s effectiveness and generalization ability.Source: §5 · p. 9
• OneTracker unifies several RGB and RGB+X tracking tasks in a general framework.The framework combines a Foundation Tracker pretrained on RGB tracking datasets with adaptation to downstream RGB+X tasks.Source: §5 · p. 9
• Downstream RGB+X adaptation uses prompt tuning and parameter-efficient finetuning to leverage the pretrained Foundation Tracker.Source: §5 · p. 9

6. Discussion

• OneTracker unifies RGB and RGB+N/D/T/E/M tracking and achieves competitive performance across six tasks and 11 benchmarks.Source: §6 · p. 15
• Earlier multi-task trackers considered RGB alone [1] [68] [87] [101] [102], while multimodal methods [108] [123] focused on specific modalities and kept RGB and RGB+X tracking separate.Source: §6 · p. 15
• OneTracker applies parameter-efficient transfer learning: it pretrains Foundation Tracker on large-scale RGB data, then uses multimodal prompts and CMT Prompters to enhance Prompt Tracker while freezing the foundation model.The approach builds on the similar temporal matching mechanisms of RGB and RGB+X tracking; TTP Transformer layers further adapt Prompt Tracker to downstream tasks.Source: §6 · p. 15
• About 2.8M additional parameters are adjusted in Prompt Tracker, which the authors say improves adaptability over full finetuning and suits resource-constrained end-side deployment.Source: §6 · p. 15
• Prompt Tracker must be trained separately for different RGB+X tasks, requiring task-specific adjustment of CMT Prompters and TTP Transformer layers; joint training is a future direction.The authors describe these modules as lightweight but note that separate adjustment remains inconvenient.Source: §6 · p. 15
• Although OneTracker handles six tasks, it does not yet cover other modalities, and the authors plan to extend it to more downstream tasks.Source: §6 · p. 15
• Adapting Prompt Tracker to emerging downstream tasks without sacrificing performance on existing tasks remains an open challenge.Source: §6 · p. 15

7.1. Foundation Tracker Training

• Foundation Tracker is trained using the training sets of LaSOT [28], TrackingNet [69], GOT-10K [45], and COCO [64], with horizontal-flip and brightness-jitter augmentations.Its training follows [35].Source: §7.1 · p. 15
• Foundation Tracker outperforms other trackers by at least 0.6 AUC on LaSOT with consistent training settings, and performs better than UNINEXT and OmniTracker despite their greater data and compute costs.The paper also reports an advantage of at least 1.3 AUC on LaSOT and says Foundation Tracker trains in about one day on four NVIDIA RTX 3090 GPUs.Source: §7.1 · pp. 15–16

7.2. Prompt Tracker Finetuning

• Prompt Tracker is finetuned separately for RGB+N/D/T/E tasks using 192 × 192 template and 384 × 384 search frames, with training sets chosen per modality.RGB+N uses OTB99, LaSOT [28], and TNL2K [91]; RGB+D uses DepthTrack [104], RGB+T uses LasHeR [61], and RGB+E uses VisEvent [90].Source: §7.2 · p. 16
• For RGB+M finetuning, Prompt Tracker uses DAVIS17 [76] and YouTube-VOS [97], with the first and previous frames as templates and no cropping of template or search frames.Finetuning details are listed in Table 7.Source: §7.2 · p. 16