Source-linked AI summary
Single-Model and Any-Modality for Video Object Tracking
Zongwei Wu, Jilai Zheng, Xiangxuan Ren, Florin-Alexandru Vasluianu, Chao Ma, Danda Pani Paudel, Luc Van Gool, Radu Timofte
TL;DR
Video tracking needs auxiliary modalities because RGB trackers can struggle in challenging conditions, yet heterogeneous representations, missing modalities, and limited multimodal data impede single-model unification. Un-Track learns a shared embedding from RGB-X pairs using low-rank factorization and prompting within one transformer architecture. Across five datasets, it surpasses unified and modality-specific trackers, with a +8.1 absolute F-score gain on DepthTrack at modest computational overhead.
Problem
Auxiliary modalities can improve RGB tracking in difficult conditions, but heterogeneous representations, missing modalities, scarce multimodal datasets, and modality-specific tuning hinder one unified model.
Method
Un-Track learns a shared embedding across modalities from RGB-X pairs using low-rank factorization, reconstruction, and cross-modal prompting in a unified transformer architecture.
Results
Un-Track surpasses unified and modality-specific trackers across five datasets and achieves a +8.1 absolute F-score gain on DepthTrack over the RGB baseline.
Takeaways & Limitations
A single parameter set can support cross-modality alignment and any-modality tracking while adding only +2.14 GFLOPs and +6.6M parameters over the RGB baseline.
Takeaways & Limitations
The evaluation focuses on settings where only one auxiliary modality is available at a time, using modality pairs such as RGB-D, RGB-T, or RGB-event.
Abstract
from arXiv · showhide
In the realm of video object tracking, auxiliary modalities such as depth, thermal, or event data have emerged as valuable assets to complement the RGB trackers. In practice, most existing RGB trackers learn a single set of parameters to use them across datasets and applications. However, a similar single-model unification for multi-modality tracking presents several challenges. These challenges stem from the inherent heterogeneity of inputs -- each with modality-specific representations, the scarcity of multi-modal datasets, and the absence of all the modalities at all times. In this work, we introduce Un-Track, a Unified Tracker of a single set of parameters for any modality. To handle any modality, our method learns their common latent space through low-rank factorization and reconstruction techniques. More importantly, we use only the RGB-X pairs to learn the common latent space. This unique shared representation seamlessly binds all modalities together, enabling effective unification and accommodating any missing modality, all within a single transformer-based architecture. Our Un-Track achieves +8.1 absolute F-score gain, on the DepthTrack dataset, by introducing only +2.14 (over 21.50) GFLOPs with +6.6M (over 93M) parameters, through a simple yet efficient prompting strategy. Extensive comparisons on five benchmark datasets with different modalities show that Un-Track surpasses both SOTA unified trackers and modality-specific counterparts, validating our effectiveness and practicality. The source code is publicly available at https://github.com/Zongwei97/UnTrack.
1. Introduction
Video tracking benefits from auxiliary modalities in difficult conditions, but their heterogeneous representations, incomplete pairings, and modality-specific designs make single-model unification challenging. Un-Track addresses these constraints with a shared embedding learned from RGB-X pairs and cross-modal prompting within one architecture.
- Motivation: Depth, thermal, and event inputs improve tracking reliability when RGB struggles with occlusion, low visibility, or fast motion.These auxiliary modalities complement RGB in challenging scenarios.
- Motivation: Modality-specific designs and fine-tuning produce multiple parameter sets, limiting adaptability and practicality across diverse applications.Existing uniform architectures still require modality-specific fine-tuning.
- Motivation: Missing modalities, scarce multimodal datasets, and absent paired combinations make joint learning across all possible inputs infeasible.These constraints prevent straightforward training with complete modality combinations.
- Un-Track: Un-Track learns a cohesive shared embedding from RGB-X pairs using low-rank factorization and reconstruction to align heterogeneous modalities.The method does not require all modalities to co-occur during training.
- Un-Track: +8.1 absolute F-score gain on DepthTrack accompanies only +2.14 GFLOPs and +6.6M parameters over the RGB baseline.The RGB baseline has 21.50 GFLOPs and 92M parameters; cross-modal features are used as prompts for token recovery.
2. Related Works
Multimodal tracking extends transformer-based and RGB tracking approaches with depth, thermal, or event inputs to address difficult visual conditions. However, modality-specific components and fine-tuning prevent a practical unified model, while current datasets rarely contain all modal combinations.
- Multimodal tracking: Transformer-based trackers have become popular for feature extraction and per-frame correlation, while large-scale datasets enable RGB parameters to transfer across applications.Additional modalities are explored because RGB tracking remains vulnerable to occlusion, low illumination, and fast motion.
- Multimodal tracking: Depth, thermal, and event modalities have been investigated to improve tracking under challenging visual conditions.These modalities supplement RGB tracking in settings where appearance information is unreliable.
- Unified tracking: Modality-specific blocks limit adaptability, and recent architectural unification still requires modality-specific fine-tuning with distinct parameter sets.This remains a central obstacle to one model serving multiple modalities.
- Unified tracking: Current tracking datasets predominantly provide a single modality rather than all possible modal combinations, complicating unified training.Examples include depth, thermal, and event datasets.
3. Methods
Un-Track unifies single-modality-at-a-time multimodal tracking through a shared embedding, modality-aware prompting, and LoRA adaptation of a pretrained RGB tracker. Its shared representation combines low-rank reconstruction with edge guidance, while prompting selectively incorporates auxiliary-modal information.
- Overall Framework: Un-Track targets multimodal tracking where only one auxiliary modality is available at a time, using one parameter set across modalities.The framework is designed for depth, thermal, and event inputs without requiring all modalities to co-occur during training.
- Overall Framework: The framework combines shared embedding, modal prompting, and LoRA-finetuned pretrained RGB tracking components.The shared embedding binds modalities, modal prompting adds modality awareness at each scale, and LoRA adapts the pretrained tracker.
- Shared Embedding: Explicit edge awareness supplements low-rank reconstruction by integrating gradient maps from auxiliary modalities and RGB features.The gradients are computed along both spatial axes and combined into a gradient feature to preserve shared object-shape cues.
- Shared Embedding: The shared embedding uses modality-specific low-rank approximations, fuses them with gradient guidance, and reconstructs a common feature space.Depth, thermal, and event features are separately projected into low-rank matrices before fusion and reconstruction.
- Outer Modal Prompting: Modal prompting categorizes RGB tokens as negative, uncertain, or positive, then exchanges, enhances, or retains them using cross-modal features.The resulting token-recovery process operates largely in low-rank space before reconstructing the fused output.
- Evaluation: Table 1 reports that the uni-model with a single parameter set outperforms previous depth-specific SOTA on the DepthTrack test set.The table caption also states that the depth-specific model sets the SOTA record.
4. Experiments
Experiments evaluate Un-Track within distribution and across datasets using one parameter set for RGB-D, RGB-T, and RGB-Event tracking. The model consistently competes with or surpasses modality-specific and unified baselines, including when auxiliary inputs are missing.
- Evaluation setup: Un-Track uses RGB-D, RGB-T, and RGB-Event training data from DepthTrack, LasHer, and VisEvent, respectively, with one model trained across modality pairs.The datasets provide single-modality auxiliary inputs, while the target setting is a Uni-model with one parameter set.
- Within distribution evaluation: +2.1% absolute precision over ViPT is achieved on DepthTrack under depth-specific training and fine-tuning.The mixed-data Uni-model also outperforms depth-specific ViPT, while jointly training ViPT across datasets substantially degrades its performance.
- Within distribution evaluation: +3.8% absolute gain over ViPT is achieved by the single-parameter Uni-model on LasHer under cross-domain joint learning.With domain-specific fine-tuning, Un-Track also surpasses ViPT and establishes a new SOTA result.
- Within distribution evaluation: +1.1% absolute precision over the current SOTA is achieved on VisEvent with one parameter set.The unified prompting method uses gradual token exchange between RGB and event modalities to preserve modality-specific clues.
- Generalization across datasets: +0.5% accuracy on VOT-RGBD2022 and +0.7% absolute precision on RGBT234 show generalization beyond the training datasets.The Uni-model surpasses depth-specific ViPT on VOT-RGBD2022 and thermal-specific ViPT on RGBT234 without dataset-specific thermal fine-tuning.
- Missing-modality evaluation: With dummy auxiliary input, Un-Track consistently outperforms RGB-only fine-tuned counterparts using +6.65M parameters and +2.14 GFLOPs.The RGB-only setting models cases where the auxiliary sensor is unavailable.
5. Ablation Studies
Ablations show that shared embedding, implicit learning, explicit edge guidance, in-domain approximation, and carefully selected low-rank representations are important to performance. Removing or misconfiguring these components harms the unified tracker.
- Component analysis: Removing the shared embedding and directly mixing modalities harms network performance.Equal treatment of heterogeneous modalities does not adequately account for their representation differences.
- Low-rank approximation: Low-rank rank selection affects both shared embedding and modal prompting: low ranks miss essential information, while high ranks capture excessive modality-specific detail.The shared-embedding study evaluates ranks 2, 4, and 8; modal prompting evaluates ranks 4, 8, and 16.
- Low-rank approximation: LoRA fine-tuning also exhibits a rank trade-off, with lower ranks performing poorly and higher ranks tending toward poorer performance from likely overfitting.The evaluated LoRA ranks are 2, 4, and 8.
- Shared embedding ablation: Removing explicit edge guidance causes a substantial performance drop, highlighting its role in binding modalities during implicit shared-embedding learning.The model otherwise learns the shared embedding without an edge prior.
- Shared embedding ablation: Using only explicit edge guidance without implicit learning yields suboptimal performance because modality-specific clues are neglected.The result supports combining explicit shared structure with learned modality information.
- Shared embedding ablation: Direct low-rank approximation of mixed modalities struggles because the domain gap creates an intricately mixed representation.The ablation bypasses initial in-domain approximation and subsequent fusion steps.
6. Conclusion and Future Work
Un-Track demonstrates a single-model tracker that supports any modality through shared embeddings, lightweight prompting, and inner finetuning. Experiments report improved tracking performance and robust generalization without substantial computational burden.
- Un-Track provides a single-model tracker for video object tracking that supports any modality.
- A shared embedding binds heterogeneous modalities, while lightweight modal prompting and inner finetuning support unification.
- The approach inherits benefits from large-scale pre-trained trackers without introducing a substantial computational burden.
- Exhaustive experiments show improved tracking performance and robust generalization with any modality input.