Source-linked AI summary
Visual Prompt Multi-Modal Tracking
Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, Huchuan Lu
TL;DR
Multi-modal tracking lacks the large-scale downstream data available for RGB tracking, challenging full fine-tuning of RGB foundation models. ViPT adapts a frozen pretrained foundation model by learning modality-specific visual prompts, achieving state-of-the-art results across RGB-D, RGB-T, and RGB-E tracking with fewer than 1% trainable parameters.
Problem
Multi-modal tracking has scarce downstream training data, while existing methods commonly rely on full fine-tuning of pretrained RGB-based trackers.
Method
ViPT freezes the foundation model and learns modality-specific visual prompts through modality-complementary prompters for downstream multi-modal tracking.
Results
ViPT achieves state-of-the-art performance across RGB-D, RGB-T, and RGB-E tracking while using fewer than 1% trainable parameters.
Takeaways & Limitations
Visual prompt learning provides a unified, parameter-efficient way to adapt RGB foundation trackers to multiple downstream multi-modal tracking tasks.
Abstract
from arXiv · showhide
Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multimodal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.
1. Introduction
Multi-modal tracking benefits from auxiliary modalities but has far less training data than RGB tracking, making full fine-tuning costly and potentially less transferable. ViPT instead adapts a frozen RGB foundation model with modality-specific visual prompts and reports strong performance with fewer than 1% trainable parameters.
- Multi-modal tracking uses auxiliary modalities to improve robustness in difficult RGB scenarios such as extreme illumination, clutter, and motion blur.
- Multi-modal datasets contain at least an order of magnitude fewer training sequences and annotated frame pairs than major RGB tracking datasets.DepthTrack, LasHeR, and VisEvent have 150, 979, and 500 training sequences, respectively.
- Full fine-tuning of extended RGB trackers is time-consuming, parameter-inefficient, and cumbersome to transfer when downstream data is scarce.
- ViPT freezes the RGB foundation model and learns a few modality-specific visual prompts to adapt it to downstream RGB-D, RGB-T, and RGB-E tracking.
- Fewer than 1% of model parameters are trainable, while extensive experiments report state-of-the-art performance across multiple downstream multi-modal tracking tasks.
2. Related Work
Related work commonly extends pretrained RGB or ImageNet models with auxiliary-modal branches and fine-tunes them, while earlier prompt-based fusion used nonlearnable prompts. ViPT pursues learnable, parameter-efficient prompt tuning for multi-modal tracking.
- Single-modal tracking has advanced through large-scale datasets and deep neural networks but remains vulnerable in challenging visual scenarios.
- Existing multi-modal trackers typically add auxiliary-modal branches to pretrained RGB-based or ImageNet models and fine-tune them on downstream data.
- Pseudo-truth generation from RGB sequences partly alleviates limited multi-modal training data, but does not eliminate the data shortage.
- Full fine-tuning requires repetitive task-specific copies and storage of entire pretrained models, making it parameter-inefficient.
- Earlier prompt-based image fusion used weighted addition without tuning, so its prompts were not learnable and produced limited improvements.
3. Methodology
ViPT adapts a frozen RGB-based foundation tracker to synchronized auxiliary modalities using learned, modality-complementary prompts rather than full fine-tuning. Its MCP generates prompts at multiple semantic levels, which are injected residually into the foundation model while updating only a small parameter set.
- Overall Architecture: ViPT adapts a pre-trained RGB foundation tracker to RGB-D, RGB-T, and RGB-E tasks using task-oriented visual prompts.The framework is designed as a general method for downstream multi-modal tracking.
- Overall Architecture: The multi-modal tracker receives spatially aligned, temporally synchronized RGB and auxiliary flows, which are separately patch-embedded and mapped into D-dimensional latent tokens.The auxiliary modality may be depth, thermal infrared, or event data.
- Modality-Complementary Prompter: The learned prompts are added as residuals to intermediate RGB features and injected stage-wise into successive foundation-model layers.This design uses representations from diverse semantic levels and can be applied directly to existing pre-trained foundation trackers.
- Modality-Complementary Prompter: MCP processes foundation and auxiliary flows through lower-dimensional projections, complementary representation generation, and projection back to the original dimension.Its two branches use 1×1 convolutions, while the foundation embeddings undergo spatial fovea processing before additive fusion with auxiliary embeddings.
- Motivation: Prompt-tuning is motivated by scarce multi-modal tracking data and the time, storage, and transfer burdens of task-oriented full fine-tuning.The method instead freezes the foundation model while learning modality complementarity through prompts.
4. Experiments
Experiments evaluate ViPT across RGB-D, RGB-T, and RGB-E tracking benchmarks, showing strong performance, parameter efficiency, and favorable behavior across ablations and visual analyses.
- RGB-D Tracking: 59.4% F-score on DepthTrack surpasses previous state-of-the-art trackers and improves 6.5% over the foundation model.DepthTrack measures localization accuracy and robustness using precision and recall, with F-score as the primary measure.
- RGB-D Tracking: 0.721 EAO on VOT-RGBD2022 exceeds the foundation model by 4.5% EAO.VOT-RGBD2022 uses an anchor-based short-term evaluation protocol with expected average overlap as its overall measure.
- RGB-T Tracking: ViPT reaches 61.7% MSR and 83.5% MPR on RGBT234, outperforming ProTrack by 1.8% on MSR.These results exceed those of the compared RGB-T trackers on the reported metrics.
- RGB-T Tracking: On LasHeR, ViPT exceeds the second-place method by 10.5% on success and 11.3% on precision.The benchmark contains 245 testing sequences and evaluates short-term RGB-T tracking with success and precision plots.
- RGB-E Tracking: On VisEvent, ViPT gains 5.8% on success and 6.3% on precision over runner-up OSTrack.The evaluation uses event images transformed from raw event data across 320 testing videos.
- Exploration Studies: ViPT beats full fine-tuning while using two orders of magnitude fewer trainable parameters, and increasing MCP blocks improves performance across benchmarks.Ablations also report favorable precision-efficiency trade-offs, while expanding data or unfreezing all parameters alone does not improve results.
5. Conclusion
ViPT is presented as a parameter-efficient prompt-learning framework for multi-modal tracking that exploits a pre-trained foundation model across downstream tasks. The authors report effectiveness and generalization across multiple tasks, while identifying extensions to vision-language tracking and joint multi-modal training as future directions.
- ViPT introduces prompt learning to multi-modal tracking as a parameter-efficient vision-tuning framework.
- ViPT aims to exploit the foundation model’s large-scale pre-training and the complementarity of multiple modalities.
- Experiments on multiple downstream tasks demonstrate ViPT’s effectiveness and generalization.
- The authors identify vision-language tracking and joint training across multiple modalities as future extensions.