PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting for Multi-Modal Tracking

Jinyu Yang, Zhe Li, Feng Zheng, Aleš Leonardis, Jingkuan Song

arXiv:2207.14571v2cs.CV

TL;DR

Multi-modal tracking lacks the large-scale, high-quality data needed to learn effective fusion across modalities. ProTrack reformulates multi-modal inputs as prompted single-modality inputs for frozen RGB trackers, achieving strong benchmark performance without extra multi-modal training. The approach also exposes boundaries where auxiliary information can disturb RGB-solvable cases and where triple-modal tracking remains unexplored.

  • Problem

    Multi-modal tracking suffers from small, low-resolution, and poorly synchronized or misaligned datasets, limiting the learning of cross-modal fusion.

  • Method

    ProTrack converts multi-modal inputs into three-channel prompted inputs so frozen pre-trained RGB trackers can process them without a fusion module or extra multi-modal training.

  • Results

    ProTrack achieves strong performance across six benchmark datasets and reaches state-of-the-art F-scores of 75.7% on CDTB and 57.8% on DepthTrack.

  • Takeaways & Limitations

    Prompting can leverage large-scale RGB tracking knowledge while incorporating complementary multi-modal information and unifying multiple tracking tasks in one model.

  • Takeaways & Limitations

    Triple-modal tracking is not explored, and auxiliary information can disturb performance when RGB trackers already solve the tracking challenge.

Abstract

from arXiv · show

Multi-modal tracking gains attention due to its ability to be more accurate and robust in complex scenarios compared to traditional RGB-based tracking. Its key lies in how to fuse multi-modal data and reduce the gap between modalities. However, multi-modal tracking still severely suffers from data deficiency, thus resulting in the insufficient learning of fusion modules. Instead of building such a fusion module, in this paper, we provide a new perspective on multi-modal tracking by attaching importance to the multi-modal visual prompts. We design a novel multi-modal prompt tracker (ProTrack), which can transfer the multi-modal inputs to a single modality by the prompt paradigm. By best employing the tracking ability of pre-trained RGB trackers learning at scale, our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data. Extensive experiments on 5 benchmark datasets demonstrate the effectiveness of the proposed ProTrack.

1 INTRODUCTION

Multi-modal tracking can improve robustness in complex scenarios, but limited and lower-quality data leaves cross-modal fusion insufficiently learned. ProTrack instead adapts frozen RGB trackers by modifying multi-modal inputs with visual prompts, avoiding additional multi-modal training.

  • Motivation: Multi-modal tracking includes RGB-D, RGB-T, and RGB-E settings intended to improve accuracy and robustness in complex scenarios.Auxiliary modalities can provide information useful for particular tracking conditions.
  • Data bottleneck: Multi-modal datasets are orders of magnitude smaller than RGB datasets and also suffer from lower resolution, desynchronization, and modality misalignment.These deficiencies constrain the development of high-performing data-driven trackers and fusion modules.
  • Data bottleneck: Pre-trained RGB trackers offer a way to exploit large-scale RGB knowledge, but fine-tuning them on small multi-modal datasets incompletely learns cross-modal fusion.Existing approaches commonly combine pre-trained RGB baselines with multi-modal fine-tuning.
  • ProTrack: ProTrack directly modifies multi-modal inputs with modality-agnostic visual prompts instead of altering or fine-tuning the pre-trained RGB tracker.The prompt paradigm transfers multi-modal data toward a form usable by RGB tracking models.
  • ProTrack: ProTrack unifies multiple multi-modal tracking tasks in one prompting framework, whereas fine-tuned models are limited to one data type.The framework is presented as a solution that avoids inappropriate fine-tuning across different multi-modal scenarios.

2 RELATED WORK

Prior multi-modal tracking methods combine modalities through representations, feature fusion, or modality-aware modules, but their development is constrained by limited large-scale training data. Prompting research motivates reformulating downstream tasks around capabilities already learned by pre-trained models.

  • Multi-Modal Tracking: Earlier multi-modal trackers used depth and thermal data for occlusion handling and scale estimation, and event features for target re-detection.These approaches focused on exploiting auxiliary modalities for specific tracking conditions.
  • Multi-Modal Tracking: Deep-learning approaches fuse visible and auxiliary modalities through concatenated features, adaptive FusionNet modules, or depth-aware convolutions.Examples include visible-thermal fusion and depth-aware adaptations of RGB tracking baselines.
  • Prompt Learning: Prompting replaces downstream fine-tuning in the described NLP paradigm by reformulating tasks to resemble those solved by pre-trained models.The passage connects prompting with high performance in few-shot or zero-shot settings.
  • Prompt Learning: Visual prompting has been explored for visual grounding, vision backbones, and task-specific image perturbations before its application to multi-modal tracking.The cited examples include CPT, VPT, and visual prompting for downstream adaptation.

3 METHOD

ProTrack reformulates multi-modal tracking as single-modal tracking by prompting inputs for a pre-trained RGB tracker. It avoids multi-modal fusion training while using auxiliary modality information and preserving large-scale RGB knowledge.

  • Prompt formulation: Prompting adapts the pre-trained-model paradigm by modifying multi-modal inputs rather than updating the tracker.The approach uses prompts to reduce the distribution gap between downstream multi-modal data and pre-trained RGB data.
  • Prompt formulation: ProTrack converts multi-modal input X={V,A} into a three-channel prompted input X′ through a prompt function f(X).V denotes the visible modality and A the auxiliary modality.
  • Framework: The framework combines a multi-modal prompt that transfers video sequences into visible single-modal ones with a pre-trained RGB tracker.The tracker supplies discriminative ability learned in visible tracking, while prompting transfers auxiliary information into its input space.
  • Framework: The tracker is trained only on existing large-scale RGB data, so no fusion module is required during training or testing.This preserves the tracker’s learned RGB associations while enabling auxiliary data to address new tracking challenges.
  • Prompt design: The prompt function uses modality-specific coloring to produce three-channel inputs, allowing pre-trained RGB trackers to process prompted videos without extra training.For one-channel data, Color(*) expands the result to three channels; for RGB input, it leaves the image unchanged.

4 EXPERIMENTS

ProTrack is evaluated across RGB-D, RGB-Thermal, and RGB-Event tracking benchmarks using pre-trained RGB trackers, with ablations examining prompts, modality inputs, colors, and λ.

  • Experimental Settings: Experiments cover RGB-D tracking on CDTB and DepthTrack, RGB-Thermal tracking on LasHeR and RGBT234, and RGB-Event tracking on VisEvent.The evaluation uses established downstream benchmarks spanning three multi-modal tracking settings.
  • Experimental Settings: ProTrack uses the pre-trained STARK spatial-temporal transformer as its RGB tracking model.STARK includes a backbone, encoder, decoder, prediction head, and score head, and is trained on large-scale RGB tracking datasets.
  • Main Results: 75.7% F-score is achieved on CDTB, while ProTrack outperforms DeT by 10% using the same ResNet-50 backbones.On DepthTrack, ProTrack also obtains a new state-of-the-art F-score of 57.8%.
  • Main Results: 50.9/41.9 precision/success is obtained on LasHeR, outperforming DMCNet by 4.9%/6.3%.The compared models are trained on the LasHeR training set.
  • Main Results: 78.6% precision and 58.7% success are obtained on RGBT234 without RGB-T pre-training or adaptive learning, improving over STARK by 4.1% and 2.4%.ProTrack reaches the top success rate reported in the passage.
  • Main Results: On VisEvent, ProTrack performs on par with current state-of-the-art trackers, while event prompts provide less information than depth or thermal maps.The passage attributes the sub-optimal performance partly to event prompts encoding only positive/negative values on some pixels.
  • Ablation Study: Ablations show that visible information is more discriminative than auxiliary modalities, while default multi-modal prompts outperform single-modal inputs.Color choices and λ also affect performance across the evaluated benchmarks.

5 ANALYSIS AND DISCUSSION

The analysis examines ProTrack’s prompting mechanism, pretrained-model compatibility, multimodal extensions, and failure modes. Prompting improves target localization and can benefit multiple RGB trackers, while failures arise from limited discrimination or prompt-induced distribution shifts.

  • Analysis of prompting: Prompting produces cleaner, less ambiguous target localization and more accurate bounding-box estimation than using no prompts.Without prompts, distractors can create multiple local maxima in the score maps.
  • More pre-trained models: ProTrack can use multiple RGB trackers pretrained on large-scale datasets, including ATOM, TransT, DiMP, PrDiMP, and KeepTrack.The analysis evaluates prompting with representative pretrained RGB trackers.
  • More pre-trained models: 5.4% and 3.3% improvements are reported for DiMP and KeepTrack, respectively, after prompting.Table 8 reports DepthTrack performance using F-score, with “_P” denoting performance after prompting.
  • Beyond Dual-Modal tracking: ProTrack is described as applicable to triple-modal tracking by modifying its prompting formulation to combine visible and two auxiliary modalities.The modalities are converged to the visible modality for use by RGB pretrained tracking models.
  • Beyond Dual-Modal tracking: Triple-modal tracking is not explored in the paper, leaving that application as a proposed future direction.The authors state that the direction is promising but remains unexplored.
  • Failed Cases: Failures result from limited tracker discrimination in challenging sequences or multimodal prompts disturbing the input distribution.The latter case can occur when auxiliary information wrongly guides a tracker that could otherwise solve the RGB tracking challenge.

6 CONCLUSION

The paper proposes multimodal visual prompts that adapt multimodal tracking inputs for large-scale pretrained RGB trackers. ProTrack modifies only the tracker inputs and reports promising results across benchmark datasets.

  • 6 CONCLUSION: ProTrack leverages large-scale RGB tracking models for a wide range of downstream multimodal tracking tasks through multimodal prompts.The approach adapts multimodal tracking tasks to pretrained RGB trackers by modifying their inputs.
  • 6 CONCLUSION: The authors hope ProTrack will spur further research on multimodal tracking and inspire related areas.
Loading 2207.14571v2…