PaperlayerView exact revision on arXiv

Source-linked AI summary

OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning

Lingyi Hong, Shilin Yan, Renrui Zhang, Wanyun Li, Xinyu Zhou, Pinxue Guo, Kaixun Jiang, Yiting Chen, Jinglun Li, Zhaoyu Chen, Wenqiang Zhang

arXiv:2403.09634v1cs.CV

TL;DR

Visual tracking needs a unified way to handle RGB and multimodal RGB+X tasks despite task-specific architectures and limited data in some settings. OneTracker pretrains a Foundation Tracker for RGB tracking, then uses frozen weights plus trainable prompting components to adapt to RGB+X tasks. It achieves state-of-the-art results across 6 tracking tasks and 11 benchmarks, while still requiring task-specific adaptation for different RGB+X tasks and not covering all modalities.

  • Problem

    Existing tracking systems are often task-specific, causing redundant architectures and parameters, while unified approaches commonly omit multimodal tracking.

  • Method

    OneTracker pretrains a Foundation Tracker on RGB tracking data and adapts it to RGB+X tasks with multimodal prompts and parameter-efficient trainable modules.

  • Results

    OneTracker achieves state-of-the-art performance on 11 benchmarks across 6 tracking tasks.

  • Takeaways & Limitations

    A pretrained RGB tracker can serve as the basis for a unified framework spanning RGB and multiple RGB+X tracking tasks.

  • Takeaways & Limitations

    Different RGB+X tracking tasks still require separate training and adjustment of the CMT Prompters and TTP Transformer layers, and additional modalities remain unconsidered.

Abstract

from arXiv · show

Visual object tracking aims to localize the target object of each frame based on its initial appearance in the first frame. Depending on the input modility, tracking tasks can be divided into RGB tracking and RGB+X (e.g. RGB+N, and RGB+D) tracking. Despite the different input modalities, the core aspect of tracking is the temporal matching. Based on this common ground, we present a general framework to unify various tracking tasks, termed as OneTracker. OneTracker first performs a large-scale pre-training on a RGB tracker called Foundation Tracker. This pretraining phase equips the Foundation Tracker with a stable ability to estimate the location of the target object. Then we regard other modality information as prompt and build Prompt Tracker upon Foundation Tracker. Through freezing the Foundation Tracker and only adjusting some additional trainable parameters, Prompt Tracker inhibits the strong localization ability from Foundation Tracker and achieves parameter-efficient finetuning on downstream RGB+X tracking tasks. To evaluate the effectiveness of our general framework OneTracker, which is consisted of Foundation Tracker and Prompt Tracker, we conduct extensive experiments on 6 popular tracking tasks across 11 benchmarks and our OneTracker outperforms other models and achieves state-of-the-art performance.

1 Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University, Shanghai, China

The authors are affiliated with Fudan University institutions in Shanghai, including the School of Computer Science and engineering research centers.

  • The listed affiliations include Fudan University’s School of Computer Science and engineering research centers in Shanghai.

1. Introduction

OneTracker unifies RGB and RGB+X tracking by pretraining a Foundation Tracker and adapting it with parameter-efficient prompt tuning. The framework targets limitations of task-specific models and reports state-of-the-art performance across six tracking tasks and 11 benchmarks.

  • Task-specific trackers require customized architectures, redundant parameters, and separate training procedures, while some unified models omit multimodal tracking.
  • OneTracker unifies RGB and RGB+X tracking within a consistent framework.
  • Foundation Tracker is pretrained on several RGB tracking datasets to provide strong target-localization ability.
  • Prompt Tracker adapts Foundation Tracker to downstream RGB+X tasks by introducing multimodal prompts and task-perception Transformer layers.
  • OneTracker achieves state-of-the-art performance on 11 benchmarks spanning 6 tracking tasks.

2. Related Works

Related work covers large-scale vision pretraining, parameter-efficient transfer learning, and visual tracking across RGB and multimodal settings. OneTracker builds on these directions to transfer a pretrained tracker across tracking tasks and modalities.

  • Large-scale Pretraining Vision Models: Large-scale vision pretraining learns transferable representations from broad data for adaptation to downstream tasks.
  • Parameter-Efficient Transfer Learning: Parameter-efficient transfer learning freezes pretrained weights and adds a small number of trainable parameters for lightweight downstream adaptation.
  • Parameter-Efficient Transfer Learning: Prior prompting methods for tracking focus on RGB+D/T/E, leaving transfer to RGB+M and RGB+N tracking unresolved.
  • Visual Tracking: Visual tracking research includes RGB and RGB+X tasks, but existing task-specific and multitask trackers lack a general multimodal temporal-matching framework.

3. Methodology

OneTracker unifies RGB and RGB+X tracking by combining a pretrained Foundation Tracker with prompt-based, parameter-efficient adaptation. Its pipeline embeds auxiliary modalities as prompts, fuses them with temporal matching, and uses lightweight adapters to specialize downstream tracking.

  • 3. Methodology: The methodology is organized around Foundation Tracker pretraining and downstream Prompt Tracker adaptation for RGB+X tracking.The paper evaluates overall performance on RGB and RGB+X tracking in Table 1.
  • 3.1. Tracking Unification: OneTracker represents RGB and RGB+X tracking in a unified format, with RGB tracking using initial-box and frame inputs and RGB+X tracking adding modality-specific information.RGB+X inputs include language, masks, depth, thermal, or event maps.
  • 3.2. Foundation Tracker: The Foundation Tracker processes template and search RGB frames with Transformer encoders, builds temporal matching, and converts correlations into box and mask predictions.It is trained on LaSOT, TrackingNet, and GOT-10K RGB tracking benchmarks.
  • 3.3. Prompt Tracker: Prompt Tracker maps downstream modalities into unified prompt tokens and uses Cross Modality Tracking Prompters to fuse them with Foundation Tracker matching representations.Language features use BERT, while depth, thermal, and event maps are patch-embedded into tokens.
  • 3.3. Prompt Tracker: CMT Prompters update each layer’s matching representation by adding the generated prompt as a residual.The layerwise operation is expressed as H_l = H_l + P_l+1.
  • 3.3. Prompt Tracker: Tracking Task Perception Transformer layers insert small trainable adapters into Transformer projections and FFN output layers to adapt the pretrained tracker efficiently.The adapter augments a linear transformation with a low-parameter update ΔW implemented through down- and up-projections.

4. Experiments

OneTracker is evaluated across six tracking tasks and shows strong performance on RGB and RGB+X benchmarks. Ablations examine parameter-efficient adaptation, multimodal prompting, layer placement, training strategy, and temporal matching.

  • Benchmark evaluation: OneTracker is evaluated on six tracking tasks, comparing its results with task-specific counterparts.The evaluation covers RGB and RGB+X tracking tasks.
  • RGB Tracking: 70.5 AUC on LaSOT and 69.7 AUC on TrackingNet surpass all other trackers, with at least 1.3 AUC improvement over several unified models on LaSOT.Foundation Tracker also exceeds GRM by 0.6 AUC on LaSOT.
  • RGB+N Tracking: Prompt Tracker exceeds existing RGB+N trackers by at least 1.7 AUC and 2.5 precision on OTB99 despite not being specifically designed for RGB+N tracking.
  • RGB+D/T/E Tracking: Prompt Tracker greatly exceeds other trackers on RGB+D, RGB+T, and RGB+E benchmarks, with CMT Prompters and TTP Transformer layers identified as effective components.The evaluated benchmarks are DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent.
  • RGB+M Tracking: Prompt Tracker achieves the best performance by a large margin on DAVIS16 and DAVIS17 while using only the first and previous frame, enabling videos of any length.The comparison includes multi-task counterparts and unified tracking models.
  • Ablation studies: Foundation Tracker provides strong visual temporal matching, while parameter-efficient Prompt Tracker adaptation further improves downstream RGB+X tracking through multimodal information and refined predictions.The ablations evaluate full finetuning against parameter-efficient finetuning with CMT Prompters and TTP Transformer layers.
  • Ablation studies: Increasing CMT Prompter layers improves most Prompt Tracker performance but substantially reduces RGB+M performance, indicating limited benefit from deeply incorporated mask embeddings.Jointly separating CMT Prompters and TTP Transformer layers improves multi-task training, while task-specific training performs better.

5. Conclusion

OneTracker unifies RGB and RGB+X tracking through a pretrained Foundation Tracker and prompt-based downstream adaptation. It achieves state-of-the-art results across 11 benchmarks spanning six tracking tasks.

  • OneTracker unifies RGB and RGB+X tracking by pretraining a Foundation Tracker and adapting it to downstream RGB+X tasks with prompt-tuning.
  • OneTracker achieves state-of-the-art results across 11 benchmarks covering six tracking tasks.

6. Discussion

OneTracker unifies RGB and multimodal RGB+X tracking in a general framework, using parameter-efficient adaptation to transfer a pretrained RGB tracker across tasks. The framework performs competitively across six tracking tasks and 11 benchmarks, but still requires task-specific adaptation and does not cover every modality.

  • OneTracker is presented as the first general framework unifying RGB and RGB+N/D/T/E/M tracking.It treats RGB and RGB+X tracking as a unified whole rather than separate entities.
  • OneTracker achieves competitive performance across 6 tracking tasks and 11 benchmarks.
  • The framework introduces parameter-efficient transfer learning for tracking instead of conventional full finetuning.Foundation Tracker is pretrained on RGB tracking, while multimodal information is incorporated as prompts for Prompt Tracker.
  • CMT Prompters add multimodal features, while TTP Transformer layers improve adaptation to downstream tracking tasks.The Foundation Tracker is frozen while additional trainable parameters are adjusted for downstream RGB+X tasks.
  • Prompt Tracker still requires separate training for different RGB+X tasks and corresponding adjustment of its adaptation modules.The authors identify joint training across multiple RGB+X tasks as an important direction for future work.
  • The framework covers six modalities or task types but leaves other modalities and adaptation to emerging tasks for future investigation.Maintaining existing performance while accommodating new downstream tasks remains an open challenge.

7. Experiment Details

Foundation Tracker is trained on established RGB tracking and detection datasets, while Prompt Tracker is finetuned separately for each downstream RGB+X task using task-specific datasets. Under comparable or lower training resources, Foundation Tracker reports stronger RGB tracking performance than prior trackers and multimodal-task models.

  • Foundation Tracker Training: Foundation Tracker is trained using LaSOT, TrackingNet, GOT-10K, and COCO training sets with standard augmentations.Horizontal flipping and brightness jittering are used during training.
  • Foundation Tracker Training: Foundation Tracker outperforms other trackers by at least 0.6 AUC on LaSOT under consistent training settings.The comparison keeps training datasets and settings, including the number of epochs, consistent with previous trackers.
  • Training Configuration: Tables 5, 6, and 7 specify the Foundation Tracker and Prompt Tracker training and finetuning settings.Table 5 covers RGB training, while Tables 6 and 7 cover RGB+N/D/T/E and RGB+M finetuning respectively.
  • Foundation Tracker Training: Foundation Tracker achieves at least 1.3 AUC higher on LaSOT than models requiring substantially more training data and cost.The comparison concerns models such as UNINEXT and OmniTracker.
  • Prompt Tracker Finetuning: Prompt Tracker is finetuned separately on task-specific datasets for RGB+N, RGB+D, RGB+T, and RGB+E tracking.The listed training sets include OTB99, LaSOT, TNL2K, DepthTrack, LasHeR, and VisEvent.
  • Prompt Tracker Finetuning: RGB+M finetuning uses DAVIS17 and YouTube-VOS, with the first and previous frames serving as templates.The method does not crop the template or search frames for this task.
Loading 2403.09634v1…