PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Universal Modal Tracking with Online Dense Temporal Token Learning

Yaozong Zheng, Bineng Zhong, Qihua Liang, Shengping Zhang, Guorong Li, Xianxian Li, Rongrong Ji

arXiv:2507.20177v1cs.CVcs.MM

TL;DR

Visual tracking must model temporal relationships while supporting RGB and multiple auxiliary modalities without maintaining separate task-specific models. UM-ODTrack addresses this with video-level sampling, temporal token propagation, and gated perceivers trained once for unified inference, and reports effectiveness across visible and multimodal benchmarks.

  • Problem

    Existing tracking methods use sparse or complex temporal modeling, while multimodal trackers commonly require separate models for different tasks.

  • Method

    UM-ODTrack models video clips with autoregressive temporal token propagation and uses gated perceivers with one-shot training to unify RGB, RGB-T, RGB-D, and RGB-E tracking.

  • Results

    Experiments on seven visible and five multimodal benchmarks show the effectiveness of UM-ODTrack.

  • Takeaways & Limitations

    Temporal token sequences provide prompts for future-frame inference, while one-shot training enables multi-task inference with the same architecture and parameters.

  • Takeaways & Limitations

    The video-level model is a global approximation because GPU-resource constraints prevent a cost-effective full-video framework.

Abstract

from arXiv · show

We propose a universal video-level modality-awareness tracking model with online dense temporal token learning (called {\modaltracker}). It is designed to support various tracking tasks, including RGB, RGB+Thermal, RGB+Depth, and RGB+Event, utilizing the same model architecture and parameters. Specifically, our model is designed with three core goals: \textbf{Video-level Sampling}. We expand the model's inputs to a video sequence level, aiming to see a richer video context from an near-global perspective. \textbf{Video-level Association}. Furthermore, we introduce two simple yet effective online dense temporal token association mechanisms to propagate the appearance and motion trajectory information of target via a video stream manner. \textbf{Modality Scalable}. We propose two novel gated perceivers that adaptively learn cross-modal representations via a gated attention mechanism, and subsequently compress them into the same set of model parameters via a one-shot training manner for multi-task inference. This new solution brings the following benefits: (i) The purified token sequences can serve as temporal prompts for the inference in the next video frames, whereby previous information is leveraged to guide future inference. (ii) Unlike multi-modal trackers that require independent training, our one-shot training scheme not only alleviates the training burden, but also improves model representation. Extensive experiments on visible and multi-modal benchmarks show that our {\modaltracker} achieves a new \textit{SOTA} performance. The code will be available at https://github.com/GXNU-ZhongLab/ODTrack.

I. INTRODUCTION

The paper identifies limitations in sparse image-pair tracking and one-model-to-one-task multimodal tracking, then proposes a universal video-level model with temporal token propagation and gated cross-modal learning.

  • Sparse image-pair sampling underrepresents temporal data and makes object-motion modeling difficult.
  • Image-pair matching emphasizes target appearance while missing continuous cross-frame associations.
  • Prior multimodal trackers use one model per task, limiting shared architectures and parameters for multi-task inference.
  • UM-ODTrack expands inputs from image pairs to video streams to capture richer temporal dependencies.
  • Temporal token association propagates target trajectory information across frames and uses token sequences as prompts for future inference.
  • Two gated perceivers learn cross-modal representations, while one-shot training supports RGB-T/D/E multi-task inference with shared parameters.

II. RELATED WORK

Related work spans image-pair, transformer, multimodal, and online-update trackers, while the paper targets their limited temporal association, complex updating, and cross-task inference.

  • Siamese and transformer trackers commonly rely on sparse image-pair sampling, limiting temporal motion modeling.
  • Auxiliary thermal, depth, and event modalities improve tracking in challenging visibility, occlusion, and fast-motion scenarios.
  • Existing multimodal methods generally use one model for one task, restricting cross-task inference and shared model parameters.
  • Online update methods improve temporal modeling but involve complex optimization, specialized losses, multi-stage training, or manual update rules.
  • The paper introduces dense token propagation to associate target motion trajectories while avoiding intricate optimization and training strategies.
  • UM-ODTrack combines video-level modeling with gated perceivers for universal RGB, RGB-T, RGB-D, and RGB-E tracking.

B. Video-level Multi-modal Tracking Formulation

The formulation extends tracking from image pairs to multimodal video clips, using autoregressive temporal tokens and shared modality encoding to model target trajectories across frames.

  • Conventional trackers process reference-search image pairs and predict bounding-box coordinates for the current search frame.
  • Temporal tokens are generated for video frames and propagated autoregressively to guide subsequent-frame inference.
  • Image-pair modeling lacks inter-frame associations and rapid extension to multimodal tracking.
  • The proposed framework models video clips autoregressively and propagates a temporal token containing target appearance, location, and trajectory information.
  • Video-level inputs use RGB reference frames and search frames of specified lengths to model spatio-temporal target relationships.
  • A shared universal modality encoder extracts and fuses RGB with auxiliary thermal, depth, or event video clips.

C. Video Sequence Sampling Strategy

The model replaces short-term image-pair sampling with video-sequence sampling to capture richer long-term multi-modal spatio-temporal information. It combines shared modality processing with gated perceivers that adaptively aggregate cross-modal representations.

  • Video Sequence Sampling Strategy: Video-sequence sampling addresses the inability of short-term image-pair methods to capture long-term motion variations and real-time multi-modal target status.The strategy uses a larger sampling interval and randomly extracts multiple frames to approximate broader video content.
  • Framework Overview: UM-ODTrack combines modality tokenizers, a shared universal modal encoder, two gated perceivers, and a prediction head for unified multi-modal representations.The gated connection performs selective multi-modal fusion by evaluating the relative importance of each modality.
  • Modality Processing: The model treats depth, thermal, and event inputs as a unified visual representation while preserving a shared sequential token format.This contrasts with using fully separate modality tokenizers for each auxiliary source.
  • Gated Perceivers: Conditional gates align visible and auxiliary features, then selectively control cross-modal learning according to modality quality.The final conditional gated layer is zero-initialized so its output initially matches the fundamental visual tracker, supporting training stability.
  • Gated Perceivers: The gated modal-scalable perceiver cross-attends learned multi-modal representations with temporal modality tokens to enhance multi-modal tracking perception.The gated attention mechanism adaptively aggregates information into a shared visual-semantic feature space.

F. Temporal Token Association Attention Mechanism

The temporal token association mechanism extends attention from image pairs to video sequences and propagates compressed target information online. This provides temporal prompts that carry appearance, localization, and trajectory information into later frames.

  • Temporal Association: The method extends 2D attention with video-sequence content and temporal token vectors to extract richer spatio-temporal features.It uses concatenated and separated temporal token attention mechanisms based on compression-propagation.
  • Concatenated Token Attention: Concatenated token attention densely models spatio-temporal relationships by concatenating reference frames, the search frame, and temporal tokens.The original image-pair formulation only establishes limited temporal correlations through independent reference-search interactions.
  • Online Token Propagation: Each temporal token compresses a frame’s target trajectory information and is propagated autoregressively into the next frame’s empty token.The updated token becomes input to subsequent frames.
  • Online Token Propagation: Temporal tokens serve as prompts for next-frame inference while implicitly propagating target appearance, localization, and trajectory information.The tracking procedure initializes temporal tokens and updates memory as frames are processed.
  • Separated Token Attention: Separated token attention decomposes processing into reference self-aggregation, reference-search cross-aggregation, and temporal-token sequence cross-aggregation.This decomposition improves computational efficiency to a certain extent.

1) Discussions with Online Update.:

UM-ODTrack replaces complex online model-update strategies with iterative propagation of token sequences. The resulting representation and computation are described as more efficient.

  • Online Update: Previous online-update trackers often require complex optimization processes and update decision rules.Examples include extra score-quality or IoU-prediction branches and sophisticated parameter-update procedures.
  • Online Update: Online iterative propagation of token sequences avoids complex update strategies while enabling more efficient model representation and computation.The approach uses propagated temporal tokens instead of repeatedly updating model parameters.

G. One-shot Training and Universal Inference

The one-shot training scheme jointly trains the universal model across multi-modal tracking tasks and enables inference with one shared parameter set. The framework uses conventional prediction heads and reports comparisons across visible benchmarks.

  • Prediction Head: The prediction head uses conventional classification and bounding-box regression heads, with three sub-convolutional networks producing prediction outputs.The supplied description identifies the head design but does not provide its full equations.
  • Visible-Benchmark Evaluation: Table I compares state-of-the-art methods on GOT10K, LaSOT, TrackingNet, and LaSOText, distinguishing GOT10K-only training and 256- versus 384-pixel search configurations.The best two results are highlighted in the table.
  • Benchmark Evaluation: The evaluation scope also includes comparisons on TNL2K, OTB100, VOT2020, DepthTrack, VOT-RGBD2022, and RGBT234.These supplied table captions identify benchmark-level comparisons but provide no numerical outcomes.
  • One-shot Training: One-shot training jointly covers RGB-T, RGB-D, and RGB-E tracking instead of training separate models on individual downstream datasets.The objective is to support multiple tracking tasks within one universal modal-awareness model.
  • Universal Inference: Universal inference performs RGB-D, RGB-T, and RGB-E tracking with the same model parameters without additional multi-shot fine-tuning.The gated modal-scalable perceiver and one-shot training scheme support inference across these tasks.

A. Implementation Details

The tracker is evaluated across visible and multi-modal benchmarks, with results attributed to video-level sampling, temporal token association, and gated attention mechanisms. Reported comparisons show state-of-the-art or improved performance across RGB, RGB-T, RGB-D, and RGB-E tracking tasks.

  • Benchmark Evaluation: The evaluation covers seven visible and five multi-modal tracking benchmarks, including RGB-T, RGB-D, and RGB-E datasets.The visible benchmarks include LaSOT, TrackingNet, GOT10K, LaSOText, VOT2020, TNL2K, and OTB100; multi-modal benchmarks include LasHeR, RGBT234, DepthTrack, VOT-RGBD2022, and VisEvent.
  • Visible Tracking: 77.0% AO, 87.9% SR0.5, and 75.1% SR0.75 on GOT10K establish a new state-of-the-art result for ODTrack384.The authors attribute the improvement to the proposed video-level sampling strategy.
  • Visible Tracking: 85.1% AUC, 90.1% PNorm, and 84.9% P on TrackingNet outperform SeqTrack by 1.2%, 1.3%, and 1.3%, respectively.Compared with VideoTrack, the gains are 1.3%, 1.4%, and 1.8% on the same metrics.
  • Visible Tracking: 60.5% EAO is achieved by ODTrack-L384 on VOT2020 mask evaluations, while ODTrack384 achieves 58.1%.The reported results are associated with injecting temporal token attention into tracking.
  • Multi-Modal Tracking: 78.0% EAO, 81.4% accuracy, and 94.8% robustness are obtained on VOT-RGBD2022, with gains of 5.9% and 7.9% over Un-Track on reported metrics.UM-ODTrack256 is reported as a new state-of-the-art result on this RGB-D benchmark.
  • Multi-Modal Tracking: 4.5% and 6.3% gains over the second place on LasHeR are reported for success and precision plots, respectively.The authors attribute these multi-modal gains to gated attention that adaptively extracts and fuses features from different modalities.

C. Ablation Study

Ablations show that temporal token association, gated modal components, multi-task training, and multimodal inputs improve tracking, while sequence length and sampling range require careful selection.

  • Token association: Removing token association decreases AUC by 1.2%, while concatenated and separated temporal attention both improve performance.The concatenation variant performs slightly better than the separated variant.
  • Video sequence length: Increasing search video-clip length from 2 to 3 improves AUC by 0.3%, but longer clips impose a learning burden without further gains.
  • Sampling range: Expanding the sampling range from 200 to 1200 improves AUC, indicating that larger ranges provide trajectory information for the video-level framework.
  • Gated perceivers: Adding the conditional gate improves DepthTrack F-score by 1.3%, and adding GMP further improves performance; tanh gating outperforms ReLU and sigmoid.
  • Multimodal inputs: Using only RGB frames decreases LasHeR SR and PR by 6.3% and 7.2%, respectively, demonstrating the value of multimodal cues.
  • Training strategy: Full fine-tuning performs slightly better than adapter fine-tuning, while adapter training does not significantly reduce resources because other gradients remain active.
  • Multi-task training: Joint training on DepthTrack, LasHeR, and VisEvent raises DepthTrack F-score from 67.8% to 69.3% compared with DepthTrack-only training.

D. Qualitative Analysis

Qualitative analyses show that dense temporal propagation helps localize targets in difficult RGB sequences, while gated perceivers support accurate multimodal localization and temporal attention follows object motion.

  • RGB tracking: Dense propagation of target trajectory information enables ODTrack to outperform SeqTrack on visualized LaSOT sequences with similar distractors.
  • Multimodal tracking: UM-ODTrack accurately localizes targets in complex LasHeR, DepthTrack, and VisEvent sequences compared with other multimodal trackers.The authors attribute this capability to gated perceivers that support arbitrary modalities.
  • Temporal attention: Temporal token attention continuously propagates and attends to object motion trajectories, aiding accurate target-instance localization.

V. CONCLUSION

The paper presents ODTrack and UM-ODTrack as video-level tracking frameworks that propagate temporal token information and aggregate multimodal cues. A stated limitation is that GPU constraints force global video modeling to remain an approximation, motivating more efficient transformer designs.

  • Framework: ODTrack reformulates visual tracking as autoregressive token propagation that densely associates contextual relationships across video frames.
  • Multimodal extension: UM-ODTrack extends this framework to multimodal perception by aggregating multimodal temporal information.
  • Limitation: GPU constraints make the video-level modeling method a global approximation, motivating lower-complexity and lightweight transformer modeling.
Loading 2507.20177v1…