PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Adaptive RGBT Tracking with Modality Prompt

Hongyu Wang, Xiaotao Liu, Yifan Li, Meng Sun, Dian Yuan, Jing Liu

arXiv:2401.01244v1cs.CV

TL;DR

RGBT tracking methods have limited use of temporal information, either relying on an initial template or online training that can neglect spatial-temporal correlations. TATrack combines modality prompts with initial and online templates in a spatio-temporal framework, and achieves state-of-the-art performance on three benchmarks.

Abstract

from arXiv · show

RGBT tracking has been widely used in various fields such as robotics, surveillance processing, and autonomous driving. Existing RGBT trackers fully explore the spatial information between the template and the search region and locate the target based on the appearance matching results. However, these RGBT trackers have very limited exploitation of temporal information, either ignoring temporal information or exploiting it through online sampling and training. The former struggles to cope with the object state changes, while the latter neglects the correlation between spatial and temporal information. To alleviate these limitations, we propose a novel Temporal Adaptive RGBT Tracking framework, named as TATrack. TATrack has a spatio-temporal two-stream structure and captures temporal information by an online updated template, where the two-stream structure refers to the multi-modal feature extraction and cross-modal interaction for the initial template and the online update template respectively. TATrack contributes to comprehensively exploit spatio-temporal information and multi-modal information for target localization. In addition, we design a spatio-temporal interaction (STI) mechanism that bridges two branches and enables cross-modal interaction to span longer time scales. Extensive experiments on three popular RGBT tracking benchmarks show that our method achieves state-of-the-art performance, while running at real-time speed.

Introduction

RGBT tracking combines visible and thermal information, but many approaches underuse temporal cues or rely on online training that can drift. TATrack uses modality prompts and online-updated templates in a spatio-temporal two-stream framework.

Introduction

  • RGB and thermal infrared images offer complementary information; thermal infrared is insensitive to extreme illumination and bad weather.RGBT tracking is used in robotics, surveillance processing, and autonomous driving.Source: Introduction · p. 1
  • Without adequate temporal information, RGBT trackers can struggle with aspect-ratio changes, deformation, and fast movement.Spatial information describes appearance, while temporal information captures object-state changes across frames.Source: Introduction · p. 1
  • Trackers that train online from tracking results can drift, neglect spatial-temporal correlations, and lose discriminative ability after unreliable updates.Trackers using only an initial template also neglect object-state changes.Source: Introduction · p. 2
  • TATrack combines initial and online templates in a spatio-temporal two-stream framework, integrating feature extraction and cross-modal interaction with modality prompts.Its spatio-temporal interaction mechanism lets cross-modal interaction span longer time scales and guides modality fusion.Source: Introduction · p. 2
  • TATrack selects reliable online templates to avoid poor-quality updates, combining temporal information with initial-template appearance for target localization.The authors report state-of-the-art performance on RGBT210, RGBT234, and LasHeR.Source: Introduction · p. 2

Related Works

Prior RGBT trackers exploit temporal information through online training, but can miss spatial-temporal correlations and make incorrect updates. TATrack instead uses a reliable online template and modality prompts to connect temporal and cross-modal information.

Related Works

  • Prior RGBT online-training methods can make incorrect updates and neglect correlations between temporal and spatial information.Some sample and train online without offline training; others update domain-specific layers during tracking after offline training.Source: Temporal Information Exploitation · pp. 2–3
  • TATrack's framework embeds the search area, initial template, and online template, with separate initial and online branches and STI for cross-frame information propagation.The prompter generates modality prompts and adjusts transformer-encoder inputs.Source: Figure 2 · p. 3
  • TATrack captures temporal information with a reliable online-updated template and learns spatio-temporal and modality information together.The online template complements the initial template's appearance information.Source: Temporal Information Exploitation · pp. 2–3
  • Prompt learning adapts models through task-oriented prompts, and prior RGBT methods have used prompts to bridge RGB and thermal modalities.ProTrack uses prompts without tuning; ViPT learns prompts but disregards object-state changes.Source: Visual Prompt Learning · p. 3

Method

TATrack extends a modality-prompt RGBT tracking baseline with initial and online template branches, using cross-branch spatio-temporal interaction and multi-modal features for target localization.

Tracking Baseline with Modality Prompt

  • Modality prompts adjust transformer inputs to adapt an RGB-pretrained model to RGBT tracking and provide inter-modal complementation without complex interaction networks.The modality-complementary prompter generates infrared prompts; prompts are added to transformer inputs.Source: Tracking Baseline with Modality Prompt · p. 4

Head and Training

Overall Architecture

Spatio-temporal Interaction

  • STI connects initial- and online-branch template features through self-attention, enabling bidirectional cross-frame propagation of spatial and temporal information.The two template sequences are concatenated before self-attention; STI is inserted before the modality-complementary prompter.Source: Temporal Adaptive RGBT Tracking · pp. 4–5

Online Template Update

  • The online template captures target-state changes, while its update uses the prediction head’s maximum classification score as confidence without additional training.The update scheme is intended to select reliable templates because poor-quality templates can mislead tracking.Source: Temporal Adaptive RGBT Tracking · pp. 4–5

Differences with other prompt-learning methods

  • TATrack combines spatio-temporal and multi-modal information, addressing ViPT’s spatial-only prompt tracking and ProTrack’s untuned prompts.The paper describes TATrack as exploring a way to combine these information types for RGBT tracking.Source: Temporal Adaptive RGBT Tracking · p. 5

Experiments

TATrack is evaluated against prior RGBT trackers on three benchmarks, with component, challenge-attribute, and response-map analyses. It outperforms the reported competitors across the benchmarks, while ablations examine modality prompts, online-template selection, and STI.

LasHeR

  • On LasHeR, TATrack exceeds ViPT by 5.1% in precision, 5.0% in normalized precision, and 3.6% in success.The comparison is reported against ViPT; the authors also state that TATrack outperforms previous state-of-the-art methods by a large margin.Source: Table 1 · p. 6, Comparison with State-of-the-art Methods · p. 6

Attribute-Based Performance

  • TATrack performs best on almost all LasHeR challenge attributes, with especially clear improvements in motion blur, high illumination, low resolution, deformation, camera motion, fast motion, and scale variation.The authors relate these results to using temporal information to address target-state changes while providing modality complementarity.Source: Table 2 · p. 6, Comparison with State-of-the-art Methods · p. 6

RGBT234

RGBT210

Component Analysis

  • The component analysis reports that modality prompts improve over an RGB-only tracker, while selecting reliable online templates outperforms updating the template every frame.Removing STI reduces success by 1.3% and precision by 1.7% relative to TATrack.Source: Ablation Study · pp. 6–7, Table 5 · p. 7

Inserting Layers of STI

  • Increasing the number of STI insertion layers makes performance improvement more pronounced, but reduces running speed to some extent.The insertion-layer study is reported in Table 6.Source: Table 6 · p. 7, Ablation Study · p. 7

Visualization

  • Response-map examples show stable localization when a car is invisible in RGB under extreme illumination and when an umbrella opens and changes shape.The authors also report robustness in examples involving occlusion, fast movement, scale change, and aspect-ratio change.Source: Figure 4 · p. 7, Ablation Study · p. 7

Conclusion

TATrack uses an online-updated template to capture temporal information and combine it with multimodal information. Experiments on three RGBT benchmarks show state-of-the-art performance; future work will explore template update options.

  • TATrack captures temporal information with an online-updated template, combining spatio-temporal and multimodal information.Source: Conclusion · p. 7
  • Experiments on three RGBT tracking benchmarks show that TATrack achieves state-of-the-art performance.Source: Conclusion · p. 7
  • Future work will explore more template update options to determine whether the tracker should be updated at a given moment.Source: Conclusion · p. 7

Related papers

Loading 2401.01244v1…