PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Transformer Tracker with Correlative Masked Modeling

Zikai Song, Run Luo, Junqing Yu, Yi-Ping Phoebe Chen, Wei Yang

arXiv:2301.10938v1cs.CV

TL;DR

The paper asks whether specialized attention structures are necessary for visual tracking, given that recent variants may reduce to packed self-attention. It uses a basic ViT with concatenated template-search inputs and a training-only correlative masked decoder, achieving strong results across five tracking benchmarks.

  • Problem

    Recent tracking attention variants may be equivalent to or subsets of standard packed self-attention, leaving the necessity of structural adaptation unresolved.

  • Method

    The method combines a basic ViT and box head with a lightweight correlative masked decoder that reconstructs masked template and search images during training.

  • Results

    The tracker achieves state-of-the-art performance across five challenging datasets, while masked modeling raises LaSOT AUC from 64.0% to 65.8%.

  • Takeaways & Limitations

    The findings support vanilla self-attention as sufficient for information aggregation and show that the decoder can improve other transformer trackers without compromising speed.

Abstract

from arXiv · show

Transformer framework has been showing superior performances in visual object tracking for its great strength in information aggregation across the template and search image with the well-known attention mechanism. Most recent advances focus on exploring attention mechanism variants for better information aggregation. We find these schemes are equivalent to or even just a subset of the basic self-attention mechanism. In this paper, we prove that the vanilla self-attention structure is sufficient for information aggregation, and structural adaption is unnecessary. The key is not the attention structure, but how to extract the discriminative feature for tracking and enhance the communication between the target and search image. Based on this finding, we adopt the basic vision transformer (ViT) architecture as our main tracker and concatenate the template and search image for feature embedding. To guide the encoder to capture the invariant feature for tracking, we attach a lightweight correlative masked decoder which reconstructs the original template and search image from the corresponding masked tokens. The correlative masked decoder serves as a plugin for the compact transform tracker and is skipped in inference. Our compact tracker uses the most simple structure which only consists of a ViT backbone and a box head, and can run at 40 fps. Extensive experiments show the proposed compact transform tracker outperforms existing approaches, including advanced attention variants, and demonstrates the sufficiency of self-attention in tracking tasks. Our method achieves state-of-the-art performance on five challenging datasets, along with the VOT2020, UAV123, LaSOT, TrackingNet, and GOT-10k benchmarks. Our project is available at https://github.com/HUSTDML/CTTrack.

1 Introduction

The paper argues that standard packed self-attention is sufficient for visual tracking, while discriminative feature extraction and information enhancement matter more than designing specialized attention variants. It therefore combines a simple ViT tracker with correlative masked modeling during training.

  • Visual tracking estimates an arbitrary target’s bounding box throughout a subsequent video sequence.
  • Advanced mix-attention and cross-attention variants are equivalent to or subsets of packed self-attention over concatenated template and search images.
  • Self-information enhancement contributes more than cross-information aggregation, although cross-information remains indispensable for visual tracking.
  • The proposed tracker uses a basic ViT encoder with concatenated template-search input, a box head, and a masked decoder that reconstructs both images during training.
  • The correlative masked decoder is skipped during inference and improves LaSOT AUC from 64.0% to 65.8%.

2 Related Work

Prior tracking methods commonly use separate feature extraction, similarity, and prediction modules, whereas this work investigates whether packed self-attention and a simpler ViT-plus-box-head pipeline suffice. Its masked modeling adapts image reconstruction to the multi-image tracking setting.

  • Siamese trackers measure similarity between template and search features using two network branches.
  • Traditional trackers typically combine separate template-search feature extraction, similarity measurement, and prediction heads.
  • The proposed compact pipeline uses only a ViT backbone and box head, without separate feature-extraction or multilayer feature-aggregation modules.
  • The correlative masked decoder addresses the absence of tracking pretraining and the need to reconstruct multiple interacting images.
  • Its self-decoder and cross-decoder reconstruct template and search images from masked tokens, extending masked image modeling to visual tracking.

3 Approach

The approach analyzes transformer attention as four information streams, showing that packed self-attention subsumes existing variants and that self-information is especially important for tracking. It then combines a ViT tracker with correlative masked reconstruction during training to strengthen self- and cross-information while preserving a simple inference pipeline.

  • 3.2 Correlative Masked Modeling: The compact tracker concatenates template and search tokens in a ViT backbone and uses a box head to estimate the target bounding box.
  • 3.1 Revisiting Transformer Tracker: The analysis decomposes attention over template and search images into four streams: template self-information, template cross-information, search cross-information, and search self-information.
  • 3.1 Revisiting Transformer Tracker: Packed self-attention and mix-attention include all four information streams, while cross-attention contains only the two streams used for cross-information aggregation.
  • 3.1 Revisiting Transformer Tracker: Removing template cross-information can improve tracking, whereas removing self-information enhancement severely degrades performance, indicating a greater role for self-information in multi-image attention.
  • 3.2 Correlative Masked Modeling: The correlative masked decoder reconstructs each image from corresponding masked tokens and reconstructs the template from search tokens to enhance self- and cross-information.
  • 3.3 Training and Inference: The decoder is used only during training, while inference uses fixed and online-updated templates with a score head controlling template updates.

4 Experiments

Experiments evaluate CTTrack’s compact design, reconstruction streams, masking ratios, template updating, attention behavior, and benchmark performance. The results support strong accuracy with a simple ViT-based tracker, while revealing trade-offs between backbone size, tracking speed, and long-term robustness.

  • Implementation and Backbone Comparison: CTTrack uses only a ViT backbone and box head, with model size and speed evaluated across ViT-Base and ViT-Large backbones.ViT-Base provides higher tracking speed, while ViT-Large provides better performance.
  • Reconstruction Streams: 1.8% AUC improvement follows adoption of all three reconstruction streams, validating the effectiveness of the masked modeling decoder.The streams reconstruct search from search tokens, template from template tokens, and template from search tokens; search-to-search reconstruction improves AUC most among individual streams, from 64.0 to 64.7.
  • Masking Ratio: 75% masking produces the best tracking results, with accuracy increasing steadily as the masking ratio rises to that level.The experiment evaluates random masking of encoded template and search tokens at different rates.
  • Online Template Updating: Fixed-interval online template updating greatly reduces template quality and causes tracking drift, whereas score-based updating improves AUC by 0.2%.The score-based strategy updates the online template only with high-confidence samples.
  • Attention Visualization: Correlative-decoder training strengthens discriminative ability: the baseline can lose the target position, while background distractors are suppressed.The comparison uses attention-map visualizations between training with and without the correlative decoder.
  • Comparison with the SOTA: CTTrack-L achieves 71.3% AUC on UAV123, 69.8% AUC and 76.2% precision on LaSOT, 84.9% AUC on TrackingNet, and 72.8% AO on GOT-10k.The reported results outperform prior trackers on UAV123, rank first in TrackingNet AUC, and exceed the best existing GOT-10k result by 2.1%; LaSOT remains below MixFormer because of long-term sequences and scale variation.

5 Conclusion

The paper finds vanilla self-attention sufficient for information aggregation and introduces correlative masked modeling to strengthen representations. Experiments support the compact tracker’s effectiveness and the decoder’s transferability to other transformer trackers.

  • Vanilla self-attention is sufficient for information aggregation in the transformer tracking framework.
  • The correlative masked decoder uses self-decoder and cross-decoder components to reconstruct original template and search-image pixels.
  • Extensive experiments demonstrate the effectiveness of correlative masked modeling and the compact tracker’s performance over previous trackers.
  • The decoder can be plugged into other transformer trackers to improve tracking performance without compromising speed.
Loading 2301.10938v1…