PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking

Ben Kang, Xin Chen, Dong Wang, Houwen Peng, Huchuan Lu

arXiv:2308.06904v1cs.CV

TL;DR

Existing transformer trackers can be too slow for resource-limited devices, and lightweight hierarchical backbones pose a challenge because tracking needs detail that large-stride downsampling can lose. HiT combines a Bridge Module with joint template-search position encoding; HiT-Base reports stronger LaSOT AUC than FEAR while running faster on AGX. The authors describe HiT as achieving competitive performance at high speed, while noting difficulty handling distractors.

  • Problem

    Many trackers prioritize performance without considering speed, and large-stride downsampling in lightweight hierarchical backbones can lose information needed for accurate tracking.

    Source: §1 · p. 1

  • Method

    HiT combines a Bridge Module that fuses deep semantic and shallow detail features with dual-image position encoding for the template and search region.

    Source: §1 · p. 2

  • Results

    HiT-Base achieves an 11.1% higher AUC score than FEAR on LaSOT while being 1.6 times faster on Nvidia Jetson AGX Xavier.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HiT is presented as an efficient tracker with competitive performance relative to state-of-the-art efficient trackers.

    Source: §1 · p. 2, §5 · p. 8

    HiT has difficulty dealing with distractors because it does not use an explicit distractor-handling module.

    Source: §5 · p. 8

Abstract

from arXiv · show

Transformer-based visual trackers have demonstrated significant progress owing to their superior modeling capabilities. However, existing trackers are hampered by low speed, limiting their applicability on devices with limited computational power. To alleviate this problem, we propose HiT, a new family of efficient tracking models that can run at high speed on different devices while retaining high performance. The central idea of HiT is the Bridge Module, which bridges the gap between modern lightweight transformers and the tracking framework. The Bridge Module incorporates the high-level information of deep features into the shallow large-resolution features. In this way, it produces better features for the tracking head. We also propose a novel dual-image position encoding technique that simultaneously encodes the position information of both the search region and template images. The HiT model achieves promising speed with competitive performance. For instance, it runs at 61 frames per second (fps) on the Nvidia Jetson AGX edge device. Furthermore, HiT attains 64.6% AUC on the LaSOT benchmark, surpassing all previous efficient trackers.

1. Introduction

High-performance trackers can be too slow for resource-limited devices, while hierarchical lightweight backbones risk losing tracking-critical detail through large-stride downsampling. HiT combines cross-stage feature fusion and joint template-search position encoding to pursue fast tracking without giving up competitive performance.

1. Introduction

  • Large-stride downsampling in hierarchical lightweight networks reduces computational expense but can lose information critical for accurate tracking.The section frames this as a tension between efficient backbones and tracking’s need for detail.Source: §1 · pp. 1–2
  • HiT-Base achieves an 11.1% higher AUC score than FEAR on LaSOT while being 1.6 times faster on Nvidia Jetson AGX Xavier.It also delivers similar performance to STARK-ST50 while being 4.7 times faster on AGX.Source: §1 · p. 2
  • The Bridge Module fuses deep semantic information into shallow, large-resolution features to mitigate information loss from large-stride downsampling.This makes hierarchical backbones with large-stride downsampling usable for tracking.Source: §1 · p. 2
  • Dual-image position encoding jointly encodes position information from the template and search region to improve positional accuracy.The method targets interaction between the two images.Source: §1 · p. 2
  • HiT is a family of efficient tracking models that combines the Bridge Module and dual-image position encoding, with experiments reporting competitive performance and fast processing.The work positions HiT as a response to research focused on tracker performance without considering speed.Source: §1 · pp. 1–2

2. Related Work

Prior tracking approaches span Siamese and one-stream frameworks, while many high-performance transformer trackers are slow on edge devices and efficient trackers can lag in performance. HiT adapts a lightweight hierarchical transformer for tracking by fusing stages and jointly encoding template and search positions.

2. Related Work

  • Siamese trackers extract template and search features with shared backbones, interact through correlation, and use prediction heads; transformer interaction improved later methods.One-stream trackers jointly perform feature extraction and fusion with the backbone, including MixFormer, SBT, SimTrack, and OS-Track.Source: §2 · p. 2
  • Efficient trackers such as LightTrack and FEAR target speed and accuracy, but the section notes a performance gap from high-performance trackers.HiT is presented as fast on edge devices with competitive results; it performs only 0.3% lower in AUC than TransT on LaSOT while being 4.7 times faster on AGX.Source: §2 · p. 2
  • Lightweight vision transformers use hierarchical architectures and high-stride downsampling to reduce computation, but HiT differs from LeViT by fusing stages and predicting on larger-resolution features.HiT also targets tracking detail and jointly encodes template and search positions, unlike LeViT’s single-image position encoding.Source: §2 · pp. 2–3

3. Method

HiT combines a lightweight hierarchical transformer, a Bridge Module, and a prediction head in a one-stream tracking framework. Its design joins multiscale features and jointly encoded template and search positions to produce tracking predictions.

3.1. Overview

  • The image pair enters the transformer; its multiresolution features are fused by the Bridge Module, while its final global vector and enhanced features feed the prediction head.The backbone produces feature maps at three resolutions and a global vector from its final stage.Source: §3.1 · p. 3, §3.2 · p. 4

3.2. Lightweight Hierarchical Vision Transformer

  • LeViT supplies HiT’s hierarchical backbone: MHA extracts and fuses image features, while SA connects stages by downsampling them.The backbone uses MHA and SA as its core modules.Source: §3.2 · pp. 3–4, §3.1 · p. 3
  • Dual-image encoding adds learned relative-position biases from pixel offsets; diagonal placement gives template and search pixels unique coordinates, avoiding position overlap confusion.Earlier separate encoding can assign the same position to the template and the search region’s upper-left area.Source: §3.2 · pp. 4–5

3.3. Bridge Module and Head

  • The Bridge Module progressively upsamples and adds Smin to Smid, then that result to Smax, fusing deep semantics with shallow detail in a minimal design.Transpose convolutions with stride 2 perform the upsampling.Source: §3.3 · p. 5
  • The corner head uses global information to reweight Bridge-enhanced local features before a fully convolutional network predicts target coordinates.An attention map between the global vector and enhanced features determines which local features are enhanced or suppressed.Source: §3.3 · p. 5

3.4. Training objective and Inference

  • Training combines ℓ1 and generalized GIoU losses; inference initializes the template in frame one and crops each search region using the preceding frame’s target box.The tracker runs end-to-end without additional post-processing such as window or scale penalties.Source: §3.4 · p. 5

4. Experiments

HiT is evaluated across tracking benchmarks, devices, and model variants, with training and inference settings reported. Ablations examine feature fusion and position encoding, while tests with another backbone assess framework generality.

4.1. Implementation Details

  • HiT variants use LeViT-384, LeViT-128, and LeViT-128S; training uses four datasets, 1500 epochs, and 60,000 sampled pairs per epoch.The input is a template-search image pair; the transformer is ImageNet-pretrained, and inference is end-to-end without hyperparameters.Source: §4.1 · pp. 5–6

4.2. State-of-the-art Comparisons

  • 64.6% AUC on LaSOT makes HiT-Base the best real-time tracker; it exceeds FEAR by 11.1% in AUC and is only 0.3% below TransT.HiT-Small is second-best in AUC; HiT-Base also achieves 73.3% PNorm and 68.1% P on LaSOT.Source: Table 2 · p. 6, §4.2 · p. 6
  • On TrackingNet, HiT-Base reaches 80.0% AUC, 3.4% above the prior best real-time tracker HCAT; it also records 64.0% AO on GOT-10k.On TrackingNet, its AUC is 80.0 versus 81.3 for STARK-ST50, while HiT-Base runs 3.5× faster on GPU and 4.7× faster on CPU and AGX.Source: Table 2 · p. 6, §4.2 · p. 6
  • On AGX, HiT-Base, Small, and Tiny run at 61, 68, and 77 fps, respectively; only HiT-Base is slower than FEAR on CPU.GPU speeds are 175, 192, and 204 fps, respectively; all three are real-time on CPU at 33, 72, and 76 fps.Source: Table 2 · p. 6, §4.2 · p. 6
  • On UAV123, HiT-Base achieves 65.6% AUC, the best among real-time trackers; on LaSOText, HiT-Base, HiT-Small, and HiT-Tiny score 44.1%, 40.4%, and 35.8% AUC, respectively.Source: §4.2 · p. 7
  • On VOT2021 using Jetson AGX, HiT-Base achieves the best result among real-time trackers, with a 25.2% EAO score.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Ablation Study and Analysis

  • The default Bridge fusion of Max, Mid, and Min features performs best; visualizations show that omitting shallow Max features can collapse attention, while deep features add semantics.The compared features are from the transformer's first, second, and third stages, respectively.Source: Table 5 · p. 7, §4.3 · pp. 7–8, Figure 5 · p. 7
  • Dual-image position encoding outperforms absolute and separate encoding; its diagonal template-search arrangement also beats vertical and horizontal arrangements.Separate encoding overlaps image positions, while the diagonal layout assigns unique horizontal and vertical positions to both images.Source: §4.3 · p. 8, Table 6 · p. 8

5. Conclusion

HiT is an efficient tracking-model family that bridges lightweight hierarchical transformers and tracking through a Bridge Module and dual-image position encoding. The authors report competitive performance and speed, while noting difficulty with distractors and that they do not design a new transformer.

  • HiT combines a Bridge Module and dual-image position encoding to connect lightweight hierarchical transformers with the tracking framework.The authors report that extensive experiments show promising performance compared with state-of-the-art efficient trackers at very fast speed.Source: §5 · p. 8
  • HiT has difficulty dealing with distractors because it does not employ an explicit distractor-handling module.Source: §5 · p. 8
  • The work focuses on bridging lightweight hierarchical transformers and tracking, making only minimal adjustments to existing transformers rather than designing a new one.The authors identify customizing a lightweight transformer for tracking as future work.Source: §5 · p. 8
Loading 2308.06904v1…