PaperlayerView exact revision on arXiv

Source-linked AI summary

HIPTrack: Visual Tracking with Historical Prompts

Wenrui Cai, Qingjie Liu, Yunhong Wang

arXiv:2311.02072v2cs.CV

TL;DR

Siamese trackers have limited use of historical information, constraining their handling of target appearance changes while often requiring extra computation or retraining. HIPTrack introduces a historical prompt network that combines refined foreground masks with historical visual features to generate prompts, achieving state-of-the-art results on several benchmarks and improving existing trackers as a plug-and-play module.

  • Problem

    Siamese trackers insufficiently use historical information for appearance changes such as deformation and occlusion, while existing solutions can require repetitive training and substantial computation.

  • Method

    HIPTrack encodes refined historical foreground masks and target visual features, then adaptively generates historical prompts for the current search region.

  • Results

    HIPTrack achieves state-of-the-art performance on LaSOT, LaSOText, GOT-10k, and NfS across evaluations on seven datasets.

  • Takeaways & Limitations

    The historical prompt network can improve existing trackers as a plug-and-play component while requiring training of only the historical prompt network and prediction head.

Abstract

from arXiv · show

Trackers that follow Siamese paradigm utilize similarity matching between template and search region features for tracking. Many methods have been explored to enhance tracking performance by incorporating tracking history to better handle scenarios involving target appearance variations such as deformation and occlusion. However, the utilization of historical information in existing methods is insufficient and incomprehensive, which typically requires repetitive training and introduces a large amount of computation. In this paper, we show that by providing a tracker that follows Siamese paradigm with precise and updated historical information, a significant performance improvement can be achieved with completely unchanged parameters. Based on this, we propose a historical prompt network that uses refined historical foreground masks and historical visual features of the target to provide comprehensive and precise prompts for the tracker. We build a novel tracker called HIPTrack based on the historical prompt network, which achieves considerable performance improvements without the need to retrain the entire model. We conduct experiments on seven datasets and experimental results demonstrate that our method surpasses the current state-of-the-art trackers on LaSOT, LaSOText, GOT-10k and NfS. Furthermore, the historical prompt network can seamlessly integrate as a plug-and-play module into existing trackers, providing performance enhancements. The source code is available at https://github.com/WenRuiCai/HIPTrack.

1. Introduction

Siamese trackers struggle when targets change appearance because they rely primarily on template–search similarity matching. HIPTrack addresses this by supplying precise, updated historical prompts without retraining the entire tracker.

  • Siamese trackers can struggle with deformation, scale variation, and occlusion when only the template guides subsequent predictions.
  • Existing history-based methods have drawbacks including inaccurate positions, extra backbones, distractors, and incomplete historical visual features.
  • HIPTrack uses a historical prompt network whose encoder represents historical target positions and visual features, while its decoder generates prompts for the current search region.
  • The method keeps the underlying tracker unchanged while using updated templates and more accurately cropped search regions to improve tracking performance.
  • The historical prompt network and prediction head are the only components trained, reducing training parameters by more than 80% compared with.
  • HIPTrack achieves state-of-the-art performance on LaSOT, LaSOText, GOT-10k, and NfS among evaluations spanning seven datasets.

2. Related Work

Related work extends Siamese tracking with historical information and prompt-learning ideas, but existing approaches face limitations in precision, computation, or historical coverage. HIPTrack frames historical information as prompts that extend similarity matching across time.

  • Siamese trackers perform template–search similarity matching and generally do not directly leverage historical information.
  • Existing history-based trackers may use excessive parameters, imprecise target-position descriptions, distractor-prone auxiliary templates, or multiple-frame backbones.
  • Prompt Learning: Prompt learning adapts pretrained models by modifying inputs or using adapters to change the input–output space.
  • Prompt Learning: In visual tracking, historical prompts can extend Siamese feature-similarity matching into the temporal dimension.
  • Prompt Learning: Figure 2 examines tracker performance as template-update intervals and current-search-region crop factors vary, with larger crop factors indicating coarser cropping.

3. Method

HIPTrack combines a frozen one-stream feature extractor with a historical prompt encoder-decoder that stores target information and generates search-region-specific prompts for prediction.

  • Overall architecture: HIPTrack comprises feature extraction, historical prompt, and prediction head networks, with the prompt network divided into encoder and decoder.The feature extractor uses a ViT initialized from existing trackers, while its parameters remain frozen.
  • Historical Prompt Encoder: The encoder combines refined foreground masks with target visual features to construct historical target features for the decoder memory bank.Masks combine candidate-elimination outputs with predicted bounding-box masks through a bitwise AND operation.
  • Historical Prompt Encoder: The encoder first concatenates the search image and foreground mask, then fuses the resulting position feature with the search-region feature through residual convolutional blocks.The input becomes a four-channel image processed by a lightweight encoder, whose output is concatenated with the search-region feature.
  • Historical Prompt Encoder: The encoder enhances fused features with channel and spatial attention before producing the encoded prompt value.The attention module uses pooled features, an MLP, convolution, sigmoid activations, and channel-wise and pixel-wise multiplication.
  • Historical Prompt Decoder: The decoder stores prompt key-value pairs, retrieves them using the current search-region feature, and adaptively aggregates values into a prompt tailored to the current frame.Keys are compressed search-region features, while the memory bank retains at most T tracked frames and updates every τ frames using FIFO.

4. Experiments

HIPTrack is evaluated across seven datasets and challenging tracking settings, where it generally outperforms state-of-the-art trackers. Ablations and integration tests further examine efficiency, historical-frame sampling, and the historical prompt network’s general applicability.

  • Generalization Ability of Historical Prompt Network: The historical prompt network significantly improves DropTrack, OSTrack, and SiamFC++, demonstrating plug-and-play gains across Transformer-based and explicit Siamese trackers.For SiamFC++, masks are formed without CE masks because of convolutional candidate-elimination limitations, yet richer historical information still improves target representation.
  • Ablation Studies: Five sampled search frames produce optimal tracking performance, while larger samples may require increased model size and training epochs.The scaling capability of the historical prompt network remains untested.
  • Ablation Studies: Candidate elimination is essential for creating refined masks with accurate historical positional information.The ablation compares configurations using candidate-elimination-filtered background patches to construct the refined mask.

5. Conclusion

The paper concludes that precise, updated historical information substantially improves Siamese tracking. HIPTrack encodes historical visual and positional information into adaptive prompts, achieves state-of-the-art performance with few trainable parameters, and can enhance existing trackers as a plug-in module.

  • Conclusion: Precise, updated historical information substantially improves trackers that follow the Siamese paradigm.The conclusion identifies this observation as the basis for the proposed historical prompt network.
  • Conclusion: The historical prompt network encodes tracked-frame visual features and positional information, then adaptively generates prompts for subsequent frames.HIPTrack uses this network as its core module to enhance tracking accuracy.
  • Conclusion: HIPTrack achieves state-of-the-art performance while requiring training for only a small number of parameters.The conclusion presents this efficiency alongside the tracker’s performance result.
  • Conclusion: The historical prompt network can function as a plug-and-play component that improves current trackers.The conclusion states that this integration extends beyond HIPTrack itself.

6. Further Analyses

The ablations examine the historical prompt encoder and decoder, showing that encoder capacity, attention branches, memory size, and update frequency affect tracking performance and efficiency.

  • 6.1. Ablation Studies on Historical Prompt Encoder: Replacing the first three ResNet-18 stages in encoder Φ with one convolutional layer degrades tracking performance, indicating that stronger initial encoding is beneficial.The encoder processes a 4-channel image tensor into feature map F.
  • 6.1. Ablation Studies on Historical Prompt Encoder: Removing either channel attention or spatial attention reduces tracking accuracy, so both branches provide positive benefits.
  • 6.2. Ablation Studies on Historical Prompt Decoder: Increasing memory bank size improves performance, while preserving more accurate early tracking results yields a slight additional gain.The selected memory bank size is 150, without careful tuning.
  • 6.2. Ablation Studies on Historical Prompt Decoder: Update intervals of 5 or 10 provide negligible gains and reduce efficiency, whereas an interval of 30 lowers performance; the chosen interval is 20.Shorter intervals require more frequent calls to the historical prompt encoder.

7. More Detailed Results in Different Attribute Scenes on LaSOT

HIPTrack is compared with leading trackers across LaSOT attribute subsets and achieves stronger performance in challenging appearance-variation scenarios, including occlusion, motion blur, and scale variation.

  • Attribute-scene comparisons: HIPTrack outperforms the compared trackers across all special-attribute subsets in the LaSOT test split.The comparisons cover success curves across multiple attribute scenes.
  • Attribute-scene comparisons: +1.5%, +2.7%, +1.4%, and +1.1% AUC over the second-best method are reported for partial occlusion, full occlusion, motion blur, and scale variation, respectively.
  • Attribute-scene comparisons: HIPTrack improves out-of-view tracking by +1.1% AUC over the second-best method and achieves the highest success and precision performance across the full LaSOT test split.

8. More Qualitative Results

Qualitative analyses show how HIPTrack constructs refined foreground masks and adaptively uses historical memory, improving tracking under deformation, occlusion, and scale variation.

  • Qualitative tracker comparisons: Figure 8 compares HIPTrack, GRM, and SeqTrack on videos involving target deformation, occlusion, and scale variation.The visual comparison focuses on challenging LaSOT test-set segments.
  • Refined foreground masks: Refined foreground masks combine predicted bounding-box masks with candidate-elimination masks through a bitwise AND operation.The resulting mask filters most background regions and more precisely depicts target position.
  • Memory-bank attention: When recent target appearance is stable, the decoder attends more to the latest memory frame; after drastic appearance changes, it shifts attention toward earlier memories.This adaptive selection is associated with improved prediction accuracy.
  • Memory-bank attention: Under severe deformation, the decoder attends to target boundary regions in historical memories, improving boundary-prediction precision.
Loading 2311.02072v2…