PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Object Modeling for Visual Tracking

Yidong Cai, Jie Liu, Jie Tang, Gangshan Wu

arXiv:2308.05140v1cs.CV

TL;DR

Visual tracking needs robust target representations despite distractors, deformation, and appearance variation, while separate and interactive template learning each have limitations. ROMTrack combines inherent and hybrid template streams with variation tokens, and reports state-of-the-art performance across multiple benchmarks. Its benchmark results span both long-term and short-term tracking settings.

  • Problem

    Separate template learning lacks template-search communication, while interactive learning can introduce distractors from cluttered search regions, limiting robust target-oriented feature extraction.

  • Method

    ROMTrack jointly models inherent and hybrid templates and integrates variation tokens into attention to represent appearance changes and object deformation.

  • Results

    ROMTrack achieves state-of-the-art performance across multiple benchmarks, including 71.4% AUC on LaSOT, 83.6% AUC on TrackingNet, and 59.3% NP on LaSOText.

  • Takeaways & Limitations

    The framework provides robust object modeling for tracking by preserving inherent target information while using hybrid features and appearance context.

Abstract

from arXiv · show

Object modeling has become a core part of recent tracking frameworks. Current popular tackers use Transformer attention to extract the template feature separately or interactively with the search region. However, separate template learning lacks communication between the template and search regions, which brings difficulty in extracting discriminative target-oriented features. On the other hand, interactive template learning produces hybrid template features, which may introduce potential distractors to the template via the cluttered search regions. To enjoy the merits of both methods, we propose a robust object modeling framework for visual tracking (ROMTrack), which simultaneously models the inherent template and the hybrid template features. As a result, harmful distractors can be suppressed by combining the inherent features of target objects with search regions' guidance. Target-related features can also be extracted using the hybrid template, thus resulting in a more robust object modeling framework. To further enhance robustness, we present novel variation tokens to depict the ever-changing appearance of target objects. Variation tokens are adaptable to object deformation and appearance variations, which can boost overall performance with negligible computation. Experiments show that our ROMTrack sets a new state-of-the-art on multiple benchmarks.

1. Introduction

Visual tracking must localize arbitrary targets despite occlusion, deformation, scale changes, and distractors. ROMTrack combines inherent and hybrid template modeling with variation tokens to improve robust target representation.

  • Visual object tracking localizes an arbitrary target in video sequences from its initial status.
  • Separate template modeling lacks template-search communication, whereas interactive modeling can introduce distractors from cluttered search regions.
  • ROMTrack simultaneously models inherent and hybrid templates, using mutual guidance to preserve target information and extract target-oriented features.
  • Variation tokens embed appearance context into attention, addressing object deformation and appearance variations with a lightweight design.
  • ROMTrack reports state-of-the-art performance on six challenging benchmarks: GOT-10k, LaSOT, TrackingNet, LaSOText, OTB100, and NFS30.

2. Related Work

Prior visual trackers evolved from separate CNN feature extraction and correlation toward Transformer-based attention for feature interaction and relation modeling. Recent methods differ in how they combine template and search-region processing.

  • Visual Object Tracking: Early Siamese trackers separately extract template and search-region features with a shared CNN backbone before correlation-based similarity modeling.
  • Visual Object Tracking: TransT replaces lightweight relation modeling with Transformer layers and enables bidirectional template-search information interaction through two-stream attention.
  • Visual Object Tracking: MixFormer uses attention for simultaneous feature extraction and relation modeling while pruning cross-attention from target queries to the search area.
  • Transformer Attention: Transformer attention models spatial features and temporal relations, with ViT, PVT, CVT, and Swin-Transformer demonstrating broad vision-task utility.

3. Method

ROMTrack combines inherent and hybrid template modeling with variation tokens in stacked object encoders to learn robust tracking features. Its design preserves target information, enables template–search interaction, and incorporates appearance changes across frames.

  • 3.1. Overall Architecture: ROMTrack processes inherent-template, hybrid-template, and search-region patches through stacked object encoder layers for robust object modeling.The inputs are linearly projected, position-embedded token sequences before entering the encoders.
  • 3.1. Overall Architecture: The prediction head selects the feature-map position with the highest classification score and derives the target bounding box from predicted offsets and size.Classification uses Gaussian weighted focal loss, while box regression uses L1 and GIoU losses.
  • 3.2. Object Encoder: Variation tokens encode contextual appearance changes from the search region and are generated after each object encoder for subsequent-frame tracking.The previous frame’s hybrid-template tokens are reused as variation tokens for the next frame, with negligible MACs from embedding assignments and lightweight attention.
  • 3.2. Object Encoder: The encoder separately self-attends to the inherent template while using cross-attention to jointly learn hybrid-template and search-region features.This preserves pure template information while enabling mutual feature interaction.
  • 3.2. Object Encoder: Cross-attention combines search-region, hybrid-template, inherent-template, and variation-token features to enhance hybrid-template and search-region representations.The output features aggregate information from inherent template and variation tokens through similarity-weighted attention.

4. Experiments

Experiments evaluate ROMTrack against state-of-the-art trackers on six benchmarks and through ablations of object modeling, variation tokens, template updating, and sampling. ROMTrack achieves strong benchmark performance, while the ablations support contributions from robust object modeling and variation tokens.

  • Comparison with State-of-the-art Trackers: 83.6% AUC and 88.4% NP on TrackingNet outperform MixFormer-22k, while ROMTrack achieves top-ranked 59.3% NP on LaSOText.On LaSOText, it surpasses ToMP by 1.2% and shows strong performance against similar distractors.
  • Comparison with State-of-the-art Trackers: ROMTrack and ROMTrack-384 establish state-of-the-art performance on the additional NFS30 and OTB100 small-scale benchmarks.These results are reported using AUC(%) comparisons for the two small-scale benchmarks.
  • Ablation Study and Analysis: ROMTrack (w/o vt) performs best against separate and hybrid template modeling under the same framework and experimental settings.The comparison isolates the inherent-template and hybrid-template components before adding variation tokens.

5. Conclusions

ROMTrack combines inherent and hybrid template streams to learn robust, discriminative representations, while variation tokens capture appearance changes during tracking.

  • ROMTrack uses two template streams: the inherent template preserves original target features, while the hybrid template learns mixed template-search features.
  • The hybrid template extracts helpful information from the inherent template to form target-oriented features.
  • Variation tokens embed appearance context to improve robustness with negligible computational cost.

A.1. Training Details

ROMTrack training uses a two-stage procedure that first learns the backbone and head, then integrates variation tokens through consecutive sampling.

  • The backbone and prediction head are trained for 300 epochs without variation tokens, followed by 100 epochs merging variation tokens into the architecture.
  • The first stage samples one 256×256 search region and two 128 × 128 template regions from the same sequence.
  • Consecutive sampling uses two search regions and four template regions from one sequence, preserving hybrid features as variation tokens for subsequent training.
  • ROMTrack training uses 4 Tesla V100 GPUs with batch size 64 per GPU, while ROMTrack-384 uses 8 GPUs with batch size 16 per GPU.
  • During inference, the initial template serves as both template streams, while preserved hybrid features provide variation tokens across frames.

B.1. Results on UAV123 Benchmark

On UAV123, ROMTrack outperforms OSTrack-256 and SwinTrack-T-224, while ROMTrack-384 achieves competitive performance with previous state-of-the-art trackers.

  • ROMTrack appreciably outperforms OSTrack-256 and SwinTrack-T-224 on UAV123.
  • ROMTrack-384 achieves competitive performance with previous state-of-the-art trackers on UAV123.

B.2. Results on VOT2020 Benchmark

On VOT2020, ROMTrack-384 achieves top-ranked bounding-box tracking performance, with an EAO score of 0.329 and a 2% margin over ToMP101.

  • VOT2020 contains 60 videos with segmentation masks and emphasizes challenges including fast motion and occlusion.
  • 0.329 EAO is achieved by ROMTrack-384, which surpasses the previous state-of-the-art ToMP101 by 2%.
  • ROMTrack and ROMTrack-384 show excellent Robustness scores on VOT2020.

C. More Analysis

ROMTrack improves tracking accuracy and robustness across LaSOT and LaSOText, including a +0.6% AUC gain over OSTrack-256 (w/o CE) on LaSOT without template updating.

  • +0.6% AUC over OSTrack-256 (w/o CE) demonstrates continued improvement on the challenging LaSOT benchmark without template updating.The authors note that LaSOT’s long-term attribute makes improvement difficult.
  • ROMTrack and ROMTrack-384 improve tracking results in both accuracy and robustness on LaSOT and LaSOText.Figures 4 and 5 provide Success, Precision, and Normalized Precision plots for the two benchmarks.
  • The analysis compares ROMTrack against ablation methods HTM and STM using visualization results.The comparison is introduced as part of the additional analysis section.

D.1. Visualization Results

Visualization results show that ROMTrack progressively enhances target features, suppresses background distractors, and concentrates attention on the target more effectively than HTM and STM.

  • Visualization Results: ROMTrack enhances the search-region object layer by layer through interaction with two template streams and variation tokens.This describes the progressive feature-processing mechanism visible in the visualizations.
  • Visualization Results: ROMTrack suppresses possible background distractors while concentrating on the target object.The attention visualizations attribute the observed behavior to more robust target-focused modeling.
  • Visualization Results: HTM and STM have difficulty distinguishing distractors from targets, whereas ROMTrack locates objects more accurately.The comparison is shown in the ablation visualizations.
  • Visualization Results: Across additional attention maps, ROMTrack concentrates on the target rather than the distractor more effectively than HTM and STM.The results cover various target categories and challenging scenarios.
  • Visualization Results: Figures 4 and 5 report Success, Precision, and Normalized Precision plots for LaSOT and LaSOText, respectively.The figures support comparison of tracking behavior on long-term tracking benchmarks.
  • Visualization Results: Figures 6–10 visualize attention or feature maps across different blocks, with red marking the target or ground-truth box and green marking predictions where specified.The visualizations use sr-to-ht and sr-to-sr attention labels in Figures 6–8.
Loading 2308.05140v1…