PaperlayerView exact revision on arXiv

Source-linked AI summary

Correlation-Aware Deep Tracking

Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, Wenjun Zeng

arXiv:2203.01666v1cs.CV

TL;DR

Visual tracking needs features that remain coherent for changing targets while discriminating against similar distractors, but common Siamese-like features do not fully achieve both. The paper proposes SBT, which deeply embeds cross-image correlation through self-/cross-attention and directly predicts locations from fused search features. Extensive experiments report state-of-the-art real-time performance, while the feature networks also improve existing tracking pipelines.

  • Problem

    Visual tracking requires robustness to target appearance variation and discrimination against similar distractors, while popular Siamese-like features do not fully model both.

  • Method

    SBT deeply fuses template and search features through self- and cross-attention across feature-extraction layers, then directly feeds fused search features to prediction heads.

  • Results

    The method achieves state-of-the-art results at real-time speed and can improve other popular visual tracking and segmentation pipelines.

  • Takeaways & Limitations

    Target-dependent feature networks simplify tracking by removing separate correlation operations and can be integrated into existing tracking pipelines.

Abstract

from arXiv · show

Robustness and discrimination power are two fundamental requirements in visual object tracking. In most tracking paradigms, we find that the features extracted by the popular Siamese-like networks cannot fully discriminatively model the tracked targets and distractor objects, hindering them from simultaneously meeting these two requirements. While most methods focus on designing robust correlation operations, we propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. In contrast to the Siamese-like feature extraction, our network deeply embeds cross-image feature correlation in multiple layers of the feature network. By extensively matching the features of the two images through multiple layers, it is able to suppress non-target features, resulting in instance-varying feature extraction. The output features of the search image can be directly used for predicting target locations without extra correlation step. Moreover, our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods. Extensive experiments show our method achieves the state-of-the-art results while running at real-time. Our feature networks also can be applied to existing tracking pipelines seamlessly to raise the tracking performance. Code will be available.

1. Introduction

Visual object tracking must preserve target identity under appearance changes while rejecting visually similar distractors. The paper addresses this tension with target-dependent, deeply fused feature extraction in Single Branch Transformer (SBT).

  • Visual tracking requires both robustness to target appearance variation and discrimination against similar background distractors.
  • Existing approaches mainly improve Siamese-like feature embeddings or correlation operations, but their feature networks can face a target-distractor dilemma.
  • SBT deeply interacts template and search features during extraction, using cross-attention to filter irrelevant features and self-attention to enrich representations.
  • The target-dependent extraction produces asymmetric, instance-varying search features that separate targets from similar distractors while preserving target coherence.
  • SBT removes a separate correlation stage by directly sending fused search features to prediction heads, and the framework improves existing tracking pipelines.
  • The paper systematically studies SBT experimentally and theoretically, then summarizes general principles for subsequent tracking work.

2. Related Work

Prior tracking methods rely on Siamese, DCF, or Transformer-based designs, each leaving limitations in feature discrimination, optimization, or training. SBT instead embeds target-aware correspondence into feature extraction.

  • Siamese trackers achieve strong performance but shallow offline target matching can lack discrimination against distractors.
  • DCF trackers use online least-squares target modeling but remain sensitive to handcrafted optimization and feature quality in challenging scenarios.
  • SBT differs from these paradigms by embedding correlation within Cross-Attention layers rather than using a standalone correlation module.
  • Its EoC blocks combine self-attention within images and cross-attention across images, enabling hierarchical target-dependent feature fusion.
  • Transformer-based trackers fuse features through long-range modeling and avoid online learning, but vision-task initialization can impose enormous training costs.

3. Architecture

The Single Branch Transformer uses Extract-or-Correlation blocks to fuse template and search features through self- and cross-attention, then directly predicts target location from the search representation. Its architecture also specifies efficient attention, positional encoding, and prediction heads.

  • Architecture: SBT processes template and search images through stacked Extract-or-Correlation blocks that alternately fuse within-image and cross-image features.EoC-SA performs self-attention within one image, whereas EoC-CA performs cross-attention between the two images.
  • Attention blocks: Self-attention enriches features within each image, while cross-attention embeds hierarchical correspondence between template and search features.The resulting prediction naturally contains hierarchical feature-correlation information.
  • Architecture comparison: Table 1 evaluates attention, positional encoding, patch embedding, dimensions, block counts, feature-map stride, parameter counts, and FLOPs under specified common settings.The baseline settings include training from scratch, interleaved EoC-SA/EoC-CA in the third stage, and 128 × 128 and 256 × 256 template/search images.
  • Attention blocks: SBT uses Spatial-Reduction Global attention for both self-attention and cross-attention because local cross-attention cannot cover targets appearing anywhere in the search image.Spatial reduction lowers the computational overhead compared with global attention over all tokens.
  • Position encoding: Conditional positional encoding adds a 3 × 3 depth-wise convolution before GELU in the MLP.This encoding is generated dynamically by convolutional layers.
  • Direct prediction: The search feature from SBT feeds classification and regression heads directly, without an additional correlation operation.The heads estimate the target’s location and shape, and are implemented by stacked Mix-MLP Blocks modeling spatial and channel dependencies.

4. Empirical Study of SBT Instantiations

The empirical studies identify design choices that improve SBT tracking, especially hierarchical structure, earlier and more frequent cross-attention, and interleaved self-/cross-attention. These gains trade off against inference speed and depend on suitable model scale and feature resolution.

  • Ablations: Hierarchical structure improves performance over a single-stage design through multi-scale representations, while position encoding has limited impact.Conditional position encoding surpasses relative position encoding by 0.4 points, whereas convolutional patch embedding is more practical and expressive than hand-crafted patch merging.
  • Cross-attention placement: More EoC-CA blocks consistently improve performance, and earlier cross-attention yields gains of 4.9 and 6.5 points in matched comparisons.The authors attribute the benefit to more comprehensive cross-attention between template and search branches and to target-dependent features from early cross-attention.
  • Cross-attention placement: Interleaving EoC-SA and EoC-CA performs better than separating them, with the best reported performance of 67.2% at an interval of 1.Increasing the interval from 1 to 2 reduces performance from 61.1% to 59.2% in the cited comparison.
  • Model variants: Three-stage models outperform four-stage models at comparable complexity, while excessive shallow-stage parameters, larger stride, and imbalanced channel dimensions harm performance.A network stride of 16 reduces performance by 11.3 points, and increasing head number slightly improves performance but decreases speed.
  • Ablations: Early cross-attention in shallow stages lowers inference speed because it destroys one-shot inference, while shallow self-attention blocks act as buffers.The study finds that early-cross designs in stages 1 and 2 provide little improvement in the tested comparisons.
  • Pre-training: Pre-training with more weights benefits SBT and produces much faster convergence than Transformer-based trackers such as TransT and STARK.The comparison is reported for SBT tracker variants in the empirical study.

5. Single Branch Transformer Driven Tracking

SBT is analyzed as a target-dependent, hierarchical feature network that deeply fuses template and search features through attention blocks. Its variants are evaluated as backbones in established trackers across standard tracking benchmarks.

  • Theoretical analysis: Cross-attention is mathematically equivalent to two dynamic convolutions and a SoftMax layer, whereas depth-wise or pixel-wise correlation corresponds to one dynamic convolution.The dynamic filters are generated from the interacting template and search features.
  • Model variants: The four SBT scales differ in convolution, MLP, attention-head, down-sampling, and stage settings, with speed measured on a single Tesla V100 GPU.The table specifies EoC-CA blocks in the third stage and defines the notation for each component.
  • Hierarchical utilization: Serial multi-level correlation intrinsically incorporates hierarchical feature information into the prediction result instead of relying on hand-crafted layer-wise aggregation.The cited three-level example combines shallow, intermediate, and deep features through successive EoC-CA blocks.
  • Evaluation: The evaluation includes comparisons on GOT-10k, LaSOT, and VOT2020, alongside success-plot comparisons on OTB-100 and GOT-10k.The VOT2020 results use AlphaRefine to generate benchmark masks.
  • Tracker integration: Replacing the backbones of SiamFCpp, DiMP, STARK, and STM with SBT produces the CAT tracker variants SiamFCpp-CA, DiMP-CA, STARK-CA, and STM-CA.The paper presents this integration to demonstrate that the feature network can be used across typical tracking pipelines.

6. Experiments

Experiments evaluate SBT’s training setup, benchmark performance, feature behavior, and transfer to other tracking pipelines. Results show strong performance on unseen objects, competitive or state-of-the-art results across benchmarks, and benefits for existing trackers.

  • Benchmark Results: On GOT-10K, SBT-base and SBT-large outperform top-performing trackers, while SBT-light and SBT-small remain competitive with much smaller size.The benchmark uses zero overlap between training and testing object classes and follows the official policy without extra training data.
  • Benchmark Results: SBT-base achieves an EAO of 0.515 on VOT2020 and outperforms recent Transformer-based methods on LaSOT without online update.SBT-large reaches 0.530 EAO-adjacent performance relative to the VOT2020 winner RPT, while SBT-base and SBT-large achieve state-of-the-art OTB results.
  • Transferability: Correlation-aware features improve other tracking pipelines with comparable model size and less computation burden, validating the feature network’s generality.The improvements are reported for box-level tracking pipelines and are also illustrated through CATs comparisons on GOT-10K and DAVIS17.
  • Transferability: STM-CA improves STM by 3.6% in J/F on DAVIS17, showing that discriminative embeddings can benefit pixel-level video object segmentation.The result extends evaluation beyond box-level tracking to multi-object video object segmentation.
  • Feature Analysis: Correlation-embedded SBT improves tracking across correlation settings and outperforms CNN-based or attention-based alternatives in the reported comparisons.The reported comparisons include 65.9% versus 60.1%, 65.0% versus 61.5%, and 39.2% versus 30.3%.
  • Feature Analysis: Correlation-embedded features preserve spatial information and better filter distractors than Siamese correlation or ResNet-50 settings in hard OTB videos.The comparison uses target localization, T-SNE, classification maps, and average negative-object measurements.
  • Pre-training: More pre-trained blocks correlate with better tracking performance, and SBT converges faster than recent Transformer-based trackers.The study also examines model variants and pre-training consistency.

7. Conclusion

The paper concludes that SBT is a target-dependent feature network for visual object tracking that simplifies the pipeline and supports fast convergence. It also reports that SBT improves other VOT and VOS trackers as a dynamic feature network.

  • Conclusion: SBT introduces a target-dependent feature network for visual object tracking and directly simplifies the tracking pipeline.The paper identifies this as the first such network for VOT.
  • Conclusion: SBT converges much faster than recent Transformer-based trackers while achieving state-of-the-art results.The conclusion summarizes extensive experimental support for these claims.
  • Conclusion: Four SBT network versions improve other popular visual object tracking and video object segmentation trackers.The conclusion presents the feature network as applicable across VOT and VOS pipelines.
  • Supplementary Material: The supplementary material provides implementation details, additional exploration studies, and visualizations.These materials are organized into Sections A, B, and C.

A.1. Training Details

The supplementary training details describe ImageNet pre-training, tracking fine-tuning, inference, and integration of SBT features into Siamese, DCF, and Transformer-based trackers. Additional experiments compare computation and model variants on GOT-10K.

  • Training Details: ImageNet pre-training uses DeiT-style augmentations, AdamW optimization, cosine learning-rate decay, and 100 training epochs.The reported augmentations include random cropping, flipping, label smoothing, mixup, CutMix, and random erasing.
  • Inference Details: During inference, classification and regression heads produce response maps, and the maximum classification confidence selects the final target prediction.The selected bounding-box size comes from the regression response map.
  • Pipeline Integration: SBT replaces feature extractors in SiamFCpp, DiMP, and STARK to provide correlation-aware search features.The modified SBT features are used in existing Siamese, DCF, and Transformer-based tracking pipelines.
  • Evaluation: The GOT-10K comparison reports model accuracy alongside model size and computation FLOPs.DCF FLOPs are omitted because those methods use online learning.
  • Exploration Studies: The supplementary experiments explore the effects of SBT model variants on tracking performance.These studies include channel dimension and block-pattern configurations.

B.1. Position Pattern.

Position-pattern ablations show that the placement and ordering of EoC-CA and EoC-SA blocks affect tracking performance. Early cross-attention can help target-dependent feature formation, but shallow early-cross placements reduce speed and may disrupt one-shot inference.

  • Position Pattern: Moving the first EoC-CA block earlier improves performance when the final two correlation operations remain fixed at blocks 9 and 10.The reported results are 63.2% for A3, 64.6% for A4, and 65.7% for A5.
  • Position Pattern: The interleaved EoC-SA/EoC-CA pattern outperforms the Siamese-style and earlier-cross designs in the reported comparison.The compared results are 65.0% for A1, 61.0% for A2, and 63.2% for A3.
  • Design Rationale: Early cross-attention generates target-dependent features, while EoC-SA refines template and search features after correlation.This ordering is presented as the basis for choosing an interleaved SBT design.

B.2. Block Number.

The ablations examine how block allocation and channel dimension across stages affect SBT tracking performance, model size, speed, and implementation flexibility.

  • Block Number: 64.0% of B2 achieves the best tracking performance with 4/4 blocks in the shallow stage and 14 blocks in the deep stage.The result indicates that a moderate shallow-stage depth and more blocks in the deep stage are practical.
  • Block Number: Stacking different numbers of EoC blocks supports practical adjustment to speed and model-size requirements.The architecture can be adapted by changing the number of EoC blocks.
  • Channel Dimension: Larger channel dimensions improve tracking performance but increase SBT model size, with C1, C3, and C7 reaching 63.6%, 63.7%, and 63.6%, respectively.The channel dimension is therefore an important model-size factor alongside performance.

C. Visualization Result

The visualizations show that SBT progressively suppresses distractors and separates target features, producing more centralized responses and better tracking in challenging cases. They also identify heavy occlusion and out-of-view targets as limitations.

  • Attention Maps: Cross-attention suppresses non-target search features layer by layer, while the final attention becomes uniform before prediction.The shallow stage focuses on background context, and the resulting attention block discriminates distractors to some extent.
  • Classification Maps: SBT makes finer-grained distinctions among visually similar distractors, including a green-clothed distractor identified using the target’s white clothing number.In the basketball case, SBT suppresses a white-clothed distractor and succeeds where the other three models fail.
  • Failure Case: The framework struggles with heavy occlusion involving distractors and appearance changes, as well as targets moving out of view.Fast attention computation is also constrained by the limited friendliness of modern scientific computation packages.
  • Feature Visualization: As SBT becomes deeper, target features separate increasingly from background and distractors, unlike target-unaware Siamese features.The Siamese extraction relies on a separate correlation step to discriminate targets from background.
  • Tracking Results: SBT shows stronger discriminative ability and better accuracy than the compared trackers on challenging OTB100 sequences, with more centralized and higher response maps.SiamFCpp-CA is also more discriminative toward distractors and background clutter than original SiamFCpp.
Loading 2203.01666v1…