PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework

Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, Xilin Chen

arXiv:2203.11991v4cs.CV

TL;DR

Two-stream, two-stage trackers extract template and search-region features separately, limiting target awareness and target-background discrimination. OSTrack unifies feature learning and relation modeling through bidirectional template-search information flow, with early candidate elimination for efficiency. It achieves state-of-the-art performance across multiple benchmarks while balancing accuracy and speed, including 73.7% AO on GOT-10k.

  • Problem

    Two-stream, two-stage tracking separates feature extraction from relation modeling, producing target-unaware features with limited target-background discriminability.

  • Method

    OSTrack uses a one-stream, one-stage architecture that jointly learns features and relations through early bidirectional template-search interaction, plus progressive candidate elimination.

  • Results

    73.7% AO on GOT-10k establishes OSTrack-384 as a new state of the art, while OSTrack achieves strong results across multiple benchmarks and balances performance with inference speed.

  • Takeaways & Limitations

    The unified framework provides target-oriented features and efficient tracking by combining mutual guidance with progressive removal of background candidates.

Abstract

from arXiv · show

The current popular two-stream, two-stage tracking framework extracts the template and the search region features separately and then performs relation modeling, thus the extracted features lack the awareness of the target and have limited target-background discriminability. To tackle the above issue, we propose a novel one-stream tracking (OSTrack) framework that unifies feature learning and relation modeling by bridging the template-search image pairs with bidirectional information flows. In this way, discriminative target-oriented features can be dynamically extracted by mutual guidance. Since no extra heavy relation modeling module is needed and the implementation is highly parallelized, the proposed tracker runs at a fast speed. To further improve the inference efficiency, an in-network candidate early elimination module is proposed based on the strong similarity prior calculated in the one-stream framework. As a unified framework, OSTrack achieves state-of-the-art performance on multiple benchmarks, in particular, it shows impressive results on the one-shot tracking benchmark GOT-10k, i.e., achieving 73.7% AO, improving the existing best result (SwinTrack) by 4.3\%. Besides, our method maintains a good performance-speed trade-off and shows faster convergence. The code and models are available at https://github.com/botaoye/OSTrack.

1 Introduction

OSTrack addresses the target-awareness and performance-speed limitations of two-stream, two-stage tracking with a unified one-stream framework that jointly learns features and models relations. An early candidate elimination module further improves efficiency, while the tracker achieves strong performance across benchmarks.

  • Two-stream, two-stage trackers separately extract template and search-region features, leaving them target-unaware and limiting target-background discriminability.
  • OSTrack bridges template and search-region images early through bidirectional information flow, enabling target-oriented features through mutual guidance.The framework concatenates flattened image pairs and processes them with self-attention layers.
  • The one-stream design avoids additional heavy relation-modeling networks and is highly parallelizable, improving the performance-speed trade-off.
  • An in-network candidate elimination module progressively discards background candidates using early similarity information, reducing inference cost and limiting noisy background effects.
  • OSTrack achieves state-of-the-art performance on multiple benchmarks while maintaining efficient inference and faster convergence than state-of-the-art Transformer trackers.

2 Related Work

Prior tracking pipelines separate feature extraction from relation modeling, often trading efficiency for stronger feature interaction. OSTrack instead unifies both operations in a one-stream pipeline and adaptively removes low-similarity background candidates during inference.

  • Tracking Pipelines: Earlier Siamese and discriminative trackers separately extract template and search-region features before applying lightweight relation modeling.
  • Tracking Pipelines: Bi-directional fusion architectures improve interaction but use heavier structures that slow inference, motivating a more efficient unified design.
  • Tracking Pipelines: OSTrack combines feature extraction and relation modeling in one pipeline, allowing free template-search information flow with minor computation costs.
  • Adaptive Inference: The early candidate elimination module progressively discards search-region candidates with low target similarity to accelerate tracking inference.
  • Adaptive Inference: Unlike classification-oriented adaptive inference methods, OSTrack treats each token as a target candidate and uses self-attention similarity to remove background candidates.

3 Method

OSTrack unifies template–search feature extraction and relation modeling in a one-stream ViT framework, enabling bidirectional target-oriented feature learning. It further uses similarity-based early candidate elimination to reduce computation and suppress background noise.

  • Joint Feature Extraction and Relation Modeling: OSTrack feeds concatenated template and search-region patch embeddings into Transformer encoder layers for joint feature extraction and relation modeling.The inputs are split, flattened, linearly projected, position-embedded, concatenated, and processed together.
  • Joint Feature Extraction and Relation Modeling: Self-attention simultaneously aggregates intra-image features and inter-image relations, allowing bidirectional mutual guidance between template and search-region representations.Cross-image attention measures template–search similarity, while within-image attention performs feature extraction.
  • Early Candidate Elimination: Attention weights from template tokens provide an early similarity prior between the target and search-region candidates.Each template token attends to search-region tokens, whose attention weights estimate candidate similarity; the center template part can serve as a representative target descriptor.
  • Joint Feature Extraction and Relation Modeling: The one-stream design is highly parallelized because concatenated features use self-attention rather than a separate heavy relation-modeling network.Compared with two-stream Transformer fusion trackers, OSTrack directly concatenates projected features before Transformer processing.
  • Early Candidate Elimination: OSTrack progressively eliminates low-similarity background candidates inside selected ViT encoder layers to reduce computation and limit noisy-background effects on feature learning.The module is inserted after multi-head attention, retains top-k candidates, records their original order, and later restores the sequence structure.

4 Experiments

Experiments evaluate OSTrack across seven benchmarks, ablate early candidate elimination and pre-training, and compare against aligned two-stream trackers. Results show strong accuracy, speed, convergence, and target-background discrimination.

  • Comparison with State-of-the-arts: OSTrack is evaluated against state-of-the-art trackers on seven benchmarks, with results reported for four large-scale and three additional datasets.The experiments cover LaSOT, LaSOText, TrackingNet, GOT-10k, NFS, UAV123, and TNL2K.
  • Comparison with State-of-the-arts: 71.1% AUC is achieved by OSTrack-384 on LaSOT, while OSTrack-256 reaches 105.4 FPS, twice SwinTrack-B’s 52 FPS.OSTrack-384 sets a new state-of-the-art on LaSOT after increasing input resolution.
  • Comparison with State-of-the-arts: OSTrack-384 sets the best performance on NFS, UAV123, and TNL2K, while OSTrack-256 and OSTrack-384 surpass SwinTrack-B on TrackingNet AUC.TrackingNet gains are 0.6% and 1.4%, respectively.
  • Ablation Study and Analysis: Adding early candidate elimination reduces OSTrack-256 MACs by 25.9%, increases tracking speed by 13.2%, and raises LaSOT AUC by 0.4%.The module progressively discards background candidates and slightly improves performance in most reported cases.
  • Ablation Study and Analysis: Visualizations show OSTrack extracts more discriminative target-oriented regions than SwinTrack-aligned, which may lose target information such as heads and helmets.The comparison attributes this difference to early relation modeling between template and search region.

5 Conclusion

The paper concludes that OSTrack unifies feature extraction and relation modeling in a one-stream framework, while early candidate elimination improves tracking efficiency. The resulting tracker balances performance and inference speed.

  • Conclusion: OSTrack combines feature extraction and relation modeling in a Vision Transformer-based one-stream tracking framework.The framework is presented as a departure from the Siamese-like pipeline.
  • Conclusion: The early candidate elimination module progressively discards search-region tokens belonging to background regions and significantly boosts tracking efficiency.Its design uses the early similarity prior available in the one-stream framework.
  • Conclusion: The proposed tracker maintains a good balance between tracking performance and inference speed.The conclusion presents this balance as a central overall property of the method.

A More Implementation Details

OSTrack combines feature extraction and relation modeling in a single ViT backbone, with implementation details covering training inputs, positional embeddings, classification loss, and discriminative-region visualization. Its early candidate elimination module uses similarity-based candidate selection, while discriminative regions are computed from channel-wise activation magnitudes and normalized importance scores.

  • Training Details: OSTrack-256 uses 128 × 128 templates and 256 × 256 search regions, while OSTrack-384 uses 192 × 192 templates and 384 × 384 search regions.The corresponding target bounding-box area ratios are 22 and 42 for OSTrack-256, and 22 and 52 for OSTrack-384.
  • Classification Loss: The tracker uses weighted focal loss for classification, with α = 2 and β = 4.The loss is defined over ground-truth target centers and their low-resolution equivalents using a Gaussian-weighted heatmap formulation.
  • Position Embeddings: Pre-trained ViT positional embeddings are resized with 2D bicubic interpolation to match the template and search-region embeddings before being added to patch embeddings.The interpolation is applied separately to the template and search-region embedding sizes.
  • Model Details: OSTrack combines feature extraction and relation modeling in a single ViT backbone, unlike aligned two-stream trackers that process template and search features separately before relation modeling.The one-stream framework is illustrated in Fig. A1(a), while the aligned two-stream framework uses extra Transformer encoder layers for relation modeling.
  • Discriminative Regions Visualization: Discriminative-region visualization first averages absolute feature activations across channels, then normalizes each pixel's relative importance using the minimum and maximum values.The channel-wise average uses C, the number of channels, and the normalized map is computed from the resulting pixel scores.

B More Ablation Studies

The early candidate elimination analysis examines how template-token selection affects similarity-based background candidate removal. Because the template includes background, token choice can introduce noise into the similarity scores and alter elimination outcomes.

  • Candidate Elimination: The early candidate elimination module ranks target–candidate similarities to identify and discard candidates belonging to background regions.Its purpose is to remove background candidates using the similarity ranking calculated between the target and each candidate.
  • Candidate Elimination: Template background regions introduce noisy information when similarity scores are calculated for candidate elimination.The template is not exclusively target content, so its background can affect the similarity calculation.
  • Candidate Elimination: Different choices of template tokens can influence the candidate elimination results.The analysis therefore studies which template parts are used for similarity calculation.

B.2 Identity Embeddings and Relative Positional Embeddings

The ablations test whether additional embeddings, relation modules, or restricted relation-modeling depth improve OSTrack. The reported evidence instead supports the sufficiency of the one-stream backbone for feature extraction and relation modeling.

  • Identity Embeddings and Relative Positional Embeddings: Adding identity embeddings and relative positional embeddings does not bring performance gain compared with the base OSTrack-256 configuration.The evaluation covers LaSOT, TrackingNet, and GOT10k benchmarks.
  • Additional Relation Modeling: Adding a transformer-based feature fusion module with 4 self-attention layers and 1 cross-attention layer degrades tracking performance.This module was added to further fuse the extracted template and search-region features.
  • Encoder-Layer Ablation: Vanilla OSTrack uses all 12 ViT-Base encoder layers for simultaneous feature extraction and relation modeling.The ablation varies this design by restricting simultaneous processing to the last n layers while using earlier layers only for feature extraction.

B.5 Different Token Drop Rate

The early candidate elimination module trades retained candidates against computational cost and tracking performance. The reported setting keeps ρ = 0.7 because smaller ratios lose information, whereas larger ratios provide limited additional savings.

  • Different Token Drop Rate: Using ρ < 0.7 causes a performance drop on LaSOT because a small keeping ratio may cause significant information loss.Here, ρ denotes the proportion of candidates retained by the early candidate elimination module.
  • Different Token Drop Rate: ρ = 0.7 decreases computational cost with a slight improvement in tracking performance.This setting provides the reported balance between candidate reduction and tracking performance.
  • Different Token Drop Rate: Larger ρ values provide limited computational-cost reduction, while ρ = 1 means that early candidate elimination is not adopted.The comparison is summarized in Table A5.

C Results on VOT2020

On VOT2020, OSTrack achieves strong segmentation-tracking performance without online template updating, exceeding a dynamic-template baseline.

  • 0.518 EAO is achieved by OSTrack-256 on VOT2020.
  • OSTrack-256 outperforms STARK-ST50 with online template updating, which reaches 0.505 EAO.
  • VOT2020 evaluates short-term tracking using target segmentation results.

D Results on ITB

On the ITB benchmark, OSTrack-384 outperforms the previous best tracker by a substantial margin in mIoU.

  • 64.8% mIoU, surpassing STARK by 7.2% on the ITB benchmark.ITB contains 9 representative scenarios and 180 diverse videos.

E More Visualization

Visualizations indicate that OSTrack progressively focuses attention on targets, distinguishes them from distractors, and eliminates background candidates across challenging tracking scenarios.

  • Attention visualization: OSTrack attends to foreground objects early and develops stronger target-distractor discrimination in later ViT layers.The visualization highlights foreground attention at Layer 4 and discriminative power at Layer 12.
  • Candidate elimination: The early candidate elimination module identifies and discards background regions across target categories and challenging scenarios.Examples include target deformation, occlusion, and motion blur.
  • Attention visualization: Attention maps progressively focus on target objects while distinguishing them from distractors.
  • Candidate elimination: Candidate elimination visualizations show discarded tokens across different tracking targets and scenarios.
Loading 2203.11991v4…